JP2017146647A - Display device, input device, head-mounted type display device, control method for display device, and program - Google Patents

Abstract

PROBLEM TO BE SOLVED: To easily increase variations of operation and to achieve improvement of operability in a configuration enabling touch operation with respect to an operation surface.SOLUTION: An HMD 100 comprising an image display section 20 having a display area displays a prescribed image by the image display section 20, detects an operation position in a track pad 14 receiving contact operation, moves a display position of the prescribed image according to the operation position detected in the track pad 14, and changes association between the operation position and the display position of the prescribed image when detecting one operation position in the track pad 14 and when detecting a plurality of operation positions in the track pad 14.SELECTED DRAWING: Figure 1

Description

  The present invention relates to a display device, an input device, a head-mounted display device, a display device control method, and a program.

  Conventionally, a device that receives a touch operation is known (for example, see Patent Document 1). The information processing apparatus described in Patent Literature 1 includes a contact finger unit that receives input from an operator's finger, a touch pen, or the like, and is configured to be able to input gesture information. The information processing apparatus can generate a command by reading a gesture and execute an operation corresponding to the command. For example, the information processing apparatus can execute an operation of turning on the power and an operation of turning off the power in accordance with the number of fingers touching the contact finger means.

Japanese Patent Laid-Open No. 9-23004

There is an advantage that various devices can be easily operated with a finger or the like by using a touch operation like the device described in Patent Document 1. On the other hand, in order to increase the types of operations, there has been a tendency for finger movements to become complicated.
The present invention has been made in view of the above circumstances, and an object thereof is to easily increase operation variations and improve operability in a configuration in which a touch operation on an operation surface is possible.

In order to achieve the above object, a display device of the present invention detects an operation surface that accepts a contact operation, a display unit that has a display area for displaying an image, and an operation position on the operation surface, and responds to the detected operation. A control unit that controls display of the display unit, and the control unit displays a predetermined image by the display unit, and moves a display position of the predetermined image according to an operation position of the operation surface, The association between the operation position and the display position of the predetermined image is changed between when one operation position is detected on the operation surface and when a plurality of operation positions are detected.
According to the present invention, the manner of movement of the display position of the predetermined image corresponding to the operation changes depending on the number of operation positions where the contact operation is performed on the operation surface. For this reason, the variation of operation can be increased by changing the number of operation positions, and operativity can be improved.

In the display device according to the aspect of the invention, when the control unit detects two operation positions on the operation surface when moving the display position of the predetermined image according to the operation position on the operation surface, the detection is performed. The display position of the predetermined image is moved to a display position corresponding to an operation position different from the two operation positions.
According to the present invention, by performing the contact operation at the two operation positions, it is possible to increase the variation of the operation for moving the display position of the predetermined image, and it is possible to improve the operability.

According to the present invention, in the display device, the control unit is configured to display the predetermined image at a display position corresponding to an operation position different from the two detected operation positions based on the two operation positions detected on the operation surface. Moving one display position to the display position corresponding to the one new operation position when one new operation position is detected on the operation surface. Features.
According to the present invention, when an operation is performed at a new position in addition to the two operation positions, the display position of the predetermined image can be controlled corresponding to the new operation position. Thereby, the variation of operation which moves the display position of a predetermined image can be increased, and operability can be improved.

According to the present invention, in the display device, the control unit detects an operation at two operation positions on the operation surface after displaying the predetermined image based on one operation position detected on the operation surface. In the case, the display of the predetermined image is stopped.
According to the present invention, it is possible to notify the user that the number of detected operation positions has changed by changing the display state of the predetermined image.

Further, according to the present invention, in the display device, the control unit executes a movement display that moves a display position of the predetermined image in accordance with an operation position of the operation surface, and sets the movement display mode on the operation surface. Switching is performed according to the number of operation positions to be detected.
According to the present invention, the operability can be further improved by switching the moving display mode of the predetermined image.

Further, the present invention is characterized in that, in the display device, the control unit switches a display mode of the predetermined image according to the number of operation positions detected on the operation surface.
According to the present invention, by changing the display mode of a predetermined image, the visibility of the predetermined image suitable for the operation can be ensured, and the operability can be further improved.

In order to achieve the above object, the display device of the present invention detects an operation surface that receives a touch operation, a display unit that has a display area for displaying an image, and an operation position on the operation surface, and detects the operation. A control unit that outputs a position for display control based on the position and displays an image on the display unit based on the position for display control, and the control unit detects one operation position; When a plurality of operation positions are detected, the association between the detected operation position and the display control position is changed.
According to the present invention, the manner of controlling the display position of the predetermined image corresponding to the operation changes depending on the number of operation positions where the touch operation is performed on the operation surface. For this reason, the variation of operation can be increased by changing the number of operation positions, and operativity can be improved.

In the display device according to the aspect of the invention, when the control unit detects a plurality of operation positions on the operation surface, the display control position corresponds to an operation position different from the detected operation positions. Is output.
According to the present invention, by performing the contact operation at a plurality of operation positions, it is possible to increase the variation of the operation for moving the display position of the predetermined image, and it is possible to improve the operability.

In the display device according to the aspect of the invention, the control unit obtains a virtual operation position from a plurality of operation positions detected on the operation surface, and outputs the display control position corresponding to the virtual operation position. It is characterized by doing.
According to the present invention, it is possible to appropriately move the display position in response to contact operations at a plurality of operation positions.

According to the present invention, in the display device, the control unit is configured to display the display control position corresponding to an operation position different from the two detected operation positions based on the two operation positions detected on the operation surface. After that, when one new operation position is detected on the operation surface, the display control position corresponding to the one new operation position is output.
According to the present invention, it is possible to increase variations of operations for moving the display position of a predetermined image in response to contact operations at a plurality of operation positions.

In the display device according to the present invention, the control unit outputs information specifying the display control position by at least one of a movement direction and a movement amount from the output display position. It is characterized by.
According to the present invention, it is not necessary to strictly associate the sizes and shapes of the display area and the operation surface, and the present invention can be applied to operation surfaces having various shapes.

In the display device according to the aspect of the invention, the display unit is mounted on a user's head, and the control unit detects an operation position on the operation surface configured separately from the display unit. It is characterized by this.
According to the present invention, when the user who wears the display device on the head performs an operation on the operation surface, the mode of movement of the display position of the predetermined image can be changed depending on the number of operation positions. For this reason, the variation of operation can be increased by changing the number of operation positions, and operativity can be improved. For example, even when it is difficult to visually recognize the operation surface, the moving state of the display position can be arbitrarily operated.

In order to achieve the above object, the input device of the present invention receives a contact operation on the operation surface, detects an operation position on the operation surface, and performs processing corresponding to the operation position based on the detected operation position. An operation information generation unit configured to generate position information, wherein the operation information generation unit detects the operation position and the operation position when detecting one operation position on the operation surface and when detecting a plurality of operation positions. The association with the processing position is changed.
According to the present invention, it is possible to change the contents of processing to be executed in response to an operation, depending on the number of operation positions where a contact operation is performed on the operation surface. For this reason, the variation of operation can be increased by changing the number of operation positions, and operativity can be improved.

In order to achieve the above object, the present invention provides a head-mounted display device including a display unit that transmits external light, and detects an operation surface that receives a contact operation and an operation position on the operation surface. A control unit that controls display of the display unit according to the detected operation, and the control unit displays a predetermined image on the display unit, and the display position of the predetermined image is set to the operation position of the operation surface. And the transmittance of the external light with respect to the predetermined image is controlled in accordance with the operation position detected on the operation surface.
According to the present invention, it is possible to display an image corresponding to an operation position where a contact operation is performed on the operation surface, and to change the transmittance of the image. For this reason, the appearance of the image displayed by the head-mounted display device with respect to the external light can be changed corresponding to the operation, and the operability can be improved.

Further, the present invention is characterized in that, in the display device, the control unit changes a display mode of the predetermined image corresponding to an operation position detected on the operation surface.
According to the present invention, it is possible to change the display mode such as the shape of an image to be displayed in addition to the appearance of the display image with respect to external light, and further improve the operability.

In order to achieve the above object, the display device control method of the present invention controls a display device including a display unit having a display area for displaying an image, displays a predetermined image on the display unit, and performs contact. Detecting an operation position on an operation surface that receives an operation, moving a display position of the predetermined image according to an operation position detected on the operation surface, and detecting one operation position on the operation surface; The association between the operation position and the display position of the predetermined image is changed when a plurality of operation positions on the operation surface are detected.
According to the present invention, the manner of movement of the display position of the predetermined image corresponding to the operation changes depending on the number of operation positions where the contact operation is performed on the operation surface. For this reason, the variation of operation can be increased by changing the number of operation positions, and operativity can be improved.

In order to achieve the above object, the present invention is a program executable by a computer that controls a display device including a display unit having a display area for displaying an image, and displays a predetermined image on the display unit. Detecting the operation position on the operation surface that accepts the contact operation, moving the display position of the predetermined image according to the operation position detected on the operation surface, and detecting one operation position on the operation surface; The program for changing the association between the operation position and the display position of the predetermined image when a plurality of operation positions on the operation surface are detected.
According to the present invention, the manner of movement of the display position of the predetermined image corresponding to the operation changes depending on the number of operation positions where the contact operation is performed on the operation surface. For this reason, the variation of operation can be increased by changing the number of operation positions, and operativity can be improved.

  The present invention can also be realized in various forms other than the above-described display device, display device control method, and program. For example, the present invention can be realized in the form of a recording medium that records the program, a server device that distributes the program, a transmission medium that transmits the program, a data signal that embodies the program in a carrier wave, and the like.

Explanatory drawing which shows the external appearance structure of HMD. The figure which shows the structure of the optical system of an image display part. Explanatory drawing which shows a response | compatibility with an image display part and an imaging range. The block diagram of each part which comprises HMD. The block diagram of a control part and a memory | storage part. The flowchart which shows operation | movement of HMD. The flowchart which shows operation | movement of HMD. Explanatory drawing which shows the change of the display corresponding to operation of HMD. Explanatory drawing which shows the change of the display corresponding to operation of HMD. Explanatory drawing which shows the change of the display corresponding to operation of HMD. Explanatory drawing which shows the change of the display corresponding to operation of HMD. Explanatory drawing which shows the change of the display corresponding to operation of HMD. Explanatory drawing which shows the change of the display corresponding to operation of HMD. Explanatory drawing which shows the change of the display corresponding to operation of HMD.

FIG. 1 is an explanatory diagram showing an external configuration of an HMD (Head Mounted Display) 100 according to an embodiment to which the present invention is applied.
The HMD 100 is a display device that includes an image display unit 20 (display unit) that allows a user to visually recognize a virtual image while being mounted on the user's head, and a control device 10 that controls the image display unit 20. In addition, the control device 10 (input device) includes various buttons, switches, and track pads 14 (operation surface) that accept user operations, and functions as a controller for the user to operate the HMD 100.

The image display unit 20 is a wearing body that is worn on the user's head, and has a glasses shape in the present embodiment. The image display unit 20 includes a right display unit 22, a left display unit 24, a right light guide plate 26, and a left light guide plate 28 in a main body having a right holding unit 21, a left holding unit 23, and a front frame 27. .
Each of the right holding unit 21 and the left holding unit 23 extends rearward from both ends of the front frame 27, and holds the image display unit 20 on the user's head like a temple of glasses. Here, of both ends of the front frame 27, an end located on the right side of the user in the mounted state of the image display unit 20 is defined as an end ER, and an end located on the left side of the user is defined as an end EL. To do. The right holding unit 21 extends from the end ER of the front frame 27 to a position corresponding to the right side of the user when the image display unit 20 is mounted. The left holding unit 23 is provided to extend from the end EL to a position corresponding to the left side of the user when the image display unit 20 is mounted.

  The right light guide plate 26 and the left light guide plate 28 are provided on the front frame 27. The right light guide plate 26 is positioned in front of the right eye of the user when the image display unit 20 is mounted, and causes the right eye to visually recognize the image. The left light guide plate 28 is positioned in front of the left eye of the user when the image display unit 20 is mounted, and causes the left eye to visually recognize the image.

  The front frame 27 has a shape in which one end of the right light guide plate 26 and one end of the left light guide plate 28 are connected to each other, and this connection position is in a wearing state in which the user wears the image display unit 20. Corresponding to the eyebrows. The front frame 27 may be provided with a nose pad portion that comes into contact with the user's nose when the image display unit 20 is mounted at a connection position between the right light guide plate 26 and the left light guide plate 28. In this case, the image display unit 20 can be held on the user's head by the nose pad, the right holding unit 21 and the left holding unit 23. Further, a belt (not shown) that contacts the back of the user when the image display unit 20 is mounted may be connected to the right holding unit 21 and the left holding unit 23. In this case, the image display unit 20 is used by the belt. Can be held on the person's head.

  The right display unit 22 is a unit related to display of an image by the right light guide plate 26 and is provided in the right holding unit 21 and is located in the vicinity of the right side of the user in the mounted state. The left display unit 24 is a unit related to display of an image by the left light guide plate 28 and is provided in the left holding unit 23 and is located in the vicinity of the left head of the user in the mounted state. The right display unit 22 and the left display unit 24 are also collectively referred to simply as a “display driving unit”.

The right light guide plate 26 and the left light guide plate 28 of the present embodiment are optical units formed of a light-transmitting resin or the like, and are, for example, prisms, and output image light output from the right display unit 22 and the left display unit 24. Lead to the eyes of the user.
A light control plate (not shown) may be provided on the surfaces of the right light guide plate 26 and the left light guide plate 28. The light control plate is an optical element on a thin plate having different transmittance depending on the wavelength region of light, and functions as a so-called wavelength filter. The light control board is arrange | positioned so that the front side of the front frame 27 which is the opposite side to the user's eyes side may be covered, for example. By appropriately selecting the optical characteristics of the light control plate, the transmittance of light in an arbitrary wavelength region such as visible light, infrared light, and ultraviolet light can be adjusted. The amount of external light incident on the light plate 28 and transmitted through the right light guide plate 26 and the left light guide plate 28 can be adjusted.

  The image display unit 20 guides the image light generated by the right display unit 22 and the left display unit 24 to the right light guide plate 26 and the left light guide plate 28, respectively, and makes the user visually recognize the virtual image by using the image light, thereby Is displayed. When the external light enters the user's eyes through the right light guide plate 26 and the left light guide plate 28 from the front of the user, the image light and the external light constituting the virtual image enter the user's eyes. In other words, the visibility of the virtual image is affected by the intensity of external light. For this reason, for example, by attaching a light control plate to the front frame 27 and appropriately selecting or adjusting the optical characteristics of the light control plate, the visibility of the virtual image can be adjusted. In a typical example, it is possible to use a light control plate having a light transmittance that allows a user wearing the HMD 100 to visually recognize at least the outside scenery. In addition, when the dimming plate is used, it is possible to protect the right light guide plate 26 and the left light guide plate 28 and to suppress the damage to the right light guide plate 26 and the left light guide plate 28, the adhesion of dirt, and the like. The light control plate may be detachable with respect to the front frame 27 or each of the right light guide plate 26 and the left light guide plate 28, or may be mounted by replacing a plurality of types of light control plates. May be omitted.

  The camera 61 is disposed on the front frame 27 of the image display unit 20. The camera 61 desirably captures an outside scene direction viewed by the user wearing the image display unit 20, and external light transmitted through the right light guide plate 26 and the left light guide plate 28 on the front surface of the front frame 27. It is provided at a position that does not block. In the example of FIG. 1, the camera 61 is disposed on the end ER side of the front frame 27. The camera 61 may be disposed on the end EL side, or may be disposed at a connection portion between the right light guide plate 26 and the left light guide plate 28.

The camera 61 is a digital camera that includes an imaging element such as a CCD or CMOS, an imaging lens, and the like. The camera 61 of the present embodiment is a monocular camera, but may be a stereo camera. The camera 61 images at least a part of the outside scene (real space) in the front side direction of the HMD 100, in other words, the user's viewing direction when the HMD 100 is worn. In another expression, the camera 61 captures a range or direction that overlaps the user's field of view, and captures the direction in which the user gazes. Although the angle of view of the camera 61 can be set as appropriate, in the present embodiment, as will be described later, the camera 61 includes the outside world visually recognized through the right light guide plate 26 and the left light guide plate 28. More preferably, the imaging range of the camera 61 is set so that the entire field of view of the user visible through the right light guide plate 26 and the left light guide plate 28 can be imaged.
The camera 61 performs imaging in accordance with the control of the imaging control unit 149 included in the control unit 150 (FIG. 5), and outputs captured image data to the imaging control unit 149.

  The HMD 100 may include a distance sensor (not shown) that detects a distance to a measurement object positioned in a preset measurement direction. The distance sensor can be disposed, for example, at a connection portion between the right light guide plate 26 and the left light guide plate 28 in the front frame 27. In this case, when the image display unit 20 is mounted, the position of the distance sensor is substantially in the middle between the user's eyes in the horizontal direction and above the user's eyes in the vertical direction. The measurement direction of the distance sensor can be, for example, the front side direction of the front frame 27, in other words, the direction overlapping the imaging direction of the camera 61. The distance sensor can be configured to include, for example, a light source such as an LED or a laser diode, and a light receiving unit that receives reflected light that is reflected from the light to be measured. The distance sensor may perform a triangulation process or a distance measurement process based on a time difference in accordance with the control of the control unit 150. The distance sensor may be configured to include a sound source that emits ultrasonic waves and a detection unit that receives ultrasonic waves reflected by the measurement object. In this case, the distance sensor may perform the distance measurement process based on the time difference until the reflection of the ultrasonic wave according to the control of the control unit 150.

FIG. 2 is a principal plan view showing the configuration of the optical system provided in the image display unit 20. FIG. 2 shows the user's left eye LE and right eye RE for explanation.
As shown in FIG. 2, the right display unit 22 and the left display unit 24 are configured symmetrically. As a configuration for allowing the user's right eye RE to visually recognize an image, the right display unit 22 includes an OLED (Organic Light Emitting Diode) unit 221 that emits image light, a lens group that guides image light L emitted from the OLED unit 221, and the like. Right optical system 251. The image light L is guided to the right light guide plate 26 by the right optical system 251.

The OLED unit 221 includes an OLED panel 223 and an OLED drive circuit 225 that drives the OLED panel 223. The OLED panel 223 is a self-luminous display configured by arranging light emitting elements that emit light of R (red), G (green), and B (blue) light by organic electroluminescence in a matrix. It is a panel. The OLED panel 223 includes a plurality of pixels, each of which includes one R, G, and B element, and forms an image with pixels arranged in a matrix. The OLED drive circuit 225 selects a light emitting element included in the OLED panel 223 and energizes the light emitting element under the control of the control unit 150 (FIG. 5), and causes the light emitting element of the OLED panel 223 to emit light. The OLED drive circuit 225 is fixed to the back surface of the OLED panel 223, that is, the back surface of the light emitting surface by bonding or the like. The OLED drive circuit 225 may be formed of a semiconductor device that drives the OLED panel 223, for example, and may be mounted on a substrate (not shown) fixed to the back surface of the OLED panel 223. A temperature sensor 217 is mounted on this substrate.
Note that the OLED panel 223 may have a configuration in which light emitting elements that emit white light are arranged in a matrix, and color filters corresponding to R, G, and B colors are stacked. Further, an OLED panel 223 having a WRGB configuration provided with a light emitting element that emits W (white) light in addition to the light emitting elements that respectively emit R, G, and B color light may be used.

  The right optical system 251 includes a collimator lens that converts the image light L emitted from the OLED panel 223 into a parallel light flux. The image light L converted into a parallel light beam by the collimator lens enters the right light guide plate 26. A plurality of reflecting surfaces that reflect the image light L are formed in an optical path that guides light inside the right light guide plate 26. The image light L is guided to the right eye RE side through a plurality of reflections inside the right light guide plate 26. A half mirror 261 (reflection surface) located in front of the right eye RE is formed on the right light guide plate 26. The image light L is reflected by the half mirror 261 and emitted from the right light guide plate 26 toward the right eye RE. The image light L forms an image on the retina of the right eye RE, and allows the user to visually recognize the image.

  In addition, as a configuration for allowing the user's left eye LE to visually recognize an image, the left display unit 24 includes a left optical system including an OLED unit 241 that emits image light, a lens group that guides the image light L emitted from the OLED unit 241, and the like. 252. The image light L is guided to the left light guide plate 28 by the left optical system 252.

  The OLED unit 241 includes an OLED panel 243 and an OLED drive circuit 245 that drives the OLED panel 243. The OLED panel 243 is a self-luminous display panel configured similarly to the OLED panel 223. The OLED driving circuit 245 causes the light emitting elements of the OLED panel 243 to emit light by selecting a light emitting element included in the OLED panel 243 and energizing the light emitting element according to the control of the control unit 150 (FIG. 5). The OLED drive circuit 245 is fixed to the back surface of the OLED panel 243, that is, the back surface of the light emitting surface by bonding or the like. The OLED drive circuit 245 may be configured by a semiconductor device that drives the OLED panel 243, for example, and may be mounted on a substrate (not shown) fixed to the back surface of the OLED panel 243. A temperature sensor 239 is mounted on this substrate.

  The left optical system 252 includes a collimating lens that converts the image light L emitted from the OLED panel 243 into a light beam in a parallel state. The image light L converted into a parallel light beam by the collimator lens enters the left light guide plate 28. The left light guide plate 28 is an optical element formed with a plurality of reflecting surfaces that reflect the image light L, and is, for example, a prism. The image light L is guided to the left eye LE side through a plurality of reflections inside the left light guide plate 28. The left light guide plate 28 is formed with a half mirror 281 (reflection surface) located in front of the left eye LE. The image light L is reflected by the half mirror 281 and emitted from the left light guide plate 28 toward the left eye LE. The image light L forms an image on the retina of the left eye LE, and allows the user to visually recognize the image.

According to this configuration, the HMD 100 functions as a see-through display device. That is, the image light L reflected by the half mirror 261 and the external light OL transmitted through the right light guide plate 26 are incident on the user's right eye RE. Further, the image light L reflected by the half mirror 281 and the external light OL transmitted through the half mirror 281 are incident on the left eye LE. In this way, the HMD 100 superimposes the image light L of the internally processed image and the external light OL so as to enter the user's eyes, and for the user, the outside light can be seen through the right light guide plate 26 and the left light guide plate 28. The image by the image light L is visually recognized over the outside scene.
The half mirrors 261 and 281 are image extraction units that extract image by reflecting image light output from the right display unit 22 and the left display unit 24, respectively, and can be called display units.

  The left optical system 252 and the left light guide plate 28 are collectively referred to as a “left light guide”, and the right optical system 251 and the right light guide plate 26 are collectively referred to as a “right light guide”. The configuration of the right light guide and the left light guide is not limited to the above example, and any method can be used as long as a virtual image is formed in front of the user's eyes using image light. It may be used, or a semi-transmissive reflective film may be used.

  Returning to FIG. 1, the control device 10 and the image display unit 20 are connected by a connection cable 40. The connection cable 40 is detachably connected to a connector (not shown) provided at the lower part of the control device 10, and is connected to various circuits provided in the image display unit 20 from the tip of the left holding unit 23. The connection cable 40 includes a metal cable or optical fiber cable that transmits digital data, and may include a metal cable that transmits an analog signal. A connector 46 is provided in the middle of the connection cable 40. The connector 46 is a jack for connecting a stereo mini-plug, and the connector 46 and the control device 10 are connected by a line for transmitting an analog audio signal, for example. In the configuration example shown in FIG. 1, a headset 30 having a right earphone 32 and a left earphone 34 and a microphone 63 constituting a stereo headphone is connected to a connector 46.

  For example, as shown in FIG. 1, the microphone 63 is arranged so that the sound collection unit of the microphone 63 faces the user's line of sight, collects sound, and outputs a sound signal to the sound interface 182 (FIG. 4). To do. For example, the microphone 63 may be a monaural microphone or a stereo microphone, may be a directional microphone, or may be an omnidirectional microphone.

The control device 10 includes a button 11, an LED indicator 12, a track pad 14, an up / down key 15, a changeover switch 16, and a power switch 18 as operated parts operated by a user.
The button 11 includes a menu key, a home key, a return key, and the like for operating the operating system 143 (FIG. 5) executed by the control device 10, and in particular, is displaced by a pressing operation of these keys and switches. Including things. The LED indicator 12 lights up or flashes in accordance with the operating state of the HMD 100. The up / down key 15 is used to input an instruction to increase / decrease the volume output from the right earphone 32 and the left earphone 34 and to input an instruction to increase / decrease the display brightness of the image display unit 20. The changeover switch 16 is a switch for changing an input corresponding to the operation of the up / down key 15. The power switch 18 is a switch for switching on / off the power of the HMD 100, and is configured by a slide switch, for example.

The track pad 14 has an operation surface for detecting a contact operation, and outputs an operation signal according to an operation on the operation surface. The detection method on the operation surface is not limited, and an electrostatic method, a pressure detection method, an optical method, or the like can be adopted.
Moreover, although not shown in figure, the control apparatus 10 is provided with the touch operation part which detects touch operation. The touch operation unit does not have a switch or the like that is displaced by the operation, and for example, an icon indicating an operation position and an operation content is arranged by display or printing on the screen. This contact (touch operation) with the touch operation unit is detected by a touch sensor 13 (FIG. 4) described later.

  FIG. 3 is a diagram illustrating a configuration of a main part of the image display unit 20, FIG. 3A is a main part perspective view of the image display unit 20 as viewed from the user's head side, and FIG. 3B is a camera. It is explanatory drawing of the angle of view of 61. FIG. Note that the connection cable 40 is not shown in FIG.

FIG. 3A shows the side of the image display unit 20 in contact with the user's head, in other words, the side visible to the user's right eye RE and left eye LE. In other words, the back sides of the right light guide plate 26 and the left light guide plate 28 are visible.
In FIG. 3A, the half mirror 261 that irradiates the user's right eye RE with the image light and the half mirror 281 that irradiates the left eye LE with the image light appear as substantially rectangular areas. Further, the entire right light guide plate 26 and left light guide plate 28 including the half mirrors 261 and 281 transmit external light as described above. For this reason, the user sees the outside scene through the entire right light guide plate 26 and the left light guide plate 28, and sees a rectangular display image at the positions of the half mirrors 261 and 281.

  The camera 61 is arranged at the right end of the image display unit 20 as described above, and images the direction in which both eyes of the user face, that is, the front for the user. FIG. 3B is a diagram schematically showing the position of the camera 61 in plan view together with the user's right eye RE and left eye LE. An angle of view (imaging range) of the camera 61 is indicated by C. FIG. 3B shows a horizontal angle of view C, but the actual angle of view of the camera 61 extends in the vertical direction as in a general digital camera.

  The optical axis of the camera 61 is a direction including the line-of-sight directions of the right eye RE and the left eye LE. The outside scene that the user can visually recognize in a state where the HMD 100 is worn is not always at infinity. For example, as shown in FIG. 3B, when the user gazes at the object OB with both eyes, the user's line of sight is directed toward the object OB as indicated by reference signs RD and LD in the figure. In this case, the distance from the user to the object OB is often about 30 cm to 10 m, and more often about 1 m to 4 m. Therefore, for the HMD 100, an upper limit and a lower limit guideline for the distance from the user to the object OB during normal use may be set. This standard may be obtained by investigation or experiment, or may be set by the user. The optical axis and the angle of view of the camera 61 are determined when the distance to the object OB during normal use corresponds to the set upper limit guide and when the target OB corresponds to the lower limit guide. It is preferably set so as to be included in the angle of view.

  In general, the viewing angle of a human is about 200 degrees in the horizontal direction and about 125 degrees in the vertical direction, and of these, the effective field of view with excellent information receiving capability is about 30 degrees in the horizontal direction and about 20 degrees in the vertical direction. Furthermore, the stable gaze field in which the gaze point that the human gazes at appears to be quickly and stably is about 60 to 90 degrees in the horizontal direction and about 45 to 70 degrees in the vertical direction. In this case, when the gazing point is the object OB in FIG. 3B, the effective visual field is about 30 degrees in the horizontal direction and about 20 degrees in the vertical direction around the lines of sight RD and LD. Further, the stable viewing field is about 60 to 90 degrees in the horizontal direction and about 45 to 70 degrees in the vertical direction, and the viewing angle is about 200 degrees in the horizontal direction and about 125 degrees in the vertical direction. Furthermore, the actual field of view that the user perceives through the image display unit 20 and through the right light guide plate 26 and the left light guide plate 28 can be referred to as a real field of view (FOV). In the configuration of the present embodiment shown in FIGS. 1 and 2, the actual visual field corresponds to an actual visual field that is transmitted through the right light guide plate 26 and the left light guide plate 28 and visually recognized by the user. The real field of view is narrower than the viewing angle and stable field of view, but wider than the effective field of view.

  The angle of view C of the camera 61 is preferably capable of capturing a wider range than the user's field of view, and specifically, the angle of view C is preferably at least wider than the effective field of view of the user. Further, it is more preferable that the angle of view C is wider than the actual field of view of the user. More preferably, the angle of view C is wider than the stable viewing field of the user, and most preferably, the angle of view C is wider than the viewing angle of both eyes of the user.

  The camera 61 may include a so-called wide-angle lens as an imaging lens so that a wide angle of view can be captured. The wide-angle lens may include a lens called a super-wide-angle lens or a quasi-wide-angle lens, may be a single focus lens or a zoom lens, and the camera 61 includes a lens group including a plurality of lenses. There may be.

FIG. 4 is a block diagram illustrating a configuration of each unit included in the HMD 100.
The control device 10 includes a main processor 140 that executes a program and controls the HMD 100. A memory 118 and a non-volatile storage unit 121 are connected to the main processor 140. The main processor 140 is connected to the track pad 14 and the operation unit 110 as input devices. In addition, a six-axis sensor 111, a magnetic sensor 113, and a GPS 115 are connected to the main processor 140 as sensors. The main processor 140 is connected to a communication unit 117, an audio codec 180, an external connector 184, an external memory interface 186, a USB connector 188, a sensor hub 192, and an FPGA 194. These function as an interface with the outside.

  The main processor 140 is mounted on the controller board 120 built in the control device 10. In addition to the main processor 140, a memory 118, a nonvolatile storage unit 121, and the like may be mounted on the controller board 120. In the present embodiment, a six-axis sensor 111, a magnetic sensor 113, a GPS 115, a communication unit 117, a memory 118, a nonvolatile storage unit 121, an audio codec 180, and the like are mounted on the controller board 120. Further, the external connector 184, the external memory interface 186, the USB connector 188, the sensor hub 192, the FPGA 194, and the interface 196 may be mounted on the controller board 120.

  The memory 118 constitutes a work area that temporarily stores a program to be executed and data to be processed when the main processor 140 executes the program. The nonvolatile storage unit 121 includes a flash memory or an eMMC (Embedded Multi Media Card). The nonvolatile storage unit 121 stores a program executed by the main processor 140 and various data processed by the main processor 140 executing the program.

The main processor 140 detects a touch operation on the operation surface of the track pad 14 based on an operation signal input from the track pad 14 and acquires an operation position.
The operation unit 110 includes a button 11, a touch sensor 13, and an LED display unit 17. The touch sensor 13 detects a touch operation on the touch operation unit included in the control device 10. When the operation of the button 11 is performed and when the touch sensor 13 detects the touch operation, an operation signal is output from the operation unit 110 to the main processor 140.
The LED display unit 17 includes an LED included in the LED indicator 12 (FIG. 1) and a drive circuit that lights the LED. The LED display unit 17 turns on, blinks, and turns off the LED according to the control of the main processor 140. Moreover, the LED display part 17 may control the brightness | luminance which LED light-emits. The LED display unit 17 may be configured to include LEDs of three colors, red, blue, and green. In this case, the LED indicator 12 is caused to emit light in an arbitrary color by adjusting the luminance of the LED of each color. May be.

The 6-axis sensor 111 is a motion sensor (inertial sensor) including a 3-axis acceleration sensor and a 3-axis gyro (angular velocity) sensor. The 6-axis sensor 111 may employ an IMU (Inertial Measurement Unit) in which the above sensors are modularized.
The magnetic sensor 113 is, for example, a triaxial geomagnetic sensor.
A GPS (Global Positioning System) 115 includes a GPS antenna (not shown), receives a radio signal transmitted from a GPS satellite, and detects the coordinates of the current position of the control device 10.
The six-axis sensor 111, the magnetic sensor 113, and the GPS 115 output detection values to the main processor 140 according to a sampling period specified in advance. Alternatively, the six-axis sensor 111, the magnetic sensor 113, and the GPS 115 output detection values to the main processor 140 at a timing designated by the main processor 140 in response to a request from the main processor 140.

The communication unit 117 performs wireless communication with an external device. The communication unit 117 includes an antenna, an RF circuit, a baseband circuit, a communication control circuit, and the like, or a device in which these are integrated. The communication unit 117 performs wireless communication conforming to standards such as Bluetooth (registered trademark) and wireless LAN (including Wi-Fi (registered trademark)).
The audio interface 182 is an interface for inputting and outputting audio signals. In the present embodiment, the audio interface 182 includes a connector 46 (FIG. 1) provided on the connection cable 40. The audio codec 180 is connected to the audio interface 182 and encodes / decodes an audio signal input / output via the audio interface 182. The audio codec 180 may include an A / D converter that converts analog audio signals to digital audio data, and a D / A converter that performs the reverse conversion. For example, the HMD 100 according to the present embodiment outputs sound by the right earphone 32 and the left earphone 34 and collects sound by the microphone 63. The audio codec 180 converts the digital audio data output from the main processor 140 into an analog audio signal and outputs the analog audio signal via the audio interface 182. The audio codec 180 converts an analog audio signal input to the audio interface 182 into digital audio data and outputs the digital audio data to the main processor 140.

The external connector 184 is a connector that connects an external device that communicates with the main processor 140. The external connector 184 is an interface for connecting an external device when, for example, an external device is connected to the main processor 140 to debug a program executed by the main processor 140 or collect a log of the operation of the HMD 100. It is.
The external memory interface 186 is an interface to which a portable memory device can be connected. For example, the external memory interface 186 includes a memory card slot in which a card type recording medium is mounted and data can be read and an interface circuit. In this case, the size, shape and standard of the card type recording medium are not limited and can be changed as appropriate.
A USB (Universal Serial Bus) connector 188 includes a connector conforming to the USB standard and an interface circuit, and can connect a USB memory device, a smartphone, a computer, and the like. The size and shape of the USB connector 188 and the compatible USB standard version can be appropriately selected and changed.

  Further, the HMD 100 includes a vibrator 19. The vibrator 19 includes a motor (not shown), an eccentric rotor (not shown), and the like, and generates vibrations under the control of the main processor 140. The HMD 100 generates vibration by the vibrator 19 with a predetermined vibration pattern, for example, when an operation on the operation unit 110 is detected, or when the power of the HMD 100 is turned on / off.

  The sensor hub 192 and the FPGA 194 are connected to the image display unit 20 via an interface (I / F) 196. The sensor hub 192 acquires detection values of various sensors included in the image display unit 20 and outputs them to the main processor 140. The FPGA 194 executes processing of data transmitted and received between the main processor 140 and each unit of the image display unit 20 and transmission via the interface 196.

  The right display unit 22 and the left display unit 24 of the image display unit 20 are each connected to the control device 10. As shown in FIG. 1, in the HMD 100, a connection cable 40 is connected to the left holding unit 23, wiring connected to the connection cable 40 is laid inside the image display unit 20, and each of the right display unit 22 and the left display unit 24 is Connected to the control device 10.

The right display unit 22 includes a display unit substrate 210. The display unit substrate 210 includes an interface (I / F) 211 connected to the interface 196, a receiving unit (Rx) 213 that receives data input from the control device 10 via the interface 211, and an EEPROM 215 (storage unit) ) Is implemented.
The interface 211 connects the receiving unit 213, the EEPROM 215, the temperature sensor 217, the camera 61, the illuminance sensor 65, and the LED indicator 67 to the control device 10.

  An EEPROM (Electrically Erasable Programmable Read-Only Memory) 215 stores various data so that the main processor 140 can read the data. The EEPROM 215 stores, for example, data relating to the light emission characteristics and display characteristics of the OLED units 221 and 241 provided in the image display unit 20, data relating to the characteristics of sensors provided in the right display unit 22 or the left display unit 24, and the like. Specifically, parameters relating to gamma correction of the OLED units 221, 241 and data for compensating the detection values of the temperature sensors 217, 239 are stored. These data are generated by the factory inspection of the HMD 100 and written in the EEPROM 215. After the shipment, the main processor 140 can perform processing using the data in the EEPROM 215.

The camera 61 executes imaging according to a signal input via the interface 211 and outputs captured image data or a signal indicating the imaging result to the control device 10.
As shown in FIG. 1, the illuminance sensor 65 is provided at the end ER of the front frame 27 and is arranged to receive external light from the front of the user wearing the image display unit 20. The illuminance sensor 65 outputs a detection value corresponding to the amount of received light (received light intensity).
As shown in FIG. 1, the LED indicator 67 is disposed near the camera 61 at the end ER of the front frame 27. The LED indicator 67 is lit during execution of imaging by the camera 61 to notify that imaging is in progress.

  The temperature sensor 217 detects the temperature and outputs a voltage value or a resistance value corresponding to the detected temperature as a detected value. The temperature sensor 217 is mounted on the back side of the OLED panel 223 (FIG. 3). For example, the temperature sensor 217 may be mounted on the same substrate as the OLED drive circuit 225. With this configuration, the temperature sensor 217 mainly detects the temperature of the OLED panel 223.

  The receiving unit 213 receives data transmitted from the main processor 140 via the interface 211. When receiving the image data of the image displayed on the OLED unit 221, the receiving unit 213 outputs the received image data to the OLED drive circuit 225 (FIG. 2).

The left display unit 24 includes a display unit substrate 210. On the display unit substrate 210, an interface (I / F) 231 connected to the interface 196 and a receiving unit (Rx) 233 that receives data input from the control device 10 via the interface 231 are mounted. In addition, a 6-axis sensor 235 and a magnetic sensor 237 are mounted on the display unit substrate 210.
The interface 231 connects the receiving unit 233, the six-axis sensor 235, the magnetic sensor 237, and the temperature sensor 239 to the control device 10.

The 6-axis sensor 235 is a motion sensor (inertial sensor) including a 3-axis acceleration sensor and a 3-axis gyro (angular velocity) sensor. The 6-axis sensor 235 may employ an IMU (Inertial Measurement Unit) in which the above sensors are modularized.
The magnetic sensor 237 is, for example, a triaxial geomagnetic sensor.

The temperature sensor 239 detects the temperature and outputs a voltage value or a resistance value corresponding to the detected temperature as a detected value. The temperature sensor 239 is mounted on the back side of the OLED panel 243 (FIG. 3). The temperature sensor 239 may be mounted on the same substrate as the OLED drive circuit 245, for example. With this configuration, the temperature sensor 239 mainly detects the temperature of the OLED panel 243.
Further, the temperature sensor 239 may be incorporated in the OLED panel 243 or the OLED drive circuit 245. The substrate may be a semiconductor substrate. Specifically, when the OLED panel 243 is mounted as an Si-OLED as an integrated circuit on an integrated semiconductor chip together with the OLED drive circuit 245 and the like, the temperature sensor 239 may be mounted on the semiconductor chip.

The camera 61, the illuminance sensor 65, the temperature sensor 217, and the 6-axis sensor 235, the magnetic sensor 237, and the temperature sensor 239 included in the left display unit 24 are connected to the sensor hub 192. The sensor hub 192 sets and initializes the sampling period of each sensor according to the control of the main processor 140. The sensor hub 192 executes energization to each sensor, transmission of control data, acquisition of a detection value, and the like in accordance with the sampling period of each sensor. The sensor hub 192 outputs detection values of the sensors included in the right display unit 22 and the left display unit 24 to the main processor 140 at a preset timing. The sensor hub 192 may have a function of temporarily holding the detection value of each sensor in accordance with the output timing to the main processor 140. In addition, the sensor hub 192 may have a function of converting to data in a unified data format and outputting the data to the main processor 140 in response to a difference in signal format or data format of the output value of each sensor.
The sensor hub 192 starts and stops energization of the LED indicator 67 according to the control of the main processor 140, and lights or blinks the LED indicator 67 in accordance with the timing when the camera 61 starts and ends imaging.

  The control device 10 includes a power supply unit 130 and operates with power supplied from the power supply unit 130. The power supply unit 130 includes a rechargeable battery 132 and a power supply control circuit 134 that detects the remaining capacity of the battery 132 and controls the charging of the battery 132. The power control circuit 134 is connected to the main processor 140 and outputs a detected value of the remaining capacity of the battery 132 or a detected value of the voltage to the main processor 140. Further, the power may be supplied from the control device 10 to the image display unit 20 based on the power supplied from the power supply unit 130. The power supply state from the power supply unit 130 to each unit of the control device 10 and the image display unit 20 may be configured to be controllable by the main processor 140.

  FIG. 5 is a functional block diagram of the storage unit 122 and the control unit 150 that constitute the control system of the control device 10. The storage unit 122 illustrated in FIG. 5 is a logical storage unit configured by the nonvolatile storage unit 121 (FIG. 4), and may include an EEPROM 215. The control unit 150 and various functional units included in the control unit 150 are formed by cooperation of software and hardware by the main processor 140 executing a program. The control unit 150 and each functional unit constituting the control unit 150 include, for example, a main processor 140, a memory 118, and a nonvolatile storage unit 121.

The control unit 150 executes various processes using data stored in the storage unit 122 and controls the HMD 100.
The storage unit 122 stores various data processed by the control unit 150. The storage unit 122 stores setting data 123, content data 124, and operation setting data 127. The setting data 123 includes various setting values related to the operation of the HMD 100. When the control unit 150 controls the HMD 100, parameters, determinants, arithmetic expressions, LUTs (Look Up Tables), and the like may be included in the setting data 123.
The content data 124 is content data including images and videos displayed by the image display unit 20 under the control of the control unit 150, and includes image data or video data. Further, the content data 124 may include audio data. The content data 124 may include image data of a plurality of images. In this case, the plurality of images are not limited to images displayed on the image display unit 20 at the same time.
The content data 124 is an interactive type in which when the content is displayed on the image display unit 20, the control device 10 receives a user operation and the control unit 150 executes a process according to the received operation. It may be content. In this case, the content data 124 may include image data of a menu screen that is displayed when an operation is accepted, data that defines processing corresponding to items included in the menu screen, and the like.

  The operation setting data 127 includes various setting data related to control executed by the control unit 150 in response to operations of the track pad 14 and the operation unit 110. Specifically, the detection accuracy (detection resolution) when the control unit 150 detects an operation on the track pad 14, and the accuracy (resolution) of the data related to the display position that the control unit 150 outputs in response to the operation of the track pad 14 ) Etc. are included.

  The control unit 150 has functions of an operating system (OS) 143, an image processing unit 145, a display control unit 147, an imaging control unit 149, and an operation detection control unit 152. The functions of the operating system 143 are functions of a control program stored in the storage unit 122, and the other units are functions of application programs executed on the operating system 143.

The image processing unit 145 generates a signal to be transmitted to the right display unit 22 and the left display unit 24 based on image data of an image or video displayed by the image display unit 20. The signal generated by the image processing unit 145 may be a vertical synchronization signal, a horizontal synchronization signal, a clock signal, an analog image signal, or the like.
Further, the image processing unit 145 may perform resolution conversion processing for converting the resolution of the image data to a resolution suitable for the right display unit 22 and the left display unit 24 as necessary. The image processing unit 145 also performs image adjustment processing for adjusting the brightness and saturation of image data, 2D image data from 3D image data, 2D / 3D conversion processing for generating 3D image data from 2D image data, and the like. May be executed. When these image processes are executed, the image processing unit 145 generates a signal for displaying an image based on the processed image data, and transmits the signal to the image display unit 20 via the connection cable 40.

  The image processing unit 145 may be configured by hardware (for example, DSP (Digital Signal Processor)) different from the main processor 140 in addition to the configuration realized by the main processor 140 executing a program.

  The display control unit 147 generates a control signal for controlling the right display unit 22 and the left display unit 24, and controls the generation and emission of image light by the right display unit 22 and the left display unit 24 based on the control signal. . Specifically, the display control unit 147 controls the OLED drive circuits 225 and 245 to cause the OLED panels 223 and 243 to display an image. The display control unit 147 controls the timing at which the OLED drive circuits 225 and 245 draw on the OLED panels 223 and 243, controls the luminance of the OLED panels 223 and 243, and the like based on the signal output from the image processing unit 145.

  The imaging control unit 149 controls the camera 61 to execute imaging, generates captured image data, and temporarily stores the captured image data in the storage unit 122. When the camera 61 is configured as a camera unit including a circuit that generates captured image data, the imaging control unit 149 acquires captured image data from the camera 61 and temporarily stores the captured image data in the storage unit 122.

The operation detection control unit 152 (operation information generation unit) detects an operation on the track pad 14 and the operation unit 110 and outputs data corresponding to the operation. For example, when a button or the like of the operation unit 110 is operated, the operation detection control unit 152 generates operation data indicating the operation content and outputs the operation data to the display control unit 147. The display control unit 147 changes the display state according to the operation data input from the operation detection control unit 152.
Further, when the operation detection control unit 152 detects an operation on the track pad 14, the operation detection control unit 152 generates and outputs movement data indicating a change in the relative position corresponding to the change in the operation position on the track pad 14. Specifically, the operation detection control unit 152 acquires the coordinates of the operation position in the operation detection area (detection area) of the track pad 14 when an operation on the track pad 14 is detected. In the present embodiment, it is assumed that the entire surface of the track pad 14 is a detection region. The operation detection control unit 152 continues the operation of repeatedly detecting the operation on the track pad 14 at a preset detection cycle (sampling cycle). The operation detection control unit 152 outputs movement data indicating the detected change amount of the operation position, not the coordinates of the detected operation position.

The movement data is data of a vector amount indicating the movement direction and movement amount of the base point position, or data indicating a change in coordinates of the base point position.
Here, the base point is a virtual operation position generated by the operation detection control unit 152, and the base point may coincide with the operation position on the track pad 14. The position of the base point corresponds to a display control position and a processing position. Further, the operation detection control unit 152 may perform a process of calculating the coordinates of the base point at a position different from the operation position based on the operation position. The movement data can include data on the amount of change in the X coordinate and the Y coordinate of the base point position as data indicating the change in the coordinate of the base point position.
The vector amount or the coordinates included in the movement data is based on the coordinates in the detection area of the track pad 14 detected by the operation detection control unit 152.

For example, the display control unit 147 displays a pointer that moves in response to an operation of the pointing device. The shape and size of the pointer are arbitrary, and the number to be displayed is also arbitrary. In the present embodiment, an example in which one pointer that moves according to the operation on the track pad 14 is displayed will be described.
The display control unit 147 moves the display position of the pointer based on the movement data output from the operation detection control unit 152. Since the movement data is data based on the coordinates in the detection area of the track pad 14, the display control unit 147 converts the coordinates or amount included in the movement data into coordinates or quantities in the display area of the image display unit 20. Parameters, arithmetic expressions, and the like when the display control unit 147 performs conversion are included in the operation setting data 127, for example.

  Further, the HMD 100 may include an interface (not shown) for connecting various external devices that are content supply sources. For example, an interface corresponding to a wired connection such as a USB interface, a micro USB interface, or a memory card interface may be used, or a wireless communication interface may be used. The external device in this case is an image supply device that supplies an image to the HMD 100, and a personal computer (PC), a mobile phone terminal, a portable game machine, or the like is used. In this case, the HMD 100 can output images and sounds based on content data input from these external devices.

6 and 7 are flowcharts showing the operation of the HMD 100. Specifically, the operation of displaying a pointer and moving the display position of the pointer in response to an operation on the track pad 14 is shown.
8, 9, and 10 are explanatory diagrams showing changes in display corresponding to the operation of the HMD 100. 8 shows a display change mode in the one-point operation mode, FIG. 9 shows a display change mode in the multi-point operation mode, and FIG. 10 shows a display change mode in the three-point operation mode.

  When the control unit 150 starts operating, the operation detection control unit 152 starts detecting an operation on the track pad 14 (step S11), and the display control unit 147 displays a pointer (step S12). Thereafter, the operation detection control unit 152 stands by until there is an operation on the track pad 14 (step S13).

When the track pad 14 is operated and an operation detection interrupt is generated (step S152), the operation detection control unit 152 acquires the operation position in the detection area of the track pad 14 (step S15).
The operation detection control unit 152 determines whether or not the operation position of the track pad 14 is one (step S16). When the operation position is one (step S16; Yes), the operation detection control unit 152 starts the one-point operation mode (step S17). The one-point operation mode is one of the operation modes of the control unit 150, particularly the operation detection control unit 152. The one-point operation mode is an operation mode in which movement data is generated in accordance with a change in one operation position on the track pad 14, and the pointer display position is changed based on the movement data. In addition, as an operation mode of the operation detection control unit 152, a multi-point operation mode corresponding to two or more operation positions is executed, and the operation position is increased to 3 points after two operations are performed. There is a three-point operation mode. The operation detection control unit 152 can switch and execute these three operation modes.

  After starting the one-point operation mode, the operation detection control unit 152 associates the operation position detected on the track pad 14 with the base position (step S18). The operation detection control unit 152 uses the coordinates of the operation position of the track pad 14 as the coordinates of the base point position.

The operation detection control unit 152 acquires the operation position of the track pad 14 (step S19). Here, the operation detection control unit 152 determines whether or not the number of operation positions is one (step S20). When the operation position is one (step S20; Yes), the operation detection control unit 152 continues the one-point operation mode and outputs the movement data of the base point position (step S21). In step S <b> 21, the operation detection control unit 152 generates and outputs movement data using the coordinates of the operation position in the detection area of the track pad 14 as the coordinates of the base point.
The display control unit 147 moves the display position of the pointer based on the movement data output from the operation detection control unit 152 (step S22). The processing in steps S21 to S22 corresponds to “moving display” of the pointer.

  FIG. 8A shows a display example of pointers displayed by the display control unit 147 in the display area V. FIG. In the example of FIG. 8A, an arrow-shaped pointer P is displayed, and the display position of the pointer P is moved according to the operation of the track pad 14.

  FIG. 8B shows an operation example on the track pad 14. In this example, the contact operation is detected at the position A1 of the track pad 14, and then the detected position moves to the position A2. The operation detection control unit 152 executes the one-point operation mode, and acquires the coordinates (Xa1, Ya1) of the position A1 and the coordinates (Xa2, Ya2) of the position A2 as the coordinates of the base point position. The operation detection control unit 152 generates and outputs movement data indicating movement from the position A1 (Xa1, Ya1) to the position A2 (Xa2, Ya2). In response to the movement data, the display control unit 147 moves the display position of the pointer P from the position P1 (Xp1, Yp1) to the position P2 (Xp2, Yp2) as shown in FIG.

  Here, the movement amount of the display position of the pointer P is calculated based on the movement data. A movement amount in the X direction (Xp2-Xp1) and a movement amount in the Y direction (Yp2-Yp1) of the display position of the pointer P, a movement amount in the X direction (Xa2-Xa1), and a movement amount in the Y direction (Ya2). -Ya1) is associated with a predetermined parameter, function, matrix, or the like. Data relating to this association is included in the operation setting data 127, for example.

Returning to FIG. 6, the operation detection control unit 152 determines whether or not the operation on the track pad 14 is released (step S23). The cancellation of the operation means that the contact operation with respect to the track pad 14 is lost.
When the operation has not been released (step S23; No), the operation detection control unit 152 returns to step S19 and continues the one-point operation mode. When the operation is released (step S23; Yes), the control unit 150 determines whether or not the detection of the operation of the track pad 14 is ended (step S24). When the power of the HMD 100 is turned off, when the operation end of the track pad 14 is instructed, or when the end of the display corresponding to the operation of the track pad 14 is instructed, the control unit 150 determines to end the detection. (Step S24; Yes), this process is terminated. If it is determined that the detection is not completed (step S24; No), the control unit 150 returns to step S13 and waits for the operation of the track pad 14.

In addition, the operation detection control unit 152 acquires the operation position on the track pad 14 and determines that the operation position is not one (step S16; No, step S20; No), and starts the multipoint operation mode ( Step S31 in FIG.
The operation detection control unit 152 calculates a base point position from the coordinates of a plurality of operation positions on the track pad 14 that has already been acquired (step S32). For example, the operation detection control unit 152 performs a predetermined calculation process on the coordinates of all the operation positions on the track pad 14 and calculates the coordinates of the base point position. An arithmetic expression, a function, a parameter, and the like for obtaining the base point position are included in the operation setting data 127, for example.

  The operation detection control unit 152 acquires the operation position of the track pad 14 (step S33). Here, the operation detection control unit 152 determines whether or not the number of operation positions has increased from two to three (step S34). That is, the operation detection control unit 152 determines whether the condition for starting the three-point operation mode is correct. The three-point operation mode is executed when the third contact operation is detected while the contact operation is performed at two positions on the track pad 14. For example, the three-point operation mode is not executed when the position of the contact operation on the track pad 14 is increased from one to three, or when three contact operations are performed from a state where there is no contact operation. The operation detection control unit 152 compares the number of operation positions on the track pad 14 detected in the past with the number of operation positions acquired in step S33, and performs the determination in step S34.

  When it is determined that the operation position does not correspond to the case where the number of operation positions is increased from 2 to 3 (step S34; No), the operation detection control unit 152 determines whether or not the operation position is decreased to one position ( Step S35). If it is determined that the operation position has decreased to one place (step S35; Yes), the operation detection control unit 152 proceeds to step S17 (FIG. 6) and starts the one-point operation mode.

  If it is determined that the operation position has not decreased to one place (step S35; No), the operation detection control unit 152 performs the same processing as in step S32, for example, so that the base point is determined from the plurality of operation positions of the track pad 14. The position is calculated (step S36). The operation detection control unit 152 generates and outputs movement data indicating the movement amount and movement direction of the base point position (step S37). The display control unit 147 moves the display position of the pointer based on the movement data output from the operation detection control unit 152, and updates the display (step S38). The processing in steps S37 to S38 corresponds to “moving display” of the pointer.

FIG. 9A shows an example of operation on the track pad 14, and FIG. 9B shows an example of change in the operation position of the track pad 14. FIG. 9C shows a change in display corresponding to the movement of the operation position.
FIG. 9A shows an example in which contact operations are detected at a plurality of positions on the track pad 14. In FIG. 9A, an operation is detected at two positions A1 (Xa1, Ya1) and B1 (Xb1, Yb1). The operation detection control unit 152 calculates the coordinates (Xs1, Ys1) of the base point S1 based on the coordinates of the position A1 and the position B1. For example, the base point S1 can be a midpoint between the position A1 and the position B1. Further, in the simplest example, when the track pad 14 is operated at three or more positions, the average of the coordinates of all the operation positions can be calculated and used as the coordinates of the base point S1. In the multipoint operation mode, the base point S1 is set at a position different from the operation position (A1, B1 here).

  FIG. 9B shows an example in which contact operations are detected at a plurality of positions on the track pad 14 and the plurality of operation positions are moved. In the example of FIG. 9B, the operation position A1 has moved to position A2 (Xa2, Ya2), and the operation position B1 has moved to B2 (Xb2, Yb2). In this case, the base point S1 moves to the position S2 (Xs2, Ys2). In this case, the operation detection control unit 152 generates and outputs movement data indicating movement from the base position S1 (Xs1, Ys1) to the position S2 (Xs2, Ys2). That is, movement data indicating the movement amount in the X direction (Xs2-Xs1) and the movement amount in the Y direction (Ys2-Ys1) of the coordinates of the base point is output. In response to the movement data, the display control unit 147 moves the display position of the pointer P from the position P1 (Xp1, Yp1) to the position P2 (Xp2, Yp2) as shown in FIG. 9C.

  The amount of movement of the display position of the pointer P is calculated based on the movement data. As described above, the movement amount in the X direction (Xp1-Xp2) and the movement amount in the Y direction (Yp2-Yp1) of the display position of the pointer P and the movement amount in the X direction and the movement amount in the Y direction of the base point position The correspondence is associated with a predetermined parameter, function, matrix, or the like. Data relating to this association is included in the operation setting data 127, for example.

  Returning to FIG. 7, the operation detection control unit 152 determines whether or not the operation on the track pad 14 has been released (step S39). If the operation is not released (step S39; No), the operation detection control unit 152 returns to step S33 and continues the multipoint operation mode. When the operation is released (step S39; Yes), the control unit 150 proceeds to step S24 (FIG. 6).

  When it is determined that the condition for starting the three-point operation mode is satisfied (step S34; Yes), the operation detection control unit 152 starts the three-point operation mode (step S41). In the three-point operation mode, the third operation position, that is, the operation position newly (finally) detected in the state where the two operation positions are detected is associated with the base point (step S42).

The operation detection control unit 152 acquires the operation position of the track pad 14 (step S43). Here, the operation detection control unit 152 determines whether or not the number of operation positions is three (step S44). If the operation position is not three (step S44; No), the operation position has increased or decreased, and the operation detection control unit 152 proceeds to step S16 (FIG. 6). If there are three operation positions (step S44; Yes), the operation detection control unit 152 continues the three-point operation mode and outputs movement data of the base point position (step S45). In step S45, the operation detection control unit 152 generates and outputs movement data using the coordinates of the third operation position in the detection area of the track pad 14 as the coordinates of the base point.
The display control unit 147 moves the display position of the pointer based on the movement data output from the operation detection control unit 152 (step S46). The processing in steps S45 to S46 corresponds to “moving display” of the pointer.

  FIG. 10A shows an example of an operation on the track pad 14, and FIG. 10B shows an example of a change in the operation position of the track pad 14. FIG. 10C shows an example of movement of the operation position, and FIG. 10D shows a change in display corresponding to the movement of the operation position.

  FIG. 10A shows an example in which operations are detected at two positions A1 (Xa1, Ya1) and B1 (Xb1, Yb1) as examples in which contact operations are detected at a plurality of positions on the track pad 14. . In the state of FIG. 10A, the operation detection control unit 152 executes the multipoint operation mode, and calculates the coordinates (Xs1, Ys1) of the base point S1 based on the coordinates of the position A1 and the position B1.

As shown in FIG. 10B, when the operation at the third operation position C1 (Xc1, Yc1) is detected from the state of FIG. 10A, the operation detection control unit 152 starts the three-point operation mode. . In this case, the coordinates (Xc1, Yc1) of the position C1 are the coordinates of the base point.
As shown in FIG. 10C, when the position C1 moves to the position C2 (Xc2, Yc2), the operation detection control unit 152 moves the X direction (Xc2-Xc1) and the Y direction (Yc2). The movement data corresponding to -Yc1) is output. In the example of FIG. 10C, the previously detected operation positions A1 and A2 are not moved, but these operation positions are not involved in the processing for obtaining the position of the base point.
In response to the movement data, the display control unit 147 moves the display position of the pointer P from the position P1 (Xp1, Yp1) to the position P2 (Xp2, Yp2) as shown in FIG.

  The operations shown in FIGS. 10B and 10C can be performed by the user using, for example, three fingers of one hand. When the user touches the trackpad 14 with two fingers and touches the third finger, the position of the third finger is the base point, and the pointer is moved according to the movement of the third finger. P is moved. That is, the user moves the display position of the pointer P by a touch operation with two fingers, and further performs the touch operation with the third finger without moving the two fingers, thereby the pointer P. Can be moved. For example, if the two fingers reach the end of the detection area during the touch operation of two fingers, or if it becomes difficult to move the two fingers due to the user's posture, the third finger The pointer P can be moved by performing a touch operation.

  In this way, by using the three-point operation mode, a specific operability can be realized. In addition, since the operation detection control unit 152 starts the three-point operation mode only when the above-described condition of step S34 is satisfied, the possibility of erroneously starting the three-point operation mode is extremely low, and erroneous operation is prevented. it can.

  Returning to FIG. 7, the operation detection control unit 152 determines whether or not the operation on the track pad 14 is released (step S47). If the operation has not been released (step S47; No), the operation detection control unit 152 returns to step S43 and continues the three-point operation mode. When the operation is released (step S47; Yes), the control unit 150 proceeds to step S24 (FIG. 6).

  In the example described with reference to FIGS. 6 to 10, the operation in which the display control unit 147 displays the pointer P and changes the display position of the pointer P has been described. The functions of the operation detection control unit 152 and the display control unit 147 are not limited to this. For example, the display control unit 147 may perform an operation of scrolling part or all of the screen displayed in the display area of the image display unit 20 based on the movement data output from the operation detection control unit 152. In this case, the display control unit 147 may determine the scroll direction corresponding to the movement direction of the base point included in the movement data, and determine the scroll amount corresponding to the movement amount of the base point. Further, for example, the display control unit 147 may perform a zoom operation for enlarging / reducing a part or the whole of the screen displayed in the display area of the image display unit 20 based on the movement data output from the operation detection control unit 152. Good. In this case, the display control unit 147 may determine the zoom direction corresponding to the movement direction of the base point included in the movement data, that is, enlargement or reduction, and determine the zoom amount corresponding to the movement amount of the base point. In this case, the center of enlargement / reduction may be, for example, a display position corresponding to the base position.

  In addition, in the multipoint operation mode of FIG. 7, when the display control unit 147 detects an operation in which the interval between the operation positions becomes large or an operation in which the interval becomes small, Reduction may be performed. This operation corresponds to a so-called pinch operation (pinch-in, pinch-out). In this case, the movement of the display position of the pointer P may be stopped, or the display of the pointer P may be stopped.

Further, the display control unit 147 may change the shape or the like of the pointer P.
FIG. 11 is a diagram illustrating another example of display in the HMD 100. FIG. 11A shows a display example when operations at a plurality of operation positions on the track pad 14 are detected. FIGS. 11B and 11C show the movement of a plurality of operation positions, and FIG. 11D shows a change in display corresponding to the movement of the operation positions.

In the example of FIG. 11A, when the operation detection control unit 152 detects a plurality of operation positions while the pointer P is displayed, the display control unit 147 stops the display of the pointer P, and two operations are performed instead. Markers Q1 and Q2 corresponding to the positions are displayed.
Further, as shown in FIG. 11B, the operation detection control unit 152 determines the coordinates (Xs1, Ys1) of the base point S1 based on the two operation positions A1 (Xa1, Ya1) and the position B1 (Xb1, Yb1). calculate.

  After that, when the operation position is moved, the operation detection control unit 152, as shown in FIG. 11C, is based on the coordinates of the moved positions A2 and B2 where the two operation positions A1 and B1 are moved. The coordinates (Xs2, Ys2) of the base point S2 are calculated. Then, the operation detection control unit 152 outputs movement data corresponding to the movement from the base point S1 to S2.

As shown in FIG. 11D, the display control unit 147 moves the display positions of the markers Q1 and Q2 in accordance with the movement data output from the operation detection control unit 152. In this case, the movement direction and movement amount of the markers Q1, Q2 correspond to the movement amount of the base points S1-S2. More specifically, the movement amount in the X direction (XPa2-XPa1) and the movement amount in the Y direction (YPa2-YPa1) by which the marker Q1 moves from the position PA1 to the position PA2 are the movement amount in the X direction of the base point (Xs2-Xs1). ) And the amount of movement in the Y direction (Ys2-Ys1). The movement of the marker Q2 from the position PB1 to the position PB2 is the same.
Thus, by changing the display of the pointer P to the markers Q1 and Q2 according to the number of touch operations on the track pad 14, the user can be notified of the number of touch operations detected by the track pad 14. . For this reason, the user can know the state of operation with respect to the track pad 14 without visually recognizing the track pad 14.

  In the present embodiment, under the control of the control unit 150, the HMD 100 displays a pointer P and markers Q1 and Q2 as predetermined images. The predetermined image is not limited to the pointer P and the markers Q1 and Q2, and may be a circle pointer configured by a circle drawn in the display area or a “◯” shaped pointer. Also, a cross-shaped pointer may be used.

  In the above embodiment, the example in which the markers Q1 and Q2 are displayed corresponding to the two operation positions on the track pad 14 and the pointer P is displayed in correspondence with the three operation positions has been described. Here, when displaying the markers Q1, Q2 or the pointer P as a predetermined image corresponding to two or more operation positions, the display mode of the predetermined image may be changed. For example, it is conceivable to change the transmittance (brightness) of a predetermined image in accordance with the distance between a plurality of detection positions detected by the track pad 14. In the image display unit 20, the higher the luminance at which the right display unit 22 and the left display unit 24 display an image, the higher the relative light quantity of the image light L with respect to the external light OL (FIG. 2). Visibility is increased. That is, the transmittance of the display image of the image display unit 20 is lowered. On the other hand, the lower the luminance at which the right display unit 22 and the left display unit 24 display an image, the lower the relative light quantity of the image light L with respect to the external light OL, and the lower the visibility of the display image. That is, the transmittance of the display image of the image display unit 20 is increased. For example, under the control of the control unit 150, it is conceivable that the brightness of a predetermined image displayed by the image display unit 20 is increased as the distance between a plurality of detection positions detected by the track pad 14 is smaller. In another expression, the control unit 150 increases the luminance of the predetermined image as the operation position interval is narrower, and thereby decreases the transmittance. Thereby, when an operation for finely adjusting the position is performed, the operability can be improved by increasing the visibility of the pointer P and the markers Q1 and Q2. On the contrary, it is conceivable that the brightness of the predetermined image displayed by the image display unit 20 is lowered as the distance between the plurality of detection positions detected by the track pad 14 is larger. In another expression, the control unit 150 decreases the brightness of the predetermined image as the interval between the operation positions is larger, thereby increasing the transmittance. In the image display unit 20 that displays an image so as to be visible through the outside scene, the transmittance of the display image has a strong influence on the visibility and operability, and can be considered as an index of visibility. For this reason, the operability can be greatly improved by changing the transmittance of the pointer or marker which is a display image corresponding to the fineness of the operation. Further, the control unit 150 may change the pointer P and the markers Q1 and Q2 when the number of operation positions detected by the track pad 14 is 1, when the operation position is 2, and when the number is 3 or more. In this case, the user can know the number of operation positions detected by the track pad 14 from the transmittance of the pointer P and the markers Q1 and Q2. Further, the operability can be further improved by changing the transmittance as the display mode of the pointer P and the markers Q1 and Q2 corresponding to the mode of operation performed by the user on the track pad 14. Can do.

  Further, for example, the shape of the predetermined image may be changed according to the distance between a plurality of detection positions detected by the track pad 14 under the control of the control unit 150. An example of this case is shown in FIGS.

  FIG. 12 is an explanatory diagram showing a change in display corresponding to the operation of the HMD 100. 12A shows an example of the operation position on the track pad 14, and FIG. 12B shows a pointer P11 displayed by the image display unit 20. FIG. 12C shows an example of the operation position on the track pad 14, and FIG. 12B shows a pointer P 12 displayed on the image display unit 20.

FIG. 12A shows an example in which the control unit 150 detects two operation positions A1 (Xa1, Ya1) and B1 (Xb1, Yb1) with the trackpad 14. FIG. 12B shows an example in which a circular pointer P11 is displayed at a display position corresponding to the operation position at the center of two points, for example, corresponding to the operation position of FIG.
When the control unit 150 performs control to make the shape of the pointer elongated as the interval between the operation positions on the track pad 14 increases, an elliptical pointer P12 is displayed as shown in FIG. That is, as shown in FIG. 12C, the control unit 150 detects two operation positions A1 (Xa1, Ya1) and B3 (Xb3, Yb3) with the track pad 14, and a distance Dis2 between the positions A1 and B3. Is longer than the distance Dis1 between the positions A1 and B1. In this case, the control unit 150 displays the pointer P12 having an ellipticity corresponding to the distance Dis3 at a display position corresponding to the midpoint between the operation positions A1 (Xa1, Ya1) and B3 (Xb3, Yb3). To do.

FIG. 13 is an explanatory diagram showing a change in display corresponding to the operation of the HMD 100. FIG. 13A shows an example of an operation position on the track pad 14, and FIG. 13B shows a pointer P 21 displayed on the image display unit 20. FIG. 13C shows an example of the operation position on the track pad 14, and FIG. 13B shows a pointer P 22 displayed on the image display unit 20.
The example of FIG. 13 shows an example in which cross-shaped pointers P21 and P22 are displayed.

  FIG. 13A shows an example in which the control unit 150 detects two operation positions A1 (Xa1, Ya1) and B1 (Xb1, Yb1) with the trackpad 14. FIG. 13B shows an example in which a cross-shaped pointer P21 is displayed at a display position corresponding to the operation position at the center of two points, for example, corresponding to the operation position of FIG.

  When the control unit 150 performs control to elongate the shape of the pointer as the interval between the operation positions on the track pad 14 increases, the pointer P22 obtained by extending the cross shape to be elongated is displayed as shown in FIG. As shown in FIG. 13C, the control unit 150 detects two operation positions A1 (Xa1, Ya1) and B3 (Xb3, Yb3) with the track pad 14, and the distance Dis2 between the positions A1 and B3 is Assume that the distance between the positions A1 and B1 is longer than the distance Dis1. In this case, the control unit 150 displays the pointer P22 whose horizontal straight line is extended corresponding to the distance Dis2, corresponding to the midpoint between the operation positions A1 (Xa1, Ya1) and B3 (Xb3, Yb3). Display in position.

  As described above, the control unit 150 may change the size of the pointer as illustrated in FIGS. 12D and 13D. In the head-mounted image display unit 20 that affects the user's visual field, the size of the display image has a strong influence on the visibility and operability, and can be considered as an easy-to-see index. For this reason, operability can be greatly improved by changing the size and shape of the pointers and markers, which are display images, corresponding to the fineness of the operation.

FIG. 14 is an explanatory diagram showing a change in display corresponding to the operation of the HMD 100.
FIG. 14A shows an example in which the control unit 150 detects two operation positions A1 (Xa1, Ya1) and B1 (Xb1, Yb1) with the trackpad 14. In this example, the operation position A1 moves to position A2 (Xa2, Ya2), and the operation position B1 moves to B2 (Xb2, Yb2). In this case, as described above, since the control unit 150 sets the base point S1 as the midpoint of the positions A1 and B1, pointers and markers are displayed corresponding to the trajectory where the base point S1 has moved to the base point S2.

  FIG. 14B shows an example of a display mode when the display position of the circular pointer P31 is moved in accordance with the movement of the base point. As shown in FIG. 14B, when the circular pointer P31 is moved, an image imitating the afterimage of the pointer P31 may be displayed so that the moving direction and movement locus of the pointer P31 can be easily understood. That is, the pointer P31 has a shape including an afterimage portion that represents a movement locus. In other words, the shape is like a comet tail, and the user can clearly know the movement of the pointer P31. The process of changing the shape of this type of pointer to a shape having a locus or an afterimage can be applied when the pointer has various shapes such as an arrow shape, a cross shape, and a circular shape.

  In addition, when the controller 150 detects two detection positions on the track pad 14, that is, when the user performs an operation of touching the two positions, the user interface is changed according to the distance between the two detection positions. May be changed. For example, the control unit 150 distinguishes the distance between two detection positions in two stages based on a preset threshold value. As an example, a distinction is made between a “normal selection state” in which the distance between two detection positions is a normal range and a “precise selection state” in which the distance between the two detection positions is small. In this case, the control unit 150 may switch the position detection resolution on the track pad 14 to a higher resolution in the “precision selection state” than in the “normal selection state”.

  Further, when the control unit 150 detects three operation positions on the track pad 14, the control unit 150 obtains the coordinates of one base point based on the three operation positions, and displays a predetermined image at the display position corresponding to the base point coordinates. May be. In this case, the center of the two operation positions may be obtained as a base point, and the obtained base point may be set to a position that is biased according to the third operation position. Alternatively, a position corresponding to the center or the center of gravity of the three operation positions may be used as the base point.

As described above, the HMD 100 worn and used by the user includes the track pad 14 that receives a touch operation and the image display unit 20 that has a display area for displaying an image. Moreover, the control part 150 which detects the operation position in the trackpad 14 and controls the display of the image display part 20 according to the detected operation is provided. The control unit 150 displays a pointer or marker as a predetermined image on the image display unit 20 and moves the display position of the predetermined image according to the operation position of the track pad 14. When moving the display position of the pointer or marker as the predetermined image, the control unit 150 detects the operation position and the predetermined position when detecting one operation position with the track pad 14 and when detecting a plurality of operation positions. The association with the display position of the image is changed.
For example, the operation detection control unit 152 changes the association between the coordinates of the operation position detected on the track pad 14 and the base point according to the number of operation positions. In the one-point operation mode, the operation position is set as the base point position, and in the multi-point operation mode, a position other than the operation position calculated from the operation position is set as the base point position. In the three-point operation mode, the third operation position is set as the base point position.

  The HMD 100 also includes a track pad 14 that accepts a contact operation, and an image display unit 20 that has a display area for displaying an image. In addition, a control unit 150 that detects an operation position on the track pad 14, outputs a display control position based on the detected operation position, and causes the image display unit 20 to display an image based on the display control position is provided. The control unit 150 changes the association between the detected operation position and the display control position when one operation position is detected and when a plurality of operation positions are detected.

According to the control device 10 as the input device, the HMD 100 as the display device, and the display device control method executed by the HMD 100, it is possible to increase the number of operation variations by changing the number of operation positions. Can be improved.
For example, the manner of controlling the display position of the pointer corresponding to the operation can be changed depending on the number of operation positions where the touch operation is performed on the track pad 14.

  When the control unit 150 detects two operation positions on the track pad 14 when moving the display position of the pointer or marker as the predetermined image, the control unit 150 sets the display position corresponding to the operation position different from the two detected operation positions. The display position of the predetermined image is moved. Accordingly, by performing the contact operation at the two operation positions, it is possible to increase the variation of the operation for moving the display position of the predetermined image, and it is possible to improve the operability.

  Based on the two operation positions detected by the track pad 14, the control unit 150 moves the display position of the predetermined image to a display position corresponding to an operation position different from the two detected operation positions. Thereafter, when one new operation position is detected by the track pad 14, the control unit 150 moves the display position of the predetermined image to the display position corresponding to the one new operation position. Accordingly, when an operation is performed at a new position in addition to the two operation positions, the display position of the predetermined image can be controlled corresponding to the new operation position. Thereby, the variation of operation which moves the display position of a predetermined image can be increased, and operability can be improved.

  The control unit 150 may stop displaying the predetermined image when an operation at two operation positions is detected on the track pad 14 after displaying the predetermined image based on one operation position detected on the track pad 14. . In this case, it is possible to notify the user that the number of detected operation positions has changed by changing the display state of the predetermined image.

  The control unit 150 executes moving display for moving the display position of a pointer or marker as a predetermined image, and switches the movement mode in this moving display according to the number of operation positions detected by the track pad 14. For example, when one operation position is detected by the track pad 14, the control unit 150 moves the display position of the predetermined image to the display position corresponding to the detected operation position in steps S21 to S22. When two operation positions are detected on the track pad 14, the control unit 150 moves the display position of the predetermined image to a display position corresponding to an operation position different from the two detected operation positions in steps S37 to S38. Let In addition, when the control unit 150 detects two operation positions with the track pad 14 and then newly detects one operation position with the track pad 14, the control unit 150 displays a predetermined image at a display position corresponding to the new operation position. Move the display position of. In this way, by switching the movement display mode of the predetermined image according to the number of operation positions of the track pad 14, the user can change the movement mode of the pointer and the marker by a simple operation. Thereby, the operativity can be improved. For example, it is possible to change the manner in which the pointer or marker is moved according to the content of the user's operation.

  The control unit 150 switches the display mode of a pointer or marker as a predetermined image according to the number of operation positions detected by the track pad 14. For example, the control unit 150 changes the pointer P1 to the markers Q1 and Q2. For example, the control unit 150 changes the shape of the circular pointer P11 to an ellipse like the pointer P12, and changes the shape of the cross-shaped pointer P21 to the pointer P22. Alternatively, the control unit 150 changes the circular pointer to a pointer P31 having a tail shape. Or the control part 150 changes the brightness | luminance (brightness) of the display of a predetermined image, and changes the transmittance | permeability of a predetermined image. Thereby, the visibility of the predetermined image suitable for operation can be ensured, and the operability can be further improved.

  When a plurality of operation positions are detected by the track pad 14, the control unit 150 sets the position of the base point corresponding to the operation position different from the detected plurality of operation positions as the display control position, and outputs movement data. . Thereby, by performing the contact operation at a plurality of operation positions, it is possible to increase the variation of the operation for moving the display position of the predetermined image, and it is possible to improve the operability.

  The control unit 150 obtains a virtual operation position from a plurality of operation positions detected by the track pad 14, and outputs a display control position corresponding to the virtual operation position. Thereby, a display position can be appropriately moved corresponding to the contact operation in a plurality of operation positions.

  Based on the two operation positions detected by the track pad 14, the control unit 150 outputs a display control position corresponding to an operation position different from the two detected operation positions. Thereafter, when one new operation position is detected by the track pad 14, the control unit 150 outputs a position for display control corresponding to the one new operation position. Thereby, the variation of operation which moves the display position of a predetermined image corresponding to contact operation of a plurality of operation positions can be increased.

  The control unit 150 outputs information specifying the display control position by at least one of the movement direction and the movement amount from the output display position. Accordingly, it is not necessary to strictly associate the size and shape of the display area and the track pad 14, and the present invention can be applied to the track pad 14 having various shapes.

  The image display unit 20 is mounted on the user's head, and the control unit 150 detects an operation position on the track pad 14 configured separately from the image display unit 20. Thereby, when the user wearing the HMD 100 on the head performs an operation on the track pad 14, the manner of movement of the display position of the predetermined image can be changed depending on the number of operation positions. For this reason, the variation of operation can be increased by changing the number of operation positions, and operativity can be improved. For example, even when the track pad 14 is difficult to visually recognize, the moving state of the display position can be arbitrarily operated.

  The HMD 100 is a head-mounted display device that includes an image display unit 20 that transmits external light, and includes a track pad 14 that receives a contact operation. The HMD 100 detects the operation position on the track pad 14 by the control unit 150 and controls the display of the image display unit 20 according to the detected operation. The control unit 150 displays a predetermined image on the image display unit 20 and moves the display position of the predetermined image according to the operation position of the track pad 14. The control unit 150 controls the transmittance of external light with respect to a predetermined image according to the operation position detected by the track pad 14. Thereby, the image of a pointer or a marker can be displayed corresponding to the operation position where the contact operation is performed on the track pad 14, and the transmittance of these images can be changed. For this reason, the appearance of the image displayed by the head-mounted display device with respect to the external light can be changed corresponding to the operation, and the operability can be improved.

  Further, the control unit 150 changes the display mode of the predetermined image corresponding to the operation position detected by the track pad 14. For this reason, in addition to how the pointers and markers are visible with respect to external light, the display mode such as the shape of the image to be displayed can be changed, and the operability can be further improved.

  In addition, this invention is not restricted to the structure of the said embodiment, In the range which does not deviate from the summary, it can be implemented in a various aspect. For example, in the above-described embodiment, the configuration in which the user visually recognizes the outside scene through the display unit is not limited to the configuration in which the right light guide plate 26 and the left light guide plate 28 transmit external light. For example, the present invention can be applied to a display device that displays an image in a state where an outside scene cannot be visually recognized. Specifically, the present invention is applied to a display device that displays a captured image of the camera 61, an image or CG generated based on the captured image, video based on video data stored in advance or video data input from the outside, and the like. Can be applied. This type of display device can include a so-called closed display device in which an outside scene cannot be visually recognized. In addition, as described in the above embodiment, AR display that displays an image superimposed on real space, MR (Mixed Reality) display that combines a captured real space image and a virtual image, or VR (VR that displays a virtual image) It can also be applied to a display device that does not perform processing such as virtual reality display. For example, a display device that displays video data or an analog video signal input from the outside is naturally included as an application target of the present invention.

  Moreover, in the said embodiment, control of the display position when displaying the pointer P and the marker Q in the image display part 20 was illustrated and demonstrated. The present invention is not limited to this, and an image that can be displayed on the image display unit 20 is arbitrary. For example, when AR display is performed in which an image is superimposed on a real object in an outside scene (real space) that is transmitted through the image display unit 20 and visually recognized by the user, the display position of the AR display is adjusted according to the operation of the track pad 14. May be. In this case, the above-described operation may be performed by appropriately switching the one-point operation mode, the multi-point operation mode, and the three-point operation mode according to the number of operation positions of the track pad 14. In this case, the control unit 150 may adjust the display position and display size of the AR display based on the movement data.

  Further, for example, instead of the image display unit 20, another type of image display unit such as an image display unit worn like a hat may be adopted, and an image is displayed corresponding to the left eye of the user. And a display unit that displays an image corresponding to the right eye of the user. The display device of the present invention may be configured as a head mounted display mounted on a vehicle such as an automobile or an airplane. Further, for example, it may be configured as a head-mounted display built in a body protective device such as a helmet. In this case, the part for positioning the position with respect to the user's body and the part positioned with respect to the part can be used as the mounting portion.

Further, in the above embodiment, the image display unit 20 and the control device 10 are separated and connected via the connection cable 40 as an example. However, the control device 10 and the image display unit 20 are integrated. It is also possible to be configured to be mounted on the user's head.
As the control device 10, a notebook computer, a tablet computer, or a desktop computer may be used. The control device 10 may be a game machine, a portable phone, a portable electronic device including a smartphone or a portable media player, other dedicated devices, or the like. Further, the control device 10 may be configured separately from the image display unit 20, and various signals may be transmitted and received between the control device 10 and the image display unit 20 by wireless communication.

In addition, as an optical system that guides image light to the user's eyes, a configuration in which virtual images are formed by the half mirrors 261 and 281 on a part of the right light guide plate 26 and the left light guide plate 28 is illustrated. The present invention is not limited to this, and an image may be displayed on a display region having an area that occupies the entire or most of the right light guide plate 26 and the left light guide plate 28. In this case, a process of reducing the image may be included in the operation of changing the display position of the image.
Furthermore, the optical element of the present invention is not limited to the right light guide plate 26 and the left light guide plate 28 having the half mirrors 261 and 281, and may be any optical component that allows image light to enter the user's eyes. A diffraction grating, a prism, or a holographic display unit may be used.

  Also, at least a part of each functional block shown in FIG. 4 or the like may be realized by hardware, or may be realized by cooperation of hardware and software, as shown in the figure. It is not limited to a configuration in which independent hardware resources are arranged. The program executed by the control unit 150 may be stored in the nonvolatile storage unit 121 or another storage device (not shown) in the control device 10. A program stored in an external device may be acquired and executed via the communication unit 117 or the external connector 184. Moreover, the operation part 110 may be formed as a user interface (UI) among the structures formed in the control apparatus 10. Further, the configuration formed in the control device 10 may be formed in the image display unit 20 in an overlapping manner. For example, a processor similar to the main processor 140 may be disposed in the image display unit 20, or the main processor 140 and the processor of the image display unit 20 included in the control device 10 perform functions separately divided. Also good.

  DESCRIPTION OF SYMBOLS 10 ... Control apparatus (input device), 14 ... Trackpad (operation surface), 20 ... Image display part (display part), 21 ... Right holding part, 22 ... Right display unit, 23 ... Left holding part, 24 ... Left display Unit: 26 ... Right light guide plate, 27 ... Front frame, 28 ... Left light guide plate, 30 ... Headset, 40 ... Connection cable, 61 ... Camera, 63 ... Microphone, 65 ... Illuminance sensor, 67 ... LED indicator, 100 ... HMD (display device, head-mounted display device), 110 ... operation section, 111 ... 6-axis sensor, 113 ... magnetic sensor, 115 ... GPS, 117 ... communication section, 118 ... memory, 119 ... vibrator, 120 ... controller board 121 ... Non-volatile storage unit 122 ... Storage unit 123 ... Setting data 124 ... Content data 127 ... Operation setting data 130 ... Power supply unit 132 ... Battery 134 ... Power supply control circuit 140 ... Main processor 143 ... Operating system 145 ... Image processing unit 149 ... Imaging control unit 150 ... Control unit 152 ... Operation detection control unit (operation information generation unit) 180 ... Audio codec, 182 ... Audio interface, 184 ... External connector, 186 ... External memory interface, 188 ... USB connector, 192 ... Sensor hub, 194 ... FPGA, 196 ... Interface, 211,231 ... Interface, 213,233 ... Reception 215 ... EEPROM, 217, 239 ... temperature sensor, 221,241 ... OLED unit, 235 ... 6-axis sensor, 237 ... magnetic sensor.

Claims (17)

An operation surface that accepts a contact operation;
A display unit having a display area for displaying an image;
A control unit that detects an operation position on the operation surface and controls display of the display unit according to the detected operation;
The control unit displays a predetermined image on the display unit, moves the display position of the predetermined image according to the operation position of the operation surface, and detects one operation position on the operation surface; Changing the association between the operation position and the display position of the predetermined image when the operation position is detected;
A display device.   When the control unit detects two operation positions on the operation surface when moving the display position of the predetermined image according to the operation position of the operation surface, the control unit sets the operation position different from the detected two operation positions. The display device according to claim 1, wherein the display position of the predetermined image is moved to a corresponding display position.   The control unit moves the display position of the predetermined image to a display position corresponding to an operation position different from the two detected operation positions based on the two operation positions detected on the operation surface, and then performs the operation. 3. The display device according to claim 2, wherein when one operation position is newly detected on the screen, the display position of the predetermined image is moved to a display position corresponding to the new operation position.   When the control unit detects an operation at two operation positions on the operation surface after displaying the predetermined image based on one operation position detected on the operation surface, the control unit stops displaying the predetermined image. The display device according to claim 1, wherein:   The control unit executes a movement display for moving the display position of the predetermined image according to the operation position of the operation surface, and switches the mode of the movement display according to the number of operation positions detected on the operation surface. The display device according to claim 1.   The display device according to claim 1, wherein the control unit switches a display mode of the predetermined image according to the number of operation positions detected on the operation surface. An operation surface that accepts a contact operation;
A display unit having a display area for displaying an image;
A control unit that detects an operation position on the operation surface, outputs a display control position based on the detected operation position, and displays an image on the display unit based on the display control position; and
The control unit changes the association between the detected operation position and the display control position when detecting one operation position and when detecting a plurality of operation positions;
A display device.   The control unit, when detecting a plurality of operation positions on the operation surface, outputs the display control position corresponding to an operation position different from the detected plurality of operation positions. 7. Display position according to 7.   9. The control unit obtains a virtual operation position from a plurality of operation positions detected on the operation surface, and outputs the display control position corresponding to the virtual operation position. Display position. The control unit outputs a position for display control corresponding to an operation position different from the two detected operation positions based on the two operation positions detected on the operation surface;
The position for display control corresponding to the new one operation position is then output when one new operation position is detected on the operation surface. Display device.   The display device according to claim 9, wherein the control unit outputs information specifying the display control position by at least one of a movement direction and a movement amount from the output display position. . The display unit is attached to a user's head,
The control unit detects an operation position on the operation surface configured as a separate body from the display unit;
The display device according to claim 1, wherein: A head-mounted display device including a display unit that transmits external light,
An operation surface that accepts a contact operation;
A control unit that detects an operation position on the operation surface and controls display of the display unit according to the detected operation;
The control unit displays a predetermined image on the display unit, moves a display position of the predetermined image in accordance with an operation position on the operation surface, and applies to the predetermined image in accordance with an operation position detected on the operation surface. Controlling the transmittance of the external light,
A head-mounted display device characterized by the above. The control unit changes a display mode of the predetermined image corresponding to an operation position detected on the operation surface;
The head-mounted display device according to claim 13. An operation information generation unit that receives a contact operation on the operation surface, detects an operation position on the operation surface, and generates information on a processing position corresponding to the operation position based on the detected operation position;
The operation information generation unit changes the association between the operation position and the processing position when a single operation position is detected on the operation surface and when a plurality of operation positions are detected.
An input device characterized by. Controlling a display device comprising a display unit having a display area for displaying an image;
A predetermined image is displayed by the display unit,
Detect the operation position on the operation surface that accepts contact operation,
The display position of the predetermined image is moved according to the operation position detected on the operation surface,
Changing the association between the operation position and the display position of the predetermined image when one operation position is detected on the operation surface and when a plurality of operation positions on the operation surface are detected;
A control method of a display device characterized by the above. A computer-executable program for controlling a display device including a display unit having a display area for displaying an image,
A predetermined image is displayed by the display unit,
Detect the operation position on the operation surface that accepts contact operation,
The display position of the predetermined image is moved according to the operation position detected on the operation surface,
A program for changing the association between the operation position and the display position of the predetermined image when one operation position is detected on the operation surface and when a plurality of operation positions on the operation surface are detected.

Images