JP2017182460A - Head-mounted type display device, method for controlling head-mounted type display device, and computer program - Google Patents

Abstract

PROBLEM TO BE SOLVED: To provide a head-mounted type display device capable of displaying a virtual object in a position intended by a user and with an inclination intended by the user.SOLUTION: A head-mounted type display device 100 includes an image display part 20, acquisition parts 61 and 62 which acquire the information of a real space, and a control part 10 which allows the image display part to form a virtual image including the virtual object which is an object which does not actually exist in the real space. The control part analyzes the acquired information of the real space, to obtain the position and inclination of a reference surface as a plane which exists in the real space and satisfies a predetermined condition, and changes the position and inclination of the virtual object visually recognized by the user on the basis of the obtained position and inclination of the reference surface.SELECTED DRAWING: Figure 1

Description

  The present invention relates to a head-mounted display device.

  2. Description of the Related Art A head-mounted display device that is worn on a user's head and forms a virtual image in the user's visual field region is known. In addition, a technique called augmented reality (AR) that adds information to a real environment using a computer is known. In Patent Literature 1, using such augmented reality technology, a virtual image representing a virtual input surface is formed in the user's visual field area, and input using the virtual input surface is enabled. A head-mounted display device is described.

JP-T-2015-519673

  On the other hand, in recent years, it has been proposed to use a transmissive head-mounted display device in which the user's view is not blocked in a worn state in various scenes such as work and life. For example, in a head-mounted display device used in the manufacturing industry or the like, the virtual input surface described above and the virtual guide surface representing the work procedure and work contents to be performed by the user are augmented reality. Presented using technology. However, in such a head-mounted display device, there is a problem that the virtual input surface and the guide surface are displayed at a position and an inclination that are not intended by the user, which is unusable. A virtual object that forms a virtual input surface or a virtual guide surface is also referred to as a “virtual object”.

  For this reason, a head-mounted display device capable of displaying a virtual object at a position and inclination intended by the user has been desired.

  SUMMARY An advantage of some aspects of the invention is to solve at least a part of the problems described above, and the invention can be implemented as the following forms.

(1) According to one aspect of the present invention, a head-mounted display device is provided. The head-mounted display device includes: an image display unit that allows a user wearing the head-mounted display device to visually recognize a virtual image; an acquisition unit that acquires real space information; A control unit that causes the image display unit to form the virtual image including a virtual object that is an object; and the control unit is in the real space by analyzing the acquired information of the real space In addition, the position and inclination of a reference plane, which is a plane satisfying a predetermined condition, are determined in the user's field of view; the position and inclination of the virtual object visually recognized by the user are determined. And change based on tilt.
According to this form of the head-mounted display device, the control unit changes the position and inclination of the virtual object visually recognized by the user based on the position and inclination of the reference plane in the real space. For this reason, the user can adjust the position and inclination of the virtual object visually recognized as a virtual image as desired by adjusting the position and inclination of the reference plane.

(2) In the head-mounted display device of the above aspect; the control unit may make the position and inclination of the virtual object visually recognized by the user the same as the position and inclination of the reference plane.
According to this form of the head-mounted display device, the position and inclination of the virtual object that the user visually recognizes as a virtual image are the same as the position and inclination of the reference plane in the real space. Therefore, the user can intuitively adjust the position and inclination of the virtual object.

(3) The head-mounted display device of the above aspect further includes: a line-of-sight detection unit that acquires the direction of the line of sight of the user; the acquisition unit; You may abbreviate | omit acquisition of the information of the said real space which does not correspond to the direction of the said gaze acquired by the gaze detection part.
According to the head-mounted display device of this aspect, the acquisition unit omits acquisition of real space information that does not correspond to the direction of the user's line of sight, and thus the amount of information in the real space information acquired by the acquisition unit Can be reduced. As a result, the processing load at the time of analysis by the control unit can be reduced.

(4) In the head-mounted display device according to the above aspect; the control unit; and; when detecting a predetermined first action by the user, the reference plane in the field of view of the user The position and inclination may be obtained again, and the changed position and inclination of the virtual object may be changed again based on the newly obtained position and inclination of the reference plane.
According to the head-mounted display device of this aspect, the control unit re-changes the position and inclination of the virtual object triggered by the first action, so that the convenience for the user can be improved.

(5) In the head-mounted display device according to the above aspect; the control unit may further: change the display mode of the virtual object when detecting a second action predetermined by the user Good.
According to the head-mounted display device of this aspect, the control unit changes the display mode of the virtual object in response to the second operation, so that convenience for the user can be improved.

(6) In the head-mounted display device according to the above aspect; the control unit; a human hand, the user's hand, an object having a predetermined shape, and an image including information having a predetermined pattern; At least one of them may be regarded as an object including the reference plane.
According to the head-mounted display device of this aspect, the control unit includes at least one of a human hand, a user's own hand, an object having a predetermined shape, and an image including information having a predetermined pattern. It is possible to improve the processing efficiency by regarding the object as an object including a reference plane.

(7) In the head-mounted display device according to the above aspect; the plane that satisfies the predetermined condition; has a flatness within a predetermined first range as a result of three-dimensional recognition; It may be a surface having an area equal to or larger than a predetermined second value.
According to the head-mounted display device of this aspect, the control unit is required to have flatness within the first range as a result of the three-dimensional recognition. By appropriately setting the first range, the control unit can regard a surface with some unevenness such as a human hand as a reference surface, and has a large unevenness such as a human face. By excluding the surface from the reference surface, erroneous recognition can be suppressed. Moreover, a control part is on condition that an area is more than 2nd predetermined value. By appropriately setting the second value, the control unit can suppress misrecognition by excluding, for example, a hand of a third party located far away from the user from the reference plane.

  A plurality of constituent elements of each embodiment of the present invention described above are not essential, and some or all of the effects described in the present specification are to be solved to solve part or all of the above-described problems. In order to achieve the above, it is possible to appropriately change, delete, replace with a new component, and partially delete the limited contents of some of the plurality of components. In order to solve some or all of the above-described problems or achieve some or all of the effects described in this specification, technical features included in one embodiment of the present invention described above. A part or all of the technical features included in the other aspects of the present invention described above may be combined to form an independent form of the present invention.

  The present invention can be realized in various modes. For example, a head-mounted display device, a head-mounted display device control method, an image display system including a head-mounted display device, and the like. The present invention can be realized in the form of a computer program for realizing the functions of the apparatus, method, and system, a server apparatus for distributing the computer program, a storage medium storing the computer program, and the like.

It is a figure which shows the external appearance structure of HMD in one Embodiment of this invention. It is a principal part top view which shows the structure of the optical system with which an image display part is provided. It is a figure which shows the principal part structure of the image display part seen from the user. It is a figure for demonstrating the angle of view of a camera. It is a block diagram which shows the structure of HMD functionally. It is a block diagram which shows functionally the structure of the control function part of HMD. It is a figure explaining a default display surface. It is a flowchart which shows the procedure of a display surface setting process. It is a figure explaining step S104-S108 of a display surface setting process. It is a figure explaining step S110 of a display surface setting process. It is a figure explaining the display surface after correction | amendment. It is a figure explaining re-change of the display surface of a virtual object. It is a figure explaining re-change of the display surface of a virtual object. It is a figure explaining the change of the display mode of a virtual object. It is a principal part top view which shows the structure of the optical system with which the image display part of a modification is provided.

A. Embodiment:
A-1. HMD configuration:
FIG. 1 is a diagram showing an external configuration of an HMD (Head Mounted Display) 100 according to an embodiment of the present invention. HMD100 is a display provided with the image display part 20 (display part) which makes a user visually recognize a virtual image in the state with which the user's head was mounted | worn, and the control apparatus 10 (control part) which controls the image display part 20. Device. The control device 10 includes various input devices for acquiring a user's operation, and functions as a controller for the user to operate the HMD 100.

  The image display unit 20 is a wearing body that is worn on the user's head, and has a glasses shape in the present embodiment. The image display unit 20 includes a right display unit 22, a left display unit 24, a right light guide plate 26, and a left light guide plate 28 in a main body having a right holding unit 21, a left holding unit 23, and a front frame 27. With.

  Each of the right holding unit 21 and the left holding unit 23 extends rearward from both end portions of the front frame 27 and holds the image display unit 20 on the user's head like a temple of glasses. Here, of both ends of the front frame 27, an end located on the right side of the user in the mounted state of the image display unit 20 is defined as an end ER, and an end located on the left side of the user is defined as an end EL. To do. The right holding unit 21 extends from the end ER of the front frame 27 to a position corresponding to the right side of the user when the image display unit 20 is worn. The left holding part 23 is provided to extend from the end EL of the front frame 27 to a position corresponding to the left side of the user when the image display part 20 is worn.

  The right light guide plate 26 and the left light guide plate 28 are provided on the front frame 27. The right light guide plate 26 is positioned in front of the right eye of the user when the image display unit 20 is mounted, and causes the right eye to visually recognize the image. The left light guide plate 28 is positioned in front of the left eye of the user when the image display unit 20 is mounted, and causes the left eye to visually recognize the image.

  The front frame 27 has a shape in which one end of the right light guide plate 26 and one end of the left light guide plate 28 are connected to each other. This connection position corresponds to the position between the eyebrows of the user when the image display unit 20 is mounted. The front frame 27 may be provided with a nose pad portion that comes into contact with the user's nose when the image display unit 20 is mounted at a connection position between the right light guide plate 26 and the left light guide plate 28. In this case, the image display unit 20 can be held on the user's head by the nose pad, the right holding unit 21 and the left holding unit 23. Further, a belt that is in contact with the back of the user's head when the image display unit 20 is mounted may be connected to the right holding unit 21 and the left holding unit 23. In this case, the image display unit 20 can be firmly held on the user's head by the belt.

  The right display unit 22 displays an image by the right light guide plate 26. The right display unit 22 is provided in the right holding unit 21 and is located in the vicinity of the right side of the user when the image display unit 20 is worn. The left display unit 24 displays an image by the left light guide plate 28. The left display unit 24 is provided in the left holding unit 23 and is located in the vicinity of the user's left head when the image display unit 20 is worn. The right display unit 22 and the left display unit 24 are also collectively referred to as “display driving unit”.

  The right light guide plate 26 and the left light guide plate 28 of the present embodiment are optical units (for example, prisms) formed of a light transmissive resin or the like, and use image light output from the right display unit 22 and the left display unit 24. Lead to the eyes of the person. A dimming plate may be provided on the surfaces of the right light guide plate 26 and the left light guide plate 28. The light control plate is a thin plate-like optical element having different transmittance depending on the wavelength region of light, and functions as a so-called wavelength filter. For example, the light control plate is disposed so as to cover the surface of the front frame 27 (the surface opposite to the surface facing the user's eyes). By appropriately selecting the optical characteristics of the light control plate, the transmittance of light in an arbitrary wavelength region such as visible light, infrared light, and ultraviolet light can be adjusted. The amount of external light incident on the light plate 28 and transmitted through the right light guide plate 26 and the left light guide plate 28 can be adjusted.

  The image display unit 20 guides the image light generated by the right display unit 22 and the left display unit 24 to the right light guide plate 26 and the left light guide plate 28, respectively, and causes the user to visually recognize a virtual image with this image light (this is referred to as “ Also called “display image”). When external light is incident on the user's eyes through the right light guide plate 26 and the left light guide plate 28 from the front of the user, image light constituting the virtual image and external light are incident on the user's eyes. To do. For this reason, the visibility of the virtual image for the user is affected by the intensity of external light.

  Therefore, for example, by attaching a light control plate to the front frame 27 and appropriately selecting or adjusting the optical characteristics of the light control plate, it is possible to adjust the visibility of the virtual image. In a typical example, it is possible to select a dimming plate having a light transmittance that allows a user wearing the HMD 100 to visually recognize at least the outside scenery. Use of the light control plate can be expected to protect the right light guide plate 26 and the left light guide plate 28, and to suppress damage to the right light guide plate 26 and the left light guide plate 28, adhesion of dirt and the like. The light control plate may be detachable from the front frame 27 or each of the right light guide plate 26 and the left light guide plate 28. In addition, a plurality of types of light control plates may be exchanged and removable, or the light control plates may be omitted.

  The camera 61 is disposed on the front frame 27 of the image display unit 20. The camera 61 is provided on the front surface of the front frame 27 at a position that does not block outside light that passes through the right light guide plate 26 and the left light guide plate 28. In the example of FIG. 1, the camera 61 is disposed on the end ER side of the front frame 27. The camera 61 may be disposed on the end EL side of the front frame 27 or may be disposed at a connection portion between the right light guide plate 26 and the left light guide plate 28.

  The camera 61 is a digital camera that includes an imaging element such as a CCD or CMOS, an imaging lens, and the like. The camera 61 of this embodiment is a monocular camera, but a stereo camera may be adopted. The camera 61 captures at least a part of the outside scene (real space) in the front side direction of the HMD 100, in other words, in the visual field direction visually recognized by the user when the image display unit 20 is mounted. In other words, the camera 61 images a range or direction that overlaps the user's field of view, and images a direction that the user visually recognizes. The angle of view of the camera 61 can be set as appropriate. In the present embodiment, the width of the angle of view of the camera 61 is set so that the entire field of view of the user that the user can see through the right light guide plate 26 and the left light guide plate 28 is imaged. The camera 61 executes imaging according to the control of the control unit 150 (FIG. 5), and outputs the obtained imaging data to the control unit 150. The camera 61 functions as an “acquisition unit” by cooperating with the control unit 150.

  Similarly to the camera 61, the inner camera 62 is a digital camera that includes an imaging element such as a CCD or CMOS, an imaging lens, and the like. The inner camera 62 images the inner direction of the HMD 100, in other words, the direction facing the user when the image display unit 20 is worn. The inner camera 62 of the present embodiment includes an inner camera for imaging the right eye of the user and an inner camera for imaging the left eye. The width of the angle of view of the inner camera 62 can be set as appropriate similarly to the camera 61. However, the width of the angle of view of the inner camera 62 is preferably set in a range in which the entire right eye or left eye of the user can be imaged. Similarly to the camera 61, the inner camera 62 executes imaging in accordance with the control of the control unit 150 (FIG. 5), and outputs the obtained imaging data to the control unit 150. The inner camera 62 functions as a “line-of-sight detection unit” in cooperation with the control unit 150.

  The HMD 100 may include a distance measuring sensor that detects a distance to a measurement object positioned in a preset measurement direction. The distance measuring sensor can be disposed, for example, at a connection portion between the right light guide plate 26 and the left light guide plate 28 of the front frame 27. The measurement direction of the distance measuring sensor can be the front side direction of the MD 100 (the direction overlapping the imaging direction of the camera 61). The distance measuring sensor can be composed of, for example, a light emitting unit such as an LED or a laser diode, and a light receiving unit that receives reflected light that is reflected from the light to be measured by the light source. In this case, the distance is obtained by triangular distance measurement processing or distance measurement processing based on a time difference. The distance measuring sensor may be configured by, for example, a transmission unit that emits ultrasonic waves and a reception unit that receives ultrasonic waves reflected by the measurement object. In this case, the distance is obtained by distance measurement processing based on the time difference. The distance measuring sensor is controlled by the control unit 150 similarly to the camera 61, and outputs the detection result to the control unit 150.

  FIG. 2 is a principal plan view showing the configuration of the optical system provided in the image display unit 20. For convenience of explanation, FIG. 2 shows a user's right eye RE and left eye LE. As shown in FIG. 2, the right display unit 22 and the left display unit 24 are configured symmetrically.

  As a configuration for allowing the right eye RE to visually recognize a virtual image, the right display unit 22 includes an OLED (Organic Light Emitting Diode) unit 221 and a right optical system 251. The OLED unit 221 emits image light. The right optical system 251 includes a lens group and the like, and guides the image light L emitted from the OLED unit 221 to the right light guide plate 26.

  The OLED unit 221 includes an OLED panel 223 and an OLED drive circuit 225 that drives the OLED panel 223. The OLED panel 223 is a self-luminous display panel configured by light emitting elements that emit light by organic electroluminescence and emit color lights of R (red), G (green), and B (blue). In the OLED panel 223, a plurality of pixels each having a unit including one R, G, and B element as one pixel are arranged in a matrix.

  The OLED drive circuit 225 selects and energizes the light emitting elements included in the OLED panel 223 under the control of the control unit 150 (FIG. 5), and causes the light emitting elements to emit light. The OLED drive circuit 225 is fixed to the back surface of the OLED panel 223, that is, the back side of the light emitting surface by bonding or the like. The OLED drive circuit 225 may be configured by a semiconductor device that drives the OLED panel 223, for example, and may be mounted on a substrate that is fixed to the back surface of the OLED panel 223. A temperature sensor 217 described later is mounted on this substrate. Note that the OLED panel 223 may employ a configuration in which light emitting elements that emit white light are arranged in a matrix, and color filters corresponding to the colors R, G, and B are stacked. Further, an OLED panel 223 having a WRGB configuration including a light emitting element that emits W (white) light in addition to the light emitting elements that respectively emit R, G, and B color light may be employed.

  The right optical system 251 includes a collimator lens that converts the image light L emitted from the OLED panel 223 into a parallel light flux. The image light L converted into a parallel light beam by the collimator lens enters the right light guide plate 26. A plurality of reflecting surfaces that reflect the image light L are formed in an optical path that guides light inside the right light guide plate 26. The image light L is guided to the right eye RE side through a plurality of reflections inside the right light guide plate 26. A half mirror 261 (reflection surface) located in front of the right eye RE is formed on the right light guide plate 26. The image light L is reflected by the half mirror 261 and then emitted from the right light guide plate 26 to the right eye RE. The image light L forms an image with the retina of the right eye RE, thereby allowing the user to visually recognize a virtual image.

  As a configuration for causing the left eye LE to visually recognize a virtual image, the left display unit 24 includes an OLED unit 241 and a left optical system 252. The OLED unit 241 emits image light. The left optical system 252 includes a lens group and the like, and guides the image light L emitted from the OLED unit 241 to the left light guide plate 28. The OLED unit 241 includes an OLED panel 243 and an OLED drive circuit 245 that drives the OLED panel 243. Details of each part are the same as those of the OLED unit 221, the OLED panel 223, and the OLED drive circuit 225. A temperature sensor 239 is mounted on the substrate fixed to the back surface of the OLED panel 243. The details of the left optical system 252 are the same as those of the right optical system 251.

  According to the configuration described above, the HMD 100 can function as a see-through display device. That is, the image light L reflected by the half mirror 261 and the external light OL transmitted through the right light guide plate 26 enter the right eye RE of the user. The image light L reflected by the half mirror 281 and the external light OL transmitted through the left light guide plate 28 enter the left eye LE of the user. As described above, the HMD 100 causes the image light L of the image processed inside and the external light OL to overlap and enter the user's eyes. As a result, for the user, the outside scene (real world) can be seen through the right light guide plate 26 and the left light guide plate 28, and the virtual image by the image light L is visually recognized so as to overlap the outside scene.

  The half mirror 261 and the half mirror 281 function as an “image extraction unit” that reflects the image light output from the right display unit 22 and the left display unit 24 and extracts an image. Further, the right optical system 251 and the right light guide plate 26 are collectively referred to as a “right light guide”, and the left optical system 252 and the left light guide plate 28 are also collectively referred to as a “left light guide”. The configurations of the right light guide unit and the left light guide unit are not limited to the above-described examples, and any method can be used as long as a virtual image is formed in front of the user's eyes using image light. For example, a diffraction grating or a transflective film may be used for the right light guide and the left light guide.

  In FIG. 1, the control device 10 and the image display unit 20 are connected by a connection cable 40. The connection cable 40 is detachably connected to a connector provided in the lower part of the control device 10, and is connected to various circuits inside the image display unit 20 from the tip of the left holding unit 23. The connection cable 40 includes a metal cable or an optical fiber cable that transmits digital data. The connection cable 40 may further include a metal cable that transmits analog data. A connector 46 is provided in the middle of the connection cable 40.

  The connector 46 is a jack for connecting a stereo mini-plug, and the connector 46 and the control device 10 are connected by a line for transmitting an analog audio signal, for example. Connected to the connector 46 are a right earphone 32 and a left earphone 34 constituting a stereo headphone, and a headset 30 having a microphone 63. For example, as shown in FIG. 1, the microphone 63 is arranged so that the sound collection unit of the microphone 63 faces the line of sight of the user. The microphone 63 collects sound and outputs a sound signal to the sound interface 182 (FIG. 5). The microphone 63 may be a monaural microphone or a stereo microphone, and may be a directional microphone or an omnidirectional microphone.

  The control device 10 includes a button 11, an LED indicator 12, a cross key 13, a track pad 14, an up / down key 15, a changeover switch 16, and a power switch 18 as an input / output interface for a user.

  The button 11 includes a menu key, a home key, a return key, and the like for operating an OS (Operating System) 143 (FIG. 6) executed by the control device 10. The button 11 of this embodiment is a type of button that is displaced by a pressing operation of these keys and switches. The LED indicator 12 is turned on or off according to the operating state of the HMD 100. The cross key 13 detects a touch or press operation on keys corresponding to the upper, lower, left, right, and center portions, and outputs a signal corresponding to the detected content. In the cross key 13 of the present embodiment, a circular determination button is arranged at the center, and an up button, a down button, a left button, and a right button are arranged in the up, down, left and right directions, respectively. Various modes can be adopted for each button, for example, a capacitance type button or a mechanical contact type button.

  The track pad 14 has an operation surface for detecting a contact operation, detects a contact operation on the operation surface, and outputs a signal corresponding to the detected content. Various methods such as an electrostatic method, a pressure detection method, and an optical method can be adopted as a detection method for the contact operation. The up / down key 15 detects an instruction operation for increasing / decreasing the volume output from the earphone of the headset 30 or an instruction operation for increasing / decreasing the luminance in the image display unit 20. The changeover switch 16 detects a changeover operation of an instruction target (volume, luminance) of the up / down key 15. The power switch 18 detects an on / off (ON, OFF) switching operation of the power of the HMD 100. As the power switch 18, for example, a slide switch can be adopted.

  FIG. 3 is a diagram illustrating a main configuration of the image display unit 20 as viewed from the user. In FIG. 3, the connection cable 40, the right earphone 32, and the left earphone 34 are not shown. In the state of FIG. 3, the back sides of the right light guide plate 26 and the left light guide plate 28 can be visually recognized, the half mirror 261 for irradiating the right eye RE with image light, and the image light for irradiating the left eye LE with image light. The half mirror 281 can be visually recognized as a substantially rectangular area. The user views the outside scene through the entire left and right light guide plates 26 and 28 including the half mirrors 261 and 281 and visually recognizes a rectangular display image at the position of the half mirrors 261 and 281.

  FIG. 4 is a diagram for explaining the angle of view of the camera 61. In FIG. 4, the camera 61 and the user's right eye RE and left eye LE are schematically shown in plan view, and the angle of view (imaging range) of the camera 61 is indicated by C. Note that the angle of view C of the camera 61 extends in the horizontal direction as shown in the figure, and also in the vertical direction as in a general digital camera.

  As described above, the camera 61 is arranged at the right end of the image display unit 20 and images the direction of the user's line of sight (that is, the front of the user). For this reason, the optical axis of the camera 61 is a direction including the line-of-sight directions of the right eye RE and the left eye LE. The outside scene that the user can visually recognize in a state where the HMD 100 is worn is not always at infinity. For example, when the user gazes at the object OB with both eyes, the user's line of sight is directed toward the object OB as indicated by reference numerals RD and LD in the figure. In this case, the distance from the user to the object OB is often about 30 cm to 10 m, and more often 1 m to 4 m. Therefore, for the HMD 100, an upper limit and a lower limit of the distance from the user to the object OB during normal use may be determined. This standard may be obtained in advance and preset in the HMD 100, or may be set by the user. The optical axis and the angle of view of the camera 61 are set so that the object OB is included in the angle of view when the distance to the object OB during normal use corresponds to the set upper and lower limits. It is preferred that

  In general, a human viewing angle is approximately 200 degrees in the horizontal direction and approximately 125 degrees in the vertical direction. Among them, the effective visual field with excellent information receiving ability is about 30 degrees in the horizontal direction and about 20 degrees in the vertical direction. A stable gaze field in which a gaze point that a person gazes at appears quickly and stably is set to 60 to 90 degrees in the horizontal direction and about 45 to 70 degrees in the vertical direction. In this case, when the gazing point is the object OB (FIG. 4), the effective visual field is about 30 degrees in the horizontal direction and about 20 degrees in the vertical direction around the lines of sight RD and LD. Further, the stable focus is about 60 to 90 degrees in the horizontal direction and about 45 to 70 degrees in the vertical direction. The actual field of view that the user permeates through the image display unit 20 and permeates through the right light guide plate 26 and the left light guide plate 28 is referred to as a real field of view (FOV). The real field of view is narrower than the viewing angle and stable focus field, but wider than the effective field of view.

  The angle of view C of the camera 61 of the present embodiment is set so that a wider range than the user's visual field can be imaged. The angle of view C of the camera 61 is preferably set so that at least a range wider than the effective visual field of the user can be captured, and more preferably set so that a range wider than the actual visual field can be captured. It is more preferable that the angle of view C of the camera 61 is set so as to be able to image a wider range than the stable viewing field of the user, and is set so as to be able to image a range wider than the viewing angle of both eyes of the user. Most preferred. For this reason, the camera 61 may include a so-called wide-angle lens as an imaging lens so that a wide angle of view can be captured. The wide-angle lens may include a lens called an ultra-wide-angle lens or a quasi-wide-angle lens. The camera 61 may include a single focus lens, a zoom lens, or a lens group including a plurality of lenses.

  FIG. 5 is a block diagram functionally showing the configuration of the HMD 100. The control device 10 includes a main processor 140 that controls the HMD 100 by executing a program, a storage unit, an input / output unit, sensors, an interface, and a power supply unit 130. The storage unit, input / output unit, sensors, interface, and power supply unit 130 are connected to the main processor 140. The main processor 140 is mounted on the controller board 120 built in the control device 10.

  The storage unit includes a memory 118 and a nonvolatile storage unit 121. The memory 118 constitutes a work area that temporarily stores a computer program executed by the main processor 140 and data to be processed. The nonvolatile storage unit 121 is configured by a flash memory or an eMMC (embedded Multi Media Card). The nonvolatile storage unit 121 stores a computer program executed by the main processor 140 and various data processed by the main processor 140. In the present embodiment, these storage units are mounted on the controller board 120.

  The input / output unit includes a track pad 14 and an operation unit 110. The operation unit 110 includes the button 11, the LED indicator 12, the cross key 13, the track pad 14, the up / down key 15, the changeover switch 16, and the power switch 18. The main processor 140 controls these input / output units and acquires signals output from the input / output units.

  The sensors include a six-axis sensor 111, a magnetic sensor 113, and a GPS (Global Positioning System) 115. The 6-axis sensor 111 is a motion sensor (inertial sensor) including a 3-axis acceleration sensor and a 3-axis gyro (angular velocity) sensor. The 6-axis sensor 111 may employ an IMU (Inertial Measurement Unit) in which these sensors are modularized. The magnetic sensor 113 is, for example, a triaxial geomagnetic sensor. The GPS 115 includes a GPS antenna (not shown), receives a radio signal transmitted from a GPS satellite, and detects the coordinates of the current position of the control device 10. These sensors (six-axis sensor 111, magnetic sensor 113, GPS 115) output detection values to the main processor 140 according to a sampling frequency designated in advance. The timing at which each sensor outputs a detection value may be in accordance with an instruction from the main processor 140.

  The interface includes a communication unit 117, an audio codec 180, an external connector 184, an external memory interface 186, a USB (Universal Serial Bus) connector 188, a sensor hub 192, an FPGA 194, and an interface 196. Yes. These function as an interface with the outside. The communication unit 117 performs wireless communication between the HMD 100 and an external device. The communication unit 117 includes an antenna (not shown), an RF circuit, a baseband circuit, a communication control circuit, and the like, or is configured as a device in which these are integrated. The communication unit 117 performs wireless communication based on a standard such as a wireless LAN including Bluetooth (registered trademark) and Wi-Fi, for example.

  The audio codec 180 is connected to the audio interface 182 and encodes / decodes an audio signal input / output via the audio interface 182. The audio codec 180 may have a function of processing audio input / output via the built-in headset 30. Furthermore, the audio codec 180 performs an A / D converter that converts an analog audio signal into digital audio data and vice versa for an audio signal that is input / output via the built-in headset 30. A D / A converter may be provided. The A / D converter converts an analog audio signal input via the microphone 63 into digital audio data and outputs the digital audio data to the main processor 140. The D / A converter converts the digital audio data output from the main processor 140 into an analog audio signal and outputs the analog audio signal to the right earphone 32 and the left earphone 34. The audio interface 182 is an interface for inputting / outputting audio signals to / from the headset 30.

  The external connector 184 is a connector for connecting an external device (for example, a personal computer, a smart phone, a game device, etc.) that communicates with the main processor 140 to the main processor 140. The external device connected to the external connector 184 can be a content supply source, and can be used for debugging a computer program executed by the main processor 140 and collecting an operation log of the HMD 100. The external connector 184 can employ various modes. As the external connector 184, for example, an interface corresponding to a wired connection such as a USB interface, a micro USB interface, a memory card interface, or an interface corresponding to a wireless connection such as a wireless LAN interface or a Bluetooth interface can be adopted.

  The external memory interface 186 is an interface to which a portable memory device can be connected. The external memory interface 186 includes, for example, a memory card slot in which a card-type recording medium is mounted and data is read and written, and an interface circuit. The size, shape, standard, etc. of the card type recording medium can be selected as appropriate. The USB connector 188 is an interface that can connect a memory device, a smartphone, a personal computer, or the like that conforms to the USB standard. The USB connector 188 includes, for example, a connector conforming to the USB standard and an interface circuit. The size, shape, USB standard version, etc. of the USB connector 188 can be selected as appropriate.

  Further, the HMD 100 includes a vibrator 19. The vibrator 19 includes a motor (not shown), an eccentric rotor, and the like, and utters vibrations under the control of the main processor 140. The HMD 100 causes the vibrator 19 to generate a vibration with a predetermined vibration pattern when, for example, an operation on the operation unit 110 is detected or when the power of the HMD 100 is turned on / off.

  The sensor hub 192 and the FPGA 194 are connected to the image display unit 20 via an interface (I / F) 196. The sensor hub 192 acquires detection values of various sensors included in the image display unit 20 and outputs them to the main processor 140. The FPGA 194 executes processing of data transmitted / received between the main processor 140 and each unit of the image display unit 20 and transmission via the interface 196. The interface 196 is connected to each of the right display unit 22 and the left display unit 24 of the image display unit 20. In the example of the present embodiment, the connection cable 40 is connected to the left holding unit 23, the wiring connected to the connection cable 40 is laid inside the image display unit 20, and each of the right display unit 22 and the left display unit 24 is It is connected to the interface 196 of the control device 10.

  The power supply unit 130 includes a battery 132 and a power supply control circuit 134. The power supply unit 130 supplies power for operating the control device 10. The battery 132 is a rechargeable battery. The power control circuit 134 detects the remaining capacity of the battery 132 and controls the charging of the OS 143. The power supply control circuit 134 is connected to the main processor 140 and outputs a detected value of the remaining capacity of the battery 132 and a detected value of the voltage of the battery 132 to the main processor 140. Note that power may be supplied from the control device 10 to the image display unit 20 based on the power supplied by the power supply unit 130. The power supply state from the power supply unit 130 to each unit of the control device 10 and the image display unit 20 may be configured to be controllable by the main processor 140.

  The right display unit 22 includes a display unit substrate 210, an OLED unit 221, a camera 61, an illuminance sensor 65, an LED indicator 67, and a temperature sensor 217. On the display unit substrate 210, an interface (I / F) 211 connected to the interface 196, a receiving unit (Rx) 213, and an EEPROM (Electrically Erasable Programmable Read-Only Memory) 215 are mounted. The receiving unit 213 receives data input from the control device 10 via the interface 211. When receiving the image data of the image to be displayed on the OLED unit 221, the receiving unit 213 outputs the received image data to the OLED drive circuit 225 (FIG. 2).

  The EEPROM 215 stores various data in a manner that can be read by the main processor 140. The EEPROM 215 stores, for example, data on the light emission characteristics and display characteristics of the OLED units 221 and 241 of the image display unit 20, data on the sensor characteristics of the right display unit 22 or the left display unit 24, and the like. Specifically, for example, parameters relating to gamma correction of the OLED units 221 and 241, data for compensating detection values of temperature sensors 217 and 239 described later, and the like are stored. These data are generated by the factory inspection of the HMD 100 and are written in the EEPROM 215. After shipment, the main processor 140 reads the data in the EEPROM 215 and uses it for various processes.

  The camera 61 executes imaging in accordance with a signal input via the interface 211 and outputs captured image data or a signal representing the imaging result to the control device 10. As shown in FIG. 1, the illuminance sensor 65 is provided at an end ER of the front frame 27 and is arranged to receive external light from the front of the user wearing the image display unit 20. The illuminance sensor 65 outputs a detection value corresponding to the amount of received light (received light intensity). As shown in FIG. 1, the LED indicator 67 is disposed near the camera 61 at the end ER of the front frame 27. The LED indicator 67 is lit during execution of imaging by the camera 61 to notify that imaging is in progress.

  The temperature sensor 217 detects the temperature and outputs a voltage value or a resistance value corresponding to the detected temperature. The temperature sensor 217 is mounted on the back side of the OLED panel 223 (FIG. 3). For example, the temperature sensor 217 may be mounted on the same substrate as the OLED drive circuit 225. With this configuration, the temperature sensor 217 mainly detects the temperature of the OLED panel 223. The temperature sensor 217 may be incorporated in the OLED panel 223 or the OLED drive circuit 225. For example, when the OLED panel 223 is mounted as an Si-OLED as an integrated circuit on an integrated semiconductor chip together with the OLED drive circuit 225, the temperature sensor 217 may be mounted on the semiconductor chip.

  The left display unit 24 includes a display unit substrate 230, an OLED unit 241, and a temperature sensor 239. On the display unit substrate 230, an interface (I / F) 231 connected to the interface 196, a receiving unit (Rx) 233, a six-axis sensor 235, and a magnetic sensor 237 are mounted. The receiving unit 233 receives data input from the control device 10 via the interface 231. When receiving the image data of the image displayed by the OLED unit 241, the receiving unit 233 outputs the received image data to the OLED drive circuit 245 (FIG. 2).

  The 6-axis sensor 235 is a motion sensor (inertia sensor) including a 3-axis acceleration sensor and a 3-axis gyro (angular velocity) sensor. The 6-axis sensor 235 may employ an IMU in which the above sensors are modularized. The magnetic sensor 237 is, for example, a triaxial geomagnetic sensor. The temperature sensor 239 detects the temperature and outputs a voltage value or a resistance value corresponding to the detected temperature. The temperature sensor 239 is mounted on the back side of the OLED panel 243 (FIG. 3). The temperature sensor 239 may be mounted on the same substrate as the OLED drive circuit 245, for example. With this configuration, the temperature sensor 239 mainly detects the temperature of the OLED panel 243. The temperature sensor 239 may be incorporated in the OLED panel 243 or the OLED drive circuit 245. Details are the same as those of the temperature sensor 217.

  The camera 61, the illuminance sensor 65, and the temperature sensor 217 of the right display unit 22, and the 6-axis sensor 235, the magnetic sensor 237, and the temperature sensor 239 of the left display unit 24 are connected to the sensor hub 192 of the control device 10. The sensor hub 192 sets and initializes the sampling period of each sensor according to the control of the main processor 140. The sensor hub 192 executes energization to each sensor, transmission of control data, acquisition of a detection value, and the like in accordance with the sampling period of each sensor. The sensor hub 192 outputs detection values of the sensors included in the right display unit 22 and the left display unit 24 to the main processor 140 at a preset timing. The sensor hub 192 may have a cache function that temporarily holds the detection value of each sensor. The sensor hub 192 may have a conversion function (for example, a conversion function to a unified format) of a signal format or a data format of a detection value of each sensor. The sensor hub 192 turns on or off the LED indicator 67 by starting and stopping energization of the LED indicator 67 according to the control of the main processor 140.

  FIG. 6 is a block diagram functionally showing the configuration of the control function unit of the HMD 100. The control function unit of the HMD 100 includes a storage unit 122 and a control unit 150. The storage unit 122 is a logical storage unit configured by the nonvolatile storage unit 121 (FIG. 5). Although FIG. 6 illustrates an example in which only the storage unit 122 is used, the EEPROM 215 or the memory 118 may be used in combination with the nonvolatile storage unit 121. The control unit 150 is configured by the main processor 140 executing a computer program, that is, hardware and software cooperate.

  The storage unit 122 stores various data used for processing in the control unit 150. Specifically, the storage unit 122 of the present embodiment stores setting data 123, content data 124, a default display screen setting 126, and a change parameter 128. Details of the default display screen setting 126 will be described later. The change parameter 128 is a parameter set in a display surface setting process described later, and 0 is stored in the initial state.

  The setting data 123 includes various setting values related to the operation of the HMD 100. For example, the setting data 123 includes parameters, determinants, arithmetic expressions, LUTs (Look Up Tables), and the like when the control unit 150 controls the HMD 100.

  The content data 124 includes content data (image data, video data, audio data, etc.) including images and video displayed by the image display unit 20 under the control of the control unit 150. The content data 124 may include interactive content data. The interactive content is acquired by a user operation by the control device 10, the process according to the acquired operation content is executed by the control unit 150, and the content according to the process content is displayed on the image display unit 20. Means the type of content In this case, the content data may include image data of a menu screen for acquiring a user's operation, data for defining a process corresponding to an item included in the menu screen, and the like.

  The control unit 150 executes various processes using the data stored in the storage unit 122, whereby the OS 143, the image processing unit 145, the display control unit 147, the imaging control unit 149, the augmented reality processing unit 151, The function as the display surface setting unit 152 is executed. In the present embodiment, each functional unit other than the OS 143 is configured as an application program executed on the OS 143.

  The image processing unit 145 generates a signal to be transmitted to the right display unit 22 and the left display unit 24 based on image / video image data displayed by the image display unit 20. The signal generated by the image processing unit 145 may be a vertical synchronization signal, a horizontal synchronization signal, a clock signal, an analog image signal, or the like. In addition to a configuration in which the main processor 140 is realized by running a computer program, the image processing unit 145 may be configured by hardware (for example, a DSP (Digital Signal Processor)) different from the main processor 140.

  Note that the image processing unit 145 may execute resolution conversion processing, image adjustment processing, 2D / 3D conversion processing, and the like as necessary. The resolution conversion process is a process for converting the resolution of the image data into a resolution suitable for the right display unit 22 and the left display unit 24. The image adjustment process is a process for adjusting the brightness and saturation of image data. The 2D / 3D conversion process is a process of generating 2D image data from 3D image data or generating 3D image data from 2D image data. When these processes are executed, the image processing unit 145 generates a signal for displaying an image based on the processed image data, and transmits the signal to the image display unit 20 via the connection cable 40.

  The display control unit 147 generates a control signal for controlling the right display unit 22 and the left display unit 24, and controls the generation and emission of image light by each of the right display unit 22 and the left display unit 24 based on the control signal. To do. Specifically, the display control unit 147 controls the OLED drive circuits 225 and 245 to cause the OLED panels 223 and 243 to display an image. The display control unit 147 controls the timing at which the OLED drive circuits 225 and 245 draw on the OLED panels 223 and 243 and controls the luminance of the OLED panels 223 and 243 based on the signal output from the image processing unit 145.

  The imaging control unit 149 controls the camera 61 to perform imaging, generates captured image data, and temporarily stores the captured image data in the storage unit 122. When the camera 61 is configured as a camera unit including a circuit that generates captured image data, the imaging control unit 149 acquires the captured image data from the camera 61 and temporarily stores the captured image data in the storage unit 122.

  The augmented reality processing unit 151 executes augmented reality processing. The augmented reality process is a process for causing the image display unit 20 to form a virtual image representing a virtual object, and includes the following procedures a1 to a3. Here, the “virtual object” means an object (image, character, graphic symbol, etc.) that does not actually exist in the surrounding environment (hereinafter also simply referred to as “real space”) surrounding the user. This virtual object forms a virtual input surface for receiving input from the user, and a virtual guide surface for guiding the user to work procedures and work contents to be performed by the user. be able to.

(A1) The augmented reality processing unit 151 determines a virtual object that is a target of virtual image formation. This virtual object can be determined depending on, for example, an application being executed in the control unit 150. For example, when the application being executed is a work support application used in the manufacturing industry or the like, a figure, a symbol, a character, an image representing a work procedure to be performed next by the user, or a software keyboard for operating the work support application And buttons can be virtual objects. For example, when the application being executed is a calling application, characters representing information on the calling party, images of the calling party, software keyboards and buttons for operating the calling application, and the like can be virtual objects. The image referred to here includes both a still image and a moving image.

(A2) The augmented reality processing unit 151 generates image data for displaying the virtual object determined in the procedure a1 on the default display surface determined by the default display surface setting 126. Specifically, the augmented reality processing unit 151 arranges a virtual object whose position and inclination are adjusted so that a user wearing the HMD 100 visually recognizes a virtual image representing the virtual object on the default display surface, and Then, image data in which black is arranged in other portions is generated. Note that the augmented reality processing unit 151 may perform image processing such as enlargement, reduction, rotation, and keystone correction on the virtual object determined in step a1 in order to adjust the position and inclination of the virtual object. .

(A3) The augmented reality processing unit 151 transmits the image data generated in the procedure a2 to the image processing unit 145. The image processing unit 145 performs the above-described processing on the received image data.

  FIG. 7 is a diagram illustrating the default display surface. As a result of the execution of the augmented reality processing described in steps a1 to a3, the user wearing the HMD 100 is determined in step a1 to the area AR1 in front of his / her own eye (the rectangle illustrated by the broken line in FIG. 7). Virtual objects (graphics, symbols, characters, images, etc.) can be visually recognized. That is, the illustrated area AR1 is the default display surface. The default display surface AR1 is specified by the gravity center position D and the surface inclinations Dx, Dy, Dz in the user's field of view. In the default display surface setting 126 (FIG. 6), the center-of-gravity position D and the surface inclinations Dx, Dy, and Dz are stored in advance as parameters for specifying the default display surface AR1.

  The display surface setting unit 152 (FIG. 6) executes the display surface setting process described later to change the display surface of the virtual object from the default display surface described in FIG. 7 to an arbitrary display surface designated by the user. Can be changed.

A-2. Display surface setting process:
FIG. 8 is a flowchart showing the procedure of the display surface setting process. The display surface setting process is a process of changing the display surface of the virtual object from the default display surface described in FIG. 7 to an arbitrary display surface designated by the user. The display surface setting process is executed by the display surface setting unit 152 of the control device 10.

  In step S <b> 100, the display surface setting unit 152 determines whether or not there is an instruction to start the display surface setting process from the user. The start instruction can be given in any manner. For example, selection execution of an icon created on the screen of the OS 143, a start instruction using a voice command, a start instruction using a gesture command, or the like can be used. The display surface setting process start instruction may be given in the form of a call from an arbitrary computer program installed in the OS 143. When there is no instruction to start the display surface setting process (step S100: NO), the display surface setting unit 152 returns to step S100 and continues monitoring.

  When there is an instruction to start the display surface setting process (step S100: YES), in step S102, the display surface setting unit 152 shifts to the display surface setting mode. First, the display surface setting unit 152 acquires the direction of the user's line of sight. Specifically, the display surface setting unit 152 analyzes the right eye image and the left eye image of the user taken by the inner camera 62, and acquires the direction of the black eye. The display surface setting unit 152 estimates the direction of the user's line of sight from the acquired direction of the black eye.

  FIG. 9 is a diagram illustrating steps S104 to S108 of the display surface setting process. In FIG. 9 and subsequent figures, the user wearing the HMD 100 and the user's hand are illustrated from the upper side in the vertical direction (the top of the user's head). In step S104, the display surface setting unit 152 causes the camera 61 to perform a three-dimensional scan (three-dimensional recognition) on the acquired line-of-sight direction. The camera 61 of the present embodiment described with reference to FIGS. 1 and 5 includes a laser light emitting unit, and performs a three-dimensional scan in the front side direction of the HMD 100 (the visual field direction visually recognized by the user when the image display unit 20 is mounted). can do.

  Specifically, the camera 61 emits slit-shaped laser light from the laser light emitting unit in the front side direction of the HMD 100. The camera 61 receives reflected light of the emitted laser light using a CCD or CMOS image sensor. The camera 61 acquires distance information with respect to the subject using the principle of triangulation, and converts it into three-dimensional data. Further, the camera 61 obtains color image data having a predetermined resolution by the same CCD or CMOS by dispersing the received light data with a rotating filter. For example, in the example of FIG. 9, the display surface setting unit 152 can acquire the three-dimensional data of the user's hand hd and the color image data by causing the camera 61 to perform a three-dimensional scan in the direction LS of the line of sight. it can. The three-dimensional data and color image data obtained in this step function as “real space information”.

  In step S106, the display surface setting unit 152 determines whether there is any object within the effective range of the three-dimensional scan. Specifically, the display surface setting unit 152 refers to the 3D data obtained in step S104 and detects an object included in the 3D data. In addition, when a plurality of objects (for example, a plurality of hands, hands and figurines, etc.) are included in the three-dimensional data, all objects are detected in this step. When there is no object (step S106: NO), the display surface setting unit 152 proceeds to step S102 and reacquires the direction of the user's line of sight. At this time, the display surface setting unit 152 may notify the user that the object has not been detected or a message for guiding the movement in the direction of the line of sight. The notification can employ any form such as display of characters or the like, voice or the like.

  When there is an object (step S106: YES), in step S108, the display surface setting unit 152 determines, for one of the detected objects, whether the object includes a reference surface. The display surface setting unit 152 of the present embodiment determines that the object includes a reference surface when both of the following conditions b1 and b2 are satisfied. The display surface setting unit 152 can determine whether or not the reference surface is included using various conditions in addition to the exemplified conditions b1 and b2. The conditions b1 and b2 correspond to “predetermined conditions”.

(B1) The determination target object includes a surface having flatness within the first range. The first range can be set as appropriate. For example, an object with some unevenness such as a human hand is allowed, and an object with large unevenness such as a human face is not allowed. preferable. By appropriately setting the first range, the display surface setting unit 152 can suppress erroneous recognition.
(B2) The area of the surface of the object to be determined is greater than or equal to the second value. The second value can be set as appropriate, but is preferably a value that can be excluded from, for example, a third party's hand, face, or any object located far away from the user. By appropriately setting the second value, the display surface setting unit 152 can suppress erroneous recognition.

  For example, in the example of FIG. 9, the display surface setting unit 152 sets a plane CA (FIG. 9, broken line) including the palm of the detected object (user's hand hd) and the aligned fingers as the reference plane CA. Authorize. In the example of FIG. 9, the reference plane CA has an elliptical shape, but the shape of the reference plane is not particularly limited, and any shape can be adopted.

  When the object to be determined does not include the reference surface (step S108: NO), the display surface setting unit 152 sequentially performs the determination in step S108 for the other detected objects. When all the detected objects do not include the reference plane, the display plane setting unit 152 may transition to step S102 and reacquire the direction of the user's line of sight. At this time, the display surface setting unit 152 may notify the user of a message indicating that no object has been detected or a message for guiding the movement in the direction of the line of sight.

  Note that the display surface setting unit 152 may perform the determination in step S108 only on an object including a specific feature among the detected objects. As specific characteristics, for example, a human hand, a user's hand, an object having a predetermined shape, an image including information having a predetermined pattern, or the like can be employed. The human hand is a human hand regardless of a user or a third party. The user's hand means the hand of the user who is currently using the HMD 100, and can be determined by collating fingerprints or palm prints. Examples of objects having a predetermined shape include smartphones (rectangular devices), notebooks (rectangular media), clock faces (rectangular and circular devices), round fans (circular media), pens ( An elongated cylindrical medium) or the like may be employed. As an image including information having a predetermined pattern, for example, a two-dimensional code such as a QR code (registered trademark), a marker, or the like may be employed. For a fingerprint or palm print for recognizing a user's hand, an object pattern for recognizing an object of a predetermined shape, and a pattern in an image for recognizing an image including information having a predetermined pattern, a storage unit 122 may be stored in advance, or may be stored in advance in an external device (for example, a cloud server) connected to the HMD 100 via a network. In this way, since the process of step S108 is executed only for objects including these specific features, the display surface setting process can be made more efficient.

FIG. 10 is a diagram illustrating step S110 of the display surface setting process. When the object to be determined includes the reference plane CA in FIG. 8 (step S108: YES), in step S110, the display plane setting unit 152 determines the center of gravity position of the reference plane CA with respect to the default display plane (FIG. 7), the plane inclination, and the like. Find the change parameter of. The gravity center position of the reference surface CA is the gravity center position C of the reference surface CA in the user's field of view (FIG. 10, black circle). The plane inclination of the reference plane CA is the inclination Cx, Cy, Cz in the user's field of view (FIG. 10, arrow extending from the black circle). For this reason, the display surface setting unit 152 in step S110 is within the user's field of view.
The amount of change from the center of gravity position D of the default display surface to the center of gravity position C of the reference surface CA;
The amount of change from the inclination Dx of the default display surface to the inclination Cx of the reference surface CA;
An amount of change from the inclination Dy of the default display surface to the inclination Cy of the reference surface CA;
The amount of change from the inclination Dz of the default display surface to the inclination Cz of the reference surface CA;
Are obtained and set as a change parameter.

  In step S112 of FIG. 8, the display surface setting unit 152 sets the display surface of the virtual object based on the obtained change parameter. Specifically, the display surface setting unit 152 stores the change parameters (the change amounts of the center of gravity and the change amounts of the inclinations x, y, and z) obtained in step S110 in the change parameter 128 (FIG. 6). . When a value other than the initial value (0) is stored in the change parameter 128, the augmented reality processing unit 151 in the above-described augmented reality processing procedure a 2 causes the default display surface setting 126 (centroid position D, surface inclination). Dx, Dy, Dz) is corrected with the change parameter 128 (the change in the center of gravity position and the change in each of the inclinations x, y, z), and the virtual object determined in step a1 is determined on the display surface. Generate image data to be displayed.

  FIG. 11 is a diagram for explaining the corrected display surface. As a result of executing the augmented reality processing (procedures a1 to a3) using the change parameter 128, the user wearing the HMD 100 has the same centroid position C as the centroid position C of the reference plane CA and the reference plane CA. The virtual objects (graphics, symbols, characters, images, etc.) determined in step a1 can be visually recognized in the area AR2 (the rectangle shown by the broken line in FIG. 11) having the same inclination as the inclinations Cx, Cy, Cz.

  In the above-described example, only the display surface of one virtual object is described as an example, but the display surface setting unit 152 sets the display surfaces of a plurality of virtual objects using the same method as described above. be able to. The display surfaces of the plurality of virtual objects include, for example, a display surface for a guide surface that guides work procedures and work contents to be performed by the user, and a display surface for an operation surface that is used by the user to operate the HMD 100. Etc., display surfaces having different properties may be included. The display surfaces of the plurality of virtual objects may include display surfaces having the same property.

  As described above, according to the present embodiment, the display surface setting unit 152 of the control unit 150 determines the position and inclination of the virtual object visually recognized by the user as the position C and the inclination Cx of the reference plane CA in the real space. It changes based on Cy and Cz (FIG. 8, step S112). For this reason, the user adjusts the position C and the inclinations Cx, Cy, and Cz of the reference surface CA during the display surface setting mode (for example, by changing the position and inclination of the hand) and the position of the virtual object that is visually recognized as a virtual image. And, in other words, the position and inclination of the display surface AR2 of the virtual object can be adjusted as intended.

  Further, according to the present embodiment, the position and inclination of the virtual object that the user visually recognizes as a virtual image (the position and inclination of the display surface AR2 of the virtual object) are the position C and inclination Cx of the reference plane CA in the real space. It becomes the same as Cy and Cz (FIG. 8, step S112). Therefore, the user can intuitively adjust the position and inclination of the virtual object.

  Furthermore, according to the present embodiment, the acquisition unit (camera 61) acquires only real space information (three-dimensional data, color image data) corresponding to the user's line-of-sight direction LS (step S104). In other words, the acquisition unit omits acquisition of real space information that does not correspond to the user's line-of-sight direction LS. For this reason, the information content in the information of the real space acquired by the acquisition part can be reduced. As a result, the processing load at the time of analysis (steps S108 and S110) by the display surface setting unit 152 of the control unit 150 can be reduced.

A-3. Additional processing:
In the display surface setting process, the display surface setting unit 152 may additionally execute processes c1 and c2 exemplified below. The processes c1 and c2 may be added alone or in combination.

(C1) Re-changing the display surface of the virtual object:
FIG. 12 is a diagram for explaining re-change of the display surface of the virtual object. After the display surface setting process described with reference to FIG. 8 is completed, the display surface setting unit 152 executes the display surface setting process again in response to the detection of the first motion by the user, so that the display surface of the virtual object is displayed. The position and inclination may be changed again based on the newly obtained position and inclination of the reference plane. The first operation can be arbitrarily determined in advance. For example, as shown in FIG. 12, the operation to make the hand hd a predetermined shape (in the example shown, the shape of standing the thumb), the operation to make the finger a predetermined shape, the operation to change the direction and front / back of the hand hd, the predetermined Gestures, utterances with predetermined contents (voice commands), etc. can be employed. Note that the display surface setting unit 152 may execute the display surface setting process again only when the first action is detected in the vicinity of the display surface AR2 of the virtual object set by the previous display surface setting process. Good.

  In the case of this re-change, the display surface setting unit 152 may overwrite the change parameter 128 with the newly obtained change parameter, or store the newly obtained change parameter together with the time and version number in the change parameter 128. May be.

  In this way, the display surface setting unit 152 of the control unit 150 re-changes the position and inclination of the virtual object (position and inclination of the display surface AR2 of the virtual object) triggered by the first operation. Convenience can be improved.

(C2) Change display mode of virtual object:
FIG. 12 and FIG. 14 are diagrams for explaining the change of the display mode of the virtual object. After the display surface setting process described with reference to FIG. 8 ends, the augmented reality processing unit 151 may change the display mode of the virtual object in response to the detection of the second action by the user. The display mode includes whether or not a virtual object is displayed, the size of the virtual object, the color of the virtual object, the transparency of the virtual object, the brightness of the virtual object, the contents of the virtual object, and the like. The second action can be arbitrarily determined in advance, such as an action for making the hand hd into a predetermined shape, an action for making the finger into a predetermined shape, an action for changing the orientation or front / back of the hand hd, a predetermined gesture, An utterance (voice command) having a predetermined content can be employed. For example, in the example of FIG. 13, the content of the virtual object OB (specifically, the displayed page) is changed by the operation of turning the page of the hand hd. For example, in the example of FIG. 14, the virtual object OB is changed to non-display by the operation of grasping the fist of the hand hd.

  In this way, since the augmented reality processing unit 151 of the control unit 150 changes the display mode of the virtual object triggered by the second operation, the convenience for the user can be improved.

B. Variations:
In the above embodiment, a part of the configuration realized by hardware may be replaced by software, and conversely, a part of the configuration realized by software may be replaced by hardware. Good. In addition, the following modifications are possible.

・ Modification 1:
In the said embodiment, it illustrated about the structure of HMD. However, the configuration of the HMD can be arbitrarily determined without departing from the gist of the present invention, and for example, addition / deletion / conversion of components can be performed.

  In the above embodiment, the so-called transmission type HMD 100 in which the right light guide plate 26 and the left light guide plate 28 transmit external light has been described. However, the present invention can also be applied to, for example, a so-called non-transmissive HMD 100 that displays an image in a state where an outside scene cannot be visually recognized. In addition, in these HMDs 100, in addition to the AR display that displays an image superimposed on the real space described in the above embodiment, an MR (Mixed Reality) display that displays a combination of the captured real space image and a virtual image, Or VR (Virtual Reality) display which displays virtual space can also be performed. The present invention can also be applied to an apparatus that does not perform any of AR, MR, and VR display.

  In the above embodiment, the functional units of the control device 10 and the image display unit 20 have been described, but these can be arbitrarily changed. For example, the following aspects may be adopted. A mode in which the storage unit 122 and the control unit 150 are mounted on the control device 10 and only the display function is mounted on the image display unit 20. The aspect which mounts the memory | storage part 122 and the control part 150 in both the control apparatus 10 and the image display part 20. FIG. The aspect which integrated the control apparatus 10 and the image display part 20. FIG. In this case, for example, the image display unit 20 includes all the components of the control device 10 and is configured as a glasses-type wearable computer. A mode in which a smartphone or a portable game device is used instead of the control device 10. The aspect which connected the control apparatus 10 and the image display part 20 by radio | wireless communication, and arranged the connection cable 40. FIG. In this case, for example, power supply to the control device 10 and the image display unit 20 may be performed wirelessly.

Modification 2
In the said embodiment, it illustrated about the structure of the control apparatus. However, the configuration of the control device can be arbitrarily determined without departing from the gist of the present invention, and for example, addition / deletion / conversion of components can be performed.

  In the above embodiment, an example of the input unit provided in the control device 10 has been described. However, the control device 10 may be configured by omitting some of the illustrated input means, and may include other input means not described above. For example, the control device 10 may include an operation stick, a keyboard, a mouse, and the like. For example, the control device 10 may include an input unit that interprets a command associated with the movement of the user's body. The user's body movement or the like can be acquired by, for example, line-of-sight detection for detecting line-of-sight, gesture detection for detecting hand movement, foot switch for detecting foot movement, or the like. The line-of-sight detection can be realized by, for example, a camera that images the inside of the image display unit 20. Gesture detection can be realized, for example, by analyzing an image taken with time by the camera 61.

  In the above embodiment, the control unit 150 is operated by the main processor 140 executing the computer program in the storage unit 122. However, the control unit 150 can employ various configurations. For example, the computer program may be replaced with or together with the storage unit 122, the nonvolatile storage unit 121, the EEPROM 215, the memory 118, another external storage device (a storage device such as a USB memory inserted in various interfaces, It may be stored in an external device such as a server connected via a network. In addition, each function of the control unit 150 may be realized using an ASIC (Application Specific Integrated Circuit) designed to realize the function.

・ Modification 3:
In the said embodiment, it illustrated about the structure of the image display part. However, the configuration of the image display unit can be arbitrarily determined without departing from the gist of the present invention, and for example, addition / deletion / conversion of components can be performed.

  FIG. 15 is a main part plan view showing the configuration of the optical system provided in the image display unit of the modification. In the image display unit of the modified example, an OLED unit 221a corresponding to the user's right eye RE and an OLED unit 241a corresponding to the left eye LE are provided. The OLED unit 221a corresponding to the right eye RE includes an OLED panel 223a that develops white color and an OLED drive circuit 225 that drives the OLED panel 223a to emit light. A modulation element 227 (modulation device) is arranged between the OLED panel 223a and the right optical system 251. The modulation element 227 is configured by, for example, a transmissive liquid crystal panel, and generates image light L by modulating light emitted from the OLED panel 223a. The image light L that has been transmitted through the modulation element 227 and is modulated is guided to the right eye RE by the right light guide plate 26.

  The OLED unit 241a corresponding to the left eye LE includes an OLED panel 243a that emits white light and an OLED drive circuit 245 that drives the OLED panel 243a to emit light. A modulation element 247 (modulation device) is arranged between the OLED panel 243a and the left optical system 252. The modulation element 247 is constituted by, for example, a transmissive liquid crystal panel, and generates image light L by modulating light emitted from the OLED panel 243a. The image light L that has been modulated through the modulation element 247 is guided to the left eye LE by the left light guide plate 28. The modulation elements 227 and 247 are connected to a liquid crystal driver circuit (not shown). The liquid crystal driver circuit (modulation device driving unit) is mounted on a substrate disposed in the vicinity of the modulation elements 227 and 247, for example.

  According to the image display unit of the modified example, the right display unit 22 and the left display unit 24 respectively include OLED panels 223a and 243a as light source units and image light including a plurality of color lights by modulating light emitted from the light source units. Are provided as modulation elements 227 and 247 that output a video signal. Note that the modulation device that modulates the light emitted from the OLED panels 223a and 243a is not limited to a configuration in which a transmissive liquid crystal panel is employed. For example, a reflective liquid crystal panel may be used instead of the transmissive liquid crystal panel, a digital micromirror device may be used, or a laser retinal projection type HMD 100 may be used.

  In the above embodiment, the spectacle-type image display unit 20 has been described, but the aspect of the image display unit 20 can be arbitrarily changed. For example, the image display unit 20 may be mounted like a hat, or may be built in a body protector such as a helmet. Further, the image display unit 20 may be configured as a HUD (Head Up Display) mounted on a vehicle such as an automobile or an airplane, or other transportation means.

  In the above embodiment, a configuration in which virtual images are formed by the half mirrors 261 and 281 on a part of the right light guide plate 26 and the left light guide plate 28 as an optical system that guides image light to the user's eyes has been exemplified. However, this configuration can be arbitrarily changed. For example, a virtual image may be formed in a region that occupies the entire surface (or most) of the right light guide plate 26 and the left light guide plate 28. In this case, the image may be reduced by an operation for changing the display position of the image. The optical element of the present invention is not limited to the right light guide plate 26 and the left light guide plate 28 having the half mirrors 261 and 281, but is an optical component (for example, a diffraction grating, a prism, or the like) that makes image light incident on the user's eyes. Any form can be adopted as long as holography or the like is used.

-Modification 4:
In the said embodiment, it illustrated about the procedure of the display surface setting process. However, the procedure of the display surface setting process can be arbitrarily determined without departing from the gist of the present invention. For example, addition / deletion of executed steps, change of processing contents in the steps, and the like are performed. Can do.

  In the above embodiment, the position of the center of gravity is used to specify the positions of the reference plane and the display surface of the virtual object. However, arbitrary information can be used as long as the positions of the reference plane and the display surface of the virtual object can be specified. For example, the reference plane and the end point of the virtual object display plane (for example, the upper left point) may be used instead of the barycentric position, or the reference plane and virtual object display plane frame may be used instead of the barycentric position. May be. Further, for example, both the center of gravity position of the reference plane and the display surface of the virtual object and the end point may be used.

  In the above embodiment, the augmented reality processing unit 151 includes the default display surface setting 126 (centroid position D, surface inclination Dx, Dy, Dz) and the change parameter 128 (center of gravity position change amount, inclination x, y, z). And the position and inclination of the display surface of the virtual object after correction are controlled to be the same as the position and inclination of the reference surface. However, the augmented reality processing unit 151 adopts any mode as long as the position and inclination of the display surface of the corrected virtual object are changed based on the position C and inclination Cx, Cy, Cz of the reference plane CA. it can. For example, the augmented reality processing unit 151 may multiply the change parameter 128 by some coefficient.

  In the embodiment described above, the camera 61 performs a three-dimensional scan (three-dimensional recognition) on the direction of the user's line of sight. However, the camera 61 may three-dimensionally scan the entire range that can be recognized by itself. Note that the three-dimensional scanning method of the above embodiment is merely an example, and other methods may be employed.

  In the embodiment described above, the display surface setting unit 152 determines the reference surface using the condition b1 (flatness within the first range) and the condition b2 (area is equal to or greater than the second value). However, the display surface setting unit 152 may omit either one or both of the conditions b1 and b2.

  Regarding the condition b2 (the area is equal to or larger than the second value) in the above embodiment, the display surface setting unit 152 may be able to use a plurality of modes (for example, the normal mode and the minute surface detection mode). The mode to be adopted is designated according to a user instruction, for example. In the case of the minute surface detection mode, the display surface setting unit 152 sets the second value in the condition b2 to be smaller than the second value in the normal mode. In this way, the display surface setting unit 152 can execute the display surface setting process using a small surface as a reference surface in accordance with a user instruction.

  Instead of the condition b2 (the area is equal to or larger than the second value) in the above embodiment, the display surface setting unit 152 determines that the distance between the surface of the object to be determined and the user is within a predetermined range. It may be adopted. Although it can determine arbitrarily with a predetermined range, it can set to about 20 cm-80 cm, for example. Even in this way, the display surface setting unit 152 can suppress misrecognition as in the above embodiment.

  In the above-described embodiment, the user's own hand may be recognized by collating fingerprints or palm prints, and the determination in step S108 may be performed only for the user's hand. As a method for recognizing the user's own hand, a captured image of a wide-angle camera may be recognized as an image instead of a fingerprint or a palm print. In this case, the HMD 100 includes a wide-angle camera instead of the camera 61 or together with the camera 61. When the display surface setting unit 152 recognizes an image captured by the wide-angle camera and determines that the object to be determined (hand) extends from the user of the HMD 100, the object to be determined (hand) It can be determined that it is the hand of the person.

  In the above-described embodiment, the display surface setting unit 152 determines whether or not the display surface setting unit 152 is a human hand by performing image analysis on an image captured by the camera 61. However, the display surface setting unit 152 may determine whether or not it is a human hand based on the detection result of the temperature detection unit (thermography) mounted on the HMD 100.

  In the above-described embodiment, the display surface setting unit 152 can set the display surfaces of a plurality of virtual objects through the display surface setting process. In setting the display surfaces of the plurality of virtual objects, the display surface setting unit 152, for example, uses a first display surface with the user's right hand as a reference surface and a second display with the user's left hand as a reference surface. A surface may be set. The display surfaces of the plurality of virtual objects may be sequentially set by executing the display surface setting process serially, or may be simultaneously set by executing the display surface setting process in parallel.

  In the above embodiment, when a value other than the initial value is stored in the change parameter 128, the augmented reality processing unit 151 immediately displays the virtual object on the display surface corrected with the change parameter 128. However, the augmented reality processing unit 151 may execute the above-described processing after a predetermined time has elapsed since a value other than the initial value is stored in the change parameter 128. In this way, the augmented reality processing unit 151 displays the virtual object on the corrected display surface after the value other than the initial value is stored in the change parameter 128 (FIG. 8, step S112). A delay can be caused in the time until. The delay time may be set as appropriate according to a request from a user or an application installed in the HMD 100.

  In the display surface setting process of the above embodiment, the display surface of a two-dimensional virtual object has been described. However, the display surface setting unit 152 may be able to set the display surface of the three-dimensional virtual object. The display surface of the three-dimensional virtual object is, for example, the center-of-gravity position A of the three-dimensional virtual object, the orientations Ax, Ay, Az of the reference points of the three-dimensional virtual object, and the size As of the three-dimensional virtual object. And can be specified by In the display surface setting process described above, the display surface setting unit 152 uses the human palm as the first reference surface, and the abdominal surface of each finger extending from the palm as the second reference surface (thumb of the thumb) to the sixth reference surface (of the little finger). As an antinode, the center of gravity position, orientation, and size of the three-dimensional virtual object can be specified. In this way, the display surface setting unit 152 can adjust not only the display surface of the two-dimensional virtual object but also the display surface of the three-dimensional virtual object as intended by the user.

-Modification 5:
The present invention is not limited to the above-described embodiments, examples, and modifications, and can be realized with various configurations without departing from the spirit thereof. For example, the technical features in the embodiments, examples, and modifications corresponding to the technical features in each embodiment described in the summary section of the invention are to solve some or all of the above-described problems, or In order to achieve part or all of the above-described effects, replacement or combination can be performed as appropriate. Further, if the technical feature is not described as essential in the present specification, it can be deleted as appropriate.

    DESCRIPTION OF SYMBOLS 10 ... Control apparatus, 11 ... Button, 12 ... LED indicator, 13 ... Cross key, 14 ... Track pad, 15 ... Up / down key, 16 ... Changeover switch, 18 ... Power switch, 19 ... Vibrator, 20 ... Image display part, 21 ... Right holding unit, 22 ... Right display unit, 23 ... Left holding unit, 24 ... Left display unit, 26 ... Right light guide plate, 27 ... Front frame, 28 ... Left light guide plate, 30 ... Headset, 32 ... Right earphone 34 ... left earphone, 40 ... connection cable, 46 ... connector, 61 ... camera, 62 ... inside camera, 63 ... microphone, 65 ... illuminance sensor, 67 ... LED indicator, 110 ... operation unit, 111 ... 6-axis sensor, 113 ... magnetic sensor, 117 ... communication unit, 118 ... memory, 120 ... controller board, 121 ... nonvolatile storage unit, 122 ... memory , 123 ... setting data, 124 ... content data, 126 ... default display surface setting, 128 ... change parameter, 130 ... power supply unit, 132 ... battery, 134 ... power supply control circuit, 140 ... main processor, 145 ... image processing unit, 147 ... display control unit, 149 ... imaging control unit, 150 ... control unit, 151 ... augmented reality processing unit, 152 ... display surface setting unit, 180 ... audio codec, 182 ... audio interface, 184 ... external connector, 186 ... external memory Interface, 188 ... USB connector, 192 ... Sensor hub, 196 ... Interface, 210 ... Display unit board, 211 ... Interface, 213 ... Receiver, 215 ... EEPROM, 217 ... Temperature sensor, 221, 221a ... OLED unit, 223, 23a ... OLED panel, 225 ... OLED drive circuit, 227 ... modulation element, 230 ... display unit substrate, 231 ... interface, 233 ... receiver, 235 ... 6-axis sensor, 237 ... magnetic sensor, 239 ... temperature sensor, 241, 241a ... OLED unit, 243, 243a ... OLED panel, 245 ... OLED drive circuit, 247 ... modulation element, 251 ... right optical system, 252 ... left optical system, 261 ... half mirror, 281 ... half mirror

Claims (9)

A head-mounted display device,
An image display unit for visually recognizing a virtual image for a user wearing the head-mounted display device;
An acquisition unit for acquiring real space information;
A control unit that causes the image display unit to form the virtual image including a virtual object that is an object that does not exist in the real space;
With
The controller is
By analyzing the information of the acquired real space, the position and inclination of the reference plane, which is a plane in the real space and satisfies a predetermined condition, in the user's field of view,
A head-mounted display device that changes the position and inclination of the virtual object visually recognized by the user based on the obtained position and inclination of the reference plane. The head-mounted display device according to claim 1,
The controller is
A head-mounted display device that makes the position and inclination of the virtual object visually recognized by the user the same as the position and inclination of the reference plane. The head-mounted display device according to claim 1 or 2, further comprising:
A line-of-sight detector that obtains the direction of the line of sight of the user;
The acquisition unit
A head-mounted display device that omits acquisition of information in the real space that does not correspond to the direction of the line of sight acquired by the line-of-sight detection unit when obtaining the position and inclination of the reference plane. The head-mounted display device according to any one of claims 1 to 3,
The control unit further includes:
When a predetermined first motion by the user is detected, the position and inclination of the reference plane in the user's field of view are obtained again, and the position and inclination of the virtual object after the change are newly determined. A head-mounted display device that changes again based on the obtained position and inclination of the reference plane. The head-mounted display device according to any one of claims 1 to 4,
The control unit further includes:
A head-mounted display device that changes a display mode of the virtual object when detecting a predetermined second action by the user. The head-mounted display device according to any one of claims 1 to 5,
The controller is
A head-mounted display device that regards at least one of a human hand, the user's hand, an object having a predetermined shape, and an image including information having a predetermined pattern as an object including the reference plane. . The head-mounted display device according to any one of claims 1 to 6,
The plane satisfying the predetermined condition is a plane having flatness within a predetermined first range as a result of three-dimensional recognition and having an area equal to or greater than a predetermined second value. A head-mounted display device. A method of controlling a head-mounted display device, wherein a computer
Forming a virtual image including a virtual object, which is an object that does not exist in real space, and causing the user to visually recognize it;
Obtaining the real space information;
Analyzing the acquired information of the real space to obtain a position and inclination in the user's field of view of a reference plane that is in the real space and satisfies a predetermined condition;
Changing the position and inclination of the virtual object in the visualizing step based on the obtained position and inclination of the reference plane;
A control method comprising: A computer program,
A function of forming a virtual image including a virtual object that is an object that does not exist in real space, and causing the user to visually recognize it;
A function of acquiring information of the real space;
By analyzing the information of the acquired real space, a function for obtaining a position and inclination in the user's field of view of a reference plane that is a plane that is in the real space and satisfies a predetermined condition;
A function of changing the position and inclination of the virtual object in the step of visually recognizing based on the obtained position and inclination of the reference plane;
A computer program comprising:

Images