KR20190070899A - Method for providing user with virtual image in head-mounted display device, machine-readable storage medium and head-mounted display device - Google Patents

Abstract

According to one aspect of the present invention, provided is a method for providing a virtual image to a user in a head-mounted display device, which comprises the steps of: detecting ambient illuminance; calculating a target transmittance of a window provided in the head-mounted display device corresponding to the ambient illuminance; adjusting the transmittance of the window in accordance with the target transmittance; and projecting light to the window using a projector, thereby providing a virtual image formed by the light reflected by the window to the user.

Description

Technical Field [0001] The present invention relates to a method for providing a virtual image to a user in a head-mounted display device, a machine-readable storage medium, and a head-mounted display device. -MOUNTED DISPLAY DEVICE}

The present invention relates to a wearable display device, and more particularly to a head-mounted display (HMD) device.

As a conventional HMD device, a video see-through HMD device has one or two cameras to acquire an image of an actual surrounding environment. The video-see-through HMD device combines a real image input from a camera with a virtual image generated by a computer using a video synthesizer, and displays the synthesized image to a user through a display device such as an LCD attached to the HMD device.

The video-see-through HMD device has the advantage that the virtual object can be accurately displayed at a desired position using computer vision technology. However, due to the small screen of the display device, both the resolution of the real environment and the resolution of the virtual environment are limited.

It is an object of certain embodiments of the present invention to at least partially solve, alleviate or eliminate at least one of the problems and / or disadvantages associated with the prior art.

The present invention can simultaneously provide a virtual image and an actual image, and can reduce the weight and size of the HMD device. The object of the present invention is to improve the outdoor visibility of a virtual image and to provide an HMD device with low power and low heat.

A method of providing a virtual image to a user in a head-mounted display device according to an aspect of the present invention includes: detecting ambient illuminance; Calculating a target transmittance of a window provided in the head-mounted display device corresponding to the ambient illuminance; Adjusting a transmittance of the window according to the target transmittance; And projecting light to the window using a projector, thereby providing a virtual image formed by the light reflected by the window to the user.

According to another aspect of the present invention, there is provided a machine-readable storage medium having recorded thereon a program for executing a method of providing a virtual image to a user in a head-mounted display apparatus, and a head-mounted display apparatus having such a storage medium do.

A head-mounted display device for providing a virtual image to a user according to another aspect of the present invention includes a sensor unit for detecting an ambient illuminance; A projector for projecting light; A window for focusing and reflecting light projected by the projector to provide a virtual image formed by the reflected light to the user; And a control unit for calculating a target transmittance of the window corresponding to the ambient illuminance and adjusting the transmittance of the window according to the target transmittance.

The present invention can simultaneously provide a virtual image and an actual image using a window as a HOE-based coupler, and the weight and size of the HMD device can be miniaturized by integrally forming glass and holographic optical devices. In addition, the present invention can improve the outdoor visibility of a virtual image and drive a window with low power by implementing a window so that the light transmittance can be controlled by an electrical signal, thereby reducing heat generation and increasing battery usage time . Further, the window of the present invention can provide a wide field of view (FoV) and a good image quality.

1 is a perspective view of a head-mounted display device according to a preferred embodiment of the present invention,
2 is a perspective view of the HMD device as seen from the inside,
3 is a diagram showing a configuration of an HMD circuit,
4 is a diagram showing a schematic configuration of a first projector,
5 is a view for explaining the configuration and functions of the first window,
6 is a view showing a configuration of a first glass,
7 is a diagram showing a holographic pattern of a first holographic optical element,
8 and 9 are views for explaining the control of the transmittance of the HMD device,
10 is a flowchart illustrating a method for providing a virtual image according to a preferred embodiment of the present invention.
11 to 13 are views for explaining a first example of a virtual image providing method,
14 is a diagram for explaining a second example of the virtual image providing method,
15 is a diagram for explaining a third example of the virtual image providing method,
16 is a diagram for explaining a fourth example of the virtual image providing method,
17 is a diagram for explaining a fifth example of the virtual image providing method.

The present invention can be variously modified and may have various embodiments, and specific embodiments will be described in detail with reference to the drawings. It is to be understood, however, that the invention is not to be limited to the specific embodiments, but includes all modifications, equivalents, and alternatives falling within the spirit and scope of the invention.

Terms including ordinals, such as first, second, etc., may be used to describe various elements, but the elements are not limited to these terms. The terms are used only for the purpose of distinguishing one component from another. For example, without departing from the scope of the present invention, the first component may be referred to as a second component, and similarly, the second component may also be referred to as a first component. And / or < / RTI > includes any combination of a plurality of related listed items or any of a plurality of related listed items.

It is to be understood that when an element is referred to as being "connected" or "connected" to another element, it may be directly connected or connected to the other element, . On the other hand, when an element is referred to as being "directly connected" or "directly connected" to another element, it should be understood that there are no other elements in between.

The terminology used in this application is used only to describe a specific embodiment and is not intended to limit the invention. The singular expressions include plural expressions unless the context clearly dictates otherwise. In the present application, the terms "comprises" or "having" and the like are used to specify that there is a feature, a number, a step, an operation, an element, a component or a combination thereof described in the specification, But do not preclude the presence or addition of one or more other features, integers, steps, operations, elements, components, or combinations thereof.

Unless defined otherwise, all terms used herein, including technical or scientific terms, have the same meaning as commonly understood by one of ordinary skill in the art to which this invention belongs. Terms such as those defined in commonly used dictionaries are to be interpreted as having a meaning consistent with the contextual meaning of the related art and are to be interpreted as either ideal or overly formal in the sense of the present application Do not.

FIG. 1 is a perspective view of a head-mounted display (HMD) device according to a preferred embodiment of the present invention, and FIG. 2 is a perspective view of the HMD device as seen from the inside.

The HMD device 100 includes a HMD housing 110 and an HMD circuit 101 mounted in the HMD housing 110. The HMD device 100 is a portable terminal (or a mobile communication terminal) having an overall appearance of glasses.

The HMD housing 110 includes a front frame 115 to which first and second windows 280 and 285 corresponding to the left eye and the right eye are fixed and a front frame 115 fixed to the inside by the first and second hinges 122 and 123. [ And first and second temple frames 120 and 121 that can be folded out or outwardly. Hereinafter, the left eye and the right eye may be referred to as a first eye and a second eye. In the present invention, the first and second windows 280 and 285 may be referred to as first and second window panels.

A camera 260 is disposed on an outer surface of the front frame 115 and a camera 260 is disposed on a portion of the front frame 115 between the first and second windows 280 and 285 Bridge).

A microphone 212 is disposed on the outer surface of the front frame 115 and a touch sensor 250 is disposed on the outer surface of the front frame 115.

A first projection light 310 output from the first projector 270 disposed in the front frame 115 is projected to the outside of the front frame 115, (Not shown). A second projecting light 320 output from the second projector 275 disposed in the front frame 115 is projected onto the inner surface of the front frame 115 as a passage through which the second projection light 320 is output to the outside of the front frame 115 Two openings 112 are disposed. At this time, the openings 111 and 112 may be provided with a transparent protective glass for preventing the dust from the outside from flowing into the inside of the front frame 115.

At least one button 213 is disposed on the outer surface of the first temple frame 120.

At least one speaker 211 is disposed on the inner surface of the first and second temple frames 120 and 121.

The first projector 270 outputs the first projection light 310 for forming the first virtual image and the first projection light 310 outputted from the first projector 270 is reflected by the first window 280 And the focused first reflected light 312 forms a first virtual image in the retina of the user's left eye 330. At this time, focusing refers to collecting light, including convergence to collect light at one point, reduction of beam spot of light, and the like. Preferably, the reflected first projected light 312 converges on the lens or pupil of the left eye 330.

The second projector 275 outputs the second projection light 320 for forming the second virtual image and the second projection light 320 outputted from the second projector 275 is outputted to the second window 285 And the focused and reflected second projection light 322 forms a second virtual image in the retina of the user's right eye 335. [ Preferably, the focused and reflected second projection light 322 converges on the lens or pupil of the right eye 335. In this example, the use of two projectors is illustrated, but only one projector may be used.

In the following description, the first virtual image and the second virtual image are identical to each other except that they are displayed in the left eye and the right eye, respectively. However, the present invention is not limited to this, Only one of them may be displayed.

3 is a diagram showing a configuration of an HMD circuit.

The HMD circuit 101 includes an input / output module 210, a memory 220, a sensor unit 225, a battery 230, a power management unit 235, a communication unit 240, a touch sensor 250, a camera 260, first and second projectors 270 and 275, first and second windows 280 and 285, and a control unit 290.

The input / output module 210 includes at least one speaker 211, at least one microphone 211, and at least one microphone 211. The input / A keyboard 212, at least one button 213, a connector, a keypad, or a combination thereof.

The speaker 211 outputs a sound corresponding to various data (e.g., wireless data, broadcast data, digital audio file, digital moving picture file, or picture photographing) to the outside of the HMD device 100 under the control of the controller 290 can do. The speaker 211 can output sound corresponding to the function performed by the HMD device 100. [ The speaker 211 may be disposed in one or more positions at appropriate positions or positions of the HMD housing 110. In this example, two speakers 211 are disposed at the ends of the first and second temple frames 120 and 121, respectively.

The microphone 212 receives voice or sound from the outside of the HMD device 100 to generate an electric signal and outputs the generated electric signal to the controller 290. The microphone 212 may be disposed in one or more positions at appropriate positions or positions of the HMD housing 110. In this example, one microphone 212 is disposed on the outer surface of the front frame 115. As shown in Fig. In this specification, a signal can be represented as data, and the data can also be represented as a data signal.

A button 213 is provided for receiving a user input, and is used for on / off of the HMD circuit 101, selection and / or retrieval of a menu item or item, and the like. The button 213 may include a power button, a volume button, a menu button, a home button, a back button, a search button (a left movement button, a right movement button, a left movement button, . ≪ / RTI > The buttons 213 may be arranged in one or more positions at appropriate positions or positions of the HMD housing 110. In this example, it is illustrated that the button 213 is disposed on the outer surface of the first temple frame 120.

The connector can be used as an interface for connecting the HMD device 100 to an external electronic device or a power source. The connector can be connected to the connector of the electronic device either directly or via a wired cable, through which the control unit 290 can send the data stored in the memory 220 to the electronic device or receive data from the electronic device . In addition, the HMD device 100 can receive power from the power source via the wired cable connected to the connector to charge the battery 230. [

The keypad may receive a key input from the user for control of the HMD device 100. The keypad includes a physical keypad disposed on the HMD device 100, a virtual keypad represented by the first and / or second projectors 270 and 275, or a combination thereof.

The sensor unit 225 includes at least one sensor for detecting the state of the HMD device 100 or the environmental condition. For example, the sensor unit 225 may be a proximity sensor that detects whether or not the user accesses the HMD device 100, or a proximity sensor that detects whether the operation of the HMD device 100 (e.g., rotation, acceleration, Acceleration / deceleration, vibration, etc.), an illuminance sensor for detecting ambient illuminance, or a combination thereof. In addition, the motion / orientation sensor may include at least one of an acceleration sensor, a gravity sensor, a geomagnetic sensor, a gyro sensor, a shock sensor, a GPS, a compass sensor, and an acceleration sensor. The sensor unit 225 may detect the state of the HMD device 100 and may transmit a signal indicating the state of the HMD device 100 to the control unit 290. For example, the GPS module receives a radio wave from a plurality of GPS satellites (not shown) on the earth orbit and calculates a time of arrival (Time of Arrival) from the GPS satellite (not shown) to the HMD device 100 The position of the HMD device 100 can be calculated. The compass sensor calculates the posture or orientation of the HMD device 100.

The power management unit 235 may supply power to the HMD device 100 under the control of the control unit 290. [ The power management unit 235 may be connected to one or a plurality of batteries 230. Also, the power management unit 235 can supply power to the HMD device 100 from an external power source (not shown) via a cable connected to the connector.

The communication unit 240 may be a wire, wireless, or wire / wireless communication unit. The communication unit 240 may transmit data from the control unit 290 by wire or wirelessly to an electronic device through an external communication network or the air, And transmits the data to the control unit 290.

The communication unit 240 may include at least one of a mobile communication module, a wireless LAN module, and a local communication module according to performance.

The mobile communication module allows the HMD device 100 to communicate with the electronic device through the mobile communication network using at least one antenna (not shown) under the control of the controller 290. The mobile communication module may communicate with a mobile phone (not shown) having a network address or telephone number, such as an IP, a smart phone (not shown), a tablet PC or other communication device SMS) or a multimedia message (MMS).

The wireless LAN module may be connected to the Internet at a place where a wireless access point (not shown) is installed under the control of the controller 290. [ The wireless LAN module supports the IEEE 802.11x standard of the Institute of Electrical and Electronics Engineers (IEEE).

The short-range communication module can perform short-range communication with an external short-range communication device wirelessly under the control of the controller 290. The local area communication method may be Bluetooth, infrared data association (IrDA), WiFi-Direct communication, NFC (Near Field Communication), or a combination thereof.

The touch sensor 250 may transmit a signal corresponding to at least one touch input to the controller 290. [ The user can touch the touch sensor 250 through a part of the body (e.g., a finger) or another touch input means, and the touch sensor 250 can receive the touch input of the user. In addition, the touch sensor 250 can receive an input (i.e., a drag input) in accordance with the continuous movement of the touch. The touch input information may include touch coordinates and / or touch states. The touch state may be a mouse down state in which the touch sensor 250 is depressed, a mouse up state in which a finger is released from the touch sensor 250, a drag state in which the touch sensor 250 is slid in a state of pressing the touch sensor 250, and the like. The control unit 290 grasps user input information such as selection or movement of a menu item or an item or handwriting input from the touch input information, and displays a function corresponding to the user input information (telephone connection, camera shooting, , Data transmission, etc.).

In the present invention, the touch is not limited to the contact between the touch sensor 250 and the touch input means, and may include non-contact (e.g., when the touch sensor 250 and the touch input means are spaced from each other). Such a contactless touch input may also be referred to as a hovering input. The touch sensor 250 may be implemented by a resistive method, a capacitive method, an infrared method, an acoustic wave method, an electromagnetic resonance (EMR) method, or a combination thereof It is possible.

The camera 260 includes a lens system and an image sensor, and may further include a flashlight. The camera 260 converts the light inputted (or photographed) through the lens system into an electrical image signal and outputs the electrical image signal to the control unit 290. The user can take a moving image or a still image through the camera 260. [ The camera 260 may also be provided to receive user input by a user's motion or gesture.

The lens system forms an image of a subject by converging light input from the outside. The lens system includes at least one lens, and each lens may be a convex lens, an aspherical lens, or the like. The lens system has symmetry with respect to an optical axis passing through its center, and the optical axis is defined as such a central axis. The image sensor detects an optical image formed by the external light input through the lens system as an electrical image signal. The image sensor has a plurality of pixel units arranged in an M x N matrix structure, and the pixel unit may include a photodiode and a plurality of transistors. The pixel unit accumulates the charge generated by the input light, and the voltage due to the accumulated charge represents the illuminance of the input light. In the case of processing an image constituting a still image or a moving image, an image signal outputted from the image sensor is composed of a set of voltages (i.e., pixel values) output from the pixel units, (I.e., a still image). Further, the frame is composed of M x N pixels. The image sensor may be a charge-coupled device (CCD) image sensor, a complementary metal-oxide semiconductor (CMOS) image sensor, or the like.

According to the control signal received from the controller 290, the image sensor can operate only the pixels of the ROI among all the pixels of the image sensor or all of the pixels of the image sensor, and the image data output from the pixels is output to the controller 200 .

The control unit 290 controls a graphic user interface (GUI) configured by the control unit 290 using a frame (frame) by using the image input from the camera 260, the image stored in the memory 220, ) And processes the converted image frame to match the screen output characteristics (size, picture quality, resolution, etc.) of the first and / or second projectors 270 and 275 to the first and / or second projector 275, Or may be stored in the memory 220. Hereinafter, a graphical user interface (GUI) is specified as an example of a virtual image formed by the first and / or second projectors 270 and 275 in the present invention, but a virtual image may be used instead of the GUI, The virtual image is used to include a non-real virtual object such as a content such as a GUI or a still image.

The control unit 290 can provide a GUI corresponding to various services (e.g., call, data transmission, broadcasting, photographing, etc.) to the user through the first and / or the second projectors 270 and 275. Also, the control unit 290 can provide a still image or a moving image to the user via the GUI. That is, in the present invention, the GUI may represent a still image or a screen represented by a moving image.

The first and second projectors 270 and 275 have the same configuration and each of the projectors 270 and 275 transmits light forming the virtual image provided by the control unit 290 through the corresponding windows 280 and 285 Of-sight. Hereinafter, the configuration of the first projector 270 will be described.

4 is a diagram showing a schematic configuration of the first projector.

The first projector 270 includes a light source 410 that outputs light, an illumination optical system 420 that illuminates the display element 440 with the light output from the light source 410, A display element 440 that reflects light reflected by the mirror 430 on a pixel-by-pixel basis to form a virtual image; and a light- As shown in FIG.

The illumination optical system 420 may have a first optical axis 271 parallel to the X axis and may include at least one collimating lens, at least one filter, at least one equalization lens, a condensing lens, or a combination thereof .

The optical elements (lens, prism, filter, etc.) of the illumination optical system 420 are aligned with the first optical axis 271. Generally, an optical axis refers to an axis having no optical fluctuation even if the optical system is rotated around the optical axis. Alignment on the optical axis means that the center of curvature of the optical element constituting the optical system is located on the optical axis, or the symmetry point (i.e., the center of symmetry) of the optical element or the center point is located on the optical axis.

The light source 410 outputs light traveling along the first optical axis 271. For example, the light source 410 may use at least one LED that outputs a combination of white light, primary light (e.g., blue light, green light, etc.), or primary light.

The illumination optical system 420 collimates, filters, and / or focuses the light input from the light source 410, and outputs the processed light to the mirror 430.

The mirror 430 reflects the input light transmitted through the illumination optical system 420 toward the display element 440 side. The mirror 430 may have a structure in which a highly reflective dielectric layer or a metal layer is deposited on the substrate.

The display element 440 displays an image on a pixel-by-pixel basis in accordance with the data input from the controller 290. The display element 440 includes pixel elements corresponding to a predetermined resolution, and performs on / off driving of the pixel elements To display the image. For example, a DMD (Digital Micro-Mirror Device) including micromirrors arranged in a MxN (e.g., 1280x720, 854x480, etc.) matrix structure may be used as the display device 440. [ Each micromirror rotates to a position corresponding to the on state and a position corresponding to the on state according to the driving signal, reflects the incident light at an angle that can be displayed outside when the on state, And reflects light incident at an angle that is not displayed.

The projection optical system 450 has a second optical axis 272 parallel to the Z axis and includes a relay lens 460 and a projection lens 470. The relay lens 460 and the projection lens 470 are aligned with the second optical axis 272.

The relay lens 460 allows light reflected from the mirror 430 to be matched to the display element 440 in consideration of overfill. That is, the relay lens 460 allows light reflected from the mirror 430 to be incident on an area larger than the area occupied by the pixel elements of the display device 440.

Further, the relay lens 460 receives the light reflected from the display element 440, reduces the beam area of the light, and outputs the light. Since the light reflected from the display element 440 has a large beam spot size, light loss due to light not transmitted to the projection lens 470 may be large. The relay lens 460 condenses the light reflected from the display element 440 and reduces the beam area so that a maximum amount of light is transmitted to the projection lens 470.

The projection lens 470 receives light whose beam area is adjusted from the relay lens 460, and collimates or focuses the received light to project the light to the outside.

Referring to FIG. 3 again, the first and second windows 280 and 285 have the same configuration. The transmissivity of each of the windows 280 and 285 varies according to the control of the controller 290, And has a functioning holographic pattern. Hereinafter, the configuration of the first window 280 will be described.

5 is a diagram for explaining the configuration and functions of the first window.

The first window 280 includes a first glass 281 capable of controlling the transmittance, that is, a transmittance varying according to an applied signal, and a first holographic optical element (HOE) 282 serving as a concave mirror do. Likewise, the second window 285 includes a second glass that can control the transmittance and a second holographic optical element that functions as a concave mirror.

The first window 280 transmits the ambient light 511 input from the outside and reflects and concentrates the first projection light 521 input from the first projector 270.

The transmitted light 512 and the first projection light 522 reflected by the first window 280 are input to the left eye 330 and the retina 333 of the left eye 330 receives ambient light 512 And the GUI 520 based on the reflected light are superposed on each other. That is, the user sees an image in which the surrounding landscape image 510 and the GUI 520 are superimposed, and the GUI 520 can be viewed as a superimposed transparent layer (i.e., GUI) on the surrounding landscape. The GUI 520 includes menu items (or icons) such as a phone call, a contact, a message, a main menu, and the like. 5, the GUI 520 is displayed opaque, but the GUI 520 may be partially or wholly transparent so that the surrounding image 510 located below it is displayed.

The first projection light 521 output from the first projector 270 may be parallel light having a specific wavelength lambda and the first projection light 521 may be a first light May be incident on the first holographic optical element 282 at a certain angle? With the normal of the optical element 282. The first holographic optical element 282 is a device having wavelength selectivity and reflects and focuses light of a specific wavelength lambda (i.e., the first projection light 521 in this example) The light of another wavelength (i.e., ambient light 511 in this example) is transmitted without convergence.

The first holographic optical element 282 reflects and focuses the input first projection light 521 and the first projection light 522 reflected from the first holographic optical element 282 is incident on the first holographic optical element 282, The pupil 331 or the lens 332 position of the left eye 330 located at a specific distance from the element 282, i.e., away from the eye relief by an eye relief. At this time, the first projected light 522 to be converged may have a specific convergence angle or a viewing angle?. The lens 332 functions to adjust the focus of light incident on the eye and the transmitted ambient light 512 is converged in the eyeball 334 by the lens 332, 510). On the other hand, the reflected first projection light 522 converges at the position of the pupil 331 or the lens 332, and is projected on the retina 333 without being converged by the lens 332 to form the GUI 520 .

Preferably, the first projection light 521 incident on the first holographic optical element 282 is parallel light. However, the present invention is not limited to this, and the first holographic optical element 282 The first projection light 521 may be a diverging light or a converging light other than the parallel light under the condition that the reflected first projected light 522 converges to the pupil 331 or the lens 332. [

When a user wearing a normal HMD device focuses the eyes to see an actual object in a surrounding landscape, a virtual object (or object) of a virtual image, such as a GUI projected by a conventional HMD device, A problem that can not be achieved can arise. In this situation, the user can not see the real object and the virtual object at the same time, so that the user can not recognize the intended augmented reality. The problem of the focus mismatch between the real object and the virtual object arises from the inconsistency between the binocular eye convergence and the monocular focus control, thereby increasing the fatigue of the observer.

The HMD device 100 of the present invention solves this conventional problem by converging the projection light forming the virtual image to the pupil or lens position. That is, the present invention minimizes the focal variation amount of the projected light by the lens by allowing the projection light to converge at or near the position of the lens.

When the projection light converges to the position of the pupil or the lens, the projection light is projected directly onto the retina irrespective of the action of the lens of the eye. The virtual image projected on the retina is perceived by the user as a sharp image regardless of the focus of the lens and the aberration caused by the lens or the like.

The first holographic optical element 282 may have a focal length corresponding to the distance between the pupil 331 or the lens 332 and the first holographic optical element 282, The virtual image is clearly formed in the retina of the eye.

6 is a view showing a configuration of the first glass. The first glass 281 has a characteristic in which the transmittance thereof is changed by a signal or a voltage applied by the control unit 290.

Electrochromic glass, SPD (Suspended Particle Device), and LC (Liquid Crystal) may be used as the first glass 281. Alternatively, as the first glass 281, a photochromic glass or a thermochromic glass which can not be actively controlled by signal application and whose transmittance changes in response to a light of a specific wavelength or a temperature change, ) Glass or the like may be used.

The first glass 281 may be formed by various methods such as applying a substance capable of controlling transmittance to glass or attaching a thin film having a transmittance control to a glass.

In this example, the use of electrochromic glass as the first glass 281 is exemplified.

The first glass substrate 281 includes first and second insulating substrates 610 and 615 and a conductive first electrode 620 laminated on the upper surface of the first substrate 610, An insulating spacer 630 that separates the first and second substrates 610 and 615 from each other and seals a space therebetween, a second electrode 625 stacked on the bottom surface of the first substrate 610, The electrochromic layer 640 and the electrolyte 650 filled in the space between the electrodes 610 and 615.

Each substrate 610, 615 can be made of clear glass or plastic, and the plastic can be, for example, polyacrylate, polyethylene ether phthalate, polyethylene naphthalate, polycarbonate, polyarylate, polyetherimide, May be one of the polyimides.

The first electrode 620 may be made of a transparent conductive material such as indium tin oxide (ITO), fluorine tin oxide (FTO) or antimony doped tin oxide ATO) or an organic conductive material such as polyacetylene or polythiophene.

The second electrode 625 may be made of a transparent or opaque conductive material such as indium tin oxide (ITO), fluorine containing tin oxide (FTO), metal such as Al, antimony doped tin oxide oxide, ATO), or a combination thereof.

An electrochromic layer 640 including an electrochromic material is disposed on the first electrode 620. The electrochromic layer 640 may be disposed on the first electrode 620 in the form of a film.

The first substrate 610 and the second substrate 615 are fixed by a spacer 630 and an electrolyte 650 is filled between the first substrate 610 and the second substrate 615. The electrolyte 650 supplies a redox substance that reacts with the electrochromic material, and may be a liquid electrolyte or a solid polymer electrolyte. As the liquid electrolyte, a solution in which a lithium salt such as LiOH or LiClO ₄ , a potassium salt such as KOH and a sodium salt such as NaOH are dissolved in a solvent may be used, but the present invention is not limited thereto. As the solid electrolyte, for example, poly (2-acrylamino-2-methylpropane sulfonic acid), polyethylene oxide (poly (ethylene oxide) It is not.

The material forming the electrochromic layer 640, that is, the electrochromic material, may include a metal-organic composite having a metal and an organic compound having a functional group capable of forming a coordination bond with the metal. The metal may be selected from the group consisting of beryllium (Be), barium (Ba), copper (Cu), zinc (Zn), cerium (Ce) ), Magnesium (Mg), aluminum (Al), titanium (Ti), or combinations thereof. The functional group may include a carboxyl group, a pyridine group, an imidazole group, or a combination thereof. The organic compound may include a viologen derivative, an anthraquinone derivative, or a combination thereof.

7 is a diagram showing a holographic pattern of a first holographic optical element.

The first holographic optical element 282, 282a includes a plurality of concentric holographic patterns 710, 710a. For example, the first holographic optical element 282, 282a may comprise a transparent substrate and a holographic pattern layer deposited on the transparent substrate.

7A shows an example of a first holographic optical element 282 in which the center of the concentric circles is located at the center of the holographic pattern 710 and FIG. Lt; RTI ID = 0.0 > 710a. ≪ / RTI >

3, the controller 290 controls the overall operation of the HMD 100, and controls other components in the HMD 100 to perform a virtual image providing method. The control unit 290 may include a single core, a dual core, a triple core, or a quad core processor. The control unit 290 receives a broadcast signal (e.g., a TV broadcast signal, a radio broadcast signal, or a data broadcast signal) transmitted from a broadcast station through the communication unit 240 and broadcast supplementary information (e.g., EPS (Electric Program Guide) Guide) can be received. Alternatively, the control unit 290 may reproduce a digital audio file stored in the memory 220 or received through the communication unit 240 (e.g., a file having a file extension of mp3, wma, ogg, or wav) through the speaker 211 . Alternatively, the control unit 290 may store a digital moving picture file (for example, a file whose file extension is mpeg, mpg, mp4, avi, mov, or mkv) stored in the memory 220 or received through the communication unit 240 And / or the second projector (270, 275). The controller 290 controls the operation of the memory 290 according to user commands, menu items, icon selection, or event information input through the sensor unit 225, the input / output module 210, the camera 260 or the touch sensor 250, (Specifically, a GUI) configured by the control unit 290 by using the data stored in the memory 220 or the data stored in the memory 220 in the first and / And can be displayed to the user through the projectors 270 and 275. At this time, the image may be a still image or a moving image.

The memory 220 may store a signal or data under the control of the controller 290. The memory 220 may store control programs and applications for the HMD device 100 or the control unit 290. [

The term memory refers to a memory card (not shown) (e.g., an SD card, a memory stick), a nonvolatile memory, a volatile memory, a solid state drive (SSD ), And the like.

Figs. 8 and 9 are views for explaining the transmittance control of the HMD device. The first and second windows 280 and 285 can increase the visibility of a virtual image such as a GUI by controlling the transmittance according to the control of the controller 290.

Since the first and second windows 280 and 285 can adjust the light transmittance according to the change of the applied voltage, the outputs of the first and / or second projectors 270 and 275 for forming a virtual image are reduced Thereby reducing the total power consumption and heat generation of the first and / or the second projectors 270 and 275, and also increasing the use time of the battery 230 of the HMD device 100.

8 shows a case where a user watches TV in the room.

Referring to FIG. 8 (a), the user watches the TV 810 while wearing the HMD device 100 indoors. Since the visibility of the first and second GUIs 820 and 825 formed by the first and second projectors 270 and 275 is high in the room with low ambient illuminance, the transmittance of the first and second windows 280 and 285 Is relatively high. For example, the controller 290 can set the transmittance of the first and second windows 280 and 285 to the maximum or to 30% or more. At this time, the first and second GUIs 820 and 825 are identical to each other only in the display position thereof.

Fig. 9 shows a case where the user views the surrounding scenery outdoors.

Referring to FIG. 9 (a), the user views the surrounding scenery 910 while wearing the HMD device 100 outdoors. Since the visibility of the first and second GUIs 920 and 925 formed by the first and second projectors 270 and 275 is low in the outdoors with high ambient illuminance, the transmittance of the first and second windows 280 and 285 Is relatively low. For example, the control unit 290 may set the transmittance of the first and second windows 280 and 285 to a minimum or 10% or less.

8 and 9, the first GUI 820, 920 and the second GUI 825, 925 are not formed on the first and second windows 280, 285 but represent images viewed by the user. In Figs. 8 and 9, although each GUI is displayed opaque, the GUI may be partially or wholly transparent so that the surrounding landscape located below it is displayed.

The control unit 290 measures the ambient illuminance through the sensor unit 225 and increases the transmittance of the first and second windows 280 and 285 when the ambient illuminance is lower than a preset reference illuminance or a reference illuminance range (I.e., the transmittance is relatively high), and the transmittance of the first and second windows 280 and 285 is reduced (i.e., the transmittance is relatively low) when the ambient illuminance is higher than the preset reference illuminance. The controller 290 may maintain the transmittance of the first and second windows 280 and 285 when the ambient illuminance does not change. The controller 290 may store the currently set ambient illuminance and / or transmittance in the memory 220.

The memory 220 may store a data table representing the corresponding ambient illuminance values and transmittances (and / or the applied voltage values of the respective windows 280 and 285), and the controller 290 may store (And / or the applied voltage value of each window 280, 285) corresponding to the ambient illuminance value through mapping, interpolation or mathematical calculation. The control unit 290 adjusts the transmittance of each of the windows 280 and 285 to the calculated target transmittance by applying a voltage corresponding to the calculated transmittance to each of the windows 280 and 285 and more specifically to each glass.

10 is a flowchart illustrating a method of providing a virtual image according to a preferred embodiment of the present invention.

The virtual image providing method includes steps S1010 to S1060, and in this example, the GUI is illustrated as a virtual image.

In operation S1010, the control unit 290 measures ambient illuminance using the sensor unit 225 or the camera 260. [ The sensor unit 225 may include an illuminance sensor and the ambient illuminance value measured by the sensor unit 225 is output from the sensor unit 225 to the control unit 290. Since the camera 260 converts the light forming the surrounding landscape input (or photographed) through the lens system into an electrical image signal and outputs the electrical image signal to the control unit 290, the control unit 290 uses the brightness value of the light So that the ambient illuminance can be measured.

The control unit 290 uses a data table representing the corresponding ambient illuminance values and transmittances (and / or the applied voltage values of the respective windows 280 and 285) stored in the memory 220 in step S1020 (And / or the applied voltage value of each window 280, 285) corresponding to the ambient illuminance value.

The control unit 290 applies the voltage corresponding to the calculated target transmittance to the glass of each of the windows 280 and 285 so that the transmittance of each of the windows 280 and 285 becomes equal to the calculated target transmittance . That is, the control unit 290 controls the windows 280 and 285 so that the transmissivity of each of the windows 280 and 285 coincides with the calculated target transmissivity.

The control unit 290 configures the first GUI using the data stored in the memory 220 and controls the first and / or the second projectors 270 and 275 to configure the first GUI, To the user. This first GUI may be the default GUI first displayed to the user when the HMD device 100 is powered on or started, for example, a GUI 520 as shown in FIG. 5 may be displayed to the user have.

The control unit 290 receives the user input through the input / output module 210, the touch sensor 250, the camera 260, or the communication unit 240 in step S1050. A user may select a button 213, icon or menu item via the input / output module 210 or the touch sensor 250, enter a voice command via the microphone 212, A motion input may be performed or a specific command may be inputted wirelessly through the communication unit 240. [ These commands may be application execution commands, which may be any application, such as a voice recognition application, a schedule management application, a document creation application, a music application, an Internet application, a map application, a camera application, An image editing application, a search application, a file search application, a video application, a gaming application, an SNS application, a phone application, a messaging application, and the like. The gesture or motion input refers to, for example, a case where the user draws a locus of a predetermined pattern such as a circle, a triangle, or a rectangle toward the camera 260 with a hand or a finger. In this example, an application is executed according to a user input. However, such an application may be automatically executed according to the occurrence of an event such as message reception, call reception, alarm event generation, and the like.

In operation S1060, the control unit 290 provides a second GUI using the data stored in the memory 220 according to a user input, and transmits the configured second GUI to the first and / or the second projector 270, 275 to the user. This second GUI may be an application window. Changing this first GUI to the second GUI may be described as updating the GUI according to user input.

In the description of examples of the virtual image providing method described below, only the GUI displayed by the first projector 270 and the first window 280 is illustrated, but this description is applicable only to the second projector 275 and the second window 274. [ The same can be applied to the second GUI displayed by the second GUI 285.

11 to 13 are views for explaining a first example of the virtual image providing method.

First, a user selects a button 213, icon or menu item, voice command, gesture or motion input, input of a touch pattern, and the like through the touch sensor 250, the input / output module 210 or the camera 260 And the control unit 290 executes the speech recognition application in response to the user input. The control unit 290 configures the application window 1110a using the data stored in the memory 220 and displays the configured application window 1110a to the user through the first projector 270 and the first window 280 .

11 to 13, the voice recognition application window 1110a is not actually displayed on the first window 280 but is visible to the user.

Hereinafter, although the speech recognition application and other applications are illustrated as subjects of the program operation, the program operation can be described as being performed by the control unit 290. [

11 is a diagram showing an initial screen of the speech recognition application.

When the speech recognition application is initially activated, a usage guide 1130 such as " What task do you want to do? &Quot;, or " What would you like to do? &Quot; is displayed in the application window 1110a.

At the lower portion of the voice recognition application window 1110a, a voice recognition button 1120 for executing a voice recognition mode and a voice guidance button for guiding voice guidance to the usage method, if selected, And a help button indicating examples of the usage method is disposed on the other side of the voice recognition button 1120. [

Referring to FIG. 12, the user inputs a desired command, in this example, a command "Seoul weather" through the microphone 212 by voice.

The speech recognition application recognizes a voice command input by the user and converts the voice command into text data.

The speech recognition application displays the converted text data 1140 in the application window 1110b.

13, the speech recognition application searches for Seoul weather using the converted text data 1140 as a search word, and outputs search results 1150 and 1160, that is, a guidance phrase 1150 and weather information 1160, And displays it on the window 1110c. At this time, the speech recognition application may search the Seoul weather using the converted text data 1140 and the current position of the HMD device 100 (e.g., Seogyo-dong) as a search term.

The voice recognition application may convert the converted text data 1140 back into voice data, transmit the converted voice data to the voice recognition server, and provide the result of the response received from the voice recognition server to the user. Alternatively, the speech recognition application may transmit the converted text data 1140 to the speech recognition server and provide the result of the response received from the speech recognition server to the user.

14 is a diagram for explaining a second example of the virtual image providing method.

First, a user selects a button 213, icon or menu item, voice command, gesture or motion input, input of a touch pattern, and the like through the touch sensor 250, the input / output module 210 or the camera 260 And the control unit 290 executes the music application in response to the user input. The control unit 290 configures the music application window 1410 using the data stored in the memory 220 and transmits the configured music application window 1410 to the user through the first projector 270 and the first window 280 Display.

The music application plays the music set in accordance with the user's selection or default, and displays the name of the music being played, the playing time, and the like in the application window 1410. In the lower portion of the music application window 1410, a menu item 1420 such as pause, fast forward, rewind, and a List button for displaying a selectable music list may be provided.

15 is a diagram for explaining a third example of the virtual image providing method.

First, a user selects a button 213, icon or menu item, voice command, gesture or motion input, input of a touch pattern, and the like through the touch sensor 250, the input / output module 210 or the camera 260 And the control unit 290 executes the call application in response to the user input. The control unit 290 configures the call application window 1510 using the data stored in the memory 220 and transmits the configured call application window 1510 to the user through the first projector 270 and the first window 280 Display.

The call application displays a keypad 1520 for entering a telephone number and menu items 1530 such as keypad translation, recent history, contacts, favorites, and so on in the currency application window 1510.

16 is a diagram for explaining a fourth example of the virtual image providing method.

First, a user selects a button 213, icon or menu item, voice command, gesture or motion input, input of a touch pattern, and the like through the touch sensor 250, the input / output module 210 or the camera 260 And the control unit 290 executes the camera application corresponding to this user input. The control unit 290 configures the camera application window 1610 using the data stored in the memory 220 and transmits the configured camera application window 1610 to the user through the first projector 270 and the first window 280 Display.

The camera application displays the camera application window 1610 with a photographing button 1620 for taking a picture, a photographing position or a focus position indicator 1630, and menu items 1640 such as environment setting and flash.

17 is a diagram for explaining a fifth example of the virtual image providing method.

First, a user selects a button 213, icon or menu item, voice command, gesture or motion input, input of a touch pattern, and the like through the touch sensor 250, the input / output module 210 or the camera 260 And the control unit 290 executes the message application in response to the user input. The control unit 290 configures the message application window 1710 using the data stored in the memory 220 and transmits the configured message application window 1710 to the user through the first projector 270 and the first window 280 Display.

The message application displays a keypad 1720 for text entry and menu items 1730, such as message transmission, file attachment, etc., in the message application window 1710.

It will be appreciated that embodiments of the present invention may be implemented in hardware, software, or a combination of hardware and software. Such arbitrary software may be stored in a memory such as, for example, a volatile or non-volatile storage device such as a storage device such as ROM or the like, or a memory such as a RAM, a memory chip, a device or an integrated circuit, , Or a storage medium readable by a machine (e.g., a computer), such as a CD, a DVD, a magnetic disk, or a magnetic tape, as well as being optically or magnetically recordable. It will be appreciated that the memory that may be included in the HMD device is an example of a machine-readable storage medium suitable for storing programs or programs containing instructions embodying the embodiments of the present invention. Accordingly, the invention includes a program comprising code for implementing the apparatus or method as claimed in any of the claims, and a machine-readable storage medium storing such a program. In addition, such a program may be electronically transported through any medium such as a communication signal transmitted via a wired or wireless connection, and the present invention appropriately includes the same.

Also, the HMD device may receive and store the program from a program providing device connected by wire or wireless. The program providing apparatus comprising: a memory for storing a program including instructions for causing the HMD device to perform a method of providing a predetermined virtual image to a user, information necessary for a method of providing a virtual image to a user, A communication unit for performing wired or wireless communication with the apparatus, and a controller for transmitting the program to the HMD apparatus upon request or automatically by the HMD apparatus.

The present invention relates to an input / output module, and more particularly, to an input / output module, which includes a memory, a sensor portion, a battery, a power management portion, a communication portion, a touch sensor, a camera, a first projector, 280: first window, 285: second window, 290: control section

Claims (10)

A method of providing a virtual image to a user in a head-mounted display device,
In a head-mounted display device including a sensor portion, a projector for projecting light, a window including a holographic optical element having a glass with a varying transmittance and a holographic pattern, and a control portion, Detecting;
Calculating a target transmittance of the glass included in the head-mounted display device corresponding to the ambient illuminance;
Adjusting a transmittance of the glass according to the target transmittance;
Projecting a first light onto the holographic optical element using a projector included in the head-mounted display device, and reflecting the first light by the holographic optical element, thereby displaying a plurality of items corresponding to the plurality of applications Displaying a first graphical user interface comprising;
Detecting a user command for selecting one item of the plurality of items displayed in the first graphical user interface; And
Projecting a second light to the holographic optical element using the projector and the holographic optical element reflecting the second light to display a second graphical user interface of the application corresponding to the selected item and,
Wherein the holographic optical element has a focal length corresponding to a distance between the pupil or lens of the user's eye and the holographic optical element, and wherein the first light and the second light are reflected by the holographic optical element Converges to the position of the pupil or the lens of the eye of the user,
Wherein the output of the projector is adjusted based on the adjusted transmittance of the glass. ≪ Desc / Clms Page number 17 > The method according to claim 1,
Wherein the transmittance of the glass corresponding to the detected ambient illuminance is calculated by using a data table representing ambient light illuminance values and transmittances of the glass corresponding to each other How to. The method according to claim 1,
The transmittance of the glass is decreased when the detected ambient illuminance is higher than the currently set ambient illuminance and the transmittance of the glass is increased when the detected ambient illuminance is lower than the currently set ambient illuminance A method for providing a virtual image to a user in a head-mounted display device. A machine-readable storage medium having recorded thereon a program for executing a method of providing a virtual image to a user in a head-mounted display device according to any one of claims 1, 2 and 3. A head-mounted display device comprising the machine-readable storage medium of claim 4. A head-mounted display device for providing a virtual image to a user,
A sensor unit for detecting ambient illuminance;
A projector for projecting light;
A window comprising a holographic optical element having a glass and holographic pattern with varying transmittance; And
The target transmittance of the glass provided in the head-mounted display device corresponding to the ambient illuminance is calculated using the sensor unit,
Adjusting a transmittance of the glass according to the target transmittance,
A first graphical user interface comprising a plurality of items corresponding to a plurality of applications by controlling the projector to project first light to the holographic optical element and the holographic optical element reflecting the first light, Display,
Detecting a user command for selecting one of the plurality of items displayed in the first graphical user interface,
And a control unit for controlling the projector to project a second light to the glass and to display a second graphic user interface of the application corresponding to the selected item by reflecting the second light,
Wherein the holographic optical element has a focal distance corresponding to a distance between the pupil or lens of the user's eye and the holographic optical element, and wherein the first light and the second light are reflected by the holographic optical element, Converges to the position of the pupil or lens of the eye,
Wherein the output of the projector is adjusted based on the adjusted transmittance of the window. The method according to claim 6,
Wherein the holographic optical device transmits ambient light input from the outside of the head-mounted display device, and reflects and collects light input from the projector. The method according to claim 6,
Further comprising a memory storing a data table indicating ambient light illuminance values corresponding to each other and transmittances of the glass,
Wherein the controller calculates the transmittance of the glass corresponding to the detected ambient illuminance using the data table. The method according to claim 6,
Wherein the control unit controls the transmittance of the glass by applying a voltage corresponding to the target transmittance to the glass. The method according to claim 6,
Wherein the control unit decreases the transmittance of the glass when the detected ambient illuminance is higher than the currently set ambient illuminance, and when the detected ambient illuminance is lower than the currently set ambient illuminance, Of the head-mounted display device.

Images