KR102070281B1 - Head mount display device and control method thereof - Google Patents

Abstract

The present invention relates to a head-mounted display device, which is disposed at a position corresponding to both eyes, and outputs a cursor image, a sensor for detecting the movement of the head, and in response to the movement of the head. And a controller for moving the cursor image to correspond to the movement of the head.

Description

Head mounted display device and control method thereof {HEAD MOUNT DISPLAY DEVICE AND CONTROL METHOD THEREOF}

The present invention relates to a head mounted display device capable of detecting a movement and a control method thereof.

As the information age evolves rapidly, the importance of a display device that realizes a realistic screen is emphasized. An example is a head mounted display (HMD).

The head mounted display device is mainly made of a safety glasses or a helmet type device to see the screen in front of the eyes, and was developed to realize a virtual reality. The head mounted display device is a method of projecting an image by installing a small display such as a liquid crystal in a position close to both eyes (both eyes), and is currently used for space development, nuclear reactors, military institutions, and medical institutions. Various developments for games and the like are in progress.

Thanks to this improvement, smart glasses and a VR headset have been released as one embodiment of the head mounted display device. The head mounted display device implemented as a wearable device may easily execute a function that was performed in a conventional mobile terminal. In addition, by simply mounting on the head, the portability can be increased than conventional mobile terminals.

The head mounted display device may include a function of detecting various movements of a human body due to wearable characteristics. At this time, it is necessary to develop a method for controlling the head mounted display device using the detection of the movement.

One object of the present invention is to provide a method for controlling the head-mounted spectacles type terminal using the movement of the head.

The present invention relates to a head mounted display device capable of detecting a movement and a control method thereof.

Head mounted display device according to an embodiment of the present invention is disposed at a position corresponding to both eyes, the display unit for outputting a cursor (cursor) image, the sensing unit for detecting the movement of the head and the movement of the head is detected In response, the control unit for moving the cursor image to correspond to the movement of the head.

In the head mounted display device according to an exemplary embodiment, the degree to which the cursor image moves is determined based on at least one of a moving speed, a distance, and a direction of the head.

The head mounted display device according to an embodiment may stop the movement of the cursor image when the head does not move, and the controller may execute a function related to screen information output to an area where the cursor image is located. .

The head mounted display device according to an embodiment may include at least one of a color, a size, and a shape of the cursor image in order to inform the user that the function may be executed in response to the movement of the head not detected for a preset time. It is characterized by changing

The head mounted display device according to an embodiment may include a first time and a second time, and the control unit may display the cursor image when the first baby time elapses while the movement of the head is stopped. Change the value, and execute the function if the second time elapses with the cursor image changed.

The head mounted display device according to an embodiment restores the appearance of the cursor image when the head image is changed after the first time elapses and when the movement of the head is detected before the second time elapses. It is characterized by.

According to an embodiment, the head mounted display apparatus may correspond to the movement of the head based on the step of outputting a keyboard image having a plurality of virtual keys, detecting the movement of the head, and detecting the movement of the head. And moving the cursor image so that the movement of the head stops for more than a preset time, and inputting the virtual key output where the cursor image is located.

The present invention can control the graphic object output to the display using the movement of the head. Accordingly, the user may control the screen currently viewed by the user only by wearing the head mounted display device without a separate touch input.

Another effect of the present invention is that when the movement of the head corresponds to a preset pattern, different functions may be executed according to the preset pattern. Through this, the user can execute various functions only by the movement of the head.

1 is a block diagram illustrating a head-mounted spectacles-type terminal according to one embodiment disclosed herein.
2 is a conceptual diagram illustrating an embodiment of a head mounted display device according to the present invention.
3 is a flowchart illustrating a method of controlling by using the movement of the head in the head mounted display device according to the present invention.
4 is a conceptual diagram illustrating the control method of FIG. 3.
5 is a conceptual diagram illustrating a state in which the cursor image is changed when the movement of the head continues for a predetermined time in a stopped state.
6 is a conceptual diagram illustrating the movement of a cursor when the movement of the head is stopped and then moved again.
7 is a conceptual diagram illustrating a state in which screen information output to a display unit is moved in accordance with movement of the head.
8 is a conceptual diagram illustrating how screen information is moved when the movement of the head corresponds to a preset pattern.
FIG. 9 is a conceptual diagram illustrating selection of screen information when the movement of the head corresponds to a preset pattern.

Hereinafter, embodiments of the present disclosure will be described in detail with reference to the accompanying drawings, and the same or similar components will be given the same reference numerals regardless of the reference numerals, and redundant description thereof will be omitted. The suffixes "module" and "unit" for components used in the following description are given or mixed in consideration of ease of specification, and do not have distinct meanings or roles. In addition, in the following description of the embodiments disclosed herein, when it is determined that the detailed description of the related known technology may obscure the gist of the embodiments disclosed herein, the detailed description thereof will be omitted. In addition, it should be noted that the accompanying drawings are only for easily understanding the exemplary embodiments disclosed herein and are not to be construed as limiting the technical spirit disclosed in the present specification by the accompanying drawings.

1 is a block diagram illustrating a head mounted display apparatus 100 according to an exemplary embodiment disclosed herein.

The head mounted display apparatus 100 includes a wireless communication unit 110, an A / V input unit 120, a user input unit 130, a detection unit 140, an output unit 150, and a memory 160. , An interface unit 170, a controller 180, and a power supply unit 190 may be included. The components shown in FIG. 1 are not essential, so that a mobile terminal having more or fewer components may be implemented.

Hereinafter, the components will be described in turn.

The wireless communication unit 110 includes one or more modules that enable wireless communication between the head mounted display apparatus 100 and a wireless communication system or between a network in which the head mounted display apparatus 100 and the head mounted display apparatus 100 are located. can do. For example, the wireless communication unit 110 may include at least one of the broadcast receiving module 111, the mobile communication module 112, the wireless internet module 113, the short range communication module 114, and the location information module 115. Can be.

The broadcast receiving module 111 receives a broadcast signal and / or broadcast related information from an external broadcast management server through a broadcast channel.

The broadcast channel may include a satellite channel and a terrestrial channel. The broadcast management server may mean a server that generates and transmits a broadcast signal and / or broadcast related information, or a server that receives a pre-generated broadcast signal and / or broadcast related information and transmits the same to the head mounted display apparatus 100. have. The broadcast signal may include not only a TV broadcast signal, a radio broadcast signal, and a data broadcast signal, but also a broadcast signal having a data broadcast signal combined with a TV broadcast signal or a radio broadcast signal.

The broadcast associated information may mean information related to a broadcast channel, a broadcast program, or a broadcast service provider. The broadcast related information may also be provided through a mobile communication network. In this case, it may be received by the mobile communication module 112.

The broadcast related information may exist in various forms. For example, it may exist in the form of Electronic Program Guide (EPG) of Digital Multimedia Broadcasting (DMB) or Electronic Service Guide (ESG) of Digital Video Broadcast-Handheld (DVB-H).

The broadcast receiving module 111 may include, for example, Digital Multimedia Broadcasting-Terrestrial (DMB-T), Digital Multimedia Broadcasting-Satellite (DMB-S), Media Forward Link Only (MediaFLO), and Digital Video Broadcast (DVB-H). Digital broadcast signals may be received using digital broadcasting systems such as handheld and integrated services digital broadcast-terrestrial (ISDB-T). Of course, the broadcast receiving module 111 may be configured to be suitable for not only the above-described digital broadcasting system but also other broadcasting systems.

The broadcast signal and / or broadcast related information received through the broadcast receiving module 111 may be stored in the memory 160.

The mobile communication module 112 transmits and receives a radio signal with at least one of a base station, an external terminal, and a server on a mobile communication network. The wireless signal may include various types of data according to transmission and reception of a voice call signal, a video call signal, or a text / multimedia message.

The mobile communication module 112 is configured to implement a video call mode and a voice call mode. The video call mode refers to a state of making a call while viewing the other party's video, and the voice call mode refers to a state of making a call without viewing the other party's image. In order to implement the video call mode and the voice call mode, the mobile communication module 112 is configured to transmit and receive at least one of audio and video.

The wireless internet module 113 refers to a module for wireless internet access and may be embedded or external to the head mounted display apparatus 100. Wireless Internet technologies include Wireless LAN (WLAN), Wireless Fidelity (WiFi) Direct, Digital Living Network Alliance (DLNA), Wireless broadband (Wibro), World Interoperability for Microwave Access (Wimax), High Speed Downlink Packet Access (HSDPA), and more. This can be used.

The short range communication module 114 refers to a module for short range communication. Short-range communication technologies include Bluetooth ™, Radio Frequency Identification (RFID), Infrared Data Association (IrDA), Ultra Wideband (UWB), ZigBee, and Near Field Communication (NFC). Can be.

The location information module 115 is a module for acquiring the location of the head mounted display apparatus 100. Examples of the location information module 115 include a global position system (GPS) module or a wireless fidelity (WiFi) module.

Referring to FIG. 1, the A / V input unit 120 is for inputting an audio signal or a video signal, and may include a camera 121 and a microphone 122. The camera 121 processes image frames such as still images or moving images obtained by the image sensor in a video call mode or a photographing mode. The processed image frame may be displayed on the display unit 151.

The image frame processed by the camera 121 may be stored in the memory 160 or transmitted to an external device through the wireless communication unit 110. In addition, the position information of the user may be calculated from the image frame acquired by the camera 121. Two or more cameras 121 may be provided according to the use environment.

The microphone 122 receives an external sound signal by a microphone in a call mode, a recording mode, a voice recognition mode, etc., and processes the external sound signal into electrical voice data. The processed voice data may be converted into a form transmittable to the mobile communication base station through the mobile communication module 112 and output in the call mode. The microphone 122 may implement various noise removing algorithms for removing noise generated in the process of receiving an external sound signal.

The user input unit 130 generates input data according to a control command for controlling the operation of the head mounted display apparatus 100 applied from the user. The user input unit 130 may include a key pad, a dome switch, a touch pad (constant voltage / capacitance), a jog wheel, a jog switch, and the like.

The sensing unit (or sensing unit) 140 may be a head mounted display such as a position of the head mounted display device 100, presence or absence of user contact, orientation of the head mounted display device 100, acceleration / deceleration of the head mounted display device 100, and the like. The current state of the device 100 is sensed to generate a detection signal (or a sensing signal) for controlling the operation of the head mounted display device 100. In addition, the detector 140 may detect whether the power supply unit 190 supplies power or whether the interface unit 170 is coupled to an external device.

The output unit 150 is used to generate an output related to sight, hearing, or tactile sense, and may include a display unit 151, an audio output module 153, an alarm unit 154, and a haptic module 155. have.

The display unit 151 displays (outputs) information processed by the head mounted display apparatus 100. For example, when the head mounted display device is in a call mode, the head mounted display device displays a user interface (UI) or a graphic user interface (GUI) related to the call. When the head mounted display apparatus 100 is in a video call mode or a photographing mode, the display unit 151 displays a photographed and / or received image, a UI, and a GUI.

The display unit 151 is a liquid crystal display (LCD), a thin film transistor-liquid crystal display (TFT LCD), an organic light-emitting diode (OLED), a flexible display (flexible) display, a 3D display, and an e-ink display.

Some of these displays may be configured to be transparent or light transmissive so that they can be seen through them. This may be referred to as a transparent display. A representative example of the transparent display is TOLED (Transparant OLED).

In addition, the display unit 151 may be configured as a stereoscopic display unit 152 for displaying a stereoscopic image.

Here, the stereoscopic image represents a three-dimensional stereoscopic image, and the three-dimensional stereoscopic image represents a gradual depth and reality in which an object is placed on a monitor or a screen. This is a video that lets you feel the same as space. 3D stereoscopic images are implemented using binocular disparity. Binocular parallax means the parallax made by the position of two eyes that are separated. When two eyes see different two-dimensional images and the images are transferred to the brain through the retina and merged, the stereoscopic image can feel the depth and reality of the stereoscopic image. .

The stereoscopic display unit 152 may be a three-dimensional display method such as a stereoscopic method (glasses method), an auto stereoscopic method (glasses-free method), a projection method (holographic method). Stereoscopic methods commonly used in home television receivers include Wheatstone stereoscope.

Examples of the auto stereoscopic method include a parallax barrier method, a lenticular method, an integrated imaging method, a switchable lens, and the like. Projection methods include reflective holographic methods and transmissive holographic methods.

Generally, a 3D stereoscopic image is composed of a left image (left eye image) and a right image (right eye image). A top-down method in which the left and right images are arranged up and down in one frame according to the way in which the left and right images are combined into a 3D stereoscopic image, and the left and right images are in the left and right in one frame. L-to-R (left-to-right, side by side) method that is arranged as a checker board method to arrange the pieces of the left and right images in the form of tiles, and the left and right images in columns Or an interlaced method of alternately arranging rows, and a time sequential (frame by frame) method of alternately displaying left and right images by time.

In addition, the 3D thumbnail image may generate a left image thumbnail and a right image thumbnail from the left image and the right image of the original image frame, respectively, and combine them to generate one 3D thumbnail image. In general, a thumbnail refers to a reduced image or a reduced still image. The left image thumbnail and the right image thumbnail generated as described above may be displayed with a three-dimensional space by displaying a left and right distance difference on the screen by a depth corresponding to the parallax of the left image and the right image.

The left image and the right image necessary for implementing the 3D stereoscopic image may be displayed on the stereoscopic display 152 by a stereoscopic processor (not shown). The stereo processing unit receives the 3D image and extracts the left and right images therefrom, or receives the 2D image and converts it to the left and right images.

On the other hand, when the display unit 151 and a sensor for detecting a touch operation (hereinafter referred to as a touch sensor) form a mutual layer structure (hereinafter referred to as a touch screen), the display unit 151 outputs In addition to the device can also be used as an input device. The touch sensor may have, for example, a form of a touch film, a touch sheet, a touch pad, or the like.

The touch sensor may be configured to convert a change in pressure applied to a specific portion of the display unit 151 or capacitance generated at a specific portion of the display unit 151 into an electrical input signal. The touch sensor may be configured to detect not only the position and area where the touch object is touched on the touch sensor, but also the pressure at the touch. Here, the touch object is an object applying a touch to the touch sensor and may be, for example, a finger, a touch pen or a stylus pen, a pointer, or the like.

If there is a touch input to the touch sensor, the corresponding signal (s) is sent to the touch controller. The touch controller processes the signal (s) and then transmits the corresponding data to the controller 180. As a result, the controller 180 can determine which area of the display unit 151 is touched.

Referring to FIG. 1, a proximity sensor 141 may be disposed in an inner region of the head mounted display device surrounded by the touch screen or near the touch screen. The proximity sensor 141 may be provided as an example of the sensing unit 140. The proximity sensor 141 refers to a sensor that detects the presence or absence of an object approaching a predetermined detection surface or an object present in the vicinity thereof without a mechanical contact by using an electromagnetic force or infrared rays. The proximity sensor 141 has a longer life and higher utilization than a contact sensor.

Examples of the proximity sensor 141 include a transmission photoelectric sensor, a direct reflection photoelectric sensor, a mirror reflection photoelectric sensor, a high frequency oscillation proximity sensor, a capacitive proximity sensor, a magnetic proximity sensor, and an infrared proximity sensor. When the touch screen is capacitive, the touch screen is configured to detect the proximity of the pointer by a change in an electric field according to the proximity of a conductive object (hereinafter, referred to as a pointer). In this case, the touch screen (touch sensor) may be classified as a proximity sensor.

Hereinafter, for convenience of explanation, an act of allowing the pointer to be recognized without being in contact with the touch screen so that the pointer is located on the touch screen is referred to as a "proximity touch", and the touch The act of actually touching the pointer on the screen is called "contact touch". The position at which the proximity touch is performed by the pointer on the touch screen means a position where the pointer is perpendicular to the touch screen when the pointer is in proximity touch.

The proximity sensor 141 detects a proximity touch and a proximity touch pattern (for example, a proximity touch distance, a proximity touch direction, a proximity touch speed, a proximity touch time, a proximity touch position, and a proximity touch movement state). Information corresponding to the sensed proximity touch operation and proximity touch pattern may be output on the touch screen.

When the stereoscopic display unit 152 and the touch sensor form a mutual layer structure (hereinafter, referred to as a “stereoscopic touch screen”), or when the stereoscopic display unit 152 and the 3D sensor detecting a touch operation are combined with each other. The stereoscopic display unit 152 may also be used as a three-dimensional input device.

As an example of the 3D sensor, the sensing unit 140 may include a proximity sensor 141, a stereoscopic touch sensing unit 142, an ultrasonic sensing unit 143, and a camera sensing unit 144.

The proximity sensor 141 measures the distance between a sensing object (for example, a user's finger or a stylus pen) and a detection surface to which a touch is applied without mechanical contact by using an electromagnetic force or infrared rays. The head mounted display apparatus 100 recognizes which part of the stereoscopic image is touched using this distance. In particular, when the touch screen is capacitive, the proximity of the sensing object is detected by a change in the electric field according to the proximity of the sensing object, and the touch screen is configured to recognize a three-dimensional touch using the proximity.

The stereoscopic touch sensing unit 142 is configured to detect the strength or duration of the touch applied to the touch screen. For example, the three-dimensional touch sensing unit 142 detects a pressure to apply a touch, and recognizes that the touch is strong as the touch on an object located farther from the touch screen toward the inside of the terminal.

The ultrasonic sensing unit 143 uses ultrasonic waves to recognize position information of the sensing object.

The ultrasonic sensing unit 143 may be formed of, for example, an optical sensor and a plurality of ultrasonic sensors. The optical sensor is configured to sense light, and the ultrasonic sensor is configured to detect ultrasonic waves. Since light is much faster than ultrasonic waves, the time that light reaches the optical sensor is much faster than the time that ultrasonic waves reach the ultrasonic sensor. Therefore, the position of the wave generation source can be calculated using the time difference from the time when the ultrasonic wave reaches the light as the reference signal.

The camera sensing unit 144 includes at least one of a camera 121, a photo sensor, and a laser sensor.

For example, the camera 121 and the laser sensor are combined with each other to detect a touch of a sensing object with respect to a 3D stereoscopic image. When the distance information detected by the laser sensor is added to the two-dimensional image captured by the camera, three-dimensional information may be obtained.

As another example, a photo sensor may be stacked on the display element. The photo sensor is configured to scan the movement of the sensing object in proximity to the touch screen. More specifically, the photo sensor mounts a photo diode and a transistor (TR) in a row / column and scans contents mounted on the photo sensor by using an electrical signal that varies according to the amount of light applied to the photo diode. That is, the photo sensor calculates the coordinates of the sensing object according to the amount of change of light, thereby obtaining the position information of the sensing object.

The sound output module 153 may output audio data received from the wireless communication unit 110 or stored in the memory 160 in a call signal reception, a call mode or a recording mode, a voice recognition mode, a broadcast reception mode, and the like. The sound output module 153 may also output a sound signal related to a function (eg, a call signal reception sound, a message reception sound, etc.) performed by the head mounted display apparatus 100. The sound output module 153 may include a receiver, a speaker, a buzzer, and the like.

The alarm unit 154 outputs a signal for notifying occurrence of an event of the head mounted display apparatus 100. Examples of events generated in the head mounted display apparatus 100 include call signal reception, message reception, key signal input, and touch input. The alarm unit 154 may output a signal for notifying occurrence of an event by using a form other than a video signal or an audio signal, for example, vibration. The video signal or the audio signal may also be output through the display unit 151 or the sound output module 153, so that the display unit 151 and the sound output module 153 may be classified as part of the alarm unit 154. .

The haptic module 155 generates various tactile effects that a user can feel. A representative example of the tactile effect generated by the haptic module 155 may be vibration. The intensity and pattern of vibration generated by the haptic module 155 may be controlled by a user's selection or setting of the controller 180. For example, the haptic module 155 may synthesize different vibrations and output them or sequentially output them.

In addition to the vibration, the haptic module 155 may be configured to apply a pin array that vertically moves with respect to the contact skin surface, a jetting force or suction force of air through the jetting or inlet, a grazing to the skin surface, contact of the electrode, electrostatic force, and the like. Various tactile effects can be generated, such as the effects of the heat-absorption and the reproduction of the sense of cold using the element capable of generating heat.

The haptic module 155 may not only deliver the haptic effect through direct contact, but may also be implemented to allow the user to feel the haptic effect through a muscle sense such as a finger or an arm. Two or more haptic modules 155 may be provided according to a configuration aspect of the head mounted display apparatus 100.

The memory 160 may store a program for the operation of the controller 180 and may temporarily store input / output data (for example, a phone book, a message, a still image, a video, etc.). The memory 160 may store data relating to various patterns of vibration and sound output when a touch is input on the touch screen.

The memory 160 may include a flash memory type, a hard disk type, a multimedia card micro type, a card type memory (for example, SD or XD memory), RAM random access memory (RAM), static random access memory (SRAM), read-only memory (ROM), electrically erasable programmable read-only memory (EEPROM), programmable read-only memory (PROM), magnetic memory, magnetic It may include a storage medium of at least one type of a disk and an optical disk. The head mounted display apparatus 100 may be operated in connection with a web storage that performs a storage function of the memory 160 on the Internet.

The interface unit 170 serves as a path to all external devices connected to the head mounted display apparatus 100. The interface unit 170 receives data from an external device, receives power, and transmits the data to each component inside the head mounted display apparatus 100, or transmits data inside the head mounted display apparatus 100 to an external device. do. For example, a port for connecting a device equipped with a wired / wireless headset port, an external charger port, a wired / wireless data port, a memory card port, and an identification module. The audio input / output (I / O) port, video input / output (I / O) port, earphone port, and the like may be included in the interface unit 170.

On the other hand, the identification module is a chip that stores a variety of information for authenticating the use authority of the head-mounted display device 100, a user identification module (UIM), subscriber identity module (SIM), general purpose It may include a universal subscriber identity module (USIM). A device equipped with an identification module (hereinafter referred to as an 'identification device') may be manufactured in the form of a smart card. Therefore, the identification device may be connected to the head mounted display device 100 through the interface unit 170.

In addition, the interface unit 170 may be a passage for supplying power from the cradle to the head mounted display apparatus 100 when the head mounted display apparatus 100 is connected to an external cradle, or may be provided by the user. Various command signals input from the cradle may be a passage through which the head mounted display apparatus 100 is transmitted. Various command signals or power input from the cradle may be operated as signals for recognizing that the head mounted display apparatus 100 is correctly mounted on the cradle.

The controller 180 typically controls the overall operation of the head mounted display apparatus 100. For example, control and processing related to voice call, data communication, video call and the like are performed. The controller 180 may include a multimedia module 181 for playing multimedia. The multimedia module 181 may be implemented in the controller 180 or may be implemented separately from the controller 180.

In addition, the controller 180 may perform a pattern recognition process for recognizing a writing input or a drawing input performed on the touch screen as text and an image, respectively.

In addition, when the state of the head mounted display apparatus 100 satisfies a set condition, the controller 180 may execute a lock state for restricting input of a user's control command to applications. In addition, the controller 180 may control the lock screen displayed in the locked state based on the touch input sensed by the display unit 151 in the locked state.

The power supply unit 190 receives an external power source and an internal power source under the control of the controller 180 to supply power for operation of each component.

Various embodiments described herein may be implemented in a recording medium readable by a computer or similar device using, for example, software, hardware or a combination thereof.

According to a hardware implementation, the embodiments described herein include application specific integrated circuits (ASICs), digital signal processors (DSPs), digital signal processing devices (DSPDs), programmable logic devices (PLDs), and field programmable gate arrays (FPGAs). It may be implemented using at least one of processors, controllers, micro-controllers, microprocessors, and electrical units for performing other functions. In some cases, the embodiments described herein may be implemented by the controller 180 itself.

According to the software implementation, embodiments such as the procedures and functions described herein may be implemented as separate software modules. Each of the software modules may perform one or more functions and operations described herein.

The software code may be implemented as a software application written in a suitable programming language. The software code may be stored in the memory 160 and executed by the controller 180.

Hereinafter, the head mounted display device 100 or the head mounted display device 100 in which the components of the head mounted display device 100 or the head mounted display device 100 are disposed according to an embodiment of the present invention described with reference to FIG. 1. Look at the structure of).

2 is a conceptual diagram illustrating an embodiment of a head mounted display device 100 according to the present invention.

Referring to FIG. 2, the head mounted display apparatus 100 according to the present invention includes a main body 310, a display unit 151, and a controller 180.

In addition, the head mounted display apparatus 100 according to the present invention may further include a motion detection unit.

The head mounted display apparatus 100 according to the present invention may be implemented as a head mounted display (HMD). As a specific example, the present invention may be implemented to be mounted on the head in various forms such as smart glasses and a VR headset.

The main body 310 is formed to be mountable on the head. For example, the frame and the leg portion of the smart glasses may be implemented.

The display unit 151 may be combined with the main body 310 and disposed at positions corresponding to both eyes. In addition, the display unit 151 may have light transparency and output visual information.

The visual information refers to a virtual object generated by the head mounted display apparatus 100 or received from an external device. For example, the virtual object may mean an application, an icon corresponding to the icon, content, or a UI in a call mode. This may be generated by the controller 180 or received from a mobile terminal such as a smart phone.

In addition, since the display unit 151 is light transmissive, the user can see the external environment through the display unit 151.

In addition, as an example, the display unit 151 may display an external environment and output information on any external object constituting the external environment. For example, the external object may mean any object that can be seen through the user's field of view, such as a tree or a building.

As described above, the controller 180 may control the head mounted display apparatus 100. In detail, the controller 180 may detect the movement of the head and control the graphic object output to the display unit 151. For example, the controller 180 may control the graphic object to correspond to the movement trajectory of the head and the movement trajectory of the graphic object output to the display unit 151.

In addition, the controller 180 may be mounted on the main body 310 of the head mounted display apparatus 100 or may be integrally formed with the main body 310. In another embodiment, it may be disposed away from the main body 310.

The camera 121 may be disposed in front of at least one of the left eye and right eye display units 151. Alternatively, the frame 310 may be disposed on one or both sides of the frame 310 to capture a space other than the wearer's view.

The user input unit 130 may be implemented as a separate touch panel on one or both sides of the frame 310. Or it may be implemented as a physical key. For example, an ON / OFF switch of the power supply may be implemented on one side of the frame 310.

The sensing unit 140 may be integrally formed with the main body. For example, the sensing unit 140 may be formed in a frame.

In another embodiment, it may be implemented as a separate external device device connected to the main body 310. Accordingly, the user may input a specific command to a separate external device device. Alternatively, the display unit 151 may be implemented as a touch screen to directly receive a control command from the user.

In another embodiment, the present invention may be implemented as a module that recognizes a voice command of a user. In this way, the user may input a specific command to the main body 310 through voice.

Meanwhile, as an embodiment of the head mounted display device, smart glasses have been released. Smart glasses implemented as a wearable device can easily execute functions that were previously executed in a mobile terminal.

The display unit 151 of the smart glasses is disposed in both eyes of the user, and it is necessary to use various control methods other than touch input. Thus, to use the characteristics of the head-mounted display device, a method of using the movement of the head can be considered.

Hereinafter, a method of controlling the head mounted display apparatus 100 using the movement of the head will be described. 3 is a flowchart illustrating a method of controlling a cursor image output to the display unit 151 using movement of the head, and FIG. 4 is a conceptual diagram illustrating the control method of FIG. 3.

The head mounted display device according to an embodiment of the present disclosure may proceed to detect a movement of the head (S310).

The head means a portion that becomes the head of the user. The sensing unit 140 may detect the movement of the head by detecting the movement of the main body 100 mounted on the head.

The detector 140 may be configured as a sensor capable of detecting a movement. The sensor for detecting the movement may be configured with various sensors capable of detecting the movement, such as a geomagnetic sensor and an acceleration sensor. The acceleration sensor is a sensor that detects motion by measuring acceleration and vibration of an object. In addition, the geomagnetic sensor is a sensor for detecting the motion by detecting the magnetic field.

The movement of the head may be defined by various variables such as movement speed, acceleration, direction of movement, distance moved, and rotation. For example, the movement of the head may be a movement of shaking the head up and down. It may also be a movement to shake the head from side to side. In addition, the shaking up and down, or left and right may be different movements according to the speed.

In this case, the head mounted display apparatus may proceed to detect the movement of the head by a preset control command of the user. The preset control command of the user may be implemented by various types of control commands such as a voice command, a button command, and a touch command. In addition, when the head mounted display device is worn on the head, the head mounted display device may detect the movement of the head in real time.

In response to the movement of the head, the head mounted display device according to an embodiment of the present invention may move the cursor image to correspond to the movement of the head (S320).

The cursor refers to a symbol indicating an object to be controlled, such as a character or a figure, on a display device screen of an electronic device such as a computer. For example, in the case of a computer, it may mean an arrow-shaped image being output to a monitor. The arrow-shaped cursor image may serve to execute a function indicated by the cursor image by an external input, for example, a mouse input.

The cursor image may have various shapes, colors, and sizes. For example, the cursor image may have an arrow shape which is a basic form of a Windows operating system. In addition, the cursor image may be circular.

The cursor image may change at least one of a shape, a size, and a color based on the movement of the head. That is, the cursor image may be a static image or a dynamic image. The cursor image may be changed by the user's selection or may be automatically changed by the controller 180 when a preset condition is satisfied. For example, the cursor image may execute an arbitrary function using the movement of the head, and may be changed to gradually increase in size to indicate that the arbitrary function is being executed.

When the movement of the head is detected, the controller 180 can move the cursor image output to the display unit 151. The cursor image may be moved to correspond to the movement of the head.

For example, when the head is moved to the right side and then moved further downward, the cursor image output to the display unit 151 is moved to the right side again after moving to the right side from the current output position. can see. That is, the controller 180 can move the cursor image to correspond to the movement trajectory of the head.

In addition, the controller 180 may determine the degree to which the cursor image is moved based on at least one of a speed, a direction, and a distance of the movement of the head. For example, when the movement speed of the head is fast, the controller 180 can quickly move the cursor image to correspond to the movement speed of the head.

When the movement of the head is no longer detected, the head-mounted display device according to an embodiment of the present invention outputs to the area where the cursor image is located when a predetermined time elapses while the movement of the head is stopped. The execution of a function related to the displayed screen information may be performed (S330).

When the movement of the head is no longer detected, that is, when the stationary state of the head continues, the controller 180 may determine that the head is not moving.

If the movement of the head is not detected by the detector 140, the controller 180 may not move the cursor image any more. In addition, the controller 180 may determine whether the stop state satisfies a preset condition. The preset condition may be whether or not a preset time has elapsed to the stop state. The controller 180 may determine whether to select screen information output in an area where the cursor image is located, according to whether the preset time has elapsed.

For example, when the controller 180 determines that a predetermined time has elapsed while the movement of the head is not detected, the controller 180 provides a function related to the screen information output in the area where the cursor image is located. Can be run.

An example of executing the above function will be described in detail. The display unit 151 of the head mounted display device according to the present invention includes an area 400 where a keyboard image having a plurality of virtual keys is output and the keyboard 400. It may include an area 410 for outputting a message according to the input. In this case, the cursor image 420 may be output on the keyboard image 400. Thereafter, when the cursor image 420 is not moved for a preset time, the controller 180 may select a virtual key output in an area where the cursor image 420 is located. That is, the controller 180 can receive the selected virtual key input.

As another example, the display unit 151 may output icons of a plurality of applications. In this case, the controller 180 may determine whether a preset time has elapsed while the movement of the head is stopped. In this case, the cursor image may not be moved but may be output in a state of being stopped for a preset time. When the cursor image is not moved and is stopped for a preset time, an icon output in an area where the cursor image is located among icons of the plurality of applications is selected, and an application corresponding to the selected icon may be executed.

Referring to FIG. 4A, a user may be wearing a head mounted display device. In this case, the user may see that screen information is output to the display unit 151 of the head mounted display device. For example, the screen information may be a screen capable of receiving input of the keyboard 400 and a screen 410 of outputting received information of the input of the keyboard 400. In this case, the cursor image 420 may be located in an area in which any one key 401 of the plurality of keys constituting the keyboard 400 is located.

At this time, referring to Figure 4 (b), the user can move the head. In this case, the sensing unit 140 may detect the movement of the head. For example, the movement of the head may be a movement of the head to the left. In this case, the controller 180 may move the cursor image output to the display unit 151 in response to the movement trajectory of the head. Thus, the cursor image may also be shifted left from the current position. For example, as shown in FIG. 4B, the cursor image 420 may be moved to an area of the other key 402 located in the left area of the one key 401.

Referring to FIG. 4C, after moving the cursor image, the user may stop the head without moving the head. In this case, the sensing unit 140 may no longer detect the movement of the head. If the movement of the head is no longer detected by the detector 140, the controller 180 may determine that the head is stopped. In this case, the controller 180 may determine whether a preset time has elapsed while the head is stopped.

 If the controller 180 determines that the predetermined time has elapsed while the head is stopped, the controller 180 outputs the other one to the area where the cursor image 430 is located, as shown in FIG. The key 402 of may be selected. In this case, the controller 180 may change the cursor image 430 to inform that the other key 402 may be selected.

For example, when the state where the head is stopped is detected, the controller 180 may gradually change the cursor image (430) gradually after the stop, in response to the time for which the head is held. For example, referring to FIG. 4C, the cursor image may be located on the “f” key of the virtual keyboard. In this case, when the cursor image 430 positioned on the virtual “f” key is stopped for a preset time, the “f” key may be selected. In this case, the controller 180 can greatly change the size of the cursor image 430 to inform that the virtual "f" key can be selected. Therefore, the “f” may be input to an area 410 in which a message is output among one areas of the display unit 151.

In addition, after the virtual key is input, the cursor image may be restored to a state before being changed again. For example, after the virtual key is input, the size of the cursor image may be returned to the size before the change. In this case, the controller 180 may again move the cursor image to correspond to the movement of the head based on the detection of the movement of the head by the sensing unit 140.

In the above, the method of moving the cursor image using the movement of the head has been described. Through this, in the head mounted display device, the user can move the cursor image only by simple movement of the head without a separate touch input, thereby increasing convenience.

Hereinafter, the state in which the cursor image is changed in response to the movement of the head will be described in more detail. 5 is a conceptual diagram illustrating that the cursor image is changed when the movement of the head is not detected.

Referring to FIGS. 5A and 5B, a user who uses the head mounted display apparatus may move the head while wearing the head mounted display apparatus.

In this case, the sensing unit 140 may detect the movement of the head. When the sensing unit 140 detects the movement of the head, the controller 180 may move a cursor image to correspond to the movement trajectory of the head. For example, when the user rotates the head to the left, the detector 140 may detect the movement of rotating the left side. In addition, when the sensing unit 140 detects a movement to rotate to the left, the controller 180 can move the cursor image to the left.

Thereafter, referring to FIG. 5C, the user may no longer move the head. In this case, the detector 140 may not detect the movement of the head. If the movement of the head is not detected by the detector 140, the controller 180 may determine that the head is stationary.

Thereafter, the controller 180 can determine whether the stop state lasts for a preset time. The preset time may include a first time and a second time. For example, when dividing the predetermined time in half, the first half time may be defined as the first time, and the second half time may be defined as the second time.

The controller 180 may change at least one of a shape, a size, and a color of a cursor image when the first time elapses. For example, as shown in (b) of FIG. 5, when the cursor image is circular, the size of the circular shape may be changed. For example, the size of the cursor images 520 and 530 may gradually increase in response to the passage of time. Through this, the user may know that a function related to the screen information output in the area where the current cursor image is located may be selected.

Thereafter, the cursor image gradually increases, and as shown in (c) of FIG. 5, when the second time elapses while the cursor image is stopped, an area where the cursor image 530 is located The screen information output on the screen may be selected. In this case, when the cursor image reaches the second time, the cursor image may be resized maximum. For example, the cursor image 530 may be output at the position of the "f" key, and when the second time elapses, the "f" key may be selected. When "f" is selected, "f" may be output to the message output area 410.

After the function related to the screen information output in the area where the cursor image is located is selected, the head mounted display device according to the present invention can restore the cursor image to the image before the change and return to a state where another function can be selected again. have. For example, after any one of the plurality of virtual keys is selected, the cursor image can be returned to its pre-change size and back to a state for selecting another key. In this case, the virtual keyboard image output to the display unit 151 may be continuously output.

In the above, the cursor image is changed according to the movement of the head. In this way, the user can check what information is currently selected on the display unit.

Hereinafter, a control method when the movement of the head is detected again after the cursor image is changed using the movement of the head. 6 is a conceptual diagram illustrating that the cursor image is changed when the movement of the head is stopped and the movement of the head is detected again.

Referring to (a) and (b) of FIG. 6, the user may move the cursor image by using the movement of the head. At this time, the cursor image is moved to correspond to the movement trajectory of the head. For example, when the head of the user is moved to the left, the cursor image may also be moved to the left from the current position.

Thereafter, as illustrated in FIG. 6B, when the movement of the head is no longer detected, the controller 180 may change the cursor image. In this case, the change 520 of the cursor image may represent that a function related to screen information currently being output in the region where the cursor is located is selected.

The controller 180 may change the cursor image to indicate that the function is being selected in response to the movement of the head not being detected for a preset time. In this case, the preset time may include a first time and a second time. For example, the first time may mean a time from the start of the preset time to half, and the second time may mean a time of the other half of the preset time.

In this case, when the first time elapses without the movement of the head being detected, the controller 180 may change the cursor image 520. Thereafter, as shown in (c) of FIG. 6, in the state where the cursor image 520 is changed, before the second time elapses, the detector 140 may detect the movement of the head. In this case, the controller 180 may determine that the head is moving again, and determine that a function that is currently selected is not selected. Accordingly, the controller 180 may restore the cursor image 520 indicating that the selection is currently made to the image 610 before the change.

In addition, the controller 180 may move the cursor image to correspond to the movement trajectory of the head. For example, when the user detects a motion of turning his head to the left again while the user is stationary, the cursor image is restored to the pre-change image 610, and the restored cursor image 610 corresponds to the movement trajectory of the head. It can be moved from the current position to the left area.

In the above, the method of controlling the cursor image when the movement of the head is detected again for a predetermined time while the movement of the head is stopped has been described. The user may not select the currently selected function by moving the head again before the preset time elapses. In addition, the user can recognize that the cursor image is restored to the image before the change again, so that the selected function is not selected.

The following describes changing (or moving) the screen using the movement of the head. 7 is a conceptual diagram illustrating a state in which a screen is changed (or moved) by using the movement of the head.

As shown in FIG. 7A, the head mounted display device according to the present invention may detect the movement of the head. The movement of the head may be detected by the detector 140.

When the movement of the head is detected by the sensor 140, the controller 180 changes at least one of screen information and a cursor image output to the display unit 151 based on the movement of the head. Can be. For example, the controller 180 may change screen information. In this case, the changed screen information may be screen information displayed on one area or the entire area of the display unit 151. In this case, only the screen information is changed by the movement of the head, and the cursor image may be at a fixed position. In addition, the controller 180 may simultaneously change the screen information and the cursor image.

For example, the controller 180 may change (or move) screen information of one region output to the display unit 151 to correspond to the movement of the head. In this case, the cursor image may be fixed at the center of the screen. That is, the controller 180 may change the screen so that the screen may appear to be moved to the eyes of the user.

Referring to FIG. 7B, the movement of the head may be a movement to the right. In this case, the display unit 151 may output a keyboard unit 410 including a character output unit and a virtual key. When the head moves to the right, the controller 180 can move the screen information output to the display unit 151 to the right as a whole. In addition, as illustrated in FIG. 7B, the controller 180 may move only one region of the display unit 151 to the right. For example, the controller 180 may move only the keyboard unit 710 to the right.

As another example, referring to FIG. 7C, the user may move the head to the left. In this case, the keyboard unit output to the display unit 151 may be moved 720 to the left to correspond to the movement of the head.

In the above, the screen information output to the display unit 151 is moved corresponding to the movement of the head. Using this, the user can select the screen information of the desired position by the movement of the head only.

In addition to moving the screen information of the display unit corresponding to the movement of the head, the movement of the head may coincide with a preset pattern. In this case, a function previously matched to the predetermined pattern may be executed.

Hereinafter, when the movement of the head coincides with the preset pattern, the function matching the pattern will be described. FIG. 8 is a conceptual diagram illustrating that a function previously matched is executed when the movement of the head corresponds to a preset pattern.

The preset pattern may have various patterns such as a movement of shaking the head up and down or left and right, a movement of tilting the head, and the like. In addition, the controller 180 may match an arbitrary function with the preset pattern in advance. For example, the controller 180 may match a function of terminating an application currently being executed with a movement of shaking the head from side to side.

For example, as shown in FIG. 8A, the head mounted display device according to the present invention may be currently executing a web browser application. In this case, the web browser page 810 may be output to the display unit 151. In the state where the web browser page 810 is output, the detector 140 may detect the movement of the head. In this case, when the head moves to the right, a function matched to the movement of the head may be executed. For example, referring to FIG. 8B, the function matched to the movement to the right may be a function of moving from the currently output web browser page 810 to the previous web browser page 820.

As another example, the sensing unit 140 may detect a movement of the head moving to the left while the web browser page 810 is output. In this case, the movement to the left may be matched with a function for outputting the next web browser page 830 of the currently output web browser page 810.

In the above, while the function of an arbitrary application is being executed, it demonstrated that the function related to the said running application is executed based on the movement of a head. Through this, the user can control the function of the application with only a simple head movement.

Hereinafter, it will be described that the preset function is executed based on the movement of the head. 9 is a conceptual diagram illustrating that a preset function is executed based on the movement of the head.

As shown in (a) and (b) of FIG. 9, the user may move the cursor image by using the movement of the head. For example, if the user moves the head to the left, the cursor image may also be moved to the left.

Thereafter, as illustrated in FIG. 9C, the user may use the movement of the head to execute a function related to the screen information output at the position where the cursor image is moved. In this case, when the movement of the head corresponds to a preset pattern, a function matching the pattern may be executed. For example, the preset pattern may include various patterns such as moving the head up and down, moving the left and right, or rotating the head. In this case, the controller 180 may match a function with a preset pattern in advance. For example, the controller 180 may match the pattern of moving the head up and down to execute a function displayed in a region where a cursor image is currently output.

For example, referring to FIG. 9C, the user may move the head up and down. In this case, a function related to screen information output in the area where the cursor image is located may be executed. In FIG. 9C, a virtual key “f” may be input.

Although not shown, unlike FIG. 9, an icon related to an application may be currently displayed on the display unit 151. In this case, when the cursor image is located on the icon, an application related to the icon may be executed by the vertical movement of the head.

In addition, various functions may be matched to various movement patterns of the head. For example, the movement of the head to the left or right may match the function of terminating the program currently being executed.

In the above, when the movement of the head corresponds to the preset pattern, the function matching the pattern has been described. The user can execute various functions using various head movements.

The present invention can control the graphic object output to the display using the movement of the head. Accordingly, the user may control the screen currently viewed by the user only by wearing the head mounted display device without a separate touch input.

Another effect of the present invention is that when the movement of the head corresponds to a preset pattern, different functions may be executed according to the preset pattern. Through this, the user can execute various functions only by the movement of the head.

In addition, according to one embodiment disclosed herein, the above-described method may be implemented as code that can be read by a processor in a medium in which a program is recorded. Examples of processor-readable media include ROM, RAM, CD-ROM, magnetic tape, floppy disk, optical data storage, and the like, and may be implemented in the form of a carrier wave (for example, transmission over the Internet). Include.

The above-described mobile terminal is not limited to the configuration and method of the above-described embodiments, but the embodiments may be configured by selectively combining all or some of the embodiments so that various modifications can be made. It may be.

Claims (7)

In the head mounted display device,
Outputs a screen disposed at a position corresponding to both eyes, a first area of the screen outputs a keyboard image having a plurality of virtual keys, and a second area of the screen outputs information received from the keyboard image A display unit configured to output a cursor image on one of the first areas of the screen;
A detector for detecting a movement of the head; And
And a controller for moving the cursor image to correspond to the movement of the head in the screen in response to the movement of the head being detected.
The control unit,
In response to the movement of the head not detected for a preset time, the size of the cursor image is changed to increase gradually,
And when the cursor image corresponds to a preset size, execute the function corresponding to the area where the cursor image is located. The method of claim 1,
And the degree of movement of the cursor image is determined based on at least one of a movement speed, a distance, and a direction of the head. The method of claim 1,
The cursor image stops movement when the head is not moving,
The control unit,
And when the movement of the head corresponds to a preset pattern, executing the function matching the pattern. The method of claim 3, wherein the control unit
And at least one of a color and a shape of the cursor image to notify the user that the function can be executed in response to the movement of the head not detected for a preset time. The method of claim 3,
The preset time includes a first time and a second time,
The control unit
When the first time elapses with the movement of the head stopped, the cursor image is changed,
And executing the function when the second time elapses in the state where the cursor image is changed. The method of claim 5,
The control unit
After the first time has elapsed and the cursor image is changed, when the movement of the head is detected before the second time has elapsed, the head-mounted display device may restore the appearance of the cursor image to before the change. . Outputting a screen disposed at a position corresponding to both eyes on the display unit, wherein the screen outputs a keyboard image having a plurality of virtual keys and information received from the keyboard image;
Detecting movement of the head;
Moving a cursor image to correspond to the movement of the head in the screen based on the movement of the head being detected; And
Changing the cursor image to gradually increase in response to a state in which the movement of the head stops is not detected for a preset time; And
And executing a function corresponding to an area in which the cursor image is located when the cursor image corresponds to a preset size.

Images