KR20140146889A - Electric device and operation method thereof - Google Patents

Abstract

Disclosed are an electronic device and a method of operating the same. According to embodiments of the present invention, the electronic device includes: a wireless communication unit to access a head-mount display which displays an augmented reality image; a controller to sequentially acquire surrounding images using a camera, extract a plurality of feature points from the acquired image, detect a moving path of the extracted feature points according to a result acquired by matching the same feature points in the acquired image, and create a three dimensional (3D) map. In addition, the controller can control the wireless communication unit to transmit an augmented image, which is acquired by synthesizing virtual objects allowing a user input in match with the position and the direction of the camera based on the created map, to the accessed head-mounted display.

Description

ELECTRONIC DEVICE AND OPERATION METHOD THEREOF FIELD OF THE INVENTION [0001]

BACKGROUND OF THE INVENTION 1. Field of the Invention [0001] The present invention relates to an electronic device and an operation method of the electronic device, and more particularly to an electronic device connectable with a head mounted display capable of displaying an augmented reality image and an operation method thereof.

The electronic device may be a mobile terminal such as a mobile phone, a smart phone, a laptop computer, a digital broadcasting terminal, a personal digital assistant (PDA), a portable multimedia player (PMP), a navigator, And a fixed terminal such as a computer.

As the functions of the electronic devices are diversified, they are implemented in the form of a multimedia device having a combination of functions such as photographing and video shooting, music and video file playback, game and broadcast reception . Further, for functional support and enhancement of the electronic device, it may be considered to improve the structural and software parts of the electronic device.

Due to such improvements, recently, augmented reality (AR) technology has been applied to the electronic devices, and they have been used for various purposes. Such augmented reality is a field of virtual reality, which means a computer graphics technique that combines virtual objects or information into a real environment to make it look like objects in the original environment. Such augmented reality screen can be displayed using a wearable device connected to the electronic device by wire or wireless, for example, a head mounted display.

On the other hand, such a head-mounted display may be connected to the above-described electronic device and function as an input / output device of the electronic device. For this purpose, the actual projected input / output device can be realized by irradiating light using the laser project placed on the head mount display.

However, this method can not protect the personal privacy of the user because the user can see the input / output device and the input / output contents through the input / output device to the users other than the user. In addition, the amplitude of the motion is increased according to the distance of the light irradiation, and the input / output device is displayed at an undesired position or the position of the input / output device is suddenly changed.

Therefore, embodiments of the present invention provide an electronic device capable of generating a three-dimensional map of a peripheral image, synthesizing a virtual object capable of being input by a user into a generated map, and displaying the acquired augmented reality image on a connected head- And an operation method of the electronic device.

To this end, an electronic device according to an embodiment of the present invention includes: a wireless communication unit connected to a head-mounted display configured to display an augmented reality image; Sequentially acquiring images of surrounding objects using a camera, extracting a plurality of feature points from the acquired image, and determining a movement path of the extracted feature points according to a result obtained by matching identical feature points in a plurality of sequentially acquired images And a control unit for generating a three-dimensional map based on the generated map, wherein the control unit controls the display unit to display the augmented reality image obtained by synthesizing a virtual object capable of user input according to the position and direction of the camera, And controls the wireless communication unit to transmit the information to the display.

In one embodiment, if the camera is fixed to the head-mounted display, the control unit tracks the movement path of the head-mounted display to capture an image of the surroundings, And controls to move and display the real image.

In one embodiment, the camera is characterized by being a single stereo camera provided in front of an amount of a user wearing the head mount display.

In one embodiment, the control unit updates the map by tracking the changed position when at least one of the positions of the minutiae points of the camera and the map is changed, and updates the virtual object on the basis of the updated map, On the map.

In one embodiment, when a user input is detected on the virtual object, the control unit tracks the position of the detected user input to execute a corresponding function, and displays the execution result in one area of the head mounted display And a control unit.

In one embodiment, the user input is performed by using a pointing device or a gesture of pointing a specific area or a specific point displayed on the virtual object using a user's finger or moving in a preset direction after pointing.

In one embodiment, the control unit detects the finger of the user from the image obtained using the camera, and tracks the position of the detected finger to determine a difference in distance between the virtual object spaced from the camera and the detected finger And recognizing whether or not to point to the specific region or the specific point based on a direction in which the detected finger is pointing.

In one embodiment, the virtual object is an input device including a plurality of keys that can be input by a user, and when the pointing coordinate pointed to the specific area or a specific point that is pointed is a specific key in the virtual object, And an output area capable of outputting text mapped to the key is displayed on the head mount display together with the virtual object.

In one embodiment, the virtual object is an input / output device including a virtual touch pad, and the control unit generates pointing coordinates for the specific area or specific point that is pointed, And combines the first augmented reality image in the augmented reality and displays it on the head-mounted display.

In one embodiment, the virtual object is an input / output device including a virtual touch pad. When the dragged input has a starting point of the pointed specific area or a specific point, The second AR object image is synthesized in the AR object in the virtual object correspondingly and displayed on the head mount display.

In one embodiment, the wireless communication unit is connected to an external device to transmit and receive data information, and the control unit transmits the three-dimensional data information of the generated map and the data information of the virtual object to the connected And controls the wireless communication unit to transmit the image data to an external device.

In one embodiment, the control unit determines whether to allow a user interaction with a virtual object displayed on the connected external device based on the transmitted data information based on a predetermined priority, and transmits the determined result And controls the wireless communication unit to transmit to the connected external device.

According to another aspect of the present invention, there is provided a method of operating an electronic device, the method comprising: connecting to a head-mounted display capable of displaying an augmented reality image; Sequentially acquiring images of a surrounding area using a camera; Extracting a plurality of feature points from the acquired image; Matching the same feature points in the sequentially acquired images and generating a three-dimensional map by grasping a movement path of the extracted feature points based on the matching; And transmitting the augmented reality image obtained by synthesizing the virtual object including the virtual touch pad to the connected head-mounted display according to the position and direction of the camera based on the generated map.

In one embodiment, updating the map in response to a change in at least one of a position of a feature point of the camera and the map; And synthesizing the virtual object on the updated map based on the updated map.

In one embodiment, if a user input is detected on the virtual object, tracking the location of the detected user input to perform a corresponding function; And controlling the display unit to display the execution result in one area of the head mount display.

According to the method of operating the electronic device and the electronic device according to the embodiment of the present invention, an augmented reality image including a virtual object capable of user input and a screen change corresponding to a user input using the augmented reality image are displayed on a head- So that only the wearer wearing the head mount display can view the information, thereby protecting the personal privacy and information of the individual.

In addition, according to the electronic device and the operation method of the electronic device according to the embodiment of the present invention, it is possible to provide a three-dimensional map generated by using a plurality of minutiae points extracted from arbitrary scenes, objects, By displaying the augmented reality image obtained by synthesizing the virtual objects capable of being input by tracking the position of the camera, the input / output device can be displayed in a desired position and direction to provide user convenience.

Further, according to the electronic device and the operation method of the electronic device according to the embodiment of the present invention, the input and output using the virtual object are displayed together on the head mount display connected to the electronic device, thereby easily controlling the operation of the electronic device .

1 is a conceptual diagram showing an example of a head-mounted display connected to an electronic device and an electronic device according to an embodiment of the present invention.
2 is a block diagram showing a detailed configuration of an electronic device according to an embodiment of the present invention.
3 is a view illustrating an example of a head mount display connectable to an electronic device according to an embodiment of the present invention.
4 and 5 are flowcharts for explaining an operation method of an electronic device according to an embodiment of the present invention.
Figs. 6A to 6G are conceptual diagrams for explaining the operation method of Figs. 4 and 5. Fig.
7 is a view showing a virtual object including a plurality of keys which can be input by a user, on a head mount display according to an embodiment of the present invention.
8A and 8B are views showing a virtual object including a virtual touch pad displayed on a head mount display according to an embodiment of the present invention.

Hereinafter, an electronic device and an operation method of an electronic device according to an embodiment of the present invention will be described in detail with reference to the accompanying drawings.

Hereinafter, embodiments of the present invention will be described with reference to the accompanying drawings so that those skilled in the art can easily carry out the technical idea of the present invention. The present invention may, however, be embodied in many different forms and should not be construed as limited to the embodiments set forth herein. In order to clearly explain the present invention, parts not related to the description are omitted, and like parts are denoted by similar reference numerals throughout the specification.

In addition, the electronic device described in this specification includes a mobile phone, a smart phone, a laptop computer, a digital broadcasting terminal, a PDA (Personal Digital Assistants), a PMP (Portable Multimedia Player), a navigator, A TV, a desktop computer, and the like.

1 is a conceptual diagram showing an example of a head-mounted display connected to an electronic device and an electronic device according to an embodiment of the present invention.

As shown in FIG. 1, an electronic device 100 according to an embodiment of the present invention can be communicably connected to a head mounted display 200 in a wired or wireless manner.

The electronic device 100 communicatively coupled to the head mount display 200 may be a mobile phone, a smart phone, a laptop computer, a digital broadcasting terminal, a personal digital assistant (PDA) ), A portable multimedia player (PMP), a navigator, a digital TV, a desktop computer, and the like.

In addition, the head mount display 200 may be various wearable devices such as portable glasses, rings, necklaces, clothes, and the like. In the case where it is not necessary to attach the electronic devices to the body according to circumstances, Or may be implemented in any one of them.

For example, contents, to be displayed on the head-mounted display 200. The display unit 200 may display the image information on the head-mounted display 200, for example. In addition, the head mount display 200 may transmit data collected through the provided sensors to the electronic device 100. In addition, the electronic device 100 may share data received from other connected external devices with the head mounted display 200.

The electronic device 100 may transmit a specific command input from a user via an interface to a connected head mount display 200 to control an associated operation. In addition, the electronic device 100 may perform related operations based on data received from the head mounted display 200, or may output predetermined screen information or sounds.

The electronic device 100 and the head mount display 200 may be connected to each other by a wireless communication device such as a wireless communication standard such as Bluetooth or the like, a wireless internet standard such as Wi-Fi, an external device interface And the like.

When the electronic device 100 and the head mount display 200 are connected as described above, the head mount display 200 can display an augmented reality screen according to a control signal transmitted from the electronic device 100, 100 as shown in FIG.

The head mount display 200 transmits image data acquired through the camera to the connected electronic device 10 so that the electronic device 100 can recognize the user's gaze tracking and gesture input . On the other hand, it is also possible to control the image data acquired through the camera provided in the electronic device 100 to be displayed on the head mount display 200. In addition, the head mount display 200 may further display status information of the electronic device 100. [

FIG. 2 is a block diagram illustrating a detailed configuration of a mobile terminal as an example of an electronic device 100 according to an embodiment of the present invention.

2, the electronic device 100 includes a wireless communication unit 110, an A / V input unit 120, a user input unit 130, a sensing unit 140, an output unit 150, A memory 160, an interface unit 170, a control unit 180, and a power supply unit 190. The components shown in Fig. 2 are not essential, so that an electronic device having more or fewer components can be implemented.

Hereinafter, the components 110 to 190 of the electronic device 100 will be described in turn.

The wireless communication unit 110 may include one or more modules that enable wireless communication between the electronic device 100 and the wireless communication system or between the electronic device 100 and the network in which the electronic device 100 is located. For example, the wireless communication unit 110 may include a broadcast receiving module 111, a mobile communication module 112, a wireless Internet module 113, a short distance communication module 114, and a location information module 115.

The broadcast receiving module 111 receives broadcast signals and broadcast related information from an external broadcast management server through a broadcast channel. Here, the broadcast-related information means information related to a broadcast channel, a broadcast program, or a broadcast service provider. The broadcast-related information can also be provided through a mobile communication network. In this case, the broadcast-related information may be received by the mobile communication module 112. The broadcast signal and the broadcast-related information received through the broadcast receiving module 111 may be stored in the memory 160.

The mobile communication module 112 transmits and receives radio signals to at least one of a base station, an external terminal, and a server on a mobile communication network. Such wireless signals may include various types of data depending on a voice call signal, a video call signal, a text message, or a multimedia message transmission / reception.

The wireless Internet module 113 is a module for wireless Internet access, and may be embedded in or enclosed in the electronic device 100. WLAN (Wi-Fi), Wibro (Wireless broadband), Wimax (World Interoperability for Microwave Access), HSDPA (High Speed Downlink Packet Access) and the like can be used as wireless Internet technologies.

The short-range communication module 114 refers to a module for short-range communication. As a short range communication technology, Bluetooth, Radio Frequency Identification (RFID), Infrared Data Association (IrDA), Ultra Wideband (UWB), ZigBee, and the like can be used.

The position information module 115 is a module for obtaining the position of the electronic device 100, and a representative example thereof is a Global Position System (GPS) module.

2, an A / V (Audio / Video) input unit 120 receives an audio signal and a video signal and tracks a user using the input signal. The audio / video input unit 120 includes a camera 121, A microphone 122, an infrared sensor (not shown), and the like. The camera 121 processes image frames such as still images and moving images obtained by the image sensor in the video communication mode or the photographing mode. The image frame processed by the camera 121 can be displayed on the display unit 151. [ The image frame may be stored in the memory 160 or transmitted to the outside through the wireless communication unit 110. [ More than two cameras 121 may be provided depending on the use environment.

The microphone 122 processes the sound signal input from the outside in the communication mode, the recording mode, the voice selection mode, and the like as electrical voice data. The voice data processed by the microphone 122 in the communication mode can be converted into a form that can be transmitted to the mobile communication base station through the mobile communication module 112 and output. The microphone 122 may be implemented with various noise reduction algorithms for eliminating noise generated when an external sound signal is input.

The user input unit 130 generates input data for controlling the operation of the electronic device 100 by a user. The user input unit 130 may include a key pad, a dome switch, a touch pad (static pressure and static electricity), a jog wheel, a jog switch, and the like.

The sensing unit 140 senses the current state of the electronic device 100 such as the presence of a user, the open / close state of the electronic device 100, the position, orientation, acceleration, deceleration, And generates a sensing signal. For example, when the electronic device 100 is in the form of a slide phone, the sensing unit 140 can detect whether the slide phone is opened or closed. The sensing unit 140 may sense whether the power supply unit 190 is powered on, whether the interface unit 170 is connected to an external device, and the like.

The sensing unit 140 may include a proximity sensor 141. The sensing unit 140 may include a touch sensor (not shown) for sensing a touch operation with respect to the display unit 151.

The touch sensor may have the form of a touch film, a touch sheet, a touch pad, or the like. The touch sensor may be configured to convert a pressure applied to a specific portion of the display portion 151 or a change in capacitance generated at a specific portion of the display portion 151 into an electrical input signal. The touch sensor can be configured to detect not only the position and area to be touched but also the touch pressure.

When the touch sensor and the display unit 151 have a mutual layer structure, the display unit 151 can be used as an input device in addition to the output device. The display unit 151 may be referred to as a 'touch screen'.

If there is a touch input via the touch screen, corresponding signals are sent to a touch controller (not shown). The touch controller processes the signals transmitted from the touch sensor, and then transmits data corresponding to the processed signals to the controller 180. Thus, the control unit 180 can know which area of the display unit 151 is touched.

In the case where the touch screen is electrostatic, it can be configured to detect the proximity of the sensing object by a change of the electric field along the proximity of the sensing target. Such a touch screen may be classified as proximity sensor 141. [

The proximity sensor 141 refers to a sensor that detects the presence or absence of an object to be sensed without mechanical contact using an electromagnetic force or infrared rays. The proximity sensor 141 has a longer life than the contact type sensor and its utilization is also high. Examples of the proximity sensor 141 include a transmission type photoelectric sensor, a direct reflection type photoelectric sensor, a mirror reflection type photoelectric sensor, a high frequency oscillation type proximity sensor, a capacitive proximity sensor, a magnetic proximity sensor, and an infrared proximity sensor.

Hereinafter, for convenience of explanation, a proximity action is referred to as " proximity touch " while an object to be sensed does not touch the touch screen, and an action of touching the sensing object on the touch screen is called & touch ".

The proximity sensor 141 detects the presence or absence of a proximity touch and a proximity touch pattern (for example, a proximity touch distance, a proximity touch direction, a proximity touch speed, a proximity touch time, a proximity touch position, The information corresponding to the presence / absence of proximity touch and the proximity touch pattern can be output to the touch screen.

The output unit 150 generates an output related to visual, auditory, tactile, and the like. The output unit 150 may include a display unit 151, an audio output module 153, an alarm unit 154, and a haptic module 155.

The display unit 151 displays (outputs) information to be processed in the electronic device 100. For example, when the electronic device 100 operates in the call mode, the display unit 151 displays a UI (User Interface) or a GUI (Graphic User Interface) related to the call. When the electronic device 100 operates in the video communication mode or the photographing mode, the display unit 151 displays the photographed image, the received image, the UI, the GUI, and the like.

The display unit 151 may be a liquid crystal display (LCD), a thin film transistor liquid crystal display (TFT-LCD), an organic light emitting diode (OLED), a flexible display display, a 3D display, and an e-ink display.

At least one display (or display element) included in the display unit 151 may be configured to be transparent or light transmissive so that the display can see the outside through it. This can be referred to as a transparent display. A typical example of such a transparent display is TOLED (Transparent OLED). The rear structure of the display unit 151 may also be of a light transmission type. With this structure, the user can see an object located behind the terminal body through the area occupied by the display unit 151 in the terminal body.

The display unit 151 may exist in two or more depending on the implementation of the electronic device 100. For example, in the electronic device 100, a plurality of display portions may be spaced apart from one another or may be located integrally with each other, or may be located on different surfaces.

Also, the display unit 151 may be configured as a stereoscopic display unit 152 for displaying a stereoscopic image.

Here, a stereoscopic image represents a 3-dimensional stereoscopic image, and a 3-dimensional stereoscopic image represents a progressive depth and reality in which objects are located on a monitor or a screen, It is an image that makes you feel the same as the space. 3D stereoscopic images are implemented using binocular disparity. The binocular parallax means a parallax caused by the position of two eyes away from each other. When two eyes see two different images and the images are transmitted to the brain through the retina and fused, the depth and real feeling of the stereoscopic image can be felt .

The stereoscopic display unit 152 may be applied to a three-dimensional display system such as a stereoscopic system (glasses system), an autostereoscopic system (no-glasses system), and a projection system (holographic system). The stereoscopic method, which is widely used in home television receivers, includes a Wheatstone stereoscopic method.

Examples of the autostereoscopic method include a parallax barrier method, a lenticular method, an integral imaging method, and a switchable lens method. The projection method includes a reflection type holographic method and a transmission type holographic method.

Generally, 3D stereoscopic images consist of left image (left eye image) and right image (right eye image). A top-down method of arranging a left image and a right image in one frame according to a method in which a left image and a right image are combined into a three-dimensional stereoscopic image, A checker board system in which pieces of a left image and a right image are arranged in a tile form, a left-to-right (right-side) Or an interlaced method in which rows are alternately arranged, and a time sequential (frame-by-frame) method in which right and left images are alternately displayed in time.

In addition, the 3D thumbnail image can generate a left image thumbnail and a right image thumbnail from the left image and right image of the original image frame, respectively, and combine them to generate one 3D thumbnail image. In general, a thumbnail means a reduced image or a reduced still image. The left image thumbnail and the right image thumbnail generated in this way are displayed on the screen with a difference of the left and right distance by the depth corresponding to the parallax between the left image and the right image, thereby exhibiting a stereoscopic spatial feeling.

The left and right images necessary for realizing the three-dimensional stereoscopic image can be displayed on the stereoscopic display unit 152 by a stereoscopic processing unit (not shown). The stereoscopic processing unit receives a 3D image and extracts a left image and a right image from the 3D image, or receives a 2D image and converts it into a left image and a right image.

In the case where the three-dimensional display unit 152 and the touch sensor have a mutual layer structure (hereinafter referred to as a 'three-dimensional touch screen') or a three-dimensional sensor that detects the touch operation and the stereoscopic display unit 152 are combined with each other The stereoscopic display unit 152 may also be used as a three-dimensional input device.

The sensing unit 140 may include a proximity sensor 141, a three-dimensional touch sensing unit 142, an ultrasonic sensing unit 143, and a camera sensing unit 144 as an example of the three-dimensional sensor.

The proximity sensor 141 measures the distance between the sensing surface (for example, a user's finger or a stylus pen) to which the touch is applied without mechanical contact using the force of the electromagnetic field or infrared rays. The terminal recognizes which part of the stereoscopic image has been touched using the distance. In particular, when the touch screen is of the electrostatic type, the proximity of the sensing object is detected by a change of the electric field according to the proximity of the sensing object, and the touch on the three-dimensional is recognized using the proximity.

The stereoscopic touch sensing unit 142 senses the strength or duration of a touch applied to the touch screen. For example, the three-dimensional touch sensing unit 142 senses a pressure to apply a touch, and when the pressing force is strong, recognizes the touch as a touch to an object located further away from the touch screen toward the inside of the terminal.

The ultrasonic sensing unit 143 is configured to recognize the position information of the sensing target using ultrasonic waves.

The ultrasound sensing unit 143 may include, for example, an optical sensor and a plurality of ultrasound sensors. The light sensor is configured to sense light, and the ultrasonic sensor is configured to sense ultrasonic waves. Since light is much faster than ultrasonic waves, the time it takes for light to reach the optical sensor is much faster than the time it takes for the ultrasonic waves to reach the ultrasonic sensor. Therefore, it is possible to calculate the position of the wave generating source using the time difference with the time when the ultrasonic wave reaches the reference signal.

The camera sensing unit 144 includes at least one of a camera 121, a photo sensor, and a laser sensor.

For example, the camera 121 and the laser sensor are combined with each other to sense a touch of a sensing target with respect to a three-dimensional stereoscopic image. When the distance information detected by the laser sensor is added to the two-dimensional image photographed by the camera, three-dimensional information can be obtained.

As another example, a photosensor may be stacked on a display element. The photosensor is configured to scan the movement of the object proximate to the touch screen. More specifically, the photosensor mounts photo diodes and TRs (Transistors) in a row / column and scans the contents loaded on the photosensor using an electrical signal that varies according to the amount of light applied to the photo diode. That is, the photo sensor performs coordinate calculation of the object to be sensed according to the amount of light change, thereby acquiring position information of the object to be sensed.

The sound output module 153 may output audio data received from the wireless communication unit 110 or stored in the memory 160 in a call signal reception mode, a call mode or a recording mode, a voice selection mode, a broadcast reception mode, The sound output module 153 also outputs sound signals associated with functions (e.g., call signal reception tones, message reception tones, etc.) performed in the electronic device 100. The sound output module 153 may include a receiver, a speaker, a buzzer, and the like.

The alarm unit 154 outputs a signal for notifying the occurrence of an event of the electronic device 100. [ Examples of events generated in the electronic device 100 include call signal reception, message reception, key signal input, touch input, and the like. The alarm unit 154 may output a signal for informing occurrence of an event in a form other than the video signal or the audio signal, for example, vibration. The display unit 151 and the audio output module 153 may be classified as a part of the alarm unit 154 because the video signal or the audio signal can be output through the display unit 151 or the audio output module 153. [

The haptic module 155 generates various tactile effects that the user can feel. A typical example of the haptic effect generated by the haptic module 155 is vibration. The intensity, pattern, and the like of the vibration generated by the tactile module 155 are controllable. For example, different vibrations may be synthesized and output or sequentially output.

In addition to the vibration, the haptic module 155 may be configured to perform various functions such as a pin arrangement vertically moving with respect to the contact skin surface, a spraying force or a suction force of the air through the injection port or the suction port, a touch on the skin surface, contact with an electrode, Various effects can be generated such as the effect of heat generation and the effect of reproducing the cold sensation using the heat absorbing or heatable element.

The haptic module 155 can be configured to not only transmit the tactile effect through the direct contact but also to allow the user to feel the tactile effect through the muscular sense of the finger or arm. The haptic module 154 may include two or more haptic modules 154 according to the configuration of the electronic device 100.

The memory 160 may store a program for the operation of the controller 180 and temporarily store input and output data (e.g., a phone book, a message, a still image, a moving picture, etc.). The memory 160 may store data related to vibrations and sounds of various patterns that are output upon touch input on the touch screen.

The memory 160 may be a flash memory, a hard disk, a multimedia card micro type, a card type memory (e.g., SD or XD memory), a random access memory (RAM), static random access memory (SRAM), read-only memory (ROM), electrically erasable programmable read-only memory (EEPROM), programmable read-only memory (PROM) And may include at least one storage medium. The electronic device 100 may operate in association with a web storage that performs storage functions of the memory 160 on the Internet.

The interface unit 170 serves as a path for communication with all external devices connected to the electronic device 100. The interface unit 170 receives data from an external device or supplies power to each component in the electronic device 100 or allows data in the electronic device 100 to be transmitted to an external device. For example, the interface unit 170 may include a wired / wireless headset port, an external charger port, a wired / wireless data port, a memory card port, a port for connecting a device having an identification module, Input / output ports, video I / O ports, earphone ports, and the like.

The identification module is a chip for storing various information for authenticating the usage right of the electronic device 100 and includes a user identification module (UIM), a subscriber identity module (SIM), a general user authentication module Universal Subscriber Identity Module (USIM)). A device equipped with an identification module (hereinafter referred to as "identification device") can be manufactured in a smart card format. Accordingly, the identification device can be connected to the terminal 100 through the port.

The interface unit 170 is a path through which electric power from the cradle is supplied to the electronic device 100 when the electronic device 100 is connected to an external cradle or various command signals input from the cradle by the user, (Not shown). The various command signals or power input from the cradle may also act as a signal to recognize that the electronic device 100 is correctly mounted in the cradle.

A controller 180 controls the overall operation of the electronic device 100. For example, control and processing related to voice communication, data communication, video communication, and the like. The control unit 180 may include a multimedia module 181 for multimedia playback. The multimedia module 181 may be implemented in the control unit 180 or may be implemented separately from the control unit 180. [

In addition, the control unit 180 may perform a pattern selection process for selecting handwriting input and drawing input on the touch screen as characters and images, respectively. In addition, if the state of the mobile terminal meets a set condition, the controller 180 can execute a lock state for restricting input of a control command of the user to the applications. Also, the controller 180 may control the lock screen displayed in the locked state based on the touch input sensed through the display unit 151 in the locked state.

The power supply unit 190 receives external power and internal power under the control of the controller 180 and supplies power necessary for operation of the respective components.

The various embodiments described herein may be implemented in a recording medium readable by a computer or similar device using software, hardware, or a combination thereof.

According to a hardware implementation, the embodiments described herein may be applied to various types of application specific integrated circuits (ASICs), digital signal processors (DSPs), digital signal processing devices (DSPDs), programmable logic devices (PLDs), field programmable gate arrays Microprocessors, microprocessors, and other electronic units for performing other functions, as will be appreciated by those skilled in the art. In some cases, the embodiments described herein may be implemented by the controller 180 itself.

According to a software implementation, embodiments such as the procedures and functions described herein may be implemented with separate software modules. Each of the software modules may perform one or more of the functions and operations described herein. Software code may be implemented in a software application written in a suitable programming language. Such software codes may be stored in the memory 160 and executed by the control unit 180. [

Hereinafter, a method of processing user input to the electronic device 100 will be described.

The user input unit 130 is operated to receive a command for controlling the operation of the electronic device 100, and may include a plurality of operation units. The operating units may also be referred to as manipulating portions and may be employed in any manner as long as they are operated in a tactile manner by the user's tactile sense.

Various kinds of time information can be displayed on the display unit 151. [ Such visual information can be displayed in the form of letters, numbers, symbols, graphics, icons, and the like, and can be formed as a three-dimensional stereoscopic image. At least one of a character, a number, a symbol, a graphic, and an icon may be displayed in a predetermined arrangement for inputting time information, thereby being implemented as a keypad. Such a keypad may be referred to as a so-called " soft key ".

The display unit 151 may operate as an entire area or may be divided into a plurality of areas and operated. In the latter case, the plurality of areas can be configured to operate in association with one another. For example, an output window and an input window may be displayed on the upper and lower portions of the display unit 151, respectively. The output window and the input window are areas allocated for outputting or inputting information, respectively. In the input window, a soft key with a number for inputting a telephone number may be output. When the soft key is touched, the number corresponding to the touched soft key is displayed in the output window. When the operating unit is operated, a call connection to the telephone number displayed in the output window may be attempted, or the text displayed in the output window may be entered into the application.

The display unit 151 or the touch pad may be configured to detect touch scrolling. The user can move the cursor or the pointer positioned on the displayed object, for example, the icon, on the display unit 151 by scrolling the display unit 151 or the touch pad. Further, when the finger is moved on the display unit 151 or the touch pad, the path along which the finger moves may be visually displayed on the display unit 151. [ This will be useful for editing the image displayed on the display unit 151. [

One function of the electronic device 100 may be executed corresponding to the case where the display unit 151 and the touch pad are touched together within a predetermined time range. In the case of being touched together, there may be a case where the user clamps the main body of the electronic device 100 using the thumb and index finger. At this time, one function of the electronic device 100 to be executed may be activation or deactivation, for example, with respect to the display unit 151 or the touch pad.

3 is a view illustrating an example of a head mount display connectable to an electronic device according to an embodiment of the present invention.

The head mounted display (HMD) 200 connected to the electronic device 100 including the above-described components may be a wearable device of various types such as a portable eyepiece type, a ring type, a necklace type, And may be implemented in any one of the examples of the above-described electronic device if it is not necessary to mount it on the body according to the environment.

Here, for the purpose of explanation, a head mount display of a perspective type (not limited to this) capable of projecting external light will be described as an example.

As shown in Fig. 3, the head-mounted display 200 in the form of a pair of glasses includes a camera 201, image projectors 205 and 207, and a head mount controller 200 for overall operation control of the head- (211).

The camera 201 may be disposed on the front or rear surface of the head mount display 200 to capture images of the outside or track the user's gaze. Eye-tracking refers to a technique of tracking a user's gaze by recognizing a user's eyes by analyzing an image input through a camera.

The image data obtained through the camera 201 can be transmitted in real time to the connected electronic device 100. [ The electronic device 100 may then analyze the image data received from the head mount display 200 to recognize predetermined user inputs. User input refers to gesture input using a user's hand, motion gesture, hand gesture, or other device.

The image output devices 205 and 207 can output images according to the control of the electronic device 100. [ Particularly, according to the embodiment of the present invention, the head mount display 200 can display an augmented reality image under the control of the electronic device 100. [

The electronic device 100 comprising the above described configurations is connected to a head mounted display 200 (e.g., a head mounted display in the form of a glasses) configured to display an augmented reality image. The image acquiring unit of the electronic device 100 connected to the head mount display 200 can acquire an image of the camera periphery by using the camera 121 provided in the electronic device 100 or the camera 201 provided in the head mount display 200. [ Can be obtained sequentially.

When an image around the camera is acquired in this manner, the feature point extracting unit of the electronic device 100 can extract a plurality of feature points from the acquired image. Then, the control unit 180 of the electronic device 100 matches the same feature points in a plurality of sequentially acquired images, that is, a frame of a previous image and a frame of a subsequent image.

The control unit 180 can generate a three-dimensional map of the camera environment by grasping the movement path of the extracted minutiae based on the matching result. At this time, a plurality of minutiae points may be extracted from any space or texture photographed by the camera rather than registered markers or objects.

When the 3D map is generated in this way, the electronic device 100 can acquire an augmented reality image obtained by synthesizing a virtual object capable of user input according to the position and direction of the camera based on the generated map. Here, the virtual object that can be input by a user is a predetermined virtual image or text including a predetermined input area and / or an output area, or a time information of the content output to the electronic device 100, ). ≪ / RTI > The acquired augmented reality image is transmitted to the head mounted display 200 connected through the wireless communication unit 110 and displayed.

As described above, according to the embodiments of the present invention, since the virtual object capable of user input can be seen only by the wearer wearing the head mount display 200, privacy and information of the individual can be protected. In addition, a three-dimensional map of the surrounding environment is generated based on feature points extracted from arbitrary space, and an augmented reality image synthesized with a user-inputable virtual object is displayed by tracking the position of the camera, Provides convenience for displaying virtual objects.

Hereinafter, an operation method of an electronic device according to an embodiment of the present invention will be described with reference to Figs. 4 and 6A to 6G.

In this regard, FIG. 4 is a flowchart of an operation method of an electronic device according to an embodiment of the present invention, and FIGS. 6A to 6G are conceptual diagrams for explaining the operation method of FIG.

Referring to FIG. 4, an electronic device 100 (FIG. 2) according to an embodiment of the present invention includes a head mounted display 200 (FIG. 2) configured to display an augmented reality image through a wireless communication unit 110 3 (S410).

In this way, in a state connected to the head mount display 200, the electronic device 100 drives the camera to sequentially acquire images of the camera periphery (S420). Here, the camera is either a camera 201 provided on the head mount display 200 or a camera 121 provided on the electronic device 100. The camera 201 or 121 is driven in accordance with the control signal transmitted from the control unit 180 of the electronic device 100.

Specifically, if the camera is fixed to the head mount display 200, the controller 180 may track the movement path of the head mount display 200 to take an image of the surroundings of the camera. That is, the controller 180 can calculate the position of the camera 201 from the motion information of the head mount display 200, and acquires the photographed image based on the calculated position of the camera 201.

For example, referring to FIG. 6A, a camera 201 fixed to a head mount display 200 may be used to photograph a surrounding environment including arbitrary textures (e.g., a user's hand, a keyboard of a PC, etc.) Can be obtained.

On the other hand, the camera may be a single stereo camera disposed in front of a user's face on which the head-mounted display 200 is mounted.

For example, if the camera 201 is disposed on the front surface of the head mount display 200, the image of the camera is acquired. If the camera 201 is disposed on the rear surface of the head mount display 200, You can track your gaze. In the latter case, the camera 121 provided in the electronic device 100 can photograph the periphery of the camera and acquire an image.

In addition, the control unit 180 may extract a plurality of feature points from the obtained image of the camera periphery (S430). In detail, the control unit 180 processes the obtained peripheral image to extract a plurality of feature points from the camera peripheral image. Here, the image processing converts the input image into a frequency domain. The controller 180 can determine a texture or a space (e.g., a human hand, a wire, a corner, and the like)

For example, referring to FIG. 6B, the electronic device 100 may extract a plurality of feature points from a user's hand contained in a plurality of images obtained or received from the head- The extracted minutiae are shown in a circular form as shown. In FIGS. 6B and 6D, the large circle represents a characteristic point where the variation width of the frequency is large, and the small circle represents a small characteristic point that is relatively the frequency variation width. 6D, the electronic device 100 may extract a plurality of feature points from any space or texture included in a plurality of images obtained or received from the head-mounted display 200. In addition,

When a plurality of feature points are extracted from the sequentially acquired images, the controller 180 matches the same feature points in the plurality of images, and recognizes the movement path of the extracted feature points based on the matching, A three-dimensional map can be generated (S440).

Specifically, the controller 180 extracts the feature points from the frames of the surrounding image photographed at time t, extracts the feature points from the frames of the surrounding images sequentially captured at the adjacent time t + 1, Match. From the matching, it is possible to recognize the positional change of the same feature point, thereby acquiring the three-dimensional information of the minutiae of the actual surrounding environment, and construct a three-dimensional map of the surrounding environment. At this time, if the number of matching feature points is insufficient for forming the map, a new image can be obtained through the camera and the process of extracting feature points from the new image can be repeated.

In addition, the process of generating such a map can be continuously executed in the background of the electronic device 100 while the electronic device 100 and the head-mounted display 200 are connected.

When the map for the camera peripheral image is generated as described above, the controller 180 displays the augmented reality image obtained by synthesizing the virtual object capable of user input according to the position and direction of the camera on the basis of the generated map, (S450).

Then, the head mount display 200 can display an augmented reality image synthesized with user-input virtual objects in the field of view of a wearer wearing the head mount display 200.

For example, referring to FIG. 6C, when a map generated based on minutiae points extracted from a user's hand contained in the acquired image is registered, an augmented reality image, Can be displayed on the display (200). In this case, since the augmented reality image is displayed by tracking the movement path of the user's hand, the user feels as though the virtual object lies on the upper part of the actual user. In addition, for example, referring to FIG. 6E, when a map generated based on minutiae extracted from an arbitrary texture included in the acquired image is registered, a virtual object may be displayed in a specific space and in a specific direction .

Meanwhile, if the camera is fixed to the head mount display 200, the controller 180 may control to move and display the augmented reality image along the movement path of the head mount display 200. In addition, when a plurality of cameras are disposed on the head mount display 200 and one of them monitors the user's gaze, the augmented reality image may be displayed along the user's gaze.

In the above description, a method of extracting feature points from an arbitrary space using a camera to generate a three-dimensional map of the surrounding environment, and outputting an augmented reality image synthesized with user inputable virtual objects to a connected head mount display I looked at it.

Hereinafter, a method for updating the generated three-dimensional map and outputting a user input for the displayed virtual object will be described with reference to FIGS. 5 and 6A to 6G.

The electronic device 100 according to the embodiment of the present invention can be connected to the head mount display 200 capable of displaying an augmented reality image through the wireless communication unit 110. [ In this way, in a state connected to the head mount display 200, the electronic device 100 drives the camera to sequentially acquire images of the surroundings. The control unit 180 may extract a plurality of feature points from the acquired image. In this manner, the controller 180 matches the same feature points in the sequentially acquired images, and obtains the three-dimensional map of the camera peripheral image by grasping the movement path of the extracted feature points based on the matching . When the map for the camera peripheral image is generated as described above, the controller 180 displays the augmented reality image obtained by synthesizing the virtual object capable of user input according to the position and direction of the camera on the basis of the generated map, Lt; / RTI >

Meanwhile, the controller 180 may detect a change in position of the cameras 201 and 121 or a change in the position of the minutiae of the map during the generation of the map for the camera periphery or during the display of the augmented reality image.

Referring to FIG. 5, the controller 180 may update the map based on the changed position when at least one of the positions of the cameras 201 and 121 and the minutiae points of the map are changed S510).

When the map is updated as described above, the controller 180 synthesizes the virtual object on the updated map based on the updated map (S520). The augmented reality image obtained by synthesizing the virtual object on the updated map can be output through the head mount display 200 as described above.

In a state where the updated augmented reality image based on the map is output to the head mount display 200, the controller 180 may detect a user input applied to the virtual object of the augmented reality image at step S530.

When a user input is detected on the virtual object, the controller 180 may track the position of the detected user input and perform a corresponding function (S540). Here, the tracking of the position of the user input may be performed through the camera 201 provided in the head mount display 200. [

Specifically, the user input may be performed by a pointing device or a gesture of pointing a specific area or a specific point in which the virtual object is displayed using the user's finger. In addition, the user input may be performed by a pointing device or a gesture of moving a user in a predetermined direction after pointing at a specific area or a specific point displayed on the virtual object using the user's finger.

In the case where the user input is performed using the user's finger, the process of detecting the user input is as follows.

The controller 180 can detect the user's finger from the image obtained using the camera 121 or 201. [ For example, a user's finger may be recognized (or a voice command may be performed in parallel), starting with a predetermined gesture (e.g., a gesture in which the palm of the user is pointed at the front of the palm by spreading the fingers of both hands).

In addition, depending on the environment, only a finger of a previously registered user can be recognized. In this case, the controller 180 may ignore the hand or the finger of the unregistered third party or output a warning alarm.

Then, the camera tracks the position of the detected finger, and based on the difference between the position of the virtual object spaced from the camera and the detected finger position and the direction of the detected finger, Or pointing to a specific point.

For example, if the distance of the virtual object from the camera and the distance of the detected finger from the camera are at least equal, or at least the position of the detected finger is farther than the virtual object (that is, overlaps or passes through the virtual object) It can be recognized that the pointing to the corresponding region or point is performed if the direction of the virtual object is perpendicular to the direction in which the virtual object is displayed.

6F and 6G, the controller 180 compares the hand image (FIG. 6F) photographed through the camera with the predetermined finger shape modeling (FIG. 6G) to detect the user's fingertip have. The detected movement path of the fingertip can be traced through the camera.

On the other hand, when the user input is performed using the pointing device, the process of detecting the user input is as follows.

The control unit 180 analyzes the image obtained through the camera 121 or 201 and the camera obtained through the camera provided in the pointing device to calculate a three-dimensional relative position, direction, and angle of the pointing device with respect to the virtual object Can be recognized. Accordingly, when the relative coordinate system of the edge of the calculated pointing device and the coordinate system of the area in which the virtual object is displayed are encountered, it can be recognized that the pointing to the corresponding area or point has been performed.

When the user input is detected in this way, the control unit 180 can execute the function corresponding to the detected user input.

Here, the function corresponding to the user input means, for example, a screen change corresponding to input to a specific key or pointing to a specific point. In addition, the screen change may be a shape change of a virtual object, and / or an output of text mapped to a key recognized as an input.

Meanwhile, the controller 180 may control the sound output module 152 (FIG. 2) to output predetermined sound information along with the change of the screen.

Also, the controller 180 may control the display of the execution result of the function corresponding to the detected user input in one area of the head mount display 200. [ According to this, since the user can confirm both the user input to the virtual object and the output corresponding to the user input on the head mount display 200, the personal information is protected and the control is convenient.

Hereinafter, specific examples of user input using a virtual object in the augmented reality image displayed through the head mount display 200 will be described with reference to Figs. 7 and 8A and 8B.

In this regard, FIG. 7 is a view showing a virtual object displayed on a head mount display including a plurality of keys that can be input by a user, according to an embodiment of the present invention. 8A and 8B are views showing a virtual object including a virtual touch pad displayed on a head mount display according to an embodiment of the present invention.

The electronic device 100 is connected to the head mount display 200 through which the augmented reality image can be displayed through the wireless communication unit 110. In this way, in a state connected to the head mount display 200, the electronic device 100 drives the camera to sequentially acquire images of the surroundings. The control unit 180 may extract a plurality of feature points from the acquired image. In this manner, the controller 180 matches the same feature points in the sequentially acquired images, and obtains the three-dimensional map of the camera peripheral image by grasping the movement path of the extracted feature points based on the matching . When the map for the camera peripheral image is generated as described above, the controller 180 displays the augmented reality image obtained by synthesizing the virtual object capable of user input according to the position and direction of the camera on the basis of the generated map, (200).

In one embodiment, the virtual object displayed on the head mounted display 200 may be an input device including a plurality of keys that are user inputable.

In this case, the controller 180 generates pointing coordinates for a specific area or a specific point that is pointed on the virtual object, and when the position of the pointing coordinate is an area of a specific key in the virtual object, Text can be output.

To this end, the control unit 180 may control the display unit 200 to further display an output area capable of outputting the text together with the virtual object. At this time, the output area may be displayed in an area distinguished from the virtual object, or may be overlapped with a part of the displayed area of the virtual object.

For example, referring to FIG. 7, an input device having a plurality of keys through the head mount display 200 may be displayed as the virtual object. The input device may be in the form of a keyboard as shown in FIG. 7, but the present invention is not limited thereto. The input device may be of any type including a key for receiving a control command for the connected electronic device 100. For example, the input device may be in the form of a musical instrument having a specific tone color. In addition, a predetermined output area 310 'may be displayed in an area distinguished from the displayed area of the virtual object.

If the pointing is recognized in the 'x' key by the user's hand while the virtual object 300 'in the form of a keyboard is displayed, the control unit 180 displays' Lt; RTI ID = 0.0 > ', < / RTI > The user can immediately confirm the output result corresponding to the input key. At this time, the inputted '?' Character may be output simultaneously to the electronic device 100 connected to the head mount display 200, for example, the display unit 151 of the mobile terminal. In the personal information protection mode, the control unit 180 may display the input 'i' character only through the head mount display 200.

In another embodiment, the virtual object displayed on the head mounted display 200 may be a content screen including a virtual touch pad.

In this case, the controller 180 generates pointing coordinates for a specific area or a specific point pointed to on the virtual object, synthesizes the first augmented reality image in the augmented reality in the virtual object corresponding to the pointing coordinate, And can be displayed on the head mount display 200. Here, the first augmented reality image may be an image and / or text of a predetermined augmented reality, for example, a mouse cursor or a hand image.

When the first augmented reality image is synthesized in the augmented reality in the virtual object, the controller 180 recognizes that the augmented reality image is pointed to the corresponding area or point. On the other hand, if the gesture of a predetermined type is detected while the first augmented reality image is pointing to a specific position, the controller 180 can perform an operation corresponding to the gesture. For example, if the location indicated by the first augmented reality image is a web address of a specific homepage, if there is a gesture knocking on the area where the first augmented reality image is displayed, the screen corresponding to the web address may be displayed as a virtual object.

For example, referring to FIG. 8A, time information output to the mobile terminal 100 through the head mount display 200, for example, an execution screen of a web application may be displayed. At this time, it is possible to include all types of letters, numbers, symbols, graphics, icons, etc. in the outputable time information.

When the execution screen of the web application is output to the virtual object 300 ', for example, when pointing to a specific point is detected by the user's hand, the control unit 180 calculates the coordinates of the pointed position, The mouse cursor icon can be synthesized into the first augmented reality image in the augmented reality in the object 300 'and displayed.

Further, in another embodiment, the virtual object displayed on the head mounted display 200 may be an input / output device including a virtual touch pad.

In this case, the controller 180 may detect a drag input having a specific area or a specific point pointed to on the virtual object as a starting point. The control unit 180 may combine the second augmented reality image with the augmented reality in the virtual object corresponding to the detected position and direction of the drag input and display the second augmented reality image on the head mounted display 200. [ Here, the second augmented reality image may be an image including the start point and the end point of the input of the drag, and the range of the trajectory corresponding to the input drag. For example, the second augmented reality image may be a handwriting input or a signature input by the user on the virtual object.

For this, the controller 180 may set the effective area of the drag input in advance. In addition, the effective area of the drag input may be enlarged or reduced according to the user gesture depending on the environment.

As described above, the second augmented reality image synthesized in the augmented reality in the virtual object can be generated in real time in accordance with the detected drag input. Meanwhile, the controller 180 can display the second augmented reality image only through the head mount display 200 in the personal information protection mode.

For example, referring to FIG. 8B, a predetermined input screen may be displayed through the head mount display 200. Specifically, the control unit 180 may synthesize the trajectory range of the drag input made by the user's hand with the second augmented reality image 320 and output the result. On the other hand, the trajectory range of the input drag input may be output as it is, or the straight line, angle, and shape of the input drag input may be determined and converted to recognized text and output.

The controller 180 may store the augmented reality image including the drag input in the memory 160 of the electronic device 100. [ In addition, the controller 180 may transmit an augmented reality image including a drag input to another external device connected to the electronic device 100.

Hereinafter, it will be described how data is exchanged with another external device to which an electronic device according to an embodiment of the present invention is connected, by sharing data of the map or the virtual object.

An electronic device 100 according to an embodiment of the present invention is connected to a head mount display 200 that is capable of displaying an augmented reality image through a wireless communication unit 110. [ In this way, in a state connected to the head mount display 200, the electronic device 100 drives the camera to sequentially acquire images of the surroundings. The control unit 180 may extract a plurality of feature points from the acquired image. The same feature points are matched in a plurality of images sequentially acquired in this manner, and a three-dimensional map of the camera peripheral image is generated by grasping the movement path of the extracted feature points based on the matching. When the map for the camera peripheral image is generated as described above, the controller 180 displays the augmented reality image obtained by synthesizing the virtual object capable of user input according to the position and direction of the camera on the basis of the generated map, To be output.

Meanwhile, the electronic device 100 may share the augmented reality image obtained by synthesizing the generated map information and the virtual object with another external device.

To this end, the electronic device 100 is connected to an external device through the wireless communication unit 110, and the control unit 180 controls the three-dimensional data information of the generated map and the virtual object Data information can be transmitted to the connected external device. At this time, the connected external device may request the electronic device 100 to transmit the three-dimensional data information and the virtual object data information of the generated map. In addition, the electronic device 100 transmits the three-dimensional data information of the generated map and the data information of the virtual object to a predetermined server (not shown) so as to upload the data to the server (not shown) The external device may receive the three-dimensional data information of the uploaded map and the data information of the virtual object.

When the three-dimensional data information of the same map is shared with the virtual object data information, the external device can display the augmented reality image synthesized with the virtual object on the display unit or through the second mount display (not shown) . At this time, the display of the augmented reality image including the virtual object may be respectively displayed on the mount display 200 connected to the electronic device 100 and the display of the external device. Or, depending on the environment, one augmented reality image may be shared on the display of the external device and the mount display 200 connected to the electronic device 100. [ In this case, a restriction condition that the mount display 200 and the display of the external device are adjacent to each other within a predetermined distance will be necessary.

When the same virtual object data information is shared with the same map in this way, it becomes possible to share a plurality of contents together according to each control of the electronic device 100 and the external device.

Meanwhile, the controller 180 may set whether or not to allow the user of the external device to interact with the virtual object.

Specifically, the control unit 180 may determine whether to allow user interaction with the virtual object displayed on the connected external device based on the transmitted data information based on the preset priority. Specifically, the user of the electronic device 100 can allow a user of the connected external device to view the augmented reality image including the virtual object, or to condition the user to interact with the user input . Here, the predetermined priority may be determined based on the information stored in the memory 160 of the electronic device 100.

The determined result may be transmitted to the connected external device together with the data information through the wireless communication unit 110. [

Meanwhile, in the embodiments described above, an example has been described in which an augmented reality image including a generated three-dimensional map and a virtual object is displayed on the head mount display 200. However, in a case where the input / output is not related to personal information, The augmented reality image including the map and the virtual object may be displayed on the display unit 151 of the electronic device 100 or may be displayed on the electronic device 100 or the projector provided on the head mounted display 200 It is needless to say that they may be displayed by using.

As described above, according to the electronic device and the operation method of the electronic device according to the embodiment of the present invention, only the user wearing the head mount display can view the screen changes corresponding to the user input and the user input using the augmented reality image By doing so, it is possible to protect personal information and personal privacy. In addition, a three-dimensional map is registered using a plurality of minutiae points extracted from arbitrary unscored scenes, objects, textures and the like around the camera, and a virtual object capable of user input is displayed thereon along the position of the camera, The input / output device can be displayed in a desired position and direction, thereby providing user convenience. Furthermore, the operation of the electronic device can be conveniently controlled by implementing input and output using a virtual object together on a head mount display connected to the electronic device.

Further, according to the embodiment disclosed herein, the above-described method can be implemented as a code that can be read by a processor on a medium on which the program is recorded. Examples of the medium that can be read by the processor include ROM, RAM, CD-ROM, magnetic tape, floppy disk, optical data storage, etc., and may be implemented in the form of a carrier wave (e.g., transmission over the Internet) .

The above-described electronic device can be applied to a configuration and a method of the embodiments described above in a limited manner, but the embodiments can be configured such that all or some of the embodiments are selectively combined so that various modifications can be made It is possible.

Claims (15)

A wireless communication unit connected to a head-mounted display configured to display an augmented reality image; And
Sequentially acquiring images of surrounding objects using a camera, extracting a plurality of feature points from the acquired image, and determining a movement path of the extracted feature points according to a result obtained by matching identical feature points in a plurality of sequentially acquired images And generating a three-dimensional map,
Wherein,
And controls the wireless communication unit to transmit an augmented reality image obtained by synthesizing a virtual object capable of user input according to the position and direction of the camera based on the generated map to the connected head-mounted display. The method according to claim 1,
If the camera is fixed to the head-mounted display,
Wherein the control unit tracks the movement path of the head mount display to capture an image of a surrounding area and controls the moving and displaying of the augmented reality image along the movement path of the head mount display. 3. The method of claim 2,
Wherein the camera is a single stereo camera provided in front of an amount of a user wearing the head mount display. The method according to claim 1,
Wherein,
Updates the map by tracking the changed position when at least one of the positions of the camera and the minutiae of the map is changed, and synthesizes the virtual object on the updated map on the basis of the updated map. Device. The method according to claim 1,
Wherein,
And controls to display a result of the execution in one area of the head mounted display when a user input is detected on the virtual object by tracking the position of the detected user input to execute a corresponding function, . 6. The method of claim 5,
Wherein the user input is performed by a pointing device or a gesture pointing a specific area or a specific point displayed by the virtual object using a user's finger or moving in a predetermined direction after the pointing. The method according to claim 6,
Wherein,
Detecting a finger of the user from the image obtained using the camera and tracking the position of the detected finger to determine a distance difference between the virtual object and the detected finger spaced from the camera, And recognizes whether or not to point to the specific region or the specific point on the basis of the determination result. The method according to claim 6,
Wherein the virtual object is an input device including a plurality of keys capable of user input,
Wherein,
When the pointing coordinate pointed to the specific area or the specific point being pointed is a specific key in the virtual object, an output area capable of outputting a text mapped to the specific key is displayed on the head mount display together with the virtual object . The method according to claim 6,
Wherein the virtual object is a content screen including a virtual touch pad,
Wherein,
Pointing coordinates for the specific area or the specific point being pointed and combining the first augmented reality image in the augmented reality in the virtual object corresponding to the pointing coordinate and displaying it on the head-mounted display Electronic device. The method according to claim 6,
Wherein the virtual object is an input / output device including a virtual touch pad,
Wherein,
When there is a drag input having the starting point of the specific area or the specific point being pointed, a second augmented reality image is synthesized in the augmented reality in the virtual object corresponding to the position and direction of the drag input and displayed on the head mount display And said control means controls said control means. The method according to claim 1,
Wherein the wireless communication unit is connected to an external device to transmit / receive data information,
Wherein,
And controls the wireless communication unit to transmit the generated three-dimensional data information of the map and the data information of the virtual object to the connected external device based on the control signal. 12. The method of claim 11,
Wherein,
Determining whether to allow a user interaction with a virtual object displayed on the connected external device based on the transmitted data information based on a predetermined priority, and transmitting the determined result to the connected external device And controls the wireless communication unit. Connecting to a head-mounted display capable of displaying an augmented reality image;
Sequentially acquiring images of a surrounding area using a camera;
Extracting a plurality of feature points from the acquired image;
Matching the same feature points in the sequentially acquired images and generating a three-dimensional map by grasping a movement path of the extracted feature points based on the matching; And
And transmitting an augmented reality image obtained by synthesizing a virtual object including a virtual touch pad in accordance with the position and direction of the camera based on the generated map to the connected head mounted display Lt; / RTI > 14. The method of claim 13,
Updating the map in correspondence with a change of at least one of positions of minutiae points of the camera and the map; And
And synthesizing the virtual object on the updated map based on the updated map. The method according to claim 13 or 14,
Tracking a location of the detected user input to perform a corresponding function when a user input is detected on the virtual object; And
Further comprising the step of controlling to display the execution result in one area of the head-mounted display.

Images