KR20180113115A - Mobile terminal and method for controlling the same - Google Patents

Abstract

The present invention relates to an electronic device and a method for controlling virtual reality content by using the same. According to an embodiment of the present invention, the electronic device includes: a body which is configured to be worn on a specific part of a user′s body; a display unit which includes a viewing area and plays virtual reality content in the viewing area; and a control unit which generates first control commands related to the displayed virtual reality content upon sensing an input in the viewing area and generates second control commands different from the first control commands after stopping the playback of the virtual reality content being displayed upon sensing an input on the outside of the viewing area. The control unit can also generate third control commands which are different from the first control commands while being related to the displayed virtual reality content upon dragging a target of the input sensed in any one of the viewing area or the areas on the outside of the viewing area to another area.

Description

BACKGROUND OF THE INVENTION 1. Field of the Invention [0001] The present invention relates to an electronic device,

The present invention relates to a method and an electronic apparatus for controlling contents of a virtual reality in an electronic device.

An HMD (Head Mounted Display) type electronic device is a display device that is worn on a head of a user and can present an image directly in front of the user's eyes. Such an electronic device may allow a user to enjoy video contents with a larger image than a TV or a screen, and in particular, display a virtual space screen so that a user can enjoy a virtual space experience.

Such an electronic device can be used in another device such as a smart phone, a laptop computer, a navigation device, a slate PC, a tablet PC, an ultrabook, a wearable device wearable device) have been actively studied.

At this time, in order to control the real world while experiencing the virtual reality using the electronic device, the virtual reality had to be stopped or the electronic device had to be removed.

It is therefore an object of the present invention to provide an electronic device capable of quickly and easily interacting with a virtual reality and a real world without removing a main body from a user wearing the electronic device and an operation method thereof The purpose is to provide.

According to an aspect of the present invention, there is provided an electronic device comprising: a main body; A display unit including a field of view and reproducing the virtual reality contents in the field of view; And generating a first control command associated with the displayed virtual reality content when an input is detected within the viewing area, and stopping playback of the displayed virtual reality content when an input is detected in an area outside the viewing area, And a controller for generating a second control command different from the first control command when the object of the input sensed in any one of the visual field area and the visual field area is dragged into another area, And generates a third control command different from the first control command in association with the virtual reality contents.

Also, in one embodiment, the viewing area is a space in which information related to the virtual reality contents is provided to a user wearing the main body, and an area outside the viewing area is a space .

In one embodiment, when the object of the input corresponding to the second control command is dragged into the view area, the suspended virtual reality contents are reproduced again according to the third control command.

Also, in one embodiment, when the second control command is generated, the displayed virtual reality contents are replaced with other virtual reality contents.

Also, in one embodiment, when a predetermined triggering operation is detected in an area outside the viewing area while the virtual reality contents are being reproduced, an indicator indicating that the second control command is input is displayed on the virtual reality contents The controller generates the second control command in response to an additional input sensed in an area outside the visual field while the indicator is being displayed.

In one embodiment, when an event is generated during reproduction of the virtual reality contents, a notification icon indicating occurrence of an event is outputted so as to correspond to the virtual reality contents, and the control unit responds to the output of the notification icon And controls the display unit to output a screen associated with processing of event information corresponding to the notification icon when an input is detected in an area outside the viewing area.

Also, in one embodiment, the event is an alarm generated in a real device, and a screen associated with the processing of the event information is a control screen for controlling an operation of a real device in which the event occurs.

In one embodiment, when an object corresponding to the first control command is dragged into an area outside the visual field while controlling an object included in the displayed virtual reality content according to the first control command, The control method of the object is changed differently.

Also, in one embodiment, the object is a virtual input pad, and when the object of the input corresponding to the first control command is dragged into an area outside the visual field, a pointer is created on the virtual input pad And controls the virtual input pad using the pointer according to the second control command.

Further, in one embodiment, when the object of the input corresponding to the second control command is dragged into the field of view, the pointer disappears from the virtual input pad.

Further, in one embodiment, in response to the fact that the object of the input corresponding to the first control command has been dragged and dropped on the specific object of the reality, the control unit processes the virtual reality content based on the attribute of the specific object .

In one embodiment, the control unit provides guide information for confirming whether the virtual reality content is processed before processing the virtual reality content based on the attribute of the specific object, And the processing is executed based on the response to the command.

Also, in one embodiment, when the object related to the real information is dragged and dropped onto a virtual object included in the view area, the control unit controls the processing of the virtual object based on the information of the reality And controls the display unit.

Also, in one embodiment, the control unit outputs guide information for requesting confirmation of a user wearing the main body before execution of the second control command, and when a response to the guide information is input in the view area, And cancel the input corresponding to the second control command.

According to another embodiment of the present invention, there is provided a virtual reality apparatus, comprising: a sensing unit for sensing a degree of movement of the main body while interacting with a virtual object included in the virtual reality content according to the first control command; The range of the visual field is adjusted based on the position of the main body and the degree of motion of the main body.

Also, in one embodiment, when the user's gaze of the user wearing the main body is fixed to a specific object included in the virtual reality contents for a predetermined time, the control unit controls the viewing area Is adjusted.

According to another aspect of the present invention, there is provided a method of operating an electronic device, the method including: reproducing a virtual reality content in a field of view including a field of view; Generating a first control command associated with the displayed virtual reality content when an input is detected within the viewing area; Stopping reproduction of the displayed virtual reality contents and generating a second control command different from the first control command when an input is detected in an area outside the visual field; And generating a third control command that is associated with the virtual reality content and is different from the first control command when an object of the input detected in any one of the view area and the out of field area is dragged into another area The method comprising the steps of:

As described above, the electronic device according to the embodiment of the present invention can easily and quickly interact with virtual reality and objects in the real world while naturally crossing over using the view area.

Figs. 1A and 1B are conceptual diagrams showing electronic devices related to the present invention in different directions. Fig.
2 is a block diagram for explaining the configuration of an electronic device related to the present invention.
FIGS. 3A, 3B, and 3C are conceptual diagrams for explaining an area outside the view area and the view area in the electronic device related to the present invention. FIG.
Figs. 4A and 4B are views showing examples of different execution results according to input in a viewing area and input in an area outside a viewing area, in an electronic device related to the present invention. Fig.
5 is a representative flowchart for explaining an operation method of an electronic device related to the present invention.
6A, 6B, 6C, 6D and 6E illustrate an example of processing an event occurring in a real world during a virtual reality content experience using a view area in an electronic device related to the present invention.
Figs. 7A, 7B, 7C, and 7D are conceptual diagrams illustrating a method of dragging an input in a field of view to a reality to control a virtual object differently in an electronic device related to the present invention.
FIGS. 8A, 8B, and 8C are conceptual diagrams for explaining a method of processing virtual reality contents by dragging and dropping an input in a viewing area to a real object in an electronic device related to the present invention. FIG.
9A and 9B are conceptual diagrams illustrating a method of processing a virtual object by dragging and dropping an object associated with real information into a view area in an electronic device related to the present invention.
FIGS. 10A, 10B, and 10C are conceptual diagrams for explaining a method of canceling an input made outside the view area in an electronic device related to the present invention. FIG.
Figs. 11A, 11B, 12A, and 12B are diagrams illustrating various examples of adjusting the range of the viewing area in an electronic device related to the present invention.

Hereinafter, embodiments of the present invention will be described in detail with reference to the accompanying drawings, wherein like or similar components are denoted by the same reference numerals, and redundant explanations thereof will be omitted. The suffix "module" and " part "for the components used in the following description are given or mixed in consideration of ease of specification, and do not have their own meaning or role. In the following description of the embodiments of the present invention, a detailed description of related arts will be omitted when it is determined that the gist of the embodiments disclosed herein may be blurred. It is to be understood that both the foregoing general description and the following detailed description are exemplary and explanatory and are intended to provide further explanation of the invention as claimed. , ≪ / RTI > equivalents, and alternatives.

Terms including ordinals, such as first, second, etc., may be used to describe various elements, but the elements are not limited to these terms. The terms are used only for the purpose of distinguishing one component from another.

It is to be understood that when an element is referred to as being "connected" or "connected" to another element, it may be directly connected or connected to the other element, . On the other hand, when an element is referred to as being "directly connected" or "directly connected" to another element, it should be understood that there are no other elements in between.

The singular expressions include plural expressions unless the context clearly dictates otherwise.

In the present application, the terms "comprises", "having", and the like are used to specify that a feature, a number, a step, an operation, an element, a component, But do not preclude the presence or addition of one or more other features, integers, steps, operations, elements, components, or combinations thereof.

A mobile phone, a smart phone, a laptop computer, a digital broadcasting terminal, a personal digital assistant (PDA), a portable multimedia player (PMP), a navigation device, A slate PC, a tablet PC, an ultrabook, a wearable device (e.g., a smartwatch, a glass glass, a head mounted display (HMD) )), And the like.

However, it should be understood that the configuration according to the embodiments described herein may be applied to fixed terminals such as a digital TV, a desktop computer, a digital signage, and the like, except in a case where the configuration is applicable only to an electronic device, It will be readily apparent to those skilled in the art.

Also, as used in the embodiments described below, 'user' may mean a person using an electronic device or an apparatus using an electronic device (for example, a robot having artificial intelligence). In addition, the embodiments described below describe techniques related to the operations of an electronic device for capturing a virtual space during a virtual experience. In addition, the electronic device described below may mean a wearable device formed to be wearable on a specific part (e.g., a head) of the user's body.

Figs. 1A and 1B are conceptual diagrams showing electronic devices related to the present invention in different directions. Fig.

The electronic device 100 according to the present invention includes a wireless communication unit, an input unit (e.g., a user input unit 123a, 123b, 123c, a microphone or the like), a sensing unit 140, an output unit 151, and an audio output unit 152), an interface unit 160, a memory, a control unit, and a power supply unit. The components shown in FIG. 1A are not essential for implementing an electronic device, so that the electronic device 100 described herein may have more or fewer components than the components listed above.

1A, an electronic device 100 according to the present invention is configured to be worn on a head (or head, face, head) of a user (human body) Cover, etc.). The frame part may be formed of a flexible material to facilitate wearing. In this figure, the frame portion includes a first frame 101 and a second frame 102 of different materials.

The first frame 101 serves to provide a space in which at least one of the components described with reference to FIG. 1A can be disposed, and the second frame 102 serves to support the first frame 101 (Or fixed) to be mountable on the head of the user (human body).

The frame portion may be referred to as a body (or an electronic device body) or a body (or an electronic device body). Here, the electronic device body (or the electronic device body) may be understood as a concept of referring to the electronic device 100 as at least one aggregate.

If the frame portion including the first frame 101 and the second frame 102 is regarded as one electronic device body, the body of the electronic device related to the present invention can be formed in various forms. Specifically, the main body may include a plurality of surfaces having predetermined angles. The plurality of surfaces refer to the surfaces located outside the body of the electronic device 100. In this regard, the plurality of surfaces may refer to the surface (outer surface, outer surface, etc.) of the electronic device 100. Each of the plurality of surfaces may be flat or curved.

In addition, the wearing portion 112 may contact a part of the user's body. For example, the wearer 112 may use a band formed of an elastic material so that the frame portion is brought into close contact with the eyes of the user's face.

The main body (frame portion) is supported on the head portion, and a space for mounting various components is provided. As shown in the figure, the first frame 101 includes a camera 121, a display unit 151, user input units 123a, 123b and 123c, a control unit 180, a sensing unit 140, an interface unit 160, The same electronic component can be mounted.

The second frame 102 may be provided with an electronic component such as an acoustic output unit 152 or the like. However, the present invention is not limited to this, and the first frame 101 and the second frame 102 may have various other components required for the electronic device, depending on the user's selection. That is, the electronic device 100 described herein may have more or fewer components than those listed above.

The control unit 180 of the electronic device is configured to control various electronic components included in the electronic device 100.

The camera 121 may be provided on the display unit 151 of the main body. For example, the camera 121 may be disposed on one side (e.g., the front side) of the main body of the electronic device 100. [ The camera 121 is disposed adjacent to at least one of the left eye and the right eye, and is configured to capture (receive, input) a forward image. Since the camera 121 is disposed adjacent to the eye and facing forward, the camera 121 can acquire a scene viewed by the user as an image.

In this figure, one camera 121 is provided, but the present invention is not limited thereto. The plurality of cameras 121 may be provided to acquire stereoscopic images.

The electronic device 100 may include a sensing unit 140. The sensing unit 140 may be a proximity sensor, an illumination sensor, a touch sensor, an acceleration sensor, a magnetic sensor, a G-sensor, a gyroscope sensor, A motion sensor, an RGB sensor, an infrared sensor, a finger scan sensor, an ultrasonic sensor, an optical sensor, a microphone, ), A battery gauge, an environmental sensor (for example, a barometer, a hygrometer, a thermometer, a radiation detection sensor, a heat sensing sensor, a gas sensing sensor, Biometric sensors, etc.), and the like.

For example, the controller 180 may detect movement of the electronic device using a gyroscope sensor, a gravity sensor, a motion sensor, and the like included in the sensing unit 140. Alternatively, the control unit 180 may detect an object approaching the periphery of the electronic device using a proximity sensor, an illuminance sensor, a magnetic sensor, an infrared sensor, an ultrasonic sensor, an optical sensor, or the like included in the sensing unit 140.

The electronic device 100 may include user inputs 123a, 123b, and 123c that are manipulated to receive control commands. The user input units 123a, 123b, and 123c can be employed in any manner as long as the user operates in a tactile manner such as touch, push, or wheel operation. In this drawing, it is exemplified that all the user inputs 123a, 123b, and 123c of the push input method, the touch input method, and the wheel operation input method are provided in the frame portion.

In addition, the electronic device 100 may include a microphone (not shown) that receives sound and processes the sound into electrical voice data and an acoustic output unit 152 that outputs sound. The sound output unit 152 may be configured to transmit sound in a general sound output mode or a bone conduction mode. When the sound output unit 152 is implemented in a bone conduction manner, when the user wears the electronic device 100, the sound output unit 152 is brought into close contact with the head and vibrates the skull to transmit sound.

The display unit 151 is mounted on the frame unit and outputs screen information (e.g., image, image, moving picture, etc.) in front of the user's eyes. When the user wears the electronic device 100, the display unit 151 may be arranged to correspond to at least one of the left and right eyes so that screen information can be displayed before the user's eyes. That is, the display unit 151 may be formed to cover at least one of the left and right eyes of the user (or to face at least one of the left and right eyes of the user).

As an example, the display portion 151 of the electronic device associated with the present invention may be located inside the electronic device body. Specifically, the display unit 151 is disposed inside the electronic device, and may be disposed at a position facing the user's eyes when the user wears the electronic device on the user's head.

Also, the electronic device 100 may include a lens unit 120 so that the user can view the screen information output from the display unit 151. That is, the electronic device 100 according to the present invention is configured such that all of the screen information (or light) output from the display unit 151 is transmitted through the lens unit 120 to the user's eyes (or a field of view) .

For example, the lens unit 120 may be disposed to correspond to at least one of the eyes of the user (i.e., the left eye and the right eye). In addition, the lens portion 120 can be arranged to lie between the user ' s eye and the display portion 151 when the user wears the electronic device on the head.

Since the viewing angle may vary depending on the distance between the user's eyeball and the display unit, the lens unit 120 may be formed to be variable in position by user control. Further, the lens unit 120 may be formed through a concave lens or a convex lens, or a combination thereof.

Also, the display unit 151 can project an image with the user's eyes using a prism. Further, the prism may be formed to be transmissive so that the user can view the projected image and the general view of the front (the range that the user views through the eyes) together.

As described above, the image output through the display unit 151 can be seen overlapping with the general view. The electronic device 100 can provide augmented reality (AR) in which a virtual image is superimposed on a real image or a background using one of the characteristics of the display.

That is, the display unit 151 is formed so that external light can not pass through to realize a virtual reality (VR), or external light passes through the display unit 151 to realize Augmented Reality (AR) .

Alternatively, the electronic device 100 may be functionally connected to the terminal device 200 (FIG. 1B), and the terminal device 200 may be used as the display portion 151 of the electronic device 100. FIG. When the terminal device 200 is used as the display portion 151 of the electronic device 100, the frame portion of the electronic device 100 includes a predetermined docking space for the terminal device 200 to be coupled thereto. In this case, the terminal device 200 may be accommodated in the docking space. When the terminal device 200 is used as the display unit 151 of the electronic device 100 as described above, the camera 121 and the sensing unit 140 are replaced with a camera and a sensing unit provided in the terminal device 200, .

In addition, when the electronic device 100 is electrically connected to the terminal device 200, the control unit 180 of the electronic device may control the terminal device 200. When the terminal device 200 is mounted on the main body 200 of the electronic device as described above, the camera 121, the sensing unit 140, the display unit 151, the control unit 180 of the electronic device 100, May be replaced by a camera, a sensing unit, a display unit, and a control unit included in the terminal device 200. [

1A and 1B illustrate that the terminal device 200 is mounted on the main body of the electronic device 100 and is used as a display portion. However, the present invention is not limited thereto. For weight and weight, the electronic device 100 itself may include a display portion 151 The present invention is not limited thereto.

The terminal device 200 is mounted on the electronic device main body 200 and the configuration of the camera 121 of the electronic device 100, the sensing unit 140, the display unit 151, the control unit 180, The detailed configuration of the electronic device 100 will be described in detail on the premise that the camera, the sensing unit, the display unit, and the control unit included in the terminal device 200 are replaced.

2 is a block diagram for explaining a configuration of a terminal device 200 mounted in an electronic device 100 related to the present invention. FIGS. 3A and 3B are conceptual views of the display configuration of the electronic device related to the present invention, which are separated from each other in the direction of the display.

The terminal device 200 includes a wireless communication unit 210, an input unit 220, a sensing unit 240, an output unit 250, an interface unit 260, a memory 270, a control unit 280, and a power supply unit 290 ), And the like. The components shown in FIG. 2 are not essential for implementing a terminal device, so that the terminal device described in this specification can have more or fewer components than those listed above.

The wireless communication unit 210 among the components is connected between the terminal apparatus 200 and the wireless communication system or between the terminal apparatus 200 and the other terminal apparatus 200 or between the terminal apparatus 200 and the external server 200. [ Lt; RTI ID = 0.0 > wireless < / RTI > In addition, the wireless communication unit 210 may include one or more modules for connecting the terminal device 200 to one or more networks.

The wireless communication unit 210 may include at least one of a broadcast receiving module 211, a mobile communication module 212, a wireless Internet module 213, a short distance communication module 214 and a location information module 215 .

The input unit 220 includes a camera 221 or an image input unit for inputting an image signal, a microphone 222 for inputting an audio signal, an audio input unit, a user input unit 223 for receiving information from a user A touch key, a mechanical key, and the like). The voice data or image data collected by the input unit 220 may be analyzed and processed by a user's control command.

The sensing unit 240 may include one or more sensors for sensing at least one of the information in the terminal device, the surrounding environment information surrounding the terminal device, and the user information. For example, the sensing unit 240 may include a proximity sensor 241, an illumination sensor 242, a touch sensor, an acceleration sensor, a magnetic sensor, A G-sensor, a gyroscope sensor, a motion sensor, an RGB sensor, an infrared sensor, a finger scan sensor, an ultrasonic sensor, A microphone 221, a battery gauge, an environmental sensor (such as a barometer, a hygrometer, a thermometer, a radiation detection sensor, a temperature sensor, A thermal sensor, a gas sensor, etc.), a chemical sensor (e.g., an electronic nose, a healthcare sensor, a biometric sensor, etc.). Meanwhile, the terminal device disclosed in the present specification can combine and utilize information sensed by at least two of the sensors.

The output unit 250 includes at least one of a display unit 251, an acoustic output unit 252, a haptrip module 253, and a light output unit 254 for generating an output related to a visual, auditory, can do. The display unit 251 may have a mutual layer structure with the touch sensor or may be integrally formed to realize a touch screen. The touch screen functions as a user input unit 223 that provides an input interface between the terminal device 200 and a user and can provide an output interface between the terminal device 200 and a user.

The interface unit 260 serves as a channel with various types of external devices connected to the terminal device 200. The interface unit 260 may be configured to connect a device having a wired / wireless headset port, an external charger port, a wired / wireless data port, a memory card port, And may include at least one of a port, an audio I / O port, a video I / O port, and an earphone port. In the terminal device 200, an appropriate control relating to the connected external device can be performed in correspondence with the connection of the external device to the interface 260.

In addition, the memory 270 stores data supporting various functions of the terminal device 200. The memory 270 may store a plurality of application programs (application programs or applications) driven by the terminal device 200, data for operation of the terminal device 200, and commands. At least some of these applications may be downloaded from an external server via wireless communication. At least a part of these application programs may exist on the terminal device 200 from the time of shipment to the terminal device 200 for basic functions (e.g., call incoming, outgoing, message reception, and origination functions). Meanwhile, the application program may be stored in the memory 270, installed on the terminal device 200, and may be driven by the control unit 280 to perform the operation (or function) of the terminal device.

In addition to the operations related to the application program, the control unit 280 typically controls the overall operation of the terminal device 200. [ The control unit 280 may process or process signals, data, information or the like inputted or outputted through the above-mentioned components or may drive an application program stored in the memory 270 to provide or process appropriate information or functions to the user.

In addition, the control unit 280 may control at least some of the components illustrated in FIG. 2 to drive an application program stored in the memory 270. FIG. Furthermore, the control unit 280 may operate at least two or more of the components included in the terminal device 200 in combination with each other for driving the application program.

The power supply unit 290 receives external power and internal power under the control of the controller 280 and supplies power to the components included in the terminal apparatus 200. The power supply unit 290 includes a battery, which may be an internal battery or a replaceable battery.

At least some of the components may operate in cooperation with each other to implement a method of operation, control, or control of a terminal device according to various embodiments described below. In addition, the operation, control, or control method of the terminal device can be implemented on the terminal device by driving at least one application program stored in the memory 270.

First, referring to the wireless communication unit 210, the broadcast receiving module 211 of the wireless communication unit 210 receives broadcast signals and / or broadcast-related information from an external broadcast management server through a broadcast channel. The broadcast channel may include a satellite channel and a terrestrial channel. More than one broadcast receiving module may be provided to the mobile terminal 200 for simultaneous broadcast reception or broadcast channel switching for at least two broadcast channels.

The mobile communication module 212 may be a mobile communication module or a mobile communication module that is capable of communicating with one or more mobile communication devices in a mobile communication environment using technology standards or communication methods (e.g., Global System for Mobile communication (GSM), Code Division Multi Access (CDMA), Code Division Multi Access 2000 (Enhanced Voice-Data Optimized or Enhanced Voice-Data Only), Wideband CDMA (WCDMA), High Speed Downlink Packet Access (HSDPA), High Speed Uplink Packet Access (HSUPA), Long Term Evolution And an external terminal, or a server on a mobile communication network established according to a long term evolution (AR), a long term evolution (AR), or the like.

The wireless signal may include various types of data depending on a voice call signal, a video call signal or a text / multimedia message transmission / reception.

The wireless Internet module 213 is a module for wireless Internet access, and may be built in or externally attached to the terminal device 200. The wireless Internet module 213 is configured to transmit and receive wireless signals in a communication network according to wireless Internet technologies.

Wireless Internet technologies include, for example, wireless LAN (WLAN), wireless fidelity (Wi-Fi), wireless fidelity (Wi-Fi) Direct, DLNA (Digital Living Network Alliance), WiBro Interoperability for Microwave Access, High Speed Downlink Packet Access (HSDPA), High Speed Uplink Packet Access (HSUPA), Long Term Evolution (LTE) and Long Term Evolution-Advanced (LTE-A) 213 transmit and receive data according to at least one wireless Internet technology in a range including internet technologies not listed above.

The wireless Internet module 213 for performing a wireless Internet connection through the mobile communication network can be used for wireless Internet access by WiBro, HSDPA, HSUPA, GSM, CDMA, WCDMA, LTE and LTE- May be understood as a kind of the mobile communication module 212.

The short-range communication module 214 is for short-range communication, and includes Bluetooth ™, Radio Frequency Identification (RFID), Infrared Data Association (IrDA), Ultra Wideband (UWB), ZigBee, NFC (Near Field Communication), Wi-Fi (Wireless-Fidelity), Wi-Fi Direct, and Wireless USB (Wireless Universal Serial Bus) technology. The short distance communication module 214 may be connected to the wireless communication system between the terminal device 200 and the other terminal device 200 or between the terminal device 200 and the terminal device 200 through wireless local area networks ) And the network in which the other terminal device 200 (or the external server) is located. The short-range wireless communication network may be a short-range wireless personal area network.

Here, the other terminal device 200 is a wearable device (e.g., a smartwatch, smart glass, etc.) capable of exchanging data with the terminal device 200 according to the present invention (smart glass), HMD (head mounted display)). The short range communication module 214 can detect (or recognize) a wearable device capable of communicating with the terminal device 200 around the terminal device 200. [ If the detected wearable device is a device authenticated to communicate with the terminal device 200 according to the present invention, the control unit 280 may transmit at least a part of the data processed by the terminal device 200 to the short- 214 to the wearable device. Therefore, the user of the wearable device can use the data processed in the terminal device 200 through the wearable device. For example, according to this, when a telephone is received in the terminal device 200, the user performs a telephone conversation via the wearable device, or when a message is received in the terminal device 200, It is possible to check the message.

The position information module 215 is a module for obtaining the position (or current position) of the terminal device, and representative examples thereof include a Global Positioning System (GPS) module or a Wireless Fidelity (WiFi) module. For example, when the terminal device utilizes the GPS module, the terminal device can acquire the position of the terminal device using a signal transmitted from the GPS satellite. As another example, when the terminal device utilizes the Wi-Fi module, the terminal device can acquire the position of the terminal device based on information of a wireless access point (AP) transmitting or receiving the wireless signal with the Wi-Fi module. Optionally, the location information module 215 may perform any of the other modules of the wireless communication unit 210 to obtain replacement or additionally data regarding the location of the terminal device. The position information module 215 is a module used for obtaining the position (or the current position) of the terminal device, and is not limited to the module for directly calculating or acquiring the position of the terminal device.

Next, the input unit 220 is for inputting image information (or signal), audio information (or signal), data, or information input from a user. For inputting image information, Or a plurality of cameras 221 may be provided. The camera 221 processes an image frame such as a still image or a moving image obtained by the image sensor in the video communication mode or the photographing mode. The processed image frame can be displayed on the display unit 251 or stored in the memory 270. [ The plurality of cameras 221 provided in the terminal device 200 may be arranged to have a matrix structure and the terminal device 200 may be provided with various angles or focuses through a camera 221 having a matrix structure A plurality of pieces of image information can be input. In addition, the plurality of cameras 221 may be arranged in a stereo structure to acquire a left image and a right image for realizing the stereoscopic image.

The microphone 222 processes the external acoustic signal into electrical voice data. The processed voice data can be utilized variously according to a function (or a running application program) being executed in the terminal device 200. [ Meanwhile, the microphone 222 may be implemented with various noise reduction algorithms for eliminating noise generated in receiving an external sound signal.

The user input unit 223 is for receiving information from the user and when the information is inputted through the user input unit 223, the control unit 280 can control the operation of the terminal apparatus 200 to correspond to the input information . The user input unit 223 may include a mechanical input unit (or a mechanical key, for example, a button located at a front / rear or side of the terminal device 200, a dome switch, a jog wheel, Jog switches, etc.) and touch-type input means. For example, the touch-type input means may comprise a virtual key, a soft key or a visual key displayed on the touch screen through software processing, The virtual key or the visual key can be displayed on the touch screen with various forms. For example, the virtual key or the visual key can be displayed on the touch screen, ), An icon, a video, or a combination thereof.

Meanwhile, the sensing unit 240 senses at least one of information in the terminal, surrounding environment information surrounding the terminal, and user information, and generates a corresponding sensing signal. The control unit 280 can control driving or operation of the terminal device 200 or perform data processing, function or operation related to an application program installed in the terminal device 200 based on the sensing signal. Representative sensors among various sensors that may be included in the sensing unit 240 will be described in more detail.

First, the proximity sensor 241 refers to a sensor that detects the presence of an object approaching a predetermined detection surface, or the presence of an object in the vicinity of the detection surface, without mechanical contact by using electromagnetic force or infrared rays. The proximity sensor 241 may be disposed in the inner area of the terminal device or the proximity sensor 241 near the touch screen.

Examples of the proximity sensor 241 include a transmission type photoelectric sensor, a direct reflection type photoelectric sensor, a mirror reflection type photoelectric sensor, a high frequency oscillation type proximity sensor, a capacitive proximity sensor, a magnetic proximity sensor, and an infrared proximity sensor. In the case where the touch screen is electrostatic, the proximity sensor 241 can be configured to detect the proximity of the object with a change in the electric field along the proximity of the object having conductivity. In this case, the touch screen (or touch sensor) itself may be classified as a proximity sensor.

On the other hand, for convenience of explanation, the act of recognizing that the object is located on the touch screen in proximity with no object touching the touch screen is referred to as "proximity touch & The act of actually touching an object on the screen is called a "contact touch. &Quot; The position at which the object is closely touched on the touch screen means a position where the object corresponds to the touch screen vertically when the object is touched. The proximity sensor 241 is capable of sensing proximity touch and a proximity touch pattern (e.g., a proximity touch distance, a proximity touch direction, a proximity touch speed, a proximity touch time, a proximity touch position, have. Meanwhile, the control unit 280 processes data (or information) corresponding to the proximity touch operation and the proximity touch pattern sensed through the proximity sensor 241 as described above, and further provides visual information corresponding to the processed data It can be output on the touch screen. Further, the control unit 280 can control the terminal device 200 to process different operations or data (or information) according to whether the touch to the same point on the touch screen is a proximity touch or a contact touch .

The touch sensor uses a touch (or touch input) applied to the touch screen (or the display unit 251) using at least one of various touch methods such as a resistance film type, a capacitive type, an infrared type, an ultrasonic type, Detection.

For example, the touch sensor may be configured to convert a change in a pressure applied to a specific portion of the touch screen or a capacitance generated in a specific portion to an electrical input signal. The touch sensor may be configured to detect a position, an area, a pressure at the time of touch, a capacitance at the time of touch, and the like where a touch object touching the touch screen is touched on the touch sensor. Here, the touch object may be a finger, a touch pen, a stylus pen, a pointer, or the like as an object to which a touch is applied to the touch sensor.

Thus, when there is a touch input to the touch sensor, the corresponding signal (s) is sent to the touch controller. The touch controller processes the signal (s) and transmits the corresponding data to the control unit 280. Thus, the control unit 280 can know which area of the display unit 251 is touched or the like. Here, the touch controller may be a separate component from the control unit 280, and may be the control unit 280 itself.

On the other hand, the control unit 280 may perform different controls or perform the same control according to the type of the touch object, which touches the touch screen (or a touch key provided in the touch screen). Whether to perform different controls or to perform the same control depending on the type of the touch object can be determined according to the operation state of the current terminal device 200 or an application program being executed.

On the other hand, the touch sensors and the proximity sensors discussed above can be used independently or in combination to provide a short touch (touch), a long touch, a multi touch, a drag touch ), Flick touch, pinch-in touch, pinch-out touch, swipe touch, hovering touch, and the like. Touch can be sensed.

The ultrasonic sensor can recognize the position information of the object to be sensed by using ultrasonic waves. On the other hand, the controller 280 can calculate the position of the wave generating source through the information sensed by the optical sensor and the plurality of ultrasonic sensors. The position of the wave source can be calculated using the fact that the light is much faster than the ultrasonic wave, that is, the time when the light reaches the optical sensor is much faster than the time the ultrasonic wave reaches the ultrasonic sensor. More specifically, the position of the wave generating source can be calculated using the time difference with the time when the ultrasonic wave reaches the reference signal.

The camera 221 includes at least one of a camera sensor (for example, a CCD, a CMOS, etc.), a photo sensor (or an image sensor), and a laser sensor.

The camera 221 and the laser sensor can be combined with each other to sense a touch of a sensing object with respect to a three-dimensional stereoscopic image. The photosensor can be laminated to the display element, which is adapted to scan the movement of the object to be detected proximate to the touch screen. More specifically, the photosensor mounts photo diodes and TRs (Transistors) in a row / column and scans the contents loaded on the photosensor using an electrical signal that varies according to the amount of light applied to the photo diode. That is, the photo sensor performs coordinate calculation of the object to be sensed according to the amount of change of light, and position information of the object to be sensed can be obtained through the calculation.

The display unit 251 displays (outputs) information processed in the terminal device 200. [ For example, the display unit 251 may display execution screen information of an application program driven by the terminal device 200 or UI (User Interface) and GUI (Graphic User Interface) information according to the execution screen information .

Also, the display unit 251 may be configured as a stereoscopic display unit for displaying a stereoscopic image.

In the stereoscopic display unit, a three-dimensional display system such as a stereoscopic system (glasses system), an autostereoscopic system (no-glasses system), and a projection system (holographic system) can be applied.

The audio output unit 252 may output audio data received from the wireless communication unit 210 or stored in the memory 270 in a call signal reception mode, a call mode or a recording mode, a voice recognition mode, a broadcast reception mode, The sound output unit 252 also outputs sound signals related to functions (e.g., call signal reception sound, message reception sound, etc.) performed in the terminal device 200. [ The sound output unit 252 may include a receiver, a speaker, a buzzer, and the like.

The haptic module 253 generates various tactile effects that the user can feel. A typical example of the haptic effect generated by the haptic module 253 may be vibration. The intensity and pattern of the vibration generated in the haptic module 253 can be controlled by the user's selection or the setting of the control unit. For example, the haptic module 253 may combine and output different vibrations or sequentially output the vibrations.

In addition to vibration, the haptic module 253 may be configured to perform various functions such as a pin arrangement vertically moving with respect to the contact skin surface, a spraying force or suction force of the air through the injection port or the suction port, a scratch on the skin surface, And various tactile effects such as an effect of reproducing a cold sensation using an endothermic or exothermic element can be generated.

The haptic module 253 can not only transmit the tactile effect through the direct contact but also can be implemented so that the user can feel the tactile effect through the muscular sense such as the finger or the arm. The haptic module 253 may include two or more haptic modules according to the configuration of the terminal device 200.

The light output unit 254 outputs a signal for notifying the occurrence of an event using the light of the light source of the terminal device 200. Examples of events generated in the terminal device 200 may include message reception, call signal reception, missed call, alarm, schedule notification, e-mail reception, information reception through an application, and the like.

The signal output by the light output unit 254 is implemented when the terminal device emits light of a single color or a plurality of colors to the front or rear surface. The signal output may be terminated by the terminal device detecting the event confirmation of the user.

The interface unit 260 serves as a path for communication with all external devices connected to the terminal device 200. The interface unit 260 receives data from an external device or receives power from the external device and transmits the data to each component in the terminal device 200 or allows data in the terminal device 200 to be transmitted to an external device. For example, a port for connecting a device equipped with a wired / wireless headset port, an external charger port, a wired / wireless data port, a memory card port, an audio input / output port, a video input / output port, an earphone port, and the like may be included in the interface unit 260.

The identification module is a chip for storing various information for authenticating the usage right of the terminal device 200 and includes a user identification module (UIM), a subscriber identity module (SIM) A universal subscriber identity module (USIM), and the like. Devices with identification modules (hereinafter referred to as "identification devices") can be manufactured in a smart card format. Accordingly, the identification device can be connected to the terminal 200 through the interface unit 260.

The interface unit 260 may be a path through which power from the cradle is supplied to the terminal device 200 when the terminal device 200 is connected to an external cradle, And various command signals may be transmitted to the terminal device 200. [ The various command signals or the power source input from the cradle can be operated with a signal for recognizing that the terminal device 200 is correctly mounted on the cradle.

The memory 270 may store a program for the operation of the control unit 280 and temporarily store input / output data (e.g., a phone book, a message, a still image, a moving picture, etc.). The memory 270 may store data on vibrations and sounds of various patterns that are output upon touch input on the touch screen.

The memory 270 may be a flash memory type, a hard disk type, a solid state disk type, an SDD type (Silicon Disk Drive type), a multimedia card micro type ), Card type memory (e.g., SD or XD memory), random access memory (RAM), static random access memory (SRAM), read-only memory (ROM), electrically erasable programmable read memory, a programmable read-only memory (PROM), a magnetic memory, a magnetic disk, and / or an optical disk. The terminal device 200 may be operated in association with a web storage that performs a storage function of the memory 270 on the Internet.

Meanwhile, as described above, the control unit 280 controls an operation related to an application program and an overall operation of the terminal device 200. For example, when the state of the terminal device satisfies a set condition, the control unit 280 can execute or release a lock state for restricting the input of a user's control command to applications.

In addition, the control unit 280 performs control and processing related to voice communication, data communication, video call, or the like, or performs pattern recognition processing capable of recognizing handwriting input or drawing input performed on the touch screen as characters and images, respectively . Further, the controller 280 may control any one or a plurality of the above-described components in order to implement various embodiments described below on the terminal device 200 according to the present invention.

The power supply unit 290 receives external power and internal power under the control of the controller 280 and supplies power required for operation of the respective components. The power supply unit 290 includes a battery, the battery may be an internal battery configured to be chargeable, and may be detachably coupled to the terminal body for charging or the like.

In addition, the power supply unit 290 may include a connection port, and the connection port may be configured as an example of an interface 260 through which an external charger for supplying power for charging the battery is electrically connected.

As another example, the power supply unit 290 may be configured to charge the battery in a wireless manner without using the connection port. In this case, the power supply unit 290 may use at least one of an inductive coupling method based on a magnetic induction phenomenon from an external wireless power transmission apparatus and a magnetic resonance coupling method based on an electromagnetic resonance phenomenon Power can be delivered.

In the following, various embodiments may be embodied in a recording medium readable by a computer or similar device using, for example, software, hardware, or a combination thereof.

The display unit 151 of the electronic device 100 according to the embodiment of the present invention includes a viewing area, and the virtual reality contents are reproduced in the viewing area. Here, the field of view means a spatial range in which information related to the virtual reality contents, i.e., VR data, can be provided or provided to a user wearing the main body of the electronic device 100. [ The user can not see a place outside the field of view with the electronic device 100 worn.

If the input is detected within the field of view during reproduction of the virtual reality content, the controller 180 of the electronic device 100 generates a first control command associated with the displayed virtual reality content. If an input is detected in an area outside the visual field, the controller 180 may stop reproduction of the displayed virtual reality content and generate a second control command different from the first control command. At this time, the input made in the view area and the area outside the view area may mean the same input.

In addition, when the object of the input detected in any one of the viewing area and the out-of-view area is dragged into the other area, the control unit displays a third control command different from the first control command, Can be generated. That is, it is possible to perform a more organic operation related to the virtual reality contents and / or a collaboration with the reality through the interaction of the virtual space and the real space.

FIGS. 3A, 3B and 3C are diagrams for explaining more specifically a view area and an area outside the view area in the electronic device related to the present invention. FIG.

3A, the viewing angle at which the user can obtain information through the binoculars is about 30 degrees (Color Differentiation) from the center line of the eye corresponding to both eyes. However, the entire viewing angle is very wide to about 200 degrees (Monovision). In FIG. 3B, a user wearing the electronic device 100 can recognize only information that enters the field of view 310. That is, the place visible to the user is the virtual reality, and the place that is invisible is the actual reality.

As described above, the present invention defines a space 310 (including a space that can be provided) in which information related to virtual reality contents is provided to a user wearing the electronic device 100. Therefore, the area outside the view area 310 means a space in which real information that is not visible to the user is provided.

As shown in FIG. 3C, the VR contents can be displayed in the viewing area 310 '. On the other hand, a position outside the visual field can be divided into a plurality of layers around the position of the visual field. Specifically, the first layer 321 closer to the viewing area 310 'may be defined as the VR menu area (or global menu area) and the second layer 322 farther from the viewing area 310' Can be defined as the real world. The 'out-of-view area' described below may mean the first layer. Alternatively, if the operation in the second layer can be detected through a sensor such as a camera, the 'out-of-view area' includes both the first layer and the second layer. Hereinafter, for the sake of convenience of explanation, it is possible to use a combination of the 'out-of-view area' and the 'real world' or 'real area'.

On the other hand, in the case of augmented reality (AR) content or mixed reality (MR) content such as a holographic lens, a part of the real world is included in the visual field because the real world and the virtual reality are simultaneously displayed. In the area, only the real world is the same as that of the virtual reality (AR) content.

It should also be noted that the examples described below can be applied to augmented reality (AR) or mixed reality (MR) content as well as to virtual reality (AR) content. Therefore, the object or real-world information existing in the real area described below is also naturally included in the visual area.

Users feel virtual reality and real world as the same space even during VR experience. Thus, a gesture performed in an area outside the field of view may be for controlling the real world, or for controlling the virtual reality. In order to avoid such confusion, if the control of the virtual reality is restricted to be performed only in the field of view, it may feel uncomfortable or uncomfortable in controlling the contents. For example, clicking 1: 1 virtual reality within the field of view can be viewed as strange to third parties in the real world with uncomfortable posture.

Thus, input in an area outside the field of view needs to be connected to control over the virtual reality.

In this regard, FIGS. 4A and 4B show different execution results according to the input in the viewing area and the independent input in the area outside the viewing area.

As shown in FIG. 4A, a specific image 431 stored in the gallery can be displayed as an example of the virtual reality contents. At this time, when the user wearing the electronic device 100 performs a specific gesture, for example, a gesture 401 moving in the horizontal direction in the viewing area 410, the execution of the 'first control command' The next / previous image 432 stored in the gallery can be displayed. That is, through the gesture in the viewing area, the displayed virtual reality contents are intuitively controlled.

4B, when the user wearing the electronic device 100 performs the same gesture as in FIG. 4A, i.e., in the lateral direction, in the area outside the viewing area 410, The reproduction / display of the display unit 431 is stopped. Then, by executing the above-described 'second control command', contents of a different kind than displayed contents 432, for example, web contents 441 can be displayed. That is, through the gesture in the real area, the displayed virtual reality contents are controlled non-intuitively or converted to completely different virtual reality contents.

Further, since the input in the real area may be inaccurate or unintentional, in one example, it may be required to precede the trigger operation for recognizing the input in the real area as the control instruction.

Specifically, when a predetermined triggering operation is detected in an area outside the viewing area while the virtual reality contents are being reproduced on the display unit 151, an indicator indicating that input of the 'second control command' Lt; / RTI > For example, when a plurality of tab operations are performed in the real area, a 'c' icon 411 indicating that the screen can be controlled in the real area may be displayed as shown in FIG. 4B.

Next, the controller 180 may generate the second control command in response to the additional input sensed in an area outside the visual field while the indicator is displayed on the virtual reality contents. For example, while the 'c' icon 411 is displayed, the controller 180 may recognize the gesture performed in the real area as the second control command. On the other hand, if the display of the 'c' icon 411 disappears as the predetermined time elapses, the gesture performed in the real area may be regarded as unintentional and ignored.

5 is a representative flowchart for explaining an operation method of the electronic device according to the present invention.

Referring to FIG. 5, the electronic device 100 reproduces / displays the virtual reality contents through the view area of the display unit 151 (S10). Here, the field of view means a space where a user wearing the electronic device 100 can acquire or acquire VR information as described above. The area outside the visual field means the real world or the real area.

In addition, the virtual reality contents to be reproduced / displayed may be 2D or 3D images, and there is no limitation to the types. If the virtual reality contents are 3D images, a virtual reality space can be formed. In this case, the user makes a misconception of space as if the virtual reality exists in the virtual reality space formed by the virtual reality contents.

If the input is detected within the viewing area while the virtual reality contents are being reproduced, the controller 180 may generate a first control command associated with the displayed virtual reality contents (S20). Here, the first control command may be an intuitive control command associated with the displayed virtual reality contents. For example, an intuitive control command can be generated, such as attacking a virtual enemy, turning a page of a three-dimensional image, touching or lifting a virtual object depending on the type of VR content displayed.

On the other hand, if the virtual reality contents include AR or MR contents, since the virtual reality and the real world are provided together in the visual field, the real world can be controlled based on the sensed input in the visual field. However, in this case, it may be limited to an object in the real world provided in the field of view.

If an input is detected in an area outside the viewing area while the virtual reality contents are being reproduced, the controller 180 stops reproduction of the displayed virtual reality contents and generates a second control command different from the first control command (S30).

Here, the second control command may include a non-intuitive control command for the displayed virtual reality content, a command related to performing another operation not related to the displayed virtual reality content, a control command related to control of an object existing in the real world, It can mean either one. For example, it may include instructions to switch the displayed virtual reality content to another content, to move to a higher / lower level in the current phase, or to control an electronic product in the real world (e.g., a smart TV, etc.) .

The input sensed in the field of view and the input sensed outside the field of view thus generate control commands independently. In other words, if the sensed input in the field of view is terminated in the field of view or if the sensed input in the real-world is terminated in the real-world, it does not affect subsequent inputs or content.

Meanwhile, when the object of the input sensed in any one of the viewing area and the viewing area is dragged into another area while the virtual reality contents are being reproduced, the controller 180 displays the virtual reality contents A third control command different from the first control command can be generated (S40). That is, the virtual reality and the real world interact or collaborate to perform control associated with the displayed virtual reality content.

Here, the input object may be the controller 200 (FIG. 6A) for sensing the motion of the user or the user's hand performing the gesture. The controller 200 will be described in detail below.

Here, the third control command can also mean a control command associated with integrating the virtual world into the real world or incorporating the virtual reality into the real world while freely moving between the virtual reality and the real world using the input object have. For example, a control command exceeding a dimension such as processing a specific object of a virtual reality by using an object of the real world or providing information of the real world to a virtual reality may be included in the third control command described above.

According to the embodiments of the present invention described above, it is possible to selectively control the virtual reality and the real world using the view area, or to perform a new type of adaptive control that goes beyond the virtual reality and the real world.

6A, 6B, 6C, 6D, and 6E illustrate an example of processing an event that occurs in a real world during a virtual reality content experience using a view area.

First, the example described here assumes that the controller 200 is used as a peripheral device of the electronic device 100 to control the virtual reality contents. The controller 200 can be an object of input in a field of view and an area outside the field of view.

Such a controller 200 can track the motion of the wearer of the electronic device 100 and interact with the electronic device 100 through various sensors, such as an IR sensor or a motion tracking sensor, can do. In addition, the controller 200 may be linked to an actual environment image that is confirmed through a camera provided in the electronic device 100. Accordingly, the user can immediately check the external environment while wearing the electronic device 100 during the virtual space experience.

In addition, the motion tracking sensor of the controller 200 may include a three-axis acceleration sensor, a three-axis gyroscope, and a digital motion processor as one aggregate. In addition, the controller 200 may include a vertically long bar-shaped grip portion that the user can easily grip, and the grip portion may be provided with a plurality of user input portions (e.g., button, track pad, etc.).

6A, while the MR game is being played in the viewing area 610, the user wearing the electronic device 100 moves the controller 200 within the viewing area 610 to fight the virtual enemy 621 . That is, interacts with the virtual enemy 621 based on the motion of the controller 200 sensed within the field of view 610.

During the MR game, the user can see part of the real world included in the field of view 610, but still can not see the real area 601 outside the field of view 610. [

At this time, when a specific event occurs, an alert icon ('EVENT!') 630 indicating the occurrence of an event is displayed in the view area 610 as shown in FIG. 6B. Here, the specific event not only occurs in the virtual reality but also includes an event occurring in the real world, for example, a case where the doorbell of the entrance is depressed as shown in FIG. 6B.

The control unit 180 controls the electronic device 100 to output a screen associated with the processing of the event information corresponding to the notification icon when an input is detected in an area outside the viewing area in response to the notification icon output in the viewing area, The display unit 151 of FIG.

In one example, the event may be an alarm generated in a real device (e.g., an electronic device such as a smart TV, a smart refrigerator, or a smart washing machine). In addition, the screen associated with the processing of the event information may be a control screen for controlling the operation of a real device in which an event occurs. To this end, a real device in which an event has occurred is wirelessly communicated with the electronic device 100 via a BLE or the like, or connected to each other through the home device system, thereby enabling the electronic device 100 to detect the occurrence of an event . ≪ / RTI >

For example, as shown in FIG. 6C, while the notification icon 630 is displayed, the controller 200 is lowered by a predetermined distance or a predetermined angle or more from the viewing area 200, and then a specific key provided in the controller 200 is pushed , It can be recognized as an input in the real area 601. [ Accordingly, as shown in FIG. 6D, the playback of the MR game is stopped (621 '), and a smart home control screen 631 for executing the door lock of the entrance can be displayed in the viewing area. At this time, though not shown in FIG. 6D, the virtual enemy 621 'and the smart home control screen 631, which are temporarily suspended, are treated as being output to different virtual layers. Accordingly, the control on the smart home control screen 631 does not affect the movement of the virtual enemy 621 '.

In addition, in one embodiment, when an object of an input detected in an area outside the viewing area is dragged into the viewing area, the virtual reality contents that were interrupted according to the third control command can be reproduced again. For example, in FIG. 6E, after the user 'opens' the door lock of the door with the electronic device 100 worn using the smart home control screen 631, (200), thereby performing the above-described third control command. Accordingly, the MR game is reproduced again, and the virtual enemy moves again in the field of view 610 (622).

Meanwhile, although not shown, the controller 200 of FIG. 6A may be modified and displayed in a different shape (e.g., a virtual knife, a gun, etc.) depending on the type of MR content. In this case, when the deformed shape size of the controller 200 is different according to the actual size of the controller 200 and the experience of the MR contents, whether or not the deformed controller 200 is out of the view area 1, the second, and the third control commands.

Alternatively, depending on the content type of VR, AR, and MR, the range of the visual field may be temporarily expanded to distinguish the inputs of the first, second, and third control commands. This is because the user feels the virtual reality and the real world in the same space, so that the control can be performed unintentionally outside the view area as the control of the virtual reality contents becomes more frequent.

Hereinafter, Figs. 7A, 7B, 7C, and 7D show examples related to pulling the input in the field of view into the real world to control the virtual objects differently.

7A, in a state where a virtual message window 721 is displayed in the viewing area 710, a user touches a key of the virtual input pad 722 to input text. At this time, when the real area 701 is in a public place as shown in FIG. 7A, a state in which the touch operation is continuously performed on the empty space may appear strange to the third party, and the user may have a sense of refusal.

In this case, when an object (e.g., a user's hand or pen) of an input for which input has been performed in the view area 710 is dragged to the area 701 outside the view area, the control method of the object included in the view area 710 Are changed differently.

7B, when the user's hand placed on the virtual input pad 722 is dragged to the real area 701, the controller 180 generates a pointer 723 on the virtual input pad . At this time, although not separately shown, the virtual input pad 722 and the pointer 723 actually exist in different layers. For example, a second layer in which a pointer 723 is displayed on an upper portion of a first layer in which a virtual input pad 722 is displayed may be overlapped.

7C, the user performs an operation of 'Moving' or 'Clicking' the finger in the body such as the object existing in the real world 701 or the user's knee, (724) to perform text input (725).

Next, when a target object (e.g., a user's hand or pen) that has performed input in the real area is dragged into the view area 710 again (713), the control method of the virtual input pad returns to the initial state. As a result, the pointer disappears on the virtual input pad. In practice, the second layer with the pointer displayed is removed, and the first layer is again activated (i.e., replayed / displayed).

8A, 8B, and 8C illustrate an example of processing virtual reality contents by dragging and dropping an input in a field of view onto a real object. Especially, here, UX using the edge area is applied.

The controller 180 can detect that the object of the input sensed in the visual field is dragged and dropped to a specific object in the real area. Specifically, it can be sensed that a target object of the input sensed in the visual area is dragged and dropped to a real object located in an edge area of the visual area. Here, the limitation is made that the real object included in the visual field is located in the boundary area of the visual field. That is, an example of UX using the edge region of the visual field is shown. Although the edge region of the visual field is a part of the visual field, the boundary between the AR and the VR can be freely moved as if it is a real area.

When the object of the input sensed in the viewing area is dragged and dropped onto a specific object in the real area existing in the edge area of the viewing area, the controller 180 displays the virtual reality content or the virtual reality Process virtual objects contained in content.

For example, in a state where a virtual web page 821 is displayed in the view area 810 in FIG. 8A, a virtual web page 821 is displayed on the edge area of the viewing area 810 as shown in FIG. Performing the gesture 822 to the trash can 811, the electronic device 100 grasps the properties of the real trash can 811. Next, the virtual web page 821 that has been displayed is deleted as shown in FIG. 8C. At this time, based on the coordinates of the real trash can 811 existing in the viewing area 810, the controller 180 displays the graphic effect such that the virtual web page 821 is drawn into the real trash can 811 Can be output.

When the user moves, the real objects located in the edge area of the viewing area are continuously changed. In this case, the controller 180 can visually recognize the real object newly included in the edge area of the viewing area So that the corresponding graphic object can be displayed around the newly included object.

In one example, guide information for confirming whether or not the virtual reality contents are processed may be provided before processing the virtual reality contents based on the attribute of the specific object to which the input object is dropped. In this case, processing is executed based on the user's response to the guide information.

As another example, Figs. 9A and 9B show an example of processing an imaginary object by dragging and dropping an object associated with real-world information into a virtual object in the field of view. Again, the information of reality may be limited to exist in the edge area of the field of view, but it can also be applied to information of reality existing at any position of the field of view through user input.

The control unit 180 controls the display unit 151 so that, when an object related to the real information is dragged and dropped onto a virtual object included in the view area, processing of the virtual object is performed based on the information of the reality.

For example, as shown in FIG. 9A, the actual handwriting input 911 input on the actual paper using the actual pen in the edge area of the viewing area 910 is referred to as an actual pen (input (922), which is popped up in the field of view 910, to the search area of the virtual web page 921. [ Then, a real handwriting input ('Kyongbok Palace') is automatically input into the search word 923 in the search area of the virtual web page 921. At this time, character recognition of a real handwriting input may be performed through a camera or the like provided in the electronic device 100. [ In some cases, the characters of the actual handwriting input can be recognized by zooming in the edge area of the viewing area.

According to the example described above, the user can quickly input data using a real-life handwriting tool, and then bring the data directly to the virtual reality through the intuitive and easy UX.

FIGS. 10A, 10B, and 10C illustrate an exemplary method for canceling an input made outside the field of view.

The unintentional input of the user can be made in an area outside the viewing area as described above. In such a case, the user may experience inconvenience because the currently viewed virtual reality contents may be interrupted.

If an input in an area outside the visual field, that is, an input in a real area is detected, the controller 180 outputs guide information for requesting confirmation of the user wearing the main body before executing the second control command corresponding thereto Can be output. When a response to the guide information is made in the visual field, the input corresponding to the second control command is canceled. That is, it is determined that the user intends to continuously view the current virtual reality contents.

For example, as shown in FIG. 10A, when a specific photo 1021 of a gallery is detected as an MR content outside the view area 1010, guide information 1031, 1032) pops up. At this time, if the user is unintentional input, a specific gesture (e.g., hand shake to the left and right) 1011 is performed in the view area 1010 as shown in FIG. 10B, No). Accordingly, as the guide information disappears as shown in FIG. 10C, the display of the specific photo 1021 of the gallery is continuously maintained.

On the other hand, if the input is intentional input by the user outside the view area 1010, no input is performed for a predetermined time. In this case, it is assumed that the user has switched back to the other contents with 'Yes', and at that moment, the specific picture 1021 is replaced with the web page 1022 of FIG. 10B.

Next, Figs. 11A, 11B, 12A, and 12B show various examples of adjusting the range of the viewing area.

11A and 11B illustrate examples in which the range of the visual field is adjusted based on the degree of motion of the main body and the position of the virtual object. The sensing unit 140 of the electronic device 100 according to the present invention may be configured to control the electronic device 100 while the user interacts with a virtual object included in the virtual reality content in the view area according to the input sensed in the viewing area. The degree of movement of the main body can be detected.

The controller 180 may adjust the range of the current field of view based on the position of the virtual object interacting with the user and the degree of motion of the main body of the electronic device 100. Specifically, for example, while interacting with a virtual keyboard 1111 displayed in the view area 1110 in FIG. 11A by performing a keystroke, the user looks up the virtual monitor 1113 The range of the visual range is extended 1110 'so that the virtual keyboard 1111 continues to enter the visual range. At this time, since there is a virtual keyboard 1111 that interacts with the lower portion of the body movement direction, the range of the visual range extends downward.

On the other hand, if the user who has not interacted with the virtual monitor (not shown) interacts with the virtual monitor to lower his / her head for viewing the virtual keyboard, the range of the viewing area expands upward by a distance corresponding to the degree of movement of the main body.

On the other hand, such range adjustment of the visual field may be provisional. Accordingly, when the predetermined time has elapsed, it is possible to automatically return to the range of the previous visual field area.

In addition, the virtual objects displayed in the extended portion in the range of the extended viewing area can be visually distinguished from the virtual objects existing in the range of the first viewing area. For example, the virtual keyboard displayed in the expanded portion in the range of the view area in FIG. 11B may appear to be fainter than other objects.

As another example, Figs. 12A and 12B are examples for adjusting the range of the viewing area according to the gesture in the real area.

When the user's eyes of the user wearing the main body of the electronic device 100 are fixed to a specific object included in the virtual reality contents in the visual region for a predetermined time, the control unit 180 sets the range of the visual field on the basis of the input sensed in the real area Can be adjusted. That is, the range of the field of view can be intentionally narrowed or broadened according to the size of a specific object the user is interested in.

For example, as shown in FIG. 12A, when the user performs a pinch-in gesture in the real area while gazing at a specific object in the view area 1210, the range of the view area is narrowed by a distance corresponding to the gesture 1221 . In this case, since the range of the visual field is narrowed, the range of the actual information is widened.

As another example, when the user performs a pinch-out gesture in the real area while gazing at a particular object in the field of view 1210, the range of the field of view is expanded by a distance corresponding to the gesture 1222, as in Fig. In this case, as the range of the view area is widened, the information of reality is removed, so that the feeling of immersion is improved.

As described above, the electronic device according to the embodiment of the present invention can easily and quickly interact with virtual reality and objects in the real world while naturally crossing the virtual reality using the view area.

The present invention described above can be embodied as computer-readable codes on a medium on which a program is recorded. The computer readable medium includes all kinds of recording devices in which data that can be read by a computer system is stored. Examples of the computer readable medium include a hard disk drive (HDD), a solid state disk (SSD), a silicon disk drive (SDD), a ROM, a RAM, a CD-ROM, a magnetic tape, a floppy disk, , And may also be implemented in the form of a carrier wave (e.g., transmission over the Internet). Also, the computer may include a control unit 180 of the terminal. Accordingly, the above description should not be construed in a limiting sense in all respects and should be considered illustrative. The scope of the present invention should be determined by rational interpretation of the appended claims, and all changes within the scope of equivalents of the present invention are included in the scope of the present invention.

Claims (17)

main body;
A display unit including a field of view and reproducing the virtual reality contents in the field of view; And
Generating a first control command associated with the displayed virtual reality content when an input is detected within the viewing area and stopping playback of the displayed virtual reality content when an input is detected in an area outside the viewing area, And a control unit for generating a second control command different from the control command,
Wherein,
And generating a third control command different from the first control command in association with the virtual reality content when the object of the input sensed in any one of the visual field area and the visual field area is dragged into another area ≪ / RTI > The method according to claim 1,
Wherein the visual field is a space in which information related to the virtual reality contents is provided to a user wearing the main body and an area outside the visual field is a space in which real information not shown to the user is provided. . The method according to claim 1,
Wherein when the object of the input corresponding to the second control command is dragged into the field of view, the suspended virtual reality content is reproduced again according to the third control command. The method according to claim 1,
And when the second control command is generated, the displayed virtual reality content is replaced with another virtual reality content. The method according to claim 1,
An indicator indicating that the second control command can be input is displayed on the virtual reality content when a predetermined triggering operation is detected in an area outside the view area while the virtual reality content is being reproduced,
Wherein,
And generates the second control command in response to an additional input sensed in an area outside the visual field while the indicator is displayed. The method according to claim 1,
When an event is generated during reproduction of the virtual reality contents, a notification icon indicating occurrence of the event is outputted so as to correspond to the virtual reality contents,
Wherein,
Wherein the control unit controls the display unit to output a screen associated with processing of event information corresponding to the notification icon when an input is detected in an area outside the viewing area in response to the output of the notification icon. The method of claim 6, wherein
Wherein the event is an alarm generated in a real device, and a screen associated with the processing of the event information is a control screen for controlling an operation of a real device in which the event occurs. The method according to claim 1,
When the object of the input corresponding to the first control command is dragged into an area outside the view area while the object included in the displayed virtual reality content is controlled according to the first control command, And changes the state of the electronic device. 9. The method of claim 8,
The object is a virtual input pad,
Wherein,
Generating a pointer on the virtual input pad when a target object of the input corresponding to the first control command is dragged into an area out of the viewing area and controlling the virtual input pad using the pointer according to the second control command And the electronic device. 10. The method of claim 9,
And when the object of the input corresponding to the second control command is dragged into the field of view, the pointer disappears from the virtual input pad. The method according to claim 1,
Wherein,
Wherein the virtual reality content processing unit processes the virtual reality content based on the attribute of the specific object in response to the fact that the object of the input corresponding to the first control command is dragged and dropped on the specific object in the real world. 12. The method of claim 11,
Wherein,
Providing guide information for confirming whether or not the virtual reality contents are processed before processing the virtual reality contents based on the attribute of the specific object and executing processing based on a response to the guide information ≪ / RTI > The method according to claim 1,
Wherein,
Controls the display unit such that, when an object related to real information is dragged and dropped on a virtual object included in the view area, processing of the virtual object is performed based on the information of the reality. The method according to claim 1,
Wherein,
Outputs guide information for requesting confirmation of a user wearing the main body before execution of the second control command, and cancels input corresponding to the second control command when a response to the guide information is input in the view area . The method according to claim 1,
And a sensing unit for sensing a degree of movement of the main body while interacting with a virtual object included in the virtual reality content according to the first control command,
Wherein,
Wherein the range of the visual field is adjusted based on the position of the virtual object and the degree of motion of the main body. The method according to claim 1,
Wherein,
And adjusts the range of the visual field based on an input corresponding to the second control command when the visual line of the user wearing the main body is fixed to a specific object included in the virtual reality contents for a predetermined time, . Reproducing a virtual reality content in the field of view, the field of view including a field of view;
Generating a first control command associated with the displayed virtual reality content when an input is detected within the viewing area;
Stopping reproduction of the displayed virtual reality contents and generating a second control command different from the first control command when an input is detected in an area outside the visual field; And
Generating a third control command different from the first control command in association with the virtual reality content when the object of the input sensed in any one of the visual field area and the visual field area is dragged into another area And a control unit for controlling the operation of the electronic device.

Images