KR20180108100A - A system including head mounted display and method for controlling the same - Google Patents

Abstract

The present invention relates to a system including a head-mounted display which provides at least one screen outputting an external environment sensed through at least one camera while using virtual reality content by using the head-mounted display and a control method thereof. Therefore, a user can freely confirm the external environment while using the virtual reality content through a camera provided in each of the head-mounted display and an external controller.

Description

Field of the Invention [0001] The present invention relates to a system including a head-mounted display,

The present invention relates to a system including a head-mounted display that provides at least one screen for outputting an external environment that is sensed by at least one camera while utilizing virtual reality content using a head-mounted display, and a method of controlling the same.

Virtual Reality (VR) is an interface that makes a particular environment or situation computerized and makes it as if the person using it interact with the actual environment or environment.

A head-mounted display (HMD) is an image display device that is worn on a user's head like a spectacle to allow a user to view the image (contents). 2. Description of the Related Art [0002] Various wearable computers have been developed in accordance with the trend of weight reduction and miniaturization of digital devices, and the HMD is also widely used. The HMD can be combined with augmented reality technology, N-screen technology, etc., beyond the simple display function, to provide various convenience to the user.

On the other hand, in order to provide a realistic VR environment, not only an HMD but also an external controller connected thereto can be provided. In this case, depending on the motion of the external controller, the VR content being output to the HMD can be controlled. However, when the camera is provided outside the HMD, the HMD outputs an image sensed by the camera regardless of the motion of the external controller, thereby confusing the user. For example, even if the external controller is rotated or moved, the user can not confirm the forward image of the desired position unless the user moves the head in the direction to be confirmed.

The present invention is directed to solving the above-mentioned problems and other problems. It is another object of the present invention to provide a system including a head-mounted display for controlling an external environment as well as an external environment through an external camera provided in the HMD, and a control method thereof .

It is still another object of the present invention to provide a system including a head-mounted display which can easily confirm an external environment through a camera provided in each of an HMD and an external controller, and a control method thereof.

According to an aspect of the present invention, there is provided an HMD system including a head-mounted display (HMD) and first and second external controllers, the HMD comprising: a display; A first camera; A first wireless communication unit for transmitting / receiving data to / from the first and second external controllers; And a control unit, wherein the first external controller comprises: a second camera; A second wireless communication unit for transmitting / receiving data to / from the HMD; And a first sensing unit for sensing movement and input of the first external controller, wherein the second external controller comprises: a third camera; A third wireless communication unit for transmitting / receiving data with the HMD; And a second sensing unit for detecting movement and input of the second external controller, wherein the control unit outputs VR (Virtual Reality) content to a first screen of the display unit, And receives control information from at least one external controller through the wireless communication unit in a state of outputting an external image sensed by the third camera to the second sub screen, And controls the first sub screen and the second sub screen based on the information.

According to another aspect of the present invention, there is provided a head-mounted display (HMD), comprising: a display; camera; A wireless communication unit for transmitting / receiving data to / from the first and second external controllers; And a controller, wherein the controller outputs VR (Virtual Reality) content to a first screen of the display unit, outputs an external image received from the first external controller to a first sub screen, Receiving control information from at least one external controller through the wireless communication unit while outputting an external image received from a second external controller, and based on the control information, the first auxiliary screen and the second auxiliary screen And the HMD is controlled by the control unit.

According to another aspect of the present invention, there is provided a method of controlling an HMD system including a head-mounted display (HMD) and first and second external controllers, Outputs an external image sensed by the camera of the first external controller to the first sub screen and outputs an external image sensed by the camera of the second external controller to the second sub screen ; Receiving control information from at least one external controller via a wireless communication unit; And controlling the first sub screen and the second sub screen based on the control information.

Effects of the system including the head-mounted display according to the present invention and the control method thereof will be described as follows.

According to at least one of the embodiments of the present invention, the user can freely check the external environment while using the VR contents through the camera provided in each of the HMD and the external controller.

In addition, according to at least one embodiment of the present invention, there is an advantage that a space outside the view angle of the camera provided in the HMD can be monitored through a camera provided in the external controller.

Further scope of applicability of the present invention will become apparent from the following detailed description. It should be understood, however, that the detailed description and specific examples, such as the preferred embodiments of the invention, are given by way of illustration only, since various changes and modifications within the spirit and scope of the invention will become apparent to those skilled in the art.

1 is a block diagram for explaining an HMD related to the present invention.
2 illustrates an HMD and a plurality of controllers according to an embodiment of the present invention.
3 is a view showing an example of a position and an angle of view of a camera provided in the HMD system according to an embodiment of the present invention.
4 is a flowchart illustrating a method of controlling an HMD system according to an embodiment of the present invention.
5 is a diagram illustrating an example of outputting a flash screen in the HMD system according to an embodiment of the present invention.
6 is a diagram illustrating another example of outputting a flash screen in the HMD system according to an embodiment of the present invention.
7 is a diagram illustrating another example of outputting a flash screen in the HMD system according to an embodiment of the present invention.
8 is a view illustrating another example of outputting a flash screen in the HMD system according to an embodiment of the present invention.
9 is a flowchart illustrating a method of controlling an HMD system according to an embodiment of the present invention.
10 is a view showing an example of providing a zoom-in of an auxiliary screen in an HMD system according to an embodiment of the present invention.
11 is a view illustrating an example of merging a plurality of sub-screens in an HMD system according to an embodiment of the present invention.
12 is a diagram illustrating an example of holding an auxiliary screen after merging in an HMD system according to an embodiment of the present invention.
13 is a diagram illustrating an example of dividing one sub screen into a plurality of sub screens in the HMD system according to an embodiment of the present invention.
FIG. 14 is a view showing an example of outputting an external image sensed by each camera in each of the sub-screens in the HMD system according to an embodiment of the present invention.
15 is a diagram illustrating an example of checking an external environment using an external controller in the HMD system according to an embodiment of the present invention.
16 is a diagram illustrating an example of checking an external environment in cooperation with an external device in the HMD system according to an embodiment of the present invention.

Hereinafter, embodiments of the present invention will be described in detail with reference to the accompanying drawings, wherein like reference numerals are used to designate identical or similar elements, and redundant description thereof will be omitted. The suffix "module" and " part "for the components used in the following description are given or mixed in consideration of ease of specification, and do not have their own meaning or role. In the following description of the embodiments of the present invention, a detailed description of related arts will be omitted when it is determined that the gist of the embodiments disclosed herein may be blurred. It is to be understood that both the foregoing general description and the following detailed description are exemplary and explanatory and are intended to provide further explanation of the invention as claimed. , ≪ / RTI > equivalents, and alternatives.

Terms including ordinals, such as first, second, etc., may be used to describe various elements, but the elements are not limited to these terms. The terms are used only for the purpose of distinguishing one component from another.

It is to be understood that when an element is referred to as being "connected" or "connected" to another element, it may be directly connected or connected to the other element, . On the other hand, when an element is referred to as being "directly connected" or "directly connected" to another element, it should be understood that there are no other elements in between.

The singular expressions include plural expressions unless the context clearly dictates otherwise.

In the present application, the terms "comprises", "having", and the like are used to specify that a feature, a number, a step, an operation, an element, a component, But do not preclude the presence or addition of one or more other features, integers, steps, operations, elements, components, or combinations thereof.

The HMD described in this specification may include a wearable device (e.g., a smart glass) or the like.

However, the configuration according to the embodiments described herein may be applied to mobile phones, smart phones, laptop computers, digital broadcasting terminals, personal digital assistants (PDAs), PMPs (personal digital assistants) It will be appreciated by those skilled in the art that the present invention may be applied to mobile terminals such as a portable multimedia player, navigation, slate PC, tablet PC, ultrabook, smart watch, You will know.

1 is a block diagram for explaining an HMD related to the present invention.

The HMD 100 includes a wireless communication unit 110, an input unit 120, a sensing unit 140, an output unit 150, an interface unit 160, a memory 170, a control unit 180, And the like. The components shown in Fig. 1 are not essential for implementing the HMD, so that the HMD described in this specification can have more or fewer components than those listed above.

More specifically, the wireless communication unit 110 among the above-described components can communicate with the HMD 100 and the wireless communication system, between the HMD 100 and another HMD 100, between the HMD 100 and the mobile or fixed terminal, May include one or more modules between the HMD 100 and the control device, between the HMD 100 and a camera installed externally and capable of wireless communication, or between the HMD 100 and an external server. have.

Also, the wireless communication unit 110 may include one or more modules that connect the HMD 100 to one or more networks.

The wireless communication unit 110 may include at least one of a broadcast receiving module 111, a mobile communication module 112, a wireless Internet module 113, a short distance communication module 114, and a location information module 115 .

The input unit 120 includes a camera 121 or an image input unit for inputting a video signal, a microphone 122 for inputting an audio signal, an audio input unit, a user input unit 123 for receiving information from a user A touch key, a mechanical key, and the like). The voice data or image data collected by the input unit 120 may be analyzed and processed by a user's control command.

The sensing unit 140 may include at least one sensor for sensing at least one of information in the HMD, surrounding environment information surrounding the HMD, and user information. For example, the sensing unit 140 may include a proximity sensor 141, an illumination sensor 142, a touch sensor, an acceleration sensor, a magnetic sensor, A G-sensor, a gyroscope sensor, a motion sensor, an RGB sensor, an infrared sensor, a finger scan sensor, an ultrasonic sensor, A microphone 226, a battery gauge, an environmental sensor (for example, a barometer, a hygrometer, a thermometer, a radiation detection sensor, A thermal sensor, a gas sensor, etc.), a chemical sensor (e.g., an electronic nose, a healthcare sensor, a biometric sensor, etc.). Meanwhile, the HMD disclosed in the present specification can combine and use information sensed by at least two of the sensors.

The output unit 150 includes at least one of a display unit 151, an acoustic output unit 152, a haptrip module 153, and a light output unit 154 to generate an output related to visual, auditory, can do.

The interface unit 160 serves as a channel with various kinds of external devices connected to the HMD 100. The interface unit 160 is connected to a device having a wired / wireless headset port, an external charger port, a wired / wireless data port, a memory card port, A video I / O port, and an earphone port, for example, as shown in FIG. 1. In the HMD 100, corresponding to the connection of the external device to the interface unit 160, appropriate control relating to the connected external device can be performed.

In addition, the memory 170 stores data supporting various functions of the HMD 100. The memory 170 may store a plurality of application programs (application programs or applications) driven by the HMD 100, data for operation of the HMD 100, and commands. At least some of these applications may be downloaded from an external server via wireless communication. At least a part of these application programs are provided for the HMD 100 from the time of shipment to the HMD 100 in order to perform basic functions (for example, a screen information output function such as an image, a moving picture and the like, a phone call reception function, a message reception function, (100). ≪ / RTI > Meanwhile, the application program may be stored in the memory 170, installed on the HMD 100, and may be driven by the controller 180 to perform the operation (or function) of the HMD.

In addition to the operations related to the application program, the control unit 180 typically controls the overall operation of the HMD 100. The control unit 180 may process or process signals, data, information, and the like input or output through the above-described components or may drive an application program stored in the memory 170 to provide or process appropriate information or functions to the user All.

In addition, the controller 180 may control at least some of the components illustrated in FIG. 1A in order to drive an application program stored in the memory 170. FIG. In addition, the controller 180 may operate at least two of the components included in the HMD 100 in combination with each other for driving the application program.

Under the control of the control unit 180, the power supply unit 190 receives external power and internal power, and supplies power to the components included in the HMD 100. The power supply unit 190 includes a battery, which may be an internal battery or a replaceable battery.

At least some of the components may operate in cooperation with each other to implement a method of operating, controlling, or controlling an HMD according to various embodiments described below. In addition, the operation, control, or control method of the HMD can be implemented on the HMD by driving at least one application program stored in the memory 170. [

Hereinafter, the components listed above will be described in more detail with reference to FIG. 1 before explaining various embodiments implemented through the HMD 100 as described above.

First, referring to the wireless communication unit 110, the broadcast receiving module 111 of the wireless communication unit 110 receives broadcast signals and / or broadcast-related information from an external broadcast management server through a broadcast channel. The broadcast channel may include a satellite channel and a terrestrial channel. More than one broadcast receiving module may be provided to the HMD 100 for simultaneous broadcast reception or broadcast channel switching for at least two broadcast channels.

The broadcast management server includes a server for generating and transmitting broadcast signals and / or broadcast-related information, a device for receiving the generated broadcast signals and / or broadcast-related information and controlling the HMD by being connected to the HMD or the HMD For example, a control device, a terminal, or the like). The broadcast signal may include a TV broadcast signal, a radio broadcast signal, a data broadcast signal, and a broadcast signal in which a data broadcast signal is combined with a TV broadcast signal or a radio broadcast signal.

The broadcasting signal may be encoded according to at least one of technical standards for transmitting and receiving a digital broadcasting signal (or a broadcasting system, for example, ISO, IEC, DVB, ATSC, etc.) It is possible to receive the digital broadcasting signal using a method appropriate to the technical standards defined by the technical standards.

The broadcast-related information may be information related to a broadcast channel, a broadcast program, or a broadcast service provider. The broadcast-related information may also be provided through a mobile communication network. In this case, it may be received by the mobile communication module 112.

The broadcast-related information may exist in various forms, for example, an Electronic Program Guide (EPG) of Digital Multimedia Broadcasting (DMB) or an Electronic Service Guide (ESG) of Digital Video Broadcast-Handheld (DVB-H). The broadcast signal and / or the broadcast-related information received through the broadcast receiving module 111 may be stored in the memory 170.

The mobile communication module 112 may be a mobile communication module or a mobile communication module such as a mobile communication module or a mobile communication module that uses technology standards or a communication method (e.g., Global System for Mobile communication (GSM), Code Division Multi Access (CDMA), Code Division Multi Access 2000 (Wideband CDMA), HSDPA (High Speed Downlink Packet Access), HSUPA (High Speed Uplink Packet Access), LTE (Long Term Evolution), LTE-A And an external terminal, or a server on a mobile communication network established according to a long term evolution (e. G., Long Term Evolution-Advanced).

The wireless signal may include various types of data depending on a voice call signal, a video call signal or a text / multimedia message transmission / reception.

The wireless Internet module 113 is a module for wireless Internet access, and can be embedded in the HMD 100 or externally. The wireless Internet module 113 is configured to transmit and receive a wireless signal in a communication network according to wireless Internet technologies.

Wireless Internet technologies include, for example, wireless LAN (WLAN), wireless fidelity (Wi-Fi), wireless fidelity (Wi-Fi) Direct, DLNA (Digital Living Network Alliance), WiBro Interoperability for Microwave Access, High Speed Downlink Packet Access (HSDPA), High Speed Uplink Packet Access (HSUPA), Long Term Evolution (LTE), and Long Term Evolution-Advanced (LTE-A) The mobile terminal 113 transmits and receives data according to at least one wireless Internet technology in a range including internet technologies not listed above.

In view of the fact that wireless Internet access by WiBro, HSDPA, HSUPA, GSM, CDMA, WCDMA, LTE, LTE-A or the like is performed through a mobile communication network, the wireless Internet module 113 may be understood as a kind of the mobile communication module 112.

The short-range communication module 114 is for short-range communication, and includes Bluetooth ™, Radio Frequency Identification (RFID), Infrared Data Association (IrDA), Ultra Wideband (UWB) (Near Field Communication), Wi-Fi (Wireless-Fidelity), Wi-Fi Direct, and Wireless USB (Wireless Universal Serial Bus) technology. The short range communication module 114 is a module for communicating with the HMD 100 and the other HMD 100 between the HMD 100 and the wireless communication system via the wireless area networks, Between the HMD 100 and the controller, between the HMD 100 and the HMD 100, and between the HMD 100 and the external server, and / or between the HMD 100 and the external server. The short-range wireless communication network may be a short-range wireless personal area network.

Here, the HMD is a device (for example, a mobile phone, a smart phone, a smartwatch) capable of interchanging data with the HMD 100 according to the present invention, , A notebook computer, a control device, etc.). The short-range communication module 114 may sense (or recognize) a device capable of communicating with the HMD 100 around the HMD 100. Further, when the sensed device is a device authenticated to communicate with the HMD 100 according to the present invention, the control unit 180 transmits at least a part of the data processed in the HMD 100 through the short- To the device, and may transmit at least a portion of the data processed by the device to the HMD 100. [

Accordingly, the user of the HMD 100 can utilize the data processed in the device through the HMD 100. [ For example, according to this, the user can make a phone call through the HMD 100 when a phone is received in the device, or confirm the received message via the HMD 100 when a message is received in the device It is possible.

The position information module 115 is a module for obtaining the position (or current position) of the HMD, and a representative example thereof is a Global Positioning System (GPS) module or a Wireless Fidelity (WiFi) module. For example, an HMD can acquire the position of an HMD by using a signal sent from a GPS satellite by using a GPS module. As another example, using the Wi-Fi module, the HMD can acquire the position of the HMD based on the information of the wireless AP (wireless access point) that transmits or receives the Wi-Fi module and the wireless signal. Optionally, the location information module 115 may perform any of the other modules of the wireless communication unit 110 to obtain data relating to the location of the HMD, in addition or alternatively. The position information module 115 is a module used to acquire the position (or current position) of the HMD, and is not limited to a module that directly calculates or acquires the position of the HMD.

Next, the input unit 120 is for inputting image information (or signal), audio information (or signal), data, or information input from a user. For inputting image information, the HMD 100 may include one or A plurality of cameras 121 may be provided. The camera 121 processes an image frame such as a still image or a moving image obtained by the image sensor in the video communication mode or the photographing mode. The processed image frame may be displayed on the display unit 151 or stored in the memory 170. [ The plurality of cameras 121 provided in the HMD 100 may be arranged to have a matrix structure and the HMD 100 may have various angles or foci through the camera 121 having the matrix structure. A plurality of image information can be input. In addition, the plurality of cameras 121 may be arranged in a stereo structure so as to obtain a left image and a right image for realizing a stereoscopic image.

The microphone 122 processes the external acoustic signal into electrical voice data. The processed voice data can be utilized variously according to a function (or an application program being executed) being performed by the HMD 100. Meanwhile, the microphone 122 may be implemented with various noise reduction algorithms for eliminating noise generated in receiving an external sound signal.

The user input unit 123 is for receiving information from the user. When information is inputted through the user input unit 123, the controller 180 can control the operation of the HMD 100 to correspond to the input information . The user input unit 123 may include a mechanical input means (or a mechanical key, for example, a button located on the front / rear or side of the HMD 100, a dome switch, a jog wheel, Jog switches, etc.) and touch-type input means. As an example, the tabular input means may be at least one of a touch pad and a touch panel.

Meanwhile, the sensing unit 140 senses at least one of information in the HMD, surrounding environment information surrounding the HMD, and user information, and generates a corresponding sensing signal. The control unit 180 may control the operation or operation of the HMD 100 or may perform data processing, function or operation related to the application program installed in the HMD 100, based on the sensing signal. Representative sensors among various sensors that may be included in the sensing unit 140 will be described in more detail.

First, the proximity sensor 141 refers to a sensor that detects the presence of an object approaching a predetermined detection surface, or the presence of an object in the vicinity of the detection surface, without mechanical contact by using electromagnetic force or infrared rays. The proximity sensor 141 may be disposed within the HMD enclosed by the touch screen or proximity sensor 141 near the touch screen.

Examples of the proximity sensor 141 include a transmission type photoelectric sensor, a direct reflection type photoelectric sensor, a mirror reflection type photoelectric sensor, a high frequency oscillation type near sensor, a capacitive proximity sensor, a magnetic proximity sensor, and an infrared proximity sensor. When the touch-type input means of the user input unit 123 is electrostatic, the proximity sensor 141 can be configured to detect the proximity of the object with the change of the electric field along the proximity of the object having conductivity. In this case, the user input unit 123 itself may be classified as a proximity sensor.

For the sake of convenience of explanation, the act of allowing the user to recognize that the object is located on the user input unit 123 without touching the object on the user input unit 123 is referred to as "proximity touch "Quot;, and an action of actually touching an object on the user input unit 123 is called "contact touch. &Quot; The position where the object is touched on the user input unit 123 means the position where the object corresponds vertically to the user input unit 123 when the object is touched. The proximity sensor 141 senses a proximity touch and a proximity touch pattern (for example, a proximity touch distance, a proximity touch direction, a proximity touch speed, a proximity touch time, a proximity touch position, . The control unit 180 processes data (or information) corresponding to the proximity touch operation and the proximity touch pattern sensed through the proximity sensor 141 as described above, and further displays visual information corresponding to the processed data Can be output on the display unit 151. Further, the control unit 180 controls the HMD 100 to process different operations or data (or information) according to whether the touch on the same point on the user input unit 123 is a proximity touch or a contact touch .

The touch sensor senses a touch (or touch input) applied to the user input unit 123 using at least one of various touch methods such as a resistance film type, a capacitive type, an infrared type, an ultrasonic type, and a magnetic field type.

For example, the touch sensor may be configured to convert a change in a pressure applied to a specific portion of the user input portion 123 or a capacitance occurring in a specific portion to an electrical input signal. The touch sensor can be configured to detect a position, an area, a pressure at the time of touch, a capacitance at the time of touch, and the like where the touch target object touching the user input unit 123 is touched on the touch sensor. Here, the touch object may be a finger, a touch pen, a stylus pen, a pointer, or the like as an object to which a touch is applied to the touch sensor.

Thus, when there is a touch input to the touch sensor, the corresponding signal (s) is sent to the touch controller. The touch controller processes the signal (s) and transmits the corresponding data to the controller 180. Thus, the control unit 180 can know which area of the user input unit 123 is touched or the like. Here, the touch controller may be a separate component from the control unit 180, and may be the control unit 180 itself.

On the other hand, the control unit 180 can perform different controls or perform the same control according to the type of the touch object, which touches the user input unit 123. Whether to perform different controls or to perform the same control according to the type of the touch target object can be determined according to the operation state of the current HMD 100 or an application program being executed.

On the other hand, the touch sensors and the proximity sensors discussed above may be used independently or in combination to provide a short (or tab) touch, a long touch, a multi touch, a drag touch touch, drag-touch, flick touch, pinch-in touch, pinch-out touch, swype touch, hovering touch, It is possible to sense touches of the same and various methods.

The ultrasonic sensor can recognize the position information of the object to be sensed by using ultrasonic waves. Meanwhile, the controller 180 can calculate the position of the wave generating source through the information sensed by the optical sensor and the plurality of ultrasonic sensors. The position of the wave source can be calculated using the fact that the light is much faster than the ultrasonic wave, that is, the time that light reaches the optical sensor is much faster than the time the ultrasonic wave reaches the ultrasonic sensor. More specifically, the position of the wave generating source can be calculated using the time difference with the time when the ultrasonic wave reaches the reference signal.

The camera 121 includes at least one of a camera sensor (for example, a CCD, a CMOS, etc.), a photo sensor (or an image sensor), and a laser sensor.

The camera 121 and the laser sensor may be combined with each other to sense a touch of the sensing object with respect to the three-dimensional stereoscopic image. The photosensor can be laminated to the display element, which is adapted to scan the movement of the object to be detected proximate to the touch screen. More specifically, the photosensor mounts photo diodes and TRs (Transistors) in a row / column and scans the contents loaded on the photosensor using an electrical signal that varies according to the amount of light applied to the photo diode. That is, the photo sensor performs coordinate calculation of the object to be sensed according to the amount of change of light, and position information of the object to be sensed can be obtained through the calculation.

The display unit 151 displays (outputs) information to be processed by the HMD 100. For example, the display unit 151 may display execution screen information of an application program driven by the HMD 100 or UI (User Interface) and GUI (Graphic User Interface) information according to the execution screen information .

In addition, the display unit 151 may be configured as a stereoscopic display unit for displaying a stereoscopic image.

 The stereoscopic display unit can be applied to a three-dimensional display system such as a stereoscopic system (glasses system), an autostereoscopic system (no-glasses system), and a projection system (holographic system).

Generally, 3D stereoscopic images consist of left image (left eye image) and right image (right eye image). A top-down method of arranging a left image and a right image in one frame according to a method in which a left image and a right image are combined into a three-dimensional stereoscopic image, A checker board system in which pieces of a left image and a right image are arranged in a tile form, a left-to-right (right-side) Or an interlaced method in which rows are alternately arranged, and a time sequential (frame-by-frame) method in which right and left images are alternately displayed in time.

In addition, the 3D thumbnail image may generate a left image thumbnail and a right image thumbnail from the left image and right image of the original image frame, respectively, and may be generated as one image as they are combined. In general, a thumbnail means a reduced image or a reduced still image. The left image thumbnail and the right image thumbnail generated in this way are displayed on the screen with a difference of the left and right distance by the depth corresponding to the parallax between the left image and the right image, thereby exhibiting a stereoscopic spatial feeling.

The left and right images necessary for realizing the three-dimensional image can be displayed on the stereoscopic display unit by the stereoscopic processing unit. The stereoscopic processing unit receives the 3D image (the base view image and the expansion view image), sets the left image and the right image therefrom, or receives the 2D image and converts it into the left image and the right image.

The sound output unit 152 may output audio data received from the wireless communication unit 110 or stored in the memory 170 in a call signal reception mode, a call mode or a recording mode, a voice recognition mode, a broadcast reception mode, The sound output unit 152 also outputs sound signals associated with functions (e.g., call signal reception sound, message reception sound, and the like) performed by the HMD 100. [ The audio output unit 152 may include a receiver, a speaker, a buzzer, and the like.

The haptic module 153 generates various tactile effects that the user can feel. A typical example of the haptic effect generated by the haptic module 153 may be vibration. The intensity and pattern of the vibration generated in the haptic module 153 can be controlled by the setting of the user's selection or control unit 180. For example, the haptic module 153 may synthesize and output different vibrations or sequentially output the vibrations.

In addition to vibration, the haptic module 153 may be configured to perform various functions such as a pin arrangement vertically moving with respect to the contact skin surface, a spraying force or suction force of the air through the injection port or the suction port, a touch on the skin surface, And various tactile effects such as an effect of reproducing a cold sensation using an endothermic or exothermic element can be generated.

The haptic module 153 can transmit the tactile effect through the direct contact, and the tactile effect can be realized by the user through the muscles of the head, face, fingers and arms. At least two haptic modules 153 may be provided according to the configuration of the HMD 100.

The light output unit 154 outputs a signal for notifying the occurrence of an event using the light of the light source of the HMD 100. Examples of the event generated in the HMD 100 include a message reception, a call signal reception, a missed call, an alarm, a schedule notification, an e-mail reception, reception of information through an application, and output of an image (image, video, etc.). That is, the light output unit 154 may notify that the HMD 100 is performing a specific operation (function) by the user.

The signal output by the light output unit 154 is implemented when the HMD emits light of a single color or a plurality of colors to the front or rear surface. The signal output may be terminated based on whether the HMD detects the user's event confirmation or ends the operation being performed in the HMD.

The interface unit 160 serves as a path for communication with all external devices connected to the HMD 100. The interface unit 160 receives data from an external device or supplies power to each component in the HMD 100 or allows data in the HMD 100 to be transmitted to an external device. For example, a port for connecting a device equipped with a wired / wireless headset port, an external charger port, a wired / wireless data port, a memory card port, an audio I / O port, a video I / O port, an earphone port, and the like may be included in the interface unit 160.

The identification module is a chip for storing various information for authenticating the usage right of the HMD 100 and includes a user identification module (UIM), a subscriber identity module (SIM) a universal subscriber identity module (USIM), and the like. Devices with identification modules (hereinafter referred to as "identification devices") may be manufactured in the form of smart cards. Therefore, the identification device can be connected to the HMD 100 via the interface unit 160. [

When the HMD 100 is connected to an external cradle, the interface unit 160 may be a path through which power from the cradle is supplied to the HMD 100, or various commands A signal may be transmitted to the HMD 100. Various command signals or power from the cradle can be operated as a signal for recognizing that the HMD 100 is correctly mounted on the cradle.

The memory 170 may store a program for the operation of the controller 180 and temporarily store input / output data (e.g., a phone book, a message, a still image, a moving picture, etc.). The memory 170 may store data related to vibration and sound of various patterns outputted when a touch is input on the touch screen.

The memory 170 may be a flash memory type, a hard disk type, a solid state disk type, an SDD type (Silicon Disk Drive type), a multimedia card micro type ), Card type memory (e.g., SD or XD memory), random access memory (RAM), static random access memory (SRAM), read-only memory (ROM), electrically erasable programmable read -only memory, a programmable read-only memory (PROM), a magnetic memory, a magnetic disk, and an optical disk. The HMD 100 may be operated in association with a web storage that performs a storage function of the memory 170 on the Internet.

Meanwhile, as described above, the control unit 180 controls the operation related to the application program and the general operation of the HMD 100. For example, when the state of the HMD satisfies a set condition, the controller 180 may execute or release a lock state for restricting input of a user's control command to applications.

The control unit 180 may perform control and processing related to voice communication, data communication, video call, or the like, or may be a pattern capable of recognizing handwriting input or drawing input performed on the user input unit 123 as characters and images, respectively Recognition processing can be performed. Further, the control unit 180 may control any one or a plurality of the above-described components in order to implement various embodiments described below on the HMD 100 according to the present invention.

The power supply unit 190 receives external power and internal power under the control of the controller 180 and supplies power necessary for operation of the respective components. The power supply unit 190 includes a battery, the battery may be an internal battery configured to be chargeable, and may be detachably coupled to the HMD body for charging or the like.

In addition, the power supply unit 190 may include a connection port, and the connection port may be configured as an example of an interface 160 through which an external charger for supplying power for charging the battery is electrically connected.

As another example, the power supply unit 190 may be configured to charge the battery in a wireless manner without using the connection port. In this case, the power supply unit 190 may use at least one of an inductive coupling method based on a magnetic induction phenomenon from an external wireless power transmission apparatus and a magnetic resonance coupling method based on an electromagnetic resonance phenomenon Power can be delivered.

In the following, various embodiments may be embodied in a recording medium readable by a computer or similar device using, for example, software, hardware, or a combination thereof.

Hereinafter, embodiments related to a control method that can be implemented in the HMD configured as above will be described with reference to the accompanying drawings. It will be apparent to those skilled in the art that the present invention may be embodied in other specific forms without departing from the spirit or essential characteristics thereof.

2 illustrates an HMD and a plurality of controllers according to an embodiment of the present invention.

Referring to FIG. 2, the HMD 100 related to the present invention is formed to be worn on a head part (or head, face, head) of a human body, and a frame part 101 (a case, a housing, Etc.). The frame portion 101 may be formed of a flexible material so as to be easily worn.

The frame portion may be referred to as a body (or an HMD body) or a body (or an HMD body). Here, the HMD body (or the HMD body) may be understood as a concept of referring to the HMD 100 as at least one aggregate.

The frame portion is supported on the head portion, and a space for mounting various components can be provided. Electronic components such as a display unit 151, a camera 121, a controller, a sensing unit, a wireless communication unit, and the like may be mounted on the frame unit 101. Here, the display unit 151 may be formed to cover at least one of the left and right eyes of the user (or at least one of the left and right eyes of the user), and may be removably formed.

The control unit 180 controls various electronic components included in the HMD 100. Also, the controller 180 can be understood as a configuration corresponding to the controller 180 described with reference to FIG. In one embodiment of the present invention, operations performed by the HMD 100 may be controlled by the control unit 180. [ However, for convenience, these operations will be collectively referred to as being performed by the HMD in the drawings and the following description.

The display unit 151 is mounted on the frame unit 101 and outputs screen information (e.g., image, image, moving picture, etc.) in front of the user's eyes. When the user wears the HMD 100, the display unit 151 may be arranged to correspond to at least one of the left and right eyes so that screen information can be displayed before the user's eyes. According to an embodiment of the present invention, the display unit 151 may be positioned to cover both the left and right eyes so that an image can be output toward both the left and right eyes of the user.

Also, the display unit 151 can project an image with the user's eyes using a prism. Further, the prism may be formed to be transmissive so that the user can view the projected image and the general view of the front (the range that the user views through the eyes) together.

As described above, the image output through the display unit 151 can be seen overlapping with the general view. The HMD 100 can provide an Augmented Reality (AR) in which a virtual image is superimposed on a real image or a background and displayed as a single image by using the characteristics of the display.

In addition, the display unit 151 of the HMD related to the present invention may be located inside the main body. Specifically, when the HMD is worn on the user's head, the display unit 151 may be disposed at a position facing the user's eyes inside the HMD.

The camera 121 is disposed adjacent to at least one of the left eye and the right eye, and is formed so as to photograph a front view angle area. Since the camera 121 is disposed adjacent to the eye and facing forward, the camera 121 can acquire a scene viewed by the user as an image.

In addition, the HMD according to an embodiment of the present invention can acquire a photographed image of at least one object outside the HMD using the camera 121. For example, the camera 121 may correspond to a camera having a predetermined field of view. In addition, the camera 121 may take a predetermined object included in a specific area located outside the HMD. Here, the predetermined object may mean all kinds of identification objects forming specific information. In this figure, the camera 121 is illustrated as being single, but the present invention is not limited thereto. For example, a plurality of cameras 121 may be provided to obtain stereoscopic images. The control unit 180 may control the camera 121 to capture an object included in a predetermined angle of view and output the captured object to the display unit 151. [

The HMD can transmit / receive data to / from at least one external controller 200a and 200b through a wireless communication unit. Referring to FIG. 2, the HMD may be connected to the first external controller 200a and the second external controller 200b through a wireless communication unit, and may receive control signals from the external controllers 200a and 200b.

Here, the external controllers 200a and 200b may correspond to a sensor for experiencing virtual reality (VR) through the display unit 151. [ The external controllers 200a and 200b may be formed in various forms such as a glove, a bar shape, and a ring shape. Referring to FIG. 2, the external controllers 200a and 200b may correspond to a bar shape in which a user can easily grasp and move his or her hand.

Although not shown in FIG. 2, each of the external controllers 200a and 200b may include a sensing unit, a wireless communication unit, a camera, and a physical button.

The sensing unit of the external controllers 200a and 200b may include an infrared sensor, a motion tracking sensor, or the like. For example, the infrared sensor senses infrared light emitted from the outside such as a base station by the external controllers 200a and 200b so that the user can grasp the movement of the external controllers 200a and 200b . For example, the motion tracking sensor may include a six-axis motion tracking sensor, such as a gyro sensor, an acceleration sensor, and the like. The wireless communication units of the external controllers 200a and 200b may be provided to transmit motion information sensed by the external controller to the HMD body.

In addition, the external controllers 200a and 200b may include first and second external cameras 210a and 210b. Each of the cameras 210a and 210b is provided on the front surface of the external controller 200a or 200b facing the screen, and is configured to photograph a foreground angle field region. Each of the cameras 210a and 210b operates independently of the camera 121 provided in the HMD 100 and transmits a still image or a moving image to the HMD 100 through a wireless communication unit, Lt; / RTI > The resolution of the image sensor provided in each of the cameras 210a and 210b may be the same as or different from that of the camera 121 of the HMD 100. [

Meanwhile, the external controllers 200a and 200b may include at least one physical button. For example, the physical buttons may be provided on the front, rear, side, and the like of the external controllers 200a and 200b. According to an embodiment of the present invention, the user can set various functions such as enlargement and switching of the screen through at least one physical button provided in the external controllers 200a and 200b.

(HMD system) according to an embodiment of the present invention includes an HMD 100, a first external controller 200a, and a second external controller 200b. In the present invention, it is assumed that the external image and the VR content output to the display unit are controlled based on the motion information sensed from the plurality of external controllers 200a and 200b through the control unit 180 included in the HMD 100 do. However, it is obvious that a separate server is provided in addition to the HMD system so that the server can perform the role of the controller 180. In the following description, it is described that the control operation by the control unit of the HMD 100 is performed by the HMD 100. [

According to an embodiment of the present invention, the camera 121 of the HMD 100 senses an external image reflecting the head tracking and the mapped user's point of view, and the camera of the external controllers 200a and 200b 210b may sense a change in the viewpoint according to the movement of the external controller.

3 to 16, it is assumed that the display portion of the HMD includes a first screen and at least one auxiliary screen. For example, the first screen is a main screen, which corresponds to a screen that is output as a full screen. In the present invention, the first screen corresponds to a screen for outputting VR (virtual reality) contents. In addition, the VR content may be provided at 360 degrees, depending on the content attributes to provide similar to the actual user experience. Also, for example, at least one auxiliary screen corresponds to a screen for outputting a real-time image sensed through the camera provided in the HMD 100 or the external controllers 200a and 200b.

According to an embodiment of the present invention, the first to third auxiliary screens may be independently output on the display unit according to user input. In the present invention, one of the sub screens is a screen for outputting an external image sensed by the camera of the HMD 100, and the other of the sub screens is for outputting an external image sensed by the camera of the first external controller 200a Screen, and the other screen is a screen for outputting an external image sensed by the camera of the second external controller 200b.

Further, in the present invention, at least one sub screen may be output as a sub screen, overlaid on the first screen. Also, the size of the at least one auxiliary screen may be enlarged or reduced, and it is assumed that the size of the at least one auxiliary screen is smaller than the size of the first screen, even if maximized.

3 is a view showing an example of a position and an angle of view of a camera provided in the HMD system according to an embodiment of the present invention.

When only the HMD 100 is provided with an external camera in the HMD system including the HMD 100 and the at least one external controller 200, the HMD 100 displays a real-time image on the sub screen based on the head movement of the user Can be output. At this time, the auxiliary screen may be located at one side of the external controller 200.

However, when the direction of the HMD 100 does not coincide with the direction of the external controller 200, the HMD 100 can not output an image sensed by the camera of the HMD 100 on the display unit. That is, when the external controller 200 is positioned outside the view angle of the camera of the HMD 100 (direction of the user's gaze), the position of the sub screen and the user view point may be inconsistent. Therefore, it becomes necessary to provide a user interface that takes both the user viewpoint and the position of the sub screen into consideration.

The present invention provides a method of outputting an auxiliary screen for sensing an external environment to a user even when the auxiliary screen is not positioned within the viewing angle of the camera of the HMD. That is, as described above with reference to FIG. 2, a camera may be provided not only in the camera of the HMD but also in the external controller, thereby further sensing the external environment. Accordingly, even if the user places the camera of the external controller in a direction different from the direction of the HMD, the external environment can be checked through the auxiliary screen.

In the embodiment of FIG. 3, it is assumed that the first screen 10 corresponds to a case where a screen corresponding to a part of the view angle of the VR contents provided at 360 degrees is output. 3, the HMD system includes an HMD 100 and a first external controller 200a and a second external controller 200b. The view angle of the camera provided in the HMD 100 is a angle, the view angle of the camera provided in the first external controller 200a is b angle, and the view angle of the camera provided in the second external controller 200b is c Corresponds to the angle. Each angle of view may be the same or different.

The external environment sensed by the camera of the first external controller 200a is output to the first sub screen 20a and the external environment sensed by the camera of the second external controller 200b is output to the second sub screen 20b. 3, the first sub-screen 20a is output to the right of the first external controller 200a and the second sub-screen 20b is output to the left of the second external controller 200b, But are not limited to. 3, the external environment sensed by the camera of the HMD 100 may be output to a predetermined area between the first sub-screen 20a and the second sub-screen 20b.

As an example, the embodiment of FIG. 3 (a) corresponds to the case where the direction of the camera of the HMD coincides with the direction of the external controller. In this case, the user confirms the external image sensed through the first external controller 200a through the first sub-screen 20a and outputs the sensed external image through the second external controller 200b to the second sub- 20b.

As another example, the embodiment of FIG. 3 (b) corresponds to a case where the direction of the camera of the HMD is inconsistent with the direction of the external controller. Unlike the prior art in which the camera is provided only in the HMD, in the case of this embodiment, even if the direction of the camera of the HMD and the external controller are inconsistent, the user can see through the first sub screen 20a and the second sub screen 20b It is possible to confirm the area which is not used.

On the other hand, the HMD 100 can output an object corresponding to the external controller on the display unit. In the following embodiments, it is assumed that the objects corresponding to the external controllers 200a and 200b are not displayed, but the objects corresponding to the external controllers 200a and 200b are displayed on the display unit .

Flashlight Camera mode

4 to 8, a flashlight camera mode utilizing the HMD system of the present invention will be described. The flashlight camera mode corresponds to a mode for directly confirming the actual environment beyond the first screen on which the VR contents are outputted by using the camera of the external controller, similarly to a situation where the user directly illuminates the light.

4 is a flowchart illustrating a method of controlling an HMD system according to an embodiment of the present invention.

More specifically, FIG. 4 is a diagram illustrating a data flow between an HMD and at least one external controller included in the system.

As shown in FIG. 4, the HMD and at least one external controller are connected to each other through wired or wireless data communication (S410). At this time, WiFi direct, Bluetooth technology, or the like may be applied for data pairing between the HMD and at least one external controller, but the present invention is not limited thereto.

First, the HMD can output the VR content to the first screen (S420). At this time, the VR contents can be outputted in a full screen on the first screen.

Also, at least one external controller may sense the control input (S430). Here, the control input may correspond to an input to a physical button provided in the external controller by the user. Alternatively, the control input may include an input to a physical button by a user and a motion input of an external controller. Next, at least one external controller can transmit control information to the HMD (S440).

The HMD may control to output a flash screen corresponding to the external environment based on the received control information (S450). Here, the flash screen corresponds to a circular auxiliary screen of predetermined size. In addition, the flash screen may be overlaid on the first screen on which the VR content is output. Through this, the user can search the real environment beyond the VR contents by using the camera of the external controller. In this regard, the embodiments of Figs. 5 to 8 will be described.

5 is a diagram illustrating an example of outputting a flash screen in the HMD system according to an embodiment of the present invention.

5, the user wears the HMD 100, holds the first external controller 200a on the left hand, and holds the second external controller 200b on the right hand. The user may wish to check the external environment beyond the first screen 10 while using the VR content through the first screen 10. [

In this case, at least one external controller may sense the control input to the physical button. For example, as shown in the first drawing of FIG. 5, the second external controller 200b may be turned on with the flashlight mode switch 510 turned on. When the flashlight mode switch is changed to the ON state, the second external controller 200b enters the flashlight mode. In the present invention, in the case of the flashlight mode, it is assumed that they are individually entered according to the physical inputs to the first external controller 200a and the second external controller 200b, respectively. Further, in a state in which the flashlight mode is entered, at least one external controller can sense movement by the user. Also, at least one external controller may transmit information about the sensed control input and motion to the HMD 100. [

In this case, referring to the second drawing of FIG. 5, the HMD 100 may output the flash screen 30 overlaid on the first screen 10 according to control information received from at least one external controller . At this time, the position at which the flash screen 30 is output may correspond to the direction in which the external controller 200b enters the flashlight mode on the first screen 10. That is, the user can easily check the external environment of the desired direction.

In the embodiment of FIG. 5, only the second external controller 200b enters the flashlight mode. Alternatively, when the first external controller 200a changes the flashlight mode switch to the on state, the first external controller 200b enters the flashlight mode together It will be possible. In this case, a plurality of flash screens may be output on the first screen 10. In addition, each of the plurality of flash screens can overlay on the first screen 10 based on the movement of the first external controller 200a and the second external controller 200b.

In the meantime, in FIG. 5, when a separate physical input is received for each of the external controllers, the flashlight mode is separately entered. However, when an external controller receives an input for a physical button, Both external controllers may enter flashlight mode.

Although not shown in FIG. 5, when the flashlight mode switch is turned off by the user, the external controller transmits a control input to the HMD 100, and the HMD 100 receiving the control input controls the flashlight mode to be terminated There will be. That is, the HMD 100 can control the flash screen 30 not to be output.

FIG. 6 and FIG. 7 illustrate another example of outputting a flash screen in the HMD system according to an embodiment of the present invention.

6 and 7 show a flash screen output based on the movement of the external controller in the state of entering the flashlight mode. In the embodiments of FIGS. 6 and 7, the description overlapping with the embodiment of FIG. 5 described above will be omitted. In the embodiment of FIG. 6, it is assumed that the second external controller 200b enters the flashlight mode, and in the embodiment of FIG. 7, the first external controller 200a enters the flashlight mode.

As an example, referring to the first drawing of Fig. 6, the user may position the second external controller 200b, which is in the right hand, to the right lower end state of the first screen 10. [ In this case, the flash screen 30 may be output on the lower right side of the first screen 10. Next, the second external controller 200b can sense the movement to the right in the flashlight mode. In this case, the position of the second external controller 200b may be located outside the view angle of the first screen 10 from which the VR contents are output.

6, when the position of the HMD 100 facing the second external controller 200b is larger than the first screen 10, the HMD 100 may fix the flash screen 30 at the final position And outputting it. Alternatively, when the position of the HMD 100 facing the second external controller 200b exceeds the first screen 10, the HMD 100 may control the flash screen 30 to be fixed and output on a predetermined area. In addition, the HMD 100 can output the real-time external environment sensed through the camera of the second external controller 200b through the fixed flash screen 30. [

As another example, referring to the first drawing of FIG. 7, the user may place the first external controller 200a in the left hand on the bottom left of the first screen. In this case, the flash screen 30 may be output on the lower left side of the first screen 10. Next, the first external controller 200a can sense the leftward movement in the flashlight mode. In this case, the position of the first external controller 200a may be located outside the field of view of the first screen 10 from which the VR contents are output.

7, when the position of the HMD 100 facing the first external controller 200a exceeds the first screen 10, the HMD 100 may fix the flash screen 30 at the final position And outputting it. Alternatively, the HMD 100 may control the flash screen 30 to be fixed and output on a predetermined area, when the position of the first external controller 200a is over the first screen 10. In addition, the HMD 100 may output the external environment sensed through the camera of the first external controller 200a through the fixed flash screen 30. [

8 is a view illustrating another example of outputting a flash screen in the HMD system according to an embodiment of the present invention.

8 shows a case where the external controller is located at the rear of the user rather than when the external controller is moved to the left or right in the embodiments of Figs. 6 and 7 described above. In the embodiment of FIG. 8, the description overlapping with the embodiments of FIGS. 5 to 7 described above will be omitted. In the embodiment of FIG. 8, it is assumed that the second external controller 200b enters the flashlight mode.

8, the user places the second external controller 200b toward the back of the user while the VR contents are output to the first screen 10 and the flash screen 30 is output . 8, when the second external controller 200b is positioned outside the viewing angle of the first screen 10 and behind the user, the HMD 100 may be positioned at the rear side of the first screen 10 And controls the flash screen 30 to be overlaid on the predetermined area. In the embodiment of FIG. 8, the predetermined area may correspond to the right lower end of the first screen 10. For example, when the first external controller 200a enters the flash mode and faces backward beyond the angle of view of the first screen 10, the pre-set area is set to the lower left of the first screen 10 , But is not limited to.

That is, the HMD 100 controls the flash screen 30 to be output on a predetermined area of the first screen 10 even when the external controller is positioned behind the first screen 10 and behind the viewing angle of the first screen 10, Allows the rear controller to see through the camera of the external controller without movement of the body.

Moving, merging and separating multiple secondary screens

9 is a flowchart illustrating a method of controlling an HMD system according to an embodiment of the present invention.

More specifically, FIG. 9 is a diagram illustrating a data flow between an HMD included in the HMD system and at least one external controller.

As shown in FIG. 9, the HMD and at least one external controller are connected to each other through wired or wireless data communication (S910). At this time, WiFi direct, Bluetooth technology, or the like may be applied for data pairing between the HMD and at least one external controller, but the present invention is not limited thereto.

The HMD may output the VR content to the first screen, output the first external image to the first sub screen, and output the second external image to the second sub screen (S920). Here, the VR content is output as a full screen on the first screen. Also, the first external image may correspond to a real-time external environment sensed by a camera provided in the first external controller. The second external image may correspond to a real-time external environment sensed by a camera provided in the second external controller. In addition, the first sub screen and the second sub screen may be overlaid on the first screen.

On the other hand, the output of the first sub screen and the second sub screen, which represent the external environment, correspond to the basic setting of the HMD system or correspond to those output when the user's control input is received. In the present invention, it is assumed that the state in which the first sub-screen and the second sub-screen are outputted is the state in which the sub-screen output mode is entered by the control input of the user.

Also, at least one external controller may detect the control input of the external controller (S930). More specifically, the at least one external controller may sense a movement of a user's hand holding an external controller or a control input to a physical button. Also, at least one external controller may transmit sensed motion information or control information to the HMD (S940).

The HMD may control the first sub screen and the second sub screen based on the received motion information or control information (S950). As an example, the HMD may move the first sub screen and the second sub screen, respectively, based on the received motion information. As another example, the HMD may merge the first and second auxiliary screens based on the received motion information. Further, as another example, the HMD is described in the embodiment of Figs. 9 to 14 in this regard, based on the received control information.

10 is a view showing an example of providing a zoom-in of an auxiliary screen in an HMD system according to an embodiment of the present invention.

In the embodiment of FIG. 10, it is assumed that the first sub screen and the second sub screen are output on the display unit. Also, the first sub-screen outputs an external image sensed by the camera of the first external controller 200a, and the second sub-screen is in the state of outputting an external image sensed by the camera of the second external controller 200b I suppose.

Referring to the first drawing of FIG. 10, the HMD can control to output the first screen 10, the first sub screen 20a, and the second sub screen 20b. In addition, the auxiliary screen 20 may be output to one side of each of the auxiliary screens. For example, in the present invention, the first auxiliary screen 20a is output to the right upper end of the first external controller 200a, and the second auxiliary screen 20b is output to the upper left of the second external controller 200b But not limited to,

Meanwhile, the first external controller 200a and the second external controller 200b may sense a control input to the physical button. In the embodiment of FIG. 10, it is assumed that the control input is an input for zoom-in to each of the first external controller 200a and the second external controller 200b. At this time, it is assumed that the position of each of the external controllers is not changed. In this case, as shown in the second drawing of FIG. 10, the HMD 100 can output an enlarged external image to each of the first sub screen 20a and the second sub screen 20b. At this time, the enlarged area of the external image being output to the sub screen may correspond to the predetermined object. For example, the predetermined object may be set according to a person recognized in the external image, a moving object, or the like.

In contrast to the above, when the control input is a zoom-out input, the HMD 100 controls the camera of the first external controller 200a and the camera of the second external controller 200b, Out image can be output. For example, when the zoom-out input is sensed in the first drawing of FIG. 10, the HMD 100 can output an image of a wider angle of view on the sub screen. This is possible because the external image output on the auxiliary screen is a result of partially cropping the sensed image from the camera of the external controller.

Also, for example, if the zoom-out input is sensed in the second drawing of FIG. 10, the HMD 100 will be able to output the image output from the auxiliary screen of the first drawing of FIG.

Although not shown in FIG. 10, each of the external controllers may move back and forth with respect to the first screen 10 without a control input to the physical buttons of the first external controller 200a and the second external controller 200b Can be sensed. In this case, the HMD 100 may enlarge or reduce the size of each sub screen, depending on the setting.

11 is a view illustrating an example of merging a plurality of sub-screens in an HMD system according to an embodiment of the present invention.

In the embodiment of Fig. 11, it is assumed that the first sub screen and the second sub screen are output on the display unit. In the embodiment shown in FIG. 11, the description overlapping with FIG. 10 is omitted.

Referring to the first drawing of FIG. 11, each of the first external controller 200a and the second external controller 200b can sense motion. For example, the first external controller 200a senses the movement to the right and the second external controller 200b can sense the movement to the left. The first and second external controllers 200a and 200b transmit motion information to the HMD 100 and the HMD 100 controls the first and second subscreens 20a and 20b based on the motion information, It is possible to move the position of the arm 20b.

At this time, as the first and second external controllers 200a and 200b are all directed toward the center of the first screen 10, the view angle area of the camera provided in the first and second external controllers 200a and 200b is partially overlapped Can be lost. When the HMD 100 overlaps the view angle area of the camera provided in the first and second external controllers 200a and 200b by a predetermined ratio, the first sub screen 20a and the second sub screen 20b may be merged to output one sub-screen 50. FIG.

At this time, the external image output to one of the sub-screens 50 may correspond to an area common to the images simultaneously sensed through the camera of the first external controller 200a and the camera of the second external controller 200b. Alternatively, an external image output to one of the sub-screens 50 may correspond to an image sensed by a camera provided in the HMD 100. In addition, a position at which one sub screen 50 is output may correspond to a position between the first external controller 200a and the second external controller 200b.

Accordingly, when the same external environment is sensed at the same time through the cameras of the first external controller 200a and the second external controller 200b, they are merged into one sub screen and output, thereby facilitating the user's recognition . In addition, when each external controller is moved to the center, it can provide intuitive interaction in which the secondary screens are combined into one.

12 is a diagram illustrating an example of holding an auxiliary screen after merging in an HMD system according to an embodiment of the present invention.

In the embodiment of Fig. 12, it is assumed that the first auxiliary screen and the second auxiliary screen in Fig. 11 described above are combined so that one auxiliary screen is output. In the embodiment of FIG. 12, the description overlapping with the embodiment of FIGS. 10 and 11 will be omitted.

12A, the first external controller 200a and the second external controller 200b are in a state in which one sub-screen 50 is output, that is, a state in which the first sub-screen 50 is directed toward the center of the first screen 10 It is possible to sense the movement in the downward direction. Here, the sensed motion is only required to move in the downward direction, and is not limited to the fact that it is perpendicular to the surface of the earth, or moves in a straight line.

In this case, referring to FIG. 12 (b), the HMD 100 can control to hold one sub screen 50 based on the movement of the first and second external controllers 200a and 200b. That is, even when the first or second external controller 200a or 200b moves freely after the movement of the first and second external controllers 200a and 200b, the one auxiliary screen 50 maintains the output state do.

13 is a diagram illustrating an example of dividing one sub screen into a plurality of sub screens in the HMD system according to an embodiment of the present invention.

In the embodiment of FIG. 13, one subscreen assumes that the first subscreen and the second subscreen discussed above in FIG. 11 are merged. In the embodiment of FIG. 13, the contents overlapping with the above-described FIG. 11 and FIG. 12 will be omitted.

Referring to the first diagram of FIG. 13, each of the first external controller 200a and the second external controller 200b can sense motion. For example, the first external controller 200a senses the movement to the left, and the second external controller 200b senses the movement to the right. In other words, the first and second external controllers 200b can sense motions moving away from each other with respect to the center of the first screen 10. [

The first and second external controllers 200a and 200b transmit motion information to the HMD 100 and the HMD 100 transmits the first auxiliary screen 50 to the first auxiliary The screen 20a and the second sub screen 20b.

In this case, although the external image sensed by the camera of the HMD 100 is outputted on one of the sub screens 50, the external images sensed by the external controllers 200a and 200b are displayed on the first and second sub screens 20a and 20b, And outputs an external image sensed by the camera.

FIG. 14 is a view showing an example of outputting an external image sensed by each camera in each of the sub-screens in the HMD system according to an embodiment of the present invention.

In the embodiment of Fig. 14, the contents overlapping with the embodiments of Figs. 11 to 13 are omitted.

First, referring to the first drawing of FIG. 14, the HMD 100 overlays the HMD 100 on the first screen 10 by outputting an HMD sub-screen 60 that outputs an external image sensed by the camera of the HMD 100 . At this time, although not shown in FIG. 14, the HMD sub-screen 60 can be activated by operation on at least one of the first external controller 200a and the second external controller 200b.

Also, each of the first and second external controllers 200a and 200b may sense a control input to the physical button. Here, the control input corresponds to an input for activating output of an external image sensed by the camera provided to each of the first and second external controllers 200a and 200b on the first and second sub screens do.

In this case, referring to the second drawing of FIG. 14, the HMD 100 controls the first sub screen 20a and the second sub screen 20b based on the control information received from the first and second external controllers 200a and 200b, (20b). The first sub screen 20a and the second sub screen 20b may be output at a separate position from the HMD sub screen 60 and overlaid on the first screen 10. For example, as shown in FIG. 14, the first sub-screen 20a and the second sub-screen 20a may be output as circular screens of predetermined sizes, but are not limited thereto.

Thus, the user can utilize the VR contents while sensing the external environment at various angles, regardless of the position of the sight line and the external controller.

15 is a diagram illustrating an example of checking an external environment using an external controller in the HMD system according to an embodiment of the present invention.

15, it is assumed that the physical input to the second external controller 200b is sensed, and that the HMD 100 outputs the auxiliary screen 20. In this case, the external environment sensed by the camera of the second external controller 200b may be output to the sub screen 20. [

The user may want to use the VR contents while monitoring the remaining space other than the indoor space in which the VR contents are being used. For example, a user may want to monitor a living room, a kitchen, a porch, etc. while using VR contents in a study. In this case, the user can input an input to the physical button and mount the second external controller 200b, which is sensing the external environment, at a desired location. On the other hand, it can be used in situations such as watching videos where the necessity of manipulating VR contents is relatively low through an external controller.

Accordingly, the user can easily confirm the VR contents without moving directly to the space requiring monitoring in the space in use.

16 is a diagram illustrating an example of checking an external environment in cooperation with an external device in the HMD system according to an embodiment of the present invention.

16 (a), it is assumed that the HMD system is paired with various devices including a camera such as a CCTV 300a, a smart phone 300b, and an action camera 300c. Here, the action camera 300c may correspond to an ultra-small camcorder which photographs while being attached to a body or equipment. However, the present invention is not limited to this, and the HMD system may perform a pairing with another external device having a camera. WiFi direct, Bluetooth technology, or the like may be applied for data pairing between the HMD system and at least one external device, but the present invention is not limited thereto.

16 (b), the HMD system is in a state in which the physical input to at least one external controller is sensed and the auxiliary screen 20 is output. At this time, the HMD system can receive data from at least one external device connected and output the external environment on the auxiliary screen 20. [ In this case, the external image sensed by the external device, rather than the external image sensed by the camera of the external controller, is output on the sub screen 20. [

For example, when receiving data from the CCTV 300a, the HMD 100 may output the external environment being monitored on the secondary screen 20. [ In addition, for example, when receiving data from the smartphone 300b, the HMD 100 can perform a video call using the auxiliary screen 20. [ Further, for example, when receiving data from the action camera 300c, the HMD 100 may output the outdoor activity image on the auxiliary screen 20. [

Furthermore, although the drawings are shown for convenience of explanation, it is also possible to design a new embodiment to be implemented by merging the embodiments described in each drawing.

Further, the system including the HMD and the control method thereof may not be limitedly applied to the configuration and method of the embodiments described above, but the embodiments may be applied to all or a part of each embodiment May be selectively combined.

The present invention described above can be embodied as computer-readable codes on a medium on which a program is recorded. The computer readable medium includes all kinds of recording devices in which data that can be read by a computer system is stored. Examples of the computer readable medium include a hard disk drive (HDD), a solid state disk (SSD), a silicon disk drive (SDD), a ROM, a RAM, a CD-ROM, a magnetic tape, a floppy disk, , And may also be implemented in the form of a carrier wave (e.g., transmission over the Internet). Also, the computer may include a control unit 180 of the terminal. Accordingly, the above description should not be construed in a limiting sense in all respects and should be considered illustrative. The scope of the present invention should be determined by rational interpretation of the appended claims, and all changes within the scope of equivalents of the present invention are included in the scope of the present invention.

100: head-mounted display 110: wireless communication unit
120: input unit 121: camera
140: sensing unit 150: output unit
160: interface unit 170: memory
180: control unit 190: power supply unit
200a: first external controller 200b: second external controller
210a: first external camera 210b: second external camera

Claims (20)

In an HMD system including a head-mounted display (HMD) and first and second external controllers,
The HMD
A display unit;
A first camera;
A first wireless communication unit for transmitting / receiving data to / from the first and second external controllers; And
And a control unit,
The first external controller
A second camera;
A second wireless communication unit for transmitting / receiving data to / from the HMD; And
And a first sensing unit for sensing movement and input signals of the first external controller,
The second external controller
A third camera;
A third wireless communication unit for transmitting / receiving data with the HMD; And
And a second sensing unit for sensing movement and input signals of the second external controller,
The control unit
A VR (Virtual Reality) content is output to a first screen of the display unit, an external image sensed by the second camera is output to the first sub screen, and an external image sensed by the third camera With the image output,
Receives control information from at least one external controller through the wireless communication unit,
And controls the first sub screen and the second sub screen based on the control information. The method of claim 1, wherein
The at least one external controller
Sensing a first control input for a physical button, transmitting control information based on the first control input to the HMD,
The control unit
Further comprising controlling to output a zoom-in image of at least one of the first sub-screen and the second sub-screen based on the received control information. The method of claim 1, wherein
The first external controller senses a first motion, the second external controller senses a second motion, transmits each first and second motion information to the HMD,
The control unit
And controlling to output the first auxiliary screen and the second auxiliary screen by merging into one auxiliary screen based on the received first and second motion information. The method of claim 3, wherein
Wherein the first motion of the first external controller and the second motion of the second external controller correspond to movements such that the distance between each other decreases. The method of claim 3, wherein
The control unit
And outputting an external image sensed by the first camera provided in the HMD to the one auxiliary screen. The method of claim 3, wherein
The control unit
And outputting the one auxiliary screen when the view angle area of the second camera and the view angle area of the third camera exceed the predetermined ratio. The method of claim 3, wherein
Wherein the first external controller senses a third motion, the second external controller senses a fourth motion, transmits respective third and fourth motion information to the HMD,
The control unit
Further comprising maintaining one sub-screen being output based on the third and fourth motion information received. The method of claim 7, wherein
Wherein the third movement of the first external controller and the fourth movement of the second external controller correspond to movements of the respective external controllers in a predetermined direction. The method of claim 3,
The first external controller senses a fifth motion, the second external controller senses a sixth motion, transmits each fifth and sixth motion information to the HMD,
The control unit
Further comprising outputting one of the outputting auxiliary screens to the first auxiliary screen and the second auxiliary screen again based on the received fifth and sixth motion information. The method of claim 9, wherein
Wherein the fifth motion of the first external controller and the fifth motion of the second external controller correspond to movements to increase the distance between each other. The method of claim 9, wherein
The control unit
Further outputting an external image sensed by the second camera to the first sub screen and outputting an external image sensed by the third camera to the second sub screen. The method of claim 1, wherein
The control unit
Further comprising outputting an HMD sub-screen representing an external image sensed by the camera of the HMD. The method of claim 1, wherein
Wherein the first sub-screen and the second sub-screen correspond to a flash screen. The method of claim 1, wherein
At least one of the plurality of external controllers senses a seventh motion, transmits seventh motion information to the HMD,
The control unit
Further comprising fixing the flash screen to a predetermined area and outputting the flash screen based on the received seventh motion information. The method of claim 14, wherein
And the seventh movement corresponds to a movement in a direction deviating from an angle of view of the first screen. In claim 1,
The control unit
And performing a pairing with the first and second external controllers via the wireless communication unit. The method of claim 1, wherein
Wherein the size of the first screen is greater than the size of the first sub screen and the second sub screen. The method of claim 1, wherein
The first sub screen and the second sub screen are overlaid on the first screen and the first sub screen and the second sub screen are outputted at a distance closer to the HMD than the first screen , HMD system. A head-mounted display (HMD)
A display unit;
camera;
A wireless communication unit for transmitting / receiving data to / from the first and second external controllers; And
And a control unit,
The control unit
And outputs a VR (Virtual Reality) content to a first screen of the display unit and outputs an external image received from the first external controller to a first sub-screen, and outputs an external image received from the second external controller to the second sub- In this state,
Receives control information from at least one external controller through the wireless communication unit,
And controls the first sub screen and the second sub screen based on the control information. A method of controlling an HMD system including a head-mounted display (HMD) and first and second external controllers,
And outputs a VR (Virtual Reality) content to the first screen of the display unit and outputs an external image sensed by the camera of the first external controller to the first sub-screen, Outputting an external image sensed through the sensor;
Receiving control information from at least one external controller via a wireless communication unit; And
And controlling the first sub screen and the second sub screen based on the control information.

Images