Hereinafter, embodiments of the present invention will be described in detail with reference to the accompanying drawings, wherein like or similar components are denoted by the same reference numerals, and redundant explanations thereof will be omitted. The suffix "module" and " part "for the components used in the following description are given or mixed in consideration of ease of specification, and do not have their own meaning or role. In the following description of the embodiments of the present invention, a detailed description of related arts will be omitted when it is determined that the gist of the embodiments disclosed herein may be blurred. It is to be understood that both the foregoing general description and the following detailed description are exemplary and explanatory and are intended to provide further explanation of the invention as claimed. , ≪ / RTI > equivalents, and alternatives.
Terms including ordinals, such as first, second, etc., may be used to describe various elements, but the elements are not limited to these terms. The terms are used only for the purpose of distinguishing one component from another.
It is to be understood that when an element is referred to as being "connected" or "connected" to another element, it may be directly connected or connected to the other element, . On the other hand, when an element is referred to as being "directly connected" or "directly connected" to another element, it should be understood that there are no other elements in between.
The singular expressions include plural expressions unless the context clearly dictates otherwise.
In the present application, the terms "comprises", "having", and the like are used to specify that a feature, a number, a step, an operation, an element, a component, But do not preclude the presence or addition of one or more other features, integers, steps, operations, elements, components, or combinations thereof.
The mobile terminal described in this specification includes a mobile phone, a smart phone, a laptop computer, a digital broadcasting terminal, a personal digital assistant (PDA), a portable multimedia player (PMP), a navigation device, a slate PC A tablet PC, an ultrabook, a wearable device such as a smartwatch, a smart glass, and a head mounted display (HMD). have.
The mobile terminal described in this specification includes a mobile phone, a smart phone, a laptop computer, a digital broadcasting terminal, a personal digital assistant (PDA), a portable multimedia player (PMP), a navigation device, a slate PC A tablet PC, an ultrabook, a wearable device such as a smartwatch, a smart glass, and a head mounted display (HMD). have.
However, it will be appreciated by those skilled in the art that the configuration according to the embodiments described herein may be applied to fixed terminals such as a digital TV, a desktop computer, a digital signage, and the like, will be.
1A to 1C are block diagrams for explaining a mobile terminal according to the present invention, and FIGS. 1B and 1C are conceptual diagrams showing an example of a mobile terminal according to the present invention in different directions.
The mobile terminal 100 includes a wireless communication unit 110, an input unit 120, a sensing unit 140, an output unit 150, an interface unit 160, a memory 170, a control unit 180, ), And the like. The components shown in FIG. 1A are not essential for implementing a mobile terminal, so that the mobile terminal described herein may have more or fewer components than the components listed above.
The wireless communication unit 110 may be connected between the mobile terminal 100 and the wireless communication system or between the mobile terminal 100 and another mobile terminal 100 or between the mobile terminal 100 and the external server 100. [ Lt; RTI ID = 0.0 > wireless < / RTI > In addition, the wireless communication unit 110 may include one or more modules for connecting the mobile terminal 100 to one or more networks.
The wireless communication unit 110 may include at least one of a broadcast receiving module 111, a mobile communication module 112, a wireless Internet module 113, a short distance communication module 114, and a location information module 115 .
The input unit 120 includes a camera 121 or an image input unit for inputting a video signal, a microphone 122 for inputting an audio signal, an audio input unit, a user input unit 123 for receiving information from a user A touch key, a mechanical key, and the like). The voice data or image data collected by the input unit 120 may be analyzed and processed by a user's control command.
In the mobile terminal according to the present invention, the camera 121 may include a 3D camera or a depth camera. In the present invention, it is possible to detect a human body located in an arbitrary space (or a predetermined space) through which a hologram image (or a hologram object) is output through an image sensor provided in a 3D camera have. Such an image sensor may be configured to sense a user's human body while sensing a user's gesture in the arbitrary space. In the mobile terminal according to the present invention, a 3D image (or a three-dimensional (3D) image) having a three-dimensional effect corresponding to a target object (or a human body) approaching the hologram image can be acquired through the 3D camera.
The sensing unit 140 may include at least one sensor for sensing at least one of information in the mobile terminal, surrounding environment information surrounding the mobile terminal, and user information. For example, the sensing unit 140 may include a proximity sensor 141, an illumination sensor 142, a touch sensor, an acceleration sensor, a magnetic sensor, A G-sensor, a gyroscope sensor, a motion sensor, an RGB sensor, an infrared sensor, a finger scan sensor, an ultrasonic sensor, A microphone 226, a battery gauge, an environmental sensor (for example, a barometer, a hygrometer, a thermometer, a radiation detection sensor, A thermal sensor, a gas sensor, etc.), a chemical sensor (e.g., an electronic nose, a healthcare sensor, a biometric sensor, etc.). Meanwhile, the mobile terminal disclosed in the present specification can combine and utilize information sensed by at least two of the sensors.
The output unit 150 includes a display unit 151, an acoustic output unit 152, a haptrip module 153, a light output unit 154, and a hologram output unit (not shown) Or a holography module 155). The display unit 151 may have a mutual layer structure with the touch sensor or may be integrally formed to realize a touch screen. The touch screen may function as a user input unit 123 that provides an input interface between the mobile terminal 100 and a user and may provide an output interface between the mobile terminal 100 and a user.
The interface unit 160 serves as a path to various types of external devices connected to the mobile terminal 100. The interface unit 160 is connected to a device having a wired / wireless headset port, an external charger port, a wired / wireless data port, a memory card port, And may include at least one of a port, an audio I / O port, a video I / O port, and an earphone port. In the mobile terminal 100, corresponding to the connection of the external device to the interface unit 160, it is possible to perform appropriate control related to the connected external device.
In addition, the memory 170 stores data supporting various functions of the mobile terminal 100. The memory 170 may store a plurality of application programs or applications running on the mobile terminal 100, data for operation of the mobile terminal 100, and commands. At least some of these applications may be downloaded from an external server via wireless communication. Also, at least a part of these application programs may exist on the mobile terminal 100 from the time of shipment for the basic functions (e.g., telephone call receiving function, message receiving function, and calling function) of the mobile terminal 100. Meanwhile, the application program may be stored in the memory 170, installed on the mobile terminal 100, and may be operated by the control unit 180 to perform the operation (or function) of the mobile terminal.
In addition to the operations related to the application program, the control unit 180 typically controls the overall operation of the mobile terminal 100. The control unit 180 may process or process signals, data, information, and the like input or output through the above-mentioned components, or may drive an application program stored in the memory 170 to provide or process appropriate information or functions to the user.
In addition, the controller 180 may control at least some of the components illustrated in FIG. 1A in order to drive an application program stored in the memory 170. FIG. In addition, the controller 180 may operate at least two of the components included in the mobile terminal 100 in combination with each other for driving the application program.
Further, the memory 170 may store information on the hologram interference pattern to support the projection of the hologram image of the hologram output unit 155. [ The controller 180 can output information visually output from the mobile terminal as a hologram image through the hologram output unit 155 using the information stored in the memory 170. [
The power supply unit 190 receives external power and internal power under the control of the controller 180 and supplies power to the components included in the mobile terminal 100. The power supply unit 190 includes a battery, which may be an internal battery or a replaceable battery.
At least some of the components may operate in cooperation with one another to implement a method of operation, control, or control of a mobile terminal according to various embodiments described below. In addition, the operation, control, or control method of the mobile terminal may be implemented on the mobile terminal by driving at least one application program stored in the memory 170. [
Hereinafter, the various components of the mobile terminal 100 will be described in detail with reference to FIG. 1A.
First, referring to the wireless communication unit 110, the broadcast receiving module 111 of the wireless communication unit 110 receives broadcast signals and / or broadcast-related information from an external broadcast management server through a broadcast channel. The broadcast channel may include a satellite channel and a terrestrial channel. Two or more broadcast receiving modules may be provided to the mobile terminal 100 for simultaneous broadcast reception or broadcast channel switching for at least two broadcast channels.
The mobile communication module 112 may be a mobile communication module or a mobile communication module such as a mobile communication module or a mobile communication module that uses technology standards or a communication method (e.g., Global System for Mobile communication (GSM), Code Division Multi Access (CDMA), Code Division Multi Access 2000 (Enhanced Voice-Data Optimized or Enhanced Voice-Data Only), Wideband CDMA (WCDMA), High Speed Downlink Packet Access (HSDPA), High Speed Uplink Packet Access (HSUPA), Long Term Evolution And an external terminal, or a server on a mobile communication network established according to a long term evolution (AR), a long term evolution (AR), or the like.
The wireless signal may include various types of data depending on a voice call signal, a video call signal or a text / multimedia message transmission / reception.
The wireless Internet module 113 is a module for wireless Internet access, and may be built in or externally attached to the mobile terminal 100. The wireless Internet module 113 is configured to transmit and receive a wireless signal in a communication network according to wireless Internet technologies.
Wireless Internet technologies include, for example, wireless LAN (WLAN), wireless fidelity (Wi-Fi), wireless fidelity (Wi-Fi) Direct, DLNA (Digital Living Network Alliance), WiBro Interoperability for Microwave Access, High Speed Downlink Packet Access (HSDPA), High Speed Uplink Packet Access (HSUPA), Long Term Evolution (LTE) and Long Term Evolution-Advanced (LTE-A) 113 transmit and receive data according to at least one wireless Internet technology, including Internet technologies not listed above.
The wireless Internet module 113 for performing a wireless Internet connection through the mobile communication network can be used for wireless Internet access by WiBro, HSDPA, HSUPA, GSM, CDMA, WCDMA, LTE or LTE- May be understood as a kind of the mobile communication module 112.
The short-range communication module 114 is for short-range communication, and includes Bluetooth ™, Radio Frequency Identification (RFID), Infrared Data Association (IrDA), Ultra Wideband (UWB) (Near Field Communication), Wi-Fi (Wireless-Fidelity), Wi-Fi Direct, and Wireless USB (Wireless Universal Serial Bus) technology. The short-range communication module 114 is connected to the mobile terminal 100 and the wireless communication system through the wireless area networks, between the mobile terminal 100 and another mobile terminal 100, or between the mobile terminal 100 ) And the other mobile terminal 100 (or the external server). The short-range wireless communication network may be a short-range wireless personal area network.
Here, the other mobile terminal 100 may be a wearable device (e.g., a smartwatch, a smart glass, etc.) capable of interchanging data with the mobile terminal 100 according to the present invention (smart glass), HMD (head mounted display)). The short range communication module 114 may detect (or recognize) a wearable device capable of communicating with the mobile terminal 100 around the mobile terminal 100. [ If the detected wearable device is a device authenticated to communicate with the mobile terminal 100 according to the present invention, the control unit 180 may transmit at least a part of the data processed by the mobile terminal 100 to the short- 114 to the wearable device. Therefore, the user of the wearable device can use the data processed by the mobile terminal 100 through the wearable device. For example, according to this, when a telephone is received in the mobile terminal 100, the user performs a telephone conversation via the wearable device, or when a message is received in the mobile terminal 100, It is possible to check the message.
The position information module 115 is a module for obtaining the position (or current position) of the mobile terminal, and a representative example thereof is a Global Positioning System (GPS) module or a Wireless Fidelity (WiFi) module. For example, when the mobile terminal utilizes the GPS module, it can acquire the position of the mobile terminal by using a signal transmitted from the GPS satellite. As another example, when the mobile terminal utilizes the Wi-Fi module, it can acquire the position of the mobile terminal based on information of a wireless access point (AP) that transmits or receives the wireless signal with the Wi-Fi module. Optionally, the location information module 115 may perform any of the other functions of the wireless communication unit 110 to obtain data relating to the location of the mobile terminal, in addition or alternatively. The location information module 115 is a module used to obtain the location (or current location) of the mobile terminal, and is not limited to a module that directly calculates or obtains the location of the mobile terminal.
Next, the input unit 120 is for inputting image information (or signal), audio information (or signal), data, or information input from a user. For inputting image information, Or a plurality of cameras 121 may be provided. The camera 121 processes image frames such as still images or moving images obtained by the image sensor in the video communication mode or the photographing mode. The processed image frame may be displayed on the display unit 151 or stored in the memory 170. [ A plurality of cameras 121 provided in the mobile terminal 100 may be arranged to have a matrix structure and various angles or foci may be provided to the mobile terminal 100 through the camera 121 having the matrix structure A plurality of pieces of image information can be input. In addition, the plurality of cameras 121 may be arranged in a stereo structure to acquire a left image and a right image for realizing a stereoscopic image.
The microphone 122 processes the external acoustic signal into electrical voice data. The processed voice data can be utilized variously according to a function (or a running application program) being executed in the mobile terminal 100. Meanwhile, the microphone 122 may be implemented with various noise reduction algorithms for eliminating noise generated in receiving an external sound signal.
The user input unit 123 is for receiving information from a user and when the information is inputted through the user input unit 123, the control unit 180 can control the operation of the mobile terminal 100 to correspond to the input information . The user input unit 123 may include a mechanical input means (or a mechanical key such as a button located on the front, rear or side of the mobile terminal 100, a dome switch, a jog wheel, Jog switches, etc.) and touch-type input means. For example, the touch-type input means may comprise a virtual key, a soft key or a visual key displayed on the touch screen through software processing, And a touch key disposed on the touch panel. Meanwhile, the virtual key or the visual key can be displayed on a touch screen having various forms, for example, a graphic, a text, an icon, a video, As shown in FIG.
Meanwhile, the sensing unit 140 senses at least one of information in the mobile terminal, surrounding environment information surrounding the mobile terminal, and user information, and generates a corresponding sensing signal. The control unit 180 may control the driving or operation of the mobile terminal 100 or may perform data processing, function or operation related to the application program installed in the mobile terminal 100 based on the sensing signal. Representative sensors among various sensors that may be included in the sensing unit 140 will be described in more detail.
First, the proximity sensor 141 refers to a sensor that detects the presence of an object approaching a predetermined detection surface, or the presence of an object in the vicinity of the detection surface, without mechanical contact by using electromagnetic force or infrared rays. The proximity sensor 141 may be disposed in the inner area of the mobile terminal or in proximity to the touch screen, which is covered by the touch screen.
Examples of the proximity sensor 141 include a transmission type photoelectric sensor, a direct reflection type photoelectric sensor, a mirror reflection type photoelectric sensor, a high frequency oscillation type proximity sensor, a capacitive proximity sensor, a magnetic proximity sensor, and an infrared proximity sensor. In the case where the touch screen is electrostatic, the proximity sensor 141 can be configured to detect the proximity of the object with a change of the electric field along the proximity of the object having conductivity. In this case, the touch screen (or touch sensor) itself may be classified as a proximity sensor.
On the other hand, for convenience of explanation, the act of recognizing that the object is located on the touch screen in proximity with no object touching the touch screen is referred to as "proximity touch & The act of actually touching an object on the screen is called a "contact touch. &Quot; The position at which the object is closely touched on the touch screen means a position where the object corresponds to the touch screen vertically when the object is touched. The proximity sensor 141 can detect a proximity touch and a proximity touch pattern (e.g., a proximity touch distance, a proximity touch direction, a proximity touch speed, a proximity touch time, a proximity touch position, have. Meanwhile, the control unit 180 processes data (or information) corresponding to the proximity touch operation and the proximity touch pattern sensed through the proximity sensor 141 as described above, and further provides visual information corresponding to the processed data It can be output on the touch screen. Furthermore, the control unit 180 can control the mobile terminal 100 such that different operations or data (or information) are processed according to whether the touch to the same point on the touch screen is a proximity touch or a touch touch .
The touch sensor senses a touch (or touch input) applied to the touch screen (or the display unit 151) by using at least one of various touch methods such as a resistance film type, a capacitive type, an infrared type, an ultrasonic type, do.
For example, the touch sensor may be configured to convert a change in a pressure applied to a specific portion of the touch screen or a capacitance generated in a specific portion to an electrical input signal. The touch sensor may be configured to detect a position, an area, a pressure at the time of touch, a capacitance at the time of touch, and the like where a touch object touching the touch screen is touched on the touch sensor. Here, the touch object may be a finger, a touch pen, a stylus pen, a pointer, or the like as an object to which a touch is applied to the touch sensor.
Thus, when there is a touch input to the touch sensor, the corresponding signal (s) is sent to the touch controller. The touch controller processes the signal (s) and transmits the corresponding data to the controller 180. Thus, the control unit 180 can know which area of the display unit 151 is touched or the like. Here, the touch controller may be a separate component from the control unit 180, and may be the control unit 180 itself.
On the other hand, the control unit 180 may perform different controls or perform the same control according to the type of the touch object touching the touch screen (or a touch key provided on the touch screen). Whether to perform different controls or to perform the same control according to the type of the touch object may be determined according to the current state of the mobile terminal 100 or an application program being executed.
On the other hand, the touch sensors and the proximity sensors discussed above can be used independently or in combination to provide a short touch (touch), a long touch, a multi touch, a drag touch ), Flick touch, pinch-in touch, pinch-out touch, swipe touch, hovering touch, and the like. Touch can be sensed.
The ultrasonic sensor can recognize the position information of the object to be sensed by using ultrasonic waves. Meanwhile, the controller 180 can calculate the position of the wave generating source through the information sensed by the optical sensor and the plurality of ultrasonic sensors. The position of the wave source can be calculated using the fact that the light is much faster than the ultrasonic wave, that is, the time when the light reaches the optical sensor is much faster than the time the ultrasonic wave reaches the ultrasonic sensor. More specifically, the position of the wave generating source can be calculated using the time difference with the time when the ultrasonic wave reaches the reference signal.
The camera 121 includes at least one of a camera sensor (for example, a CCD, a CMOS, etc.), a photo sensor (or an image sensor), and a laser sensor.
The camera 121 and the laser sensor may be combined with each other to sense a touch of the sensing object with respect to the three-dimensional stereoscopic image. The photosensor can be laminated to the display element, which is adapted to scan the movement of the object to be detected proximate to the touch screen. More specifically, the photosensor mounts photo diodes and TRs (Transistors) in a row / column and scans the contents loaded on the photosensor using an electrical signal that varies according to the amount of light applied to the photo diode. That is, the photo sensor performs coordinate calculation of the object to be sensed according to the amount of change of light, and position information of the object to be sensed can be obtained through the calculation.
The display unit 151 displays (outputs) information processed by the mobile terminal 100. For example, the display unit 151 may display execution screen information of an application program driven by the mobile terminal 100 or UI (User Interface) and GUI (Graphic User Interface) information according to the execution screen information .
Also, the display unit 151 may be configured as a stereoscopic display unit for displaying a stereoscopic image.
In the stereoscopic display unit, a three-dimensional display system such as a stereoscopic system (glasses system), an autostereoscopic system (no-glasses system), and a projection system (holographic system) can be applied.
The sound output unit 152 may output audio data received from the wireless communication unit 110 or stored in the memory 170 in a call signal reception mode, a call mode or a recording mode, a voice recognition mode, a broadcast reception mode, The sound output unit 152 also outputs sound signals related to functions (e.g., call signal reception sound, message reception sound, etc.) performed in the mobile terminal 100. [ The audio output unit 152 may include a receiver, a speaker, a buzzer, and the like.
The haptic module 153 generates various tactile effects that the user can feel. A typical example of the haptic effect generated by the haptic module 153 may be vibration. The intensity and pattern of the vibration generated in the haptic module 153 can be controlled by the user's selection or the setting of the control unit. For example, the haptic module 153 may synthesize and output different vibrations or sequentially output the vibrations.
In addition to vibration, the haptic module 153 may be configured to perform various functions such as a pin arrangement vertically moving with respect to the contact skin surface, a spraying force or suction force of the air through the injection port or the suction port, a touch on the skin surface, And various tactile effects such as an effect of reproducing a cold sensation using an endothermic or exothermic element can be generated.
The haptic module 153 can transmit the tactile effect through the direct contact, and the tactile effect can be felt by the user through the muscles of the finger or arm. The haptic module 153 may include two or more haptic modules 153 according to the configuration of the mobile terminal 100.
The light output unit 154 outputs a signal for notifying the occurrence of an event using the light of the light source of the mobile terminal 100. Examples of events that occur in the mobile terminal 100 may include message reception, call signal reception, missed call, alarm, schedule notification, email reception, information reception through an application, and the like.
The signal output from the light output unit 154 is implemented as the mobile terminal emits light of a single color or a plurality of colors to the front or rear surface. The signal output may be terminated by the mobile terminal detecting the event confirmation of the user.
Next, the hologram output unit (or holography module 155) is configured to output a hologram image in a predetermined space. The structure of the holographic module 155 and the projection method of the holographic image using the same will be described in detail with reference to FIGS. 2A to 5B.
The interface unit 160 serves as a path for communication with all external devices connected to the mobile terminal 100. The interface unit 160 receives data from an external device or supplies power to each component in the mobile terminal 100 or transmits data in the mobile terminal 100 to an external device. For example, a port for connecting a device equipped with a wired / wireless headset port, an external charger port, a wired / wireless data port, a memory card port, an audio I / O port, a video I / O port, an earphone port, and the like may be included in the interface unit 160.
The identification module is a chip for storing various information for authenticating the use right of the mobile terminal 100 and includes a user identification module (UIM), a subscriber identity module (SIM) A universal subscriber identity module (USIM), and the like. Devices with identification modules (hereinafter referred to as "identification devices") can be manufactured in a smart card format. Accordingly, the identification device can be connected to the terminal 100 through the interface unit 160. [
The interface unit 160 may be a path through which power from the cradle is supplied to the mobile terminal 100 when the mobile terminal 100 is connected to an external cradle, And various command signals may be transmitted to the mobile terminal 100. The various command signals or the power source input from the cradle may be operated as a signal for recognizing that the mobile terminal 100 is correctly mounted on the cradle.
The memory 170 may store a program for the operation of the controller 180 and temporarily store input / output data (e.g., a phone book, a message, a still image, a moving picture, etc.). The memory 170 may store data on vibration and sound of various patterns outputted when a touch is input on the touch screen.
The memory 170 may be a flash memory type, a hard disk type, a solid state disk type, an SDD type (Silicon Disk Drive type), a multimedia card micro type ), Card type memory (e.g., SD or XD memory), random access memory (RAM), static random access memory (SRAM), read-only memory (ROM), electrically erasable programmable read memory, a programmable read-only memory (PROM), a magnetic memory, a magnetic disk, and / or an optical disk. The mobile terminal 100 may operate in association with a web storage that performs the storage function of the memory 170 on the Internet.
Meanwhile, as described above, the control unit 180 controls the operations related to the application program and the general operation of the mobile terminal 100. [ For example, when the state of the mobile terminal meets a set condition, the control unit 180 can execute or release a lock state for restricting input of a user's control command to applications.
In addition, the control unit 180 performs control and processing related to voice communication, data communication, video call, or the like, or performs pattern recognition processing to recognize handwriting input or drawing input performed on the touch screen as characters and images, respectively . Further, the controller 180 may control any one or a plurality of the above-described components in order to implement various embodiments described below on the mobile terminal 100 according to the present invention.
The power supply unit 190 receives external power and internal power under the control of the controller 180 and supplies power necessary for operation of the respective components. The power supply unit 190 includes a battery, the battery may be an internal battery configured to be chargeable, and may be detachably coupled to the terminal body for charging or the like.
In addition, the power supply unit 190 may include a connection port, and the connection port may be configured as an example of an interface 160 through which an external charger for supplying power for charging the battery is electrically connected.
As another example, the power supply unit 190 may be configured to charge the battery in a wireless manner without using the connection port. In this case, the power supply unit 190 may use at least one of an inductive coupling method based on a magnetic induction phenomenon from an external wireless power transmission apparatus and a magnetic resonance coupling method based on an electromagnetic resonance phenomenon Power can be delivered.
In the following, various embodiments may be embodied in a recording medium readable by a computer or similar device using, for example, software, hardware, or a combination thereof.
Referring to FIGS. 1B and 1C, the disclosed mobile terminal 100 includes a bar-shaped terminal body. However, the present invention is not limited thereto and can be applied to various structures such as a folder type, a flip type, a slide type, a swing type, and a swivel type in which a watch type, a clip type, a glass type or two or more bodies are relatively movably coupled . A description of a particular type of mobile terminal, although relevant to a particular type of mobile terminal, is generally applicable to other types of mobile terminals.
Here, the terminal body can be understood as a concept of referring to the mobile terminal 100 as at least one aggregate.
The mobile terminal 100 includes a case (for example, a frame, a housing, a cover, and the like) that forms an appearance. As shown, the mobile terminal 100 may include a front case 101 and a rear case 102. Various electronic components are disposed in the inner space formed by the combination of the front case 101 and the rear case 102. At least one middle case may be additionally disposed between the front case 101 and the rear case 102.
A display unit 151 is disposed on a front surface of the terminal body to output information. The window 151a of the display unit 151 may be mounted on the front case 101 to form a front surface of the terminal body together with the front case 101. [
In some cases, electronic components may also be mounted on the rear case 102. Electronic parts that can be mounted on the rear case 102 include detachable batteries, an identification module, a memory card, and the like. In this case, a rear cover 103 for covering the mounted electronic components can be detachably coupled to the rear case 102. Therefore, when the rear cover 103 is separated from the rear case 102, the electronic parts mounted on the rear case 102 are exposed to the outside.
As shown, when the rear cover 103 is coupled to the rear case 102, a side portion of the rear case 102 can be exposed. In some cases, the rear case 102 may be completely covered by the rear cover 103 during the engagement. Meanwhile, the rear cover 103 may be provided with an opening for exposing the camera 121b and the sound output unit 152b to the outside.
These cases 101, 102, and 103 may be formed by injection molding of synthetic resin or may be formed of metal such as stainless steel (STS), aluminum (Al), titanium (Ti), or the like.
The mobile terminal 100 may be configured such that one case provides the internal space, unlike the above example in which a plurality of cases provide an internal space for accommodating various electronic components. In this case, a universal mobile terminal 100 in which a synthetic resin or metal is connected from the side to the rear side can be realized.
Meanwhile, the mobile terminal 100 may include a waterproof unit (not shown) for preventing water from penetrating into the terminal body. For example, the waterproof portion is provided between the window 151a and the front case 101, between the front case 101 and the rear case 102, or between the rear case 102 and the rear cover 103, And a waterproof member for sealing the inside space of the oven.
The mobile terminal 100 is provided with a display unit 151, first and second sound output units 152a and 152b, a proximity sensor 141, an illuminance sensor 142, a light output unit 154, Cameras 121a and 121b, first and second operation units 123a and 123b, a microphone 122, an interface unit 160, and the like.
1B and 1C, a display unit 151, a first sound output unit 152a, a proximity sensor 141, an illuminance sensor 142, an optical output unit (not shown) A second operation unit 123b, a microphone 122 and an interface unit 160 are disposed on a side surface of the terminal body, And a mobile terminal 100 having a second sound output unit 152b and a second camera 121b disposed on a rear surface thereof.
However, these configurations are not limited to this arrangement. These configurations may be excluded or replaced as needed, or placed on different planes. For example, the first operation unit 123a may not be provided on the front surface of the terminal body, and the second sound output unit 152b may be provided on the side surface of the terminal body rather than the rear surface of the terminal body.
The display unit 151 displays (outputs) information processed by the mobile terminal 100. For example, the display unit 151 may display execution screen information of an application program driven by the mobile terminal 100 or UI (User Interface) and GUI (Graphic User Interface) information according to the execution screen information .
The display unit 151 may be a liquid crystal display (LCD), a thin film transistor-liquid crystal display (TFT LCD), an organic light-emitting diode (OLED), a flexible display display, a 3D display, and an e-ink display.
In addition, the display unit 151 may exist in two or more depending on the embodiment of the mobile terminal 100. In this case, the mobile terminal 100 may be provided with a plurality of display portions spaced apart from each other or disposed integrally with one another, or may be disposed on different surfaces, respectively.
The display unit 151 may include a touch sensor that senses a touch with respect to the display unit 151 so that a control command can be received by a touch method. When a touch is made to the display unit 151, the touch sensor senses the touch, and the control unit 180 generates a control command corresponding to the touch based on the touch. The content input by the touch method may be a letter or a number, an instruction in various modes, a menu item which can be designated, and the like.
The touch sensor may be a film having a touch pattern and disposed between the window 151a and a display (not shown) on the rear surface of the window 151a, or may be a metal wire . Alternatively, the touch sensor may be formed integrally with the display. For example, the touch sensor may be disposed on a substrate of the display or inside the display.
In this way, the display unit 151 can form a touch screen together with the touch sensor. In this case, the touch screen can function as a user input unit 123 (see FIG. 1A). In some cases, the touch screen may replace at least some functions of the first operation unit 123a.
The first sound output unit 152a may be implemented as a receiver for transmitting a call sound to a user's ear and the second sound output unit 152b may be implemented as a loud speaker for outputting various alarm sounds or multimedia playback sounds. ). ≪ / RTI >
The window 151a of the display unit 151 may be provided with an acoustic hole for emitting the sound generated from the first acoustic output unit 152a. However, the present invention is not limited to this, and the sound may be configured to be emitted along an assembly gap (for example, a gap between the window 151a and the front case 101) between the structures. In this case, the appearance of the mobile terminal 100 can be made more simple because the hole formed independently for the apparent acoustic output is hidden or hidden.
The optical output unit 154 is configured to output light for notifying the occurrence of an event. Examples of the event include a message reception, a call signal reception, a missed call, an alarm, a schedule notification, an email reception, and reception of information through an application. The control unit 180 may control the light output unit 154 to terminate the light output when the event confirmation of the user is detected.
Furthermore, a hologram output unit 155 may be provided in the terminal body. The hollogram output unit 155 may be configured to output a hologram image 155 '(see FIG. 2A) to a space on the front surface of the terminal body, for example, the display unit 151. In this figure, it is illustrated that the hologram output unit 155 is provided in one area of the display unit 151. In the mobile terminal according to the present invention, as shown in the drawing, the hologram output unit 155 may be arranged not only in one area of the display unit 151 but also in an arbitrary area of the main body of the terminal, The position can be changed variously according to the design.
The first camera 121a processes an image frame of a still image or a moving image obtained by the image sensor in the photographing mode or the video communication mode. The processed image frame can be displayed on the display unit 151 and can be stored in the memory 170. [
The first and second operation units 123a and 123b may be collectively referred to as a manipulating portion as an example of a user input unit 123 operated to receive a command for controlling the operation of the mobile terminal 100 have. The first and second operation units 123a and 123b can be employed in any manner as long as the user is in a tactile manner such as touch, push, scroll, or the like. In addition, the first and second operation units 123a and 123b may be employed in a manner that the user operates the apparatus without touching the user through a proximity touch, a hovering touch, or the like.
In this figure, the first operation unit 123a is a touch key, but the present invention is not limited thereto. For example, the first operation unit 123a may be a mechanical key, or a combination of a touch key and a touch key.
The contents input by the first and second operation units 123a and 123b can be variously set. For example, the first operation unit 123a receives a command such as a menu, a home key, a cancellation, a search, and the like, and the second operation unit 123b receives a command from the first or second sound output unit 152a or 152b The size of the sound, and the change of the display unit 151 to the touch recognition mode.
On the other hand, a rear input unit (not shown) may be provided on the rear surface of the terminal body as another example of the user input unit 123. The rear input unit is operated to receive a command for controlling the operation of the mobile terminal 100, and input contents may be variously set. For example, commands such as power on / off, start, end, scrolling, and the like, the size adjustment of the sound output from the first and second sound output units 152a and 152b, And the like can be inputted. The rear input unit may be implemented as a touch input, a push input, or a combination thereof.
The rear input unit may be disposed so as to overlap with the front display unit 151 in the thickness direction of the terminal body. For example, the rear input unit may be disposed at the rear upper end of the terminal body such that when the user holds the terminal body with one hand, the rear input unit can be easily operated using the index finger. However, the present invention is not limited thereto, and the position of the rear input unit may be changed.
When a rear input unit is provided on the rear surface of the terminal body, a new type of user interface using the rear input unit can be realized. When the first operation unit 123a is not disposed on the front surface of the terminal body in place of at least a part of the functions of the first operation unit 123a provided on the front surface of the terminal body, The display unit 151 may be configured as a larger screen.
Meanwhile, the mobile terminal 100 may be provided with a fingerprint recognition sensor for recognizing the fingerprint of the user, and the controller 180 may use the fingerprint information sensed through the fingerprint recognition sensor as authentication means. The fingerprint recognition sensor may be embedded in the display unit 151 or the user input unit 123.
The microphone 122 is configured to receive the user's voice, other sounds, and the like. The microphone 122 may be provided at a plurality of locations to receive stereophonic sound.
On the other hand, the hologram output unit 155 may be provided on the rear surface of the terminal body to output the hologram image 155 '(see FIG. In the mobile terminal according to the present invention, the hologram output unit 155 may be provided on at least one of the front surface, the side surface, and the rear surface of the main body.
The interface unit 160 is a path through which the mobile terminal 100 can be connected to an external device. For example, the interface unit 160 may include a connection terminal for connection with another device (for example, an earphone or an external speaker), a port for short-range communication (for example, an infrared port (IrDA Port), a Bluetooth port A wireless LAN port, or the like), or a power supply terminal for supplying power to the mobile terminal 100. The interface unit 160 may be implemented as a socket for receiving an external card such as a SIM (Subscriber Identification Module) or a UIM (User Identity Module) or a memory card for storing information.
And a second camera 121b may be disposed on a rear surface of the terminal body. In this case, the second camera 121b has a photographing direction which is substantially opposite to that of the first camera 121a.
The second camera 121b may include a plurality of lenses arranged along at least one line. The plurality of lenses may be arranged in a matrix form. Such a camera can be named an 'array camera'. When the second camera 121b is configured as an array camera, images can be taken in various ways using a plurality of lenses, and a better quality image can be obtained.
The flash 124 may be disposed adjacent to the second camera 121b. The flash 124 shines light toward the subject when the subject is photographed by the second camera 121b.
And a second sound output unit 152b may be additionally disposed in the terminal body. The second sound output unit 152b may implement a stereo function together with the first sound output unit 152a and may be used for implementing a speakerphone mode in a call.
The terminal body may be provided with at least one antenna for wireless communication. The antenna may be embedded in the terminal body or formed in the case. For example, an antenna constituting a part of the broadcast receiving module 111 (see FIG. 1A) may be configured to be able to be drawn out from the terminal body. Alternatively, the antenna may be formed in a film type and attached to the inner surface of the rear cover 103, or a case including a conductive material may be configured to function as an antenna.
The terminal body is provided with a power supply unit 190 (see FIG. 1A) for supplying power to the mobile terminal 100. The power supply unit 190 may include a battery 191 built in the terminal body or detachable from the outside of the terminal body.
The battery 191 may be configured to receive power through a power cable connected to the interface unit 160. In addition, the battery 191 may be configured to be wirelessly chargeable through a wireless charger. The wireless charging may be implemented by a magnetic induction method or a resonance method (magnetic resonance method).
The rear cover 103 is configured to be coupled to the rear case 102 so as to cover the battery 191 to restrict the release of the battery 191 and to protect the battery 191 from external impact and foreign matter . When the battery 191 is detachably attached to the terminal body, the rear cover 103 may be detachably coupled to the rear case 102.
The mobile terminal 100 may be provided with an accessory that protects the appearance or supports or expands the function of the mobile terminal 100. [ One example of such an accessory is a cover or pouch that covers or accommodates at least one side of the mobile terminal 100. [ The cover or pouch may be configured to interlock with the display unit 151 to expand the function of the mobile terminal 100. Another example of an accessory is a touch pen for supplementing or extending a touch input to the touch screen.
Hereinafter, the principle and method of outputting the hologram image through the hologram output unit 155 will be further described with reference to the accompanying drawings. FIGS. 2A and 2B are conceptual diagrams illustrating the implementation of the hologram image through the hologram output unit, and FIGS. 3A and 3B are conceptual diagrams illustrating the principle of the hologram. 4A to 4C are conceptual diagrams for explaining a transmission type hologram system, and FIGS. 5A to 5C are conceptual diagrams for illustrating a reflection type hologram system.
As described above, the mobile terminal according to the present invention includes the hologram output unit 155 and outputs the hologram image through the hologram output unit 155. In order to support the projection of the hologram image of the hologram output unit 155, the memory 170 may store information on the hologram interference pattern. The controller 180 can output information visually output from the mobile terminal as a hologram image through the hologram output unit 155 using the information stored in the memory 170. [
The hologram output unit 155 may be provided on at least one of a front surface, a rear surface, and a side surface of the terminal body to output a hologram image 155 'in a space on the rear surface.
FIGS. 2A and 2B are conceptual diagrams showing a hologram image 155 'implemented through the hologram output unit 155 shown in FIGS. 1B and 1C.
Referring to FIG. 2A, as an example, the hologram output unit 155 may be disposed on the front surface of the mobile terminal 100. The hologram output unit 155 can be driven independently from the display unit 151. The hologram output unit 155 can output the hologram image 155 even when the display unit 151 does not display visual information, (155 '). Meanwhile, the hologram output unit 155 may be mounted on a bezel portion for enclosing the display unit 151.
Further, as shown in the drawing, an output section (or haptic module 153) for transmitting a feedback signal is disposed close to the position where the hologram output section 155 is disposed, and is output through the hologram output section 155 In response to the user's gesture with respect to the hologram object, a feedback signal can be delivered to the user.
On the other hand, as shown in the drawing, the light source unit for outputting light (or light ()) for outputting the hologram object and the haptic module 153 for outputting the feedback signal can be disposed independently of each other.
Meanwhile, although not shown, the light source for outputting the hologram object and the haptic module for outputting the feedback signal may be composed of the same light source. At this time, the controller 180 controls the light source for outputting the hologram object By adjusting the intensity of the laser, the feedback signal can be realized with the intensity of the laser.
2B, the hologram output unit 155 may be disposed on the rear surface of the mobile terminal 100 to output a hologram image 155 'in a predetermined space on the rear surface thereof.
The position where the hologram output unit 155 is mounted and the space where the hologram image 155 'is outputted are not limited to the above examples. The hologram output unit 155 may be configured to be rotated or pop-up, or may be detachably attached to the terminal body as a separate device. The hologram image 155 'may be output in a space independent of the installation direction of the hologram output unit 155 through tilting, a separate reflection structure, or the like.
The hologram image 155 ', which can be implemented through the hologram output unit 155, may include both a two-dimensional (2D) plane image and a three-dimensional (3D) stereoscopic image.
The planar imaging system is a monoscopic method of providing the same image in both eyes, and it is possible to arrange a polyhedron generated through one or more points, lines, faces, or a combination thereof on a virtual three-dimensional space, Thereby displaying an image.
Next, the stereoscopic method is a method of providing different images in both eyes (stereo scopic), and is a method using a principle of feeling a stereoscopic feeling when a human is looking at an object with the naked eye. In other words, the two eyes of a person see different planar images when they see the same object by the distance between them. These different planar images are transmitted to the brain through the retina, and the brain fuses them to feel the depth and reality of the stereoscopic image. Therefore, binocular disparity caused by the distance between the two eyes is a three-dimensional feeling, and there is a slight difference in each person.
The hologram image 155 'generated through the hologram output unit 155 to be described later may include both the planar image and the stereoscopic image. Hereinafter, for the sake of convenience of description, what is expressed by the planar imaging system includes a case where the stereographic imaging system is expressed.
Hereinafter, a method of representing a hologram image 155 'and a structure for implementing the hologram image 155' will be described in detail.
The image outputted through the display unit 151 records only the distribution of the light and dark planes of the object whereas the hologram image 155 'is the image that simultaneously accumulates and reproduces all the information of the wave, that is, Can be understood.
3A and 3B are conceptual diagrams for explaining the principle of the hologram.
Referring to FIG. 3A, the coherent light from the laser light source 201 is divided into two parts by a splitter 202.
One of the light beams illuminates the object 207, and the diffused light from the surface of the object 207 reaches the holographic photosensitive material 205. Hereinafter, this ray is referred to as an object wave. Mirrors 203 and 204 may be used for path switching of the light beam.
The other one of the rays is switched to the mirror 208 and diffused through the lens 206 to reach the front of the holographic photosensitive material 205 directly. Hereinafter, this light beam is referred to as a reference wave.
The object wave and the reference wave interfere with each other on the holographic photosensitive material 205 to produce a very delicate and complex interference pattern of about 500 to 1,500 per mm. The holographic storage medium on which such interference fringes are recorded is called a hologram.
3B, the interference fringe serves as a diffraction grating, so that light is diffracted at a position different from the direction in which the reference wave is incident when the interference fringe acts as a diffraction grating Diffracted, and diffracted light is gathered and formed as light generated by reflection from the initial object, thereby projecting the hologram image 209. That is, the first object light can be reproduced through the hologram and can be expressed as the hologram image 209. [
At this time, when you look inside the reproduced wavefront, the first object appears, but the object appears to be inside. When you move the viewing point, the position of the object changes and it looks as though you are viewing a stereoscopic image. In addition, since the original wavefront of the object is reproduced, it can also interfere with the wavefront from an object that is slightly deformed.
The hologram image 209 may be classified into a transmission type hologram image display method and a reflection type polarographic image display method according to a reproduction method. Figs. 4A to 4C are conceptual diagrams for explaining a transmission holographic method, and Figs. 5A to 5C are conceptual diagrams for explaining a reflection holographic method.
The transmission type hologram image representation method is a method in which the light emerging from the back of the hologram is viewed and viewed from the front of the hologram. In the transmission type hologram image representation method, the object wave and the reference wave are exposed to the holographic photosensitive material in the same direction at the time of manufacture, and the generated hologram image is characterized in that the color is clear and bright.
Referring to FIG. 4A, the light emitted from the laser light source 301 passes through a spatial filter 302 and then spreads into a smooth spherical wave. In the beam splitter 305, one of the spherical waves divided into two beams is illuminated by the object 308 to produce an object wave, and the other is illuminated to the holographic photosensitive material 307 to produce a reference wave. The illuminated object wave on the object 308 is illuminated onto the holographic photosensitive material 307. [ Mirrors 304, 306, and 309 may be used to switch the light path.
At this time, the illuminated object wave and the reference wave interfere with each other in the holographic photosensitive material 307 to form an interference fringe, and the interference fringe is recorded in the holographic photosensitive material 307.
That is, as shown in FIG. 4B, the object wave and the reference wave are projected together on the same side of the holographic photosensitive material 307 to generate an interference fringe.
Then, as shown in Fig. 4C, when a reproduction wave which is the same as the reference wave is projected onto the holographic photosensitive material 307, the object wave is transmitted in the direction opposite to the plane of the previous object wave and the reference wave incident to generate a hologram image .
Next, the reflection type hologram image display system is constructed such that the object wave and the reference wave are incident on the holographic photosensitive material in mutually opposite directions in such a manner that an image reflected by the light in front of the hologram can be observed in front of the hologram. The hologram image generated by the reflection type hologram image display method is characterized by excellent stereoscopic effect.
Referring to FIG. 5A, light emitted from a laser light source 401 passes through a spatial filter 402 and then spreads as a smooth spherical wave, and is transmitted through a beam splitter 405, One illuminated by the object 408 to produce an object wave, and the other illuminated to the holographic photosensitive material 407 to produce a reference wave. Mirrors 404, 406, and 409 may be used to switch the light path. However, unlike FIG. 4A, the reference wave and the object wave are illuminated with the holographic photosensitive material 407 at positions opposite to each other.
5B, the reference wave is projected onto the left surface of the holographic photosensitive material 407, and the object wave is projected through the right upper surface of the holographic photosensitive material 407. [ Thereafter, as shown in FIG. 5C, when the reproduction wave which is the same as the reference wave is projected onto the holographic photosensitive material 407, the object wave is transmitted in the opposite direction to generate a hologram image.
Meanwhile, the hologram image may be displayed according to the set holography pattern in accordance with an embodiment of the present invention. The holographic pattern means that the hologram image projected through the hologram output unit 155 changes in a predetermined pattern with time and is provided to the user.
The holographic pattern can be variously set in the manner described below.
First, the holographic pattern can be set by changing the distance difference between the hologram output portion and the hologram image with time. According to the above configuration, since the hologram image projected through the hologram output unit 155 can be moved up and down, a predetermined holographic pattern can be set.
Next, the holographic pattern can be set by changing the shape of the hologram image projected by the hologram output unit 155 with time. For example, the control unit 180 may control the hologram image projected from the hologram output unit 155 to have a circular shape at first, and then control the shape of the hologram image to change into a rectangular shape with time.
Also, a method of moving or rotating the projected hologram image through the hologram output unit 155 may be applied. That is, the holographic image can be set by rotating the hologram image while moving the hologram image horizontally, horizontally, or horizontally while maintaining the same distance difference between the hologram output unit 155 and the hologram image.
Further, the hologram pattern can be set by modifying the color or size of the projected hologram image with time or adjusting the hologram image to blink. Furthermore, it is possible to set the holographic pattern through projection brightness, refresh rate, illumination, vibration feedback, sound insertion, image insertion, and repeated projection.
In the above description, it is assumed that a holographic pattern is set by an individual factor, but a holographic pattern may be set by a plurality of elements. For example, it is also possible to set the holographic pattern so that the projected hologram image moves left and right while changing the distance between the hologram output unit 155 and the hologram image according to time.
In the above description, it is assumed that a holographic pattern is set for the whole hologram image. However, this is merely an example, and it is also possible that a holographic pattern is applied to only a part of the hologram image.
Although the transmission type holographic image display method and the reflection type holographic image display method have been described above, the method of outputting the hologram image may be various.
On the other hand, holograms can be classified into three types according to the method of generating and reproducing holograms. As described above, i) analog holography method such as transmission type holographic image display method and reflection type holographic image display method, and 2) 3-dimensional digital data photographed with an image sensor such as CCD camera or CMOS sensor are acquired to generate a hologram Ii) a digital holography system having a form capable of storing, processing, and editing; iii) a technique of generating a hologram effect by photographing a super-resolution stereoscopic image though not a complete hologram, (Iii) a pseudo hologram (pseudo hologram) method in which the hologram image effect is projected to imitate the hologram image effect. Examples of similar hologram systems are preformat, Peppers ghost, and Leia display system.
Examples of the 3D holography method include Project Vermeer, a voixe box, and a 3D midair plasma display method.
Meanwhile, in the mobile terminal according to the present invention, a hologram image can be implemented through any of the various schemes described above or in a manner other than the schemes described above.
On the other hand, such a hologram image may not only be a three-dimensional image output in an arbitrary space but also provide a user experience such that the user actually touches an object corresponding to the hologram image or the hologram image.
Hereinafter, the hologram image output through the hologram output unit 155 is collectively referred to as a 'hologram object'.
The hologram object is output through the hologram output unit 155 and may be output on a predetermined space (or a predetermined three-dimensional space). Here, the predetermined space may correspond to the projection range of the hologram output unit 155. [ Then, in a state in which the hologram object is output, a process of sensing a human body (e.g., 'hand' of a user) located in the predetermined space may be performed. Here, it is possible not to sense a human body located in the predetermined space, but to sense a human body located in a region where the hologram object is output and a peripheral region.
On the other hand, the placement position of the sensing unit 140 may be set so that the predetermined space corresponding to the projection range in which the hologram output unit 155 projects the hologram object is sensed.
Here, the sensing unit (or sensing means) for sensing a human body located in the predetermined space may exist in a variety of ways. For example, the sensing unit may be a 3D camera.
More specifically, the camera 121 may include a 3D camera or a depth camera. For example, a human body located in an arbitrary space (or a predetermined space) through which a hologram image (or a hologram object) is output through an image sensor provided in a 3D camera (depth camera) . Such an image sensor may be configured to sense a user's human body while sensing a user's gesture in the arbitrary space. (Or 3D) image corresponding to a target object (or a human body) located near the hologram object or a hologram object (or a user human body) approaching the hologram object is acquired through the 3D camera .
The coordinate information of the hologram object can be compared with the coordinate information of the human body from the image obtained through the image sensor. The control unit 180 can determine the relative position between the hologram object and the human body by comparing coordinate information between them. If the coordinate information of the human body is equal to or close to the coordinate information of the hologram object as a result of the comparison, the human body i) approaches (ii) touches, or (iii) passes through the hologram object It can be judged.
Here, the coordinate information of the hologram object and the coordinate information of the human body may be obtained through modeling for the hologram object and the human body, respectively. When the human body approach to the hologram object is detected as described above, the controller 180 controls the hologram object so that the feedback signal is transmitted to the human body adjacent to or close to the hologram object based on the detection result The feedback output unit 156 can be controlled.
Here, various examples of the feedback signal may exist. As an example, the feedback output section 156 may include at least one of a laser output module and a sonic output module. The feedback signal may include at least one of a laser output through the laser output module and a sound wave output through the sound output module. The laser or sound wave has a constant intensity (e.g., intensity or amplitude), a period (e.g., frequency), a waveform (e.g., pulse wave, sine wave, etc.). The laser or sound wave outputted through the feedback output unit 156 is transmitted to the human body through which the user can obtain an experience of actually touching the hologram object. This feedback signal may also be referred to as a " haptic signal ". It is also possible that the feedback output unit 156 comprises the haptic module 153.
FIG. 6 illustrates an example in which the feedback output unit 156 transmits a feedback signal according to the approach of the human body to the hologram object.
The control unit 180 transmits the feedback signal 620 output to the feedback output unit 156 based on the coordinate information obtained as a result of modeling the human body 610 approaching or adjacent to the hologram object 600 The coordinate information of the target point to be obtained can be obtained. The control unit 180 may use the modeled human coordinate information so that the feedback signal 620 is transmitted to the human body 610. In this case, the control unit 180 does not transmit the feedback signal 620 to all of the areas where the hologram object 600 is output, but may transmit the feedback signal 620 only at a position where the human body 610 is located, (620).
On the other hand, the hologram output unit 155 and the feedback output unit 156 may have different configurations. That is, the hologram output unit 155 can be driven independently of the feedback output unit 156. [ In this case, the hologram output unit 155 and the feedback output unit 156 may be physically separated from each other. That is, the hologram output unit 155 may be a light source such as an LED or a laser in the visible light region, and the feedback output unit 156 may include a laser output module provided separately from the hologram output unit 155, An output module, an ultrasonic output module, a wind output module, and a mist output module.
On the other hand, when the feedback output unit 156 is deactivated, the control unit 180 outputs the feedback signal 620 only when the proximity or proximity of the human body 610 to the hologram object 600 is detected The feedback output unit 156 may be activated or deactivated. In other words, the feedback output unit 156 can maintain the deactivation state at all times, and can be switched to the active state only when the feedback signal 620 is to be output, and output the feedback signal 620. Here, the deactivated state may mean an off state, and the activated state may mean an on state.
Accordingly, only when the user's access to the hologram object 600 is detected, the feedback output unit 156 controls the feedback signal 620 to be output, thereby further reducing the power consumption. In addition, by using the coordinate information of the human body 610 adjacent to or close to the hologram object 600, not only the entire region of the hologram object 600 but also the feedback signal 620 By controlling the feedback output unit 156 so as to output the feedback signal 620 to all the regions.
Meanwhile, in the above description, the hologram object, which is output through a hologram output unit 155 to a region of a predetermined space, is called a hologram object. However, the hologram image may be augmented reality (Augmented Reality) image. That is, when a user wears an HMD (Head Mounted Display) or a Smart Glass smart lens, an augmented reality object corresponding to the hologram image is displayed on a device (HMD, smart Glass, smart lens, or the like) in the vicinity of the corresponding device.
Hereinafter, embodiments related to a control method that can be implemented in the mobile terminal 100 configured as above will be described with reference to the accompanying drawings. It will be apparent to those skilled in the art that the present invention may be embodied in other specific forms without departing from the spirit or essential characteristics thereof.
7 is a flowchart illustrating an operation of displaying a hologram object related to a currently executed function in a mobile terminal according to an exemplary embodiment of the present invention.
7, the controller 180 of the mobile terminal 100 according to the embodiment of the present invention can perform an operation according to at least one function executable in the mobile terminal 100 according to a user's selection have.
In this state, the control unit 180 can detect the state where the mobile terminal 100 is stationed (S700). This is to determine whether to display the hologram object related to the function of the mobile terminal 100 currently being executed, based on the state where the current user is stationed on the mobile terminal 100. [ For example, the control unit 180 can identify whether the mobile terminal 100 is in a state of being mounted in a direction perpendicular to the ground or in a horizontal direction using the gravity sensor.
That is, when the direction of gravity sensed by the gravity sensor is the direction toward the front or back of the mobile terminal 100, the controller 180 determines whether the current mobile terminal 100 is horizontally mounted or gripped . On the other hand, when the detected gravity direction is directed toward the upper or lower end of the mobile terminal 100, the controller 180 may determine that the mobile terminal 100 is currently held or gripped perpendicularly to the ground.
The front surface or the rear surface of the mobile terminal 100 may include a front surface or a rear surface of the mobile terminal 100 on which the display unit 151 is formed with reference to a display unit 151 of the mobile terminal 100, The face of the mobile terminal 100 in the opposite direction of the front face may be the rear face. A portion of the upper or lower portion of the mobile terminal 100 where the first sound output portion 152a of the mobile terminal 100 is formed is connected to the upper end of the mobile terminal 100, May be the lower end of the mobile terminal 100.
In step S702, the controller 180 may determine whether the state of the door being held or gripped in step S700 is a state corresponding to a predetermined first holding state (or holding state). Here, the first stationary state may be a state in which the mobile terminal 100 is mounted or held in a horizontal direction with respect to the ground. That is, when a user performs a specific function in the mobile terminal 100, the user places the mobile terminal 100 on a structure such as a floor or a table, When the mobile terminal 100 is gripped by the gravity sensor of the terminal 100 so that the gravity direction detected by the gravity sensor of the terminal 100 is vertical or similar to the angle of the display unit 151, It can be determined that it is in the first stationary state.
On the other hand, if it is determined in step S702 that the mounting state is not in a state corresponding to the predetermined first mounting state, the controller 180 displays the image information related to the currently executing function on the display unit 151 . Then, the process proceeds to step S700, where it is detected whether the state of mounting or holding of the mobile terminal 100 has been changed.
However, if it is determined in step S702 that the mobile terminal 100 is in the first stationary state, the controller 180 may display the image information related to the currently executed function as a hologram object in step S704.
For example, the control unit 180 can check whether the currently executing function includes image information that can be generated by the hologram object. For example, the control unit 180 determines whether or not the image information related to the currently executed function includes information about a stereoscopic shape (hereinafter, referred to as stereoscopic information), and whether the image information can be generated as a hologram object It can be judged.
Here, the stereoscopic information may be various information. For example, the stereoscopic information may be provided for generating a hologram object from a function or a provider providing the image information. In this case, the stereoscopic information may be provided in advance with the function being executed, or may be received from a predetermined server or an external server (for example, a cloud server, etc.) according to a user's selection.
On the other hand, the stereoscopic information may be stored together with the image information when the image information is sensed. For example, when an image to be sensed through the camera 121 is stored, depth information, that is, depth information of each part of the subject, together with an image of the subject, is stored, It can be information. This depth information is information on the depth of curvature formed by the surface of the subject according to the difference in distance between each part of the subject from the camera 121 lens, that is, the three-dimensional shape of the subject, And the distances of the respective parts of the subject from the camera lens 121 are different from each other.
If the function currently executed in the mobile terminal 100 includes image information capable of generating a hologram object, the control unit 180 of the mobile terminal 100 according to the embodiment of the present invention may include a function It is possible to generate a corresponding hologram object. The generated hologram object may be output in a predetermined space (S704). Here, the 'predetermined space' may be a space around the mobile terminal 100, which is preset so that the hologram output unit 155 outputs a hologram object. Therefore, in the mobile terminal 100 according to an embodiment of the present invention, when the mobile terminal 100 is in a state of being held or held in a state corresponding to the first stationary state, A first hologram object).
Accordingly, if the function currently being executed in the mobile terminal 100 is a video call function, the control unit 180 of the mobile terminal 100 according to an embodiment of the present invention can convert the image of the other party currently on the call to a hologram object Can be displayed. This is also because the mobile terminal of the calling party can also detect the depth information of the sensed image through the camera sensing the image of the calling party and transmit the depth information together with the image of the calling party. Accordingly, when the currently executing function is the video call function, the control unit 180 of the mobile terminal 100 according to the embodiment of the present invention generates the hologram object of the communication partner's image using the received depth information, and displays can do. Hereinafter, an operation of outputting a video of a calling party as a hologram object when performing a video call function in the mobile terminal 100 according to an embodiment of the present invention will be described in detail with reference to FIG.
Meanwhile, in this state, the controller 180 may control the function of the mobile terminal 100 currently being executed according to the input of the user applied to the first hologram object. That is, the controller 180 may detect a user's input to a specific region in which the first hologram object is displayed. Here, the input of the user may be various gestures of the user applied to the first hologram object. When the user's gesture applied to the first hologram object is sensed, the controller 180 may change the display state of the first hologram object according to the sensed gesture. For example, the controller 180 may change the size of the first hologram object or change the display position of the first hologram object according to the detected gesture. Hereinafter, various examples in which the first hologram object changes its shape and position according to the gesture of the user will be described in detail with reference to FIGS. 12A to 12D.
In addition, the controller 180 of the mobile terminal 100 according to the embodiment of the present invention can sense a user's input. For example, the input of the user may be a finger of a user approaching a region in which the first hologram object is displayed. In this case, the control unit 180 may sense the user's finger entering the predetermined space, and may provide a feedback signal to the user's finger through the feedback output unit 156. This is to enable the user to recognize that the user has entered the predetermined space by a tactile sense.
On the other hand, the feedback signal may be provided differently depending on the position of a user's body part (user's finger) approaching the first hologram object. That is, for example, if the detected position of the user's finger is located outside a predetermined first separation distance from the center of one area in which the first hologram object is displayed, A feedback signal can be provided. On the other hand, if the detected position of the user's finger is located within a predetermined first separation distance from the center of one region in which the first hologram object is displayed, it is possible to provide a stronger feedback signal. If the finger of the user is sensed as touching the first hologram object as a result of sensing the position of the detected finger of the user or in the area where the first hologram object is displayed, Signal.
Meanwhile, the control unit 180 of the mobile terminal 100 according to an embodiment of the present invention may display not only the tactile feedback signal but also a separate hologram object to visually detect that the user's finger is detected. For example, when the user's finger is detected within a predetermined distance from the center of one area in which the first hologram object is displayed, the controller 180 may display various hologram objects around the detected position of the finger have. For example, the hologram object may have a wavy shape or a wavelength shape formed around the detected position of the finger, and may be displayed in a form of concentric circles around the detected position of the finger. Accordingly, the mobile terminal 100 according to the embodiment of the present invention may allow the user to visually recognize that the mobile terminal 100 senses the user's finger through the hologram object. Of course, the control unit 180 may provide a feedback signal to the user's finger to recognize whether or not the user's finger is detected by not only the time but also the tactile sense.
If a user's input (e.g., a finger of a user approaching within a predetermined distance from the first hologram object) is detected, the controller 180 displays a menu screen related to a function currently being executed in the mobile terminal 100 (S706). Here, the menu screen may include a plurality of different control functions for controlling functions performed in the mobile terminal 100, and may be generated and displayed as a new hologram object (second hologram object).
The second hologram object may be displayed as a different hologram object from the first hologram object, and may be displayed in an area different from the area in which the first hologram object is displayed. Here, the hologram object corresponding to the second hologram object, that is, the 'menu screen', may be displayed in one area within the preset space in which the input of the user is sensed.
The control unit 180 may sense a user's input to the second hologram object. For example, the input of the user may be a body part (e.g., a finger) of a user accessing the second hologram object, as described above. If the position at which the body part of the user is sensed is detected within a specific area in which the second hologram object is displayed, the controller 180 can detect that the user's input is applied to the second hologram object have.
In this case, the controller 180 may determine that the control function corresponding to one point of the second hologram object detected by the user is selected by the input of the user. The function being executed by the mobile terminal 100 may be controlled according to the selected control function (S708).
On the other hand, any one of the plurality of control functions included in the second hologram object may be a function of terminating the function currently being executed in the mobile terminal 100. Accordingly, if the function selected in step S708 is a termination function, the controller 180 can terminate the function currently being executed in the mobile terminal 100. [ In this case, the display of the hologram object displayed through the hologram output unit 155 may also be terminated.
However, if the function selected by the user is not the end of the currently executing function, the control unit 180 may detect whether the mounting state of the mobile terminal 100 has been changed. When the stationary state of the mobile terminal 100 is a state corresponding to the first stationary state or a state that can be regarded as the first stationary state (e.g., a state in which the mobile terminal 100 can be regarded as being in a horizontal state Or when the angle of the gravity direction detected by the gravity sensor and the angle of the direction of the front or rear face of the mobile terminal 100 is within an angle of predetermined angles) The operation state in which the function of the mobile terminal 100 is controlled according to the input of the user applied through the hologram object can be maintained.
On the other hand, if it is determined that the mobile terminal 100 is not in the first stationary state due to the change of the state of the mobile terminal 100, the controller 180 proceeds to step S710 to display image information related to the currently executed function Dimensional image to be displayed on the display unit 151. [0154] FIG. The control unit 180 may detect whether the state of the mobile terminal 100 is changed or not, and output the hologram object corresponding to the image information currently being executed to a predetermined space according to a result of the detection. In this case, the control unit 180 may repeat the process of FIG. That is, according to the result detected in step S700, the image information is displayed as a two-dimensional image displayed on the display unit 151, or the hologram object is displayed through steps S704 to S708, So that the functions executed in the mobile terminal 100 can be controlled.
In the above description, the first stationary state is a state in which the mobile terminal 100 is held or held horizontally with the ground. However, the first stationary state can be changed in accordance with the user's choice. For example, the first stationary state may be a state in which the mobile terminal 100 is vertically mounted or held. In this case, the controller 180 determines whether the angle between the direction of the front or rear surface of the mobile terminal 100 and the direction of the gravity sensed by the gravity sensor is a vertical angle or a similar angle When the mobile terminal 100 is held or gripped, the hologram object can be generated and output according to the operation procedure described with reference to FIG. And controls a function executed in the mobile terminal 100 according to a user's input applied to the hologram object.
7, the control unit 180 of the mobile terminal 100 according to the embodiment of the present invention can display the image of the calling party as a hologram object when the currently executing function is the video call function . 8 is a flowchart illustrating an operation of displaying a hologram object for the video call function in the mobile terminal 100 according to an embodiment of the present invention.
Referring to FIG. 8, when the currently executing function is the video call function, the control unit 180 of the mobile terminal 100 according to the embodiment of the present invention determines whether the currently- The image of the calling party and the stereoscopic information of the image of the calling party can be received (S800). For example, the control unit 180 may receive the depth information of the communication partner image sensed through the front camera of the communication partner mobile terminal, together with the image of the communication partner, in step S800.
In step S802, the controller 180 may generate a hologram object corresponding to the image of the calling party using the stereoscopic information (depth information) obtained in step S800. Here, the hologram object may be a hologram object corresponding to a previous face of the calling party. This is because the stereoscopic information obtained from the mobile terminal of the calling party may be the previous face image of the calling party.
However, in the present invention, it is needless to say that the hologram object corresponding to the image of the calling party is not limited to the previous image of the calling party. That is, if there is any stereoscopic information obtained, a hologram object including not only the front side but also the back side of the calling party may be generated. That is, for example, when the stereoscopic information obtained from the mobile terminal 100 of the called party includes the depth information obtained from all of the front and rear sides of the other party, the movement according to the embodiment of the present invention The control unit 180 of the terminal 100 may generate a hologram object including not only the front side but also the back side of the communication party.
Then, the controller 180 may output the generated hologram object to one region of a predetermined space (S804). Accordingly, a three-dimensional hologram object formed according to the stereoscopic information of the other party to which the current video call is connected can be displayed in a predetermined space, and the user can perform a video call by confirming the other party through the hologram object.
Meanwhile, when generating and displaying a hologram object in step S704 of FIG. 7, the controller 180 of the mobile terminal 100 according to an exemplary embodiment of the present invention displays the hologram object in a predetermined area, It is possible to detect whether there is an obstacle. If there is an obstacle, the size or position of the hologram object can be displayed differently. FIG. 9 is a flowchart illustrating an operation process of the mobile terminal 100 according to the embodiment of the present invention.
9, when a hologram object (for example, a first hologram object or a second hologram object) is generated, the controller 180 of the mobile terminal 100 according to the embodiment of the present invention determines whether the hologram object One area within the predetermined space to be displayed can be calculated (S900). For example, the one area may be calculated according to the size of the hologram object currently generated from a predetermined point so that the hologram object is displayed in the predetermined space.
In operation S900, if the hologram object is to be displayed in the predetermined space, the controller 180 may detect whether there is an obstacle in the calculated area in operation S902. For example, the control unit 180 can detect an object around the mobile terminal 100 using an infrared sensor or a laser sensor. The control unit 180 compares the position of the sensed object with one area calculated in step S900, It is possible to identify whether the sensed object is an obstacle obstructing display of the hologram object.
If it is determined that there is no obstacle in the area where the hologram object is to be displayed as a result of the detection in step S902, the controller 180 controls the controller 180 to display the hologram currently generated in one area within the predetermined space calculated in step S900. You can output the object.
On the other hand, if it is determined in step S902 that there is an obstacle in the area where the hologram object is to be displayed, the controller 180 may change the size or position of the hologram object in step S904. For example, when an obstacle is detected in a part of the predetermined area calculated in step S900, the controller 180 may display the hologram object in another area in the predetermined space where the obstacle is not detected have.
For example, when an obstacle is detected, the control unit 180 determines whether or not an area corresponding to the size of the generated hologram object is positioned around another point in the predetermined space, (Second area), and can again detect whether or not there is an obstacle in the calculated second area. If an obstacle is detected, it can be repeatedly performed.
Alternatively, if the distance between the detected obstacle and the mobile terminal 100 is within a certain distance, or if it is not possible to find another area in which the obstacle is not detected in the preset space, the control unit 180 sets the size of the generated hologram object It can also be changed. Here, 'when no other area in which an obstacle is not detected' can be found in the predetermined space 'means that the process of calculating the second area and detecting whether or not an obstacle is present in the calculated second area is repeated a predetermined number of times or more It may be the case that the second region in which an obstacle is not detected can not be found.
Accordingly, in the present invention, when there is an obstacle around the mobile terminal 100, the hologram object can be displayed in a region other than the place where the obstacle exists, or in a size smaller than that of the obstacle. The function of the mobile terminal 100 may be controlled according to the input of the user applied to the hologram object.
According to the above description, the mobile terminal 100 according to the embodiment of the present invention can generate the hologram object by using the depth information stored together with the image information when the image information is sensed. The mobile terminal 100 according to an embodiment of the present invention uses the feature of the present invention to generate an image storing depth information of an image stored in the mobile terminal 100 and a hologram object generated using the depth information So that at least a part thereof can be changed or modified.
FIG. 10 illustrates an operation of changing a shape of an object included in a specific image using a hologram object in the mobile terminal 100 according to an exemplary embodiment of the present invention.
10, the controller 180 of the mobile terminal 100 according to the embodiment of the present invention can display any one of a plurality of previously stored images on the display unit 151 according to a user's selection . And may enter an operation state in which the image displayed on the display unit 151 can be modified or changed according to the user's selection.
In this state, the control unit 180 can sense the touch input of the user applied on the display unit 151, and identifies the object corresponding to the touch input among the objects included in the displayed image as the user's selection can do. If the specific object is identified, the control unit 180 may display the identified specific object so as to be distinguished from other objects (S1000).
Meanwhile, the image displayed on the display unit 151 may be an image including depth information about the subject. For example, the camera 121 of the mobile terminal 100 according to the embodiment of the present invention may be an array camera in which a plurality of lenses are arrayed in a matrix on a curved surface. When a plurality of lenses are arranged on the curved surface as described above, the photographing angles for photographing the subject in each lens may be different from each other. Accordingly, images in which one subject is photographed in different directions can be stored, and the depth difference, i.e., depth information, of each part of the subject can be stored for each of the images taken in the different directions . When the images taken in different directions are used, stereoscopic information about the entire surface of the subject may be obtained.
In operation S1002, the controller 180 may generate a hologram object for the image displayed on the display unit 151 using the depth information. Here, the control unit 180 may generate only the hologram object corresponding to the identified object.
If the hologram object is generated as described above, the controller 180 can display the generated hologram object in one area within a predetermined space (S1004). Here, one region in the predetermined space in which the hologram object is displayed may be a region corresponding to one region on the display unit 151. For example, the hologram object corresponding to the selected object may be formed directly on the display unit 151. In particular, the hologram object corresponding to the selected object may be vertically spaced from a point on the display unit 151, May be displayed.
Meanwhile, if the hologram object is displayed in step S1004, the controller 180 can detect a user's input to the hologram object. The state in which the hologram object is displayed can be changed according to the input of the sensed user (S1006). For example, the input of the user to the hologram object may be a part of the user's body sensed in one area of the space where the hologram object is displayed, and a gesture of the user through the body part of the sensed user.
The control unit 180 may change the state of the hologram object displayed according to the input of the user. For example, changing the display state of the hologram object may be to rotate the hologram object in a specific direction by a certain angle, or to change the size of the hologram object.
On the other hand, if the state in which the hologram object is displayed is changed according to the input of the user, the controller 180 reflects the display state of the changed hologram object and displays the display state of the selected specific object displayed on the display unit 151 Can be changed. That is, when the hologram object is rotated according to the input of the user, the controller 180 may rotate the specific object displayed on the display unit 151 according to the rotated hologram object. Alternatively, when the size of the hologram object is changed, the size of a specific object displayed on the display unit 151 may be changed according to the changed hologram object.
Accordingly, when the size of the specific object displayed on the display unit 151 is changed, or when one side of the specific object different from one side of the specific object displayed on the display unit 151 is displayed, can be changed. That is, even if the specific object is not in a front-facing state on the image, in response to a user's input through a hologram object corresponding to the specific object, The image displayed on the display 151 may be modified or changed. In this state, the control unit 180 may store an image including the specific object whose display state has been changed according to the user's selection, and may update the new image or the existing image so that the modified or changed image can be stored (S1010).
In the above description, the mobile terminal 100 according to the embodiment of the present invention displays the image information related to the currently executing function in the state where the mobile terminal 100 is stationary or at the user's selection, as a hologram object, An operation process in which a function executed in the mobile terminal 100 is controlled based on an input of a user applied to an object has been described in detail with reference to a plurality of flowcharts.
In the following description, in the mobile terminal 100 according to the present invention, a hologram object is displayed, and the displayed hologram object is controlled according to various gestures of the user or is displayed on the mobile terminal 100 using the hologram object. Will be described in more detail with reference to a plurality of exemplary diagrams. In the following description, a video call function is performed as an example of a function executed in the mobile terminal 100, and image information related to the video call function is displayed in accordance with the user's selection (state of mounting the mobile terminal 100, etc.) Is displayed in one area within a predetermined space will be described as an example.
However, it is needless to say that the present invention is not limited thereto, and it is needless to say that an album function capable of displaying any other function, for example, a game or a previously stored image of a photograph, can be performed. When another function is executed as described above, the mobile terminal 100 according to the embodiment of the present invention may display the image information of the currently executing other function as a hologram object.
11 is an exemplary diagram illustrating an example in which a mobile terminal 100 according to an embodiment of the present invention displays a hologram object of a video call partner according to a user's selection.
Referring to FIG. 11, the first diagram of FIG. 11 illustrates an example in which a mobile terminal 100 according to an exemplary embodiment of the present invention performs a video call.
The control unit 180 can detect whether the mobile station 100 is in a stationary state as described above with reference to FIG. When the stationary state of the mobile terminal 100 is held or held in the predetermined first stationary state, the mobile station 100 outputs a hologram object related to the currently executing function, i.e., a video call function, according to the stationary state of the mobile terminal 100 . The second diagram of FIG. 11 shows this example.
11, the second diagram of FIG. 11 shows an example in which the user places the mobile terminal 100 on a table during a video call as shown in the first diagram of FIG. 11 It is. When the user puts down the mobile terminal 100 as described above, the mobile terminal 100 can be placed in a horizontal state with respect to the ground, as shown in the second drawing of FIG. In this case, the control unit 180 of the mobile terminal 100 can determine that the mobile station 100 is currently in the first stationary state, and determines the hologram object related to the currently executing function Can be output.
Meanwhile, as described above, when the function currently being executed in the mobile terminal 100 is a video call function, the controller 180 can output image information related to the video call function as a hologram object. That is, in this case, the image information related to the video call function may be the image of the communication partner connected to the current video call. Accordingly, the control unit 180 can generate a hologram object image of the communication partner connected to the current video call and output the hologram object. The third diagram in Fig. 11 shows this example.
Meanwhile, in order to generate the hologram object, the controller 180 may use the stereoscopic information received from the mobile terminal of the called party. For example, the stereoscopic information may be depth information acquired with a video of the calling party at the mobile terminal of the calling party, or when the mobile terminal of the calling party includes an array camera , The images of the communication party captured in different directions through respective camera lenses included in the array camera may be the stereoscopic information. The controller 180 of the mobile terminal 100 according to the embodiment of the present invention can generate and output the hologram object corresponding to the image of the communication partner using the stereoscopic information.
Accordingly, as shown in the third drawing of FIG. 11, the hologram object 1100 corresponding to the image of the other party to which the current video call is connected can be output to one area within a predetermined space. As shown in the fourth diagram of FIG. 11, the user can continue the video call with the calling party while viewing the hologram object 1100. Meanwhile, in this case, the control unit 180 transmits the image of the user sensed through another camera (for example, a separately connected camera) connected to the mobile terminal 100 or a camera provided in the direction in which the user is located to the communication party So that the user's image can be displayed to the mobile terminal 100 of the called party.
On the other hand, the size and displayed position of the hologram objects displayed as shown in FIG. 11 can be freely changed according to the user's input. FIGS. 12A to 12D show examples in which the size and the display position are changed based on the user input applied to the hologram object.
12A shows an example of changing the size of a hologram object based on a user's input applied to the hologram object.
For example, as shown in the first drawing and the second drawing of FIG. 12A, in a state in which the hologram object 1100 is displayed, the control unit 180 determines whether a user's body part (for example, user's fingers) It is possible to detect that the user's fingers are approaching the hologram object by a predetermined level or more based on the sensed position and one area in the predetermined space in which the hologram object is displayed. In this case, the controller 180 may provide a feedback signal to the detected user's fingers to recognize that the user has accessed the hologram object with a tactile sense sensed by the fingers.
In this state, the control unit 180 can recognize the posture taken by the detected user fingers. For example, the control unit 180 compares the modeling information of the previously stored human body with the position of the user fingers sensed around the current hologram object 1100 and the shape taken by the fingers, The posture can be recognized. In addition, functions that can be performed from a gesture consisting of a body part of a user taking the specific posture, i.e., the user's fingers, may be performed according to the recognized postures.
Accordingly, as shown in the second diagram of FIG. 12A, when two fingers of a user who are spaced apart from each other by a predetermined distance or less are detected in the vicinity of the hologram object 1100, the controller 180 detects And recognizable. According to the recognized posture, the user can perform the motion, that is, the function corresponding to the gesture.
That is, when the posture taken by a user's body part (two fingers of a user) as shown in the second drawing of FIG. 12A is detected, the controller 180 enlarges the size of the hologram object 1100 It can be judged that the gesture for reducing the size is to be applied. Thus, if the user's gesture (pinch-out gesture) that causes the distance between the two detected user fingers to be distanced is sensed, as shown in the third diagram of Figure 12A, May further enlarge the size of the hologram object 1100 currently displayed according to the pinch-out gesture of the user. In this case, the magnification at which the hologram object 1100 is magnified can be determined according to the distance between the two farther apart user fingers or the distances between the changed positions of each of the fingers.
Meanwhile, in the third drawing of FIG. 12A, a case where a gesture of a user for distancing the distance between two fingers is detected has been described as an example, but conversely, the distance between two detected fingers is narrowed It goes without saying that the controller 180 may sense the gesture of the user (pinch-in gesture). In this case, the controller 180 may further reduce the size of the currently displayed hologram object 1100 according to the pinch-in gesture of the user. And the magnification by which the hologram object 1100 is shrunk can be determined according to the distance between the two closer user fingers or the distances between the changed positions of each of the fingers.
The gesture for enlarging or reducing the size of the hologram object 1100 may include various gestures in addition to the pinch-out or pinch-in gesture. For example, such a gesture may be made by the user's hand, which is sensed above the hologram object 1100.
For example, as shown in the first drawing of FIG. 12B, in a state in which the hologram object 1100 is displayed, the controller 180 determines whether the hologram object 1100 is displayed in the predetermined space Some (user's hand) can be detected. For example, as shown in the second drawing of FIG. 12B, when the user's hand is positioned vertically upward in one area within a predetermined space in which the hologram object 1100 is displayed, the control unit 180 can detect have.
In this state, the controller 180 can sense the detected part of the user's body, that is, the movement of the hand, by the user's gesture. In order to distinguish the part of the user's body sensed by chance in the predetermined space and the body part of the user sensed according to the motion of the user for controlling the hologram object 1100, It can be determined whether a part of the body of the user has reached a predetermined level or more in the area where the hologram object 1100 is displayed or whether the body part of the detected user meets predetermined conditions. If the detected part of the user's body is approaching a predetermined level or more in a region where the hologram object 1100 is displayed or if the body part of the detected user meets predetermined conditions, The body part can be detected as being for the user's input to the hologram object 1100. [ Here, the predetermined condition may be whether the detected time of the body part of the user is longer than a preset time and / or whether there is no movement for the predetermined time.
If it is determined that the detected part of the user's body is for the input of the user to the hologram object 1100, the controller 180 controls the hologram according to the gesture made by the body part of the user, The size or position of the object 1100 can be changed.
That is, as shown in the second and third drawings of FIG. 12B, when the user's hand is detected above the hologram object 1100 within a predetermined distance from the hologram object 1100, When the hand moves and the distance between the sensed user's hand and the mobile terminal 100 approaches, the controller 180 detects that the gesture of the user is to reduce the size of the hologram object 1100 . Accordingly, the controller 180 can reduce the size of the hologram object 1100, as shown in the third diagram of FIG. 12B. The scaling factor by which the hologram object 1100 is reduced may be determined according to the changed distance between the sensed user's hand and the mobile terminal 100 according to the gesture. That is, as the user's hand moves closer to the mobile terminal 100, the hologram object 1100 may be reduced in size.
On the other hand, when the detected user's hand is further away from the hologram object 1100, it is needless to say that the controller 180 may enlarge the size of the hologram object 1100. As described above, the magnification at which the hologram object 1100 is enlarged can be determined according to the changed distance between the sensed user's hand and the mobile terminal 100 according to the gesture.
In addition to changing the size of the hologram object 1100, the position at which the hologram object 1100 is displayed may be changed according to various gestures of the user. Figures 12C and 12D show these examples.
Referring to FIG. 12C, when the hologram object 1100 is displayed, the controller 180 controls the hologram object 1100 to display the hologram object 1100 at a predetermined distance or more (E.g., a finger) of the approaching user. In this case, the control unit 180 provides a feedback signal to the sensed user's finger to allow the user to recognize that the finger of the controller 180 has approached the hologram object 1100 within a certain distance It is possible. Alternatively, as shown in the second drawing of FIG. 12C, a separate hologram object may be further displayed around the sensed user's finger position so that the controller 180 may detect a finger approaching the hologram object 1100 within a certain distance The user may be able to visually identify the user.
In this state, the controller 180 may detect movement of the user's finger. 12C, the controller 180 may change the display position of the hologram object 1100 according to the movement of the user's finger, as shown in FIG. 12C.
12C, the controller 180 of the mobile terminal 100 according to the embodiment of the present invention changes the position at which the hologram object 1100 is displayed according to a specific gesture made by the user's finger It is possible. 12D illustrates an example in which the position of the hologram object 1100 is changed in the mobile terminal 100 according to the embodiment of the present invention.
Referring to FIG. 12D, when the hologram object 1100 is displayed as shown in the first drawing of FIG. 12D, the controller 180 can detect a part of the body of the user who is taking a predetermined attitude have. That is, as shown in the second drawing of FIG. 12D, when the user extends one of the fingers and indicates the direction in which the hologram object 1100 is located, the controller 180 detects the posture taken by the user's finger can do.
In this state, the controller 180 can determine a position at which the hologram object 1100 is displayed according to the attitude of the detected user's finger. For example, as shown in the fourth diagram of FIG. 12D, when the user takes a gesture in which a finger extending in the direction in which the hologram object 1100 is located is closed, the control unit 180 controls the hologram The position where the object 1100 is displayed can be changed. That is, the controller 180 may move the hologram object 1100 in a direction in which the user's finger is tilted.
In this case, as shown in the fourth diagram of FIG. 12D, the position where the hologram object 1100 is displayed in the direction in which the fingers are pulled out, that is, the direction in which the user's finger is sensed, may be changed. It goes without saying that the controller 180 may cause the hologram object 1100 to be displayed at a position closer to the user's finger based on the number of times the finger is worn or the time when the finger is held in the worn state.
Meanwhile, it is needless to say that the mobile terminal 100 according to the exemplary embodiment of the present invention may perform various functions related to a currently performed function, i.e., a video call function, by using the hologram object. 13 is an exemplary view illustrating an example of transmitting an image stored in a calling party using a hologram object in the mobile terminal 100 according to the embodiment of the present invention.
First, the first diagram of FIG. 13 shows an example in which a hologram object 1100 is displayed in the mobile terminal 100 according to an embodiment of the present invention. In this state, the controller 180 can detect a body part of the user who approaches the hologram object 1100 within a certain distance. As shown in the first diagram of FIG. 13, it is determined whether the control unit 180 detects a finger approaching the hologram object 1100 within a predetermined distance using a separate hologram object or a feedback signal. Or tactile sense.
In this state, the control unit 180 may display a menu including various functions related to the currently executing function, that is, a video call, as a separate hologram object, in close proximity to the detected user's finger. 13, the controller 180 may output the menu hologram object 1300 including the various functions in a state that the hologram object 1100 (first hologram object) is displayed . Here, the menu hologram object 1300 may be displayed at a position where the user's finger is sensed or around the sensed position. The menu hologram object 1300 may be displayed while the input of the user is detected. That is, when the user approaches the hologram object 1300 with the finger, the menu hologram object 1300 can be displayed according to this approach, and when the user again places a finger at a position distant from the hologram object 1100 The display of the menu hologram object 1300 can be terminated.
Here, the menu hologram object 1300 may include various functions related to the video call as described above. The 'related various functions' may include a function of transmitting a pre-stored image, a function of terminating a call, or a function of holding a video call function. The various functions may be displayed in different areas ('Photo', 'Hold', 'Decline') that are distinguished from each other in the menu hologram object 1300.
Here, among the menu hologram objects 1300, a function corresponding to an area indicated by 'Photo' may be a function of displaying a plurality of images stored in the mobile terminal 100. When a user's input (e.g., a user's finger) is detected in the region indicated by 'Photo', the controller 180 may display a plurality of images stored in the mobile terminal 100 as a hologram object . 13, the controller 180 may display a hologram object 1310 including thumbnail holograms corresponding to each of the images stored in the mobile terminal 100 in a preset space have. And may detect a user's input to one of the hologram thumbnails included in the hologram object 1310 (e.g., the first thumbnail hologram 1312).
On the other hand, in the hologram object 1310 including the thumbnail holograms, when one of the thumbnail holograms is selected according to the user's input, as shown in the third diagram of FIG. 13, the controller 180 controls the selected thumbnail hologram Can be transmitted to the calling party. In this case, the control unit 180 transmits the image by using the currently selected thumbnail hologram, i.e., the third hologram object 1330 corresponding to the first thumbnail hologram 1312, as shown in the fourth diagram of FIG. 13 To the hologram object 1100 corresponding to the currently displayed communication party.
Meanwhile, among the menu hologram objects 1300, the function corresponding to the area indicated by 'Decline' may be a function for terminating the currently connected video call. In this case, if the input of the user is applied in one area of the menu hologram object 1300 in which 'Hold' is indicated, the controller 180 may terminate the currently connected video call and terminate the display of the related hologram object. Accordingly, the display of both the hologram object 1100 and the menu hologram object 1300 of the called party can be terminated.
Meanwhile, among the menu hologram objects 1300, the function corresponding to the area indicated by 'Hold' may be for holding the display state of the currently connected video call or menu hologram object 1300.
Here, 'hold' of the menu hologram object 1300 may be for fixing the display state of the menu hologram object 1300. That is, when a user's input is applied to one area of the menu hologram object 1300 in which the 'Hold' is displayed, the controller 180 displays the currently displayed menu hologram object 1300 regardless of whether or not the user's input is sensed The display state can be maintained until the hold state is released. Here, the release of the hold state may occur when a user's input is applied to one area of the menu hologram object 1300 in which 'Hold' is displayed again, or when a preset user's input (for example, a specific gesture) have. In addition, when the display of the menu hologram object 1300 is held, the controller 180 displays the menu hologram object 1300 in a different color or uses another hologram object to indicate to the user that the menu hologram object 1300 is in a held state It is possible.
The control unit 180 of the mobile terminal 100 according to the embodiment of the present invention displays alarm information related to various events occurring in the mobile terminal 100 as a separate hologram object Hereinafter referred to as an alarm hologram object). The alarm hologram object may be displayed differently according to the distance from the user.
14A and 14B are views illustrating examples in which the mobile terminal 100 according to the embodiment of the present invention displays different alarm hologram objects for displaying alarm information according to the distance from the user admit.
For example, the control unit 180 of the mobile terminal 100 according to the embodiment of the present invention can detect various events occurring in the mobile terminal 100. For example, various events that occur in the mobile terminal 100 may be various kinds of messages, such as reception of a message, reception of a call, missed call, email reception, alarm for battery shortage or data usage, . The alarm hologram object 1100 may include information related to such an event. 14A and 14B illustrate examples in which a certain alarm is displayed among these alarms.
When an event (for example, a schedule alarm) occurs in the mobile terminal 100, as shown in FIG. 14A, an alarm hologram object 1400 for notifying the user of the generated event may be displayed. In this case, the controller 180 may display the alarm hologram object 1400 in one area in a direction corresponding to the detected position of the user, within a predetermined space capable of displaying the hologram object.
Meanwhile, the size of the alarm hologram object 1400 can be determined according to the distance from the user. 14A, when the distance between the mobile terminal 100 and the user is the first distance d1, the controller 180 determines that the alarm hologram object < RTI ID = 0.0 > (First alarm hologram object: 1400). However, when the distance between the mobile terminal 100 and the user is different, the size of the alarm hologram object may be different according to the distance.
14B, when the distance between the mobile terminal 100 and the user is a second distance d2 that is longer than the first distance d1, the controller 180 determines that the longer user The alarm hologram object can be displayed in a larger size. That is, as shown in FIG. 14B, the second hologram object 1450 whose magnitude is larger than that of the first alarm hologram object 1400 can be displayed in one area corresponding to the detected position of the user have.
According to the above description, the mobile terminal 100 according to the embodiment of the present invention can change the display state of a specific object included in a previously stored image using a hologram object. FIG. 15 illustrates an example of changing the display state of a specific object included in a previously stored image using a hologram object in the mobile terminal 100 according to an exemplary embodiment of the present invention.
15, the first diagram of FIG. 15 illustrates an example in which one of the images stored in the mobile terminal 100 according to the user's selection in the mobile terminal 100 according to the embodiment of the present invention Is displayed on the display unit 151. [0064] In this state, the control unit 180 can sense a touch input of a user applied on the display unit 151 and identify a specific object 1500 on the image corresponding to the position where the touch input is applied . In addition, the identified specific object 1500 can be separately displayed. The identified specific object 1500 may be displayed in a different color as shown in the first drawing of FIG. 15, or may be displayed separately from other graphic objects through another graphic object.
If the specific object 1500 of the image displayed on the display unit 151 is selected as described above, the controller 180 may generate the hologram object 1550 corresponding to the specific object 1500. For example, as described above, when the currently selected image is captured, the controller 180 controls the depth of the hologram object 150 using the depth information sensed by the bending according to the shape of the object corresponding to the specific object 1500 Lt; RTI ID = 0.0 > 1550 < / RTI > Alternatively, when the camera 121 is an array camera including a plurality of lenses, the controller 180 may control the hologram object 1550 corresponding to the specific object 1500 using the images received from the plurality of lenses It can also be displayed.
Here, the hologram object 1550 may be displayed on a predetermined spatial location corresponding to a region in which the specific object 1500 is displayed on the display unit 151. [ That is, the hologram object 1550 may be formed on the display unit 151 on the area where the specific object 1500 is displayed. The second diagram of FIG. 15 shows an example in which the hologram object 1550 is displayed above the area in which the specific object 1500 is displayed.
In this way, when the hologram object 1550 is displayed, the controller 180 can sense the user's input to the hologram object 1550. For example, such user input may be made by the attitude or movement of a part of the user's body that is applied to the hologram object 1100. [
15, when the user takes a gesture that the user hits the hologram object 1500 in one direction, the controller 180 controls the hologram object 1550 according to the gesture of the user, Can be changed. That is, as shown in the third drawing of FIG. 15, the controller 180 may rotate the currently displayed hologram object 1550 from left to right according to the gesture of the user. Accordingly, as shown in the third drawing of FIG. 15, the hologram object 1550 may be changed from a state in which the hologram object 1550 is looking in the front direction to a state in which the hologram object 1550 is viewed in a direction tilted to the right.
Meanwhile, when the display state of the hologram object 1550 is changed, the controller 180 may reflect the state of the hologram object 1550 whose display state is changed to the specific object 1500. That is, as shown in the third drawing of FIG. 15, when the hologram object 1550 is changed to a state of looking in the rightward tilted state while looking at the front, the controller 180 determines that the hologram object 1550 The display state of the specific object 1500 displayed on the display unit 151 can be changed according to the changed state. Thus, as shown in the fourth drawing of FIG. 15, the specific object 1500 displayed on the display unit 151 is in a state of looking in the direction tilted to the right while looking in the front, Can be changed. In this way, the image whose display state of the specific object 1500 is changed can be saved as a new image or an image replacing the existing image according to the user's selection.
In the above description, the user's body part (e.g., a finger) touches the hologram object as a user's input to the hologram object, and the controller (e.g., 180 detects the hologram object to which the user's input is applied.
When a user's body part is in contact with the hologram object, the controller 180 may output a feedback signal to a part of the body of the user to interact with the hologram object, to be. That is, in the above description, when the user's finger is detected in the area where the hologram object is displayed, or in the vicinity of the area where the hologram object is displayed, the feedback output unit 156 outputs the tactile sense corresponding to the hologram object It is possible to output a specific feedback signal corresponding to the hologram object. Accordingly, when the user touches a specific hologram object or takes a specific gesture (for example, a gesture as if picking up with a finger), the controller 180 controls the hologram object Of course, it is also possible to feel the tactile sense of touching the object directly.
According to the above description, the hologram image may be displayed as an augmented reality image through a device worn by the user. In this case, the mobile terminal according to the embodiment of the present invention displays an augmented reality object corresponding to the hologram object through a display unit of a device (HMD, smart glass, smart lens, etc.) You may. Fig. 16 shows this example.
First, referring to the first drawing of FIG. 16, the first drawing of FIG. 16 shows an example of a state in which a user is wearing a wearable device (smart glass: 1600) and making a video call through the mobile terminal 1650 FIG. 16, when the user holds the mobile terminal 1650, the image of the calling party connected to the current video call may be displayed on the display unit of the mobile terminal 1650 .
Meanwhile, in this state, the control unit of the mobile terminal 1650 can detect whether or not the stationary state of the mobile terminal 1650 has been changed. The augmented reality object corresponding to the image information currently being executed in the mobile terminal 1650 may be displayed through another predetermined device according to the state of the mobile terminal 1650. And the predetermined other device may be a device that is currently connected to the mobile terminal 1650 to perform a function related to a function previously designated from the user or currently performed in the mobile terminal 1650. [ 16, when a user performs a video call through the mobile terminal 1650 while the smart glass 1600 is worn, the smart glass 1600 is connected to the predetermined other device .
In this case, the control unit of the mobile terminal 1650 may display the augmented reality object related to the currently executed function through the smart glass 1600 according to the state of the mobile terminal 1650. 16, when the mobile terminal 1650 is laid horizontally on the table, the control unit of the mobile terminal 1650 controls the operation of the mobile terminal 1650 in such a manner that, The augmented reality object related to the function currently being performed in the mobile terminal 1660, that is, the video call function, may be displayed through the smart glass 1600 according to the change of the stationary state. Here, the augmented reality object related to the video call function may be an image of the other party connected to the video call.
In this case, the control unit of the mobile terminal 1650 can transmit the stereoscopic information, which is acquired together with the image of the communication partner and the image of the communication partner, to the smart glass 1600. Then, the smart glass 1600 can display an image generated using the video and stereoscopic information of the received call destination as an augmented reality object. The third diagram of FIG. 16 shows this example.
Referring to FIG. 16, the third diagram of FIG. 16 illustrates an example of displaying an augmented reality image through a transparent display unit of the smart glass 1600. That is, as shown in the third drawing of FIG. 16, the image 1660, i.e., the augmented reality object 1660, generated from the image and the stereoscopic information received from the mobile terminal 1650, And can be displayed around the image of the mobile terminal 1650 viewed through the display unit. Accordingly, an effect similar to the effect of outputting the hologram object may be implemented using a wearable device (HMD, smart glass, smart lens, or the like) equipped by the user.
In the above description, the smart glass 1600 generates the augmented reality object 1660 according to the image and the stereoscopic information received from the mobile terminal 1650, And may be generated in the mobile terminal 1650. In this case, the smart glass 1600 outputs the augmented reality object 1660 generated by the mobile terminal 1650 around the mobile terminal 1650 seen through the transparent display unit, thereby outputting the hologram object It is needless to say that similar effects may be obtained.
16, the augmented reality object 1660 is displayed according to the state of the mobile terminal 1650. However, the augmented reality object 1660 may be displayed according to the input of various users Of course. For example, the input of such a user may be a preset gesture or an input of a specific key. In this case, the control unit of the smart glass 1600 may be arranged around the mobile terminal 1650 visible through the transparent display unit of the smart glass 1600 according to whether the predetermined gesture is detected or a specific key is input. It is also possible to display the augmented reality object 1660 corresponding to the image.
In the above description, the smart glass having the transparent display unit has been described as an example, but the present invention is not limited thereto. That is, it is needless to say that the present invention may similarly be applied even when the user wears an HMD.
For example, when the user wears the HMD, the user can identify an external object through the camera provided in the HMD, or identify a graphic object corresponding to an external object through the display unit of the HMD have. In this case, the control unit of the HMD may display the augmented reality object in the external object image (when the image obtained through the camera of the HMD is displayed as it is), or in the vicinity of the graphic object corresponding to the external object Can be displayed.
The present invention described above can be embodied as computer-readable codes on a medium on which a program is recorded. The computer readable medium includes all kinds of recording devices in which data that can be read by a computer system is stored. Examples of the computer readable medium include a hard disk drive (HDD), a solid state disk (SSD), a silicon disk drive (SDD), a ROM, a RAM, a CD-ROM, a magnetic tape, a floppy disk, , And may also be implemented in the form of a carrier wave (e.g., transmission over the Internet). Also, the computer may include a control unit 180 of the terminal. The foregoing detailed description, therefore, should not be construed in a limiting sense in all respects and should be considered illustrative. The scope of the present invention should be determined by rational interpretation of the appended claims, and all changes within the scope of equivalents of the present invention are included in the scope of the present invention.
