KR20160031819A - Mobile terminal and method for controlling the same - Google Patents

Abstract

Disclosed are a mobile terminal, and a control method of a mobile terminal. The mobile terminal of the present invention comprises: a main sensor for obtaining a main image including at least one object; a first sub-sensor for obtaining a first image which includes at least one object in a first resolution lower than a resolution of the main sensor; a second sub-sensor for obtaining a second image which includes at least one object in a second resolution lower than the resolution of the main sensor; and a control unit for calculating three-dimensional depth information on the object included in the main image by using the first and second images, and compensating the three-dimensional depth information on the object based on the calculated three-dimensional depth information by using the main and first images or the main and second images. According to the present invention, the three-dimensional depth information in an image can be effectively obtained by using the sub-sensor while applying the main sensor in which the resolution increases.

Description

[0001] MOBILE TERMINAL AND METHOD FOR CONTROLLING THE SAME [0002]

The present invention relates to a mobile terminal and a control method thereof, which can calculate three-dimensional depth information of an image acquired through a sensor and control the three-dimensional image characteristics, further considering the convenience of the user.

A terminal can be divided into a mobile terminal (mobile / portable terminal) and a stationary terminal according to whether the terminal can be moved. The mobile terminal can be divided into a handheld terminal and a vehicle mounted terminal according to whether the user can directly carry the mobile terminal.

The functions of mobile terminals are diversified. For example, there are data and voice communication, photographing and video shooting through a camera, voice recording, music file playback through a speaker system, and outputting an image or video on a display unit. Some terminals are equipped with an electronic game play function or a multimedia player function. In particular, modern mobile terminals can receive multicast signals that provide visual content such as broadcast and video or television programs.

Such a terminal has various functions, for example, in the form of a multimedia device having multiple functions such as photographing and photographing of a moving picture, reproduction of a music or video file, reception of a game and broadcasting, etc. .

In order to support and enhance the functionality of such terminals, it may be considered to improve the structural and / or software parts of the terminal.

As a result of various functions performed in a terminal, an attempt has been made to implement a three-dimensional image reflecting the shape of an actual object in image-forming a specific object. As an example, there is an attempt to implement a three-dimensional image by applying an array camera.

However, if the resolution is increased in order to meet the user's demand for the high resolution of the increased image, the array camera method may cause problems such as cost increase and algorithm redesign. Accordingly, there is a need for a method capable of effectively acquiring three-dimensional depth information at the same time while increasing resolution of a conventional camera in response to a demand for high resolution.

The present invention is directed to solving the above-mentioned problems and other problems. Another object of the present invention is to provide a mobile terminal and a control method thereof that can effectively acquire three-dimensional depth information in an image using a sub sensor while applying a main sensor with an increased resolution.

According to an aspect of the present invention, there is provided an image processing apparatus including a main sensor for acquiring a main image including at least one object, a first sensor having a first resolution lower than the resolution of the main sensor, A first sub-sensor for acquiring a first image including the at least one object, a second sub-sensor for acquiring a second image including the at least one object at a second resolution lower than the resolution of the main sensor, Dimensional depth information of the at least one object included in the main image using the sub-sensor and the first image and the second image, and outputs the three-dimensional depth information of the at least one object included in the main image, And a controller for correcting the three-dimensional depth information of the at least one object based on the three-dimensional depth information calculated using the second image Provides a mobile terminal.

The main sensor, the first sub-sensor, and the second sub-sensor may be implemented as a single module.

The first sub-sensor and the second sub-sensor may be positioned symmetrically with respect to the main sensor.

The first sub-sensor and the second sub-sensor may be located at different distances with respect to the main sensor.

The first sub-sensor and the second sub-sensor may be positioned on different straight lines with respect to the main sensor.

The first sub-sensor and the second sub-sensor may be RGB Bayer sensors.

The first sub-sensor and the second sub-sensor may be monochrome sensors.

The first sub-sensor and the second sub-sensor may be an array camera.

The first sub sensor and the second sub sensor may be a sensor composed of any one of RGB channels.

The mobile terminal may further include an infrared light projecting infrared rays on the at least one object, wherein at least one of the first sub sensor or the second sub sensor senses an infrared image reflected from the at least one object can do.

The controller may adjust the three-dimensional characteristics of the main image based on the three-dimensional depth information of the at least one object included in the main image.

According to another aspect of the present invention, there is provided an image processing apparatus including a main sensor for acquiring a main image including at least one object, an infrared light for projecting infrared rays on the at least one object, A sub-sensor for acquiring a sub-image including an object of the at least one object, a sub-sensor for sensing an infrared image reflected by the at least one object, and a controller for calculating three-dimensional depth information of the at least one object using the sensed infrared image, And a controller for correcting the calculated 3D depth information using the main image and the sub-image.

The infrared light can project the structured light added with a predetermined pattern to the object.

The sub-sensor may include an infrared channel.

The controller may adjust the three-dimensional characteristics of the main image based on the three-dimensional depth information of the at least one object included in the main image.

According to another aspect of the present invention, there is provided a method of acquiring an image, the method comprising: acquiring a main image including at least one object; acquiring a first image including the at least one object at a first resolution lower than the resolution of the main image Obtaining a second image including the at least one object at a second resolution lower than the resolution of the main image by using the first image and the second image, Dimensional depth information of the at least one object based on the three-dimensional depth information calculated using the main image and the first image or the main image and the second image, And a step of correcting the depth information.

According to another aspect of the present invention, there is provided a method of generating a sub-image, the method comprising: obtaining a main image including at least one object; obtaining a sub-image including the at least one object at a lower resolution than the main image; Projecting an infrared ray on an object, sensing an infrared image reflected from the at least one object, calculating three-dimensional depth information for the at least one object using the sensed infrared image, And correcting the calculated three-dimensional depth information using the sub-image.

Effects of the mobile terminal and the control method according to the present invention will be described as follows.

According to at least one of the embodiments of the present invention, there is an advantage that three-dimensional depth information in an image can be effectively acquired by using a sub sensor while applying the main sensor with increased resolution.

In addition, according to at least one embodiment of the present invention, it is possible to acquire a three-dimensional depth value according to various baselines by adjusting the position of the sub sensor based on the main sensor.

In addition, according to at least one of the embodiments of the present invention, various kinds of sub-sensors are applied, and various 3D depth information can be obtained according to sub-sensors.

In addition, according to at least one embodiment of the present invention, it is possible to apply the structured optical system using infrared illumination, and to acquire 3D depth information easily at night.

Further scope of applicability of the present invention will become apparent from the following detailed description. It should be understood, however, that the detailed description and specific examples, such as the preferred embodiments of the invention, are given by way of illustration only, since various changes and modifications within the spirit and scope of the invention will become apparent to those skilled in the art.

1 is a block diagram illustrating a mobile terminal according to the present invention.
2 is a flowchart of a method of controlling a mobile terminal according to an embodiment of the present invention.
3 to 6 are views for explaining acquisition of 3D depth information using a main sensor and a sub sensor according to an embodiment of the present invention.
FIGS. 7 to 10 are views for explaining acquiring three-dimensional depth information according to positions of sub-sensors according to an embodiment of the present invention.
11 to 16 are views for explaining acquiring three-dimensional depth information according to the type of sub-sensor according to an embodiment of the present invention.
17 is a flowchart of a method of controlling a mobile terminal according to another embodiment of the present invention.
18 to 22 are diagrams for explaining acquisition of 3D depth information by applying infrared illumination according to an embodiment of the present invention.
FIGS. 23 to 27 illustrate acquisition of three-dimensional depth information by further including infrared illumination on two sub-sensors according to an embodiment of the present invention.

Hereinafter, embodiments of the present invention will be described in detail with reference to the accompanying drawings, wherein like reference numerals are used to designate identical or similar elements, and redundant description thereof will be omitted. The suffix "module" and " part "for the components used in the following description are given or mixed in consideration of ease of specification, and do not have their own meaning or role. In the following description of the embodiments of the present invention, a detailed description of related arts will be omitted when it is determined that the gist of the embodiments disclosed herein may be blurred. It is to be understood that both the foregoing general description and the following detailed description are exemplary and explanatory and are intended to provide further explanation of the invention as claimed. , ≪ / RTI > equivalents, and alternatives.

Terms including ordinals, such as first, second, etc., may be used to describe various elements, but the elements are not limited to these terms. The terms are used only for the purpose of distinguishing one component from another.

It is to be understood that when an element is referred to as being "connected" or "connected" to another element, it may be directly connected or connected to the other element, . On the other hand, when an element is referred to as being "directly connected" or "directly connected" to another element, it should be understood that there are no other elements in between.

The singular expressions include plural expressions unless the context clearly dictates otherwise.

In the present application, the terms "comprises", "having", and the like are used to specify that a feature, a number, a step, an operation, an element, a component, But do not preclude the presence or addition of one or more other features, integers, steps, operations, elements, components, or combinations thereof.

The mobile terminal described in this specification includes a mobile phone, a smart phone, a laptop computer, a digital broadcasting terminal, a personal digital assistant (PDA), a portable multimedia player (PMP), a navigation device, a slate PC A tablet PC, an ultrabook, a wearable device such as a smartwatch, a smart glass, and a head mounted display (HMD). have.

However, it will be appreciated by those skilled in the art that the configuration according to the embodiments described herein may be applied to fixed terminals such as a digital TV, a desktop computer, a digital signage, and the like, will be.

Referring to FIG. 1, FIG. 1 is a block diagram illustrating a mobile terminal according to the present invention.

The mobile terminal 100 includes a wireless communication unit 110, an input unit 120, a sensing unit 140, an output unit 150, an interface unit 160, a memory 170, a control unit 180, and a power supply unit 190 ), And the like. The components shown in FIG. 1 are not essential for implementing a mobile terminal, so that the mobile terminal described herein may have more or fewer components than the components listed above.

The wireless communication unit 110 may be connected between the mobile terminal 100 and the wireless communication system or between the mobile terminal 100 and another mobile terminal 100 or between the mobile terminal 100 and the external server 100. [ Lt; RTI ID = 0.0 > wireless < / RTI > In addition, the wireless communication unit 110 may include one or more modules for connecting the mobile terminal 100 to one or more networks.

The wireless communication unit 110 may include at least one of a broadcast receiving module 111, a mobile communication module 112, a wireless Internet module 113, a short distance communication module 114, and a location information module 115 .

The input unit 120 includes a camera 121 or an image input unit for inputting a video signal, a microphone 122 for inputting an audio signal, an audio input unit, a user input unit 123 for receiving information from a user A touch key, a mechanical key, and the like). The voice data or image data collected by the input unit 120 may be analyzed and processed by a user's control command.

The sensing unit 140 may include at least one sensor for sensing at least one of information in the mobile terminal, surrounding environment information surrounding the mobile terminal, and user information. For example, the sensing unit 140 may include a proximity sensor 141, an illumination sensor 142, a touch sensor, an acceleration sensor, a magnetic sensor, A G-sensor, a gyroscope sensor, a motion sensor, an RGB sensor, an infrared sensor, a finger scan sensor, an ultrasonic sensor, A microphone 226, a battery gauge, an environmental sensor (for example, a barometer, a hygrometer, a thermometer, a radiation detection sensor, A thermal sensor, a gas sensor, etc.), a chemical sensor (e.g., an electronic nose, a healthcare sensor, a biometric sensor, etc.). Meanwhile, the mobile terminal disclosed in the present specification can combine and utilize information sensed by at least two of the sensors.

The output unit 150 includes at least one of a display unit 151, an acoustic output unit 152, a haptic tip module 153, and a light output unit 154 to generate an output related to visual, auditory, can do. The display unit 151 may have a mutual layer structure with the touch sensor or may be integrally formed to realize a touch screen. The touch screen may function as a user input unit 123 that provides an input interface between the mobile terminal 100 and a user and may provide an output interface between the mobile terminal 100 and a user.

The interface unit 160 serves as a path to various types of external devices connected to the mobile terminal 100. The interface unit 160 is connected to a device having a wired / wireless headset port, an external charger port, a wired / wireless data port, a memory card port, And may include at least one of a port, an audio I / O port, a video I / O port, and an earphone port. In the mobile terminal 100, corresponding to the connection of the external device to the interface unit 160, it is possible to perform appropriate control related to the connected external device.

In addition, the memory 170 stores data supporting various functions of the mobile terminal 100. The memory 170 may store a plurality of application programs or applications running on the mobile terminal 100, data for operation of the mobile terminal 100, and commands. At least some of these applications may be downloaded from an external server via wireless communication. Also, at least a part of these application programs may exist on the mobile terminal 100 from the time of shipment for the basic functions (e.g., telephone call receiving function, message receiving function, and calling function) of the mobile terminal 100. Meanwhile, the application program may be stored in the memory 170, installed on the mobile terminal 100, and may be operated by the control unit 180 to perform the operation (or function) of the mobile terminal.

In addition to the operations related to the application program, the control unit 180 typically controls the overall operation of the mobile terminal 100. The control unit 180 may process or process signals, data, information, and the like input or output through the above-mentioned components, or may drive an application program stored in the memory 170 to provide or process appropriate information or functions to the user.

In addition, the controller 180 may control at least some of the components illustrated in FIG. 1 in order to drive an application program stored in the memory 170. In addition, the controller 180 may operate at least two of the components included in the mobile terminal 100 in combination with each other for driving the application program.

The power supply unit 190 receives external power and internal power under the control of the controller 180 and supplies power to the components included in the mobile terminal 100. The power supply unit 190 includes a battery, which may be an internal battery or a replaceable battery.

At least some of the components may operate in cooperation with one another to implement a method of operation, control, or control of a mobile terminal according to various embodiments described below. In addition, the operation, control, or control method of the mobile terminal may be implemented on the mobile terminal by driving at least one application program stored in the memory 170. [

Hereinafter, the components listed above will be described in more detail with reference to FIG. 1 before explaining various embodiments implemented through the mobile terminal 100 as described above.

First, referring to the wireless communication unit 110, the broadcast receiving module 111 of the wireless communication unit 110 receives broadcast signals and / or broadcast-related information from an external broadcast management server through a broadcast channel. The broadcast channel may include a satellite channel and a terrestrial channel. Two or more broadcast receiving modules may be provided to the mobile terminal 100 for simultaneous broadcast reception or broadcast channel switching for at least two broadcast channels.

The mobile communication module 112 may be a mobile communication module or a mobile communication module such as a mobile communication module or a mobile communication module that uses technology standards or a communication method (e.g., Global System for Mobile communication (GSM), Code Division Multi Access (CDMA), Code Division Multi Access 2000 (Enhanced Voice-Data Optimized or Enhanced Voice-Data Only), Wideband CDMA (WCDMA), High Speed Downlink Packet Access (HSDPA), High Speed Uplink Packet Access (HSUPA), Long Term Evolution (LTE) And an external terminal, or a server on a mobile communication network established according to a long term evolution (e. G., Long Term Evolution-Advanced).

The wireless signal may include various types of data depending on a voice call signal, a video call signal or a text / multimedia message transmission / reception.

The wireless Internet module 113 is a module for wireless Internet access, and may be built in or externally attached to the mobile terminal 100. The wireless Internet module 113 is configured to transmit and receive a wireless signal in a communication network according to wireless Internet technologies.

Wireless Internet technologies include, for example, wireless LAN (WLAN), wireless fidelity (Wi-Fi), wireless fidelity (Wi-Fi) Direct, DLNA (Digital Living Network Alliance), WiBro Interoperability for Microwave Access, High Speed Downlink Packet Access (HSDPA), High Speed Uplink Packet Access (HSUPA), Long Term Evolution (LTE) and Long Term Evolution-Advanced (LTE-A) 113 transmit and receive data according to at least one wireless Internet technology, including Internet technologies not listed above.

The wireless Internet module 113 for performing a wireless Internet connection through the mobile communication network can be used for wireless Internet access by WiBro, HSDPA, HSUPA, GSM, CDMA, WCDMA, LTE or LTE- May be understood as a kind of the mobile communication module 112.

The short-range communication module 114 is for short-range communication, and includes Bluetooth ™, Radio Frequency Identification (RFID), Infrared Data Association (IrDA), Ultra Wideband (UWB) (Near Field Communication), Wi-Fi (Wireless-Fidelity), Wi-Fi Direct, and Wireless USB (Wireless Universal Serial Bus) technology. The short-range communication module 114 is connected to the mobile terminal 100 and the wireless communication system through the wireless area networks, between the mobile terminal 100 and another mobile terminal 100, or between the mobile terminal 100 ) And the other mobile terminal 100 (or the external server). The short-range wireless communication network may be a short-range wireless personal area network.

Here, the other mobile terminal 100 may be a wearable device (e.g., a smartwatch, a smart glass, etc.) capable of interchanging data with the mobile terminal 100 according to the present invention (smart glass), HMD (head mounted display)). The short range communication module 114 may detect (or recognize) a wearable device capable of communicating with the mobile terminal 100 around the mobile terminal 100. [ If the detected wearable device is a device authenticated to communicate with the mobile terminal 100 according to the present invention, the control unit 180 may transmit at least a part of the data processed by the mobile terminal 100 to the short- 114 to the wearable device. Therefore, the user of the wearable device can use the data processed by the mobile terminal 100 through the wearable device. For example, according to this, when a telephone is received in the mobile terminal 100, the user performs a telephone conversation via the wearable device, or when a message is received in the mobile terminal 100, It is possible to check the message.

The position information module 115 is a module for obtaining the position (or current position) of the mobile terminal, and a representative example thereof is a Global Positioning System (GPS) module or a Wireless Fidelity (WiFi) module. For example, when the mobile terminal utilizes the GPS module, it can acquire the position of the mobile terminal by using a signal transmitted from the GPS satellite. As another example, when the mobile terminal utilizes the Wi-Fi module, it can acquire the position of the mobile terminal based on information of a wireless access point (AP) that transmits or receives the wireless signal with the Wi-Fi module. Optionally, the location information module 115 may perform any of the other functions of the wireless communication unit 110 to obtain data relating to the location of the mobile terminal, in addition or alternatively. The location information module 115 is a module used to obtain the location (or current location) of the mobile terminal, and is not limited to a module that directly calculates or obtains the location of the mobile terminal.

Next, the input unit 120 is for inputting image information (or signal), audio information (or signal), data, or information input from a user. For inputting image information, Or a plurality of cameras 121 may be provided. The camera 121 processes an image frame such as a still image or moving image obtained by the image sensor in the video communication mode or the photographing mode. The processed image frame may be displayed on the display unit 151 or stored in the memory 170. [ A plurality of cameras 121 provided in the mobile terminal 100 may be arranged to have a matrix structure and various angles or foci may be provided to the mobile terminal 100 through the camera 121 having the matrix structure A plurality of pieces of image information can be input. In addition, the plurality of cameras 121 may be arranged in a stereo structure to acquire a left image and a right image for realizing a stereoscopic image.

The microphone 122 processes the external acoustic signal into electrical voice data. The processed voice data can be utilized variously according to a function (or a running application program) being executed in the mobile terminal 100. Meanwhile, the microphone 122 may be implemented with various noise reduction algorithms for eliminating noise generated in receiving an external sound signal.

The user input unit 123 is for receiving information from a user and when the information is inputted through the user input unit 123, the control unit 180 can control the operation of the mobile terminal 100 to correspond to the input information . The user input unit 123 may include a mechanical input means (or a mechanical key such as a button located on the front, rear or side of the mobile terminal 100, a dome switch, a jog wheel, Jog switches, etc.) and touch-type input means. For example, the touch-type input means may comprise a virtual key, a soft key or a visual key displayed on the touch screen through software processing, And a touch key disposed on the touch panel. Meanwhile, the virtual key or the visual key can be displayed on a touch screen having various forms, for example, a graphic, a text, an icon, a video, As shown in FIG.

Meanwhile, the sensing unit 140 senses at least one of information in the mobile terminal, surrounding environment information surrounding the mobile terminal, and user information, and generates a corresponding sensing signal. The control unit 180 may control the driving or operation of the mobile terminal 100 or may perform data processing, function or operation related to the application program installed in the mobile terminal 100 based on the sensing signal. Representative sensors among various sensors that may be included in the sensing unit 140 will be described in more detail.

First, the proximity sensor 141 refers to a sensor that detects the presence of an object approaching a predetermined detection surface, or the presence of an object in the vicinity of the detection surface, without mechanical contact by using electromagnetic force or infrared rays. The proximity sensor 141 may be disposed in the inner area of the mobile terminal or in proximity to the touch screen, which is covered by the touch screen.

Examples of the proximity sensor 141 include a transmission type photoelectric sensor, a direct reflection type photoelectric sensor, a mirror reflection type photoelectric sensor, a high frequency oscillation type proximity sensor, a capacitive proximity sensor, a magnetic proximity sensor, and an infrared proximity sensor. In the case where the touch screen is electrostatic, the proximity sensor 141 can be configured to detect the proximity of the object with a change of the electric field along the proximity of the object having conductivity. In this case, the touch screen (or touch sensor) itself may be classified as a proximity sensor.

On the other hand, for convenience of explanation, the act of recognizing that the object is located on the touch screen in proximity with no object touching the touch screen is referred to as "proximity touch & The act of actually touching an object on the screen is called a "contact touch. &Quot; The position at which the object is closely touched on the touch screen means a position where the object corresponds to the touch screen vertically when the object is touched. The proximity sensor 141 can detect a proximity touch and a proximity touch pattern (e.g., a proximity touch distance, a proximity touch direction, a proximity touch speed, a proximity touch time, a proximity touch position, have. Meanwhile, the control unit 180 processes data (or information) corresponding to the proximity touch operation and the proximity touch pattern sensed through the proximity sensor 141 as described above, and further provides visual information corresponding to the processed data It can be output on the touch screen. Furthermore, the control unit 180 can control the mobile terminal 100 such that different operations or data (or information) are processed according to whether the touch to the same point on the touch screen is a proximity touch or a touch touch .

The touch sensor uses a touch (or touch input) applied to the touch screen (or the display unit 151) by using at least one of various touch methods such as a resistance film type, a capacitive type, an infrared type, an ultrasonic type, Detection.

For example, the touch sensor may be configured to convert a change in a pressure applied to a specific portion of the touch screen or a capacitance generated in a specific portion to an electrical input signal. The touch sensor may be configured to detect a position, an area, a pressure at the time of touch, a capacitance at the time of touch, and the like where a touch object touching the touch screen is touched on the touch sensor. Here, the touch object may be a finger, a touch pen, a stylus pen, a pointer, or the like as an object to which a touch is applied to the touch sensor.

Thus, when there is a touch input to the touch sensor, the corresponding signal (s) is sent to the touch controller. The touch controller processes the signal (s) and transmits the corresponding data to the controller 180. Thus, the control unit 180 can know which area of the display unit 151 is touched or the like. Here, the touch controller may be a separate component from the control unit 180, and may be the control unit 180 itself.

On the other hand, the control unit 180 may perform different controls or perform the same control according to the type of the touch object touching the touch screen (or a touch key provided on the touch screen). Whether to perform different controls or to perform the same control according to the type of the touch object may be determined according to the current state of the mobile terminal 100 or an application program being executed.

On the other hand, the touch sensors and the proximity sensors discussed above can be used independently or in combination to provide a short touch (touch), a long touch, a multi touch, a drag touch ), Flick touch, pinch-in touch, pinch-out touch, swipe touch, hovering touch, and the like. Touch can be sensed.

The ultrasonic sensor can recognize the position information of the object to be sensed by using ultrasonic waves. Meanwhile, the controller 180 can calculate the position of the wave generating source through the information sensed by the optical sensor and the plurality of ultrasonic sensors. The position of the wave source can be calculated using the fact that the light is much faster than the ultrasonic wave, that is, the time when the light reaches the optical sensor is much faster than the time the ultrasonic wave reaches the ultrasonic sensor. More specifically, the position of the wave generating source can be calculated using the time difference with the time when the ultrasonic wave reaches the reference signal.

The camera 121 includes at least one of a camera sensor (for example, a CCD, a CMOS, etc.), a photo sensor (or an image sensor), and a laser sensor.

The camera 121 and the laser sensor may be combined with each other to sense a touch of the sensing object with respect to the three-dimensional stereoscopic image. The photosensor can be laminated to the display element, which is adapted to scan the movement of the object to be detected proximate to the touch screen. More specifically, the photosensor mounts photo diodes and TRs (Transistors) in a row / column and scans the contents loaded on the photosensor using an electrical signal that varies according to the amount of light applied to the photo diode. That is, the photo sensor performs coordinate calculation of the object to be sensed according to the amount of change of light, and position information of the object to be sensed can be obtained through the calculation.

The display unit 151 displays (outputs) information processed by the mobile terminal 100. For example, the display unit 151 may display execution screen information of an application program driven by the mobile terminal 100 or UI (User Interface) and GUI (Graphic User Interface) information according to the execution screen information .

Also, the display unit 151 may be configured as a stereoscopic display unit for displaying a stereoscopic image.

In the stereoscopic display unit, a three-dimensional display system such as a stereoscopic system (glasses system), an autostereoscopic system (no-glasses system), and a projection system (holographic system) can be applied.

The sound output unit 152 may output audio data received from the wireless communication unit 110 or stored in the memory 170 in a call signal reception mode, a call mode or a recording mode, a voice recognition mode, a broadcast reception mode, The sound output unit 152 also outputs sound signals related to functions (e.g., call signal reception sound, message reception sound, etc.) performed in the mobile terminal 100. [ The audio output unit 152 may include a receiver, a speaker, a buzzer, and the like.

The haptic module 153 generates various tactile effects that the user can feel. A typical example of the haptic effect generated by the haptic module 153 may be vibration. The intensity and pattern of the vibration generated in the haptic module 153 can be controlled by the user's selection or the setting of the control unit. For example, the haptic module 153 may synthesize and output different vibrations or sequentially output the vibrations.

In addition to vibration, the haptic module 153 may be configured to perform various functions such as a pin arrangement vertically moving with respect to the contact skin surface, a spraying force or suction force of the air through the injection port or the suction port, a touch on the skin surface, And various tactile effects such as an effect of reproducing a cold sensation using an endothermic or exothermic element can be generated.

The haptic module 153 can transmit the tactile effect through the direct contact, and the tactile effect can be felt by the user through the muscles of the finger or arm. The haptic module 153 may include two or more haptic modules 153 according to the configuration of the mobile terminal 100.

The light output unit 154 outputs a signal for notifying the occurrence of an event using the light of the light source of the mobile terminal 100. Examples of events that occur in the mobile terminal 100 may include message reception, call signal reception, missed call, alarm, schedule notification, email reception, information reception through an application, and the like.

The signal output from the light output unit 154 is implemented as the mobile terminal emits light of a single color or a plurality of colors to the front or rear surface. The signal output may be terminated by the mobile terminal detecting the event confirmation of the user.

The interface unit 160 serves as a path for communication with all external devices connected to the mobile terminal 100. The interface unit 160 receives data from an external device or supplies power to each component in the mobile terminal 100 or transmits data in the mobile terminal 100 to an external device. For example, a port for connecting a device equipped with a wired / wireless headset port, an external charger port, a wired / wireless data port, a memory card port, an audio I / O port, a video I / O port, an earphone port, and the like may be included in the interface unit 160.

The identification module is a chip for storing various information for authenticating the use right of the mobile terminal 100 and includes a user identification module (UIM), a subscriber identity module (SIM) A universal subscriber identity module (USIM), and the like. Devices with identification modules (hereinafter referred to as "identification devices") can be manufactured in a smart card format. Accordingly, the identification device can be connected to the terminal 100 through the interface unit 160. [

The interface unit 160 may be a path through which power from the cradle is supplied to the mobile terminal 100 when the mobile terminal 100 is connected to an external cradle, And various command signals may be transmitted to the mobile terminal 100. The various command signals or the power source input from the cradle may be operated as a signal for recognizing that the mobile terminal 100 is correctly mounted on the cradle.

The memory 170 may store a program for the operation of the controller 180 and temporarily store input / output data (e.g., a phone book, a message, a still image, a moving picture, etc.). The memory 170 may store data related to vibration and sound of various patterns outputted when a touch is input on the touch screen.

The memory 170 may be a flash memory type, a hard disk type, a solid state disk type, an SDD type (Silicon Disk Drive type), a multimedia card micro type ), Card type memory (e.g., SD or XD memory), random access memory (RAM), static random access memory (SRAM), read-only memory (ROM), electrically erasable programmable read memory, a programmable read-only memory (PROM), a magnetic memory, a magnetic disk, and / or an optical disk. The mobile terminal 100 may operate in association with a web storage that performs the storage function of the memory 170 on the Internet.

Meanwhile, as described above, the control unit 180 controls the operations related to the application program and the general operation of the mobile terminal 100. [ For example, when the state of the mobile terminal meets a set condition, the control unit 180 can execute or release a lock state for restricting input of a user's control command to applications.

In addition, the control unit 180 performs control and processing related to voice communication, data communication, video call, or the like, or performs pattern recognition processing to recognize handwriting input or drawing input performed on the touch screen as characters and images, respectively . Further, the controller 180 may control any one or a plurality of the above-described components in order to implement various embodiments described below on the mobile terminal 100 according to the present invention.

The power supply unit 190 receives external power and internal power under the control of the controller 180 and supplies power necessary for operation of the respective components. The power supply unit 190 includes a battery, the battery may be an internal battery configured to be chargeable, and may be detachably coupled to the terminal body for charging or the like.

In addition, the power supply unit 190 may include a connection port, and the connection port may be configured as an example of an interface 160 through which an external charger for supplying power for charging the battery is electrically connected.

As another example, the power supply unit 190 may be configured to charge the battery in a wireless manner without using the connection port. In this case, the power supply unit 190 may use at least one of an inductive coupling method based on a magnetic induction phenomenon from an external wireless power transmission apparatus and a magnetic resonance coupling method based on an electromagnetic resonance phenomenon Power can be delivered.

In the following, various embodiments may be embodied in a recording medium readable by a computer or similar device using, for example, software, hardware, or a combination thereof.

Hereinafter, embodiments related to a control method that can be implemented in a mobile terminal configured as above will be described with reference to the accompanying drawings. It will be apparent to those skilled in the art that the present invention may be embodied in other specific forms without departing from the spirit or essential characteristics thereof.

2 is a flowchart of a method of controlling a mobile terminal according to an embodiment of the present invention. 3 to 6 are views for explaining acquisition of 3D depth information using a main sensor and a sub sensor according to an embodiment of the present invention.

The method for controlling a mobile terminal according to an embodiment of the present invention may be implemented in the mobile terminal 100 described with reference to FIG. Hereinafter, a method of controlling a mobile terminal according to an exemplary embodiment of the present invention and an operation of the mobile terminal 100 for implementing the method will be described in detail with reference to the accompanying drawings.

Referring to FIG. 2, the controller 180 may acquire a main image including at least one object through a main sensor 10 (S100).

3, a camera 121 provided on one side of the mobile terminal 100 is shown. The camera 121 may include a main sensor 10, a first sub-sensor 30 and a second sub-sensor 50.

The main sensor 10 may correspond to an image sensor generally used in the mobile terminal 100. For example, it may be an RGB Bayer sensor with high resolution. However, the present invention is not limited thereto, and the main sensor 10 can be implemented with various sensors having high resolution according to the technology development.

The main image is an image obtained through the main sensor 10, and may correspond to another image depending on a sensor to which it is applied. At least one object may be included in the main image. The object is not limited to a specific object.

Referring again to FIG. 2, the controller 180 may obtain a first image including the at least one object at a first resolution lower than the resolution of the main image through a first sub-sensor [ S110]. Also, the controller 180 may acquire a second image including the at least one object at a second resolution lower than the resolution of the main image through the second sub-sensor (S120).

In FIG. 2, steps S110 and S120 are shown as two steps, but the present invention is not limited thereto. The control unit 180 may simultaneously perform the acquisition of the first image and the acquisition of the second image, or may acquire any one of the images first.

The first image and the second image may include the at least one object included in the main image. However, if necessary, only one of the first image and the second image may include the at least one object.

The first resolution and the second resolution may be any values as long as they are lower than the resolution of the image obtained through the main sensor 10. However, if necessary, at least one of the first resolution and the second resolution may be higher than the resolution of the image obtained through the main sensor 10. The first resolution and the second resolution may be the same value or different values.

3, the camera 121 may be embodied as one module including a main sensor 10, a first sub sensor 30, and a second sub sensor 50, according to an example. However, the present invention is not limited thereto, and the main sensor 10, the first sub sensor 30, and the second sub sensor 50 may be implemented as different modules as needed.

Referring back to FIG. 2, the controller 180 may calculate the 3D depth information of the at least one object included in the main image using the first image and the second image [S130].

4 illustrates the acquisition of an image including the same object 200 in the main sensor 10, the first sub-sensor 30, and the second sub-sensor 50 included in the camera 121. FIG. The dotted line in each sensor means an angle that can be imaged by each sensor, and is not limited to a specific angle.

Referring to FIG. 5, FIG. 5 (a) shows an image 230 of the object acquired by the first sub-sensor 30. FIG. 5B shows an image 210 of the object acquired by the main sensor 10. FIG. 5C shows an image 250 of the object acquired by the second sub-sensor 50. FIG.

6 (a), the images 230 and 250 of the objects shown in FIGS. 5 (a) and 5 (c) are superimposed. The control unit 180 determines the distance between the values of the baseline which is the distance between the first sub sensor 30 and the second sub sensor 50 and the values of the images 230 and 250 of each object according to the distances of the sensors 30 and 50 The depth information of the object up to the object can be calculated using the parallax d1 which is the distance between the centers. According to an example, the calculation of the three-dimensional depth information can be performed by a known stereoscopic vision method, and a detailed description thereof will be omitted.

Referring again to FIG. 2, the control unit 180 determines the three-dimensional depth information of the at least one object based on the three-dimensional depth information calculated using the main image and the first image or the main image and the second image. The depth information can be corrected (S140).

6 (b), the images 210 and 250 of the objects shown in FIGS. 5 (b) and 5 (c) are superimposed. 6 (c), the images 210 and 230 of the objects shown in FIGS. 5 (a) and 5 (b) are overlaid. Each of the parallaxes d1, d2, (10, 30, 50).

The intervals of the 3D depth values are also determined corresponding to various parallaxes. Accordingly, the interval of the three-dimensional depth value using the image obtained based on the baseline by the main sensor 10 and the first sub-sensor 30 corresponds to the depth of the three-dimensional depth value using the image obtained based on the different baseline Section.

The controller 180 uses the three-dimensional depth information calculated using the main image (FIG. 5B) and the first image (FIG. 5B) The three-dimensional depth information calculated using the second image ((c) of FIG. 5) can be corrected.

 Alternatively, the control unit 180 may correct the three-dimensional depth information calculated using the first image and the second image calculated in step S130 by using the three-dimensional depth information calculated using the main image and the second image can do. This will be described in detail in Figs. 7 to 10.

Accordingly, the user can acquire an image with high resolution using the main sensor, acquire the three-dimensional depth in the obtained image using the sub sensor, The 3D depth can be further subdivided.

FIGS. 7 to 10 are views for explaining acquiring three-dimensional depth information according to positions of sub-sensors according to an embodiment of the present invention.

Referring to FIG. 7, the first sub-sensor 30 and the second sub-sensor 50 may be positioned symmetrically with respect to the main sensor 10. The controller 180 may calculate the 3D depth information of the object included in the image using the first sub sensor 30 and the second sub sensor 50. [

Since the first sub sensor 30 and the second sub sensor 50 are positioned to be symmetrical with respect to each other, the base line between the first sub sensor 30 and the main sensor 10, the second sub sensor 50, 10 have the same value as each other. Accordingly, the controller 180 can calculate the same three-dimensional depth value using either the first sub-sensor 30, the main sensor 10, the second sub-sensor 50, or the main sensor 10 have.

The control unit 180 uses the first and second sub sensors 30 and 50 using the three dimensional depth value calculated using the main sensor 10 and the first or second sub sensor 30 or 50 The calculated three-dimensional depth value can be corrected.

Referring to FIG. 10, FIG. 10 (a) shows a section of a three-dimensional depth value calculated using the first and second sub-sensors 30 and 50. In the digitized sensor, the parallax d1 in Fig. 6 (a) is measured in pixel units. Accordingly, the controller 180 measures the same three-dimensional depth value for the parallax in the same pixel of the sensor. That is, the intervals a1 to a5 become three-dimensional depths corresponding to one pixel, respectively.

The control unit 180 can not calculate the three-dimensional depth difference within the same pixel of the sensor. That is, even if there is a difference in the depth value, there is a section that is calculated to the same depth, so that the depth resolution is reduced. The depth resolution indicating the degree of calculation of the three-dimensional depth can be understood as the number of pixels in the sensor corresponding to the unit length of the three-dimensional depth. Alternatively, the depth resolution can be understood as a three-dimensional depth value corresponding to one pixel.

When a digitized sensor is used, the position shift value of an object included in the acquired image is represented by the number of pixels. Therefore, it can be said that as the same three-dimensional depth is represented by a larger number of pixels, the depth resolution is larger. Alternatively, the smaller the three-dimensional depth value corresponding to one pixel, the greater the depth resolution.

Referring to FIG. 10B, the sections b1 to b4 correspond to the three-dimensional depth values calculated using the main sensor 10 and the first or second sub sensor 30 or 50. FIG. The control unit 180 can further subdivide the sections a1 to a5 using the sections b1 to b4. For example, the control unit 180 may divide the interval a2 into the intervals c2 and c3 using the interval b1. Therefore, more accurate three-dimensional depth information can be obtained.

Referring to FIG. 8, the first sub-sensor 30 and the second sub-sensor 50 may be located at different distances with respect to the main sensor 10. The controller 180 may calculate the 3D depth information of the object included in the image using the first sub sensor 30 and the second sub sensor 50. [

Since the first sub sensor 30 and the second sub sensor 50 are located at different distances from each other, the base line between the first sub sensor 30 and the main sensor 10, the second sub sensor 50, And the base line between the base plates 10 have different values. Accordingly, the controller 180 can calculate different three-dimensional depth values using the first sub-sensor 30, the main sensor 10, the second sub-sensor 50, and the main sensor 10, respectively.

The controller 180 controls the three-dimensional depth values calculated using the first and second sub-sensors 30 and 50 using the three-dimensional depth values calculated using the main sensor 10 and the first sub- Value can be corrected. In addition, the three-dimensional depth value calculated by using the first and second sub-sensors 30 and 50 using the three-dimensional depth value calculated by using the main sensor 10 and the second sub- Can be corrected.

That is, in FIG. 10 (b), the three-dimensional depth section shown in FIG. 10 (a) is further subdivided. It is assumed that the sections b1 to b4 in FIG. 10 (b) are three-dimensional depth sections calculated by the main sensor 10 and the second sub-sensor 50. In this case, the control unit 180 can further subdivide the sections c1 to c9 using another three-dimensional depth section calculated by the main sensor 10 and the first sub-sensor 30. Therefore, more accurate three-dimensional depth information can be obtained.

Referring to FIG. 9, the first sub-sensor 30 and the second sub-sensor 50 may be positioned on different straight lines with respect to the main sensor 10. The controller 180 may calculate the 3D depth information of the object included in the image using the first sub sensor 30 and the second sub sensor 50. [

Since the first sub sensor 30 and the second sub sensor 50 are located on different straight lines, the base line between the first sub sensor 30 and the main sensor 10, the second sub sensor 50, The baselines between the sensors 10 have different values. Accordingly, the controller 180 can calculate different three-dimensional depth values using the first sub-sensor 30, the main sensor 10, the second sub-sensor 50, and the main sensor 10, respectively.

The controller 180 controls the three-dimensional depth values calculated using the first and second sub-sensors 30 and 50 using the three-dimensional depth values calculated using the main sensor 10 and the first sub- Value can be corrected. In addition, the three-dimensional depth value calculated by using the first and second sub-sensors 30 and 50 using the three-dimensional depth value calculated by using the main sensor 10 and the second sub- Can be corrected.

That is, in FIG. 10 (b), the three-dimensional depth section shown in FIG. 10 (a) is further subdivided. It is assumed that the sections b1 to b4 in FIG. 10 (b) are three-dimensional depth sections calculated by the main sensor 10 and the second sub-sensor 50. In this case, the control unit 180 can further subdivide the sections c1 to c9 using another three-dimensional depth section calculated by the main sensor 10 and the first sub-sensor 30.

In this case, the directions of the base lines of the main sensor 10 and the sub sensors 30 and 50 further include components in the direction perpendicular to the base line direction in FIG. Accordingly, since the controller 180 calculates the three-dimensional depth information in consideration of the vertical direction, more accurate three-dimensional depth information can be obtained.

Through this, the user can obtain the three-dimensional depth value according to various baselines by adjusting the position of the sub-sensor based on the main sensor. Further, it is possible to calculate more precise three-dimensional depth information by using this.

Although the two sub-sensors have been described with reference to Figs. 7 to 9, the number of sub-sensors is not limited to two. If necessary, three or more sub-sensors having different distances from the main sensor 10 may be applied. Using this, the control unit 180 can calculate more detailed three-dimensional depth information.

7 to 9, the sub sensors 30 and 50 are located in different directions with respect to the main sensor 10, respectively. This is because it is preferable that the distance between the sub-sensors 30 and 50, which are mainly used for three-dimensional depth measurement, is long. However, the positions of the sub-sensors 30 and 50 are not limited thereto, and the sub-sensors 30 and 50 may be positioned in the same direction with respect to the main sensor 10 as necessary.

Hereinafter, calculation of three-dimensional depth information according to the types of sub-sensors will be described. In this case, the description of the position of the sub sensor described above can be applied equally regardless of the type of sub sensor.

11 to 16 are views for explaining acquiring three-dimensional depth information according to the type of sub-sensor according to an embodiment of the present invention.

Referring to FIG. 11, the first and second sub-sensors 31 and 51 may be an RGB Bayer sensor in the same manner as the main sensor 10. As shown in FIG. 11, the first and second sub-sensors 31 and 51 may be sensors having a lower resolution as compared with the main sensor 10. That is, in the case of the main sensor 10, the main purpose is to obtain a high-resolution image, and the sub-sensors 31 and 51 can mainly calculate the three-dimensional depth information.

11, the main sensor 10 and the first and second sub-sensors 31 and 51 are shown in arbitrary sizes. However, it will be appreciated that this is for convenience of description, and can be implemented by sensors of various resolutions as required. 11, the ratios of the Red (11), Green (12), and Blue (13) filters applied to each sensor may be 1: 2: 1. However, this is according to an example, but is not limited thereto. Various types of filters can be applied as needed.

11, when the first and second sub-sensors 31 and 51 are implemented by the RGB Bayer sensor, the controller 180 uses the first and second sub-sensors 31 and 51 by the stereo vision method It is possible to calculate three-dimensional depth information.

7, the controller 180 may calculate the 3D depth information using the images obtained by the first and second sub sensors 31 and 51 and the main sensor 10, respectively . To this end, the controller 180 may convert the image acquired by the main sensor 10 into the same resolution information as that of the first and second sub sensors 31 and 51.

The controller 180 calculates the three-dimensional depth information calculated using the first and second sub sensors 31 and 51 by using the first and second sub sensors 31 and 51 and the main sensor 10 And can be corrected using a three-dimensional depth information.

In this case, when the specific surface or the like of the same object is acquired only by one of the sub sensors, the control unit 180 calculates the output of the main sensor 10 by using the first and second sub sensors 31 and 51 and the main sensor 10 It is possible to compensate it by using a 3D depth information.

16, the controller 180 can use images (FIG. 16A and FIG. 16B) having different resolutions in order to increase the spatial resolution. The control unit 180 may convert the resolution information so that the image obtained by the main sensor 10 and the image obtained by the first and second sub sensors 31 and 51 represent the same area. Referring to FIG. 16 (c), more detailed information can be obtained for the same area, so that the controller 180 can increase the spatial resolution.

In addition, the controller 180 can improve the quality of the image acquired by the main sensor 10 by using the pixel information of the image acquired by the first and second sub sensors 31 and 51. In this case, the controller 180 may improve the quality of the image acquired by the main sensor 10 by applying a known super resolution algorithm.

Also, the controller 180 may perform the refocusing function using the 3D depth information calculated for the image acquired by the main sensor 10. [ The control unit 180 can adjust the focus of any part of the image acquired by the main sensor 10 as if it were in focus.

Accordingly, the controller 180 can acquire more precise 3D depth information, and thus, the user can be provided with various applications such as refocusing based on 3D depth information.

Referring to FIG. 12, the first and second sub-sensors 32 and 52 may be monochrome sensors. In the case of a monochrome sensor, an image can be obtained even in a low-illuminance environment. The control unit 180 can display the image of the main sensor 10 obtained in the low illumination environment using the three-dimensional depth value calculated from the image obtained by the monochrome sensor 32,

In addition, the controller 180 may convert the image acquired by the main sensor 10 into the same channel and resolution information as the image acquired by the monochrome sensor 32, 52. The control unit 180 may calculate the three-dimensional depth value using the converted image, and may correct the three-dimensional depth value calculated using the black and white sensors 32 and 52. [

According to one example, when using the black and white sensors 32 and 52, a relatively simple algorithm can be applied in calculating the three-dimensional depth information, and the controller 180 can quickly calculate the three-dimensional depth information.

Through this, the user can be provided with an improved image even in a low-illuminated environment. Further, by using a monochrome sensor, it is possible to more easily calculate three-dimensional depth information.

Referring to FIG. 13, the first and second sub-sensors 33 and 53 may be an array camera. The first and second sub-sensors 33 and 53 may be implemented as a set of four (2 * 2) independent sensors as array cameras. 13, each of the first and second sub sensors 33 and 53 includes independent sensors 33a to 33c and 53a to 53c to which RGB filters are applied and independent sensors 33d and 53d to which a black and white filter is applied Lt; / RTI > This can be distinguished from the fact that the first and second sub-sensors 31 and 51 of FIG. 11 are applied to the RGB filter in pixel units in one sensor.

However, this is an example, and the form of the array camera is not limited thereto. And may be configured with a different number of sensors as needed. In addition, the arrangement of the independent sensors 33a to 33c, 53a to 53c to which the RGB filters are applied and the independent sensors 33d and 53d to which the monochrome filter is applied can be set differently as needed.

The controller 180 can acquire three-dimensional depth information using sensors in the array camera constituting the first and second sub-sensors 33 and 53. In addition, the controller 180 may combine sensors having baselines between the first and second sub-sensors 33 and 53 to obtain three-dimensional depth information. When some sensor of the array camera is implemented as a monochrome sensor, the control unit 180 can easily acquire three-dimensional depth information in a low-illuminance environment.

In addition, the controller 180 may convert the image acquired by the main sensor 10 into the same channel and resolution information as the image acquired by the array cameras 33 and 53. The control unit 180 may calculate the three-dimensional depth value using the converted image, and may correct the three-dimensional depth value calculated using the array cameras 33 and 53.

Through this, the user can acquire the 3D depth information using the array camera and receive the application accordingly. In addition, more accurate three-dimensional information can be provided by using the 3D depth information calculated using the image acquired by the main sensor.

In FIG. 14, first and second sub-sensors 34 and 54 are shown which are composed of an array camera consisting of two sensors according to an example. As shown in FIG. 14, the first and second sub-sensors 34 and 54 may be implemented by sensors 34a and 54a and black and white sensors 34b and 54b, respectively, to which one green filter is applied. However, this is an example, and if necessary, a red filter or a blue filter may be used in addition to a green filter and a black filter.

The control unit 180 can calculate the three-dimensional depth information by using the sensor to which the green filter is applied by the first and second sub sensors 34 and 54 in an environment of good illuminance. The controller 180 can calculate the 3D depth information using the monochrome sensor in the first and second sub sensors 34 and 54 in the low illumination environment.

14, the correction and application of the three-dimensional depth information using the main sensor 10 described above can be practically applied in the same manner, and a detailed description thereof will be omitted here.

By applying the sub sensor having the simple structure shown in FIG. 14, the user can easily receive the three-dimensional depth information regardless of the illumination environment.

Referring to FIG. 15, the first sub-sensor 35 and the second sub-sensor 55 may be sensors composed of any one of RGB channels. FIG. 15 shows a sensor to which red and green filters are applied, but the present invention is not limited thereto. If desired, the first and second sub-sensors 35 and 55 may be implemented in various combinations of RGB. For example, the first sub-sensor 35 may be implemented with a Blue filter, and the second sub-sensor 55 may be implemented with a sensor to which a Red filter is applied.

The controller 180 can quickly calculate three-dimensional depth information by using a sensor having a simple structure when compared with the first and second sub sensors 31 and 51 shown in FIG. In this case, the implementation cost of the sensor can be lowered. Further, the desired color effect can be further added to the image acquired through the main sensor 10 according to the RGB filter applied to the sub-sensor.

In the above description, the acquisition and correction of the three-dimensional depth information according to the types of the first and second sub-sensors are described, but the present invention is not limited thereto. Although the first and second sub-sensors are shown to be of the same type, the present invention is not limited thereto, and the sub-sensors described above may be combined. In addition, although two sub-sensors have been described as a reference, substantially the same three or more sub-sensors may be applied.

17 is a flowchart of a method of controlling a mobile terminal according to another embodiment of the present invention. 18 to 22 are diagrams for explaining acquisition of 3D depth information by applying infrared illumination according to an embodiment of the present invention.

The control method of the mobile terminal according to another embodiment of the present invention can be implemented in the mobile terminal 100 described with reference to FIG. Hereinafter, the control method of the mobile terminal according to another embodiment of the present invention and the operation of the mobile terminal 100 for implementing the same will be described in detail with reference to the necessary drawings.

Referring to FIG. 17, the controller 180 may acquire a main image including at least one object through the main sensor 10 (S200).

Referring to FIG. 18, the camera 121 may include a main sensor 10, a sub sensor 53, and an infrared light 70. Acquisition of the main image in the main sensor 10 is substantially the same as that in Fig. 2, and thus a further explanation will be omitted.

Referring again to FIG. 17, the controller 180 may acquire a sub-image including the at least one object with a lower resolution than the main image using the sub-sensor 53 (S210).

Referring to FIG. 18, the sub-sensor 53 may be implemented as a set of four (2 * 2) independent sensors as an array camera, as described above with reference to FIG. The sub-sensor 53 may be constituted by an independent sensor to which RGB and black-and-white filters are applied, respectively.

The control unit 180 may acquire the sub image including the at least one object at a lower resolution than the main image by using the sensors 53a to 53c to which the RGB filters are applied.

Referring again to FIG. 17, the control unit 180 may project infrared rays to the at least one object using the infrared ray illumination 70 (S220).

Referring to FIG. 18, the infrared light 70 may be positioned in a direction different from the sub-sensor 53 about the main sensor 10. However, as an example, the description of the arrangement of the first and second sub-sensors described in Figs. 7 to 10 may be applied substantially equally to the arrangement of the infrared light 70 and the sub-sensor 53 .

The infrared light 70 may be a light source capable of projecting infrared light to a specific object, and is not limited to a specific light source. Further, according to one example, the infrared light 70 can project the structured light with a predetermined pattern added thereto.

Referring again to FIG. 17, the controller 180 may detect the infrared image reflected by the at least one object using the monochrome sensor included in the sub-sensor 53 (S230).

Referring to FIG. 18, the sub-sensor 53 may include a monochrome sensor 53d for an infrared channel. The monochrome sensor 53d can sense infrared rays. Since the sub sensor 53 is substantially the same in the description of FIG. 13, a detailed description thereof will be omitted.

Referring again to FIG. 17, the controller 180 may calculate the 3D depth information of the at least one object using the sensed infrared image (S240).

The control unit 180 may calculate the 3D depth information in the image by combining the image obtained using the monochrome sensor 53d included in the sub sensor 53 with the image acquired from the main sensor 10. [ 14 may be practically the same, so a detailed description thereof will be omitted.

According to an example, the controller 180 may acquire three-dimensional depth information using the infrared light 70 projecting the structured light to which a predetermined pattern is added. Referring to FIG. 19, a method of measuring three-dimensional depth using structured light according to an example is shown.

19, b means a baseline which is the distance between the infrared light 310 and the lens 320 in the monochrome sensor. The reference plane 330 refers to any plane that is set as a reference in measuring the depth of a specific object. ZO corresponds to the distance between the mobile terminal 100 and the reference plane 330 and can be predetermined.

f is the focal length and corresponds to the distance between the lens 320 in the monochrome sensor and the image plane 350 of the imaging device. The light projected from the infrared light 310 enters through the lens 320 and is converted into an electrical image signal in the image sensing apparatus. The imaging device can be applied to any device capable of converting light such as a charge-coupled device (CCD) into a video signal.

The reference image, which is a specific reference in the three-dimensional depth measurement, is configured to project the structured light in the infrared light 310 with respect to an arbitrary reference plane 330, detect the structured light reflected from the reference plane in a monochrome sensor, . ≪ / RTI >

19, in order to obtain the reference image, the structured light is projected onto the reference plane 330 spaced apart by a predetermined distance ZO from the infrared light 310. The projected structured light is reflected by the reference plane 330 and is detected by the monochrome sensor.

An image of the pattern of the structured light projected on the reference plane 330 becomes the reference image. In this case, the reference image may be generated in advance and stored in the memory 170.

In order to measure the three-dimensional depth of the object 340, structured light having the same pattern as that used when acquiring the reference image may be projected onto the object 340. In FIG. 19, the object 340 is shown as one plane, but this is for the convenience of explanation. Actually, the structured light will be projected on each surface constituting the object 340.

When the structured light reflected by the object 340 is obtained from the monochrome sensor, the controller 180 may measure the positional shifts of the corresponding patterns in the obtained image based on the reference image. The movement of the pattern corresponds to d in Fig. 19, which means disparity corresponding to k points of the object 340 with reference to the reference image.

According to an example, it can be seen that the relationship of D / b = (ZO-ZK) / ZO, d / f = D / ZK is obtained by using the resemblance of triangle in FIG. The above two equations can be summarized as follows.

As described above, in the above equation (1), f is a focal length, b is a base line, and ZO is a distance to the reference plane 330. Thus, if the parallax d for an arbitrary point k of the object 340 is measured in the acquired image, the 3D depth ZK, which is the distance to the object 340, can be calculated.

Referring again to FIG. 17, the controller 180 may correct the calculated 3D depth information using the main image and the sub-image (S250).

The control unit 180 can correct the three-dimensional depth information calculated using the infrared light 70 and the monochrome sensor 53d using the three-dimensional depth information calculated using the main image and the sub-image. The subsequent correction of the three-dimensional depth information and adjustment and application of the three-dimensional characteristics of the image through the correction are substantially the same as those of the step S140 described in Figs. 2 to 16, and thus a detailed description thereof will be omitted.

Accordingly, the user can acquire an image at high resolution using the main sensor, acquire three-dimensional depth in the obtained image using infrared illumination, The 3D depth can be further subdivided. Also, by using the structured optical system, the depth information of the planar region in which the tecture information is lacking in the image acquired by the main sensor can be obtained more precisely.

Referring to FIG. 20, there is shown a sub-sensor 54 composed of an array camera composed of two sensors according to an example. As shown in FIG. 20, the sub-sensor 54 may be implemented by a sensor 54a and a monochrome sensor 54b, respectively, to which one green filter is applied. However, this is an example, and if necessary, a red filter or a blue filter may be used in addition to a green filter and a black filter.

The control unit 180 can calculate the 3D depth information using the image obtained using the sensor 54a to which the green filter is applied and the image obtained from the main sensor 10 in the sub sensor 54. [ When the structure light projected from the infrared light 70 is detected by the monochrome sensor 54b, the controller 180 can calculate the 3D depth information using the sensed image.

20, the correction and application of the three-dimensional depth information using the main sensor 10 described above can be practically applied in the same manner, and a detailed description thereof will be omitted here.

By applying the sub sensor having the simple structure shown in FIG. 20, the user can easily receive the three-dimensional depth information calculated by applying the stereoscopic vision system and the structural optical system scheme together.

Referring to Fig. 21, the sub-sensor 52 may be a monochrome sensor. In the case of a monochrome sensor, infrared light projected from the infrared light 70 can be sensed to acquire an image. The control unit 180 can control the three-dimensional characteristic of the image obtained by the main sensor 10 using the three-dimensional depth value calculated from the image obtained by the monochrome sensor 52. [

Through this, the user can acquire a high-resolution image through the main sensor and control the three-dimensional characteristics of the high-resolution image acquired by the main sensor using the 3D depth information calculated using the structural optical system .

Referring to FIG. 22, the sub-sensor 56 may be implemented as an array camera of color channels. 22, the sub sensor 56 may be implemented as an array camera including a sensor 56a to which a green filter is applied, a sensor 56b to which a blue filter is applied, and a sensor 56c to which a red filter is applied . However, this may be implemented in other RGB combinations as required, for example.

When the sub-sensor 56 is implemented as an array camera of color channels, the sub-sensor 56 can further detect the infrared wavelength as RGB color. The control unit 180 may calculate the three-dimensional depth information of the image obtained by the main sensor 10 using the infrared-detected image. In addition, the color information of the image obtained by the main sensor 10 can be controlled using information according to infrared rays.

In the case of the configuration shown in FIG. 22, the details that can be substantially applied to the correction and application of the three-dimensional depth information using the main sensor 10 are not further described.

FIGS. 23 to 27 illustrate acquisition of three-dimensional depth information by further including infrared illumination on two sub-sensors according to an embodiment of the present invention.

Referring to FIG. 23, the first and second sub-sensors 31 and 51 may be an RGB Bayer sensor in the same manner as the main sensor 10. As shown in FIG. 23, the first and second sub-sensors 31 and 51 may be sensors having a lower resolution as compared with the main sensor 10. That is, in the case of the main sensor 10, the main purpose is to obtain a high-resolution image, and the sub-sensors 31 and 51 can mainly calculate the three-dimensional depth information.

In addition, the camera 121 may further include an infrared light (70). With the addition of the infrared illumination 70, the first and second sub-sensors 31 and 51 can further detect the infrared wavelength, such as RGB color. The control unit 180 may calculate the three-dimensional depth information of the image obtained by the main sensor 10 using the infrared-detected image. In addition, the color information of the image obtained by the main sensor 10 can be controlled using information according to infrared rays.

The main sensor 10 and the first and second sub sensors 31 and 51 in FIG. 23 are substantially the same as those described with reference to FIG. 11, and a detailed description thereof will be omitted.

As shown in FIG. 23, when the first and second sub-sensors 31 and 51 are implemented by RGB Bayer sensors, the controller 180 uses the first and second sub-sensors 31 and 51 It is possible to calculate three-dimensional depth information.

7, the controller 180 may calculate the 3D depth information using the images obtained by the first and second sub sensors 31 and 51 and the main sensor 10, respectively . To this end, the controller 180 may convert the image acquired by the main sensor 10 into the same resolution information as that of the first and second sub sensors 31 and 51.

The controller 180 calculates the three-dimensional depth information calculated using the first and second sub sensors 31 and 51 by using the first and second sub sensors 31 and 51 and the main sensor 10 And can be corrected using a three-dimensional depth information.

In this case, when the specific surface or the like of the same object is acquired only by one of the sub sensors, the control unit 180 calculates the output of the main sensor 10 by using the first and second sub sensors 31 and 51 and the main sensor 10 It is possible to compensate it by using a 3D depth information.

In addition, the controller 180 can improve the quality of the image acquired by the main sensor 10 by using the pixel information of the image acquired by the first and second sub sensors 31 and 51. In this case, the controller 180 may improve the quality of the image acquired by the main sensor 10 by applying a known super resolution algorithm.

Also, the controller 180 may perform the refocusing function using the 3D depth information calculated for the image acquired by the main sensor 10. [ The control unit 180 can adjust the focus of any part of the image acquired by the main sensor 10 as if it were in focus.

The control unit 180 controls the first and second sub sensors 31 and 51 and the first and second sub sensors 31 and 51 using an image sensed by the first and second sub sensors 31 and 51, The three-dimensional depth information calculated using the main sensor 10 can be corrected. On the other hand, the controller 180 controls the infrared rays projected from the infrared ray lamp 70 to the first (first) and second Dimensional depth information calculated using the images sensed by the second sub-sensors 31 and 51 can be corrected.

Accordingly, the controller 180 can acquire more precise 3D depth information, and thus, the user can be provided with various applications such as refocusing based on 3D depth information.

Referring to FIG. 24, the first and second sub-sensors 32 and 52 may be monochrome sensors. In the case of a monochrome sensor, an image can be obtained even in a low-illuminance environment. The control unit 180 can display the image of the main sensor 10 obtained in the low illumination environment using the three-dimensional depth value calculated from the image obtained by the monochrome sensor 32,

In addition, the controller 180 may convert the image acquired by the main sensor 10 into the same channel and resolution information as the image acquired by the monochrome sensor 32, 52. The control unit 180 may calculate the three-dimensional depth value using the converted image, and may correct the three-dimensional depth value calculated using the black and white sensors 32 and 52. [

According to one example, when using the black and white sensors 32 and 52, a relatively simple algorithm can be applied in calculating the three-dimensional depth information, and the controller 180 can quickly calculate the three-dimensional depth information.

The control unit 180 controls the first and second sub sensors 32 and 52 and the first and second sub sensors 32 and 52 using the image sensed by the first and second sub sensors 32 and 52, The three-dimensional depth information calculated using the main sensor 10 can be corrected. On the other hand, the control unit 180 controls the infrared rays projected from the infrared ray illumination 70 to the first (first) and the second (second) subspace using the three-dimensional depth information calculated using the first and second sub- , It is possible to correct the three-dimensional depth information calculated using the image sensed by the second sub-sensor 32 or 52.

Through this, the user can be provided with an improved image even in a low-illuminated environment. Further, by using a monochrome sensor, it is possible to more easily calculate three-dimensional depth information. In addition, more accurate three-dimensional depth information can be calculated by further adding infrared light, and the three-dimensional characteristics of the image can be precisely displayed.

Referring to FIG. 25, the first and second sub-sensors 33 and 53 may be array cameras. The first and second sub-sensors 33 and 53 may be implemented as a set of four (2 * 2) independent sensors as array cameras. 13, each of the first and second sub sensors 33 and 53 includes independent sensors 33a to 33c and 53a to 53c to which RGB filters are applied and independent sensors 33d and 53d to which a black and white filter is applied Lt; / RTI > This can be distinguished from the fact that the first and second sub-sensors 31 and 51 of FIG. 11 are applied to the RGB filter in pixel units in one sensor. However, this is an example, and the form of the array camera is not limited thereto. And may be configured with a different number of sensors as needed.

The controller 180 can acquire three-dimensional depth information using sensors in the array camera constituting the first and second sub-sensors 33 and 53. In addition, the controller 180 may combine sensors having baselines between the first and second sub-sensors 33 and 53 to obtain three-dimensional depth information. When some sensor of the array camera is implemented as a monochrome sensor, the control unit 180 can easily acquire three-dimensional depth information in a low-illuminance environment.

In addition, the controller 180 may convert the image acquired by the main sensor 10 into the same channel and resolution information as the image acquired by the array cameras 33 and 53. The control unit 180 may calculate the three-dimensional depth value using the converted image, and may correct the three-dimensional depth value calculated using the array cameras 33 and 53.

The controller 180 controls the first and second sub sensors 33 and 53 and the main sensor 10 by using the image sensed by the black and white sensors 33d and 53d with the infrared rays projected from the infrared light 70. [ Dimensional depth information calculated using the three-dimensional depth information. Conversely, the controller 180 controls the infrared light projected from the infrared light 70 to the black-and-white sensor 70 using the three-dimensional depth information calculated using the first and second sub-sensors 33 and 53 and the main sensor 10, Dimensional depth information calculated using the images sensed by the three-dimensional depth sensors 33d and 53d.

Through this, the user can acquire the 3D depth information using the array camera and receive the application accordingly. In addition, more accurate three-dimensional information can be provided by using the 3D depth information calculated using the image acquired by the main sensor. In addition, more accurate three-dimensional depth information can be calculated by further adding infrared light, and the three-dimensional characteristics of the image can be precisely displayed.

In FIG. 26, first and second sub-sensors 34 and 54, each of which is an array camera composed of two sensors, are shown according to an example. As shown in FIG. 26, the first and second sub-sensors 34 and 54 may be implemented by sensors 34a and 54a and black and white sensors 34b and 54b, respectively, to which one green filter is applied. However, this is an example, and if necessary, a red filter or a blue filter may be used in addition to a green filter and a black filter.

The controller 180 can calculate the 3D depth information using the sensors 34a and 54a to which the Green filter is applied by the first and second sub sensors 34 and 54 in an environment with good illumination. The controller 180 can calculate three-dimensional depth information using the black and white sensors 34b and 54b in the first and second sub-sensors 34 and 54 in a low-illuminance environment.

26, the correction and application of the three-dimensional depth information using the main sensor 10 described above can be practically applied in the same manner, and thus a detailed description thereof will be omitted.

The control unit 180 uses the images sensed by the monochrome sensors 34b and 54b with infrared rays projected from the infrared light 70 and uses the sensors 34a and 54a to which the green filter is applied and the main sensor 10 Thereby correcting the calculated three-dimensional depth information. On the other hand, the control unit 180 transmits the infrared rays projected from the infrared light 70 to the monochrome sensor 34b (34b) by using the three-dimensional depth information calculated using the sensors 34a and 54a to which the Green filter is applied and the main sensor 10, And 54b can be used to correct the calculated 3D depth information.

By applying the sub-sensor having the simple structure shown in FIG. 25, the user can easily receive the three-dimensional depth information regardless of the illumination environment. In addition, more accurate three-dimensional depth information can be calculated by further adding infrared light, and the three-dimensional characteristics of the image can be precisely displayed.

Referring to FIG. 27, the first sub sensor 35 and the second sub sensor 55 may be a sensor composed of any one of RGB channels. FIG. 27 shows a sensor to which red and green filters are applied, but is not limited thereto. If desired, the first and second sub-sensors 35 and 55 may be implemented in various combinations of RGB. For example, the first sub-sensor 35 may be implemented with a Blue filter, and the second sub-sensor 55 may be implemented with a sensor to which a Red filter is applied.

The control unit 180 controls the first and second sub sensors 35 and 55 and the first and second sub sensors 35 and 55 using the image sensed by the first and second sub sensors 35 and 55, The three-dimensional depth information calculated using the main sensor 10 can be corrected. On the other hand, the controller 180 controls the infrared rays projected from the infrared ray lamp 70 to the first (first) and second Dimensional depth information calculated using the image sensed by the second sub-sensor 35 or 55 can be corrected.

Since the correction of the three-dimensional depth information is substantially the same as the above description, a detailed description thereof will be omitted.

The controller 180 can quickly calculate three-dimensional depth information using a sensor of a simple structure when compared with the first and second sub sensors 31 and 51 shown in Fig. In this case, the implementation cost of the sensor can be lowered. Further, the desired color effect can be further added to the image acquired through the main sensor 10 according to the RGB filter applied to the sub-sensor. In addition, more accurate three-dimensional depth information can be calculated by further adding infrared light, and the three-dimensional characteristics of the image can be precisely displayed.

The control unit 180 may adjust the three-dimensional characteristics of the main image based on the three-dimensional depth information of the at least one object included in the main image acquired by the main sensor 10. [

The three-dimensional characteristic may be a compensation of an occlusion in which depth resolution enhancement, spatial resolution enhancement, image quality improvement using a super-resolution algorithm, refocusing in an image, specific surface of the same object, Acquisition of depth information of a planar region using a structured light pattern, and the like. However, the present invention is not limited thereto, and any three-dimensional characteristic that can be adjusted using three-dimensional depth information can be used.

The present invention described above can be embodied as computer-readable codes on a medium on which a program is recorded. The computer readable medium includes all kinds of recording devices in which data that can be read by a computer system is stored. Examples of the computer readable medium include a hard disk drive (HDD), a solid state disk (SSD), a silicon disk drive (SDD), a ROM, a RAM, a CD-ROM, a magnetic tape, a floppy disk, , And may also be implemented in the form of a carrier wave (e.g., transmission over the Internet). Also, the computer may include a control unit 180 of the terminal. Accordingly, the above description should not be construed in a limiting sense in all respects and should be considered illustrative. The scope of the present invention should be determined by rational interpretation of the appended claims, and all changes within the scope of equivalents of the present invention are included in the scope of the present invention.

100: mobile terminal 110: wireless communication unit
120: input unit 140: sensing unit
150: output unit 160: interface unit
170: memory 180:
190: Power supply

Claims (17)

A main sensor for acquiring a main image including at least one object;
A first sub sensor for acquiring a first image including the at least one object at a first resolution lower than the resolution of the main sensor;
A second sub-sensor for acquiring a second image including the at least one object at a second resolution lower than the resolution of the main sensor; And
Dimensional depth information of the at least one object included in the main image using the first image and the second image, and calculates the three-dimensional depth information of the at least one object included in the main image and the first image or the main image and the second image Dimensional depth information of the at least one object based on the 3D depth information calculated using the 3D depth information;
. The method according to claim 1,
Wherein the main sensor, the first sub-sensor, and the second sub-sensor are implemented as a single module. The method according to claim 1,
Wherein the first sub-sensor and the second sub-sensor are positioned symmetrically with respect to the main sensor. The method according to claim 1,
Wherein the first sub-sensor and the second sub-sensor are positioned at different distances with respect to the main sensor. The method according to claim 1,
Wherein the first sub-sensor and the second sub-sensor are positioned on different straight lines with respect to the main sensor. The method according to claim 1,
Wherein the first sub-sensor and the second sub-sensor are RGB Bayer sensors. The method according to claim 1,
Wherein the first sub-sensor and the second sub-sensor are monochrome sensors. The method according to claim 1,
Wherein the first sub-sensor and the second sub-sensor are array cameras. The method according to claim 1,
Wherein the first sub-sensor and the second sub-sensor are sensors composed of any one of RGB channels. 10. The method according to any one of claims 1 to 9,
An infrared light illuminating the at least one object with infrared light;
Further comprising:
Wherein at least one of the first sub sensor or the second sub sensor senses an infrared image reflected from the at least one object. The method according to claim 1,
Wherein the controller adjusts the three-dimensional characteristics of the main image based on three-dimensional depth information of the at least one object included in the main image. A main sensor for acquiring a main image including at least one object;
An infrared light illuminating the at least one object with infrared light;
A sub sensor for acquiring a sub image including the at least one object at a resolution lower than the resolution of the main sensor and for sensing an infrared image reflected by the at least one object; And
Dimensional depth information of the at least one object using the sensed infrared image, and correcting the calculated three-dimensional depth information using the main image and the sub-image;
. 13. The method of claim 12,
Wherein the infrared illumination projects the structured light to which the predetermined pattern is added to the object. 13. The method of claim 12,
Wherein the sub-sensor comprises an infrared channel. 13. The method of claim 12,
Wherein the controller adjusts the three-dimensional characteristics of the main image based on three-dimensional depth information of the at least one object included in the main image. Obtaining a main image including at least one object;
Obtaining a first image including the at least one object at a first resolution lower than the resolution of the main image;
Obtaining a second image including the at least one object at a second resolution lower than the resolution of the main image;
Calculating three-dimensional depth information for the at least one object included in the main image using the first image and the second image; And
Correcting the three-dimensional depth information for the at least one object based on the three-dimensional depth information calculated using the main image and the first image or the main image and the second image;
And transmitting the control information to the mobile terminal. Obtaining a main image including at least one object;
Obtaining a subimage including the at least one object at a lower resolution than the main image;
Projecting infrared light onto the at least one object;
Sensing an infrared image reflected from the at least one object;
Calculating three-dimensional depth information for the at least one object using the sensed infrared image; And
Correcting the calculated three-dimensional depth information using the main image and the sub-image;
And transmitting the control information to the mobile terminal.

Images