WO2020085525A1

WO2020085525A1 - Mobile terminal and method for controlling same

Info

Publication number: WO2020085525A1
Application number: PCT/KR2018/012562
Authority: WO
Inventors: 송민우; 안예한; 진동철; 김민호; 전상국
Original assignee: 엘지전자 주식회사
Priority date: 2018-10-23
Filing date: 2018-10-23
Publication date: 2020-04-30

Abstract

The purpose of the present invention is to provide a time of flight (TOF) camera operating in a first or second mode to reduce power consumption. To this end, a time of flight (TOF) camera according to the present invention is characterized by comprising: a light source for emitting a light source onto an object; an image sensor for receiving light reflected from the object and obtaining intensity data; an operation unit for processing the intensity data; and a controller connected to the light source, the image sensor, and the operation unit, wherein the controller controls the light source to emit a light source of a periodic single pulse in the first mode, switches to the second mode when the operation unit detects a change in the intensity data, controls the light source to periodically vary the phase of the single pulse in the second mode, and controls the operation unit to generate depth data on the object through the intensity data.

Description

Mobile terminal and control method thereof

The present invention relates to a control method for a mobile terminal. More specifically, it is applicable to the technical field of driving at least one camera (for example, a depth camera) included in the mobile terminal with low power.

Terminals can be divided into mobile terminals (mobile / portable terminals) and stationary terminals according to their mobility. Again, the mobile terminal may be divided into a handheld terminal and a vehicle mounted terminal according to whether the user can directly carry it.

The functions of mobile terminals are diversifying. For example, there are functions for data and voice communication, photo and video shooting through a camera, voice recording, music file playback through a speaker system, and image or video output on the display. In some terminals, an electronic game play function is added or a multimedia player function is performed. In particular, recent mobile terminals can receive multicast signals that provide visual content such as broadcast and video or television programs.

As the function is diversified, such a terminal is a multimedia player having multiple functions, such as taking a picture or a video, playing music or a video file, receiving a game, or broadcasting. Is being implemented.

The complex functions of the mobile terminal can be performed without the user directly touching the mobile terminal. The user may interact without directly touching the mobile terminal through a voice or vision-based user interface provided in the mobile terminal.

The user interface based on the touch input cannot be interacted with when the user is away from the mobile terminal. Also, in the process of touch input, the position of the mobile terminal in the stand state may be changed or may be overturned. However, a voice or vision-based user interface may be advantageous over a touch input-based user interface in that a user can interact while away from the mobile terminal and does not apply any external force to the mobile terminal.

The voice-based user interface may be limited by space and time. For example, in a noisy space, it may be difficult for the mobile terminal to discern the user's voice. In addition, the user may be inconvenient to interact with the mobile terminal through a voice at the late night or with another person.

Accordingly, the vision-based user interface may provide convenience for the user to interact with the mobile terminal in the limited situation as described above. Accordingly, recently, a technology in which a mobile terminal interacts with a user through a vision-based user interface is actively being researched.

With a vision-based user interface, the mobile terminal can be equipped with a depth camera. However, in order to recognize the user's movement using the depth camera, there is a problem in that power consumption of the mobile terminal is unnecessary since the depth camera must always be kept in the ON state.

The present invention aims to solve the above-mentioned problems and other problems through the specification of the present invention.

The present invention intends to propose a solution in which a mobile terminal provides a user interface (UI) based on vision instead of touch input through a depth camera.

The present invention is to provide a solution to prevent excessive power consumption by varying the mode depending on the condition while the depth camera included in the mobile terminal is always ON.

The present invention is to provide a solution for determining the presence or absence of a user with low power and obtaining a depth image through a Time Of Flight (TOF) camera included in a mobile terminal.

To achieve the above object, according to an embodiment of the present invention, a TOF (Time Of Flight) camera operating in a first or second mode receives a light source irradiating a light source to an object, and receives light reflected from the object An image sensor for obtaining raw data, an operation unit for processing the raw data, and a controller connected to the light source, the image sensor, and the operation unit, wherein the controller is a light source of a specific signal in the first mode. The light source is controlled to irradiate, and when the change of the raw data is sensed through the operation unit, the device is switched to the second mode, and the light source is controlled to vary the phase of the specific signal in the second mode. , Controlling the operation unit to generate depth data of the object through the raw data Is done.

In addition, according to an embodiment of the present invention for achieving the above object, the image sensor includes first and second photogates to which signals having different phases are applied and receive reflected light corresponding thereto, The controller controls the image sensor to acquire the raw data through the first photo gate in the first mode, and the image to acquire the raw data through the first and second photo gates in the second mode. It is characterized by controlling the sensor.

In addition, according to an embodiment of the present invention for achieving the above object, the operation unit is a sub-operation unit for determining the intensity change of the raw data obtained in the first mode, and the obtained in the second mode It characterized in that it comprises a main operation unit for generating the depth data through the intensity ratio of the raw data.

In addition, according to an embodiment of the present invention for achieving the above object, when the controller detects an intensity change of the raw data in the sub-operation unit, it is characterized in that the main operation unit wakes up do.

In addition, according to an embodiment of the present invention for achieving the above object, the controller is characterized in that the specific signal is periodically varied to have a phase difference of 0 degrees, 90 degrees, 180 degrees and 270 degrees in the second mode Is done.

In addition, according to an embodiment of the present invention for achieving the above object, the controller is characterized in that when the depth data does not change over a preset time in the second mode, the first mode.

A mobile terminal according to an embodiment of the present invention for achieving the above object operates in a first or second mode, a light source for irradiating light, an image sensor for receiving reflected light, and obtaining raw data, and A TOF (Time Of Flight) camera including a calculation unit for generating depth data through the raw data, a memory storing a command corresponding to the depth data, a display providing graphic feedback in response to the command stored in the memory, The controller includes a TOF camera, a memory, and a controller connected to the display. The controller controls the light source to irradiate a light source of a specific signal in the first mode, and the raw data is calculated by the operation unit. ), If it detects a change, it switches to the second mode, and the specific signal in the second mode. Control the light source to change the phase of the, control the TOF camera to obtain the depth data through the raw data, and when the provided graphic feedback ends, switch from the second mode to the first mode It is characterized by.

In addition, according to an embodiment of the present invention for achieving the above object, the image sensor is applied with a signal having a different phase during a preset frame, and correspondingly, first and second photogates receiving the reflected light And in the first mode, the raw data is obtained through the first photo gate, and in the second mode, the raw data is obtained through the first and second photo gates.

In addition, according to an embodiment of the present invention for achieving the above object, the operation unit is a sub-operation unit for determining the intensity change of the raw data obtained in the first mode, the raw data obtained in the second mode It characterized in that it comprises a main operation unit for calculating the intensity ratio, and thereby generating the depth data.

A method of controlling a mobile terminal including a time of flight (TOF) camera according to an embodiment of the present invention for achieving the above object, by operating the time of flight (TOF) camera in a first mode, moving Checking the existence of the object, and if the existence of the moving object in the first mode is confirmed, operating the TOF camera in a second mode to obtain depth data of the object, wherein the first mode includes The TOF camera irradiates a light source of a specific signal to an object, receives a light source reflected by the object, obtains raw data, detects a change in the raw data, determines the existence of the moving object, The second mode irradiates a light source by varying the phase of the specific signal in the TOF camera, and the light source reflected by the object Receiving data to obtain a row, and is characterized in that for obtaining the depth data of the objects through the raw data.

In addition, according to an embodiment of the present invention for achieving the above object, providing a graphic feedback corresponding to the depth data, and when the provided graphic feedback is terminated, the first mode in the second mode It characterized in that it comprises a step of switching to.

The effects of the mobile terminal and its control method according to the present invention are as follows.

According to at least one of the embodiments of the present invention, a user may interact with a mobile terminal through a user's movement instead of a touch input.

According to at least one of the embodiments of the present invention, the TOF camera included in the mobile terminal may be operated in different modes, and thus may consume excessive power.

According to at least one of the embodiments of the present invention, the presence or absence of a user may be checked with low power through a TOF camera included in a mobile terminal, and a depth image may be acquired to prevent excessive power consumption.

1A is a block diagram illustrating a mobile terminal related to the present invention.

1B and 1C are conceptual views of an example of a mobile terminal related to the present invention viewed from different directions.

2 to 6 are views for explaining the driving method of the TOF camera.

7 is a view for explaining the problem of the existing TOF camera 200 and the effect of the TOF camera according to an embodiment of the present invention which improves the problem.

8 is a schematic view for explaining the configuration of a TOF camera according to an embodiment of the present invention.

9 is a flowchart illustrating a control method for controlling a TOF camera according to an embodiment of the present invention.

FIG. 10 is a flowchart illustrating a method of determining the presence or absence of an object by comparing frames in FIG. 9.

11 is a view for explaining FIG. 9.

12 is a schematic diagram for explaining the configuration of a TOF camera according to another embodiment of the present invention.

13 is a view for explaining a method and a problem of recognizing a user's finger through a depth image acquired by a conventional depth camera.

14 is a diagram for schematically explaining a method in which a mobile terminal according to an embodiment of the present invention recognizes a point targeted by a user's hand through a depth camera.

15 is a flow chart for explaining FIG. 14.

FIG. 16 is a view for explaining an interaction region in FIG. 15.

17 is a diagram for describing a depth image acquired in the interaction area of FIG. 16.

18 is a flowchart illustrating a method of implementing a segment tree corresponding to a user's hand in FIG. 15.

19 to 22 are views for explaining FIG. 18.

23 is a flowchart illustrating a method of determining an effective end through a segment tree in FIG. 15.

24 is a view for explaining FIG. 23.

25 is a flowchart illustrating a method of extracting a targeting point and depth corresponding to the effective end determined in FIG. 15.

26 and 27 are views for explaining FIG. 25.

28 to 30 are diagrams for explaining video feedback provided corresponding to the targeting point extracted through FIG. 15.

Hereinafter, exemplary embodiments disclosed herein will be described in detail with reference to the accompanying drawings, but the same or similar elements are assigned the same reference numbers regardless of the reference numerals, and overlapping descriptions thereof will be omitted. The suffixes "modules" and "parts" for the components used in the following description are given or mixed only considering the ease of writing the specification, and do not have meanings or roles distinguished from each other in themselves. In addition, in describing the embodiments disclosed in the present specification, when it is determined that detailed descriptions of related known technologies may obscure the gist of the embodiments disclosed herein, detailed descriptions thereof will be omitted. In addition, the accompanying drawings are only for easy understanding of the embodiments disclosed in the present specification, and the technical spirit disclosed in the specification is not limited by the accompanying drawings, and all modifications included in the spirit and technical scope of the present invention , It should be understood to include equivalents or substitutes.

Mobile terminals described herein include mobile phones, smart phones, laptop computers, digital broadcasting terminals, personal digital assistants (PDAs), portable multimedia players (PMPs), navigation, and slate PCs. , Tablet PC (tablet PC), ultrabook (ultrabook), wearable device (wearable device, for example, a watch-type terminal (smartwatch), glass-type terminal (smart glass), HMD (head mounted display), etc. may be included have.

However, the configuration according to the embodiment described in the present specification may be applied to a fixed terminal such as a digital TV, a desktop computer, and a digital signage, except when applicable only to a mobile terminal. It is easy for a technician to know.

1A to 1C, FIG. 1A is a block diagram illustrating a mobile terminal related to the present invention.

The mobile terminal 100 includes a wireless communication unit 110, an input unit 120, a sensing unit 140, an output unit 150, an interface unit 160, a memory 170, a control unit 180, and a power supply unit 190 ) And the like. The components shown in FIG. 1A are not essential for implementing the mobile terminal 100, so the mobile terminal 100 described herein may have more or fewer components than those listed above. have.

More specifically, the wireless communication unit 110 among the components, between the mobile terminal 100 and the wireless communication system, between the mobile terminal 100 and another mobile terminal 100, or the mobile terminal 100 and an external server It may include one or more modules that enable wireless communication between. Also, the wireless communication unit 110 may include one or more modules connecting the mobile terminal 100 to one or more networks.

The wireless communication unit 110 may include at least one of a broadcast receiving module 111, a mobile communication module 112, a wireless Internet module 113, a short-range communication module 114, and a location information module 115. .

The input unit 120 may include a camera 121 for inputting a video signal or a video input unit, a microphone for inputting an audio signal (microphone 122), or an audio input unit, a user input unit 123 for receiving information from a user, for example , A touch key, a push key, and the like. The voice data or image data collected by the input unit 120 may be analyzed and processed by a user's control command.

The sensing unit 140 may include one or more sensors for sensing at least one of information in the mobile terminal 100, surrounding environment information surrounding the mobile terminal 100, and user information. For example, the sensing unit 140 includes a proximity sensor 141, an illumination sensor 142, a touch sensor, an acceleration sensor, a magnetic sensor, and gravity G-sensor, gyroscope sensor, motion sensor, RGB sensor, infrared sensor (IR sensor), fingerprint scan sensor, ultrasonic sensor , Optical sensor (e.g., camera (see 121)), microphone (refer to 122), battery gauge, environmental sensor (e.g. barometer, hygrometer, thermometer, radioactivity sensor, Thermal sensing sensor, gas sensing sensor, etc.), chemical sensors (eg, electronic nose, health care sensor, biometric sensor, etc.). Meanwhile, the mobile terminal 100 disclosed in this specification may combine and use information sensed by at least two or more of these sensors.

The output unit 150 is for generating output related to vision, hearing, or tactile sense, and includes at least one of a display unit 151, an audio output unit 152, a haptic module 153, and an optical output unit 154 can do. The display unit 151 may form a mutual layer structure with the touch sensor or may be integrally formed, thereby realizing a touch screen. The touch screen may function as a user input unit 123 that provides an input interface between the mobile terminal 100 and the user, and at the same time, provide an output interface between the mobile terminal 100 and the user.

The interface unit 160 serves as a passage with various types of external devices connected to the mobile terminal 100. The interface unit 160 connects a device equipped with a wired / wireless headset port, an external charger port, a wired / wireless data port, a memory card port, and an identification module. It may include at least one of a port, an audio input / output (I / O) port, a video input / output (I / O) port, and an earphone port. In the mobile terminal 100, in response to an external device being connected to the interface unit 160, appropriate control related to the connected external device may be performed.

Also, the memory 170 stores data supporting various functions of the mobile terminal 100. The memory 170 may store a number of application programs (application programs) that are driven in the mobile terminal 100, data for operation of the mobile terminal 100, and instructions. At least some of these applications may be downloaded from external servers via wireless communication. In addition, at least some of these application programs may exist on the mobile terminal 100 from the time of shipment for basic functions of the mobile terminal 100 (for example, an incoming call, an outgoing function, a message reception, and an outgoing function). Meanwhile, the application program may be stored in the memory 170 and installed on the mobile terminal 100 to be driven by the controller 180 to perform an operation (or function) of the mobile terminal 100.

The controller 180 controls the overall operation of the mobile terminal 100 in addition to the operations related to the application program. The controller 180 may provide or process appropriate information or functions to the user by processing signals, data, information, etc. input or output through the above-described components or by driving an application program stored in the memory 170.

In addition, the controller 180 may control at least some of the components discussed with reference to FIG. 1A to drive an application program stored in the memory 170. Furthermore, the controller 180 may operate by combining at least two or more of the components included in the mobile terminal 100 for driving the application program.

Under the control of the controller 180, the power supply unit 190 receives external power and internal power to supply power to each component included in the mobile terminal 100. The power supply unit 190 includes a battery, and the battery may be a built-in battery or a replaceable battery.

At least some of the components may operate in cooperation with each other to implement an operation, control, or control method of the mobile terminal 100 according to various embodiments described below. Also, the operation, control, or control method of the mobile terminal 100 may be implemented on the mobile terminal 100 by driving at least one application program stored in the memory 170.

The input unit 120 is for input of image information (or signals), audio information (or signals), data, or information input from a user. For input of image information, the mobile terminal 100 may include one or more A camera 121 may be provided.

The camera 121 may be part of the mobile terminal 100 of the present invention, or may be configured to include the mobile terminal 100. That is, the camera 121 and the mobile terminal 100 of the present invention may include at least some common features or configurations.

The camera 121 processes image frames such as still images or moving images obtained by an image sensor in a video call mode or a shooting mode. The processed image frame may be displayed on the display unit 151 or stored in the memory 170. Meanwhile, the plurality of cameras 121 provided in the mobile terminal 100 may be arranged to form a matrix structure, and through the camera 121 forming the matrix structure, various angles or focuses may be applied to the mobile terminal 100. A plurality of image information may be input. In addition, the plurality of cameras 121 may be arranged in a stereo structure to acquire left and right images for realizing a stereoscopic image.

Meanwhile, the sensing unit 140 senses at least one of information in the mobile terminal 100, surrounding environment information surrounding the mobile terminal 100, and user information, and generates a sensing signal corresponding thereto. The controller 180 may control driving or operation of the mobile terminal 100 based on the sensing signal, or perform data processing, function, or operation related to an application program installed in the mobile terminal 100. Representative sensors among various sensors that may be included in the sensing unit 140 will be described in more detail.

First, the proximity sensor 141 refers to a sensor that detects the presence or absence of an object approaching a predetermined detection surface or an object in the vicinity using mechanical force or infrared rays, etc., without mechanical contact. The proximity sensor 141 may be provided with the proximity sensor 141 in the inner region of the mobile terminal 100 wrapped by the touch screen described above or in the vicinity of the touch screen.

Examples of the proximity sensor 141 include a transmission type photoelectric sensor, a direct reflection type photoelectric sensor, a mirror reflection type photoelectric sensor, a high frequency oscillation type proximity sensor, a capacitive type proximity sensor, a magnetic type proximity sensor, and an infrared proximity sensor. When the touch screen is capacitive, the proximity sensor 141 may be configured to detect the proximity of the object by a change in electric field according to the proximity of the conductive object. In this case, the touch screen (or touch sensor) itself may be classified as a proximity sensor.

On the other hand, for convenience of description, the act of allowing an object to be recognized as being located on the touch screen without being touched on the touch screen is called a “proximity touch”, and the touch The act of actually contacting an object on the screen is called "contact touch". The location where an object is touched close on the touch screen means a location where the object is perpendicular to the touch screen when the object is touched close. The proximity sensor 141 may detect a proximity touch and a proximity touch pattern (eg, proximity touch distance, proximity touch direction, proximity touch speed, proximity touch time, proximity touch position, proximity touch movement state, etc.). have. Meanwhile, as described above, the controller 180 processes data (or information) corresponding to the proximity touch operation and the proximity touch pattern detected through the proximity sensor 141, and further, visual information corresponding to the processed data. It can be output on the touch screen. Furthermore, the controller 180 may control the mobile terminal 100 such that different operations or data (or information) are processed according to whether a touch on the same point on the touch screen is a proximity touch or a touch touch. .

The touch sensor uses a touch (or touch input) applied to the touch screen (or the display unit 151) using at least one of various touch methods such as a resistive film method, a capacitive method, an infrared method, an ultrasonic method, and a magnetic field method. Detect.

As an example, the touch sensor may be configured to convert a change in pressure applied to a specific part of the touch screen or capacitance generated in the specific part into an electrical input signal. The touch sensor may be configured to detect a position, an area, a pressure when a touch is touched, a capacitance when a touch object is applied on the touch screen, and the like. Here, the touch object is an object that applies a touch to the touch sensor, and may be, for example, a finger, a touch pen or a stylus pen, a pointer, and the like.

As such, when there is a touch input to the touch sensor, the signal (s) corresponding thereto is sent to the touch controller. The touch controller processes the signal (s) and then transmits corresponding data to the controller 180. Accordingly, the control unit 180 can know which area of the display unit 151 has been touched, and the like. Here, the touch controller may be a separate component from the controller 180, or may be the controller 180 itself.

Meanwhile, the controller 180 may perform different controls or perform the same control according to the type of the touch object that touches the touch screen (or a touch key provided in addition to the touch screen). Whether to perform different control or the same control according to the type of the touch object, it may be determined according to an operating state of the current mobile terminal 100 or an application program being executed.

On the other hand, the touch sensors and proximity sensors described above are independently or in combination, such as short (or tap) touch on the touch screen, long touch, multi touch, drag touch ), Flick touch, pinch-in touch, pinch-out touch, swipe touch, hovering touch, etc. You can sense the touch.

The ultrasonic sensor may recognize location information of a sensing target using ultrasonic waves. Meanwhile, the control unit 180 may calculate the position of the wave generating source through information sensed by the optical sensor and the plurality of ultrasonic sensors. The position of the wave generator can be calculated by using a property in which light is much faster than ultrasound, that is, a time when light reaches the optical sensor is much faster than a time when the ultrasound reaches the ultrasonic sensor. More specifically, the position of the wave generating source may be calculated by using a time difference from the time at which ultrasonic waves reach the light as a reference signal.

On the other hand, the camera 121, as seen with the configuration of the input unit 120, includes at least one of a camera sensor (eg, CCD, CMOS, etc.), a photo sensor (or image sensor), and a laser sensor.

The camera 121 and the laser sensor may be combined with each other to detect a touch of a sensing target for a 3D stereoscopic image. The photo sensor may be stacked on the display element, which is configured to scan the movement of the sensing object close to the touch screen. More specifically, the photo sensor mounts photo diodes and TRs (transistors) in rows / columns to scan the contents loaded on the photo sensor using electrical signals that change according to the amount of light applied to the photo diode. That is, the photo sensor performs coordinate calculation of the sensing object according to the change amount of light, and through this, location information of the sensing object may be obtained.

1B and 1C, the disclosed mobile terminal 100 has a bar-shaped terminal body. However, the present invention is not limited to this, and may be applied to various structures such as a watch type, a clip type, a glass type, or a folder type, a flip type, a slide type, a swing type, a swivel type to which two or more bodies are movably coupled. . Although it will relate to a particular type of optical device, the description of the specific type of optical device can be applied to other types of optical devices in general.

Here, the terminal body may be understood as a concept of referring to the mobile terminal 100 as at least one aggregate.

The mobile terminal 100 includes a case (for example, a frame, a housing, a cover, etc.) forming an exterior. As illustrated, the mobile terminal 100 may include a front case 101 and a rear case 102. Various electronic components are disposed in the inner space formed by the combination of the front case 101 and the rear case 102. At least one middle case may be additionally disposed between the front case 101 and the rear case 102.

The display unit 151 is disposed on the front of the terminal body to output information. As illustrated, the window 151a of the display unit 151 is mounted on the front case 101 to form the front surface of the terminal body together with the front case 101.

In some cases, electronic components may also be mounted on the rear case 102. Electronic components that can be mounted on the rear case 102 include a removable battery, an identification module, and a memory card. In this case, the rear case 102 may be detachably coupled to the rear cover 103 for covering the mounted electronic components. Therefore, when the rear cover 103 is separated from the rear case 102, the electronic components mounted on the rear case 102 are exposed to the outside.

As shown, when the rear cover 103 is coupled to the rear case 102, a part of the side surface of the rear case 102 may be exposed. In some cases, the rear case 102 may be completely covered by the rear cover 103 when the coupling is performed. Meanwhile, an opening for exposing the camera 121b or the sound output unit 152b to the outside may be provided in the rear cover 103.

These cases (101, 102, 103) may be formed by injection of synthetic resin or may be formed of metal, for example, stainless steel (STS), aluminum (Al), titanium (Ti), or the like.

The mobile terminal 100 may be configured such that one case provides the inner space, unlike the above example in which a plurality of cases provide an inner space accommodating various electronic components. In this case, the mobile terminal 100 of a unibody in which synthetic resin or metal extends from the side to the rear may be implemented.

On the other hand, the mobile terminal 100 may be provided with a waterproof portion (not shown) to prevent water from entering the terminal body. For example, the waterproof portion is provided between the window 151a and the front case 101, between the front case 101 and the rear case 102, or between the rear case 102 and the rear cover 103, the combination thereof It may include a waterproof member for sealing the interior space of the city.

The mobile terminal 100 includes a display unit 151, first and second

sound output units

152a and 152b, a proximity sensor 141, an illuminance sensor 142, a light output unit 154, the first and

second units Cameras

121a and 121b, first and

second operation units

123a and 123b, a microphone 122, and an interface unit 160 may be provided.

Hereinafter, as illustrated in FIGS. 1B and 1C, a display unit 151, a first sound output unit 152a, a proximity sensor 141, an illuminance sensor 142, and an optical output unit on the front surface of the terminal body ( 154), the first camera (121a) and the first operation unit (123a) is disposed, the second operation unit (123b), the microphone 122 and the interface unit 160 is disposed on the side of the terminal body, the terminal body The mobile terminal 100 in which the second sound output unit 152b and the second camera 121b are disposed on the rear surface of the mobile terminal 100 will be described as an example.

However, these configurations are not limited to this arrangement. These configurations may be excluded or replaced as necessary, or may be disposed on different sides. For example, the first operation unit 123a may not be provided on the front of the terminal body, and the second sound output unit 152b may be provided on the side of the terminal body rather than the rear of the terminal body.

The display unit 151 displays (outputs) information processed by the mobile terminal 100. For example, the display unit 151 may display execution screen information of an application program driven by the mobile terminal 100, or UI (User Interface) or GUI (Graphic User Interface) information according to the execution screen information. .

The display unit 151 includes a liquid crystal display (LCD), a thin film transistor-liquid crystal display (TFT LCD), an organic light-emitting diode (OLED), and a flexible display (flexible). display), a three-dimensional display (3D display), and an electronic ink display (e-ink display).

Also, two or more display units 151 may be present depending on the implementation form of the mobile terminal 100. In this case, a plurality of display units may be spaced apart from one surface or integrally disposed in the mobile terminal 100, or may be respectively disposed on different surfaces.

The display unit 151 may include a touch sensor that senses a touch on the display unit 151 so that a control command can be input by a touch method. Using this, when a touch is made to the display unit 151, the touch sensor detects the touch, and the controller 180 can be configured to generate a control command corresponding to the touch based on the touch. The content input by the touch method may be a letter or a number, or an instruction or designable menu item in various modes.

On the other hand, the touch sensor is composed of a film having a touch pattern, is disposed between the display (not shown) on the back surface of the window 151a and the window 151a, or a metal wire that is directly patterned on the back surface of the window 151a. It can be. Alternatively, the touch sensor may be formed integrally with the display. For example, the touch sensor may be disposed on the substrate of the display, or may be provided inside the display.

As such, the display unit 151 may form a touch screen together with a touch sensor, and in this case, the touch screen may function as a user input unit 123 (see FIG. 1A). In some cases, the touch screen may replace at least some functions of the first operation unit 123a.

The first sound output unit 152a may be implemented as a receiver that delivers a call sound to the user's ear, and the second sound output unit 152b is a loud speaker that outputs various alarm sounds or multimedia playback sounds. ).

An acoustic hole for emitting sound generated from the first sound output unit 152a may be formed in the window 151a of the display unit 151. However, the present invention is not limited thereto, and the sound may be configured to be emitted along an assembly gap between structures (for example, a gap between the window 151a and the front case 101). In this case, the appearance of the mobile terminal 100 may be simpler because the holes formed independently for the sound output are not visible or hidden.

The light output unit 154 is configured to output light to notify when an event occurs. Examples of the event include message reception, call signal reception, missed calls, alarm, schedule notification, email reception, information reception through an application, and the like. The control unit 180 may control the light output unit 154 to end the output of light when the user's event is detected.

The first camera 121a processes image frames of still images or moving pictures obtained by an image sensor in a shooting mode or a video call mode. The processed image frame may be displayed on the display unit 151, and may be stored in the memory 170.

The first and

second operation units

123a and 123b are examples of the user input unit 123 that is operated to receive a command for controlling the operation of the mobile terminal 100, and may also be collectively referred to as a manipulating portion. have. The first and

second operation units

123a and 123b may be adopted in any manner as long as the user operates the device while receiving a tactile feeling, such as touch, push, scroll. In addition, the first and

second manipulation units

123a and 123b may be employed in such a way that the user operates without a tactile feeling through a proximity touch, a hovering touch, or the like.

In this figure, the first operation unit 123a is illustrated as a touch key, but the present invention is not limited thereto. For example, the first operation unit 123a may be a mechanical key or a combination of a touch key and a push key.

Contents input by the first and

second operation units

123a and 123b may be variously set. For example, the first operation unit 123a receives commands such as a menu, home key, cancel, search, etc., and the second operation unit 123b is output from the first or second

sound output units

152a, 152b. Commands such as adjusting the volume of the sound and switching to the touch recognition mode of the display unit 151 may be received.

Meanwhile, as another example of the user input unit 123 on the rear surface of the terminal body, a rear input unit (not shown) may be provided. The rear input unit is operated to receive a command for controlling the operation of the mobile terminal 100, and the input content may be variously set. For example, commands such as power on / off, start, end, scroll, etc., control the volume of sound output from the first and second

sound output units

152a, 152b, and touch recognition mode of the display unit 151. You can receive commands such as conversion of. The rear input unit may be implemented in a form capable of input by touch input, push input, or a combination thereof.

The rear input unit may be disposed to overlap the front display unit 151 in the thickness direction of the terminal body. For example, when the user holds the terminal body with one hand, the rear input unit may be arranged at the upper rear portion of the terminal body so that it can be easily operated using the index finger. However, the present invention is not necessarily limited to this, and the position of the rear input unit may be changed.

When the rear input unit is provided on the rear of the terminal body, a new type of user interface using the rear input unit may be implemented. In addition, when the above-described touch screen or rear input unit replaces at least some functions of the first operation unit 123a provided on the front surface of the terminal body, when the first operation unit 123a is not disposed on the front surface of the terminal body, The display unit 151 may be configured as a larger screen.

Meanwhile, the mobile terminal 100 may be provided with a fingerprint recognition sensor for recognizing a user's fingerprint, and the controller 180 may use fingerprint information detected through the fingerprint recognition sensor as an authentication means. The fingerprint recognition sensor may be embedded in the display unit 151 or the user input unit 123.

The microphone 122 is configured to receive a user's voice, other sounds, and the like. The microphone 122 may be provided at a plurality of locations and configured to receive stereo sound.

The interface unit 160 is a passage through which the mobile terminal 100 can be connected to an external device. For example, the interface unit 160 is a connection terminal for connection with other devices (eg, earphones, external speakers), a port for short-range communication (eg, an infrared port (IrDA Port), a Bluetooth port (Bluetooth) Port, Wireless LAN Port, etc.], or at least one of a power supply terminal for supplying power to the mobile terminal 100. The interface unit 160 may be implemented in the form of a socket that accommodates an external card such as a subscriber identification module (SIM) or a user identity module (UIM) or a memory card for storing information.

A second camera 121b may be disposed on the rear side of the terminal body. In this case, the second camera 121b has a shooting direction substantially opposite to the first camera 121a.

The second camera 121b may include a plurality of lenses arranged along at least one line. The plurality of lenses may be arranged in a matrix format. Such a camera may be referred to as an 'array camera'. When the second camera 121b is configured as an array camera, images may be captured in a variety of ways using a plurality of lenses, and better quality images may be obtained.

The flash 124 may be disposed adjacent to the second camera 121b. When the flash 124 photographs the subject with the second camera 121b, the light is directed toward the subject.

A second sound output unit 152b may be additionally disposed on the terminal body. The second sound output unit 152b may implement a stereo function together with the first sound output unit 152a, or may be used to implement a speakerphone mode during a call.

The terminal body may be provided with at least one antenna for wireless communication. The antenna may be built in the terminal body or may be formed in the case. For example, an antenna forming part of the broadcast receiving module 111 (see FIG. 1A) may be configured to be pulled out from the terminal body. Alternatively, the antenna may be formed of a film type and attached to the inner surface of the rear cover 103, or a case including a conductive material may be configured to function as an antenna.

The terminal body is provided with a power supply unit 190 (see FIG. 1A) for supplying power to the mobile terminal 100. The power supply unit 190 may include a battery 191 built in the terminal body or configured to be detachable from the outside of the terminal body.

The battery 191 may be configured to receive power through a power cable connected to the interface unit 160. Also, the battery 191 may be configured to be wirelessly charged through a wireless charger. The wireless charging may be implemented by a magnetic induction method or a resonance method (magnetic resonance method).

Meanwhile, in this drawing, the rear cover 103 is coupled to the rear case 102 so as to cover the battery 191 to limit the detachment of the battery 191, and is configured to protect the battery 191 from external impact and foreign matter. For example. When the battery 191 is configured to be detachably attached to the terminal body, the rear cover 103 may be detachably coupled to the rear case 102.

2 to 5 are views for explaining a driving method of the TOF camera 200.

Specifically, FIG. 2 is a basic configuration diagram of the TOF camera 200, and FIG. 3 is a diagram specifically showing the image sensor 220 in the TOF camera 200. 4 and 5 are diagrams illustrating a method of acquiring depth data by receiving light from the image sensor 220, and FIG. 6 is a light source 210 for removing ambient light and offset. A diagram for explaining a method of irradiating light 211 modularized in four phases and acquiring data therefrom.

The present invention relates to a mobile terminal that provides a vision-based user interface using a depth camera.

Depth cameras can be classified into three types according to a method of extracting a depth value. The first is a method of obtaining the distance of each pixel (Pixel) through the time that the light irradiated by the depth camera is reflected on the object and returned by the Time Of Flight (TOF) method related to the present invention. The second is a stereo method in which a depth camera obtains the distance of each pixel using binocular parallax. Third, the structured pattern method is a method in which the depth camera irradiates patterned light to an object and obtains the distance of each pixel through the degree of distortion of the pattern.

For convenience of explanation, the present invention will be described below in connection with a TOF camera, but is not limited thereto.

The TOF camera 200 may include a light source 210 that irradiates light toward the object 400 and an image sensor 220 that receives light reflected by the object 400. have. In addition, the TOF camera 200 may include a main processor 230a internally or externally to calculate the depth of the object 400 using raw data obtained through the image sensor 220.

The light source 210 may irradiate the modularized light 211 through a timing generator (not shown). The light source 210 may irradiate a light source of a specific signal through a timing generator, and in some cases, periodically change the phase of a specific signal and irradiate the light source.

The image sensor 220 may include at least one cell 221 having two

photogates

221a and 221b. The two

photo gates

221a and 221b may respectively receive the light 212 reflected by the object to obtain raw data corresponding thereto.

The method in which the image sensor 220 acquires raw data of the light 212 received through the two

photo gates

221a and 221b is as follows. The

photo gates

221a and 221b initialize the voltage of the cell by applying the reset signal Rst (step ①), and apply the modulation signals DMIX0 and DMIX1 signals to accumulate the received light (

,

) (Step ②), and apply the address decode signal (Address Decode) to

,

) Reading the charged amount of charge (step ③).

The modulation signal DMIX0 applied to the first photogate 221a may have a phase opposite to the modulation signal DMIX1 applied to the second photogate 221b, and the modulation applied to the first photogate 211a One of the signal DMIX0 and the modulation signal DMIX1 applied to the second photo gate 221b may have the same phase as the light 211 emitted from the light source 210.

Looking specifically through Figure 4, Figure 4 (a) is a light 211 irradiated from the light source 210

4 (b) is reflected by the object 400 and light 212 received from the image sensor 220 and irradiated from the light source 210 with light 211

As much as it has a phase difference. 4 (c) and 4 (b) show the modulation signal DMIX0 applied to the first photo gate 221a and the modulation signal DMIX1 applied to the second photo gate 221b. In addition, the modulation signal DMIX1 applied to the second photo gate 221b has the same phase as the light 211 emitted from the light source 210 and the modulation signal DMIX0 applied to the first photo gate 221a. And may have an opposite phase.

The light 212 received from the image sensor 220 is the light irradiated from the light source 210.

Since it has a phase difference as much as, the capacitor of the first photo gate 221a through the modulation signals DMIX0, DMIX1 (

), The amount of charge removed from Node-A (Node-A) and the capacitor of the second photo gate 221b (

), The amount of charge removed from Node-B is as shown in FIGS. 4 (c) and 4 (b).

Specifically, as shown in FIG. 5, the image sensor 220 is a capacitor of the first photogate 221a and the second photogate 221b (

,

), The phase difference data (= depth data) of the light 212 reflected by the object 400 may be acquired.

5 (a) shows that there is no phase difference between light 211 (optical) irradiated from the light source 210 and light 212 (electrical) reflected by the object 400. If there is no phase difference between light 211 (optical) irradiated from the light source 210 and light 212 (electrical) reflected by the object 400, the capacitor (

) Can be charged.

FIG. 5 (b) shows a case where light 211 (optical) irradiated from the light source 210 and light 212 (electrical) reflected by the object 400 have a phase difference of 90 degrees. When the light 211 (optical) irradiated from the light source 210 and light 212 (electrical) reflected from the object 400 have a phase difference of 90 degrees, the capacitor (

) The amount of charge charged and the capacitor (

) May have the same amount of charge.

FIG. 5 (c) shows a case where light 211 (optical) irradiated from the light source 210 and light 212 (electrical) reflected by the object 400 have a phase difference of 180 degrees. When the light 211 (optical) irradiated from the light source 210 and light 212 (electrical) reflected from the object 400 have a phase difference of 180 degrees, the capacitor (

) Can be charged.

4 and 5 illustrate an embodiment in which the light source 210 irradiates a light source of a specific signal, and the image sensor 220 receives light corresponding thereto. However, the image sensor 220 may receive ambient light in addition to the light 212 emitted from the light source 210. Therefore, in order to eliminate errors caused by ambient light, the light source 210 may periodically change the phase of a specific signal to irradiate the light source.

6 illustrates an embodiment in which a phase of a specific signal is periodically varied in the light source 210 to irradiate a light source to eliminate errors caused by ambient light.

Specifically, FIG. 6 (a) shows a light source having a phase difference of 0 degrees, that is, a light source irradiated with a single preset pulse, and FIG. 6 (b) shows a light source having a phase difference of 90 degrees from the light source of FIG. 6 (a). 6 (c) shows a light source having a phase difference of 180 degrees from the light source of FIG. 6 (a), and FIG. 6 (d) shows a light source having a phase difference of 270 degrees from the light source of FIG. 6 (a).

6 (e) shows light reflected by the object 400 and received by the image sensor 220. The modulation signal DMIX1 applied to the second photo gate 221b may be synchronized with the signals of FIGS. 6 (a) to 6 (d), wherein FIG. 6 (e) is synchronized with the signal, respectively. 221b) capacitor (

) Shows charge amounts 213a to 213b.

That is, the light source 210 periodically varies the phase of a specific signal, and the capacitor of the image sensor 220 (

,

), The error caused by ambient light can be corrected through the difference in the percentage charged.

In the above, the driving method of the TOF camera 200 and the method in which the TOF camera 200 acquires phase difference data (= depth data) have been described. Hereinafter, an improvement according to the present invention will be described.

In order for the mobile terminal to interact with the user through the TOF camera 200, it is necessary to keep the TOF camera 200 always ON. This is to provide a function for a user to interact with a mobile terminal at any time without being directly touched by a mobile terminal at a certain distance.

Specifically, FIG. 7 (a) shows a state in which the TOF camera 200 always maintains an ON state and acquires a depth image by irradiating light to the tracking area 300 even when the user 400 is not present. However, in this case, there is a problem in that power consumption of the mobile terminal is unnecessary.

Accordingly, as shown in FIG. 7 (b), obtaining a depth image is technically effective in preventing excessive power consumption only when the user 400 is present in the tracking area 300.

That is, the TOF camera 200 and the mobile terminal including the same according to an embodiment of the present invention are driven with a low power and operate in a first mode to confirm the existence of the user 400, and the user 400 in the first mode When the existence is confirmed, it may operate in a second mode for acquiring a depth image. In this case, it is possible to prevent excessive power consumption even if the TOF camera 200 is always in an ON state.

8 is a schematic diagram for explaining the configuration of the TOF camera 200 according to an embodiment of the present invention.

The TOF camera according to an embodiment of the present invention operates in a first or second mode, and receives light reflected from the illumination source 210 and the object 400 irradiating the light source to the object 400 An image sensor (ToF Image sensor) 220 for obtaining raw data,

arithmetic units

230a and 230b for processing the raw data, and a light source 210, image sensor 220 and

arithmetic unit

230a, 230b) and a controller (not shown) connected to the controller, the controller controls the light source 210 to irradiate a light source of a specific signal in the first mode, and detects changes in the raw data through the

calculators

230a and 230b. In one case, the operation unit 230a switches to the second mode, controls the light source 210 to vary the phase of the specific signal in the second mode, and generates depth data of the object 400 through the raw data. 230b) can be controlled.

Furthermore, in FIG. 8, although each component module is illustrated separately, it is also within the scope of the present invention to design some modules by combining them.

The controller may include a timing generator that provides a pulse signal to the light source 210 and the image sensor 220, and in some cases, may be configured to include

arithmetic units

230a and 230b. Also, the controller may have a configuration corresponding to the controller 180 shown in FIG. 1A.

6, the TOF camera 200 controls the light source 210 that irradiates a light source of a specific signal in a second mode in order to correct errors in ambient light or offset in the depth data, as described in FIG. 6. The phase of a specific signal can be changed periodically.

However, in the first mode, it is not necessary to correct the error of the ambient light or the offset (Off Set) by confirming the existence of the moving object 400, rather than acquiring the depth data of the object 400. Accordingly, in the first mode, the light source 400 is controlled to irradiate a light source of a specific signal without a phase change, and power consumed for the phase change can be reduced.

As illustrated in FIG. 3, the image sensor 220 applies first and

second photo gates

221a and 221b that receive signals having different phases during a predetermined frame and receive reflected light 212 correspondingly. It can contain.

Specifically, the first and

second photo gates

221a and 221b constitute one cell 221, and the image sensor 220 may include a plurality of cells 221.

The controller may control the image sensor 220 to acquire the raw data through the first photo gate 221a in the first mode. That is, the controller applies the modulation signal DMIX0 (refer to FIG. 4) only to the first photo gate 221a and does not apply the modulation signal DMIX1 to the second photo gate 221a. The controller uses only one photo gate in the cell 211 including the first and

second photo gates

221a and 221b in the first mode to drive both the first and second photo gates 221a and 221b. Power consumption can be reduced.

Specifically, in the first mode, the mode for checking the existence of the moving object 400 does not need to calculate the depth data of the object 400. In order to calculate the depth data, capacitors of the first and

second grape gates

221a and 221b (

,

) Is required. However, in order to confirm the existence of the moving object 400, a capacitor of one photo gate (

or

). Capacitor of one photogate (

or

Change in the amount of charge charged in) refers to a change in the intensity of the reflected light, through which the existence of the moving object 400 can be confirmed. That is, in the first mode, power consumed to drive both the first and

second photo gates

221a and 221b may be reduced by driving only one photo gate.

In addition, the TOF camera 200 of the present invention includes a sub-operation unit 230b for determining a change in intensity of raw data obtained by the

operation units

230a and 230b through the first photo gate 221a and the first And a main operation unit 230a that generates (calculates) the depth data of the object through the intensity ratio through the raw data obtained from the

second photo gates

221a and 221b, respectively.

The main operation unit 230a is the same as the main operation unit 230a included in the existing TOF camera 200 of FIG. 2 and may correspond to a Mobile Industry Processor Interface (MIPI). The main operation unit 230a generates depth data through raw data acquired in the second mode.

That is, the main operation unit 230a may have a lot of power consumption since the calculation process is complicated and processes a lot of data.

However, the sub-operation unit 230a is an operation unit that determines only the change in the intensity of raw data acquired through one photo gate, and may correspond to a Serial Peripheral Interface (SPI) with simple processing data and low power consumption.

That is, in the first mode, instead of using the main operation unit 230a having a high power consumption, using the sub operation unit 230a having a small power consumption, the TOF camera 200 consumes excessive power even if it is always in an ON state. Can be prevented.

When the controller detects a change in the intensity data in the sub-operation unit 230b, the controller may wake up the main operation unit 230a. That is, when the main operation unit 230a maintains a state in which the power is not applied in the first mode and then switches to the second mode, power may be applied.

A method of controlling a TOF camera according to an embodiment of the present invention in a first mode and a second mode is as follows.

The TOF camera may be driven in a first mode (= low power Object Detection mode = MP (Main Processor) sleep mode = TOF Intensity mode) to determine the presence or absence of an object. (S201)

In the first mode, the TOF camera can irradiate a light source of a specific signal. The specific signal may be a periodic single pulse signal. (S202) The periodic single pulse light source may be a light source that maintains only the pulse corresponding to FIG. 6 (a) as a pulse that does not periodically change phase in order to correct errors due to ambient light and offset. .

The image sensor may acquire raw data by receiving light reflected by the object. The raw data may include intensity data that can measure the intensity change. (S203)

Here, the image sensor may include the first and second photogates 211a and 211b as described in FIG. 3, and in the first mode, intensity data may be acquired using only one photogate. have. That is, in the first mode, intensity data corresponding to the amount of charge charged in the capacitor of one photogate can be obtained.

The TOF camera may include a sub-operation unit 230b as described in FIG. 8, and may transmit the intensity data acquired in one photo gate to the sub-operation unit 230b. (S204)

The sub-operation unit 230b may determine whether a moving object exists by determining whether a difference between the N-1th intensity data and the N-th intensity data acquired at predetermined intervals is greater than or equal to the preset intensity. (S205)

The TOF camera according to the present invention is the first mode state when the difference between the N-1st intensity data and the N-th intensity data obtained at a preset period interval in the first mode is equal to or less than the preset intensity (S205, NO). Can keep.

In addition, the TOF camera according to the present invention, when the difference between the N-1st intensity data and the N-th intensity data acquired at a predetermined period interval in the first mode is greater than or equal to the preset intensity (S205, YES), the main operation unit Wake up (230a) and switch to a second mode (= TOF 3D depth mode) for acquiring depth data. (S206)

The TOF camera changes the phase of a specific signal in the second mode and can irradiate a light source to the object. Specifically, in the second mode, a specific signal may be irradiated with light sources in phases of 0 degrees, 90 degrees, 180 degrees, and 270 degrees (4 phases). (S207)

In the second mode, the image sensor may receive a light source reflected by the object to obtain raw data (= 4 phase raw data). (S208) The raw data acquired in the second mode may be data (= 3D raw data) used to acquire 3D depth data.

The raw data (= 3D raw data) received in the second mode may be transmitted to the main operation unit (= MIPI inferface, 230a). (S209)

The main operation unit generates depth data (= Depth data) through raw data so that it can be utilized in an AP (Appication Processr) of the mobile terminal, and the mobile terminal provides a user UX / UI corresponding to the depth data (= Depth data). You can do it (S210)

The generating of depth data and providing a user UX / UI corresponding to the mobile terminal (S210) may be a step of providing graphic feedback through a display in response to the depth data. Specifically, the mobile terminal includes a memory that stores commands corresponding to the depth data, and the display can provide graphic feedback corresponding to the commands stored in the memory.

When the user UX / UI provided by the mobile terminal is terminated, the TOF camera may be switched from the second mode to the first mode (S211).

However, switching from the second mode to the first mode may be possible when there is no change in depth data over a preset time. That is, the mode can be switched through whether the depth data acquired in the second mode is changed regardless of whether the UX / UI provided by the mobile terminal is terminated.

FIG. 10 is a flow chart for explaining a method of determining the presence or absence of an object by comparing frames in FIG. 9, and FIG. 11 is a view for explaining FIG. 9.

The method of determining the existence of the moving object in the first mode is as follows. The raw data obtained through the first or second photo gate is intensity data, and may be measured at predetermined time intervals for each pixel 212a.

That is, in FIG. 9, the sub-operation unit 230b determines whether the difference between the N-1th intensity data and the N-th intensity data acquired at predetermined intervals is greater than or equal to the preset intensity (determining the presence or absence of an object ( S205) may include the following steps.

Referring to FIG. 3, the image sensor 220 may include cells 221 including the first photogate 211a and the second photogate 211b to form a plurality of rows and columns.

The plurality of cells 221 may correspond to the pixel 212a shown in FIG. 11, and the intensity data (the intensity data obtained from the first or second photo gate) obtained by each of the plurality of cells 221 may be pixels. It may correspond to the size of the value indicated in (212a).

That is, the sub-operation unit 230b calculates the sum of the pixel values for each column / row, and (S301) calculates the difference between the N-1 frame and the N frames obtained at predetermined time intervals (S301).

11 (a) shows an N-1 frame, and 11 (b) shows an N frame.

Here, the step of calculating the difference between the N-1 frame and the N frames obtained at predetermined time intervals is based on the sum 212b of the pixel values included in each column as shown in FIGS. 11 (a) and (b). And calculating a difference between a sum 212b of pixel values included in each column of the -1 frame and the N frame. Referring to FIG. 11, the difference between the sum of the pixel values included in each column of the N-1 frame and the N frame 212b is 1 + 3 + 14 + 1 + 12, which is 30.

In addition, the step of calculating the difference between the N-1 frame and the N frames obtained at predetermined time intervals is based on the sum 212c of the pixel values included in each row as shown in FIGS. 11 (a) and (b). And calculating a difference between sums 212c of pixel values included in each row of the N-1 frame and the N frame. Referring to FIG. 11, the difference between the sum of the pixel values included in each row of the N-1 frame and the N frame 212c is 1 + 4 + 12 + 11, which is 28.

The sub-operation unit 230b adds the sum 212b of the pixel values included in each column of the N-1 frame and the N frame, and the sum 212c of the pixel values included in each row of the N-1 frame and the N frame. Calculate the difference between 1 frame and N frame intensity data (= TOTAL_DIFF). (S303)

If the difference between the N-1 frame and the N frame intensity data (= TOTAL_DIFF) is equal to or less than the preset intensity (= THRES), the first mode is maintained (S304, No), and the N-1 frame and N frame intensity data difference ( = TOTAL_DIFF) is greater than or equal to the preset intensity (= THRES) (S304, Yes), it can be switched to the second mode. That is, it is possible to wake up the main operation unit 230a. (S304)

12 is a schematic diagram for explaining the configuration of the TOF camera 200 according to another embodiment of the present invention.

Hereinafter, description will be made focusing on differences from the configuration described in FIG. 8.

The TOF camera 200 includes a separate photodiode 260 and receives the light source 212 irradiated and reflected by the light source 210 in the first mode through the separate photodiode 260 can do.

In the first mode, the raw data obtained by receiving light through the photodiode 260 corresponds to the intensity data (= photo diode light amount data), and the intensity data is transmitted to the sub-operation unit 230b to determine whether there is a change. can do.

In the first mode, if the change in the intensity data received through the photodiode 260 is greater than or equal to a preset difference, it is determined that a moving object exists, and the main operation unit 230a can be wakeed up.

That is, the TOF camera 200 illustrated in FIG. 12 separately includes a configuration for receiving light in the first mode, and a feature for receiving light in the second mode to the image sensor 220.

In the above, the features of the present invention for sensing the power consumed by the TOF camera 200 have been described, and the following is another characteristic of the present invention for accurately and quickly tracking a user's finger through depth data acquired in the second mode. Let me explain.

In order to enable the mobile terminal to implement a vision-based interface through a depth camera, an image of a portion corresponding to the user's hand 410 is quickly and accurately extracted from the entire depth image, and the finger 414 can be accurately and quickly tracked. It should be possible.

To date, most finger tracking algorithms use an edge 413 on the hand 410 to obtain the position of the finger 414, as shown in FIG. 13 (a).

Since this method is to track the finger 414 by looking at the edge 413 of the hand 410, there is a problem that the shape of the obtained hand 410 must be a shape viewed from the palm or the hand. Thus, to date, products that provide UI / UX based on the gesture of the hand 410 always install the camera 200 in a position where the palm or hand is visible, as shown in FIG. 13 (b), and the hand as shown in FIG. 13 (a). (410) Acquire an image.

However, it may be difficult to track the finger 414 in a state in which the edge of the finger is unclear toward the camera as shown in FIG. 13B. In addition, the method of tracking the shortest distance point through the depth camera has a problem that is not suitable for tracking a plurality of fingers 414.

Accordingly, there is a need for a new technique for tracking finger 414 using a depth camera.

Hereinafter, an algorithm for quickly and accurately tracking a finger through a depth camera by a mobile terminal according to an embodiment of the present invention will be described.

14 is a diagram for schematically illustrating a method in which a mobile terminal according to an embodiment of the present invention recognizes a point targeted by a user's hand 410 through a depth camera 200.

In order to interact with the user, the mobile terminal needs to quickly and accurately track a finger targeted at the user's hand 410 and points 411a and 411b targeted by the finger. Further, even if the mobile terminal protrudes from the user's hand 410 toward the depth camera, it is necessary to exclude the targeting point 412 from tracking.

14 (a) shows an embodiment in which the user's hand 410 is directed toward the depth camera 200, and FIG. 14 (b) shows that the depth camera 200 in FIG. 14 (a) is the user's hand 410. Is captured, and is a segment 500 obtained corresponding to a distance h from the depth camera 200 in the depth image. In addition, FIG. 14 (c) shows the segment tree 600 implemented through the inclusion relationship between the corresponding reference distance and the segment 500 of FIG. 14 (b).

The mobile terminal may extract targeting points 411a and 411b from the depth image of the user's hand 410 through the segment tree 600.

Looking specifically at the segment tree 600 of FIG. 14C, the segment tree 600 may include at least one node 630. The at least one node 630 may be a specific segment including a plurality of segments corresponding to neighboring distances. That is, when the segment corresponding to the N + 1st reference distance (h_N + 1) includes a plurality of segments corresponding to the Nth reference distance (h_N), the segment corresponding to the N + 1st reference distance (h_N + 1) The segment is a specific segment, and may correspond to the first node 630a in the segment tree 600.

Points 411a, 411b targeted at the user's hand 410 are

effective ends

610a, 610b spaced apart by a predetermined distance or more from the node (= specific segment, 630) of the segment tree 600 toward the depth camera 200. ). Specifically, it is possible to correspond to points 411a and 411b targeting a segment located at a predetermined distance or more toward a mobile terminal or a depth camera 200 at a reference distance corresponding to a specific segment.

The first effective end 610a is connected to the first node 630a and corresponds to the first point 411a targeted by the user's hand because the first node 630a and the distance ha1 are greater than a preset distance. You can.

The second effective end 610b is connected to the second node 630b, and the second node 630b and the second point each of which is targeted at the user's hand 410 with a distance ha2 greater than or equal to a preset distance ( 411b).

Even if it is located at the end in the segment tree 600, if the distance from the node 630 is less than a preset distance, the end is classified as an invalid end 620, which is a point 412 that is not targeted at the user's hand 410 ). Specifically, the invalid end 620 is connected to the second node 630b, the second node 630b and the distance hb1 is less than a predetermined distance, a point 412 that is not targeted at the user's hand 410 Can correspond to

That is, whether the end is valid or invalid in the segment tree can be discriminated through whether the distance from the connected node is greater than or equal to a preset distance. Therefore, the present invention has the advantage of being able to track fingers with different node depths, such as the thumb, with targeting fingers, and to accurately track targeting points of multiple fingers.

Hereinafter, a method of tracking a user's finger through the depth camera 200 in detail will be described in detail.

15 is a flow chart for explaining FIG. 14.

The mobile terminal acquires a depth image through a depth camera to track a user's finger. (S401)

In order to discriminate the user's hand from the depth image, the mobile terminal sets an interaction region and removes an image outside the interaction region from the depth image. (S402) The interaction area may be set within a first preset distance from the mobile terminal, and will be described in detail with reference to FIG. 16 below.

The mobile terminal may extract the target distance of the object through a pixel having a distance value closest to the depth image. (S403) The target distance may be a starting reference distance for acquiring a segment according to the distance from the mobile terminal in the depth image, and will be specifically described with reference to FIG. 17 below.

The depth camera may acquire a segment corresponding to the distance from the depth camera in the depth image, and implement a segment tree through the segmentation relationship. (S404) The segment tree may be implemented through an inclusion relationship between segments obtained according to a distance from the mobile terminal, and will be described in detail with reference to FIGS. 18 to 22.

The mobile terminal determines the effective end through the distance from the node in the implemented segment tree. (S405) The mobile terminal may identify a specific segment corresponding to the node in the segment tree, and determine an effective end through a segment located at a distance greater than or equal to a second preset distance from the distance at which the specific segment is obtained. In this regard, it will be described in detail with reference to FIGS. 23 and 24 below.

The mobile terminal extracts a targeting point and depth (= 3D Finger Position) corresponding to the determined effective end. (S406) The mobile terminal may extract the location of at least one target point corresponding to the effective end, and provide video feedback through the display in response to the extracted location. In this regard, the method of extracting the location of the target point will be described with reference to FIGS. 26 and 27, and the method of providing video feedback through the display will be described in detail with reference to FIGS. 28 to 30.

In order for the mobile terminal to track a user's finger through the above method, the mobile terminal may store a memory storing at least one command, a depth camera capturing an object to obtain a depth image, and at least one command stored in the memory, A display for outputting video feedback corresponding to an object captured by the depth camera, and a controller connected to the memory, the depth camera, and the display, wherein the controller is configured according to at least one command stored in the memory. In response to the object within a first predetermined distance from the mobile terminal, the depth camera controls to track the object, and according to at least one command stored in the memory, the distance from the mobile terminal in the depth image To Acquiring at least one segment according to, and controlling the display to output at least one target point located above a second preset distance to the mobile terminal from a distance at which a specific segment is acquired according to at least one command stored in the memory It can be characterized by doing.

FIG. 16 is a diagram for explaining an interaction region in FIG. 15, and FIG. 17 is a diagram for describing a depth image obtained in the interaction region of FIG. 16.

The depth camera 200 may extract depth data for each pixel, and through this, it is possible to extract an object within a specific distance.

The interaction area 320 used in the present invention may correspond to an area within a first preset distance (= maximum recognition distance = Max distance, 330) as shown in FIG. 16. In some cases, the interaction area 320 may be an area corresponding to a minimum recognition distance 310 and a first preset distance 330 that the depth camera 200 can recognize the object.

The mobile terminal can track only objects that have entered the interaction area 320. Since the user reaches out to the mobile terminal and interacts with the mobile terminal through hand movement, the object entering the interaction area 320 may be the user's hand, and the mobile terminal tracking the object that is likely to be the user's hand desirable.

The mobile terminal removes the pixel value of the remaining pixels except for the pixel having the pixel value corresponding to the interaction region 320 in the acquired depth image.

In FIG. 17, a pixel 321 having a value of 0 is a pixel having a pixel value that does not correspond to the interaction area 320, and corresponds to a pixel from which a pixel value is removed. Also, the pixel 322 having a pixel value may be a pixel having a pixel value corresponding to the interaction region 320. That is, the pixel 322 having a pixel value may be a pixel having a pixel value corresponding to the first reference distance 330 or less.

The mobile terminal is designed to start tracking when an object enters the interaction area 320. That is, when an object does not enter the interaction area 320 and all pixel values are removed as 0, the mobile terminal does not perform tracking described below, thereby increasing energy and data efficiency.

When an object enters the interaction area 320, the mobile terminal determines the shortest distance from the acquired depth image. The shortest distance may be a distance corresponding to a minimum pixel value excluding 0 in FIG. 17. The determined shortest distance may be used to form a segment tree of the object by entering the object into the interaction area 320. Hereinafter, a method of forming a segment tree will be described.

FIG. 18 is a flowchart for describing a method of implementing a segment tree corresponding to a user's hand in FIG. 15, and FIGS. 19 to 22 are diagrams for describing FIG. 18.

The mobile terminal may implement a segment tree when starting tracking when an object enters the interaction area 620 of FIG. 16.

To implement the segment tree, the mobile terminal determines the shortest distance (= START) and reference distance (= END) to obtain a segment. (S501)

The shortest distance (= START) may be a distance corresponding to the smallest pixel value 323 except 0 in FIG. 17. That is, the shortest distance (= START) is the distance h1 between the depth camera 200 and the point 411c closest to the depth camera 200 from the object (or the user's hand 410) as described in FIG. 19. Can be

The reference distance (= END) is a distance for acquiring a segment, and the segment may be a group of pixels having a pixel value corresponding to a reference distance (= END) or less. The mobile terminal may determine a reference distance (= END) based on a distance exceeding the shortest distance (= START) and less than or equal to the first preset distance 330. As described in FIG. 19, the reference distance (= END) may be sequentially determined with a predetermined distance gap (Gap) between the first preset distance (h1) and the shortest distance (h1).

The mobile terminal filters only pixels having pixel values greater than or equal to the shortest distance (= START) and less than or equal to the reference distance (= END) from the acquired depth image, and removes the remaining pixel values. (S502)

20 (a) shows an embodiment of a user's hand captured in a depth image and a shortest distance (= START) and a first reference distance (= END 1) to a sixth reference distance (= END 6) in the user's hand. It is a marked drawing.

In addition, FIG. 20 (b) shows an embodiment in which only pixels having pixel values between each of the first reference distance (= END 1) to the sixth reference distance (= END 6) and the shortest distance (= START) are filtered. It is a drawing. When the reference distance END 1 to 6 is close to the depth camera (or the shortest distance (= START)), it can be confirmed that the number of pixels to be filtered increases.

The mobile terminal may filter the depth image according to the determined reference distance (= END) and the shortest distance (= START), and then segment the segment. (S503)

21 is a diagram illustrating a process of dividing a segment. FIG. 21 (a) shows the shortest distance (= START) and the first reference distance (= END1), the shortest distance (= START) and the third reference distance (= END3), and the shortest distance (= START) and FIG. Each depth image is filtered according to 5 reference distances (= END5). FIG. 21 (b) shows segments segmented by grouping pixels having pixel values in the filtered image.

Specifically, the depth image filtered according to the shortest distance (= START) and the first reference distance (= END1) may be divided into one pixel group, and one pixel group may correspond to segment A. Also, the depth image filtered according to the shortest distance (= START) and the third reference distance (= END3) may be divided into three pixel groups, and the three pixel groups may correspond to segments B, C, and D, respectively. You can. Also, the depth image filtered according to the shortest distance (= START) and the fifth reference distance (= END5) may be divided into two pixel groups, and the two pixel groups may correspond to segments E and F.

The mobile terminal determines a specific segment (= Parent) after segmenting. (S504)

22 is a diagram illustrating a method of implementing a segment tree through the segmented segment of FIG. 21 and determining a specific segment. In FIG. 22 (a), segment A has a corresponding reference distance longer than a corresponding reference distance, and the occupied area includes segments B, C, and D as shown in the upper view of FIG. 22 (b). Can be compared. In FIG. 22 (a), the reference distances corresponding to segments B, C, and D are longer than the reference distances corresponding to segments E, F, and are included in the areas occupied by segments B and C, respectively, as shown in the lower diagram of FIG. 22 (b). Can be compared. In addition, the upper and lower views of FIG. 22 (b) may be implemented as a segment tree as shown in FIG. 22 (c).

22 (c), segments A, segments B, C, D, and segments E, F are located above, middle, and bottom through corresponding reference distances, and segment A indicates segments B, C, and D. Including, it can be connected to segments B, C, and D. In addition, segments B and C include segments E and F, respectively, so that each can be interconnected.

That is, the mobile terminal according to the present invention can reduce the reference distance from the depth image at a predetermined interval from the first preset distance to obtain a corresponding segment, and form a segment tree through the intersegment inclusion relationship.

The specific segment (= parent) may be a segment including a plurality of segments corresponding to adjacent reference distances. That is, in FIG. 22, segment A includes a plurality of segments B, C, and D corresponding to adjacent reference distances, and thus may correspond to a specific segment (= parent) of the present invention.

The mobile terminal sequentially decreases the reference distance (= END) with a predetermined distance gap (Gap) at the first preset distance (h1) as described in FIG. 19 to obtain a corresponding segment, and determines a specific segment Then, when the reference distance (= END) is less than or equal to the shortest distance (= Start, h1) (S505, YES), the implementation of the segment tree is stopped (S506).

23 is a flowchart illustrating a method of determining an effective end through a segment tree in FIG. 15, and FIG. 24 is a diagram for describing FIG. 23.

When the mobile terminal implements a segment tree through depth data, it determines the end of the segment tree. (S601) Here, the end may correspond to the target point of the present invention, in this regard will be described in detail with reference to Figures 24 and 25.

The mobile terminal extracts a segment corresponding to the reference distance, and if the extracted segment does not include another segment, it may determine that segment as the end of the segment tree.

23 is an example of the segment tree 600, and may be implemented as described through FIGS. 18 to 22. The segment tree 600 according to an example included in FIG. 23 includes three ends 610b and 610a. , 620).

When the mobile terminal determines the end, the length between each end and the node is extracted. (S602) The length between each end and the node may be a difference between a reference distance corresponding to each end and a reference distance corresponding to the node. . Here, the node may correspond to the characteristic segment of FIG. 22.

The segment tree 600 according to the example of FIG. 23 includes one node 630, and three

ends

610a, 610b, and 620 are implemented by connecting the nodes 630.

The length between each end (610a, 610b, 620) and the node 630 is a reference distance (End b, End c) corresponding to each end (610a, 610b, 620) and a reference distance (End) corresponding to the node (630) It may be the difference of a).

Accordingly, the distance h4 between the first end 610a and the second end 610b and the node 630 is a reference distance End b and a node corresponding to the first end 610a and the second end 610b. It may be a difference of the reference distance End a corresponding to 630.

In addition, the distance h3 between the third end 620 and the node 630 is the difference between the reference distance End c corresponding to the third end 620 and the reference distance End a corresponding to the node 630. Can be

The mobile terminal determines an effective end based on whether the length between each end and the node is greater than or equal to the second preset distance THRES (S603). That is, the mobile terminal sets a second preset value at a reference distance corresponding to the node. An end obtained at a reference distance spaced over a distance may be determined as an effective end.

Looking through FIG. 23, when the length h4 between the first end 610a and the second end 610b and the node 630 satisfies a condition greater than or equal to a second preset distance, it may be determined as an effective end. In addition, when the length h3 between the third end 620 and the node 630 does not satisfy the condition greater than or equal to the second preset distance, it may be determined as an invalid end.

However, the end determined as an effective end is further checked whether the invalid end is through metadata. (S604)

The method of additionally checking whether the invalid end is through metadata may be determined through a size increase rate from the effective end to the node, and if the effective end suddenly increases from the node, it may be determined as the invalid end.

If the effective end is determined through the segment tree, the algorithm ends (S605).

25 is a flow chart for explaining a method of extracting a targeting point and depth corresponding to the effective end determined in FIG. 15, and FIGS. 26 and 27 are views for explaining FIG. 25.

The mobile terminal extracts the midpoint (x, y) of the segment corresponding to the effective end. (S701) The mobile terminal extracts the midpoint (x, y) of the segment through the position of the pixels constituting the segment corresponding to the effective end. can do.

The mobile terminal extracts the depth (Depth (z)) of the effective end. (S702), and ends the procedure (S703).

26 is an example of the segment tree 600, and may be implemented as described through FIGS. 18 to 22. Segments E and G may be segments corresponding to effective ends. FIG. 27 is a depth image obtained at a reference distance End e corresponding to segments E and F of FIG. 26 and includes segments E and F through filtering.

The mobile terminal may extract the midpoint (x, y) of the segment E through the position of the pixel group 610e constituting the segment E in the segment E corresponding to the effective end.

Further, the mobile terminal may extract the depth h6 through the reference distance End e corresponding to segment E. In this case, the depth difference h7 between the segment G and the segment may be extracted through the depth h5 extracted through the reference distance End g corresponding to the segment G.

The mobile terminal may acquire a targeting point by the user's hand through the midpoint (x, y) and depth (y) corresponding to the effective end, and provide video feedback to the display correspondingly. The video feedback provided in response to the targeting point will be described below.

The video feedback may be video feedback indicating that a display area corresponding to the at least one target point is selected when at least one target point moves over a third preset distance toward the mobile terminal (or depth camera). .

Further, the video feedback may be video feedback indicating that a display area corresponding to the at least one target point is selected when at least one target point moves within a fourth preset distance from the mobile terminal (or depth camera). have.

28 (a) is a diagram illustrating an embodiment in which a mobile terminal acquires a targeting point through a segment tree corresponding to a user's hand 410 and displays the target point 720 on the display 700.

FIG. 28 (b) is a diagram illustrating an embodiment in which the degree to which the targeting point approaches the display is displayed through the graphic feedback 740 approaching the selection target 730.

28 (c) is a diagram illustrating a graphic feedback 750 indicating that the selection target 730 is selected in FIG. 28 (b). That is, the user can know that the selection target is selected because the graphic feedback 750 matches the contour of the selection target 730.

In addition, when the at least one target point is divided into a plurality of target points, the video feedback may be video feedback that enlarges or zooms in the content displayed on the display.

Conversely, the video feedback may be video feedback that reduces or zooms out the size of content displayed on the display when the target points are plural and the plurality of target points are combined into one target point. .

FIG. 29 (a) shows a user's hand 410 action that provides zoom-in or zoom-out video feedback, where two

target points

720a, 720b are merged into one target point 721, or vice versa. One target point value 721 illustrates an embodiment in which two

target points

720a and 720b are divided.

FIG. 29 (a) shows a user's hand 410 action that provides zoom-in or zoom-out video feedback, wherein five

target points

720a, 720b, 720c, 720d, and 720e are one target point 721 ), Or, conversely, one target point 721 shows an embodiment in which the target points are divided into five

target points

720a, 720b, 720c, 720d, and 720e.

Also, the video feedback may be video feedback that moves content displayed on a display or displays other content in response to a direction in which the at least one target point moves.

30 illustrates an action of a user hand 410 corresponding to video feedback displaying other content, and an embodiment in which five

target points

720a, 720b, 720c, 720d, and 720e move from side to side. have.

In addition, the video feedback may be video feedback that rotates content displayed on a display corresponding to the rotation direction of the at least two target points.

The above detailed description should not be construed as limiting in all respects, but should be considered illustrative. The scope of the present invention should be determined by rational interpretation of the appended claims, and all changes within the equivalent scope of the present invention are included in the scope of the present invention.

Claims

In a TOF (Time Of Flight) camera operating in the first or second mode,

A light source that irradiates an object with a light source;

An image sensor that receives raw light reflected by the object and obtains raw data;

An operation unit that processes the raw data; And,

And a controller connected to the light source, the image sensor, and the calculation unit.

The controller

In the first mode, the light source is controlled to irradiate a light source of a specific signal, and when a change in the raw data is detected through the operation unit, the light source is switched to the second mode,

A time of flight (TOF) camera, characterized in that in the second mode, the light source is controlled to vary the phase of the specific signal, and the operation unit is controlled to generate depth data of the object through the raw data.
According to claim 1,

The image sensor

Signals having different phases are applied during a preset frame, and correspondingly include first and second photo gates that receive the reflected light,

The controller

In the first mode, the image sensor is controlled to acquire the raw data through the first photo gate,

A time of flight (TOF) camera, characterized in that the image sensor is controlled to acquire the raw data through the first and second photo gates in the second mode.
According to claim 2,

The operation unit

A sub-operation unit to determine an intensity change of the raw data acquired in the first mode; And,

And a main operation unit generating the depth data through the intensity ratio of the raw data acquired in the second mode.
According to claim 3,

The controller

A time-of-flight (TOF) camera, characterized in that when the sub-operation unit senses the change in intensity of the raw data, the main operation unit wakes up.
The method of claim 5,

The controller

In the second mode, if the depth data does not change more than a preset time, the TOF (Time Of Flight) camera, characterized in that for switching to the first mode.
In the mobile terminal,

A TOF (time of operation) including a light source irradiating light, an image sensor receiving reflected light to obtain raw data, and an operation unit generating depth data through the raw data, operating in the first or second mode Flight) camera;

A memory for storing a command corresponding to the depth data;

A display providing graphic feedback in response to the command stored in the memory; And,

And a controller connected to the TOF camera, the memory, and the display.

The controller

In the first mode, the light source is controlled to irradiate a light source of a specific signal, and when the operation unit detects a change in the raw data, it switches to the second mode,

In the second mode, the light source is controlled to vary the phase of the specific signal, and the operation unit is controlled to generate the depth data through the raw data,

And when the provided graphic feedback ends, switching from the second mode to the first mode.
The method of claim 7,

The image sensor

Signals having different phases are applied during a preset frame, and correspondingly include first and second photo gates that receive the reflected light,

In the first mode

The raw data is acquired through the first photo gate,

In the second mode

The raw data is obtained through the first and second photo gate mobile terminal.
The method of claim 10,

The operation unit

A sub-operation unit to determine an intensity change of the raw data acquired in the first mode;

And a main operation unit which calculates an intensity ratio of the raw data acquired in the second mode and generates the depth data through it.
The method of claim 8,

The operation unit

A sub-operation unit to determine an intensity change of the raw data acquired in the first mode;

And a main operation unit which calculates an intensity ratio of the raw data acquired in the second mode and generates the depth data through it.
The method of claim 9,

The controller

When the sub-operation unit senses the change in the intensity of the raw data, the mobile terminal characterized in that the main operation unit wakes up (wake up).
In the method for controlling a mobile terminal including a TOF (Time of Flight) camera,

Operating the Time Of Flight (TOF) camera in a first mode to confirm the existence of a moving object;

Includes; when the existence of the moving object is confirmed in the first mode, operating the Time Of Flight (TOF) camera in the second mode to obtain depth data of the object;

The first mode

The TOF camera irradiates a light source of a specific signal to an object, receives a light source reflected by the object, obtains raw data, detects a change in the raw data, determines the existence of the moving object,

The second mode

The TOF camera shifts the phase of the specific signal to irradiate a light source, receives the light source reflected by the object, obtains raw data, and obtains depth data of the object through the raw data. Terminal control method.
The method of claim 11,

Providing graphic feedback in response to the depth data; And,

And when the provided graphic feedback ends, switching from the second mode to the first mode.