WO2020166739A1

WO2020166739A1 - Mobile terminal

Info

Publication number: WO2020166739A1
Application number: PCT/KR2019/001773
Authority: WO
Inventors: 진동철; 안예한; 윤동한; 김현규; 송민우; 진소슬
Original assignee: 엘지전자 주식회사
Priority date: 2019-02-13
Filing date: 2019-02-13
Publication date: 2020-08-20

Abstract

Provided is a mobile terminal, according to the present invention, for performing a three-dimensional vision-based UI at low power without direct touching and clearly identifying a time point at which the three-dimensional vision-based UI is performed, the mobile terminal comprising: a proximity sensor for sensing whether an object approaches the mobile terminal; a depth camera for acquiring a depth image by capturing the object; a memory for storing at least one command; and a controller connected to the proximity sensor, the depth camera, and the memory, wherein, when the approach of the object is sensed on the basis of an initial value of the proximity sensor, the controller acquires a depth image through the depth camera according to the at least one command stored in the memory and updates the initial value of the proximity sensor on the basis of the acquired depth image.

Description

Mobile terminal

The present invention relates to a mobile terminal. In more detail, it is applicable to the technical field of temporarily driving a 3D depth camera provided in a mobile terminal to prevent power consumption, and providing a user with a convenient UI by driving a depth camera without a user's direct touch.

Terminals can be divided into mobile/portable terminals and stationary terminals depending on whether they can be moved. Again, mobile terminals can be divided into handheld terminals and vehicle mounted terminals depending on whether the user can directly carry them.

The functions of mobile terminals are diversifying. For example, there are functions of data and voice communication, taking pictures and videos through a camera, recording voice, playing music files through a speaker system, and outputting images or videos to the display. Some terminals add an electronic game play function or perform a multimedia player function. In particular, recent mobile terminals can receive multicast signals providing visual content such as broadcasting and video or television programs.

Along with the development of 3D depth camera technology, mobile terminals are also developing a user interface (UI) that controls a device by detecting a user's motion or gesture based on a 3D vision technology. UO based on 3D vision can be applied to various applications by supplementing the existing 2D touch based UI. For example, in an augmented reality (AR) application, an object can be controlled in three dimensions, even when a device is located in a position that the user cannot touch, and the user's hand is contaminated or wearing gloves. It allows you to control the device even when it is difficult to touch, such as when there is one. Accordingly, 3D vision-based gesture recognition technology is in the spotlight.

A gesture recognition technology based on 3D vision is implemented based on a 3D depth camera that can recognize 3D coordinates. The 3D depth camera has a difference in recognizing the distance between the camera and the object compared to the existing 2D camera, and for this, a calculation process for converting the measured raw image into a depth image is added. The processing load of this calculation process is large, so high processing power is used, and accordingly, power consumption according to the operation of the 3D depth camera and the processor processing the 3D image occurs. If this operation is always performed in a battery-operated device with a high level, the operating time of the device is significantly reduced. Accordingly, a separate operation for activating the 3D camera is required, and this additional operation causes inconvenience to the user, thereby deteriorating the practical usability of the 3D vision-based UI.

An object of the present invention is to temporarily drive a 3D depth camera to prevent excessive power consumption, and to provide a user with a convenient UI by driving the depth camera without a user's direct touch.

In order to achieve the above or other objects, according to an aspect of the present invention, in a mobile terminal, in a mobile terminal, a proximity sensor that senses whether an object is close, a depth camera that captures an object to obtain a depth image, and stores at least one command. A memory, the proximity sensor, the depth camera, and a controller connected to the memory, wherein the controller detects proximity by an object based on an initial value of the proximity sensor, according to at least one command stored in the memory And obtaining a depth image through the depth camera, and updating an initial value of the proximity sensor based on the acquired depth unknown.

In addition, according to an aspect of the present invention, the proximity sensor sets the initial value based on the acquired data, and sets a threshold value by adding a preset value based on the set initial value. Provide a terminal.

In addition, according to an aspect of the present invention, when the average value of the acquired data is smaller than the threshold value, the proximity sensor determines that the object is separated, and the average value of the acquired data is greater than the threshold value. In the case of case, it is determined that the object is adjacent.

In addition, according to an aspect of the present invention, when the controller detects the shape of the user's hand through the acquired depth image, it provides a mobile terminal, characterized in that maintaining the initial value of the proximity sensor.

In addition, according to an aspect of the present invention, the controller provides a mobile terminal characterized in that it wakes up when the mobile terminal is in a sleep state.

In addition, according to an aspect of the present invention, the controller extracts an index finger from the acquired depth image based on at least one command stored in the memory, and detects the movement of the index finger. It provides a mobile terminal.

In addition, according to an aspect of the present invention, if the controller cannot detect the user's hand through the depth image continuously acquired for a preset time, it stops acquiring the depth image through the depth camera, and the It provides a mobile terminal, characterized in that acquiring data through a proximity sensor.

In addition, according to an aspect of the present invention, when the controller stops acquiring the depth image through the depth camera, it provides a mobile terminal, characterized in that the mobile terminal is switched to a sleep mode. .

In addition, according to an aspect of the present invention, when detecting an object other than the user's hand through the acquired depth image, the controller checks the depth of the other object, and the depth set distance of the other object In the above case, it provides a mobile terminal, characterized in that updating the initial value of the proximity sensor.

Further, according to an aspect of the present invention, the controller acquires data through the proximity sensor, increases the initial value based on the acquired data, and increases the threshold value in response to the increased initial value. A mobile terminal, characterized in that.

In addition, according to an aspect of the present invention, the controller provides a mobile terminal, characterized in that it updates an initial value of the proximity sensor when an object is not detected within a preset distance through the acquired depth image.

In addition, according to an aspect of the present invention, the controller provides a mobile terminal, wherein the controller obtains data through the proximity sensor and updates the initial value when the acquired data is less than the threshold value.

In addition, according to an aspect of the present invention, the mobile terminal further includes a motion sensor to check the motion of the mobile terminal, and the controller detects that the mobile terminal is in a stable state through the motion sensor. It provides a mobile terminal, characterized in that acquiring data through.

In addition, according to an aspect of the present invention, when the controller fails to sense an object that is close to the data acquired through the proximity sensor for more than a preset time, the controller stops acquiring data through the proximity sensor, and the motion sensor It provides a mobile terminal, characterized in that to check the motion of the mobile terminal through the.

In addition, according to an aspect of the present invention, the stable state is at least one of a state in which the position of the mobile terminal is fixed, a state in which the mobile terminal is moved below a preset speed, and a state in which the mobile terminal rotates below a preset angle. Provides.

The effect of the mobile terminal according to the present invention will be described as follows.

According to at least one of the embodiments of the present invention, it is possible to reduce power consumption required to execute a 3D vision-based UI through a depth camera.

According to at least one of the embodiments of the present invention, the depth image may be acquired through the depth camera without the user directly touching the device.

According to at least one of the embodiments of the present invention, it is possible to prevent the depth camera from being driven due to an error by updating a reference value for determining whether an object is close by the proximity sensor.

According to at least one of the embodiments of the present invention, it is possible to prevent the depth camera from being unnecessarily driven by selectively and statically acquiring data through the proximity sensor according to the state of the mobile terminal.

Further scope of applicability of the present invention will become apparent from the detailed description below. However, since various changes and modifications within the spirit and scope of the present invention can be clearly understood by those skilled in the art, detailed description and specific embodiments such as preferred embodiments of the present invention are understood to be given by way of example only. Should be.

1A is a block diagram illustrating a mobile terminal related to the present invention.

1B and 1C are conceptual diagrams of an example of a mobile terminal according to the present invention viewed from different directions.

2 is a block diagram of the present invention for performing a 3D vision-based UI based on a depth camera, a proximity sensor, and a proximity sensor according to an embodiment of the present invention.

3 is a flowchart illustrating a processor for performing a 3D vision-based UI according to an embodiment of the present invention.

4 is a flowchart illustrating a processor for selectively activating a proximity sensor through a motion sensor according to an embodiment of the present invention.

5 is a flowchart illustrating a processor for setting a threshold value of a proximity sensor and activating a depth camera based on the threshold value of the proximity sensor according to an embodiment of the present invention.

6 is a flowchart illustrating a processor in which a mode for resetting an initial value of a proximity sensor is changed according to an object classified according to an embodiment of the present invention.

7 is a diagram for explaining the necessity of resetting an initial value of a proximity sensor according to an embodiment of the present invention.

8 to 10 are flowcharts illustrating a processor for updating a threshold value of a proximity sensor according to the presence or absence of an object or the type of an object, according to an embodiment of the present invention.

11 and 12 are diagrams for explaining an embodiment of executing a 3D vision-based UI through the present invention.

Hereinafter, exemplary embodiments disclosed in the present specification will be described in detail with reference to the accompanying drawings, but identical or similar elements are denoted by the same reference numerals regardless of reference numerals, and redundant descriptions thereof will be omitted. The suffixes "module" and "unit" for components used in the following description are given or used interchangeably in consideration of only the ease of preparation of the specification, and do not have meanings or roles that are distinguished from each other by themselves. In addition, in describing the embodiments disclosed in the present specification, when it is determined that a detailed description of related known technologies may obscure the subject matter of the embodiments disclosed in the present specification, the detailed description thereof will be omitted. In addition, the accompanying drawings are for easy understanding of the embodiments disclosed in the present specification, and the technical idea disclosed in the present specification is not limited by the accompanying drawings, and all modifications included in the spirit and scope of the present invention It should be understood to include equivalents or substitutes.

Mobile terminals described in this specification include mobile phones, smart phones, laptop computers, digital broadcasting terminals, personal digital assistants (PDAs), portable multimedia players (PMPs), navigation systems, and slate PCs. , Tablet PC (tablet PC), ultrabook (ultrabook), wearable device (wearable device, for example, smartwatch, glass-type terminal (smart glass), HMD (head mounted display)), etc. may be included. have.

However, it is understood that the configuration according to the embodiment described in the present specification may also be applied to a fixed terminal such as a digital TV, a desktop computer, and a digital signage, except for a case applicable only to a mobile terminal. If you are a technician, you can easily find out.

1A to 1C, FIG. 1A is a block diagram illustrating a mobile terminal related to the present invention, and FIGS. 1B and 1C are conceptual diagrams of an example of a mobile terminal related to the present invention viewed from different directions.

The mobile terminal 100 includes a wireless communication unit 110, an input unit 120, a sensing unit 140, an output unit 150, an interface unit 160, a memory 170, a control unit 180, and a power supply unit 190. ), etc. The components shown in FIG. 1A are not essential for implementing the mobile terminal, and thus, the mobile terminal described in the present specification may have more or fewer components than those listed above.

More specifically, among the components, the wireless communication unit 110 may be configured between the mobile terminal 100 and the wireless communication system, between the mobile terminal 100 and another mobile terminal 100, or between the mobile terminal 100 and an external server. It may include one or more modules that enable wireless communication between. In addition, the wireless communication unit 110 may include one or more modules for connecting the mobile terminal 100 to one or more networks.

The wireless communication unit 110 may include at least one of a broadcast reception module 111, a mobile communication module 112, a wireless Internet module 113, a short-range communication module 114, and a location information module 115. .

The input unit 120 includes a camera 121 or an image input unit for inputting an image signal, a microphone 122 for inputting an audio signal, or an audio input unit, and a user input unit 123 for receiving information from a user, for example, , A touch key, a mechanical key, etc.). The voice data or image data collected by the input unit 120 may be analyzed and processed as a user's control command.

The sensing unit 140 may include one or more sensors for sensing at least one of information in the mobile terminal, information on surrounding environments surrounding the mobile terminal, and user information. For example, the sensing unit 140 includes a proximity sensor 141, an illumination sensor 142, a touch sensor, an acceleration sensor, a magnetic sensor, and gravity. G-sensor, gyroscope sensor, motion sensor, RGB sensor, infrared sensor (IR sensor), fingerprint sensor (finger scan sensor), ultrasonic sensor (ultrasonic sensor) , Optical sensor (for example, camera (see 121)), microphone (microphone, see 122), battery gauge, environmental sensor (for example, barometer, hygrometer, thermometer, radiation detection sensor, It may include at least one of a heat sensor, a gas sensor, etc.), and a chemical sensor (eg, an electronic nose, a healthcare sensor, a biometric sensor, etc.). Meanwhile, the mobile terminal disclosed in the present specification may combine and utilize information sensed by at least two or more of these sensors.

The output unit 150 is for generating an output related to visual, auditory, or tactile sense, and may include at least one of a display 151, an audio output unit 152, a hap tip module 153, and a light output unit. The display 151 may form a layer structure with the touch sensor or be integrally formed, thereby implementing a touch screen. Such a touch screen can function as a user input unit 123 that provides an input interface between the mobile terminal 100 and a user, and can provide an output interface between the mobile terminal 100 and a user.

The interface unit 160 serves as a passage between various types of external devices connected to the mobile terminal 100. The interface unit 160 connects a wired/wireless headset port, an external charger port, a wired/wireless data port, a memory card port, and a device equipped with an identification module. It may include at least one of a port, an audio input/output (I/O) port, an input/output (video I/O) port, and an earphone port. The mobile terminal 100 may perform appropriate control related to the connected external device in response to the connection of the external device to the interface unit 160.

In addition, the memory 170 stores data supporting various functions of the mobile terminal 100. The memory 170 may store a plurality of application programs or applications driven by the mobile terminal 100, data for operation of the mobile terminal 100, and commands. At least some of these application programs may be downloaded from an external server through wireless communication. In addition, at least some of these application programs may exist on the mobile terminal 100 from the time of delivery for basic functions of the mobile terminal 100 (eg, incoming calls, outgoing functions, message reception, and outgoing functions). Meanwhile, the application program may be stored in the memory 170, installed on the mobile terminal 100, and driven by the controller 180 to perform an operation (or function) of the mobile terminal.

In addition to the operation related to the application program, the controller 180 generally controls the overall operation of the mobile terminal 100. The controller 180 may provide or process appropriate information or functions to a user by processing signals, data, information, etc. input or output through the above-described components or by driving an application program stored in the memory 170.

Also, in order to drive an application program stored in the memory 170, the controller 180 may control at least some of the components examined together with FIG. 1A. Furthermore, in order to drive the application program, the controller 180 may operate by combining at least two or more of the components included in the mobile terminal 100 with each other.

The power supply unit 190 receives external power and internal power under the control of the controller 180 and supplies power to each of the components included in the mobile terminal 100. The power supply unit 190 includes a battery, and the battery may be a built-in battery or a replaceable battery.

At least some of the components may operate in cooperation with each other to implement an operation, control, or control method of a mobile terminal according to various embodiments described below. In addition, the operation, control, or control method of the mobile terminal may be implemented on the mobile terminal by driving at least one application program stored in the memory 170.

1B and 1C describe basic features of a foldable mobile terminal in an unfolded state.

The mobile terminal 100 includes a display 151, first and second

sound output units

152a and 152b, proximity sensor 141, illuminance sensor 142, light output unit 154, and first and second cameras. (121a, 121b), first and second operation units (123a, 123b), a microphone 122, an interface unit 160, and the like may be provided.

In the following, as shown in FIGS. 1B and 1C, the display 151, the first sound output unit 152a, the proximity sensor 141, the illuminance sensor 142, the light output unit, and 1 The camera 121a and the first operation unit 123a are disposed, the second operation unit 123b, the microphone 122 and the interface unit 160 are disposed on the side of the terminal body, and 2 A mobile terminal 100 in which the sound output unit 152b and the second camera 121b are disposed will be described as an example.

However, these configurations are not limited to this arrangement. These configurations may be excluded or replaced as necessary, or may be disposed on different sides. For example, the first operation unit 123a may not be provided on the front surface of the terminal body, and the second sound output unit 152b may be provided on the side of the terminal body rather than on the rear surface of the terminal body.

The display 151 displays (outputs) information processed by the mobile terminal 100. For example, the display 151 may display execution screen information of an application program driven in the mobile terminal 100, or UI (User Interface) and GUI (Graphic User Interface) information according to such execution screen information.

The display 151 includes a liquid crystal display (LCD), a thin film transistor-liquid crystal display (TFT LCD), an organic light-emitting diode (OLED), and a flexible display. ), a 3D display, and an e-ink display.

In addition, two or more displays 151 may exist depending on the implementation type of the mobile terminal 100. In this case, in the mobile terminal 100, a plurality of displays may be spaced apart or integrally disposed on one surface, or may be disposed on different surfaces, respectively.

The display 151 may include a touch sensor that senses a touch on the display 151 so as to receive a control command by a touch method. Using this, when a touch is made to the display 151, the touch sensor may sense the touch, and the controller 180 may generate a control command corresponding to the touch based on this. Content input by the touch method may be letters or numbers, or menu items that can be indicated or designated in various modes.

Meanwhile, the touch sensor is configured in the form of a film having a touch pattern and is disposed between the window 151a covering the display 151 and a plurality of layers constituting the display 151, or directly on the rear surface of the window 151a. It can also be a metal wire to be patterned. Alternatively, the touch sensor may be integrally formed with the display. For example, the touch sensor may be disposed on a substrate of the display or may be provided inside the display.

As such, the display 151 may form a touch screen together with a touch sensor, and in this case, the touch screen may function as a user input unit 123 (see FIG. 1A). In some cases, the touch screen may replace at least some functions of the first manipulation unit 123a.

The first sound output unit 152a may be implemented as a receiver that transmits a call sound to the user's ear, and the second sound output unit 152b is a loud speaker that outputs various alarm sounds or multimedia playback sounds. ) Can be implemented.

A sound hole for emitting sound generated from the first sound output unit 152a may be formed in the window 151a of the display 151. However, the present invention is not limited thereto, and the sound may be configured to be emitted along an assembly gap between structures (eg, a gap between the window 151a and the front case 101). In this case, the externally formed hole for sound output is not visible or hidden, so that the appearance of the mobile terminal 100 may be more simple.

The light output unit is configured to output light to notify when an event occurs. Examples of the event include message reception, call signal reception, missed call, alarm, schedule notification, e-mail reception, and information reception through an application. When the user's event confirmation is detected, the controller 180 may control the light output unit so that light output is terminated.

The first camera 121a processes an image frame of a still image or moving image obtained by an image sensor in a photographing mode or a video call mode. The processed image frame may be displayed on the display 151 and may be stored in the memory 170.

The first and second manipulation units 123a and 123b are an example of a user input unit 123 that is manipulated to receive a command for controlling the operation of the mobile terminal 100, and may also be collectively referred to as a manipulating portion. have. The first and second operation units 123a and 123b may be employed in any manner as long as the user operates while receiving a tactile feeling such as touch, push, and scroll. In addition, the first and second manipulation units 123a and 123b may also be employed in a manner in which the first and second manipulation units 123a and 123b are operated without a user's tactile feeling through proximity touch, hovering touch, or the like.

In this drawing, the first manipulation unit 123a is illustrated as being a touch key, but the present invention is not limited thereto. For example, the first operation unit 123a may be a push key (mechanical key) or may be configured as a combination of a touch key and a push key.

Contents input by the first and second manipulation units 123a and 123b may be variously set. For example, the first operation unit 123a receives commands such as menu, home key, cancel, search, etc., and the second operation unit 123b is output from the first or second

sound output units

152a, 152b. Commands such as adjusting the volume of sound and switching to the touch recognition mode of the display 151 may be input.

Meanwhile, as another example of the user input unit 123, a rear input unit (not shown) may be provided on the rear surface of the terminal body. This rear input unit is manipulated to receive a command for controlling the operation of the mobile terminal 100, and input contents may be variously set. For example, commands such as power on/off, start, end, scrolling, etc., adjusting the volume of sound output from the first and second

sound output units

152a and 152b, and entering the touch recognition mode of the display 151 Commands such as conversion can be input. The rear input unit may be implemented in a form capable of inputting by a touch input, a push input, or a combination thereof.

The rear input unit may be disposed to overlap the front display 151 in the thickness direction of the terminal body. For example, when the user holds the terminal body with one hand, the rear input unit may be disposed on the rear upper end of the terminal body so that the user can easily manipulate it using the index finger. However, the present invention is not necessarily limited thereto, and the position of the rear input unit may be changed.

When the rear input unit is provided on the rear surface of the terminal body, a new type of user interface can be implemented using the rear input unit. In addition, when the touch screen or the rear input unit described above replaces at least some functions of the first operation unit 123a provided on the front of the terminal body, and the first operation unit 123a is not disposed on the front of the terminal body, The display 151 may be configured with a larger screen.

Meanwhile, the mobile terminal 100 may be provided with a fingerprint recognition sensor 143 for recognizing a user's fingerprint, and the controller 180 may use fingerprint information detected through the fingerprint recognition sensor 143 as an authentication means. . The fingerprint recognition sensor may be embedded in the display 151 or the user input unit 123, or may be provided in a separate location.

The microphone 122 is configured to receive a user's voice and other sounds. The microphone 122 may be provided in a plurality of locations and configured to receive stereo sound.

The interface unit 160 becomes a passage through which the mobile terminal 100 can be connected to an external device. For example, the interface unit 160 is a connection terminal for connection with other devices (eg, earphones, external speakers), a port for short-range communication (eg, an infrared port (IrDA Port), a Bluetooth port (Bluetooth Port), a wireless LAN port, etc.], or at least one of a power supply terminal for supplying power to the mobile terminal 100. The interface unit 160 may be implemented in the form of a socket for accommodating an external card such as a subscriber identification module (SIM) or a user identity module (UIM), or a memory card for storing information.

A second camera 121b may be disposed on the rear surface of the terminal body. In this case, the second camera 121b has a photographing direction substantially opposite to that of the first camera 121a.

The second camera 121b may include a plurality of lenses arranged along at least one line. The plurality of lenses may be arranged in a matrix format. Such a camera may be referred to as an'array camera'. When the second camera 121b is configured as an array camera, an image may be photographed in various ways using a plurality of lenses, and an image of better quality may be obtained.

The flash 124 may be disposed adjacent to the second camera 121b. The flash 124 illuminates light toward the subject when photographing the subject with the second camera 121b.

A second sound output unit 152b may be additionally disposed on the terminal body. The second sound output unit 152b may implement a stereo function together with the first sound output unit 152a, and may be used to implement a speakerphone mode during a call.

At least one antenna for wireless communication may be provided in the terminal body. The antenna may be embedded in the terminal body or may be formed in a case. For example, an antenna forming a part of the broadcast receiving module 111 (refer to FIG. 1A) may be configured to be retractable from the terminal body. Alternatively, the antenna may be formed in a film type and attached to the inner surface of the rear cover 103, or a case including a conductive material may be configured to function as an antenna.

The terminal body is provided with a power supply unit 190 (refer to FIG. 1A) for supplying power to the mobile terminal 100. The power supply unit 190 may include a battery 191 built in the terminal body or configured to be detachable from the outside of the terminal body.

The battery 191 may be configured to receive power through a power cable connected to the interface unit 160. In addition, the battery 191 may be configured to enable wireless charging through a wireless charger. The wireless charging may be implemented by a magnetic induction method or a resonance method (magnetic resonance method).

On the other hand, in this drawing, the rear cover 103 is coupled to the rear case 102 so as to cover the battery 191 to limit the separation of the battery 191, and to protect the battery 191 from external shock and foreign substances. It is illustrated. When the battery 191 is configured to be detachably attached to the terminal body, the rear cover 103 may be detachably coupled to the rear case 102.

Cameras applied to the present invention (camera 121 shown in Fig. 1 and camera 121a shown in Fig. 2A) are located around the touch screen of the mobile device. Accordingly, it is possible to detect depth information on an object (eg, a user's finger) within a certain distance from the touch screen of the mobile device. Such a camera can be referred to as a depth camera.

Two methods are proposed as a method of implementing the depth camera. The first method is a method using a multi-camera (or lens), which captures visible light by using two or more cameras, and generates a 3D image using depth information. The second method is to mount a separate sensor for depth sensing in the camera module, and more specifically, SL (Structured Light) and ToF (Time of Flight) methods are applied.

In the SL method described above, a laser having a specific pattern, such as a straight line or a grid pattern, is radiated onto an object to be photographed, and then the information in which the pattern is deformed according to the shape of the object surface is analyzed. Furthermore, after calculating the depth information, the 3D-based photographing result is derived by combining it with the photograph taken by the image sensor. To implement this, a laser infrared (IR) projector that transmits a specific pattern, an infrared depth sensor, an image sensor, and a 3D processor may be used.

In the above-described ToF method, depth information is calculated by measuring the time the laser travels to the object to be photographed and returns after being reflected, and then synthesizes it with a photograph taken by an image sensor to derive a 3D-based photographing result. To implement this, a laser infrared (IR) projector, a reception sensor, an image sensor, and a 3D processor may be used.

The present invention relates to a system capable of detecting a user's hand through a 3D depth camera 200 and a proximity sensor 300 and activating or deactivating a device accordingly.

The device of the present invention may include a mobile terminal, but is not necessarily limited thereto, and may include a television, an artificial intelligence device, a tablet, etc. that can utilize a 3D vision-based UI.

The depth camera 200 of the present invention may be a TOF (Time-of-Flight) type camera, but is not necessarily limited to this type of camera.

In the proximity sensor 300 of the present invention, a sensor of a method of irradiating light and determining whether or not proximity is determined through the intensity of reflected light may be used, but is not necessarily limited to this type of sensor. A method in which the proximity sensor 300 determines whether an object is in proximity will be described in detail with reference to FIG. 7.

In the present invention, when the proximity sensor 300 detects that an object is close, a depth image is obtained through the depth camera 200, and it is possible to discriminate whether the object corresponds to the user's hand from the acquired depth image.

The present invention may additionally include a motion sensor 400 capable of detecting the state of the device. The motion sensor 400 may detect whether the device exists in a state suitable for the device to perform a 3D based UI.

For example, the motion sensor 400 may detect whether the device is in a stable state suitable for performing a 3D-based UI with the device, such as when the device is on a table or is fixed toward the user.

To this end, the motion sensor 400 may include an acceleration sensor, a gyro sensor, or the like.

The motion sensor 400 may serve as a trigger for activating the proximity sensor 300. That is, only when the device is stable through the motion sensor 400, data for determining whether an object is in proximity may be obtained through the proximity sensor 300.

However, when the device is fixed such as a television, the motion sensor 400 may not be required. In this case, the motion sensor 400 may always be driven in an active state.

Specifically, the motion sensor 400 and the proximity sensor 300 may be controlled through a low power processor 500 that operates with low power.

The low power processor 500 may include a device state checker 510 connected to the motion sensor 400 to detect a motion state of a device through data acquired from the motion sensor 400.

When the device state detection unit 510 detects that the motion of the device is in a stable state for performing a 3D vision-based UI with the user, it activates the proximity sensor 300 through a proximity sensor trigger 520 and whether the object is in proximity. It is possible to obtain data for determining the.

The proximity sensor trigger 520 sets an initial value (=baseline, baseline) with data acquired through the proximity sensor 300, and determines whether an object is in proximity based on the set initial value. I can. In some cases, the initial value may be a value set under controlled experimental conditions.

When detecting that the object is in proximity through the proximity sensor trigger 520, the depth camera 300 may be activated.

The depth camera 300 is connected to a high performance processor 600, and the high performance processor 600 controls the depth camera 300 to obtain a depth image and indexes a user to perform a 3D vision-based UI. A function of tracking a finger may be provided.

Specifically, the high specification processor 600 may include an object classifying unit 610 that controls the depth camera 300 to acquire a depth image, and discriminates whether an object corresponds to a user's hand with the acquired depth image. have.

The depth camera 300 may acquire a depth image in a ready mode and discriminate the shape of the depth image through the object classifier 610.

The ready mode may be activated when an object is close through the proximity sensor 300.

The object classifier 610 may classify an object, and the initial value of the proximity sensor 300 may be updated using a depth image according to the classifying result.

This is an active mode in which the initial value of the proximity sensor 300 is incorrectly set or the vicinity of the proximity sensor 300 is contaminated and the depth camera 300 unnecessarily tracks the index finger of the user's hand. This is to prevent it from working. In this regard, it will be described in detail in FIG. 7.

When the user's hand is recognized through the object classification unit 610, the high-end processor 600 may detect a gesture of the user's hand or activate a tracking unit 620 that tracks an index finger of the user's hand.

In some cases, the high-end processor 600 may further include an object detector 630 that acquires a two-dimensional image through the depth camera 200 and detects the presence or absence of an object. The object detection unit 630 may be a component that replaces the role of the proximity sensor 300.

When the depth camera 200 acquires a two-dimensional image corresponding to the intensity data, the depth camera 200 may be operated in a low power mode through the object detector 630.

While operating the depth camera 200 in the low power mode, when detecting that an object exists, the object classifying unit 610 may be activated to switch to the ready mode.

Since the object detection unit 630 plays the same role as the proximity sensor 300, when the object detection unit 630 is included in the present invention, the proximity sensor 300 and the proximity sensor trigger 520 are omitted, and object detection The unit 630 may be configured to be activated by the device state detection unit 510.

In addition, the low-power processor 500 and the high-end processor 600 may not necessarily be divided into a configuration divided into calculation and processing capabilities, but may be included in one controller.

3 is a flowchart illustrating a processor for performing a 3D vision-based UI according to an embodiment of the present invention. FIG. 3 is a flowchart illustrating a processor for performing a 3D vision based UI based on the system of the present invention disclosed in FIG. 2.

The processor according to the present invention may check the motion of the device through the data acquired by the motion sensor 400 and activate the proximity sensor 300 when the device is in motion suitable for performing a 3D vision-based UI. (S210)

However, if the device is a stationary type other than a portable terminal, such as a mobile terminal, the step of checking the motion of the device through the motion sensor 400 may be omitted. In this case, the proximity sensor 300 may always be in an activated state.

In addition, when a two-dimensional image is obtained through the depth camera 200 according to the present invention and the presence or absence of an object can be detected (= when the object detection unit 630 is present), the object detection unit 630 It may replace the proximity sensor 300.

However, hereinafter, the present invention will be described on the basis of a processor for discriminating whether or not an object is adjacent through the proximity sensor 300.

When the proximity sensor 300 is activated, data is acquired to check whether an object is in proximity, and the depth camera 300 may be activated. (S220) Here, activating the depth camera 300 means operating the depth camera in a ready mode.

The proximity sensor 300 may use various types of sensors, but it is typically possible to discriminate whether or not the object is in proximity by detecting the intensity of light reflected and returned from the object.

The proximity sensor 300 may detect light reflected from an object and return to it, set a base line through an average value, and set a threshold value by adding a preset value from the set base line. When a threshold value is set, the intensity of light detected thereafter can be compared with the threshold value to discriminate whether an object is close.

The object classifying unit 610 is activated when the depth camera 200 is operated in a ready mode, and may identify an object by acquiring a depth image from the depth camera 200. (S230)

The object can be identified through the shape of the depth image and a previously stored database.

In the present invention, since the user's hand is tracked to perform a 3D vision-based UI, whether an object corresponds to a hand or an object other than a hand can be identified, and in some cases, it is possible to discriminate if there is no object.

The initial value of the proximity sensor 300 may be reset according to whether the object corresponds to a hand, whether it is an object other than a hand, or when there is no object. (S240) This is to correct an incorrectly set initial value of the proximity sensor 300, and to prevent an error in sensing due to contamination of adjacent portions.

When the initial value of the proximity sensor 300 is reset and the object is identified by the depth camera 200 activated based on the reset proximity sensor 300, when the object corresponds to the user's hand, the state of the device is determined accordingly. Can be changed (S250)

For example, when the device is in the sleep mode, it wakes up, tracks the index finger of the user's hand, and performs a 3D vision-based UI with the user.

4 is a flowchart illustrating a processor for selectively activating a proximity sensor through a motion sensor according to an embodiment of the present invention. 4 is a flowchart illustrating a processor for executing a 3D vision-based UI through the motion sensor 400 based on the system of the present invention disclosed in FIG. 2.

When the device of the present invention is portable such as a mobile terminal, activating the proximity sensor 300 through the motion sensor 400 may be preferable in terms of power consumption prevention and processor stabilization.

The motion sensor 400 may include an acceleration sensor, a gyro sensor, a magnetic sensor, and the like, and through this, data for determining the motion of the device may be obtained. (S211)

Although 3-axis data may be used to determine device motion, 9 or more data may be used in some cases.

The motion state of the device may be determined through data acquired at preset time intervals through the motion sensor 400 (S212).

It is possible to determine the movement of the device by comparing the data difference between the Nth frame and the N-1th frame with a predetermined reference value, and whether the device is placed on a fixed floor or in a fixed state while looking at the user through axis movement. Can discern whether or not.

When the device is in a stable state suitable for performing a 3D vision-based UI by tracking the user's hand (S213, Yes), the proximity sensor 300 may be activated to obtain data for determining whether an object is in proximity.

When the proximity sensor 300 is activated, acquiring data for detecting motion of the device may be temporarily stopped.

On the other hand, when the device is in a state in which it is difficult to perform a 3D vision-based UI by tracking the user's hand (S213, No), it is not necessary to acquire a depth image, so the proximity sensor 300 is in an inactive state and the device is stable. Until the state is reached, the operation of acquiring data through the motion sensor can be performed.

5 is a flowchart illustrating a processor for setting a threshold value of a proximity sensor and activating a depth camera based on the threshold value of the proximity sensor according to an embodiment of the present invention. Hereinafter, FIG. 5 will be described based on the system of the present invention disclosed in FIG. 2.

When the proximity sensor 300 is activated, light may be irradiated to determine whether an object is in proximity, and data corresponding to the intensity of light reflected and returned may be obtained (S221).

However, the method for the proximity sensor 300 to determine whether the object is in proximity is not limited to detecting the reflected light.

The proximity sensor 300 may set an initial value (=baseline) by averaging data of a preset number of frames acquired after being activated. (S222) In some cases, the present invention may set a preset value as an initial value.

By adding a preset value to the initial value, a threshold value, which is a reference value for discriminating whether an object is close or not, can be set. (S223)

When the data acquired by the proximity sensor 300 is greater than the threshold value, the object may be identified as a proximity state. Conversely, when the data acquired by the proximity sensor 300 is smaller than the threshold value, the object may be identified as being far away.

When the data acquired by the proximity sensor 300 exceeds the threshold value (S224, Yes), the object may be identified as being close and the depth camera 200 may be activated (in a ready mode). (S225)

On the other hand, when the data acquired by the proximity sensor 300 does not exceed the threshold value, data may be continuously acquired through the proximity sensor 300 without activating the depth camera 200. (S224, No) At this time, when the motion of the device is out of the stable state, the proximity sensor 300 may be deactivated and data acquisition may be stopped.

6 is a flowchart illustrating a processor in which a mode for resetting an initial value of a proximity sensor is changed according to an object classified according to an embodiment of the present invention. Hereinafter, FIG. 6 will be described based on FIG. 2.

When it is determined that the object is close through the proximity sensor 300, the depth carrera 200 is activated in the ready mode. In the ready mode, the depth camera 200 acquires a depth image. (S231)

The object classification unit 610 determines whether an object exists within a preset distance through the acquired depth image. (S232)

When there is no object within a preset distance (S232, No), the initial value of the proximity sensor 200 may be reset in response to the absence of the object. (S234) If there is no object within the preset distance, the initial value of the proximity sensor 300 may be set too low, or a contaminant may be trapped near the proximity sensor 300

When there is an object within a preset distance (S232, Yes), the object classifying unit 610 may discriminate whether the object is a hand or not through a previously stored database. (S233)

The initial value of the proximity sensor may be reset according to whether the object corresponds to a hand or an object other than a hand. (S234)

Previously, the threshold values 302 and 303 were set as absolute values as shown in FIG. 7(a), and the data 301 acquired through the proximity sensor 200 and the threshold values 302 and 302 were compared to the object. It was judged whether or not the proximity of

You can set one threshold value, but you can set two threshold values for more clarity. That is, when the data is higher than the first threshold value 302, it is determined that the object is near, and when the data is lower than the second threshold value 303, it is determined that the object is far away.

However, when the threshold value is absolutely set as shown in FIG. 7(b), data existing between the first threshold value 302 and the second threshold value 303 is also located near the proximity sensor 300. If contamination occurs and the overall data value increases, it may be incorrectly determined that the object is near.

Therefore, in order to prevent such misjudgment, the proximity sensor 200 sets an initial value (= base line, 304) by averaging a preset frame as shown in FIG. 7(c), and based on the initial value 304 When the threshold value is relatively set above the set value, even if the data value increases as a whole due to contamination of the adjacent portion of the proximity sensor 300, whether or not the object is close can be identified without mistake.

However, the initial value of the proximity sensor 300 may be reset based on the depth image, which will be described below.

8 to 10 are flowcharts illustrating a processor for updating a threshold value of a proximity sensor according to the presence or absence of an object or the type of an object, according to an embodiment of the present invention. Hereinafter, FIGS. 8 to 10 will be described based on FIG. 2.

Specifically, FIG. 8 is a flowchart of a case where a nearby object corresponds to a user's hand, FIG. 9 is a flowchart of a case where a nearby object corresponds to an object other than the user's hand, and FIG. 10 is a case where there is no adjacent object. It is a flow chart.

When a nearby object is identified by the user's hand, since it was appropriate for the proximity sensor 200 to activate the depth camera 300, the initial value of the proximity sensor 200 is not separately reset.

Specifically, referring to FIG. 8, when a nearby object is identified by a user's hand (S2411), the object classifying unit 610 transmits it to the proximity sensor trigger 520, and the proximity sensor trigger 520 is a proximity sensor ( 300) to obtain data (S2412)

The data acquired through the proximity sensor 300 is compared with the previously set threshold (S2413), when it is small, the object is determined to be far away (S2414), and the motion is checked through the motion sensor 300 (S2145). The existing flow can be maintained.

In this case, the depth camera 300 operates in an active mode and can track the index finger of the user's hand.

However, when the adjacent object is an object other than the user's hand, the depth camera 200 may not operate in an active mode and may reset an initial value of the proximity sensor 300 based on the acquired depth image.

Specifically, referring to FIG. 9, when a nearby object is classified as an object other than a user's hand (S2421), the object classification unit 610 determines whether the object exists less than a preset distance (S2422). This is transmitted to the proximity sensor trigger 520.

The proximity sensor trigger 520 may maintain an existing flow without resetting the initial value of the proximity sensor when the object exists within a preset distance (S2422, Yes).

That is, it maintains that the object is determined to be in proximity (S2423), and then, through data acquired through the proximity sensor (S2424), compares it with the existing threshold value (S2425), and continues to acquire data according to proximity. Or (S2424), it is possible to stop acquiring data through the proximity sensor (=proximity state definition (Far), S2426) and check the motion through the motion sensor (S2427).

However, the proximity sensor trigger 520 may update the baseline and threshold values based on the data (S2424) acquired through the proximity sensor 300 after the object exists when the object exceeds a preset distance. have. (S2425)

That is, since the proximity state is redefined by the updated threshold value (S2426), acquiring data through the proximity sensor may be stopped, and motion may be confirmed through the motion sensor. (S2427)

If there is no nearby object, the proximity sensor 300 incorrectly activates the depth camera, and thus the initial value of the proximity sensor 300 needs to be reset.

Specifically, referring to FIG. 10, when it is determined that there is no object in proximity through the depth image (S2431), the object classification unit 610 transmits this to the proximity sensor trigger 520 and again through the proximity sensor 2431. Acquire the data. (S2432)

If the re-acquired data is less than the existing threshold (S2433, Yes), the existing threshold value is correct, but it is determined that the depth camera has been incorrectly active due to another factor, and the proximity of the object is redefined as far away (S2435). , You can proceed to the step of checking the motion through the motion sensor (S2436).

However, if the re-acquired data is greater than or equal to the existing threshold (S2433, Yes), this means that the existing threshold value is not correct, and the baseline and threshold values are reset through the re-acquired data (S2434).

Since the proximity state is redefined by the updated threshold value (S2435), it is possible to stop acquiring data through the proximity sensor and proceed to the step of checking the motion through the motion sensor. (S2436)

11 and 12 are diagrams for explaining an embodiment of executing a 3D vision-based UI through the present invention. Hereinafter, FIGS. 11 and 12 will be described based on FIG. 2.

11(a), when the user's hand 700 approaches the device 100, the proximity sensor 300 detects the proximity of the user's hand 700 and activates the depth camera 300 (ready Mode). When the depth image acquired in the ready mode includes the shape of the user's hand 700, the device may be switched from the sleep mode to the active mode.

That is, while the device is in the sleep state, the user can switch the device to the active state without touching it.

In this case, when the device 100 is portable, the proximity sensor 300 may detect whether an object is in proximity only in a stable state suitable for the device to execute a 3D vision-based UI using the motion sensor 400.

As shown in FIG. 11(b), when the user's hand 700 approaches the device and then moves away from the device in the active mode, the device may be switched from the active mode to the sleep mode.

When the device is in an active mode, the depth camera 300 is not always in a ready mode or an active mode. That is, the proximity sensor 300 may detect the proximity of an object in the device active mode, and may switch the depth camera to the ready mode. After the user's hand is identified from the acquired depth image, and when it is sensed that the identified user's hand is moving away, the device may be switched from the active mode to the sleep mode.

That is, while the device is unnecessarily active, the user can switch the device to the sleep mode without touching it.

Looking at a series of processes at once, as shown in FIG. 11, the device 100 in a sleep state detects whether an object is in proximity through the proximity sensor 300, and when the object is close, the depth camera 200 is activated so that the object is When the user's hand is identified and a nearby object is identified by the user's hand, the device is switched to a mode that tracks the index finger of the user's hand and executes the 3D vision-based UI. It switches to sleep mode and stops acquiring depth images through the depth camera.

That is, the user can be active without touching the device 100, and can selectively acquire a depth image through the depth camera and execute the 3D vision board UI at low power. In addition, it is possible to prevent the depth camera from being unnecessarily activated by resetting the initial value of the proximity sensor to prevent an error from determining whether an object is in proximity.

The above detailed description should not be construed as restrictive in all respects and should be considered as illustrative. The scope of the present invention should be determined by reasonable interpretation of the appended claims, and all changes within the equivalent scope of the present invention are included in the scope of the present invention.

Claims

In the mobile terminal,

A proximity sensor that senses whether an object is in proximity;

A depth camera that captures an object to obtain a depth image;

A memory for storing at least one command; And

Includes; a controller connected to the proximity sensor, the depth camera, and the memory,

The controller is

When the proximity of the object is sensed based on the initial value of the proximity sensor, a depth image is obtained through the depth camera according to at least one command stored in the memory, and the proximity sensor is based on the acquired depth image. Mobile terminal, characterized in that to update the initial value of.
The method of claim 1,

The proximity sensor

And setting the initial value based on the acquired data, and setting a threshold value by adding a preset value based on the set initial value.
The method of claim 2,

The proximity sensor

And if the average value of the acquired data is less than the threshold value, it is determined that the object is separated, and if the average value of the acquired data is greater than the threshold value, it is determined that the object is close.
The method of claim 1,

The controller is

When detecting the shape of the user's hand through the acquired depth image, the mobile terminal, wherein the initial value of the proximity sensor is maintained.
The method of claim 4,

The controller is

A mobile terminal, characterized in that wake-up when the mobile terminal is in a sleep state.
The method of claim 4,

The controller is

And extracting an index finger from the acquired depth image based on at least one command stored in the memory, and detecting a movement of the index finger.
The method of claim 6,

The controller is

A mobile terminal, characterized in that, when the user's hand cannot be detected through the depth image continuously acquired for a preset time, it stops acquiring the depth image through the depth camera and acquires data through the proximity sensor. .
The method of claim 7,

The controller is

And when the acquisition of the depth image is stopped through the depth camera, the mobile terminal is switched to a sleep mode.
The method of claim 1,

The controller is

When an object other than the user's hand is detected through the acquired depth image, the depth of the other object is checked,

And updating the initial value of the proximity sensor when the depth of the other object is greater than or equal to a preset distance.
The method of claim 8,

The controller is

Obtaining data through the proximity sensor, increasing the initial value based on the obtained data, and increasing the threshold value in response to the increased initial value.
The method of claim 1,

The controller is

When the object is not detected within a preset distance through the acquired depth image, the mobile terminal, wherein the initial value of the proximity sensor is updated.
The method of claim 11,

The controller is

Acquiring data through the proximity sensor, and updating the initial value when the acquired data is less than the threshold value.
The method of claim 1,

The mobile terminal

Further comprising a motion sensor for checking the motion of the mobile terminal,

The controller is

And when the motion sensor detects that the mobile terminal is in a stable state, data is obtained through the proximity sensor.
The method of claim 13,

The controller is

When it is not possible to sense an object that is close to the data acquired through the proximity sensor for more than a preset time, the acquisition of data through the proximity sensor is stopped, and motion of the mobile terminal is checked through the motion sensor. Mobile terminal.
The method of claim 13,

The stable state is

A mobile terminal, characterized in that at least one of a fixed position of the mobile terminal, a state of moving below a preset speed, and a state of rotating below a preset angle.