WO2021033809A1 - Mobile terminal - Google Patents

Abstract

The present invention provides a mobile terminal comprising: a plurality of depth cameras for acquiring depth images by photographing at particular angles of view, respectively, wherein photographing areas are partially overlapped; an output unit for outputting audio-visual data; and a control unit for recognizing a motion of a user's hand by using the acquired depth images, and controlling the output unit in response to the recognized motion of the user's hand, wherein, when a part of the shape of the user's hand is included in a first depth image acquired via a first depth camera among the plurality of depth cameras, the control unit acquires the entire shape of the user's hand by merging, into the first depth image, a second depth image acquired via a second depth camera among the plurality of depth cameras, which captures an adjacent area of an area captured by the first depth camera.

Description

Mobile terminal

The present invention relates to a mobile terminal. More specifically, it can be applied to a technical field that provides an input signal through a user's hand motion without a direct touch.

Terminals can be divided into mobile/portable terminals and stationary terminals depending on whether they can be moved. Again, mobile terminals can be divided into handheld terminals and vehicle mounted terminals depending on whether the user can directly carry them.

The functions of mobile terminals are diversifying. For example, there are functions of data and voice communication, taking pictures and videos through a camera, recording voice, playing music files through a speaker system, and outputting images or videos to the display. Some terminals add an electronic game play function or perform a multimedia player function. In particular, recent mobile terminals can receive multicast signals providing visual content such as broadcasting and video or television programs.

Along with the development of 3D depth camera technology, mobile terminals are also developing a user interface (UI) that controls a device by detecting a user's motion or gesture based on a 3D vision technology. UO based on 3D vision can be applied to various applications by supplementing the existing 2D touch based UI. For example, in an augmented reality (AR) application, an object can be controlled in three dimensions, even when a device is located in a position that the user cannot touch, and the user's hand is contaminated or wearing gloves. It allows you to control the device even when it is difficult to touch, such as when there is one. Accordingly, 3D vision-based gesture recognition technology is in the spotlight.

The present invention is characterized in expanding the recognition range of a user's hand motion through a plurality of depth cameras.

In order to achieve the above or other objects, the present invention, according to an embodiment, obtains a depth image by photographing each at a constant angle of view, a plurality of depth cameras in which the photographing area partially overlaps, an output unit for outputting audiovisual data, the A control unit that recognizes a motion of a user's hand using the acquired depth image, and controls the output unit in response to the recognized user hand motion, wherein the control unit is acquired through a first depth camera among the plurality of depth cameras. When a part of the shape of the user's hand is included in the first depth image, the second depth image obtained through a second depth camera that photographs an area adjacent to the area photographed by the first depth camera among the plurality of depth cameras is It provides a mobile terminal, characterized in that the merger with the first depth image to obtain the overall shape of the user's hand.

In addition, according to an embodiment of the present invention, the control unit is a movement characterized in that the second depth image acquired by the second depth camera includes the rest of the shape of the user's hand, but does not include the entire shape of the user's hand. Provide a terminal.

In addition, according to an embodiment of the present invention, when the second depth camera cannot be identified among the plurality of depth cameras, the control unit provides an alarm for guiding the user's hand into the field of view of the first depth camera. It provides a mobile terminal characterized in that it controls the output unit.

In addition, according to an embodiment of the present invention, when the entire shape of the user's hand is included in the second depth image, the controller skips merging of the depth image and determines the overall shape of the user's hand through the second depth image. It provides a mobile terminal characterized in that the acquisition.

In addition, according to an embodiment of the present invention, the plurality of depth cameras overlap the photographing area at a specific distance or more, and the control unit includes the first depth image and the first depth image when the user's hand is positioned at the specific distance or more. It provides a mobile terminal, characterized in that a second depth image is merged to obtain an overall shape of a user's hand.

In addition, according to an embodiment of the present invention, when the controller recognizes that the user's hand is located within the specific distance through the first depth image, the user's hand is positioned within the field of view of the first depth camera. It provides a mobile terminal, characterized in that providing the alarm.

In addition, according to an embodiment of the present invention, when the controller recognizes that the user's hand is located within the specific distance through the first depth image, the control unit generates the alarm so that the user's hand is positioned at a distance greater than or equal to the specific distance. It provides a mobile terminal characterized in that to provide.

In addition, according to an embodiment of the present invention, the plurality of depth cameras acquire depth images for each successive frame, and the control unit uses the first depth image and the second depth image for each successive frame. It provides a mobile terminal, characterized in that acquiring the overall shape of the user's hand and recognizing the motion of the user's hand through a change in the overall shape of the user's hand acquired for each successive frame.

The effect of the mobile terminal according to the present invention will be described as follows.

The present invention can widen the range of motion recognition of the user's hand through a plurality of depth cameras.

The present invention can recognize the entire shape of the user's hand even when the user's hand is divided and photographed through a plurality of depth cameras.

The present invention provides a guide according to the position of the user's hand, and the overall shape of the user's hand can be recognized through a plurality of depth cameras.

Further scope of applicability of the present invention will become apparent from the detailed description below. However, since various changes and modifications within the spirit and scope of the present invention can be clearly understood by those skilled in the art, detailed description and specific embodiments such as preferred embodiments of the present invention are understood to be given by way of example only. Should be.

1A is a block diagram illustrating a mobile terminal related to the present invention.

1B and 1C are conceptual diagrams of an example of a mobile terminal according to the present invention viewed from different directions.

FIG. 2 is a conceptual diagram illustrating a photographing area photographed through a single depth camera in a mobile terminal related to the present invention.

3 illustrates a depth image acquired through a single depth camera in a mobile terminal related to the present invention.

4 to 6 are conceptual diagrams for describing a photographing area photographed through a plurality of depth cameras in a mobile terminal related to the present invention.

7 to 9 illustrate depth images acquired through a plurality of depth cameras in a mobile terminal related to the present invention.

10 is a flowchart illustrating an entire processor of the present invention that senses a motion of a user's hand through a plurality of depth cameras and provides a corresponding function.

11 is a diagram for describing a method of merging depth images acquired through a plurality of depth cameras.

12 illustrates an embodiment of acquiring an overall shape of a user's hand by merging depth images captured through a plurality of depth cameras in a mobile terminal related to the present invention.

13 and 14 are diagrams for explaining another application example of expanding a photographing area through a plurality of depth cameras.

Hereinafter, exemplary embodiments disclosed in the present specification will be described in detail with reference to the accompanying drawings, but identical or similar elements are denoted by the same reference numerals regardless of reference numerals, and redundant descriptions thereof will be omitted. The suffixes "module" and "unit" for components used in the following description are given or used interchangeably in consideration of only the ease of preparation of the specification, and do not have meanings or roles that are distinguished from each other by themselves. In addition, in describing the embodiments disclosed in the present specification, when it is determined that a detailed description of related known technologies may obscure the subject matter of the embodiments disclosed in the present specification, the detailed description thereof will be omitted. In addition, the accompanying drawings are for easy understanding of the embodiments disclosed in the present specification, and the technical idea disclosed in the present specification is not limited by the accompanying drawings, and all modifications included in the spirit and scope of the present invention It should be understood to include equivalents or substitutes.

Mobile terminals described in this specification include mobile phones, smart phones, laptop computers, digital broadcasting terminals, personal digital assistants (PDAs), portable multimedia players (PMPs), navigation systems, and slate PCs. , Tablet PC (tablet PC), ultrabook (ultrabook), wearable device (wearable device, for example, smartwatch, glass-type terminal (smart glass), HMD (head mounted display)), etc. may be included. have.

However, it is understood that the configuration according to the embodiment described in the present specification may also be applied to a fixed terminal such as a digital TV, a desktop computer, and a digital signage, except for a case applicable only to a mobile terminal. If you are a technician, you can easily find out.

1A to 1C, FIG. 1A is a block diagram illustrating a mobile terminal related to the present invention, and FIGS. 1B and 1C are conceptual diagrams of an example of a mobile terminal related to the present invention viewed from different directions.

The mobile terminal 100 includes a wireless communication unit 110, an input unit 120, a sensing unit 140, an output unit 150, an interface unit 160, a memory 170, a control unit 180, and a power supply unit 190. ), etc. The components shown in FIG. 1A are not essential for implementing the mobile terminal, and thus, the mobile terminal described in the present specification may have more or fewer components than those listed above.

More specifically, among the components, the wireless communication unit 110 may be configured between the mobile terminal 100 and the wireless communication system, between the mobile terminal 100 and another mobile terminal 100, or between the mobile terminal 100 and an external server. It may include one or more modules that enable wireless communication between. In addition, the wireless communication unit 110 may include one or more modules for connecting the mobile terminal 100 to one or more networks.

The wireless communication unit 110 may include at least one of a broadcast reception module 111, a mobile communication module 112, a wireless Internet module 113, a short-range communication module 114, and a location information module 115. .

The input unit 120 includes a camera 121 or an image input unit for inputting an image signal, a microphone 122 for inputting an audio signal, or an audio input unit, and a user input unit 123 for receiving information from a user, for example, , A touch key, a mechanical key, etc.). The voice data or image data collected by the input unit 120 may be analyzed and processed as a user's control command.

The sensing unit 140 may include one or more sensors for sensing at least one of information in the mobile terminal, information on surrounding environments surrounding the mobile terminal, and user information. For example, the sensing unit 140 includes a proximity sensor 141, an illumination sensor 142, a touch sensor, an acceleration sensor, a magnetic sensor, and gravity. G-sensor, gyroscope sensor, motion sensor, RGB sensor, infrared sensor (IR sensor), fingerprint sensor (finger scan sensor), ultrasonic sensor (ultrasonic sensor) , Optical sensor (for example, camera (see 121)), microphone (microphone, see 122), battery gauge, environmental sensor (for example, barometer, hygrometer, thermometer, radiation detection sensor, It may include at least one of a heat sensor, a gas sensor, etc.), and a chemical sensor (eg, an electronic nose, a healthcare sensor, a biometric sensor, etc.). Meanwhile, the mobile terminal disclosed in the present specification may combine and utilize information sensed by at least two or more of these sensors.

The output unit 150 is for generating an output related to visual, auditory, or tactile sense, and may include at least one of a display 151, an audio output unit 152, a hap tip module 153, and a light output unit. The display 151 may form a layer structure with the touch sensor or be integrally formed, thereby implementing a touch screen. Such a touch screen can function as a user input unit 123 that provides an input interface between the mobile terminal 100 and a user, and can provide an output interface between the mobile terminal 100 and a user.

The interface unit 160 serves as a passage between various types of external devices connected to the mobile terminal 100. The interface unit 160 connects a wired/wireless headset port, an external charger port, a wired/wireless data port, a memory card port, and a device equipped with an identification module. It may include at least one of a port, an audio input/output (I/O) port, an input/output (video I/O) port, and an earphone port. The mobile terminal 100 may perform appropriate control related to the connected external device in response to the connection of the external device to the interface unit 160.

In addition, the memory 170 stores data supporting various functions of the mobile terminal 100. The memory 170 may store a plurality of application programs or applications driven by the mobile terminal 100, data for operation of the mobile terminal 100, and commands. At least some of these application programs may be downloaded from an external server through wireless communication. In addition, at least some of these application programs may exist on the mobile terminal 100 from the time of delivery for basic functions of the mobile terminal 100 (eg, incoming calls, outgoing functions, message reception, and outgoing functions). Meanwhile, the application program may be stored in the memory 170, installed on the mobile terminal 100, and driven by the controller 180 to perform an operation (or function) of the mobile terminal.

In addition to the operation related to the application program, the controller 180 generally controls the overall operation of the mobile terminal 100. The controller 180 may provide or process appropriate information or functions to a user by processing signals, data, information, etc. input or output through the above-described components or by driving an application program stored in the memory 170.

Also, in order to drive an application program stored in the memory 170, the controller 180 may control at least some of the components examined together with FIG. 1A. Furthermore, in order to drive the application program, the controller 180 may operate by combining at least two or more of the components included in the mobile terminal 100 with each other.

The power supply unit 190 receives external power and internal power under the control of the controller 180 and supplies power to each of the components included in the mobile terminal 100. The power supply unit 190 includes a battery, and the battery may be a built-in battery or a replaceable battery.

At least some of the components may operate in cooperation with each other to implement an operation, control, or control method of a mobile terminal according to various embodiments described below. In addition, the operation, control, or control method of the mobile terminal may be implemented on the mobile terminal by driving at least one application program stored in the memory 170.

1B and 1C describe basic features of a foldable mobile terminal in an unfolded state.

The mobile terminal 100 includes a display 151, first and second sound output units 152a and 152b, proximity sensor 141, illuminance sensor 142, light output unit 154, and first and second cameras. (121a, 121b), first and second operation units (123a, 123b), a microphone 122, an interface unit 160, and the like may be provided.

In the following, as shown in FIGS. 1B and 1C, the display 151, the first sound output unit 152a, the proximity sensor 141, the illuminance sensor 142, the light output unit, and 1 The camera 121a and the first operation unit 123a are disposed, the second operation unit 123b, the microphone 122 and the interface unit 160 are disposed on the side of the terminal body, and 2 A mobile terminal 100 in which the sound output unit 152b and the second camera 121b are disposed will be described as an example.

However, these configurations are not limited to this arrangement. These configurations may be excluded or replaced as necessary, or may be disposed on different sides. For example, the first operation unit 123a may not be provided on the front surface of the terminal body, and the second sound output unit 152b may be provided on the side of the terminal body rather than on the rear surface of the terminal body.

The display 151 displays (outputs) information processed by the mobile terminal 100. For example, the display 151 may display execution screen information of an application program driven in the mobile terminal 100, or UI (User Interface) and GUI (Graphic User Interface) information according to the execution screen information.

The display 151 includes a liquid crystal display (LCD), a thin film transistor-liquid crystal display (TFT LCD), an organic light-emitting diode (OLED), and a flexible display. ), a 3D display, and an e-ink display.

In addition, two or more displays 151 may exist depending on the implementation type of the mobile terminal 100. In this case, in the mobile terminal 100, a plurality of displays may be spaced apart or integrally disposed on one surface, or may be disposed on different surfaces, respectively.

The display 151 may include a touch sensor that senses a touch on the display 151 so as to receive a control command by a touch method. Using this, when a touch is made to the display 151, the touch sensor may sense the touch, and the controller 180 may generate a control command corresponding to the touch based on this. Content input by the touch method may be letters or numbers, or menu items that can be indicated or designated in various modes.

Meanwhile, the touch sensor is configured in the form of a film having a touch pattern and is disposed between the window 151a covering the display 151 and a plurality of layers constituting the display 151, or directly on the rear surface of the window 151a. It can also be a metal wire to be patterned. Alternatively, the touch sensor may be integrally formed with the display. For example, the touch sensor may be disposed on a substrate of the display or may be provided inside the display.

As such, the display 151 may form a touch screen together with a touch sensor, and in this case, the touch screen may function as a user input unit 123 (see FIG. 1A). In some cases, the touch screen may replace at least some functions of the first manipulation unit 123a.

The first sound output unit 152a may be implemented as a receiver that transmits a call sound to the user's ear, and the second sound output unit 152b is a loud speaker that outputs various alarm sounds or multimedia playback sounds. ) Can be implemented.

A sound hole for emitting sound generated from the first sound output unit 152a may be formed in the window 151a of the display 151. However, the present invention is not limited thereto, and the sound may be configured to be emitted along an assembly gap between structures (eg, a gap between the window 151a and the front case 101). In this case, the externally formed hole for sound output is not visible or hidden, so that the appearance of the mobile terminal 100 may be more simple.

The light output unit is configured to output light to notify when an event occurs. Examples of the event include message reception, call signal reception, missed call, alarm, schedule notification, e-mail reception, and information reception through an application. When the user's event confirmation is detected, the controller 180 may control the light output unit so that light output is terminated.

The first camera 121a processes an image frame of a still image or moving image obtained by an image sensor in a photographing mode or a video call mode. The processed image frame may be displayed on the display 151 and may be stored in the memory 170.

The first and second manipulation units 123a and 123b are an example of a user input unit 123 that is manipulated to receive a command for controlling the operation of the mobile terminal 100, and may also be collectively referred to as a manipulating portion. have. The first and second operation units 123a and 123b may be employed in any manner as long as the user operates while receiving a tactile feeling such as touch, push, and scroll. In addition, the first and second manipulation units 123a and 123b may also be employed in a manner in which the first and second manipulation units 123a and 123b are operated without a user's tactile feeling through proximity touch, hovering touch, or the like.

In this drawing, the first manipulation unit 123a is illustrated as being a touch key, but the present invention is not limited thereto. For example, the first operation unit 123a may be a push key (mechanical key) or may be configured as a combination of a touch key and a push key.

Contents input by the first and second manipulation units 123a and 123b may be variously set. For example, the first operation unit 123a receives commands such as menu, home key, cancel, search, etc., and the second operation unit 123b is output from the first or second sound output units 152a, 152b. Commands such as adjusting the volume of sound and switching to the touch recognition mode of the display 151 may be input.

Meanwhile, as another example of the user input unit 123, a rear input unit (not shown) may be provided on the rear surface of the terminal body. This rear input unit is manipulated to receive a command for controlling the operation of the mobile terminal 100, and input contents may be variously set. For example, commands such as power on/off, start, end, scrolling, etc., adjusting the volume of sound output from the first and second sound output units 152a and 152b, and entering the touch recognition mode of the display 151 Commands such as conversion can be input. The rear input unit may be implemented in a form capable of inputting by a touch input, a push input, or a combination thereof.

The rear input unit may be disposed to overlap the front display 151 in the thickness direction of the terminal body. As an example, when the user holds the terminal body with one hand, the rear input unit may be disposed on the rear upper end of the terminal body so that the user can easily manipulate the terminal body using the index finger. However, the present invention is not necessarily limited thereto, and the position of the rear input unit may be changed.

When the rear input unit is provided on the rear surface of the terminal body, a new type of user interface can be implemented using the rear input unit. In addition, when the touch screen or the rear input unit described above replaces at least some functions of the first operation unit 123a provided on the front of the terminal body, and the first operation unit 123a is not disposed on the front of the terminal body, The display 151 may be configured with a larger screen.

Meanwhile, the mobile terminal 100 may be provided with a fingerprint recognition sensor 143 for recognizing a user's fingerprint, and the controller 180 may use fingerprint information detected through the fingerprint recognition sensor 143 as an authentication means. . The fingerprint recognition sensor may be embedded in the display 151 or the user input unit 123, or may be provided in a separate location.

The microphone 122 is configured to receive a user's voice and other sounds. The microphone 122 may be provided in a plurality of locations and configured to receive stereo sound.

The interface unit 160 becomes a passage through which the mobile terminal 100 can be connected to an external device. For example, the interface unit 160 is a connection terminal for connection with other devices (eg, earphones, external speakers), a port for short-range communication (eg, an infrared port (IrDA Port), a Bluetooth port (Bluetooth Port), a wireless LAN port, etc.], or at least one of a power supply terminal for supplying power to the mobile terminal 100. The interface unit 160 may be implemented in the form of a socket for accommodating an external card such as a subscriber identification module (SIM) or a user identity module (UIM), or a memory card for storing information.

A second camera 121b may be disposed on the rear surface of the terminal body. In this case, the second camera 121b has a photographing direction substantially opposite to that of the first camera 121a.

The second camera 121b may include a plurality of lenses arranged along at least one line. The plurality of lenses may be arranged in a matrix format. Such a camera may be referred to as an'array camera'. When the second camera 121b is configured as an array camera, an image may be photographed in various ways using a plurality of lenses, and an image of better quality may be obtained.

The flash 124 may be disposed adjacent to the second camera 121b. The flash 124 illuminates light toward the subject when the subject is photographed by the second camera 121b.

A second sound output unit 152b may be additionally disposed on the terminal body. The second sound output unit 152b may implement a stereo function together with the first sound output unit 152a, and may be used to implement a speakerphone mode during a call.

At least one antenna for wireless communication may be provided in the terminal body. The antenna may be embedded in the terminal body or may be formed in a case. For example, an antenna forming a part of the broadcast receiving module 111 (refer to FIG. 1A) may be configured to be retractable from the terminal body. Alternatively, the antenna may be formed in a film type and attached to the inner surface of the rear cover 103, or a case including a conductive material may be configured to function as an antenna.

The terminal body is provided with a power supply unit 190 (refer to FIG. 1A) for supplying power to the mobile terminal 100. The power supply unit 190 may include a battery 191 built in the terminal body or configured to be detachable from the outside of the terminal body.

The battery 191 may be configured to receive power through a power cable connected to the interface unit 160. In addition, the battery 191 may be configured to enable wireless charging through a wireless charger. The wireless charging may be implemented by a magnetic induction method or a resonance method (magnetic resonance method).

On the other hand, in this drawing, the rear cover 103 is coupled to the rear case 102 so as to cover the battery 191 to limit the separation of the battery 191, and to protect the battery 191 from external shock and foreign substances. It is illustrated. When the battery 191 is configured to be detachably attached to the terminal body, the rear cover 103 may be detachably coupled to the rear case 102.

The cameras applied to the present invention (the camera 121 shown in Fig. 1 and the camera 121a shown in Fig. 1A) are located around the touch screen of the mobile device. Accordingly, it is possible to detect depth information on an object (eg, a user's finger) within a certain distance from the touch screen of the mobile device. Such a camera can be referred to as a depth camera.

Two methods are proposed as a method of implementing the depth camera. The first method is a method using a multi-camera (or lens), which captures visible light by using two or more cameras, and generates a 3D image using depth information. The second method is to mount a separate sensor for depth sensing in the camera module, and more specifically, SL (Structured Light) and ToF (Time of Flight) methods are applied.

In the SL method described above, a laser having a specific pattern, such as a straight line or a grid pattern, is radiated onto an object to be photographed, and then information in which the pattern is deformed according to the shape of the object surface is analyzed. Furthermore, after calculating the depth information, the 3D-based photographing result is derived by combining it with the photograph taken by the image sensor. To implement this, a laser infrared (IR) projector that transmits a specific pattern, an infrared depth sensor, an image sensor, and a 3D processor may be used.

In the above-described ToF method, depth information is calculated by measuring the time the laser travels to the object to be photographed and returns after being reflected, and then synthesizes it with a photograph taken by an image sensor to derive a 3D-based photographing result. To implement this, a laser infrared (IR) projector, a reception sensor, an image sensor, and a 3D processor may be used.

FIG. 2 is a conceptual diagram illustrating a photographing area photographed through a single depth camera in a mobile terminal related to the present invention.

The mobile terminal 200 of the present invention includes a depth camera 300 and may recognize a 3D motion of the user's hand 400 through the depth camera 300. The mobile terminal 200 of the present invention may receive a 3D motion of the user hand 400 recognized through the depth camera 300 as an input signal controlling the mobile terminal 200.

The depth camera 300 of the present invention may photograph a subject located within the angle of view 301 and obtain 3D information of the subject. The photographing area of the depth camera 300 may be narrower as it approaches the depth camera 300. Specifically, FIG. 2 shows a recognition area (a_1) capable of acquiring 3D information in a virtual plane 410 having a normal vector toward the depth camera 300 according to the distance L1 from the depth camera 300. A limited embodiment is shown. The recognition area a_1 from which the depth camera 300 can acquire 3D information decreases as the distance L1 to the depth camera 300 decreases, and the unrecognized area a_2 in which 3D information cannot be obtained. ) Can be large.

When a single depth camera 300 is included in the mobile terminal 300, a problem of erroneously recognizing the motion of the user's hand may occur because the area for recognizing the user's hand 400 at a near distance is narrow. Hereinafter, the above problem is checked through a depth image acquired through a single depth camera.

3 shows a depth image acquired through a single depth camera in a mobile terminal related to the present invention. Specifically, FIG. 3(a) shows an embodiment in which the user's hand shape 302 is completely included on the depth image b_1 corresponding to the recognition area a_1 of FIG. 2. FIG. 3(b) shows an embodiment in which a part of the shape of a common hand 302 is included on the depth image b_1 corresponding to the recognition area a_1 of FIG. 2.

The depth camera of the present invention acquires the depth image b_1 at preset time intervals, discriminates whether the shape included in the depth image b_1 corresponds to the shape of the user's hand, and the shape of the user hand included in the depth image b_1 It is possible to recognize the user's hand motion through the change of and receive an input signal in response to the user's hand motion.

In the case where the user hand shape 302 is completely included in the depth image b_1 as shown in FIG. 3(a), the mobile terminal of the present invention can correctly recognize the user hand motion and receive an input signal corresponding thereto. to be.

However, in the case where the user hand shape 302 is partially included in the depth image b_1 as shown in FIG. 3(b), the mobile terminal of the present invention incorrectly recognizes the user's hand motion and inputs irrelevant to the user's intention. Can receive signals.

When a part of the shape of the user's hand 302 is included in the depth image b_1 as shown in FIG. 3B, the entire shape of the user's hand may be recognized by guiding the user's hand into the angle of view of the depth camera. However, it may not be easy to guide the user's hand because the area in which the entire shape of the user's hand can be detected is narrow at a short distance. Accordingly, an object of the present invention is to expand an area in which a user's hand can be recognized by including a plurality of depth cameras in a mobile terminal.

4 to 6 are conceptual diagrams for describing a photographing area photographed through a plurality of depth cameras in a mobile terminal related to the present invention.

The mobile terminal 200 of the present invention includes a plurality of depth cameras 310 to 340 each acquiring a depth image, an output unit (not shown) for outputting audio-visual data, and a control unit connecting the plurality of depth cameras 310 to 340 (Not shown) may be included.

The mobile terminal 200 of the present invention may include a plurality of depth cameras 310 to 340 so that the photographing range partially overlaps. 4 to 6 illustrate an embodiment in which a plurality of depth cameras 310 to 340 are provided in the mobile terminal 200 to face the same direction. Optical axes of the plurality of depth cameras 310 to 340 provided in the mobile terminal 200 may be parallel to each other. In some cases, the plurality of depth cameras 310 to 340 may be provided in the mobile terminal 200 so that the optical axes form a mutual inclination while partially overlapping the shooting range.

4 to 5 illustrate an embodiment in which four depth cameras 310 to 340 are provided in the mobile terminal 200. However, the number of depth cameras 310 to 340 included in the mobile terminal 200 of the present invention is not limited to four.

4 to 5 illustrate embodiments in which regions photographed by a plurality of depth cameras 310 to 340 in the mobile terminal 200 correspond to each quadrant or are separated by a grid. However, the mobile terminal 200 of the present invention may be provided with a plurality of depth cameras 310 to 340 so as to be arranged in one direction.

Areas each photographed by the plurality of depth cameras 310 to 340 may be in contact with, spaced apart, or overlap according to a distance to the mobile terminal 200. 4 to 6 show each embodiment.

Specifically, FIG. 4 shows an embodiment in which a photographing area photographed by a plurality of depth cameras 310 to 340 contacts. FIG. 4(a) is a conceptual diagram showing an area photographed by a plurality of depth cameras 310 and 340 from the side, and FIG. 4(b) is a front view of an area photographed by a plurality of depth cameras 310 to 340 It shows a conceptual diagram viewed from the (multiple depth cameras 310 to 340). The plurality of depth cameras 310 to 340 are provided to be spaced apart from each other, and may capture regions within the angles of view 311 to 341, respectively. When the virtual plane 410 is located at the minimum length (L_min) where the photographing regions of the plurality of depth cameras 310 to 340 overlap, the photographing region of the plurality of depth cameras 310 to 340 is the virtual plane 410 They can be contacted without being separated from each other or overlapping. Specifically, FIG. 4(a) shows an embodiment in which the recognition area a1_1 of the first depth camera 310 and the recognition area a4_1 of the fourth depth camera 340 contact on a virtual plane 410 Are doing. In addition, FIG. 4B shows a recognition area a1_1 of the first depth camera 310, a recognition area a2_1 of the second depth camera 320, and a third depth camera 330 on the virtual plane 410. The recognition area a3_1 of) and the recognition area a4_1 of the fourth depth camera 340 correspond to each quadrant and are in contact with each other.

In addition, FIG. 5 illustrates an embodiment in which a plurality of depth cameras 310 to 340 photographing areas are spaced apart. FIG. 5(a) is a conceptual diagram showing an area photographed by a plurality of depth cameras 310 and 340 from the side, and FIG. 5(b) is a front view of an area photographed by the plurality of depth cameras 310 to 340 It shows a conceptual diagram viewed from the (multiple depth cameras 310 to 340). The plurality of depth cameras 310 to 340 are provided to be spaced apart from each other, and may capture regions within the angles of view 311 to 341, respectively. When the virtual plane 410 is located closer than the minimum length (L_min) in which the shooting areas of the plurality of depth cameras 310 to 340 overlap, the shooting areas of the plurality of depth cameras 310 to 340 are separated from the virtual plane Can be. Specifically, FIG. 5(a) shows an unrecognized area between the recognition area a1_1 of the first depth camera 310 and the recognition area a4_1 of the fourth depth camera 340 on the virtual plane 410. a12_2) is shown. In addition, FIG. 5B shows a recognition area a1_1 of the first depth camera 310, a recognition area a2_1 of the second depth camera 320, and a third depth camera 330 on the virtual plane 410. ) Of the recognition area a3_1 and the recognition area a4_1 of the fourth depth camera 340 correspond to each quadrant, and the recognition area a1_1 of the first depth camera 310 and the second depth camera 320 An unrecognized area a12_2 is included between the recognition areas a2_1, and an unrecognized area between the recognition area a2_1 of the second depth camera 320 and the recognition area a3_1 of the third depth camera 330 ( a23_2), including an unrecognized area a34_2 between the recognition area a3_1 of the third depth camera 330 and the recognition area a4_1 of the fourth depth camera 340, and a fourth depth camera ( An embodiment including an unrecognized area a41_2 between the recognition area a4_1 of 340 and the recognition area a1_1 of the first depth camera 310 is illustrated.

In addition, FIG. 6 shows an embodiment in which a photographing area photographed by a plurality of depth cameras 310 to 340 is overlapped. FIG. 6(a) is a conceptual diagram showing an area photographed by a plurality of depth cameras 310 and 340 from the side, and FIG. 6(b) is a front view of an area photographed by the plurality of depth cameras 310 to 340 It shows a conceptual diagram viewed from the (multiple depth cameras 310 to 340). The plurality of depth cameras 310 to 340 are provided to be spaced apart from each other, and may capture regions within the angles of view 311 to 341, respectively. When the virtual plane 410 is located farther than the minimum length (L_min) where the shooting areas of the plurality of depth cameras 310 to 340 overlap, the shooting areas of the plurality of depth cameras 310 to 340 overlap on the virtual plane Can be. Specifically, in FIG. 6(a), the recognition area a1_1 of the first depth camera 310 and the recognition area a4_1 of the fourth depth camera 340 are overlapped areas a12_3 on the virtual plane 410 It shows an embodiment of forming. 6(b) shows a recognition area a1_1 of the first depth camera 310, a recognition area a2_1 of the second depth camera 320, and a third depth camera 330 on the virtual plane 410. ) Of the recognition area a3_1 and the recognition area a4_1 of the fourth depth camera 340 correspond to each quadrant, and the recognition area a1_1 of the first depth camera 310 and the second depth camera 320 The recognition area a2_1 forms an overlap area a12_3, and the recognition area a2_1 of the second depth camera 320 and the recognition area a3_1 of the third depth camera 330 form an overlap area a23_3 And, the recognition area a3_1 of the third depth camera 330 and the recognition area a4_1 of the fourth depth camera 340 form an overlapping area a34_3, and the recognition area of the fourth depth camera 340 ( a4_1) and the recognition area a1_1 of the first depth camera 310 form an overlapping area a41_3.

When the subject of the present invention is located on the virtual plane 410 of FIG. 4, depth images captured through the first to fourth depth cameras 310 to 340 may be merged without editing to obtain 3D information. . However, substantially, the virtual plane 410 of FIG. 4 is formed only in a limited position (a position separated from the mobile terminal 200 by a minimum length (L_min) where the photographing areas of the plurality of depth cameras 310 to 340 overlap) It is meaningful as a boundary.

When the subject of the present invention is located on the virtual plane 410 of FIG. 5 (that is, when the subject is located closer than the minimum length (L_min)), photographing through the first to fourth depth cameras 310 to 340 Even if the obtained depth image is merged, it may not be possible to obtain 3D information of the subject. Specifically, when the subject is located in the unrecognized areas (a12_2, a23_2, a34_2, and a41_2), the mobile terminal 200 of the present invention is limited in obtaining 3D information of the subject through a plurality of cameras 310 to 340. This can be. In this case, the present invention may guide the subject to move within the angle of view of a specific depth camera, or guide the subject to be located farther than the minimum length L_min where the photographing areas of the plurality of depth cameras 310 to 340 overlap.

When the subject of the present invention is located on the virtual plane 410 of FIG. 5 (that is, when the subject is located farther than the minimum length (L_min)), photographing through the first to fourth depth cameras 310 to 340 3D information of the subject may be obtained by merging the obtained depth images. However, when the subject is located in the overlapping areas a12_3, a23_3, a34_3, and a41_3, it is necessary to process (edit) the redundant information. Hereinafter, when the subject is located on the virtual plane 410 of FIG. 5, an embodiment of the depth image captured by the first to fourth depth cameras 310 to 340 will be described.

7 to 9 illustrate depth images acquired through a plurality of depth cameras in a mobile terminal related to the present invention. Specifically, FIGS. 7 to 9 illustrate depth images captured by the first to fourth depth cameras 310 to 340 when the subject is positioned on the virtual plane 410 of FIG. 5.

When the entire shape of the user's hand is included in the photographing area of one of the plurality of depth cameras 310 to 340, the mobile terminal of the present invention analyzes the depth image of the corresponding depth camera 310 to 340 The motion of the user's hand can be detected.

Specifically, FIG. 7 illustrates an embodiment in which the entire user's hand is included in a photographing area of one of the plurality of depth cameras 310 to 340. 7A illustrates an embodiment in which the overall shape 302 of a subject is included in the second depth image b2_1 acquired through the second depth camera. FIG. 7B shows an embodiment in which the overall shape 302 of the subject is included in the first depth image b1_1 acquired through the first depth camera. 7(c) shows an example in which the overall shape 302 of the subject is included in the third depth image b3_1 acquired through the third depth camera. 7(d) shows an embodiment in which the overall shape 302 of the subject is included in the fourth depth image b4_1 acquired through the fourth depth camera. As shown in FIG. 7, when the overall shape 302 of the subject is included in the depth image acquired by a specific depth camera, the depth images acquired from a plurality of depth cameras do not need to be merged, By analyzing the depth image, the motion of the user's hand can be detected.

When the entire shape of the user's hand is not completely included in the specific depth image, but is divided and included in the plurality of depth images acquired by each of the plurality of depth cameras, the entire shape of the user's hand cannot be recognized through the specific depth image. .

Specifically, FIG. 8 shows an embodiment in which the entire subject is divided and included in a photographing area of a plurality of depth cameras 310 to 340. 8(a) shows that the overall shape 302 of the subject is divided and included in a second depth image b2_1 acquired through a second depth camera and a third depth image b3_1 acquired through a third depth camera. An example is shown. 8(b) shows that the overall shape 302 of the subject is divided and included in the first depth image b1_1 acquired through the first depth camera and the fourth depth image b4_1 acquired through the fourth depth camera. An example is shown. 8(c) shows that the overall shape 302 of the subject is divided and included in the third depth image b3_1 acquired through the third depth camera and the fourth depth image b4_1 acquired through the fourth depth camera. An example is shown. 8(d) shows that the overall shape 302 of the subject is divided and included in the second depth image b2_1 acquired through the second depth camera and the first depth image b1_1 acquired through the first depth camera. An example is shown.

In the case of FIG. 8A, the second depth image b2_1 and the third depth image b3_1 may be merged to identify the overall shape of the user's hand, and the user's hand motion may be sensed. In the case of FIG. 8B, the first depth image b1_1 and the fourth depth image b4_1 may be merged to identify the entire shape of the user's hand, and the user's hand motion may be sensed. In the case of FIG. 8C, the third depth image b3_1 and the fourth depth image b4_1 may be merged to identify the entire shape of the user's hand, and the user's hand motion may be sensed. In the case of FIG. 8D, the second depth image b2_1 and the first depth image b1_1 may be merged to discriminate the entire shape of the user's hand and detect the user's hand motion.

In the present invention, the entire shape of the user's hand may be obtained by merging the depth image including the entire shape of the user's hand from among the depth images obtained by each of the plurality of depth cameras. That is, among the depth images each acquired by the plurality of depth cameras, a depth image that does not include a part of the user's hand does not need to be merged otherwise. That is, it is not necessary to merge the first depth image b1_1 and the fourth depth image b4_1 in FIG. 8A. In addition, in FIG. 8B, the second depth image b2_1 and the third depth image b3_1 do not need to be merged. In addition, in FIG. 8C, the second depth image b2_1 and the first depth image b1_1 need not be merged. In addition, in FIG. 8D, the third depth image b3_1 and the fourth depth image b4_1 do not need to be merged.

In some cases, the overall shape of the user's hand may be obtained by merging three or more depth images. Specifically, FIG. 9 shows an embodiment in which the shape of a user's hand is divided and included in four depth images. In this case, the present invention combines all of the first depth image (b1_1), the second depth image (b2_1), the third depth image (b3_1), and the fourth depth image (b4_1) to discriminate the entire shape of the user's hand, and Hand motion can be detected.

That is, the mobile terminal of the present invention obtains a depth image by photographing each at a constant angle of view, a plurality of depth cameras in which the photographing area partially overlaps, an output unit for outputting audio-visual data, and the user's hand using the acquired depth image. And a control unit for controlling the output unit in response to the motion of the user hand recognized and recognized, wherein the control unit includes a part of the shape of the user hand in the first depth image acquired through the first depth camera among the plurality of depth cameras. If included, a second depth image acquired through a second depth camera that photographs an area adjacent to an area photographed by the first depth camera among the plurality of depth cameras is merged with the first depth image to form the entire user's hand. Shape can be acquired.

The mobile terminal of the present invention may discriminate the second depth camera from among the plurality of depth cameras through a position in which a part of the shape of the user's hand is included in the first depth image. For example, when a part of the shape of the user's hand is located in the lower area in the first depth image, the present invention may identify a camera that photographs adjacent to the lower area in the shooting area of the first depth camera by the second depth camera. I can. The present invention may obtain the second depth image to be merged into the first depth image through the second depth camera. That is, in the present invention, when the user's hand is divided and included in the depth image, it is possible to discriminate that a part of the user's hand is included through any one depth image, and the depth image including the remaining part may be discriminated.

In the plurality of depth cameras of the present invention, a partial shape of the user's hand may be included in the first depth image, and the entire shape of the user's hand may be included in the second depth image because the photographing area partially overlaps. In this case, it is not necessary to acquire the entire shape of the user's hand by merging the second depth image with the first depth image. In this case, the present invention may acquire the entire shape of the user's hand through the second depth image, and recognize the motion of the user's hand through the second depth image continuously photographed. That is, in the present invention, the first depth image includes a partial shape of the user's hand, the second depth image includes the remaining shape of the user's hand, and the second depth image does not include the entire shape of the user's hand. In this case, the entire shape of the user's hand may be obtained by merging the first depth image and the second depth image.

10 is a flowchart illustrating an entire processor that senses a motion of a user's hand through a plurality of depth cameras and provides a corresponding function.

The mobile terminal of the present invention may each acquire a depth image through a plurality of depth cameras. (S210) Here, the plurality of depth cameras may acquire a depth image for each successive frame, and the photographing area may partially overlap.

The present invention can discriminate whether a shape of a user's hand is recognized from a depth image acquired from a plurality of depth cameras. (S220) In the present invention, it is possible to discriminate whether or not the shape of the user's hand is included in the depth image by using the feature of the shape of the user's hand. It is possible to discriminate whether a subject photographed through the depth camera corresponds to the user's hand by using a partial feature of the user's hand of the present invention.

In the present invention, when the shape of the user's hand is not recognized in the depth image obtained from each of the plurality of depth cameras (S220, No), the depth image obtained through the plurality of depth cameras is not merged and continues to be acquired. Can be done with The present invention can discriminate whether the shape of the user's hand is included in the continuously acquired depth image.

In the present invention, when at least one part of the shape of the user's hand is recognized from at least one of the depth images each obtained from a plurality of depth cameras (S220, Yes), the entire shape of the user's hand is included in the depth image acquired from a single depth camera. I can discern whether it works. (S230)

In the present invention, when the entire user's hand is located within one depth camera photographing area, and the entire shape of the user's hand can be discriminated from the depth image acquired through the corresponding depth camera, (S230, Yes) the user's hand motion through the corresponding depth camera. Can detect and provide a corresponding function. In this case, the present invention may omit merging of depth images acquired through different depth cameras.

However, in the present invention, when the user's hand is positioned over the overlapping area of the plurality of depth cameras, and the entire shape of the user's hand cannot be discriminated from the depth image acquired through a single depth camera, (S230, No) a plurality of depths The depth image acquired from the camera can be merged to recognize the entire user's hand. (S240) The present invention can recognize the entire shape of the user's hand through a plurality of depth cameras, and provide a function corresponding to the motion of the user's hand. (S250)

11 is a diagram for describing a method of merging depth images acquired through a plurality of depth cameras.

Each of the plurality of depth cameras 310 and 320 acquires a depth image corresponding to the photographing region at a constant angle of view, but the photographing regions may overlap from a specific distance. Here, the specific distance may correspond to a minimum distance L_min at which regions recognizable through a plurality of depth cameras start to overlap.

In the present invention, as shown in FIG. 5, an unrecognized area may occur between areas photographed by a plurality of depth cameras below the specific distance. According to the present invention, when the user's hand is located in the unrecognized area, the entire shape of the user's hand may not be obtained even if depth images respectively acquired through a plurality of depth cameras are merged. In this case, the present invention can guide the position of the user's hand through the output unit of the present invention, without merging depth images acquired from a plurality of depth cameras. For example, according to the present invention, the user's hand may be guided to be positioned within the angle of view of a specific camera, or may be guided to be positioned at a specific distance or more.

According to the present invention, when the user's hand is positioned more than the specific distance as described in FIG. 6, the entire shape of the user's hand may be obtained by merging depth images acquired through a plurality of depth cameras. Specifically, in the present invention, when it is determined that the user's hand is located on the overlapping shooting area of a plurality of depth cameras, the shooting areas of the plurality of depth cameras are divided so that they do not overlap, and the depths respectively acquired according to the divided areas By merging the images, the overall shape of the user's hand can be obtained.

Specifically, a method of dividing photographing regions of the plurality of depth cameras 310 and 320 so as not to overlap with each other will be described with reference to FIG. 11.

The first depth camera 310 and the second depth camera 320 of the present invention define recognition regions a1_1 and a2_1 corresponding to the angle of view on a virtual plane 410 having a normal vector toward the mobile terminal 200. Can have. The recognition area a1_1 of the first depth camera 310 and the recognition area a2_1 of the second depth camera 320 may include an overlapping area a12_3 corresponding to a distance from the mobile terminal 200. Specifically, the recognition area a1_1 of the first depth camera 310 and the recognition area a2_1 of the second depth camera 320 at a distance greater than or equal to the minimum distance L_min may include an overlapping area a12_3. . When the user's hand is located in the overlapping area a12_3, it may not be difficult to properly acquire the entire shape of the user's hand by merging the first depth image and the second depth image. This is because a part of the user's hand included in the first depth image may be repeated in the second depth image. Accordingly, according to the present invention, a depth camera that recognizes each of the overlapping regions a12_3 can be divided, and a part of the user's hand can be prevented from being repeated in the merged image.

Specifically, the overlapping area a12_3 of the present invention is a center plane a12_4 in the optical axis of each depth camera compared to the minimum distance L_min at which the recognition areas of the first depth camera 310 and the second depth camera 320 start to overlap. It can be divided using the ratio of the minimum distance to ). Here, the center plane (a12_4) is a plane including a start point when the photographing areas of the first depth camera 310 and the second depth camera 320 overlap, and the first depth camera 310 and the second depth camera It may be a plane perpendicular to a straight line connecting 320.

For example, the present invention sets a divided area (a2_1') excluding an area to be recognized through the first depth camera 310 among the overlapping areas a12_3 in the recognition area a2_1 of the second depth camera 320 Then, a second edited image b2_1 ′ corresponding to the divided area a2_1 ′ may be obtained. Specifically, the present invention is from the optical axis of the second depth camera 320 to the center plane (a12_4) compared to the minimum distance (L_min) at which the recognition regions of the first depth camera 310 and the second depth camera 320 start to overlap. The ratio (rate 2) of the minimum distance (H2_min) can be calculated. The present invention uses the distance (x_r) from the minimum point where the recognition regions of the first depth camera 310 and the second depth camera 320 starts to overlap to the virtual plane 410 and the calculated ratio (rate 2). Thus, an area y2_r to be recognized through the first depth camera 310 in the overlapping area a12_3 may be calculated. The present invention may set the divided area a2_1 ′ excluding the area y2_r to be recognized through the first depth camera 310 in the recognition area a2_1 of the second depth camera 320. In the same manner as described above, the present invention provides a divided area (a1_1') excluding an area to be recognized by the second depth camera 320 among the overlapping areas (a12_3) in the recognition area (a1_1) of the first depth camera 310. After setting, a first edited image b1_1 ′ corresponding to the divided region a1_1 ′ may be obtained.

12 illustrates an embodiment of acquiring an overall shape of a user's hand by merging depth images captured through a plurality of depth cameras in a mobile terminal related to the present invention.

Specifically, FIG. 12A illustrates depth images b1_1, b2_1, b3_1, and b4_1 respectively acquired through first to fourth depth cameras. Here, the overall shape 302 of the user's hand is shown in an embodiment in which the second depth image b2_1 and the third depth image b3_1 are divided and included in the depth images.

FIG. 12(b) illustrates an embodiment in which the entire shape of the user's hand is obtained by merging the second depth image b2_1 and the third depth image b3_1 of FIG. 12(a). Specifically, FIG. 12(b) shows the second depth in the second edited image b2_1' and the third depth image b3_1 excluding a portion to be implemented through the third depth image b3_1 in the second depth image b2_1. An exemplary embodiment in which the merged image 303 is obtained by merging the third edited image b3_1' excluding a portion to be implemented through the image b2_1 is illustrated.

The merged image 303 may correspond to a depth image captured in a continuous area without an unrecognized area or an overlapping area. In the present invention, when the shape of the user's hand is divided and included in a plurality of depth images, the merged image 303 is acquired for each shooting frame, and the motion of the user's hand is recognized through the acquired merged image 303, and a corresponding function Can provide.

13 and 14 are diagrams for explaining another application example of expanding a photographing area through a plurality of depth cameras.

The present invention may include a plurality of depth cameras in devices other than a mobile terminal, and expand a photographing area for obtaining a depth image.

Specifically, FIG. 13 illustrates an embodiment including a plurality of depth cameras 310 to 340 in the display device 200. The display device 200 of the present invention may acquire a depth image corresponding to a wider area by providing a plurality of depth cameras 310 to 340 at each corner of the display device 200. The plurality of depth cameras 310 to 340 may obtain corresponding depth images by dividing the photographing area. In this case, the divided photographing areas a1_1 ′ to a4_1 ′ are continuous without an unrecognized area, and may not overlap.

Specifically, FIG. 14 shows an embodiment including a plurality of depth cameras 310 to 340 in a vehicle. The plurality of depth cameras 310 to 340 may be provided at a corner of a front windshield of a vehicle to divide and photograph a region where a driver's hand is located. The plurality of depth cameras 310 to 340 may be provided such that an optical axis direction for acquiring a depth image faces an area where a driver's hand is located. The plurality of depth cameras 310 to 340 may obtain corresponding depth images by dividing the photographing area. In this case, the divided photographing areas a1_1 ′ to a4_1 ′ are continuous without an unrecognized area, and may not overlap.

The above detailed description should not be construed as restrictive in all respects and should be considered as illustrative. The scope of the present invention should be determined by reasonable interpretation of the appended claims, and all changes within the equivalent scope of the present invention are included in the scope of the present invention.

Claims (10)

1. A plurality of depth cameras each capturing at a constant angle of view to obtain a depth image, and partially overlapping the capturing area;

   An output unit for outputting audio-visual data;

   Including; a control unit for recognizing the motion of the user's hand using the acquired depth image and controlling the output unit in response to the recognized user hand motion; and

   The control unit

   When a part of the shape of the user's hand is included in the first depth image acquired through the first depth camera among the plurality of depth cameras,

   Acquiring the overall shape of the user's hand by merging a second depth image acquired through a second depth camera that photographs an area adjacent to an area photographed by the first depth camera among the plurality of depth cameras into the first depth image Mobile terminal characterized by.
2. The method of claim 1,

   The control unit

   The mobile terminal, characterized in that the second depth camera is identified among the depth cameras of the abdomen through a position in the first depth image including a part of the shape of the user's hand.
3. The method of claim 2,

   The control unit

   And when the second depth camera cannot be identified among the plurality of depth cameras, the output unit is controlled to provide an alarm for guiding the user's hand into the field of view of the first depth camera.
4. The method of claim 2,

   The second depth image acquired by the second depth camera includes the rest of the shape of the user's hand, but does not include the entire shape of the user's hand.
5. The method of claim 4,

   The control unit

   When the entire shape of the user's hand is included in the second depth image, merging of the depth image is omitted, and the entire shape of the user's hand is obtained through the second depth image.
6. The method of claim 1,

   The plurality of depth cameras

   Over a certain distance, the shooting area overlaps,

   The control unit

   When the user's hand is positioned more than the specific distance, the first depth image and the second depth image are merged to obtain an overall shape of the user's hand.
7. The method of claim 6,

   The control unit

   And when the user's hand is located within the specific distance, the output unit is controlled to provide an alarm guiding the position of the user's hand.
8. The method of claim 7,

   The control unit

   When it is recognized that the user's hand is located within the specific distance through the first depth image, the alarm is provided so that the user's hand is positioned within a field of view of the first depth camera.
9. The method of claim 7,

   The control unit

   When it is recognized that the user's hand is located within the specific distance through the first depth image, the alarm is provided so that the user's hand is positioned at a distance greater than or equal to the specific distance.
10. The method of claim 1,

    The plurality of depth cameras

    Each successive frame acquires a depth image,

    The control unit

    Acquiring the entire shape of the user's hand by using the first depth image and the second depth image for each successive frame,

    A mobile terminal, characterized in that the motion of the user's hand is recognized through a change in the overall shape of the user's hand acquired for each successive frame.

Images