KR20160142207A - Electronic device and Method for controlling the electronic device - Google Patents

Abstract

An electronic device and a control method thereof are provided. The control method of the electronic device comprises the steps of: obtaining a depth image using a depth camera; extracting a hand area including a hand of a user from the obtained depth image; modeling a finger and a palm of the user included in the hand area to a plurality of points; and detecting a touch input based on depth information of the modeled points.

Description

BACKGROUND OF THE INVENTION 1. Field of the Invention [0001] The present invention relates to an electronic device and a control method thereof,

BACKGROUND OF THE INVENTION 1. Field of the Invention [0001] The present invention relates to an electronic device and a control method thereof, and more particularly to an electronic device for sensing a touch input of a user using depth information of a user's hand obtained by a depth camera and a control method thereof.

Various researches and products are being developed to make a large screen composed of a beam projector as an interactive touch screen. Among them, a method of detecting a user's touch by adopting a depth camera in a beam projector is being developed. Specifically, the beam projector senses the user's touch input using the difference between the depth image and the flat depth image acquired by the depth camera.

In this case, when the palm is placed on the flat surface, a touch caused by the palm causes the inconvenience of holding the palm of the hand when the touch is inputted. In addition, when noise occurs due to environmental factors such as light generated in the surroundings, it is difficult to distinguish it from a touch due to a finger, which may cause a noise touch.

SUMMARY OF THE INVENTION The present invention has been conceived to solve the above-mentioned problems, and it is an object of the present invention to provide an electronic device for modeling a user's hand obtained by a depth camera into a plurality of points and sensing a touch input of a user based on depth information of a plurality of points And a control method thereof.

According to an aspect of the present invention, there is provided a method of controlling an electronic device, including: acquiring a depth image using a depth camera; Extracting a hand region including a user's hand from the obtained depth image; Modeling a finger and a palm of a user included in the hand region into a plurality of points; And sensing the touch input based on the depth information of the plurality of points modeled.

The modeling may be performed by modeling each of the fingers of the user into a plurality of points, modeling each of the fingers of the user as one point, and the palm of the user is one As shown in FIG.

The sensing step may include sensing a touch input at the touched point when it is detected that only an end point of at least one of the plurality of points of the detection and stop is touched, When a plurality of points of at least one finger among the points are detected as being touched, the touch input may not be detected.

The sensing step senses the multi-touch input to the touched point when it is detected that only the end points of the two fingers of the thumb and index finger are touched, If one point of the thumb is detected as being all touched, the touch input may not be detected.

Also, in the sensing step, when it is detected that only the end points of two fingers of the plurality of points of the forensic hands of the user are touched, the multi-touch input may be sensed at the touched point.

In addition, when only the end points of all of the plurality of points of all the fingers of both hands of the user are touched, the multi-touch input can be detected.

The method may further include analyzing a movement direction and a velocity of a hand included in the hand region, wherein the extracting step extracts the hand of the user based on the direction and velocity of the hand analyzed in the previous frame .

Determining whether an object in the depth image obtained by analyzing the obtained depth image is a human hand or an object; And determining the type of the extracted object if it is determined that the object in the depth image is an object.

The method may further include performing the function of the electronic device based on the type of the determined object and the touch position of the object.

According to another aspect of the present invention, there is provided an electronic device including: a depth camera for acquiring a depth image; Extracting a hand region including a user's hand from the obtained depth image, modeling a finger and a palm of a user included in the hand region into a plurality of points, and based on the depth information of the plurality of modeled points And a control unit for sensing a touch input.

The control unit models each of the fingers of the user such as the detection, stop, and ring finger as a plurality of points, and models each of the fingers of the user as one point, and the palm of the user is one point . ≪ / RTI >

The control unit senses the touch input at the touched point when it is detected that only the end point of at least one finger among the plurality of points of the detection and stop is touched, When a plurality of points of at least one finger are sensed as being touched, the touch input may not be sensed.

The control unit senses the multi-touch input to the touched point when it is detected that only the end points of the two fingers of the thumb and index finger are touched, If it is detected that one point is all touched, the touch input may not be detected.

Also, the controller may detect the multi-touch input at the touched point when it is detected that only the end points of the two fingers of the plurality of points of the hands of the user's hands are touched.

In addition, when only the end points of all of the plurality of points of all the fingers of both hands of the user are touched, the multi-touch input can be detected.

In addition, the controller may analyze the direction and velocity of the hand included in the hand region, and extract the hand of the user based on the direction and velocity of the hand analyzed in the previous frame

The controller determines whether an object in the depth image obtained by analyzing the obtained depth image is a human hand or an object, and when it is determined that an object in the depth image is an object, You can judge the kind of thing.

In addition, the controller may perform the function of the electronic device based on the type of the determined object and the touch position of the object.

And an image projecting unit for projecting an image onto the touch area.

According to various embodiments of the present invention as described above, user convenience of a touch input using a depth camera can be improved. In addition, the electronic device can provide various user inputs using a depth camera.

BRIEF DESCRIPTION OF THE DRAWINGS FIG. 1 is a block diagram schematically showing the configuration of an electronic device according to an embodiment of the present invention;
2 is a block diagram showing details of the configuration of an electronic device according to an embodiment of the present invention;
FIGS. 3A to 4 illustrate an embodiment in which a hand region is extracted from a depth image obtained using a depth camera, and a finger and a palm of the extracted hand region are modeled into a plurality of points according to an embodiment of the present invention The drawings,
FIGS. 5A and 8B are diagrams for explaining an embodiment for determining a touch input based on depth information of a plurality of points, according to various embodiments of the present invention,
Figure 9 illustrates a touch region according to one embodiment of the present invention;
10A to 11C are diagrams for explaining an embodiment for controlling an electronic device using an object according to an embodiment of the present invention;
12 and 13 are flowcharts for explaining a control method of an electronic device according to an embodiment of the present invention;
14 is a diagram for explaining an embodiment in which an electronic device is controlled through an external user terminal according to another embodiment of the present invention,
15A and 15B are views showing a stand-type electronic apparatus according to an embodiment of the present invention.

The terms used in the embodiments of the present invention will be briefly described, and these embodiments will be described in detail.

Although the terms used in the embodiments of the present invention have been selected in consideration of the functions of the present invention, the present invention is not limited thereto and can be varied depending on the intention or the precedent of the artisan skilled in the art, . Also, in certain cases, some terms are arbitrarily selected by the applicant, and in this case, the meaning thereof will be described in detail in the description of the corresponding embodiments. Therefore, the terms used in the embodiments should be defined based on the meaning of the terms, not on the names of simple terms, and on the contents of the embodiments throughout.

In an embodiment of the present invention, terms including ordinal numbers such as first, second, etc. may be used to describe various elements, but the elements are not limited to these terms. The terms are used only for the purpose of distinguishing one component from another. For example, without departing from the scope of the present invention, the first component may be referred to as a second component, and similarly, the second component may also be referred to as a first component. And / or < / RTI > includes any combination of a plurality of related listed items or any of a plurality of related listed items.

Further, in the embodiments of the present invention, the singular expressions include plural expressions unless the context clearly indicates otherwise.

Furthermore, in the embodiments of the present invention, terms such as "comprises" or "having ", etc. are intended to specify the presence of stated features, integers, steps, operations, elements, parts, or combinations thereof, Steps, operations, elements, components, or combinations of elements, numbers, steps, operations, components, parts, or combinations thereof.

Also, in the embodiments of the present invention, 'module' or 'sub' performs at least one function or operation, and may be implemented in hardware or software, or a combination of hardware and software. In addition, a plurality of 'modules' or a plurality of 'parts' may be integrated into at least one module except for 'module' or 'module' which needs to be implemented by specific hardware, and may be implemented by at least one processor.

Further, in the embodiment of the present invention, when a part is referred to as being "connected" with another part, it is not limited to a case where it is " directly connected " And the like.

Further, in an embodiment of the present invention, " touch input " may include a touch gesture that the user makes on the display and the cover to control the device. In addition, the " touch input " may include a touch (e.g., floating or hovering) that is not touched on the display and is spaced apart by a certain distance.

Further, in the embodiment of the present invention, " application " refers to a set of computer programs designed to perform a specific task. In an embodiment of the invention, the application may be varied. But are not limited to, a gaming application, a video playback application, a map application, a memo application, a calendar application, a phonebook application, a broadcast application, a motion support application, a payment application, and a photo folder application.

Hereinafter, the present invention will be described in more detail with reference to the drawings. 1 is a block diagram briefly showing a configuration of an electronic device 100 according to an embodiment of the present invention. As shown in FIG. 1, the electronic device 100 includes a depth camera 110 and a control unit 120.

The depth camera 110 acquires a depth image for a specific area. Specifically, the depth camera 110 can capture a depth image of a touch region where an image is projected.

The control unit 120 controls the overall operation of the electronic device 100. In particular, the control unit 120 extracts a hand region including a user's hand from the depth image obtained through the depth camera 110, models a finger and a palm of the user included in the hand region as a plurality of points, The touch input can be detected based on the depth information of the plurality of points.

Specifically, the controller 120 may analyze the depth image obtained through the depth camera 110 to determine whether an object in the depth image is a hand or an object in the depth image. More specifically, the control unit 120 can determine the shape of the object in the depth image by measuring the difference between the plane depth image and the captured depth image for a display area having no object.

If it is determined that the hand shape of the user exists in the depth image, the control unit 120 can detect the hand area in the depth image. At this time, the control unit 120 can detect the hand region including the user's hand by removing the noise in the depth image.

The controller 120 may model the user's palm and fingers included in the extracted hand region as a plurality of points. Specifically, the control unit 120 models each of the fingers of the user, such as the finger, the stop, and the finger, into a plurality of points, and models each of the fingers of the user as one point and the palm of the user as one point Can be modeled.

The controller 120 may sense the touch input of the user based on the depth information of the plurality of points modeled. Specifically, when it is detected that only one end point of one of the plurality of points of detection and stop is sensed as being touched, the control section 120 senses the touch input at the touched point and detects at least one of the plurality of points of detection and stop The controller 120 may not detect the touch input when a plurality of points of the finger of the user are detected as being touched.

In addition, when it is detected that only the end points of the two fingers of the user's thumb and index finger are touched, the control unit 120 detects the multi-touch input using the thumb and index finger, The controller 120 may not detect the touch input using the thumb and index finger.

In addition, when it is detected that only the end points of two of the plurality of points of the detectors of both hands of the user are touched, the controller 120 can detect the multi-touch input using the detection of both hands, The control unit 120 can sense the multi-touch input using both hands.

In addition, the controller 120 analyzes the direction and velocity of the hand included in the hand region so as to determine the touch action of the user more quickly. Based on the direction and velocity of the hand analyzed in the previous frame, The region can be extracted.

However, if it is determined that an object in the depth image is an object, the control unit 120 can determine the type of the extracted object. That is, the control unit 120 can determine the type of object placed in the touch area by comparing the object placed in the touch area and the shape of the previously registered object. The control unit 120 may perform the function of the electronic device 100 based on at least one of the type of the determined object and the touch position of the object.

By using the electronic device 100 as described above, the user can more efficiently perform the touch input using the depth camera.

Hereinafter, the present invention will be described in more detail with reference to FIGS. 2 to 11C.

2 is a block diagram showing a detailed configuration of an electronic device 200 according to an embodiment of the present invention. 2, the electronic device 200 includes a depth camera 210, an image input unit 220, a display unit 230, a storage unit 240, a communication unit 250, and a control unit 260 .

FIG. 2 is a diagram illustrating various components in an exemplary case where the electronic device 200 is a device having various functions such as a content providing function, a display function, and the like. Therefore, depending on the embodiment, some of the components shown in Fig. 2 may be omitted or changed, and other components may be further added.

The depth camera 210 acquires a depth image for a specific area. In particular, when the electronic device 200 displays an image using a beam projector, the depth camera 210 may acquire a depth image of a display area in which an image is displayed by projecting light from the beam projector.

The image input unit 220 receives image data through various sources. For example, the video input unit 220 can receive broadcast data from an external broadcasting station, input VOD data in real time from an external server, and receive video data from an external device.

The display unit 230 may display image data input through the image input unit 220. At this time. The display unit 230 can output image data in a beam projector manner. In particular, the display unit 230 can project light using the DLP method, but this is merely an embodiment, and light can be projected using another method.

In addition, the display unit 230 may be implemented as a general display rather than a beam projector. For example, the display unit 230 may have various forms such as a liquid crystal display (LCD), an organic light emitting diode (OLED) display, an active matrix organic light-emitting diode (AM-OLED), and a plasma display panel . ≪ / RTI > The display unit 230 may further include an additional configuration depending on the implementation method. For example, when the display unit 230 is a liquid crystal type, the display unit 230 may include an LCD display panel (not shown), a backlight unit (not shown) for supplying light thereto, and a panel (Not shown) for driving the display panel.

The storage unit 240 may store various programs and data necessary for the operation of the electronic device 200. Non-volatile memory, volatile memory, flash-memory, hard disk drive (HDD), or solid state drive (SSD). The storage unit 240 is accessed by the control unit 260, and the control unit 260 can read / write / modify / delete / update the data. The storage unit 240 may be a memory card (not shown) (for example, a micro SD card or a memory stick) mounted on the ROM 262, the RAM 261 or the electronic device 200 in the controller 260, . ≪ / RTI > In addition, the storage unit 240 may store programs and data for configuring various screens to be displayed in the display area.

In addition, the storage unit 240 may store and store the calculated result based on the type of object and the depth information of the object.

The communication unit 250 is configured to perform communication with various types of external devices according to various types of communication methods. The communication unit 250 includes a Wi-Fi chip, a Bluetooth chip, a wireless communication chip, an NFC chip, and the like. The control unit 260 communicates with various external devices using the communication unit 250.

In particular, Wi-Fi chips and Bluetooth chips communicate via WiFi and Bluetooth, respectively. When a Wi-Fi chip or a Bluetooth chip is used, various connection information such as an SSID and a session key may be transmitted and received first, and communication information may be used to transmit and receive various information. The wireless communication chip refers to a chip that performs communication according to various communication standards such as IEEE, ZigBee, 3G (3rd Generation), 3rd Generation Partnership Project (3GPP), LTE (Long Term Evolution) The NFC chip refers to a chip operating in an NFC (Near Field Communication) system using 13.56 MHz band among various RF-ID frequency bands such as 135 kHz, 13.56 MHz, 433 MHz, 860 to 960 MHz, and 2.45 GHz.

The control unit 260 controls the overall operation of the electronic device 200 using various programs stored in the storage unit 240.

2, the control unit 260 includes a RAM 261, a ROM 262, a graphics processing unit 263, a main CPU 264, first through n interfaces 265-1 through 265-n, Bus 266. < / RTI > At this time, the RAM 261, the ROM 262, the graphics processing unit 263, the main CPU 264, the first to n interfaces 265-1 to 265-n, etc. may be connected to each other via the bus 266 .

The ROM 262 stores a command set for booting the system and the like. When the turn-on command is input and power is supplied, the main CPU 264 copies the O / S stored in the storage unit 240 to the RAM 261 according to the instruction stored in the ROM 262, To boot the system. When the booting is completed, the main CPU 264 copies various application programs stored in the storage unit 240 to the RAM 261, executes the application programs copied to the RAM 261, and performs various operations.

The graphic processing unit 263 generates a screen including various information such as an item, an image, text, and the like using an operation unit (not shown) and a rendering unit (not shown). The operation unit computes an attribute value such as a coordinate value, a shape, a size, and a color to be displayed according to the layout of the screen by using the control command input to the user. The rendering unit generates a screen composed of various layouts including information based on the attribute values calculated by the operation unit. The screen generated by the rendering unit is displayed in the display area of the display unit 230.

The main CPU 264 accesses the storage unit 240 and performs booting using the O / S stored in the storage unit 240. [ The main CPU 264 performs various operations using various programs stored in the storage unit 240, contents, data, and the like.

The first to n interfaces 265-1 to 265-n are connected to the various components described above. One of the interfaces may be a network interface connected to an external device via a network.

In particular, the control unit 260 extracts a hand region including the user's hand from the depth image obtained through the depth camera 210, models the finger and the palm of the user included in the hand region as a plurality of points, And detects the touch input based on the depth information of the plurality of points.

Specifically, the control unit 260 acquires a depth image of a display area on which the image is projected by the display unit 230. First, the control unit 260 acquires a plane depth image on which no object is placed on the display area. The control unit 260 acquires a depth image of a display area on which a specific object (e.g., user's hand, object, etc.) is placed. The control unit 260 may measure the difference between the photographed depth image and the flat depth image to obtain a depth image as shown in FIG. 3A.

The controller 260 removes noise from the depth image as shown in FIG. 3A based on the convex hull, and generates a hand region 310 including a human hand, as shown in FIG. 3B, Can be extracted.

The controller 260 may model the palm and fingers of the user as a plurality of points based on the depth information and shape of the hand area 310, as shown in FIG. 3C. In one embodiment of the present invention, the controller 260 models the palm as the first point 410-1, the thumb as the second point 410-2, as shown in FIG. 4, 410-4, 410-4, and model the stop to the fifth and sixth points 410-5, 410-6, and the ring finger to the seventh and eighth points 410-7, 410- 8), and the substrate may be modeled as the ninth point 410-9. That is, the user's palm and finger model simplifies the natural shape of the hand when the user types on the desk plane. For example, there is no distinction between the thumb and the underexposed, and the index, the stop, and the ring finger can each represent one node.

The controller 260 can sense the touch input of the user based on the depth information of the plurality of points modeled. This will be described in more detail with reference to Figs. 5A to 8B. For reference, in Figs. 5A and 8B, the reference plane is expressed as? When it is outside the touch recognition distance, and may be expressed as? When the reference plane is within the touch recognition distance.

First, when it is detected that only one of the finger end points 410-4 and 410-5 is touched, the control unit 260 can detect the large touch input at the touched point. Specifically, as shown in FIG. 5A, only the stop end point 410-5 is within the touch recognition distance, or the stop end point 410-5 and the palm point 410- 1) is within the touch recognition distance, the control unit 260 can sense the touch input to the point where the stop end point 410-5 touches.

However, when a plurality of points of the finger of one of the detection and stop is detected as being touched, the control unit 260 may not detect the touch input. Specifically, as shown in FIG. 5C, if the plurality of points of the stop 410-5, 410-6 are all within the touch recognition distance, or the plurality of points of stop 410-5, 410-6 And the palm point 410-1 are both within the touch recognition distance, the control unit 260 may not detect the touch input. In other words, when the user touches the end portion of the pause, only the end portion of the pause is not touched. Therefore, when the plurality of pause points 410-5 and 410-6 of the pause are detected as being all touched, It is determined that the touch input is unintentional and the touch input may not be sensed.

5A to 5D have been described using a pause, but this is merely an embodiment, and when the detection is used, it can operate in the same manner as in Figs. 5A to 5D.

In addition, when multi-touch is performed using the thumb and index finger, if it is detected that only the end points of two fingers out of the plurality of points 410-2 to 410-4 of the thumb and index finger are touched, The multi-touch input can be sensed at the point where it is detected. Specifically, as shown in FIG. 6A, only the end point of the detection 410-4 and the end point of the thumb 410-2 are within the touch recognition distance, or the end point of the detection 410-4, thumb end point 410-2 and palm point 410-1 are within the touch recognition distance, the control unit 260 can detect the multi-touch input using the stop and thumb. That is, the control unit 260 may provide various functions (e.g., zooming in, zooming out of the image, etc.) according to the stop and change of the distance between the thumbs.

However, if a plurality of points 410-3 and 410-4 of the detection and one point 410-2 of the thumb are detected as being touched, the control unit 260 may not detect the touch input. Specifically, as shown in FIG. 6C, the plurality of points of detection 410-3 and 410-4 and the end point of the thumb 410-2 are within the touch recognition distance, or, as shown in FIG. 6D, When the plurality of points 410-3 and 410-4, the thumb end point 410-2 and the palm point 410-1 are both within the touch recognition distance, the control unit 260 may not detect the multi- have.

6A to 6D have been described using the detection and the thumb. However, the multi-touch input using the stop and the thumb can also operate in the same manner as in FIGS. 6A to 6D.

When it is detected that only the end points of the two fingers of the plural points of the fingers of both hands of the user are touched, when the user wants to input multi-touch using the fingers of both hands of the user, Multi-touch input can be detected. More specifically, as shown in Fig. 7A, only the end point 710-4 of the left hand detection and the end point 720-4 of the right hand detection are within the touch recognition distance, or, as shown in Fig. 7B, When the end point 710-4, the left hand palm point 710-1, the right hand palm end point 720-4 and the right hand palm point 720-1 are within the touch recognition distance, Touch input using the stop of the multi-touch input. That is, the control unit 260 may provide various functions (e.g., zooming in, zooming out, etc.) according to the distance change between the stops of both hands.

7B, it is determined that all of the palm spots 710-1 and 720-1 of both hands are within the touch recognition distance. However, only one of the palm spots 710-1 and 720-1 of both hands touch Multi-touch input using both hands can be detected even within the recognition range.

7A and 7B illustrate the detection of both hands. However, the present invention is not limited to this embodiment, and it is also possible to use the stopping of both hands in the same manner as in FIGS. 7A and 7B.

If it is detected that only the end points of all the fingers of all the fingers of all the fingers of the user's hands are touched, the controller 260 determines whether the multi- Touch input can be detected. Specifically, as shown in Fig. 8A, the end points of all the fingers of the left hand (710-2, 710-4, 710-5, 710-7, 710-9) and the right hand of all fingers (720-2,720-4,720-5,720-7,720- 9) is within the touch recognition distance, or as shown in Fig. 8B, all of the left hand finger end points 710-2, 710-4, 710-5, 710-7, 710-9, the left hand palm point 710-1, When the end points 720-2, 720-4, 720-5, 720-7, 720-9 of the finger and the right hand palm point 720-1 are within the touch recognition distance, the control unit 260 can detect the multi-touch input using both hands . That is, the control unit 260 may provide various functions (e.g., zooming in, zooming out, etc.) according to the distance between the hands.

5A to 8B, the electronic device 200 can sense the touch input by touching the finger irrespective of whether the user's palm touches the floor or not, The touch input can not be detected.

In addition, according to an embodiment of the present invention, the controller 260 can analyze the direction and velocity of the hand movement to detect the touch input of the user more quickly. The controller 260 may determine the location of the user's hand area in the next frame based on the hand direction and speed of the hand analyzed in the previous frame, and extract the position of the determined hand area. At this time, the control unit 260 may extract the hand region by cropping the hand region in the depth image.

Meanwhile, in the above-described embodiment, the user's hand is detected in the display area. However, this is an example only, and objects other than human hands can be extracted.

Specifically, the controller 260 may analyze the depth image obtained through the depth camera 210 to determine whether the object in the depth image is a human hand or an object. Specifically, the controller 260 can determine the type of object located within the display area using the difference between the plane depth image and the depth image photographed through the depth camera 210. [ At this time, the controller 260 extracts a color area of an object in the depth image, and determines whether the object is a human hand or an object using an image of the separated object according to the extracted outer area. 9, when there is a difference in the depth image in the determination region 910 located at the edge of the image, the control unit 260 determines that the human hand is located, and the determination region 910 , The controller 260 can determine that the object is located.

If it is determined that the object in the depth image is an object, the control unit 260 can determine the type of the extracted object. Specifically, the controller 260 may calculate the depth of the object, the depth interval, the depth average, and the depth deviation of the object based on the depth information of the object, multiply the calculated value by the weight, . The control unit 260 can determine the type of objects in the depth image by comparing the stored result values with the result values matched with the type of object.

The control unit 260 can control the function of the electronic device 100 according to the determined type of object. For example, as shown in FIG. 10A, when it is determined that the type of object placed on the display area during the first screen is displayed as the cup 1010, the control unit 260 displays an instruction For example, executing a moving image application). That is, the controller 260 may control the display unit 230 to display the second screen (moving image application execution screen) as shown in FIG. 10B. As another example, if it is determined that the type of object placed on the display area during the display of the first screen is a notebook, the controller 260 may execute an instruction (for example, executing a memo application) matching the notebook .

 In addition, the function of the electronic device 200 may be performed according to the type of object regardless of the position of the object, but this is merely an example, and the control unit 260 may provide different functions depending on the position of the object . That is, when the object 1010 is within the display area, as shown in Fig. 11A, when the object 1010 is on the boundary between the display area and the outline as shown in Fig. 11B, As described above, when the object 1010 is located outside the display area, all of the functions can be provided. For example, as shown in FIG. 11A, when the object 1010 is within the display area, the control unit 260 can execute the moving image application, and the object 1010 is displayed on the display area 1010, The control unit 260 can execute a music application and if the object 1010 is located outside the display area as shown in FIG. 11C, the control unit 260 controls the electronic device 100) to the standby mode. It goes without saying that the object 1010 may provide different functions depending on the position in the display area.

In addition, when the object 1010 is located outside the display area, the control unit 260 may control the display unit 230 to display a short axis icon near the location of the object 1010 in the display area.

Hereinafter, a control method of the electronic device 100 will be described with reference to FIGS. 12 and 13. FIG. 12 is a flowchart for explaining a control method of the electronic device 100 according to an embodiment of the present invention.

First, the electronic device 100 acquires a depth image using a depth camera (S1210). Specifically, the electronic device 100 may acquire a depth image within the display area.

Then, the electronic device 100 extracts a hand region including a user's hand from the photographed depth image (S1220). At this time, the electronic device 100 can extract noise from the depth image and extract the hand region of the user.

Then, the electronic device 100 models the finger and the palm of the user included in the hand region as a plurality of points (S1230). Specifically, the electronic device 100 models each of the fingers of the user such as the index finger, the stop finger, and the finger finger as a plurality of points, and models each of the fingers of the user as one point and the palm of the user as one point . ≪ / RTI >

Then, the electronic device 100 senses the touch input based on the depth information of the plurality of points modeled (S1240). Specifically, the electronic device 100 can sense the touch input, as in various embodiments, such as in FIGS. 5A-B.

13 is a flowchart for explaining a control method of the electronic device 100, according to another embodiment of the present invention.

First, the electronic device 100 acquires a depth image using a depth camera (S1310).

Then, the electronic device 100 analyzes the obtained depth image (S1315). Specifically, the electronic device 100 can analyze the depth image using the difference between the plane depth image and the photographed depth image.

Then, the electronic device 100 determines whether the object in the obtained depth image is a human hand or a human hand (S1320).

If it is determined that the object is a human hand, the electronic device 100 removes noise from the depth image and extracts the hand region (S1325).

Then, the electronic device 100 models the finger and palm of the user included in the hand area as a plurality of points (S1330), detects the touch input based on the depth information of the plurality of points modeled (S1335) And controls the electronic device 100 according to the touch input (S1340).

However, if it is determined that the object is an object, the electronic device 100 analyzes the depth information of the object (S1345), determines the type of the object based on the analysis result (S1350) And controls the electronic device 100 according to at least one (S1355).

According to various embodiments of the present invention as described above, user convenience of a touch input using a depth camera can be improved. In addition, the electronic device 100 can provide various types of user input using a depth camera.

Meanwhile, in the above-described embodiment, the electronic device 100 directly displays an image, detects a touch input, and performs a function according to a touch input. However, the present invention is not limited to this, Function can be performed through the external portable terminal 1400. [ 14, the electronic device 100 simply outputs an image using a beam project and acquires a depth image using a depth camera, and the external portable terminal 1400 can acquire a depth image using an electronic device And the functions of the portable terminal 1400 and the electronic device 100 can be controlled by providing the image to the portable terminal 100 and analyzing the depth image. That is, the external portable terminal 1400 can perform the function of the control unit 120 described above.

In addition, the electronic device 100, according to one embodiment of the present invention, may be implemented as a stand-up beam projector. Specifically, Fig. 15A is a front view of the stand-type beam projector, and Fig. 15B is a side view of the stand-type beam projector. 15A and 15B, the stand-type beam projector includes a beam projector 1510 and a depth camera 1520 at an upper end thereof, a foldable frame 1530 (Foldable Frame) and a docking base 1540 (Docking Base) can support the beam projector 1510 and the depth camera 1520. The electronic device 100 can project light to the display area using the upper beam projector 1510 and sense the touch input to the display area using the depth camera 1520. [ Then, the user can adjust the display area by adjusting the folder blur frame 1530. Also, the external portable terminal 1400 can be mounted on the docking base 1540. [

Meanwhile, the above-described method can be implemented in a general-purpose digital computer that can be created as a program that can be executed by a computer and operates the program using a computer-readable recording medium. In addition, the structure of the data used in the above-described method can be recorded on a computer-readable recording medium through various means. The computer-readable recording medium includes a storage medium such as a magnetic storage medium (e.g., ROM, floppy disk, hard disk, etc.), optical reading medium (e.g., CD ROM,

It will be understood by those skilled in the art that various changes in form and details may be made therein without departing from the spirit and scope of the invention as defined by the appended claims. Therefore, the disclosed methods should be considered from an illustrative point of view, not from a restrictive point of view. The scope of the present invention is defined by the appended claims rather than by the foregoing description, and all differences within the scope of equivalents thereof should be construed as being included in the present invention.

110, 210: Depth camera 120, 260:
220: image input unit 230: display unit
240: storage unit 250: communication unit

Claims (19)

A method of controlling an electronic device,
Acquiring a depth image using a depth camera;
Extracting a hand region including a user's hand from the obtained depth image;
Modeling a finger and a palm of a user included in the hand region into a plurality of points; And
And sensing a touch input based on the depth information of the plurality of points modeled. The method according to claim 1,
Wherein the modeling comprises:
Wherein each of the fingers of the user is modeled as a plurality of points, and each of the fingers of the user is modeled as one point, and the palm of the user is modeled as one point. Lt; / RTI > 3. The method of claim 2,
Wherein the sensing comprises:
Detecting a touch input to the touched point if it is detected that only the end point of at least one of the plurality of points of the detection and stop is touched,
And does not detect the touch input when a plurality of points of at least one of the plurality of points of the detection and stop are sensed as being touched. 3. The method of claim 2,
Wherein the sensing comprises:
Touch input is detected at the touch point when it is detected that only the end points of the two fingers of the thumb and forefinger are touched,
Wherein the touch input is not detected when a plurality of points of the detection and one point of the thumb are detected as being touched. 3. The method of claim 2,
Wherein the sensing comprises:
Wherein the multi-touch input is detected at the touched point when it is detected that only the end points of the two fingers of the plurality of points of the forensic hands of the user are touched. 3. The method of claim 2,
Wherein the multi-touch input is detected when only the end points of all of the plurality of points of all the fingers of both hands of the user are touched. The method according to claim 1,
And analyzing a moving direction and speed of a hand included in the hand region,
Wherein the extracting comprises:
Wherein the hand of the user is extracted based on the direction and velocity of the hand analyzed in the previous frame. The method according to claim 1,
Determining whether an object in the depth image obtained by analyzing the obtained depth image is a human hand or an object;
And determining the type of the extracted object if it is determined that the object in the depth image is an object. 9. The method of claim 8,
And performing the function of the electronic device on the basis of the kind of the determined object and the touch position of the object. In an electronic device,
A depth camera for acquiring a depth image; And
Extracting a hand region including a user's hand from the obtained depth image, modeling a finger and a palm of a user included in the hand region as a plurality of points, and performing a touch operation based on the depth information of the plurality of modeled points And a control unit for detecting an input. 11. The method of claim 10,
Wherein,
Wherein each of the fingers of the user is modeled as a plurality of points, and each of the fingers of the user is modeled as one point, and the palm of the user is modeled as one point. Lt; / RTI > 12. The method of claim 11,
Wherein,
Detecting a touch input to the touched point if it is detected that only the end point of at least one of the plurality of points of the detection and stop is touched,
And does not detect the touch input when a plurality of points of at least one of the plurality of points of the detection and stop are sensed as being touched. 12. The method of claim 11,
Wherein,
Touch input is detected at the touch point when it is detected that only the end points of the two fingers of the thumb and forefinger are touched,
And does not detect the touch input when a plurality of points of the detection and one point of the thumb are detected as being touched. 12. The method of claim 11,
Wherein,
Wherein the multi-touch input detecting unit detects the multi-touch input at the touched point when it is detected that only the end points of two of the plurality of points of the forensic hands of the user's hands are touched. 12. The method of claim 11,
Touch input when only the end points of all the fingers of the plurality of points of all the fingers of the user's both hands are touched. 11. The method of claim 10,
Wherein,
Analyzing a movement direction and a velocity of a hand included in the hand region and extracting the hand of the user based on a movement direction and velocity of the hand analyzed in the previous frame. 11. The method of claim 10,
Wherein,
And determines whether the object in the depth image obtained by analyzing the obtained depth image is a human hand or an object. If it is determined that the object in the depth image is an object, the type of the extracted object is determined . 18. The method of claim 17,
Wherein,
And performs the function of the electronic device based on the type of the determined object and the touch position of the object. 11. The method of claim 10,
And an image projecting part for projecting an image to the touch area.

Images