KR101662022B1 - Apparatus and Method for Device Control using Gesture - Google Patents

Abstract

The present invention relates to an apparatus and a method for controlling a device using a gesture, capable of controlling the device with reliability by accurately recognizing a gesture of a user under various lighting environments. To this end, according to the present invention, a method for controlling a device using a gesture includes an indication object recognition step in which an indication object recognition unit receives an image obtained by photographing a space where a control target object of a user is present, detects at least one object from the image, detects a body of the user from the image to recognize an indication direction of the user according to the posture of the detected body of the user, and specifies an indication object indicated by the user among the objects according to an indication direction of the user to recognize a specified indication object; and a gesture recognition step in which a gesture recognition unit analyzes a pose of a part of the body of the user in the image and recognizes a gesture to control the indication object.

Description

[0001] Apparatus and Method for Device Control Using Gesture [0002]

The present invention relates to an apparatus and a method for controlling a device according to a gesture of a person.

Conventionally, in order to control various devices such as a TV and an air conditioner in a home or an interior, a method using a remote controller by a user has been widely used. In the conventional device control method, it is inconvenient to operate a plurality of remote controllers corresponding to respective devices in order to control various kinds of devices existing in an indoor space, or to use a single remote controller There is an inconvenience in that the user must find the integrated remote control and operate it by holding it.

Accordingly, various methods of recognizing the gesture of the user's body and controlling the device according to the gesture have been developed. For this purpose, existing inventions have proposed various methods of acquiring information according to a user's body motion using various sensors and analyzing acquired operation information to recognize a user's predetermined promised gesture.

However, the existing inventions of the related art are not able to provide a means for integrally controlling a plurality of devices existing in a living space because a target device to be controlled is determined in advance according to a user's gesture, There is a limit in that it can not propose a method of efficiently distinguishing and specifying and controlling the target objects when they exist. Also, the existing inventions have a problem that the recognition probability of the gesture varies depending on the lighting conditions of the room.

Korean Patent Publication No. 10-2010-0131213 (December 15, 2010)

The object of the present invention is to provide a device and a method for selecting a target device to be controlled through a gesture instructed by a user and executing a predetermined control command through a gesture with respect to the selected device, The present invention provides a device control apparatus and a method therefor that allow a user to precisely select a target device to be controlled among a plurality of devices with which the user can control and accurately recognize the gesture of a user in various lighting environments to reliably control the device will be.

According to an aspect of the present invention, there is provided a method of controlling a device using a gesture according to an aspect of the present invention, the method comprising: receiving an image of a space in which a user's controlled object exists, Detecting a body of the user on the image, recognizing a direction of a user according to a body posture of the detected user, detecting an instruction direction of the user among the objects according to the direction of the user, An instruction object recognition step of recognizing the specific instruction object by specifying an object; And a gesture recognition step of recognizing a gesture for controlling the pointing object by analyzing a pose of a part of the user's body in the image.

The device control method using the gesture may further include an object control command output step in which the object control command output unit generates and outputs a control command for the instruction object according to the recognized gesture.

Here, the image may be a depth image having three-dimensional depth information.

Here, the instruction object recognition step may include detecting at least one or more body parts predetermined in the body of the detected user, generating an instruction line according to a line direction indicated by the user using the detected body parts, Direction, and specifies the instruction object using the instruction line.

Wherein the step of recognizing the instruction object comprises the steps of: detecting the body part in the user's body, generating the instruction line using the body part, and recognizing the instruction direction of the user; An object detecting step of detecting the object in the image; And an instruction object specifying step of specifying, as the pointing object, the object among the detected objects existing on the pointing line or within a predetermined distance from the pointing line.

Here, the instruction object recognition step may include an object recognition step of recognizing the type of the detected object or the specific instruction object using an already-learned object classifier.

Here, the directing direction recognizing step may detect the body part including at least one of the elbow part, the hand part, the shoulder part, and the eye part in the body of the user, and may use the position of the body part, And the like.

Wherein if the position of at least two joints of the user's elbow joint, hand joint or shoulder joint among the detected body parts satisfies a predetermined rule, the instruction direction recognizing step recognizes the position of the eye part and the hand And a line connecting the positions of the points is generated by the instruction line.

Wherein the object detecting step detects an object by detecting an outline of the object in the image, and the step of specifying the instruction object compares an outline of the detected object with a locus of the instruction line, And the like.

Wherein the step of specifying an instruction object includes comparing a locus of the instruction line according to the movement of the instruction line generated in the instruction direction recognition step and an outline of the object detected in the object detection step and comparing the locus of the instruction line and the outline The object specifying unit may specify the object having the outline corresponding to the locus of the instruction line as the instruction object.

Wherein if the plurality of objects exist on the instruction line or exist within a predetermined distance from the instruction line, the outline of each of the plurality of objects is compared with the locus of the instruction line, And selects the object having the outline corresponding to the locus of the instruction line among the plurality of objects, and specifies the selected object as the instruction object.

Wherein the step of specifying an instruction object includes the step of projecting the locus of the instruction line onto the space of the image and performing a transformation including a magnitude transformation or a rotation transformation on the trajectory of the projected instruction line or the outline of the object, Comparing the result of the comparison with a trajectory of the instruction line and an outline of the object according to whether the projected trajectory of the instruction line and the outline of the object coincide within a predetermined error.

Wherein the image is a depth image having three-dimensional depth information, and wherein the step of specifying the pointing object is a step of selecting a movement path of a contact point between the projection plane set at a predetermined position in the three- And comparing the obtained trajectory of the instruction line with the outline of the object to determine whether the trajectory of the instruction line and the outline of the object correspond to each other.

Wherein the object detecting step obtains an outline of the object in accordance with the outline information of a predetermined three-dimensional model corresponding to the object detected in the image, and the step of specifying the instruction object comprises the step of: And specifies the instruction object according to the comparison result.

Wherein the gesture recognition step includes: an illumination determination step of analyzing the image to acquire illumination state information according to the illumination environment in which the image is photographed; A gesture region extracting step of extracting a body part region for performing the gesture from the body of the user existing in the image according to the acquired illumination state information; And a control gesture recognition step of recognizing the gesture for controlling the pointing object by analyzing a pose of a body part region performing the extracted gesture.

Here, the illumination determination step may include inputting the image to the previously learned illumination classifier to determine which of the plurality of illumination states is the predetermined illumination environment, and acquiring the illumination state information according to the determination result Extracting an area having a predetermined color or brightness range according to the illumination state information in the image as a body part area for performing the gesture, and the control gesture recognition step extracting the control gesture Extracting a predetermined feature from a body part region, inputting a feature vector according to the extracted feature to a learned gesture classifier, and recognizing the gesture.

In order to solve the above-described problems, another type of the present invention can be a computer program stored in a medium for executing the device control method using the gesture.

According to still another aspect of the present invention, there is provided a device control apparatus using a gesture according to another aspect of the present invention. The device control method includes receiving an image of a space in which a user's controlled object exists, A body of the user is detected from the image, and a direction instruction of the user according to the body posture of the detected user is recognized, and an instruction object indicated by the user among the objects is specified An instruction object recognition unit for recognizing the specific instruction object; A gesture recognition unit for recognizing a gesture for controlling the pointing object by analyzing a pose of a part of the body of the user in the image; And an object control command output unit for generating and outputting a control command for the instruction object according to the recognized gesture.

Here, the instruction object recognition unit may detect at least one or more body parts predetermined in the body of the detected user, generate an instruction line according to a line direction instructed by the user using the detected body parts, An instruction direction recognizing unit for recognizing a direction; An object detection unit for detecting and recognizing the object in the image; And specifying the object, which is present on the instruction line or within a predetermined distance from the instruction line, as the instruction object, wherein the locus of the instruction line and the outline of the object detected by the object detection unit And an instruction object specifying unit for specifying the object having the outline corresponding to the locus of the instruction line as the instruction object when it is determined that the locus of the instruction line and the outline of the object correspond to each other.

Wherein the gesture recognition unit analyzes the image and acquires illumination state information according to the illumination environment in which the image is photographed; A gesture region extracting unit for extracting a body part region that performs the gesture from the body of the user existing in the image according to the obtained illumination state information; And a control gesture recognition unit for recognizing the gesture for controlling the instruction object by analyzing a pose of a body part region performing the extracted gesture.

According to the method and the method for controlling a device using a gesture according to the present invention, it is possible to select a target device to be controlled by a natural arm operation by a user, and to select, from among a plurality of devices, In addition, it is possible to accurately recognize a hand gesture of a user in various lighting environments, thereby enabling reliable device control.

1 is a flowchart of a device control method using a gesture according to an embodiment of the present invention.
2 is a flowchart of a device control method using a gesture according to another embodiment of the present invention.
3 is a detailed flowchart of an instruction object recognition step according to an embodiment.
4 to 6 are detailed flowcharts of the instruction object recognition step according to still another embodiment.
7 and 8 are reference diagrams for explaining the operation of the instruction object recognition step.
9 is a detailed flowchart of the gesture recognition step.
10 is a block diagram of a device control apparatus using a gesture according to another embodiment of the present invention.
11 is a detailed block diagram of the instruction object recognition unit.
12 is a detailed block diagram of the gesture recognition unit.

Hereinafter, preferred embodiments of the present invention will be described in detail with reference to the accompanying drawings. In the drawings, the same reference numerals are used to designate the same or similar components throughout the drawings. In the following description of the present invention, a detailed description of known functions and configurations incorporated herein will be omitted when it may make the subject matter of the present invention rather unclear. In addition, the preferred embodiments of the present invention will be described below, but it is needless to say that the technical idea of the present invention is not limited thereto and can be variously modified by those skilled in the art.

Conventionally, in the method of controlling various devices including a TV and an air conditioner by using a remote controller, it has been inconvenient to operate a plurality of remote controllers corresponding to respective devices in order to control various kinds of devices existing in an indoor space , Or even if a plurality of devices are operated by one integrated remote controller, the user has to find the integrated remote control and operate it with the inconvenience. Accordingly, devices have been developed that provide an intuitive device control interface that recognizes the gesture of the user's body and controls the device according to the gesture.

However, the device control apparatus using the existing gesture and the related methods have a limitation in that the target device to be controlled is predetermined according to the gesture of the user, and thus it is not possible to provide a means for integrally controlling a plurality of devices existing in the living space there was. A device control apparatus and methods using a conventional gesture can not provide a method of efficiently distinguishing and identifying and controlling a plurality of objects existing in a space in which a user exists. Also, existing apparatuses and methods have a problem that the recognition probability of the gesture varies depending on the lighting conditions of the room.

Accordingly, the present invention solves the problems of device control apparatuses and methods using existing gestures and selects a target device to be controlled by a user from a plurality of objects existing in an indoor space through a user's instruction gesture And outputting a predetermined control command through a user's input gesture to the selected device, and a method therefor. The present invention also proposes a configuration for accurately recognizing a user's gesture in various lighting environments and for reliably controlling a device.

1 is a flowchart of a device control method using a gesture according to an embodiment of the present invention.

The device control method using the gesture may include an instruction object recognition step (S100) and a gesture recognition step (S200).

Here, the device control method using the gesture may further include an object control command output step (S300). 2 is a flowchart of a device control method using a gesture according to another embodiment of the present invention.

Meanwhile, a device control apparatus using a gesture according to another embodiment of the present invention may include an instruction object recognition unit 100 and a gesture recognition unit 200, and may further include an object control command output unit 300 . 10 is a block diagram of a device controlling apparatus using a gesture according to another embodiment of the present invention.

Here, the instruction object recognition unit 100 and the gesture recognition unit 200 may operate according to the device control method using the gesture according to the present invention, as described below. The same is true of the case where the object control command output unit 300 is further included.

Here, all the components of the device control apparatus using the gesture according to the present invention can be implemented as one independent hardware. Or a device control device using a gesture according to the present invention may be embodied as a computer program having a program module in which some or all of the components are selectively combined to perform a part or all of functions combined in one or a plurality of hardware It is possible. Further, the device control apparatus using the gesture according to the present invention may be implemented as a software program and operated on a processor or a signal processing module, or may be implemented in a form of hardware to be included in a chip, an element, or the like. Further, the device control apparatus using the gesture according to the present invention may be included in a hardware or software module. For example, the device control apparatus using a gesture according to the present invention may be a home network server or a control system, a device for controlling various devices existing in an indoor space, a device for controlling each device, . And may be implemented in hardware or software in these devices, or may operate in hardware or software operating in conjunction with these devices.

Hereinafter, a device control method using a gesture according to the present invention will be described in detail.

In the instruction object recognition step S100, the instruction object recognition unit 100 receives an image of a space in which the user's control object exists, and detects at least one or more objects in the image. Also, the instruction-object recognition unit 100 detects the body of the user on the image and recognizes the direction of the user according to the detected body posture of the user. Also, the instruction object recognition unit 100 identifies the instruction object indicated by the user among the objects according to the instruction direction of the user, and recognizes the specific instruction object.

In the gesture recognition step S200, the gesture recognition unit 200 analyzes a pose of a part of the user's body in the image and recognizes a gesture for controlling the pointing object.

In the object control command output step S300, the object control command output unit 300 generates and outputs a control command for the instruction object according to the recognized gesture.

Hereinafter, the instruction object recognition step S100 will be described in more detail.

In the instruction object recognition step S100, the instruction object recognition unit 100 receives an image of a space in which the user's control object exists, and detects at least one or more objects in the image.

Here, the image received in the indication object recognition step S100 may be an image of a space in which object devices to be controlled by the user exist. And the user may be included in the image.

Here, the image may be a depth image having three-dimensional depth information. For example, the image may be an image obtained using an RGB-D camera. Alternatively, the image may be a stereo image acquired using a stereo camera, or may be a depth image obtained by analyzing the stereo image and calculating a depth value. Alternatively, the image may be a two-dimensional image as needed. In the present invention, the pointing object recognition step S100 receives a depth image having three-dimensional depth information, analyzes the object, detects an object as described in detail below, recognizes the pointing direction according to the pointing line of the user, It is desirable to specify an instruction object to be an instruction target in accordance with the instruction direction. However, if necessary, the instruction object recognition step S100 according to the present invention may receive a two-dimensional image, detect an object on a two-dimensional image plane, generate an instruction line, specify an instruction object according to the instruction line have.

The objects may be various devices and devices that can be electronically controlled or operated. For example, considering the indoor space of a home, various devices such as a TV, a projector such as a video projector such as a set-top box, a sound device such as an audio player or a speaker, a computer device, a kitchen appliance such as a gas range or a microwave, . Here, the object includes all the devices and devices that are present in various spaces such as the indoor space of the other company as well as the home space, and can operate electronically according to the user's operation.

Here, the pointing object recognition step S100 can detect an object in the image using the existing various object recognition methods, and can recognize the detected object. For example, the pointing object recognition step S100 may detect TV and settop box objects existing in the image, and recognize that each detected object corresponds to a TV and a set-top box.

Here, the pointing object recognition step S100 can detect and recognize the object using the predetermined feature information by analyzing the image. Here, in the instruction object recognition step S100, various existing types of features may be extracted from the image to detect and recognize the object, and object detection and recognition may be performed using the same. For example, the pointing object recognition step S100 may extract and use various features such as edge characteristics, corner characteristics, LoG (Laplacian of Gaussian), and DoG (Difference of Gaussian). Here, the pointing object recognition step S100 may include object recognition using various existing feature description methods including Scale-invariant feature transform (SIFT), Speeded Up Robust Features (SULF), and Histogram of Oriented Gradients And recognition techniques. Alternatively, the instruction object recognition step S100 may detect and recognize the object by comparing the template image with a certain area within the target image. Also, the instruction object recognition step S100 may detect and recognize the objects in the image using a classifier previously learned based on the features or templates extracted as described above. The classifier used here may be an SVM machine or a classifier using various other conventional classification functions. The anomaly indication recognition step S100 detects or recognizes a target object by using various known object detection and recognition techniques in addition to the method exemplified above along with the concrete name, And the operation of the instruction object recognition step S100 is not limited to the above-exemplified techniques.

Further, in the instruction object recognition step S100, the instruction object recognition unit 100 detects the user's body in the image, recognizes the direction of the user according to the detected body posture of the user, And recognizes the specific instruction object by specifying the instruction object indicated by the user among the objects according to the direction. Herein, the direction of the user is a predetermined direction in which the user instructs the instruction object to be controlled by the user using a part of the body. For example, the direction indicated by the user with the finger may be the direction of the instruction.

The instruction object recognizing step S100 detects at least one or more body parts predetermined in the body of the detected user and generates an instruction line according to a line direction instructed by the user using the detected body parts, The instruction direction can be recognized, and the instruction object can be specified using the instruction line. The instruction object specifying operation using the instruction line will be described in detail with reference to a detailed flowchart of the instruction object recognition step S100.

FIG. 3 is a detailed flowchart of an instruction object recognition step S100 according to an embodiment.

Here, the instruction object recognition step S100 may include an instruction direction recognition step S110, an object detection step S120, and an instruction object specification step S130.

The instruction direction recognition step S110 detects the body part in the user's body and generates the instruction line using the body part to recognize the instruction direction of the user.

Herein, the directing direction recognizing step (S110) may detect the body part in the same manner as described above in detail for the object detecting step S100 (step S100). That is, in this case, what is detected in the image is a specific body part of the user, and a conventional image detecting method can be used to detect a specific body part.

Here, the directing direction recognizing step (S110) may detect the body part including at least one of the elbow part, the hand part, the shoulder part, and the eye part in the body of the user, and, using the position of the body part, You can create lines. For example, the instruction direction recognizing step (S110) may detect a user's elbow joint, wrist joint, and shoulder joint, and may detect a user's eye area and a user's fingertip area. Here, the eye region of the user may be set to a position spaced apart from the central point of the head by a predetermined distance after detecting the head region of the user. For example, the eye region of the user can be set at a position (for example, a position separated by 5 cm) from the center of the user's head by a predetermined distance in the hand direction. Further, the user's fingertip can be set at a position (for example, a position separated by 15 cm) away from the wrist joint by a predetermined distance in a direction opposite to the elbow joint.

In this case, when the position of at least two joints of the user's elbow joint, hand joint or shoulder joint among the detected parts of the body satisfies predetermined rules, And a line connecting the position of the hand part with the indication line. Here, the hand joints can be various joints included in the wrist joints, palms and fingers.

In this case, if the position of at least two of the elbow joints or the hand joint or the shoulder joint of the user satisfies a preset rule among the detected body parts, the instruction direction recognizing step (S110) It can be determined that the user is an activating operation. Here, the predetermined rule may be a rule that is set according to the positional relationship of the elbow joint, the hand joint, or the shoulder joint, and preferably, the degree to which the elbow joint, the hand joint, and the shoulder joint are aligned on a straight line It can be a rule to judge.

For example, in the instruction direction recognition step (S110), a case where the user's shoulder joint, elbow joint, and wrist joint are in a straight line may be determined as an activation operation of the user for recognizing the instruction direction. Here, whether the user's shoulder joint, the elbow joint and the wrist joint are in a straight line can be determined by measuring the angular difference between the line connecting the shoulder joint and the elbow joint and the line connecting the elbow joint and the wrist joint, It can be judged according to the difference in the direction of the line or the degree of the position where the upper line is spaced apart from the space mountain.

If it is determined that the user's pose of the arm indicates the activation operation of the user for recognizing the direction, the instruction direction recognizing step (S110) recognizes the direction of the user, Can be generated by the instruction line. The user can extend his / her hand around the eyes and direct the target object to be controlled by the user. In this case, the pointing object is generally positioned on the instruction line connecting the user's eye and the hand. Therefore, in the instruction direction recognition step S110, it is preferable that a line connecting the position of the user's eye part and the position of the hand part is generated as a pointing line. Here, the eye region used for generating the instruction line may be set based on the head after the user's head is detected as described above, and the position of the hand region may be determined by detecting a user's hand and selecting a certain point included in the hand . Here, it is preferable to set the position of the hand part to the position of the fingertip part.

Next, the object detection step (S120) detects the object in the image. Here, the object detection step (S120) may detect the outline information of the object.

Here, the object detection step (S120) may generate a gradient image in each channel of the image, and may calculate the gradient images generated in each channel to detect the outline of the object.

과 같이 산출될 수 있다. 여기서 f _x 는 기준 픽셀의 좌우 값의 차, f _y 는 기준 픽셀의 상하 값의 차이다. 이와 같이 R, G, B 채널에 대하여 각각 구한 그래디언트 크기를 다시 OR 연산하여 영상에 포함된 객체의 외곽선을 구할 수 있다. 그리고 이를 이진화 시켜 최종적으로 객체의 외곽선을 구할 수 있다.For example, the gradient size can be obtained from images separated by R, G, and B channels, respectively. In this case, the gradient size is calculated for one pixel located at (x, y)

Can be calculated as follows. Where f _x is the difference between the left and right values of the reference pixel, and f _y is the difference between the upper and lower values of the reference pixel. In this manner, the outline of the object included in the image can be obtained by ORing the gradient size obtained for each of the R, G, and B channels. Finally, the outline of the object can be obtained by binarizing it.

Furthermore, the object detection step (S120) may recognize information including the type of object or identification information. At this time, the object detection step (S120) can detect and recognize the object in the image according to the method described in detail above. However, when the instruction object recognition step S100 further includes the object recognition step S140, the operation of recognizing the information including the type of the object or the identification information is performed in the object recognition step S140 It is possible.

Herein, either one of the instruction direction recognition step (S110) and the object detection step (S120) may be selectively performed first, or may be performed in parallel if necessary.

4 is a detailed flowchart of an instruction object recognition step S100 according to another embodiment in which the instruction direction recognition step S110 and the object detection step S120 are performed in parallel.

The indicating object specifying step (S130) specifies, as the pointing object, the object existing on the pointing line or within a predetermined distance from the pointing line among the detected objects. Here, the operation of the indicating object specifying step S130 will be described in more detail below.

Here, the pointing object recognition step S100 may further include an object recognition step S140 as described above.

When the object recognition step S 140 further includes the object recognition step S 100, only the basic information such as the position and the outline of the object is detected in the object detection step S 120, It is possible to recognize concrete information including the type of the object. Here, information to be acquired in the object detection step (S120) and the object recognition step (S140) can be set as needed.

The object recognizing step S140 recognizes the type of the detected object or the specific instruction object by using a pre-learned object classifier. Here, the object recognizing step (S140) can recognize the object in the image as described in detail above. Here, the detection and recognition of the object may be performed in a single process as needed, or the recognition of the detected object may be performed after the object is detected.

In the case of including the object recognition step S140, the instruction object recognition step S100 may operate as shown in the detailed flowcharts of FIGS. 5 and 6, corresponding to the cases of FIGS.

FIG. 5 is a detailed flowchart of an instruction object recognition step S100 according to another embodiment.

In the case of FIG. 5, the operation of the object recognizing step S140 may be performed on the object specified by the instruction object in the instruction object specifying step S130. In this case, there is an effect that the calculation amount can be reduced by performing only the object specified by the instruction object, without performing the object recognition on all the objects existing in the image.

FIG. 6 is a detailed flowchart of the instruction object recognition step S100 according to this another embodiment.

In the case of FIG. 6, object recognition may be performed on the object detected in the object detection step S120. In this case, the object recognition step S140 is first performed on the detected objects, the instruction object is specified in the instruction object specification step S130, and the object recognition information corresponding to the specified instruction object is acquired There is an effect that can be done.

Next, the operation of the indicating object specifying step S130 will be described in more detail.

The indicating object specifying step (S130) may identify the object existing on the instruction line generated in the direction recognition step (S110) of the object detected in the image or existing within a predetermined distance from the instruction line as the indication object can do. That is, when a specific object exists on the instruction line, the corresponding object can be specified as a pointing object, and even if the object does not exist on the pointing line, the pointing object can be specified among objects existing within a predetermined distance from the pointing line . For example, an object existing at a distance closest to the instruction line may be specified as a pointing object, or an object existing within a predetermined distance may be specified as a pointing object.

However, there may be a case where there are a plurality of objects existing within a predetermined distance from the object existing on the instruction line or the instruction line.

FIG. 7 is a reference diagram for explaining the operation of the indication object recognition step S100 when a plurality of objects exist on the instruction line.

Referring to FIG. 7, it can be seen that there are two objects, object 1 and object 3, on the instruction line. Also, if the predetermined distance is set to be larger than the distance from the instruction line to the object 2, the candidate objects that can be indicated by the instruction line may be three objects including up to the object 2.

In this case, in order to specify an accurate pointing object indicated by the user among the candidate objects that can be pointing objects, the pointing object specifying step S130 calculates the distance between the center point of each object and the pointing line, The object having the smallest distance can be identified by the pointer object.

Or the instruction object specification step S130 compares the outline of the detected object with the locus of the instruction line to specify an accurate instruction object indicated by the user among the candidate objects that can be the instruction object, The instruction object may be specified. In other words, in an actual daily living space, objects are not always present, and when two or more objects overlap one another, it is difficult to know which object among two or more objects is selected by simply using a fingertip. In order to solve this problem, an outline of an object to be pointed is drawn by a fingertip, rather than a simple hand point instruction, so that the user can know which object among the overlapping objects is selected as the pointing object by using the locus of the instruction line generated accordingly.

Here, the locus of the instruction line may be obtained as a locus corresponding to the movement of the instruction line generated in the instruction direction recognition step (S110). The trajectory along the movement of the instruction line may be a movement path of a contact between the predetermined surface and the instruction line. Here, the predetermined plane set to generate the locus of the instruction line may be set as a virtual plane located on the space in which the image is captured. For example, the predetermined face may be a virtual face set at a position where an object exists, or may be a virtual face set at a position of a wall, floor, or ceiling existing in the space. And the predetermined surface is set to be perpendicular to the direction of the line of sight of the user. To this end, the predetermined plane may be set perpendicular to a line connecting the center of the outline of the user's head and the object.

The indicating object specifying step S130 compares the outline of the indication line and the outline of the object detected in the object detecting step S120 as described above. When determining that the outline of the object corresponds to the outline of the indication line, The object having the outline corresponding to the locus of the instruction line can be specified as the instruction object.

That is, if the plurality of objects exist on the instruction line or exist within a predetermined distance from the instruction line, the outline of the plurality of objects is compared with the locus of the instruction line, The object having the outline corresponding to the locus of the instruction line among the plurality of objects can be selected and specified as the instruction object.

FIG. 8 is a reference diagram for explaining the operation of an instruction object specifying step (S130) for specifying an instruction object by comparing the outline of the instruction line with the outline of the object.

Referring to FIG. 8, in step S130, the user compares the outline of a plurality of objects with the locus of an instruction line that moves together as the user moves his or her hand. Then, the object 1 having an outline corresponding to the locus of the instruction line, It can be specified as an object.

In this case, when comparing the locus of the instruction line with the outline of the object, it is desirable to consider the similarity and the degree of coincidence between the locus of the instruction line and the outline of the object in determining the outline of the object corresponding to the locus of the instruction line.

Here, the similarity between the trajectory of the instruction line and the outline of the object can be obtained by calculating the distance between the trajectory of the instruction line and the outline of the object. Here, for example, when the mutually spaced distances are within a predetermined error, it can be determined that the locus of the instruction line corresponds to the outline of the object. Alternatively, the similarity between the locus of the instruction line and the outline of the object may be obtained by performing predetermined filtering on the locus of the instruction line and the outline of the object, and then comparing the filtered signals. Alternatively, it is also possible to perform signal conversion such as frequency conversion or wavelet conversion on the locus of the instruction line and the outline of the object, and then compare the converted signals to obtain the similar degree. It is needless to say that various conventional methods of obtaining similarity between two different two-dimensional or three-dimensional lines can be used.

Here, the similarity between the locus of the instruction line and the outline of the object can be compared in various spaces. For example, the locus of the instruction line and the outline of the object can be projected onto the space of the image and compared. In this case, the outline of the object can be used as the outline detected in the image, and the locus of the instruction line can be generated in accordance with the locus of the instruction line moving on the plane of the image. Alternatively, the locus of the instruction line and the outline of the object may be projected onto the captured three-dimensional space and compared. In this case, the locus of the instruction line may be generated in accordance with the locus of movement of the instruction line in the 3D space, and the outline of the object may be generated by using the outline on the 3D space corresponding to the outline detected in the image, An outline determined according to the outline information of the three-dimensional model that is known in advance for the detected object may be used as described below.

Here, in comparing the similarity between the trajectory of the instruction line and the outline of the object, it is also possible to compare the result of transformation such as size conversion or rotation conversion of the trajectory of the instruction line or the outline of the object. This is because, in the case of the instruction line generated according to the hand motion of the user, the size and the direction of the actual detected object may differ from the outline thereof, so that the correction is performed considering this. For example, the locus of the instruction line or the outline of the object can be resized so that the locus of the instruction line and the size of the outline of the object become the same. If necessary, the size of the locus of the instruction line or the outline of the object can be further increased by a predetermined multiple (for example, 0.8, 0.9, ≪ / RTI > 1.2 or the like). For example, it is also possible to carry out a conversion in which the trajectory of the instruction line or the outline of the object is rotated in the range of 0 to 360 degrees, and then the comparison is performed.

For example, the instruction object specification step S130 may include projecting the locus of the instruction line onto the space of the image, performing a transformation including a magnitude transformation or a rotation transformation on a trajectory of the projected instruction line or an outline of the projected object, It is possible to compare the result of the conversion and determine whether or not the locus of the instruction line and the outline of the object correspond to each other according to whether the locus of the projected instruction line and the outline of the object coincide within a predetermined error. In such a case, it can be said that the case of comparing the trajectory of the instruction line and the outline of the object on the space according to the image.

As another example, when the image is a depth image having three-dimensional depth information, the step of specifying an indicating object (S130) may include a step of determining a point of contact between the projection plane set at a predetermined position in the three- The trajectory of the instruction line and the outline of the object are compared with each other to determine whether the trajectory of the instruction line and the outline of the object correspond to each other. In this case, the object detection step S120 may obtain the outline of the object according to the outline information of the predetermined three-dimensional model corresponding to the object detected in the image. In this case, the contour of the object can be acquired according to the contour information according to the model using the three-dimensional model stored in advance for the type of the detected object and the identification information. In this case, the instruction object specifying step S130 may compare the outline of the obtained object with the locus of the instruction line, and specify the instruction object according to the result.

Next, the gesture recognition step (S200) will be described in more detail.

In the gesture recognition step S200, the gesture recognition unit 200 analyzes a pose of a part of the user's body in the image and recognizes a gesture for controlling the pointing object.

Here, the gesture may be a form, an attitude or a pose of a part of the user's body at a specific point in time, or a form or an attitude or a movement or a change of a pose of a part of the user's body for a predetermined time. Here, a part of the body of the user who analyzes to recognize the gesture may be a user's hand, and may be a user's arm, upper body, lower body, or body, if necessary. Preferably, the gesture can be recognized by analyzing the pose of the user's body according to the user's hand or arm motion.

9 is a detailed flowchart of the gesture recognition step S200.

The gesture recognition step S200 may include an illumination determination step S210, a gesture region extraction step S220, and a control gesture recognition step S230.

For example, in order to recognize the gesture of a human hand, it is necessary to detect the hand with the skin color and recognize the pose. However, in the indoor lighting environment, various colors may appear even with the hand of the same person due to the presence or absence of illumination and the difference in illumination. In the present invention, first, the illumination state information is obtained in the illumination determination step (S210), and then the body part for performing the gesture using the acquired illumination state information in the gesture region extraction step (S220) Extract the region.

The illumination determination step S210 analyzes the image and acquires illumination state information according to the illumination environment in which the image is captured.

Here, the illumination determination step (S210) may input the image to the previously learned illumination classifier to determine which of the plurality of illumination states the illumination environment photographed corresponds to and determine the illumination state information Can be obtained. It is needless to say that the illumination classifier can set the parameters of the classification function by learning basically the same methods as the existing methods described in the object classification.

For example, the illumination state may include natural light, fluorescent light, or incandescent light according to a predetermined order. In order to grasp the illumination state in the image, a plurality of learning images having the above illumination state are machine-learned using SVM or the like The illumination classifier can be learned. Here, the feature information for learning may include image signal information including contrast, brightness, hue, and saturation. Using the learned classifier, the illumination determination step S210 may obtain the illumination state information of the image in any of the illumination states including natural light, fluorescent light, and incandescent light.

The gesture region extraction step S220 extracts a body part region that performs the gesture in the user's body existing in the image according to the obtained illumination state information. Here, the region of the body part performing the gesture may be, for example, a user's hand.

Here, the region extracting step S220 may extract an area having a predetermined color or brightness range according to the illumination state information in the image as a body part region for performing the gesture. That is, for example, the Hue value corresponding to the human hand and the range of the saturation can be obtained according to the predetermined value in the illumination state information obtained from the above, and the region having the above range can be extracted as the region of the hand.

Next, the control gesture recognition step (S230) recognizes the gesture for controlling the pointing object by analyzing the pose of the body part region performing the extracted gesture.

Herein, the control gesture recognition step (S230) may extract a predetermined feature from the body part region in which the extracted gesture is performed, and input the feature vector according to the extracted feature to the learned gesture classifier to recognize the gesture. It is needless to say that the gesture classifier can also set parameters of the classification function by learning basically the same methods as the existing methods described in the object classification.

For example, in the control gesture recognition step S230, a gesture of a hand is defined using the SIFT or HOG feature, and the learned gesture using the learning image obtained for each hand gesture according to each predefined gesture to be recognized The gesture recognizer can recognize the gesture by classifying the image according to the region of the body part performing the extracted gesture using the gesture classifier.

Next, in the object control command output step S300, the object control command output unit 300 can generate and output a control command for the instruction object according to the recognized gesture.

Here, the object control command output step S300 may generate a predetermined control command for the instruction object identified and recognized in the instruction object recognition step S100 according to the gesture recognized in the gesture recognition step S200. For example, if a TV of a specific manufacturer is identified and recognized as an instruction object, if the predefined gesture is recognized, the recognized gesture means the recognized TV, A control command can be generated and output.

Yet another embodiment of the present invention may be a computer program stored in a medium for executing a device control method using a gesture according to the present invention.

Next, a device control apparatus using a gesture according to the present invention will be described.

10 is a block diagram of a device control apparatus using a gesture according to another embodiment of the present invention.

The device control apparatus using the gesture may include an instruction object recognition unit 100 and a gesture recognition unit 200, and may further include an object control command output unit 300 as needed. Here, the instruction object recognition unit 100, the gesture recognition unit 200, and the object control command output unit 300 can operate in the same manner as the device control method using the gesture described in detail with reference to FIGS. 1 to 9 . The overlapping portions will be omitted and briefly explained.

The instruction object recognition unit 100 receives an image of a space in which a user's controlled object exists, detects at least one or more objects in the image, detects the body of the user in the image, Recognizes the direction of the user according to the body posture of a user and identifies the instruction object indicated by the user among the objects according to the direction of the user and recognizes the specific instruction object.

The gesture recognition unit 200 recognizes a gesture for controlling the pointing object by analyzing a pose of a part of the user's body in the image.

The object control command output unit 300 generates and outputs a control command for the instruction object according to the recognized gesture.

11 is a detailed block diagram of the instruction object recognition unit 100. As shown in FIG.

The instruction object recognition unit 100 may include an instruction direction recognition unit 110, an object detection unit 120 and an instruction object specification unit 130 and may further include an object recognition unit 140 as needed .

The instruction-direction recognizing unit 110 detects at least one or more body parts predetermined in the body of the detected user, generates an instruction line according to the direction indicated by the user using the detected body parts, The instruction direction can be recognized.

The object detection unit 120 can detect and recognize the object in the image.

The instruction object specifying unit 130 specifies the object existing on the instruction line or within a predetermined distance from the instruction line among the detected objects as the instruction object, The controller can compare the detected outline of the object and specify the object having the outline corresponding to the locus of the instruction line as the instruction object when it determines that the locus of the instruction line and the outline of the object correspond.

The object recognition unit 140 can recognize the type of the detected object or the specific instruction object by using an object classifier previously learned.

12 is a detailed block diagram of the gesture recognition unit 200. As shown in FIG.

The gesture recognition unit 200 may include an illumination determination unit 210, a gesture region extraction unit 220, and a control gesture recognition unit 230.

The illumination determination unit 210 analyzes the image and acquires illumination state information according to the illumination environment in which the image is captured.

The gesture region extracting unit 220 extracts a body part region that performs the gesture from the body of the user existing in the image according to the acquired illumination state information.

The control gesture recognition unit 230 recognizes the gesture for controlling the pointing object by analyzing a pose of a body part region performing the extracted gesture.

It is to be understood that the present invention is not limited to these embodiments, and all elements constituting the embodiment of the present invention described above are described as being combined or operated in one operation. That is, within the scope of the present invention, all of the components may be selectively coupled to one or more of them.

In addition, although all of the components may be implemented as one independent hardware, some or all of the components may be selectively combined to perform a part or all of the functions in one or a plurality of hardware. As shown in FIG. In addition, such a computer program may be stored in a computer readable medium such as a USB memory, a CD disk, a flash memory, etc., and read and executed by a computer to implement an embodiment of the present invention. As the recording medium of the computer program, a magnetic recording medium, an optical recording medium, a carrier wave medium, and the like can be included.

Furthermore, all terms including technical or scientific terms have the same meaning as commonly understood by one of ordinary skill in the art to which this invention belongs, unless otherwise defined in the Detailed Description. Commonly used terms, such as predefined terms, should be interpreted to be consistent with the contextual meanings of the related art, and are not to be construed as ideal or overly formal, unless expressly defined to the contrary.

It will be apparent to those skilled in the art that various modifications, substitutions and substitutions are possible, without departing from the scope and spirit of the invention as disclosed in the accompanying claims. will be. Therefore, the embodiments disclosed in the present invention and the accompanying drawings are intended to illustrate and not to limit the technical spirit of the present invention, and the scope of the technical idea of the present invention is not limited by these embodiments and the accompanying drawings . The scope of protection of the present invention should be construed according to the following claims, and all technical ideas within the scope of equivalents should be construed as falling within the scope of the present invention.

S100: Instruction object recognition step
S110: instruction direction recognition step
S120: Object detection step
S130: Instruction object specific step
S140: Object recognition step
S200: Gesture recognition step
S210: Lighting determination step
S220: Gesture region extraction step
S230: Control Gesture Recognition Step
S300: Object control command output step
100: Instruction object recognition unit
110: instruction direction recognition unit
120: Object detection unit
130: Instruction Object Specification
200: Gesture recognition unit
210:
220: Gesture region extracting unit
230: control gesture recognition unit
300: Object control command output unit

Claims (20)

An instruction object recognition unit receives an image of a space in which a user's control object exists, detects at least one object in the image, detects the body of the user in the image, Recognizing an instruction direction of a user according to an attitude of the user and identifying an instruction object indicated by the user among the objects according to the instruction direction of the user and recognizing the specific instruction object; And
A gesture recognition step of recognizing a gesture for controlling the pointing object by analyzing a pose of a part of the user's body in the image;
Wherein the instruction object recognition step comprises:
A direction line according to a line direction indicated by a user is generated according to the body posture of the user to recognize a direction of the user,
Detecting an outline of the object in the image to detect the object,
Comparing an outline of the detected object with a locus of the generated instruction line, and specifying an instruction object instructed by the user according to the comparison result. The method according to claim 1,
Wherein the object control command outputting step further comprises an object control command outputting step of generating and outputting a control command for the instruction object according to the recognized gesture. The method according to claim 1,
Wherein the image is a depth image having three-dimensional depth information. The method according to claim 1,
The instruction object recognition step may include detecting at least one or more body parts predetermined in the body of the detected user, generating an instruction line according to a line direction indicated by the user using the detected body part, And the instruction object is identified using the instruction line. 5. The method of claim 4,
An instruction direction recognition step of detecting the body part in the body of the user, generating the instruction line using the body part, and recognizing the instruction direction of the user;
An object detecting step of detecting the object in the image; And
And an instruction object specifying step of specifying the object existing on the instruction line or within the predetermined distance from the instruction line as the instruction object among the detected objects. 5. The method of claim 4,
Wherein the instruction object recognition step comprises an object recognition step of recognizing the type of the detected object or the specific instruction object using a pre-learned object classifier. 6. The method of claim 5,
The instruction direction recognizing step detects the body part including at least one of the elbow part, the hand part, the shoulder part, and the eye part in the body of the user, and generates the instruction line using the position of the body part And the device control method using the gesture. 8. The method of claim 7,
Wherein the instruction direction recognition step includes a step of recognizing the direction of the user when at least two joints of the user's elbow joint, hand joint or shoulder joint among the detected body parts satisfy predetermined rules, And a line connecting the positions of the gestures is generated as the instruction line. delete 6. The method of claim 5,
Wherein the instruction object specifying step compares the locus of the instruction line with the movement of the instruction line generated in the instruction direction recognition step and the outline of the object detected in the object detection step to determine whether the outline of the instruction line and the outline of the object Wherein the determining step determines that the object having the outline corresponding to the locus of the instruction line is identified as the instruction object. 11. The method of claim 10,
Wherein the instructing object specifying step compares the outline of each of the plurality of objects with the locus of the instruction line when there are a plurality of the objects existing on the instruction line or within a predetermined distance from the instruction line, And selecting the object having the outline corresponding to the locus of the instruction line among the objects as the instruction object. 11. The method of claim 10,
Wherein the step of specifying the instruction object includes the step of projecting the locus of the instruction line onto the space of the image, performing a transformation including a magnitude transformation or a rotation transformation on the trajectory of the projected instruction line or the outline of the object, And judges whether or not the trajectory of the instruction line and the outline of the object coincide with each other according to whether or not the trajectory of the projected instruction line and the outline of the object coincide within a predetermined error. . 11. The method of claim 10,
The image is a depth image having three-dimensional depth information,
Wherein the step of specifying the pointing object includes the step of obtaining a path of movement of a point of contact between the projection plane and the pointing line set at a predetermined position within a three-dimensional space corresponding to the depth image as a locus of the pointing line, And comparing the outline of the object to determine whether or not the locus of the instruction line corresponds to the outline of the object. 14. The method of claim 13,
Wherein the object detecting step obtains an outline of the object according to outline information of a predetermined three-dimensional model corresponding to the object detected in the image,
Wherein the instruction object specifying step compares the outline of the obtained object with the locus of the instruction line, and specifies the instruction object according to a result of the comparison. The method of claim 1, wherein the gesture recognition step
An illumination determination step of analyzing the image to acquire illumination state information according to the illumination environment in which the image is captured;
A gesture region extracting step of extracting a body part region for performing the gesture from the body of the user existing in the image according to the acquired illumination state information; And
And a control gesture recognition step of recognizing the gesture for controlling the instruction object by analyzing a pose of a body part region performing the extracted gesture. 16. The method of claim 15,
The illumination determination step may include inputting the image into a previously learned illumination classifier to determine which of a plurality of illumination states the illumination environment in which the image is photographed corresponds to, and acquiring the illumination state information according to the determination result ,
Wherein the region extracting step extracts an area having a predetermined color or brightness range according to the illumination state information in the image as a body part region for performing the gesture,
Wherein the control gesture recognition step recognizes the gesture by extracting a predetermined feature from an area of a body part performing the extracted gesture and inputting a feature vector according to the extracted feature to a learned gesture classifier. A method of controlling a device. A computer program stored in a medium for executing a device control method using a gesture according to any one of claims 1 to 8 and 10 to 16. A method of detecting a user's body, comprising the steps of: receiving an image of a space in which a user's controlled object exists; detecting at least one or more objects in the image; detecting the body of the user in the image; An instruction object recognizing unit for recognizing the instruction direction of the user and identifying the instruction object indicated by the user among the objects according to the instruction direction of the user and recognizing the specific instruction object;
A gesture recognition unit for recognizing a gesture for controlling the pointing object by analyzing a pose of a part of the body of the user in the image; And
An object control command output unit for generating and outputting a control command for the instruction object according to the recognized gesture;
Wherein the instruction object recognition unit comprises:
A direction line according to a line direction indicated by a user is generated according to the body posture of the user to recognize a direction of the user,
Detecting an outline of the object in the image to detect the object,
Compares an outline of the detected object with a locus of the generated instruction line, and specifies an instruction object instructed by the user according to the comparison result. 19. The apparatus of claim 18,
Detecting at least one body part predetermined in the body of the detected user and generating an instruction line according to a line direction indicated by the user using the detected body part to recognize the instruction direction of the user part;
An object detection unit for detecting and recognizing the object in the image; And
The object being present on the instruction line or within a predetermined distance from the instruction line as the instruction object among the detected objects, wherein the locus of the instruction line and the outline of the object detected by the object detection unit are compared And an instruction object specifying unit for specifying the object having the outline corresponding to the locus of the instruction line as the instruction object when it is determined that the locus of the instruction line and the outline of the object correspond to each other, Device. 19. The apparatus of claim 18, wherein the gesture recognition unit
An illumination determination unit for analyzing the image to acquire illumination state information according to the illumination environment in which the image is captured;
A gesture region extracting unit for extracting a body part region that performs the gesture from the body of the user existing in the image according to the obtained illumination state information; And
And a control gesture recognition unit for recognizing the gesture for controlling the indication object by analyzing a pose of a body part region performing the extracted gesture.

Images