KR102192051B1 - Device and method for recognizing motion using deep learning, recording medium for performing the method - Google Patents

Abstract

An object is photographed using an infrared camera, and a learning network model is used to recognize whether a user is present in the image captured by the camera, and an image that detects the area where the user is located is generated, and the user's movement changes according to time. Initiated a motion recognition-based manipulation device that recognizes and determines the type of motion using a learning network model, generates a motion determination result, and generates and manipulates a manipulation command for an external device in consideration of characteristic information for the external device. Are doing.

Description

Motion recognition-based operation device and method {DEVICE AND METHOD FOR RECOGNIZING MOTION USING DEEP LEARNING, RECORDING MEDIUM FOR PERFORMING THE METHOD}

The present invention relates to a motion recognition-based manipulation device and method, and more particularly, to a motion recognition-based manipulation device and method for manipulating the device by recognizing a user, a user's motion, and a type of motion from an image captured in real time. .

The content described in this section merely provides background information on the present embodiment and does not constitute the prior art.

Computers and smartphones have been widely distributed and digital signal processing technologies have been developed, and in recent years, technologies such as artificial intelligence, video/motion recognition, deep learning, and machine learning have developed. By doing so, intelligent services that automatically recognize data such as voice, image, video, or text to provide data-related information or provide data-related services are being used in various fields.

In particular, research on the natural user interface (NUI) technology between users and devices is active. In particular, interfaces using hands such as hand shape and hand gesture recognition technology enable natural and intuitive interaction. The advantage of not requiring additional equipment is being actively used in HCI (Human-Computer Interaction), HRI (Human-Robot Interaction), and HMI (Human-Machine Interaction) fields.

The artificial intelligence system is a computer system that implements human-level intelligence, and unlike the existing rule-based smart system, the machine learns, judges, and becomes smarter. As the artificial intelligence system is used, the recognition rate improves and user preferences can be understood more accurately, and the existing rule-based recognition method is gradually being replaced by a deep-learning-based artificial intelligence method.

In this regard, learning-based recognition technology acquires a database of the hand shape to be recognized and recognizes the hand shape through a model created by learning it, but the existing machine learning-based recognizer It was difficult to learn the characteristics of the level, and there was a limitation in that stable performance could not be obtained in environments other than the learning environment.

Accordingly, there is a need for a method capable of obtaining high performance even in various environments that are actually used.

Korean Patent Registration No. 10-1386592 (2014.04.11.) Korean Patent Publication No. 10-2012-0076477 (2012.07.09.) Korean Patent Publication No. 10-2008-0036423 (2008.04.28.)

An aspect of the present invention provides a motion recognition-based manipulation apparatus and method for improving recognition performance by separating a process of recognizing a user, a user's motion, and a type of motion using a learning network model.

The technical problem of the present invention is not limited to the technical problem mentioned above, and other technical problems that are not mentioned will be clearly understood by those skilled in the art from the following description.

An image input unit for photographing an object according to an embodiment of the present invention; A user detection unit that recognizes whether an object photographed by the image input unit is a user using the learning network model and detects an area in which the user is located; A motion recognition unit that calculates a change amount by comparing a frame at a specific time point in a region in which the user is located detected by the user detection unit and a frame after a threshold time has elapsed from the specific time point, and recognizes a user's motion when the change amount is greater than a threshold value; A motion determination unit for recognizing what kind of motion is the user's motion using the learning network model and generating a motion determination result corresponding to the recognized motion; And a manipulation unit that manipulates at least one external device according to an operation determination result of the operation determination unit.

In an embodiment, when a plurality of users are recognized, the user detection unit may detect all regions in which the user is located.

In an embodiment, if there are a plurality of areas in which the detected user is located, the motion recognition unit may recognize whether there is a user's movement in each area.

In one embodiment, the motion determination unit, when the motion recognition unit recognizes the motions of a plurality of users, recognizes what kind of motion is the user's motion for each, and determines the plurality of motions corresponding to each recognized motion. Can be created.

In an embodiment, if the operation determination result is plural, the manipulation unit may manipulate the external device according to the operation determination result.

In an embodiment, if the plurality of operation determination results are to operate separate external devices, respectively, the operation unit may operate the external device according to the operation determination result.

In one embodiment, if the plurality of operation determination results are intended to operate the same external device, and the plurality of operation determination results are to perform the same operation, the external device according to the operation determination result in the order of the user close to the external device. Can be manipulated.

In one embodiment, if the plurality of operation determination results are intended to manipulate the same external device, and the plurality of operation determination results are to perform contradictory operations, the operation unit is based on the operation determination result of the user located closest to the external device. External devices can be operated.

In an embodiment, if the operation determination result is capable of operating a plurality of external devices, the manipulation unit may operate one external device according to a set mode.

In an embodiment, when the set mode is a distance priority mode, the manipulation unit may manipulate an external device located close to the user.

In an embodiment, when the set mode is a gaze priority mode, the manipulation unit may manipulate an external device located in the gaze direction of the user.

In an embodiment, when the set mode is a hand direction priority mode, the manipulation unit may manipulate an external device positioned in the direction indicated by the user's hand.

In accordance with an embodiment of the present invention, there is provided a motion recognition-based manipulation method performed by a motion recognition-based manipulation device, comprising: photographing an object by the gesture recognition-based manipulation device; Recognizing, by the motion recognition-based manipulation device, whether the captured object is a user using the learning network model, and detecting a region where the user is located; Calculating a change amount by comparing, by a motion recognition-based manipulation device, a frame at a specific point in time in a region where a user is located and a frame after a threshold time, and recognizing that the change amount is greater than a threshold value as a user's movement; Recognizing, by the motion recognition-based manipulation device, what kind of motion is the user's motion using the learning network model, and generating a motion determination result corresponding to the recognized motion; The motion recognition-based manipulation device includes: selecting one or more motion determination results when two or more motion determination results generated simultaneously; The motion recognition-based manipulation device includes: selecting one external device when there are two or more external devices to be manipulated based on a result of the motion determination; And manipulating, by the motion recognition-based manipulation device, a selected external device according to a result of the motion determination.

In an embodiment, the operation of the external device may further include selecting one or more results of the operation determination when two or more results of the operation determination simultaneously generated are selected.

In an embodiment, the operating of the external device may further include selecting one external device when there are two or more external devices to be operated based on the operation determination result.

According to an aspect of the present invention described above, the effect provided by the motion recognition-based manipulation device and method of the present invention is, through deep-learning, high recognition performance can be obtained even in a user environment in which changes in pixels, brightness, and saturation vary. Therefore, it can provide versatility and expandability that can be applied to the entire gesture recognition-based input system.

The technical problem of the present invention is not limited to the technical problem mentioned above, and other technical problems that are not mentioned will be clearly understood by those skilled in the art from the following description.

1 is a block diagram showing a configuration of a motion recognition-based manipulation device according to an embodiment of the present invention.
2 is a diagram illustrating a state in which a user detection unit according to the present invention recognizes a user.
3 is a diagram illustrating a state in which a motion recognition unit according to the present invention recognizes a user's hand.
4 is a diagram illustrating a state in which a motion determination unit according to the present invention recognizes a change in a user's hand.
5 is a diagram illustrating a state in which an external device is operated when there are a plurality of users.
6 is a diagram illustrating a state in which a target manipulates one external device when a result of determining an operation according to the present invention can manipulate a plurality of external devices.
7 is a flowchart illustrating a gesture recognition-based manipulation method according to an embodiment of the present invention.
FIG. 8 is a flow chart subdividing the step of selecting one or more motion determination results when there are two or more simultaneous motion determination results when there are multiple users according to the present invention.
9 is a flowchart illustrating a method of subdividing a method in which a target manipulates one external device when the operation determination result according to the present invention is capable of operating a plurality of external devices.

DETAILED DESCRIPTION OF THE INVENTION The detailed description of the present invention to be described later refers to the accompanying drawings, which illustrate specific embodiments in which the present invention may be practiced. These embodiments are described in detail sufficient to enable a person skilled in the art to practice the present invention. It is to be understood that the various embodiments of the present invention are different from each other, but need not be mutually exclusive. For example, certain shapes, structures, and characteristics described herein may be implemented in other embodiments without departing from the spirit and scope of the present invention in relation to one embodiment. In addition, it is to be understood that the location or arrangement of individual components within each disclosed embodiment may be changed without departing from the spirit and scope of the present invention. Accordingly, the detailed description to be described below is not intended to be taken in a limiting sense, and the scope of the present invention, if properly described, is limited only by the appended claims, along with all scopes equivalent to those claimed by the claims. Like reference numerals in the drawings refer to the same or similar functions over several aspects.

Hereinafter, preferred embodiments of the present invention will be described in more detail with reference to the drawings.

1 is a block diagram showing the configuration of a motion recognition apparatus according to an embodiment of the present invention.

Referring to FIG. 1, a motion recognition-based manipulation device 10 may include an image input unit 100, a user detection unit 300, a motion determination unit 700, and a manipulation unit 900.

The image input unit 100 photographs an object using an infrared camera, and the object includes a user to be recognized and an object such as furniture, and the distance to the subject may be set differently according to the environment of the house.

The user detection unit 300 may recognize whether an object photographed by the image input unit is a user using the learning network model, and detect an area in which the user is located.

The user detection unit 300 will be described below with reference to FIG. 2.

2 is a diagram illustrating a state in which a user detection unit according to the present invention recognizes a user.

Referring to FIG. 2, when there is one user (a3, a4) in an image captured by the image input unit 100 of the motion recognition-based manipulation device 10 (see FIG. 2A) and when there are more than one user (see FIG. 2B). You can see the state of

The user detection unit 300 may recognize a user from an image captured by the image input unit 100 by using a learning network model such as a 2D convolutional neural network (CNN).

The user detection unit 300 performs a down sampling process and an up sampling process to recognize the user a3.

For example, when the size (L) of an image captured by the image input unit 100 is 416Х416, the user detection unit 300 uses down sampling to have an image size of 13Х13 (=L/32). Is performed until a first order feature map is obtained.

The user detection unit 300 determines whether the object is an area in the primary feature map, determines whether it is an object, obtains an objectness score, and determines which object is a class score. ) To find a large object such as a user (a3), a sofa (a1), and a dining table (a2).

After finding a large object, the user detection unit 300 performs up-sampling, and a second feature map having an image size of 26Х26 (=L/16) and an image size of 52Х52 A third-order feature map of (=L/8) is obtained again.

In this case, when up-sampling is performed, previous information is included by adding features extracted in the down-sampling process.

The third feature map is used to find a small object such as the hand of the user a3.

According to the above-described process, the learning network model of the user detection unit 300 determines whether the photographed object is an object such as a sofa (a1) or a dining table (a2), the location of the object, and the type of the object to determine the user (a3, a5). ), and the areas a4 and a6 in which the user is located can be detected.

The motion recognition unit 500 calculates the amount of change by comparing the frame at a specific point in time in the region where the user is located detected by the user detection unit and the frame after a threshold time has elapsed from the specific point in time, and if the amount of change is greater than the threshold value, it is recognized as the user's movement can do.

The motion recognition unit 500 will be described below with reference to FIGS. 3 and 4.

The motion recognition unit 500 compares the difference between a frame at a specific point in time T and a frame after a critical time t has elapsed from the specific point in time T to calculate a frame difference. I can.

The frame difference is a sum of differences between all frames existing in an area in which the user is located, and the motion recognition unit 500 may recognize that there is a motion when the frame difference is greater than a threshold value.

In this case, the threshold value and the threshold time t are preset values.

The frame difference can be calculated according to Equation 1 below.

[Equation 1]

Frame difference = |frame(T+t)-frame(T)|

Here, T denotes the current time, t denotes a critical time, and frame (T) denotes a frame appearing at a specific time T.

3 is a diagram illustrating a state in which a motion recognition unit according to the present invention recognizes a user's hand.

Referring to FIGS. 3A to 3C, the motion recognition unit 500 may recognize a hand that is a small object within the recognized area.

The user detection unit 300 may recognize the user's hand in a third feature map, which is a 52×52 (=L/8) sized image, and the area b1, recognized according to the shape of the user's hand. The sizes of b2 and b3) are different.

It can be seen that the size of the image recognized in a state in which the user's hand is clenched (see FIG. 3A), a state with only two fingers open (see FIG. 3B), and a state with all fingers (see FIG. 3C) are different.

4 is a diagram illustrating a state in which a motion determination unit according to the present invention recognizes a change in a user's hand.

Referring to FIGS. 4A to 4E, the motion recognition unit 500 may recognize a change in the user's movement over time in an image in which the user detection unit 300 detects an area where the user is located.

Although the motion recognition unit 500 can recognize various motions, a representative motion, left and right movement (refer to FIG. 4A), will be described as an example.

The motion recognition unit 500 may recognize a movement of a user's hand by calculating a change in a frame from an up-sampled image.

The motion recognition unit 500 may determine that there is no motion because there is no frame difference in the images of FIGS. 3A to 3C, and in this case, the recognition process is performed without transmitting the image to the motion determination unit 700. It ends.

As in the case of FIGS. 4A and 4B, the motion recognition unit 500 determines that there is a user's movement when there is a frame difference, and in this case, the motion determination unit 700 may transmit an image.

The motion determination unit 700 may recognize what kind of motion the user's motion is by using the learning network model, and may generate a motion determination result corresponding to the recognized motion.

The operation determination unit 700 will be described below with reference to FIG. 4.

The motion determination unit 700 uses a learning network model such as a 3D convolutional neural network (CNN) to determine the type of the user's motion with respect to an image in which a change in motion is recognized by the motion recognition unit 500. It is possible to determine what kind of motion the user's motion is.

The motion determination unit 700 is composed of a 3D convolution layer, a 3D max pooling layer, and an inception layer including these two layers. .

The motion determination unit 700 may obtain a feature map from a 3D convolutional neural network (CNN) and recognize the type of a user's motion from the feature map.

The motion determination unit 700 determines the motion using a softmax function as an activation function of a 3D convolutional neural network (CNN) for dimensional recognition from a feature map. I can.

Referring to FIG. 4A, it can be seen that the user's hand has moved from the right (c2) to the left (c3) based on the user.

The motion determination unit 700 may first recognize the hand located on the right (c2), recognize the process of moving from the right (c2) to the left, and finally recognize the hand located on the left (c3). I can.

The motion determination unit 700 performs three-dimensional movements such as moving from right (c2) to left (c3), moving from top (c4) to bottom (c5), and moving forward/backward with only two fingers open. I can recognize it.

The motion determination unit 700 may generate a motion determination result according to the type of motion of the user's motion.

For example, moving left/right with only two fingers open can generate a motion judgment result by changing the channel, and moving up/down can generate a motion judgment result by adjusting the volume. In addition, the motion of moving forward/backward may generate a motion determination result by stopping or restarting the image.

The above motion determination result is merely an example for understanding, and the motion determination result may vary according to a finger spreading state, left/right, vertical movement.

5 is a diagram illustrating a state in which an external device is operated when there are a plurality of users.

Referring to FIG. 5, when there are multiple users of the motion recognition-based manipulation device 10, the user detection unit 300, the motion recognition unit 500, the motion determination unit 700, and the operation unit 900 perform separate operations. Is done.

When a plurality of users are recognized, the user detection unit 300 may detect all regions in which the user is located.

If there are a plurality of regions in which the detected user is located, the motion recognition unit 500 may recognize whether or not there is a user's movement in each region.

When the motion recognition unit recognizes the motions of a plurality of users, the motion determination unit 700 recognizes what kind of motion is the user's motion for each, and generates a plurality of motion determination results in response to each recognized motion. can do.

If the operation determination result is two or more, the operation determination result may be selected according to a predetermined algorithm, the operation unit 900 may select an external device to be operated according to the selected operation determination result, and then operate the corresponding external device.

The operation unit 900 can perform the same operation or contradictory operation on the same device when there are two or more results of operation determination simultaneously generated through parallel processing, so it determines whether the operation determination result is one or two or more. can do.

When the operation determination result is one, the manipulation unit 900 may select the operation determination result as a result of determining the operation to be performed because the operation may be performed according to the operation determination result.

When the operation determination result is two or more, the operation unit 900 may determine whether each operation determination result is to operate a separate external device or the same external device.

If the result of determining the plurality of operations is to manipulate separate external devices, the manipulation unit 900 may manipulate the external devices according to the results of each operation determination. Therefore, each operation determination result can be selected as an operation determination result to perform the manipulation.

If the result of determining the plurality of operations is that the same external device is operated, the manipulation unit 900 may determine whether to perform the same operation. This is because when the same operation is performed, the operation according to the result of determining the simultaneously generated operation must be continuously performed.

In the case of performing the same operation on a device having a plurality of operation determination results, the operation unit 900 selects the user's operation determination result in the order of close distance to the external device to be operated according to the plurality of operation determination results and performs the operation. I can do it.

If the result of determining a plurality of operations is to manipulate the same external device, the operation unit 900 may determine whether to perform contradictory operations. When performing contradictory operations, if both are performed, as a result, the same result as not performing one Therefore, it is to select only one operation determination result and perform the operation.

When a plurality of operation determination results are different from each other on the same device, the operation unit 900 may select a user's operation determination result located at the closest distance to the external device to be operated.

Otherwise, it is not the same operation for the same device, and it is not an opposite operation, so you can perform each operation. Therefore, the operation is performed by selecting each operation determination result.

A first example is a case where a plurality of operation determination results attempt to manipulate separate external devices.

User 1 (d1) wants to change the channel of the set-top box (d3), and user 2 (d2) wants to change the volume of the radio (d4).

In this case, the channel of the set-top box d3 is changed according to the operation determination result of user 1 (d1), and the volume of the radio d4 is changed according to the operation determination result of user 2 (d2).

As a second example, if a plurality of operation determination results are intended to operate the same external device, and a plurality of operation determination results are intended to perform the same operation, the operation may be performed by selecting the operation determination result in the order of the user close to the external device. have.

For example, when both user 1 (d1) and user 2 (d2) want to increase the volume of the set-top box d3.

Since the external device to be operated is the set-top box (d3), first, the distance between the set-top box (d3) and user 1 (d1) (dist_d1) and the set-top box (d3) and user 2 (d2) (dist_d2) is obtained. .

By comparing each distance (dist_d1, dist_d2), the volume of the set-top box (d3) is increased according to the operation determination result of the user 2 (d2) located close to the set-top box (d3), and then the volume of the set-top box (d3) is increased. ), the volume of the set-top box (d3) is raised again.

A third example is a case where a plurality of operation determination results are attempted to operate an external device with the same, and a case where the opposite operation is to be performed.

If the plurality of operation determination results are intended to manipulate the same external device, and the plurality of operation determination results are intended to perform contradictory operations, the operation unit 900 may perform the operation determination result of a user located closest to the external device. .

For example, user 1 (d1) wants to increase the volume of the set-top box (d3), and user 2 (d2) wants to decrease the volume of the set-top box (d3).

Since the external device to be operated is the set-top box (d3), first, the distance between the set-top box (d3) and user 1 (d1) (dist_d1) and the set-top box (d3) and user 2 (d2) (dist_d2) is obtained. .

The user 2 (d2) located near the set-top box (d3) by comparing each distance (dist_d1, dist_d2) is the user 2 (d2), so the volume of the set-top box (d3) is lowered according to the operation determination result of the user 2 (d2). .

A fourth example is a case in which a plurality of operation determination results are attempted to operate an external device having the same, and a completely different operation is performed.

For example, it is assumed that user 1 (d1) wants to increase the volume of the set-top box d3, and user 2 (d2) wants to change the channel of the set-top box d3.

In this case, the volume of the set-top box d3 is increased according to the result of determining the operation of the user 1 (d1), and the channel of the set-top box d3 is changed according to the result of the determination of the operation of the user 2 (d2).

The manipulation unit 900 may manipulate at least one external device according to an operation determination result.

In the case of a general home appliance, the external device can be manipulated by an infrared remote control, so the operation unit 900 can operate the external device by transmitting a signal in infrared.

The operation unit 900 may determine whether the operation determination result of the operation determination unit 700 is to operate one device or two or more devices.

When there is one external device that can be manipulated, the manipulation unit 900 may select the corresponding external device as an external device to be operated.

If there are two or more external devices that can be manipulated, the manipulation unit 900 may determine whether there are two or more powered devices among the corresponding external devices.

In general, when a user moves his or her hand, it can be determined that he intends to manipulate one external device. Therefore, the operation is performed by distinguishing whether there is one external device or two or more external devices.

In this case, when power is applied to only one external device, the manipulation unit 900 may select the external device to which the power is applied as the manipulation target device.

When there are two or more external devices to which power is applied, one external device may be selected as a target device by checking a preset mode. The mode can be set in advance and can be changed later according to the movement of the user's hand.

A method of operating one external device according to the set mode will be described below with reference to FIG. 6.

6 is a diagram illustrating a state in which a target manipulates one external device when a result of determining an operation according to the present invention can manipulate a plurality of external devices.

Since the channel change can be applied to all of the projector (e2), television (e3), set-top box (e4), and radio (e5), the operation unit 900 must select one external device from among them, and thus the operation determination unit ( A case where the result of determining the operation of 700) is a channel change will be described as an example.

Referring to FIG. 6A, the manipulation unit 900 may calculate location information and distance of the object by calculating the location and depth of the object such as a user and an object in the image.

Before receiving a user's command, the control unit 900 can determine the positional relationship with external devices such as the projector (e2), television (e3), set-top box (e4), and radio (e5), and according to the identified positional relationship. The location of the external device can be specified in the X coordinate and Y coordinate on the plane.

In an embodiment, when the set mode is a distance priority mode, the manipulation unit 900 may operate an external device located close to the user (see FIG. 6C ).

The manipulation unit 900 may obtain a distance between a user and an external device using the above-described location information and distance, and select and operate an external device located at the nearest location.

In this case, the distance can be calculated from the coordinates determined in advance, and according to the result, the radio e5, which is the device having the smallest value, can be selected and operated.

In an embodiment, when the set mode is the gaze priority mode, the manipulation unit 900 may manipulate an external device located in the user's gaze direction (see FIG. 6D ).

The manipulation unit 900 may obtain a user's location according to the above-described location information, obtain a user's gaze direction from an image of an area where the user's face is located, and select and manipulate an external device located in the user's gaze direction.

Referring to FIG. 6D, since the user e1 is looking at the projector e2 located on the left, the operation unit 900 can select and operate the projector.

In an embodiment, when the set mode is a hand direction priority mode, the manipulation unit 900 may manipulate an external device positioned in the direction indicated by the user's hand (see FIG. 6E).

The manipulation unit 900 may obtain a user's location according to the above-described location information, obtain a direction indicated by the user's hand, and select and manipulate an external device located in the direction of the user's hand.

In FIG. 6E, since the user e1 points to the television e3 located in front, the operation unit 900 can select and operate the television e3.

7 is a flowchart illustrating a gesture recognition-based manipulation method according to an embodiment of the present invention.

The gesture recognition-based manipulation method according to the present invention may be performed by the gesture recognition-based manipulation device 10 according to the present invention. To this end, in the motion recognition-based manipulation device 10 according to the present invention, an application (software) for performing each step of configuring a motion recognition-based manipulation method to be described later may be installed in advance. For example, a platform for a motion recognition-based manipulation method according to the present invention may be pre-installed in the user's computer in the form of software, and the user executes the software installed on the computer to provide a motion recognition-based manipulation method according to the present invention. You can receive various services provided.

Referring to FIG. 7, the motion recognition-based manipulation device 10 includes a photographing step (S1100), a step of recognizing and detecting a user (S1200), a step of recognizing a user's motion (S1300), and the type of the user's motion. Recognizing and generating an operation determination result (S1400), selecting an operation determination result (S1500), selecting an external device (S1600), and operating an external device according to the operation determination result (S1700) Can include.

The motion recognition-based manipulation device 10 may capture an object (S1100), recognize whether the captured object is a user using a learning network model, and detect an area in which the user is located (S1200).

The motion recognition-based manipulation device 10 calculates a change amount by comparing a frame at a specific point in time and a frame after a threshold time in the region where the user is located, and recognizes that the change amount is greater than the threshold value as the user's movement (S1300).

The motion recognition-based manipulation device 10 may recognize what kind of motion is the user's motion using the learning network model, and may generate a motion determination result corresponding to the recognized motion (S1400).

The motion recognition-based manipulation device 10 may select one or more motion determination results when there are two or more simultaneous motion determination results (S1500).

The motion recognition-based manipulation device 10 may select one external device when there are two or more external devices to be manipulated based on a result of the motion determination (S1600).

The motion recognition-based manipulation device 10 may manipulate at least one external device according to a result of the motion determination (S1700).

A detailed description of the above-described steps S1100 to S1700 has been described above with reference to FIGS. 1 to 6, and thus repeated descriptions will be omitted.

FIG. 8 is a flow chart subdividing the step of selecting one or more motion determination results when there are two or more simultaneous motion determination results when there are multiple users according to the present invention.

Referring to FIG. 8, in the operation recognition-based manipulation method, when there are multiple users, recognizing a user and detecting a plurality of regions (S1200), recognizing movements of a plurality of users (S1300), and Recognizing the type and generating a plurality of motion determination results (S1400), selecting one or more motion determination results from among a plurality of motion determination results (S1500), one subject to manipulation according to the selected motion determination result Selecting the external device of (S1600) and manipulating the selected external device according to the selected operation determination result (S1700) are performed.

When there are two or more users, the step of recognizing and detecting the user (S1200), the step of recognizing the user's movement (S1300), and the step of recognizing the type of the user's movement and generating a motion determination result (S1400) are For each user, the same operation as in the case of one user is performed, but can be performed simultaneously through parallel processing.

When two or more users simultaneously move their hands to attempt manipulation, a separate procedure is required (S1500).

If there are two or more results of determining an operation simultaneously generated through parallel processing, the same operation or a contrary operation may be performed on the same device, so it is determined whether there is one or more than one operation determination result (S1510).

If there is only one operation determination result, the operation may be performed according to the corresponding operation determination result, so that one operation determination result may be selected as the operation determination result to perform the operation (S1590).

When the result of determining the operation is more than two, it is determined whether the operation determination result is to operate a separate external device or the same external device (S1530).

If the result of determining a plurality of motions is to manipulate separate external devices, the external devices may be operated according to the results of each operation determination. Accordingly, each operation determination result may be selected as an operation determination result to perform the manipulation (S1590).

If the result of determining the plurality of operations is to manipulate the same external device, it may be determined whether the same operation is performed (S1550). This determination is because when the same operation is performed, the operation according to the result of determining the simultaneously generated operation must be continuously performed (S1550).

When the same operation is performed on a device with a plurality of operation determination results, the operation may be performed by selecting the user's operation determination result in an order of close distance to the external device to be operated according to the plurality of operation determination results ( S1560).

If the result of determining a plurality of operations is to manipulate the same external device, it may be determined whether or not contradictory operations are performed (S1570) In case of performing contradictory operations, if both are performed, the result is the same as not performing one of them. The operation is performed by selecting only the result of determining any one operation (S1570).

When a plurality of operation determination results are performed on the same device in contradictory operation, the operation determination result of the user located at the closest distance to the external device to be operated may be selected (S1580).

Otherwise, it is not the same operation for the same device, and it is not an opposite operation, so you can perform each operation. Therefore, each operation determination result is selected as an operation determination result to perform the manipulation (S1590).

A detailed description of each of the above-described steps S1500 to S1590 has been described with reference to FIG. 5, and thus a repeated description will be omitted.

9 is a detailed flowchart illustrating a method of selecting one external device when there are two or more external devices to be manipulated based on a result of operation determination according to the present invention.

Referring to FIG. 9, when the operation determination result is that a plurality of external devices can be manipulated, one external device may be manipulated according to a set mode.

According to the operation determination result, it may be determined whether there are two or more devices to which power is applied among the external devices that can be manipulated (S1610).

In general, when a user moves his or her hand, it can be determined that he intends to manipulate one external device. Therefore, the operation is performed by distinguishing whether there is one external device or two or more external devices.

In this case, when there is only one external device that can be operated, the external device may be selected as an external device to be operated (S1620).

When there are two or more external devices that can be manipulated, it is determined whether power is applied to the external device (S1630), and one of them must be selected.

This is because the device to which the power is not applied cannot perform the operation, and the intention of the user may be determined to attempt to operate the device to which the power is currently applied.

In this case, when power is applied to only one external device, the external device to which the power is applied may be selected as the manipulation target device (S1640).

When there are two or more external devices to which power is applied, one external device may be selected as the manipulation target device by checking a preset mode (S1650). The mode can be set in advance and can be changed later according to the movement of the user's hand.

Modes that can be set for selecting an external device include a distance priority mode, a hand direction priority mode, and a gaze priority mode.

When the set mode is the distance priority mode, an external device located close to the user may be selected as the manipulation target device (S1661).

When the set mode is the hand direction priority mode, an external device located in the direction indicated by the user's hand may be selected as the manipulation target device (S1663).

When the set mode is the gaze priority mode, an external device located in the user's gaze direction may be selected as the manipulation target device (S1663).

Although the above has been described with reference to embodiments, those skilled in the art will understand that various modifications and changes can be made to the present invention without departing from the spirit and scope of the present invention described in the following claims. I will be able to.

A detailed description of each of the above-described steps S1610 to S1665 has been described with reference to FIG. 6, and thus a repeated description will be omitted.

10: motion recognition based manipulation device
100: video input unit
200: user detection unit
300: motion recognition unit
400: operation determination unit
500: control panel

Claims (15)

An image input unit for photographing an object;
A user detection unit that recognizes whether an object photographed by the image input unit is a user using a learning network model and detects an area in which the user is located;
A motion recognition unit that calculates a change amount by comparing a frame at a specific time point in a region where the user is located detected by the user detection unit and a frame after a threshold time has passed from the specific time point, and recognizes the change amount as a user's motion when the change amount is greater than a threshold value;
A motion determination unit that recognizes what kind of motion is the user's motion using a learning network model and generates a motion determination result corresponding to the recognized motion; And
Including; an operation unit for operating at least one external device according to the operation determination result of the operation determination unit,
When a plurality of users are recognized, the user detection unit detects all areas in which the user is located,
If there are a plurality of areas in which the detected user is located, the motion recognition unit recognizes the movements of the plurality of users in each area,
When the motion of the plurality of users is recognized, the motion determination unit recognizes what kind of motion is the user's motion for each, and generates a plurality of motion determination results in response to each recognized motion,
If the operation determination result is a plurality, the operation of the external device according to each operation determination result, but if the plurality of operation determination results are operating the same external device, compares the respective operation determination results Check whether the operation is the same or contradictory operation, and if the operation is performed the same, operate the external device according to the result of determining the operation of all of the plurality of users in order according to the distance between the external device and the plurality of users, and perform the opposite operation. If it is, a motion recognition-based manipulation device that manipulates an external device according to a result of determining a motion of a user located at the nearest location according to a distance between the external device and a plurality of users.
delete delete delete delete The method of claim 1, wherein the operation unit,
If the plurality of operation determination results are to operate separate external devices, respectively, the operation recognition-based manipulation device is configured to operate the external device according to each operation determination result.
The method of claim 1, wherein the operation unit,
If the plurality of operation determination results are intended to operate the same external device, and the plurality of operation determination results are to perform the same operation,
A motion recognition-based manipulation device for manipulating an external device according to a result of motion determination of all of a plurality of users in an order of a user located near the external device.
delete The method of claim 1, wherein the operation unit,
If the result of the operation determination is that a plurality of external devices can be manipulated, and when power is applied to the plurality of external devices, one external device is operated according to a set mode.
The method of claim 9, wherein the operation unit,
When the set mode is a distance priority mode, the operation recognition-based manipulation device operates an external device located close to a user.
The method of claim 9, wherein the operation unit,
When the set mode is a gaze priority mode, a gesture recognition-based manipulation device for manipulating an external device located in a user's gaze direction.
The method of claim 9, wherein the operation unit,
When the set mode is a hand direction priority mode, a gesture recognition-based manipulation device for manipulating an external device positioned in a direction indicated by a user's hand.
In the motion recognition-based manipulation method performed by the motion recognition-based manipulation device,
Photographing an object by the motion recognition-based manipulation device;
Recognizing, by the motion recognition-based manipulation device, whether the captured object is a user using a learning network model, and detecting an area in which the user is located;
Calculating a change amount by comparing, by the motion recognition-based manipulation device, a frame at a specific point in time in a region in which the user is located and a frame after a threshold time, and recognizing that the change amount is greater than a threshold value as a user's movement;
Recognizing, by the motion recognition-based manipulation device, what kind of motion is the user's motion using a learning network model, and generating a motion determination result corresponding to the recognized motion;
Including, the operation recognition-based operation device manipulating an external device according to the operation determination result;
The step of detecting the area in which the user is located,
When the motion recognition-based manipulation device recognizes a plurality of users, it detects all areas where the user is located
Recognizing as the user's movement,
If there are a plurality of areas in which the detected user is located, movements of the plurality of users are recognized in each area,
Generating an operation determination result corresponding to the recognized operation,
When the motion recognition-based manipulation device recognizes the motions of the plurality of users, it recognizes what kind of motion the user's motion is for each, and generates a plurality of motion determination results in response to each recognized motion,
The step of operating the external device,
If the motion recognition-based manipulation device has a plurality of motion determination results, the external device is operated according to each motion determination result, but if the plurality of motion determination results are the same external device, the respective motion determination results If the operation is the same or contrary to each other, and if the same operation is performed, the external device is operated according to the result of the operation determination of all of the plurality of users in order according to the distance between the external device and the plurality of users, If the operation is performed, the operation recognition-based operation method for manipulating the external device according to the result of the operation determination of the user located at the nearest location according to the distance between the external device and a plurality of users.
The method of claim 13, wherein the step of operating the external device,
When the result of determining the motion generated at the same time is two or more, selecting one or more results of the motion determination;
The method of claim 13, wherein the step of operating the external device,
The motion recognition-based manipulation device further comprises: selecting one external device when there are two or more external devices to be manipulated based on the result of the motion determination.

Images