KR100692526B1 - Gesture recognition apparatus and methods for automatic control of systems - Google Patents

Abstract

An apparatus and a method for recognizing a gesture in automatically controlling a system are provided to extract motion from a continuous image frame of a moving object and detect a correct shape of a user by using image feature information, thereby tracking the user and recognizing a gesture of the user in real time. An image input unit(100) obtains a gesture image of a user from a camera attached to a robot. A shape detection unit(200) extracts a motion candidate region from a sequence of the gesture image by using image difference between neighbored image frames. Based on the motion candidate region and image feature information of the gesture image, the shape detection unit detects a human shape of the user. A gesture recognition unit(300) extracts a gesture interval of the user from the gesture image and thereby recognizes a gesture of the user. A system/robot automatic control unit(400) automatically controls a corresponding system or operation of the robot in response to the recognized gesture of the user.

Description

GESTURE RECOGNITION APPARATUS AND METHODS FOR AUTOMATIC CONTROL OF SYSTEMS}

1 is an overall configuration diagram of a gesture recognition apparatus for automatic system control according to an embodiment of the present invention;

2 is a flowchart illustrating that a gesture recognition process and a result of recognition are utilized for robot motion control or system automatic control in a gesture recognition apparatus for system automatic control according to an embodiment of the present invention;

Figure 3a shows an example of the environment for automatic robot control, 3b is to bring the robot to the user with a call gesture, 3c is to make a gesture to stop the movement of the robot in front of a certain distance between the user and the robot Degrees representing the environment of the time,

FIG. 4 detects a user's motion information (b) and a skin color region (c) from an original image (a) at a t- th frame in order to detect a user in a system control gesture recognition apparatus according to an embodiment of the present invention. (a) shows the tracking of the movement of the user's face area and the hand (d),

FIG. 5 is a diagram illustrating an example used for selection (a) and button control (b) of a system to be controlled using a gesture recognition result.

6 is a diagram illustrating extracting a structural feature vector (b) by tracking (a) movement coordinates of a face and a hand for a gesture section of a user for gesture recognition.

<Explanation of symbols for the main parts of the drawings>

100. Image input unit 200. Shape detector unit

210. Motion region extractor 220. Image feature information extractor

230. Shape extraction unit 300. Gesture recognition unit

310. Face area extraction unit 320. Hand area extraction unit

330. Gesture feature extraction and recognition unit 400. System and robot motion automatic control unit

410. System selector 420. Button control

430. Robot operation automatic control unit 440. System automatic control unit

The present invention relates to a gesture recognition apparatus and method for automatic system control, and more particularly, to a gesture recognition apparatus and method for automatically controlling a system by extracting a gesture from an input image obtained from a camera attached to a robot. It is about.

Conventionally, gesture recognition has been studied for the purpose of natural computer interaction and robot control. Recently, in the field of virtual reality, a research is being conducted to move a virtual image in an arbitrary direction on a PC by extracting and recognizing three-dimensional gesture data.

Gesture recognition for the purpose of natural computer interaction is made by attaching a camera near the monitor and receiving and recognizing user's gesture so that it can be used as an input device such as a mouse or keyboard. That is, the research has been conducted to implement an automatic interface function that enables a user to automatically input keyboard keys without directly hitting the keyboard keys by recognizing a consonant and a vowel or a hand of the alphabet. In addition, researches on gesture recognition that can automatically control a document such as flipping back and forth or moving to a specific page without using a keyboard or other tools have been conducted.

Therefore, the direction of the camera and the distance between the hand and the camera are very important for gesture recognition to precisely recognize gestures and automatically move the computer keyboard or extract hand direction information to automatically control the turning of documents such as presentation materials. Do.

However, the user gesture recognition method is mostly installed in the camera used in gesture recognition for computer interaction, such as the ceiling or the wall in a specific space, the camera is installed and the user moves to the specific space to perform the gesture input Received and recognized. In this case, a lot of errors occur in extracting accurate hand movement information or hand shape information, and thus there is a problem that the gesture recognition result is not utilized for computer interaction purposes.

Accordingly, an object of the present invention is to solve the above problems of the prior art, the object of the present invention is to acquire and recognize the gesture image of the user with a camera attached to the mobile robot regardless of any position to control The present invention provides a gesture recognition apparatus and method for automatically controlling a system that can automatically control a system by selecting a specific system.

Meanwhile, another object of the present invention is to extract the correct user shape by performing the motion extraction and the human shape extraction using the associative feature information from the moving objects of the continuous frame in order to recognize the user's gesture, and to face the user's call The present invention provides a gesture recognition apparatus and method for automatic control of a system that can naturally perform a specific user tracking and gesture recognition by utilizing detection and tracking and gesture recognition results together.

On the other hand, another object of the present invention is to move the robot to the user's space for automatic control of a specific system to recognize the gesture to automatically execute the system selection button and control button, as well as the user's gestures and actions The present invention provides a gesture recognition apparatus and method for automatically controlling a system that can automatically play and stop a system.

In order to achieve the above object, a gesture recognition apparatus for automatic system control according to the present invention includes an image input unit for obtaining and transferring a gesture image of a user to be recognized from a camera attached to a robot, and a sequence of images transmitted from the image input unit. A shape detector unit for extracting a shape of a person with respect to a target user based on the motion candidate region extracted from the image difference between neighboring frames from the image and the image characteristic information of the transmitted image, that is, an outline, a skin color region, etc. And a gesture recognition unit for extracting and recognizing a user's gesture section from a gesture image of the user and a system for automatically controlling a system and a robot operation according to a result of the recognition of the user's gesture, and a robot operation automatic control unit. Gestures from images acquired from cameras attached to robots It characterized in that the extraction and recognition to automatically control the system.

On the other hand, the gesture recognition method for automatic control of the system of the present invention for achieving the above object, as a method for detecting a gesture by detecting the target user from the input image obtained from the camera attached to the robot, and then controls a specific system, Acquiring an input image through a camera mounted on the robot, detecting a user from the obtained input image, extracting a movement of the detected user's hand, and a gesture by the extracted hand movement Determining whether it is a gesture for controlling or a gesture for automatic system control, and when the determination result is a gesture for controlling the robot, controlling to approach the user until a predetermined distance between the user and the robot is maintained. And recognizing a gesture to stop the user's robot movement. Surface, characterized in that it comprises a step of controlling the robot to stop in front of the user.

On the other hand, the gesture recognition method for system automatic control according to the present invention, in the gesture determination step, when the gesture is for automatically controlling the system, the system automatic control may include displaying a button for system selection, and when the user gesture is recognized. And controlling the selected button according to the user's gesture to play the system, and stopping the play of the selected system when the user makes a play stop gesture of the system or leaves the seat for a predetermined time or more. Characterized in that made.

Meanwhile, in the gesture recognition method for automatic system control of the present invention, in the step of detecting a user from the obtained input image and extracting the movement of the detected user's hand, the user detection is predetermined in the obtained input image. Checking whether there is a motion region greater than or equal to a threshold value; if there is a motion region greater than or equal to the predetermined threshold value, extracting a motion candidate region from an image frame and a previous input image frame; image characteristic information from the motion candidate region; Detecting the silhouette shape of the person by extracting the extract; and detecting the user by extracting a face region of the user from the detected silhouette shape of the person.

Meanwhile, in the gesture recognition method for automatic system control of the present invention, in the step of detecting a user from the obtained input image and extracting the movement of the detected user's hand, the movement of the user's hand is extracted. And extracting a hand region by inversely estimating a skin color region in a face region of the face region and detecting a gesture section by tracking a movement of a hand from the extracted hand region.

Hereinafter, a gesture recognition apparatus and method for controlling a system of the present invention will be described in detail with reference to the accompanying drawings.

1 is a block diagram of a system control gesture recognition apparatus according to an embodiment of the present invention.

As shown in FIG. 1, a gesture recognition apparatus for controlling a system according to an exemplary embodiment of the present invention includes an image input unit 100, a shape detector unit 200, a gesture recognition unit 300, and an automatic system and robot motion controller 400. It includes.

In detail, the image input unit 100 includes, for example, a camera attached to the robot, and acquires a gesture image of a user to be detected and transmits the gesture image to the system.

The shape detector 200 includes a motion region extractor 210, an image feature information extractor 220, and a shape extractor 230, and the motion region extractor 210 is transferred from the image input unit 100. The user's motion information is extracted using the image difference between neighboring frames from the sequence of consecutive images. That is, when the difference in brightness value of the pixels between three frames ( t-1, t, t + 1 ) , which are consecutive images from the image sequence , is larger than the threshold value, the motion is regarded as a pixel (hereinafter referred to as a motion pixel) in which the motion occurs. Extract the information.

The image feature information extractor 220 extracts image feature information such as edge, color, and shape information when the motion pixels obtained above are the user's motion region.

The shape extractor 230 extracts the shape of the person using the silhouette information of the person and skin color information of the person obtained by obtaining a difference between the background image and the current image from the image feature information extracted by the image feature information extractor 220. .

The gesture recognizer 300 includes a face region extractor 310, a hand region extractor 320, and a gesture feature extractor and recognizer 330, and the face region extractor 310 is a structural feature of a human shape. The face region is detected using the information, and the hand region extractor 320 extracts the hand region by extracting the skin color region and the movement region from the extracted face region. The gesture section is detected by continuously tracking the user's hand region.

The gesture feature extraction and recognition unit 330 uses a feature extraction method to recognize a user's gesture regardless of the direction, temporality, or spatiality of the gesture. Extract the user's gestures. In addition, the system and the robot motion automatic control unit 400 may control the movement of the robot or automatically control a specific system according to the recognition result of the user gesture.

The system and robot automatic controller 400 includes a system selector 410 for automatically controlling the system, a system automatic controller 440 composed of a button controller 420, and an automatic robot motion controller 430. When the robot recognizes a gesture for moving the robot toward the user, the robot motion controller 430 first identifies a user who makes a gesture to call the robot to move the robot toward the user and controls the system to control the movement of the robot. When the distance of the hand gesture for the hand enough to recognize the movement of the robot to control.

 When the robot stops operation when the distance between the robot and the user is maintained as the distance for recognizing the gesture by the control of the gesture feature extraction and recognition unit 330, the system selector 410 may control the specific system. A button is displayed to allow a user to select a system, and a specific system to be controlled is selected according to a result of the user gesture recognition. The button controller 420 controls the selected button that can automatically control the system according to the user's gesture to play the system. That is, the system and the robot motion automatic control unit 400 provide a function of automatically playing and stopping the system by analyzing a user's gesture recognition result or user's behavior.

In addition, the system and the robot motion controller 400 may extract the gesture data from the continuous image sequence, extract the feature, recognize the feature, and then use the interface to automatically control a specific system.

2 is a flowchart illustrating a gesture recognition process in a system control gesture recognition apparatus according to an exemplary embodiment of the present invention.

When the user takes a call gesture to focus the attention of the robot (S1), the robot acquires a gesture image of the user as an input image through a camera attached as the image input unit 100 (S2).

If there is a motion region having a predetermined threshold or more in the input image acquired from the image input unit 100 (S3), and if a motion region having a predetermined threshold or more exists, the motion region extractor 210 moves from the continuous input image frame. The candidate region is extracted and image feature information is extracted from the input image (S4). A shape such as a silhouette of a person is detected using the extracted motion candidate region and image feature information (S5).

A user's face region is extracted from the extracted silhouette shape of the person (S6). Then, the skin region is estimated by inversely extracting the skin region from the extracted user's face region, the user's hand is tracked from the extracted hand region, and a gesture section is detected (S7).

It is determined whether the detected gesture is a robot control gesture (S8), and if the robot control gesture, the movement of the robot is controlled. When the user makes a gesture for a robot call, the movement control of the robot recognizes the robot and approaches the user until the distance between the user and the robot is maintained (S9). Then, when the user makes a gesture to stop the robot movement, the robot is controlled to stop in front of the user (S10).

If it is not the robot control gesture, it is determined whether the gesture is for automatic system control (S11). If the determination result is a gesture for automatic system control, the robot displays a button for system selection (S12), and controls the system selection button according to a result of recognizing a user's gesture (S13) to play the system (S14). Then keep track of your hand area. When the user makes a play stop gesture of the selected system or when the user is away for a predetermined time or more, the play of the selected system is indicated (S14).

3 shows an environment for detecting a user through automatic robot control. FIG. 3A illustrates an environment for automatic robot control, in which three people are lined up to perform a robot, and FIG. 3B illustrates moving a robot by a call gesture. The user calls a robot by hand gestures (call gestures). This indicates that the robot approaches the user while tracking the face area of the user. 3C illustrates an environment when a user makes a gesture of stopping a robot's movement in front of a predetermined distance between the user and the robot. When the user makes a stop gesture, the robot stops.

FIG. 4 detects a user's face area b from an original image (a) of a t- th frame and detects a user's face from the original image (a) in order to detect the user in the system control gesture recognition apparatus according to an exemplary embodiment of the present invention. After detecting the motion information (c) and the skin color region (d), it is a diagram showing the detection and tracking of the movement of the user's hand in the original image (a).

FIG. 5 is a diagram illustrating an example of use of selection (a) and button control (b) of a system to be controlled using a gesture recognition result. FIG. 5A illustrates selecting a DVD from among selectable buttons, and FIG. 5B displays buttons corresponding to sub-items after the DVD is selected, and shows that List 3 is selected.

6 is a diagram illustrating extracting a structural feature vector of a gesture from a hand coordinate extracted in a gesture section. The gesture feature extraction and recognition unit 330 extracts and recognizes a structural feature pattern that enables the user to recognize a gesture regardless of the direction, spatiality, and temporality of the gesture (a).

In the above description, the mobile robot equipped with a camera extracts a human shape from a moving person, tracks a hand, and then recognizes a gesture to control the system. The present invention is not limited to the embodiments, and various modifications can be made without departing from the spirit of the present invention.

As described above, the gesture recognition apparatus and method for automatic system control according to the present invention had to approach near a camera fixed at a specific position and perform a gesture only in the region and recognize the gesture. There is an effect that can operate the specific system by acquiring the user's gesture through the camera mounted on the robot.

In addition, the gesture recognition apparatus and system for automatic system control according to the present invention, the user does not approach the specific location or system to receive the service, but the approach to the robot that can provide the service to the service that the user wants, namely By enabling services such as system automatic control by gesture recognition, it is possible to replace the remote control function, which is necessary for controlling the system, to solve the inconvenience of using the remote control, thereby providing convenience to the user.

Claims (11)

An image input unit 100 which acquires and transmits a gesture image of a user from a camera attached to the robot; Shape detector unit for extracting the shape of the person to the user by using the motion candidate region extracted through the image difference between neighboring frames from the sequence of the image transmitted from the image input unit 100 and the image characteristic information of the transmitted image 200; A gesture recognition unit 300 extracting and recognizing a gesture section of the user from the gesture image of the user; And A system for automatically controlling the system by automatically recognizing the gesture of the user or automatically controlling the operation of the robot and a robot motion controller 400; Gesture recognition device for automatic system control, characterized in that comprising a. The method of claim 1, wherein the shape detector 200, A motion region extraction unit 210 for extracting a user's motion region by using an image difference between neighboring frames from a sequence of consecutive images transmitted from the image input unit 100; An image feature information extracting unit 220 for extracting image feature information from the motion region of the user; And A shape extraction unit 230 for extracting a shape of a person from the extracted image feature information; Gesture recognition device for automatic system control, characterized in that comprising a. The method of claim 1 or 2, wherein the gesture recognition unit 300, A face region extracting unit 310 for detecting a face region using structural feature information from the shape of the person; A hand region extraction unit 320 for detecting a hand region by estimating a skin color region and a movement region from the extracted face region; And A gesture feature extraction and recognition unit 330 which detects a gesture region of the user from the detected hand region and extracts a gesture feature and recognizes a gesture of the user using the extracted gesture feature; Gesture recognition device for automatic system control, characterized in that comprising a. According to claim 3, The system and robot operation automatic control unit 400, A system automatic controller 440 including a system selector 410 for displaying a button for selecting a specific system and a button controller 420 for automatically controlling a system by controlling a selected button according to a gesture of the user; And Robot operation automatic control unit 430 for automatically controlling the operation of the robot by the user's gesture; Gesture recognition device for automatic system control, characterized in that comprising a. The method of claim 2, wherein the movement area of the user in the movement area extractor 210, Recognizing gestures for automatic control of a system, when the difference in brightness of pixels among consecutive three frames of images from the image sequence acquired through the image input unit 100 is greater than a threshold, it is regarded as a pixel in which movement occurs and extracted into a motion region. Device. The method of claim 4, wherein the robot motion control of the robot motion control unit 430, When the user makes a gesture of calling the robot, the robot moves the robot toward the user and stops the movement of the robot when the user approaches the distance enough to recognize the gesture of the hand for controlling the system. Recognition device. According to claim 3, Gesture recognition in the gesture feature extraction and recognition unit 330, And a structural feature pattern for recognizing a user's gesture regardless of the directionality, spatiality, and temporality of the gesture made by the user. In a method for recognizing a gesture of a user, Obtaining an input image through a camera mounted on the robot; Detecting a user from the obtained input image and extracting a movement of the detected user's hand; Determining whether the gesture by the movement of the extracted hand is a gesture for controlling the robot or a system automatic control; If the result of the determination is a gesture for controlling the robot, controlling the user to approach the user until a predetermined distance between the user and the robot is maintained; And Recognizing a gesture to stop the robot movement of the user, controlling the robot to stop in front of the user; Gesture recognition method for automatic system control, characterized in that made. The method of claim 8, wherein the detecting of the user from the acquired input image and extracting the movement of the detected user's hand comprises: detecting the user; Checking whether a motion area greater than or equal to a threshold exists in the obtained input image; Extracting a motion candidate area from consecutive image frames when there is a motion area above the threshold as a result of the check; Extracting image feature information from the motion candidate region to detect a silhouette shape of a person; And Detecting the user by extracting a face region of the user from the detected silhouette shape of the person; The gesture recognition method for system automatic control, characterized in that further comprises. The method of claim 9, wherein the movement of the user's hand is extracted. Extracting a hand region by inversely estimating a skin color region in the extracted face region of the user; And Detecting a gesture section by tracking a hand movement from the extracted hand region; The gesture recognition method for system automatic control, characterized in that further comprises. The method of claim 8, wherein in the gesture determination step, when the gesture is for automatically controlling the system, the system automatic control is performed. Displaying a button for system selection; If the gesture of the user is recognized, controlling the selected button according to the gesture of the user to play the system; And Stopping the play of the selected system when the user makes a play stop gesture of the system or is away for a predetermined time; A gesture recognition method for system automatic control using a robot, characterized in that further comprises.

Images