KR20110125524A - System for object learning through multi-modal interaction and method thereof - Google Patents

Abstract

PURPOSE: An object learning system of a robot using multi-modal interaction and method thereof are provided to enable a user to teach object information to the robot by enabling a user to communicate with the robot through the multi-modal interaction. CONSTITUTION: An object extraction means(122) extracts a video from objects. An interaction unit performs interaction with a user. A robot learns the object using object information in which is collected from the interaction means and the object information in which is extracted from the object extraction means. A gesture recognition means(120) recognizes the gesture of a user in the video. The robot learns the object using the object information collected from the gesture recognition means and the interaction means.

Description

System for object learning through multi-modal interaction and method

The present invention relates to a system and method for a robot to learn objects in an environment through multimodal interaction with a user. More specifically, the robot recognizes the user's gesture using the image input device, finds the object to be registered by the user in the environment, and learns the object to store in the object database and the environment map, and its It is about a method.

Research is being actively conducted for robots to learn and store objects in the environment. Gestures for the object learning of robots are mainly using hands or fingers, and Kahn and three others pointed at the object to register by hand (Roger E. Kahn, Michael J. Swain, Peter N. Prokopowicz, and R). James Firby, “Gesture Recognition Using the Perseus Architecture,” Proceedings of IEEE Computer Society Conference on Computer Vision and Pattern Recognition, pp. 734-741, 1996.), Roth and two others picked up objects by hand (PM). Roth, M. Donoser, H. Bischof, “On-line Learning of Unknown Hand Held Objects via Tracking,” Proceedings of Internationl Conference on Computer Vision Systems, 2006.), Arsenio published a study of waving hands or fingers in front of objects. (Artur M. Arsenio, “Figure / Ground Segregation from Human Cues,” Proceedings of IEEE / RSJ International Conference on Intelligent Robots and Systems, Vol. 4, pp. 3244-3250, 2004.)

In addition, Korean Patent Application Publication No. 10-2005-0029161 (published: March 24, 2005) discloses an object recognition system using a spatial interaction apparatus and a method thereof.

The present invention, SIDOR module (Spatial Interaction Device Object Recognition Module) capable of object recognition by acquiring a plurality of images according to the appearance of the object by the user interaction; An image input unit for inputting a plurality of images obtained from the SIDOR module into a computer; An image stitching unit which connects a plurality of input images to form a stitch image and deletes an unknown background of the stitch; A template image storage unit for storing a Hue Histogram of a plurality of template images to be compared with a stitch image; And an image comparison unit for performing object recognition by comparing the stitch image and the template image to each other. According to the above disclosure, by directly pointing to an object to be recognized by the user and stitching the image by the user's interaction, the amount of computation of the object recognition of the computer is reduced since the user does not need time to find the object to be recognized from the beginning of the image. The recognition speed will be faster.

However, the published research results assume that the object learning method through the interaction between the user and the robot assumes that the robot is always looking at the user, and to learn only an object of a specific size without deciding a gesture according to the size of the object. It has a system limitation.

In addition, in the prior art, there is a problem in that an object of interest to the user is indicated in space through a space mouse which is a separate space interaction device.

Accordingly, there is a need for an invention relating to a system and a method for a robot to recognize a person's face and hands and automatically find and learn an object of interest in space through a human gesture.

Publication No. 10-2005-0029161

Roger E. Kahn, Michael J. Swain, Peter N. Prokopowicz, and R. James Firby, “Gesture Recognition Using the Perseus Architecture,” Proceedings of IEEE Computer Society Conference on Computer Vision and Pattern Recognition, pp. 734-741, 1996. P. M. Roth, M. Donoser, H. Bischof, “On-line Learning of Unknown Hand Held Objects via Tracking,” Proceedings of Internationl Conference on Computer Vision Systems, 2006. Artur M. Arsenio, “Figure / Ground Segregation from Human Cues,” Proceedings of IEEE / RSJ International Conference on Intelligent Robots and Systems, Vol. 4, pp. 3244-3250, 2004.

The present invention is to solve the problems of the prior art, an object of the present invention, the robot communicates with the robot through the multi-modal interaction in order for the robot to learn and store the object in the environment, the object in the environment To provide a system and method for learning the robot.

Another object of the present invention is to provide a system and method for easily and simply learning robots of various sizes by determining different gestures according to the size (large / medium / small) of the object to be learned.

Another object of the present invention is to not only inform the robot in advance of the size of the object to be learned by the user, but also to report the gesture of the user to automatically estimate the size of the object and perform the learning accordingly. It is to provide a system and a method thereof.

As a technical solution for achieving the object of the present invention, in the present invention, the speaker / voice recognition unit for recognizing the direction of the speaker, the user, the voice command through a sound input device such as a microphone, and a sound output device such as a speaker Voice recognition unit for voice conversation through the user, User recognition and tracking unit for recognizing and tracking the user's face / hand through the robot head rotation device such as pan / tilt unit and the image input device such as camera, and user gesture A gesture recognition unit for estimating the position of the learning object in space by recognizing the symbol, an object detecting unit for dividing only the object more accurately in the part where the object is estimated to be located in the image, and information such as a database by learning the detected object. Object learning unit to store in the storage device and the environment map, and through the parts through a dialogue with the user And it is through the image output device such as a monitor object, the present learning system and method of a robot with a multi-modal interaction with the user, including a multi-modal control unit for feeding back the results to the user.

In addition, in the present invention, the user gives an object learning command to the robot through voice using a voice and video signal for multi-modal interaction, the robot interprets the command and performs the command, and the robot acquires the information of the object through dialogue. And the object learning system and method of the robot through a multi-modal interaction with the user of the configuration receiving the object learning results are presented.

In addition, the present invention automatically recognizes the user's gesture through the image signal to automatically find the object to be registered by the user in the environment, each gesture separately indicating the object according to the size (large / medium / small) of the object It can be configured to efficiently find objects in the environment, and the size of the object can be directly taught by the user during the conversation with the robot. An object learning system and method for a robot through multimodal interaction with a user configured to automatically recognize whether a gesture is being taken is provided.

The present invention is an alternative to the existing method for the user to take a picture of the object in the environment by hand manually to learn the object to the robot, cut or display only the portion corresponding to the object to learn the robot and store it in the environment map. The modal interaction communicates with the robot, making it easier for the robot to learn objects in the environment.

1 is a schematic block diagram of an embodiment of an object learning system through multimodal interaction between a user and a robot of the present invention.
FIG. 2 is a schematic diagram of a finite state machine existing inside a multi-modal control unit, which is a main part of an embodiment of the present invention.
3 is a flowchart illustrating an embodiment of an object learning method through multimodal interaction between a user and a robot of the present invention.
FIG. 4 is an explanatory diagram for a gesture method according to an object size (large / medium / small) of the user in the embodiment of FIG. 3.
5 is an exemplary view illustrating a gesture method according to the size (large / medium / small) of the object of FIG. 4.
6 is a flowchart illustrating a process of automatically estimating the size (large / medium / small) of an object by a gesture recognition unit in an embodiment of the present invention.

Hereinafter, embodiments of the present invention will be described in detail with reference to the accompanying drawings.

1 is a schematic diagram of an object learning system through a multimodal interaction between a user and a robot. As shown in FIG. 1, the present invention provides an audio input unit 100 for receiving a user's audio signal, an audio output unit 102 for outputting sound, and an image output unit 104 for outputting an image. ), An image capturing unit 106 for capturing an external image, a pan and tilt unit 108 for changing the direction of the image capturing unit 106, a storage unit 110 for storing information, and In the robot system comprising an environment map unit 112 for creating information about the surrounding environment,

A multi-modal control unit 114, a speaker and sound recognition unit 116 for recognizing a speaker's direction and sound input from the sound input unit 100 and transmitting the sound to the multi-modal control unit 114, and the multi-modal control unit Receives a control signal from the 114 and transmits a control signal to the pan and tilt unit 108 to change the direction of the image capturing unit 106 to the user's direction, recognizes the user's face and hand, the user's face And a user recognition and tracking unit 118 for tracking the position of the hand and the hand, and input from the image photographing unit 106 based on the position information of the user's face and hand received from the user recognition and tracking unit 118. A gesture recognition unit 120 for recognizing a user's gesture by receiving the image, an object detection unit 122 for extracting an object from the input image based on the recognition information of the gesture recognition unit 120, The water An object learning unit for learning an object from the object image extracted by the detection unit 122 and storing the object in the storage unit 110, and creating the environment map based on the learned contents and storing the object in the environment guidance unit 112. 124 and a voice expression unit 126 for outputting the voice according to the control signal of the multi-modal control unit 114 to the sound output unit 102.

The speaker and voice recognition unit 116 receives a user's voice command from the sound input unit 100 such as a microphone, recognizes in which direction the user is, and helps the user to draw attention to the robot and focus on the user. It allows you to recognize who you are and what your voice commands are. The direction of the user, the user information, and the voice command recognized by the speaker and the voice recognition unit 116 are transmitted to the multi-modal control unit 114.

The voice expression unit 126 notifies the user of the conversation contents transmitted from the multi-modal control unit 114 by using the sound output unit 102 such as a speaker, together with the speaker and the voice recognition unit 116. The interaction between the user and the robot can be made possible through dialogue.

The multi-modal control unit 114 serves as a central control for controlling and monitoring each component of the above-described object learning system of the present invention. That is, the multi-modal control unit 114 transmits the user direction received from the speaker and the voice recognition unit 116 to the user recognition and tracking unit 118 to control the pan and tilt unit 108 to control the robot. The head is rotated to draw attention to the user, and the voice command received from the user is interpreted to interpret and perform what the current user wants, and the voice expression unit 126 controls to induce a natural conversation with the user. At this time, the image output unit 104 such as a monitor may provide a variety of information to the user as well as the audio image. In addition, the multi-modal control unit 114 instructs the user recognition and tracking unit 118, the gesture recognition unit 120, the object detection unit 122, the object learning unit 124 and the like to start and end the execution. It interprets the error message transmitted from the configurations and controls to take appropriate action.

The user recognition and tracking unit 118 recognizes the user's face and hand in the image received from the image capturing unit 106 and continuously tracks each time the user moves the pan and tilt unit (pan / tilt unit) ( 108 can be used to track the user by continuously rotating the head along the user.

The gesture recognition unit 120 receives the three-dimensional position of the user's face and hand from the user recognition and tracking unit 118, interprets the user's gesture to grasp the user's intention, and the user learns the robot in the environment. Estimate the approximate location of the object you want to make. In this case, the approximate position of the object may be expressed as a region of interest such as a rectangle on the screen.

The object detector 122 is configured to automatically divide the learning object in more detail in the region where the learning object is estimated to exist in the screen. The divided learning object is transmitted to the multi-modal control unit 114 to be displayed on the image output unit 104 to be shown to the user and to check whether the object indicated by the user in the environment is correct.

The object learner 124 extracts and learns feature points, colors, shapes, etc., of the texture of the object part detected by the object detector 122. ) And store it in the storage unit 110 such as a database, and display it on the environment map in the environment map unit 112.

2 is a flowchart illustrating an object learning method of the present invention in a finite state machine included in the multi-modal control unit 114.

As shown in FIG. 2, in the object learning method of the present invention, the state of the multi-modal control unit 114 is transitioned by an event transmitted from respective parts connected to the multi-modal control unit 114. It is composed.

Waiting step (S100) which is the initial state of the multi-modal control unit 114 for starting the object learning through the multi-modal interaction of the user and the robot,

When the user gives a learning start command to the robot, the learning start signal is transmitted from the speaker and the voice recognition unit 116 to the multi-modal control unit 114 and is transitioned in the waiting step. Face and hand tracking step (S102) to transmit a start command signal of the recognition and tracking to the user recognition and tracking unit 118 in the robot to recognize and continuously track the user's face and hands;

 In the face and hand tracking step, if the user recognition and tracking unit 118 fails to recognize or track the user's face and hand, a failure signal is transmitted to the multi-modal control unit 114 to transition to an error state. do.

In the face and hand tracking step (S102), if the user gives an object registration start command to the robot while the robot pays attention to the user, a learning start signal is transmitted from the speaker and the voice recognition unit 116 to the multi-modal control unit 114. A gesture recognition step (S104) of recognizing a user's gesture in the gesture recognition unit 120 as a state of being transferred and transitioned in the face and hand tracking step (S102);

In this case, when the gesture recognition unit 120 fails to recognize the gesture of the user, a failure signal is transmitted to the multi-modal control unit 114 to transition to an error state.

The object detection state S106 is the gesture recognition when the gesture recognition unit 120 recognizes the user's gesture successfully in the gesture recognition step S104 and the estimation of the approximate location of the learning object in the environment is successful. An object detection step (S106) of detecting the object by dividing an object existing in the image by the object detecting unit 122 as a state transitioned in the step (S104);

In this case, when the object detector 122 fails to divide the learning object in the image or incorrectly divides the learning object in the image, a failure signal is transmitted to the multi-modal control unit 114 to transition to an error state.

If the object detection step (S106) successfully performs the object detection in the state transition to the object detection step (S106), the object learning step 124 includes an object learning step (S108) for performing the object learning; have.

When the object learning is successful in the object learning step (S108), the process returns to the first waiting step (S100) and transitions to the standby state until the user commands the learning of another object.

The error state in each of the above-described steps of the present invention is reached when the face and hand tracking fails, the gesture recognition fails, or the object detection fails. After transmitting to the user by using the) and the appropriate error processing is transferred to the first waiting step (S100).

3 is a flowchart illustrating an example of object learning through multimodal interaction between a user and a robot according to an embodiment of the object learning method of the present invention.

When the user commands the object learning to "learn a new object", the speaker and voice recognition unit 116 estimates the direction of the speaker and interprets the command and transmits the command to the multi-modal control unit 114 (S200). The multi-modal control unit 114 commands a user attention to the user recognition and tracking unit 118 (S202). According to the command signal of the multi-modal control unit 114, the user recognition and tracking unit 118 recognizes the speaker's face and hand (S204). After the user recognition and tracking unit 118 recognizes the speaker's face and hand, tracking of the speaker's face and hand is started according to the control signal of the multi-modal control unit 114 (S206). At this time, if the face and hand fails to recognize or track, the state is transferred to the error state.

The voice expression unit 126 asks the user, "What is the object name?" (S208). A voice signal of "This is XXX" is received from the user through the speaker and the voice recognition unit 116 (S210). The voice expression unit 126 asks the user, "How big is the object?" (S212). A voice signal of “large (medium or small) object” is received from the user through the speaker and the voice recognition unit 116 (S214). Through the voice expression unit 126, the answer is "Yes ready" (S216). At this time, the user can directly ask the user of the size of the object as described above, but as described below, the user's gesture may be used to automatically determine how large an object is currently registered among the large / medium / small object.

While the robot continuously tracks the user's face and hand, the user receives a voice signal "register this object" from the user through the speaker and voice recognition unit 116 (S218). The multi-modal control unit 114 commands a gesture recognition to the gesture recognition unit 120 (S220). If the gesture recognition fails, the state transitions to an error state. Otherwise, if the gesture recognition is successful (S222), the multi-modal control unit 114 instructs the object detector 122 to extract a learning object from the image (S224). ). The object detection unit 122 completes the object extraction (S226). The extracted object image is output to the image output unit 104, and the question is “is this object correct” through the voice expression unit 126 (S228). The speaker and voice recognition unit 116 receives a voice signal of the user "yes, the object is correct" (S230). If the user answers "no", object extraction fails and the state transitions to an error state.

The multi-modal control unit 114 commands the object learning unit 124 to learn the object (S232). The object learning unit 124 completes the object learning (S234). When the object learning is completed, the initial state is in the standby state, and the voice expression unit 126, the answer to the "learned successfully." After completing all the learning process (S236).

4 shows an example of a gesture method according to the object size (large / medium / small).

Small objects such as books, pencil sharpeners, table clocks, mugs, and dolls are small and light and can be placed on hands or picked up by hand to tell the robot what to learn.

Medium objects such as wall clocks, picture frames, TVs, and dishwashers are heavy and can't be lifted or reached out of reach.

Large objects such as desks, tables, sofas, beds, and refrigerators are so large that they don't appear on one screen, so the robot needs to look at them by turning their heads.

5 illustrates an example of a gesture method according to the size (large / medium / small) of an object according to the embodiment of the present invention described above.

In the case of a small object, the face and hand appear on one screen, so the number of screens for taking an object is one.

In the case of a medium object, the face and hand appear on one screen, but the pointing object may appear on another screen, so the number of screens required for taking an object is one or two.

In the case of large objects, multiple images are required because the object must be photographed at both ends while moving along the person.

6 is a flowchart illustrating a process of automatically estimating the size (large / medium / small) of an object in the gesture recognition unit 120.

When the distance between the face and the hand is compared by the position of the user's face and the hand received from the user recognition and tracking unit 118 (S300), if the distance is smaller than a predetermined distance, the user places a small object in the hand. The small object area estimation (S302) is performed by determining that the robot is showing to the robot by lifting or picking up. In other words, the area above the hand or the hand is searched to determine a region including pixels located at a position similar to the position of the hand in the disparity map of the camera as an area where the object is expected to exist.

If the distance between the face and the hand is greater than a predetermined distance (S300), it is determined that the user is a medium or large object assuming that the user reaches out to point to the object or touch one end of the object.

The medium and large objects are again masked by determining whether or not the user moves after reaching the hand (S304). In other words, if the user extends his or her hand and does not move, the user recognizes that the object is a medium object, calculates a three-dimensional vector in the direction indicated by the human hand, and estimates an area in which the medium object may exist in space (S306).

If the user extends his hand and then moves his body to reach another point, it touches both ends of a large object, so project both ends to the floor in three-dimensional space to see the area that the four points point to. It is assumed that there is a large object space (S308).

100: sound input unit
102: sound output unit
114: multi-modal control unit
116: speaker and speech recognition unit
118: user recognition and tracking unit
120: gesture recognition unit
122: object detection unit
124: object learning unit
126: voice expression unit

Claims (12)

In the robot system that can recognize the received voice and video by receiving a voice signal and video of the speaker,
Object extracting means for extracting an object from the received image;
It includes an interaction means for interacting with the speaker,
Robot object learning system using a multi-modal interaction, characterized in that for learning the object using the object information extracted from the object extraction means and the object information collected from the interaction means. The method according to claim 1,
Further comprising a gesture recognition means for recognizing the gesture of the speaker in the received image,
Learning an object using a multi-modal interaction, characterized in that for learning the object using the object information extracted from the object extraction means, the object information collected from the interaction means and the object information collected from the gesture recognition means. system. The method according to claim 1,
The object information extracted from the object extraction means is shape information of the object,
Object information collected from the interaction means is the object learning system of the robot using multi-modal interaction, characterized in that the information and the name of the object size. The method according to claim 2,
The object information extracted from the object extraction means is shape information of the object,
The object information collected from the interaction means is name information of the object,
The object information collected from the gesture recognition means is a robot object learning system using multi-modal interaction, characterized in that the size information of the object. The method according to claim 1 or 2,
The interaction means is a robot object learning system using a multi-modal interaction, characterized in that the means for performing a dialogue with the speaker. The method according to claim 2,
The gesture recognition means is a robot object learning system using multi-modal interaction, characterized in that the means for recognizing the gesture of the speaker's face and hand. The robot recognizes the speaker's direction and voice;
Photographing an image by changing a direction of an image photographing unit of the robot to a speaker direction according to the recognized speaker direction and voice;
Collecting, by the robot, object information through interaction with the speaker;
Extracting an object from the image by the robot;
And learning by the robot based on object information collected through interaction with the speaker and object information extracted from the image. The robot recognizes the speaker's direction and voice;
Photographing an image by changing a direction of an image photographing unit of the robot to a speaker direction according to the recognized speaker direction and voice;
Collecting, by the robot, object information by recognizing a gesture of a speaker;
Collecting, by the robot, object information through interaction with the speaker;
Extracting an object from the image by the robot;
Learning based on object information collected from the gesture information, object information collected through interaction with the speaker, and object information extracted from the image; . The method according to claim 7,
The object information extracted from the image is shape information of the object,
The object information collected through the interaction with the speaker is a name and size information of the object characterized in that the object learning method of the robot using multi-modal interaction. The method according to claim 8,
The object information collected from the gesture information is the size information of the object,
The object information extracted from the image is shape information of the object,
The object information collected through the interaction with the speaker is the object information method of the robot, characterized in that the name information of the object. The method according to claim 8,
The robot may recognize the gesture of the speaker and collect object information.
Comparing the distance between the face and the hand by the robot by the position of the speaker's face and the hand in three-dimensional space;
Recognizing a small object if the distance between the face and the hand is smaller than a certain distance, and if the distance between the face and the hand is greater than a medium or large object comprising the step of recognizing a robot using multi-modal interaction. The method according to claim 8,
The robot may recognize the gesture of the speaker and collect object information.
Determining whether the speaker is moving after reaching out,
Recognizing a medium object if the speaker does not move after reaching the hand, and the object learning method of the robot using multi-modal interaction, characterized in that it comprises a large object.

Images