KR20180129301A - Method and apparatus for emulating behavior of robot - Google Patents

Abstract

The present invention relates to a behavior emulating apparatus of a robot which recognizes and learns behavior of a demonstrator for the behavior to be emulated. According to the present invention, the behavior emulating apparatus of a robot is able to control a robot to recognize behavior of an object and a person based on the behavior of the demonstrator, and learn and emulate a sequence of the behavior of the demonstrator, in which the behavior is demonstrated for a predetermined time.

Description

METHOD AND APPARATUS FOR EMULATING BEHAVIOR OF ROBOT < RTI ID = 0.0 >

BACKGROUND OF THE INVENTION 1. Field of the Invention [0002] The present invention relates to a behavior mimic device and a method thereof, and more particularly, to a behavior mimic learning system for a robot that recognizes and imitates a behavior of a performer.

In general, in order for a robot to perform a task having a specific purpose, the robot must understand the task and be able to remember the order of the task.

However, until now, a person has used a method of specifying and inputting all operations and sequences of robots. In this case, robots are sensitive to environmental changes, and there is a limit to the difficulty of performing tasks even in small environment changes.

As a conventional behavior recognition technology, there is Bobick, A. F. et al., "The Recognition of Human Movement Using Temporal Templates," IEEE Transactions on Pattern Analysis and Machine Intelligence, 2001. In this paper, A technique of recognizing a behavior using MHI (Motion History Image) which accumulates a silhouette of a person on the front side is disclosed. However, since this technique uses one fixed video camera, There is a problem that the behavior recognition can not be performed if the direction of the performer's motion in the motion recognizer does not coincide with the direction learned by the behavior recognizer.

As a conventional behavior recognition technology for solving such a problem, there is "Free Viewpoint Action Recognition using Motion History Volumes", Computer Vision and Image Understanding, 2006, published by Weinland, D. et al. However, since the process of matching the four camera data requires complicated calculation, it is necessary to perform a real-time operation This conventional technique has a problem that it is not possible to perform the behavior recognition only when all the four cameras capture the object (the performer), and even if one camera can not capture the object, the behavior recognition can not be performed In addition, since four cameras should be installed in every space where we want to recognize the behavior, There is a problem that too much cost is required to construct an expression system.

In addition, the above-described conventional behavior recognition technology has a disadvantage in that it requires a large number of cameras to detect the behavior of the performer, and thus it is difficult to apply it to robots.

In order to solve this problem, the present invention proposes a behavior simulation learning system that recognizes the behavior of the performer using the sensor, and imitates the behavior of the robot by learning it.

Korean Patent Publication No. 1020090079655 (published on July 22, 2009), " Robot Responsive Behavior Method & Korean Patent Publication No. 1020100104272 (published on Sep. 29, 2010), "Behavior recognition system and method"

An object of the present invention is to provide a behavior mimic device of a robot capable of detecting an object zone and a body zone of a person based on a behavior of a demonstrator to recognize a behavior and learning a sequence of behaviors of the demonstrator demonstrated for a predetermined period of time, Method.

It is also an object of the present invention to provide a behavior mimic device and method of a robot capable of controlling a robot to imitate a behavior in accordance with a behavior sequence of a learned performer.

A behavior mimic device of a robot according to an embodiment of the present invention includes a detection unit that detects an object zone and a human body zone from sensed data according to a behavior of a viewer sensed by a sensor, An object storage unit for storing and holding the object recognition result, and an information storage unit for storing the object recognition result, and an information storage unit for storing the object recognition result, And a behavior recognition unit for learning a sequence of behaviors based on the object recognition result and the body region of the object region and the human body region according to the behavior of the performer stored and held in the body region.

Further, the behavior mimic device of the robot according to the embodiment of the present invention may further include a controller for controlling the mimic according to the sequence of the learned behaviors.

The detection unit may use the RGBDT (color distance temperature) sensor to obtain the sensed data on the behavior of the demonstrator.

The detecting unit may include an object detecting module for detecting the object region from the depth information in the sensing data, and a body detecting module for detecting the human body region from the color information, the depth information, and the temperature information in the sensing data have.

The object detection module may detect a plane by applying a Random Access Consensus (RANSAC) algorithm to the sensed data, and may detect the object region based on the depth information by removing a bottom region of the detected plane .

The body detection module detects a human skin region using color information to which a skin color detection algorithm is applied and the temperature information and detects the human body region using the depth information have.

The object recognition unit may recognize the position and type of the object according to the color information using a deep neural network.

The information storage unit may store the position and type of the recognized object in the object zone and the position of the body part in the body zone of the person.

The information storage unit may store and maintain relative distances and relative velocities between the object and the object or between the object and the body on the basis of the position of the recognized object and the position of the body detected for a predetermined period of time.

The behavior recognition unit may learn the sequence of the behavior according to the behavior of the performer that is demonstrated for a predetermined period of time.

The behavior recognition unit may recognize the sequence of the behavior for each frame of the sensing data based on the object region and the human body region using the Recurrent Neural Network (RNN) have.

A method of mimicking a behavior of a robot using a behavior mimic device of a robot according to an embodiment of the present invention includes detecting an object zone and a human body zone from sensed data according to a behavior of a player sensed by a sensor Extracting the detected object zone from the color information in the sensed data to recognize the position and type of the object, detecting the object zone and the body zone of the person detected for a predetermined period of time, Storing and maintaining the object region, the object region and the body region of the person according to the behavior of the performer stored and maintained, learning a sequence of behaviors based on the object recognition result, And controlling the robot to imitate the robot.

Wherein the detecting of the object region and the human body region includes detecting the object region from the depth information in the sensed data and detecting the body region of the person from the color information, .

The step of recognizing the position and the type of the object may recognize a position and a type of the object according to the color information using a deep neural network.

The step of storing and storing may store the position and type of the recognized object in the object zone and the position of the body part in the body zone of the person.

The step of learning the sequence of actions may comprise the steps of: using the Recurrent Neural Network (RNN) to determine the position of the object region and the body region of the person; and, based on the object recognition result, The sequence can be recognized.

According to the embodiment of the present invention, it is possible to detect an object zone and a body zone of a person based on a behavior of a performer, recognize a behavior, and learn a sequence of behaviors of the demonstrator demonstrated for a predetermined period of time.

Further, according to the embodiment of the present invention, the robot can be controlled to imitate the behavior according to the action sequence of the learned performer.

1 is a block diagram showing a configuration of a behavior mimic device of a robot according to an embodiment of the present invention.
2 illustrates an execution algorithm of a behavior mimic device of a robot according to an embodiment of the present invention.
FIGS. 3A to 3C are photographs of an object recognition process according to an embodiment of the present invention.
FIG. 4 is a diagram illustrating a photograph result and an algorithm structure in a behavior recognition process according to an embodiment of the present invention.
5 is a flowchart illustrating a method of simulating behavior of a robot according to an embodiment of the present invention.

Hereinafter, embodiments according to the present invention will be described in detail with reference to the accompanying drawings. However, the present invention is not limited to or limited by the embodiments. In addition, the same reference numerals shown in the drawings denote the same members.

Also, terminologies used herein are terms used to properly represent preferred embodiments of the present invention, which may vary depending on the viewer, the intention of the operator, or the custom in the field to which the present invention belongs. Therefore, the definitions of these terms should be based on the contents throughout this specification.

1 is a block diagram showing a configuration of a behavior mimic device of a robot according to an embodiment of the present invention.

1, a behavior mimic device of a robot according to an embodiment of the present invention recognizes an object zone, a body zone of a person, and a position and a type of an object using sensed data of a behavior of a viewer sensed through a sensor , And learn sequences of behaviors from recognition results.

The behavior mimic device 100 of the robot according to the embodiment of the present invention includes a detection unit 110, an object recognition unit 120, an information storage unit 130, and a behavior recognition unit 140.

The detection unit 110 detects the object region and the human body region from the sensed data according to the behavior of the performer sensed by the sensor.

For example, the detection unit 110 may acquire sensed data on the behavior of the performer using an RGBDT (color distance temperature) sensor. Here, the RGBDT sensor may include a color sensor for sensing color information, a distance sensor for sensing depth information, and a temperature sensor for sensing temperature information.

According to an embodiment, the color sensor may be an RGB sensor, and the distance sensor may be at least one of a laser sensor, a distance measuring sensor, an ultrasonic sensor, and a position sensor, Lt; / RTI > However, the present invention is not limited to the above-described types of sensors, but may be a sensor capable of sensing color information, depth information, and temperature information from the behavior or demonstration image of the performer.

According to another embodiment, the behavior mimic device 100 of the robot according to the embodiment of the present invention may include the image information obtained by using the camera in the sensed data.

The sensing data may include color information (RGB), depth information (D), and temperature information (T) sensed by the RGBDT sensor.

According to the embodiment, the color information, depth information, and temperature information sensed from the RGBDT sensor may represent the same coordinate system through calibration.

The detection unit 110 includes an object detection module 111 that detects an object region from depth information in the sensed data, and a body detection module 112 that detects a human body region from color information, depth information, and temperature information in the sensed data can do.

The object detection module 111 may detect a plane by applying a RANDAM SAmple Consensus (RANSAC) algorithm to the sensed data, and may detect the object area based on the depth information by removing the bottom area of the detected plane.

The residual algorithm is a method for detecting a data set most similar to a user-defined model, and can use a model called a plane for floor removal. For example, the arbitrary three points are selected as candidate points from the sensing data to be sensed, and the number of inliers in which the distance between the plane and the remaining pixels are less than the threshold value is obtained, The candidate point having the inlier of the floor can be recognized as the plane of the floor.

The object detection module 111 can detect the object area from the rough algorithm based on the depth information in the sensed data. Here, it is assumed that an object exists on a plane.

According to an embodiment, the object detection module 111 may obtain the dominant plane in the sensed data using the LAC algorithm, remove all areas under the plane, and then, based on the depth information in the sensed data, Area can be detected. The detected region can be expressed by two types of bounding box and object area and output.

The body detection module 112 can detect a human skin region using color information and temperature information to which a skin color detection algorithm is applied and detect a human body region using depth information .

For example, the body detection module 112 may integrate a skin color detection algorithm using color information and a map obtained through detection of an exothermic region in sensed data using temperature information, Can be detected. Then, the body detection module 112 can detect the body region of the person by defining the end point of the skin region as the position of the hand using the depth information.

According to an embodiment, the body detection module 112 performs hand detection primarily on a human body area, but can detect head, leg, and arm in addition to a hand, The body region can be detected.

Describing in more detail a method for detecting a human skin area and a body area, the body detection module 112 may classify each of the plurality of pixels from color information in the sensed data using a skin color sensing algorithm. The body detection module 112 then determines the color value of the pixel as a sequence of color information sensed by the sensor based on the over time and the color value of the skin tone A region having color values within a determined range can be tracked to detect a skin region or a body region.

Depending on the embodiment, the predetermined range may refer to a range of color values close to the skin tone that can detect a general person's skin tone, which may be preset by the user.

The object recognizing unit 120 extracts the object region detected from the color information in the sensed data, and recognizes the position and type of the object.

More specifically, the object recognition unit 120 can recognize the position and type of an object according to color information using a deep neural network.

For example, the object recognizing unit 120 can recognize the position and the type of the object using the color information detected from the detecting unit 110 and a bounding box based on the object region, Object recognition can be performed using depth neural network for robust characteristics of environmental change. The in-depth neural network can perform learning by collecting learning data about an object of a task to be demonstrated and learned.

The information storage unit 130 stores and maintains an object region, a human body region, and an object recognition result that are detected for a predetermined period of time. The preset predetermined time may be set by the user in units of seconds, minutes, or hours according to the embodiment, and is not limited.

The information storage unit 130 may store the position and type of the recognized object in the object area and the position of the body part in the human body area.

More specifically, the behavior mimic device 100 of the robot according to the embodiment of the present invention should perform behavior recognition for simulated learning based on a change of an object zone and a body zone for a predetermined time. For example, since the same image and positional relationship can be represented in the sensed data even for different behaviors of the performer, temporal feature change should be applied for more accurate behavior recognition.

To this end, the information storage unit 130 stores the detection results of the object region and the body region detected from the detection unit 110 and the object recognition results recognized by the object recognition unit 120 for a predetermined period of time, And may provide information to be stored and maintained in the storage unit 140.

The information storage unit 130 may store and maintain relative distances and relative velocities between the object and the object or between the object and the body on the basis of the position of the object and the position of the body detected for a predetermined period of time.

The behavior recognition unit 140 learns the sequence of behaviors based on the object region and the human body region according to the behavior of the performer stored and held in the information storage unit 130 and the object recognition result.

The behavior recognition unit 140 may learn a sequence of behaviors according to the behavior of the performer that is demonstrated for a predetermined period of time.

More specifically, the behavior recognition unit 140 recognizes a sequence of actions every frame of the sensed data based on the object region and the human body region using the Recurrent Neural Network (RNN) . According to an embodiment, the behavior recognition unit 140 may recognize a sequence of actions using an object and an object stored in and held in the information storage unit 130, or a relative distance and relative speed between the object and the body.

For example, since the behavior recognition unit 140 must perform the behavior recognition based on the characteristics of the behavior of the performer for a predetermined period of time, the behavior recognition unit 140 receives the object region and the body region of the person, Can be learned.

Based on the information received from the information storage unit 130, the behavior recognition unit 140 uses a cyclic-type neural network to recognize the continuous information for a predetermined period of time, The performer's behavior can be recognized.

Here, the Recurrent Neural Network (RNN) may be sequentially input by modifying the model to allow recursive input in a hidden layer of a neural network, or a sequence of features may be input It may be a neural network model considering temporal aspects that can effectively learn patterns from data given as input.

In addition, the behavior mimic device 100 of the robot according to the embodiment of the present invention may further include a controller 150 that controls the robot to imitate the robot according to a sequence of learned behaviors.

The control unit 150 may include a detection unit 110, an object recognition unit 120, an information storage unit 130, and a behavior recognition unit 140 in the behavior mimic device 100 of the robot according to the embodiment of the present invention. ) Can be performed and controlled.

2 illustrates an execution algorithm of a behavior mimic device of a robot according to an embodiment of the present invention.

2, the behavior mimic device of the robot according to an embodiment of the present invention uses color information 211, depth information 212, and temperature information 213 according to the behavior of the performer using the RGBDT sensor 210 The sensing data can be sensed.

Then, the behavior mimic device of the robot according to the embodiment of the present invention can detect (220) the object zone from the depth information 212. [ More specifically, the behavior mimic device of the robot according to the embodiment of the present invention detects a plane of sensing data sensed from the RGBDT sensor 210 by applying a Random Access Consensus (RANSAC) algorithm, The region of interest (ROI) and the object position can be detected 221 based on the depth information 212.

In addition, the behavior mimic device of the robot according to the embodiment of the present invention can detect (222) a human skin area using color information 211 and temperature information 213, and use depth information 212 The body region can be detected (223).

More specifically, the behavior mimic device of a robot according to an embodiment of the present invention includes a skin color detection algorithm using color information 211 and a detection of an area to be heated using temperature information 213 The acquired map can be fused to detect a portion corresponding to the human skin area 222. [ Then, the human body region can be detected 223 by defining the end point of the skin region as the position of the hand or the body part using the depth information 212.

The behavior mimic device of the robot according to the embodiment of the present invention recognizes an object 230 from the color information 211 using a deep neural network and recognizes the position and type of the object 231 )can do.

For example, a behavior mimic device of a robot according to an embodiment of the present invention may use a depthing network and a bounding box based on detected color information 211, object area 221 and body area 223 The object recognition can be performed 230, and the position and the type of the object can be recognized 231. Herein, the neural network can collect and learn learning data about objects of a task to be demonstrated and learned.

Thereafter, the behavior mimic device of the robot according to the embodiment of the present invention includes an object region 221, a position and kind of the recognized object in the object region 231, a position of the body portion 223 in the human body region (240).

In addition, the behavior mimic device of the robot according to the embodiment of the present invention is configured to measure the position and orientation of an object and an object based on a recognized object position 221 and a body part position 223, The relative distance and the relative velocity between the two points can be stored (241).

Hereinafter, the behavior mimic device of the robot according to the embodiment of the present invention is configured such that the behavior recognition 250 is performed based on the characteristics of the behavior of the performer for a preset predetermined time, so that the stored information 241, (Human body area 223 and object recognition result 231) and learn the sequence 251 of actions.

For example, the behavior mimic device of the robot according to the embodiment of the present invention may be configured to perform a behavioral simulation based on the stored information 241, a recognition performance of continuous information for a predetermined period of time, (RNN) to recognize the behavior sequence 251 of the performer for each frame.

FIGS. 3A to 3C are photographs of an object recognition process according to an embodiment of the present invention.

More specifically, FIG. 3A shows a photograph result of sensing data sensed from an RGBDT sensor according to an embodiment of the present invention, FIG. 3B shows a photograph result of eliminating the background of sensed data according to an embodiment of the present invention And Fig. 3C shows a photograph result of object recognition.

Referring to FIG. 3A, a behavior mimic device of a robot according to an embodiment of the present invention can acquire sensing data including color information, depth information, and temperature information using an RGBDT sensor.

Referring to FIG. 3B, the behavior mimic device of the robot according to the embodiment of the present invention can remove the background to the sensed data obtained in FIG. 3A.

3C, the behavior mimic device of the robot according to an embodiment of the present invention can recognize the position and type of an object using a plurality of bounding boxes (red squares) based on detected color information and object zones Object recognition can be performed using a deep neural network for robustness to object recognition performance and environmental changes. The in-depth neural network can perform learning by collecting learning data about an object of a task to be demonstrated and learned.

Accordingly, the behavior mimic device of the robot according to the embodiment of the present invention can recognize the position and the type of the object using the sensed data as shown in FIGS. 3A to 3C.

FIG. 4 is a diagram illustrating a photograph result and an algorithm structure in a behavior recognition process according to an embodiment of the present invention.

More specifically, Fig. 4 (a) shows photograph results of an object zone and a bodily zone detected according to an embodiment of the present invention, and Fig. 4 (b) (RNN) algorithm of the present invention.

Referring to FIG. 4A, a behavior mimic device of a robot according to an embodiment of the present invention can detect an object zone and a human body zone from sensed data according to a behavior of a performer.

For example, in FIG. 4 (a), a behavior mimic device of a robot according to an embodiment of the present invention can sense a hand of a performer as a body area. In addition, the cup GC located in the hand of the performer can be detected as an object region, and a bowl located on a tabletop can be detected as an object region.

Hereinafter, the behavior mimic device of the robot according to the embodiment of the present invention may store the information of the sequential object region and the body region detected for a preset predetermined time based on the three-dimensional position information.

4 (b), the behavior mimic device of the robot according to the embodiment of the present invention learns a sequence of behaviors of a performer using a recurrent neural network (RNN) can do.

For example, the behavior mimic device of a robot according to an embodiment of the present invention may include an object region and a human body region of information stored using a Recurrent Neural Network (RNN), object recognition results (Hand, Bowl, GC, etc.), it is possible to recognize the behavior of the performer by recognizing the sequence of action every frame of the sensed data.

More specifically, referring to FIG. 4B, the Recurrent Neural Network (RNN) can be modeled as a sequential input by modifying the model so that recursive input is possible in a hidden layer of a neural network Or a neural network structure considering temporal aspects that can effectively learn patterns from data (1, 2, 3, ..., n2, n1, n) given as inputs of sequences of features.

5 is a flowchart illustrating a method of simulating behavior of a robot according to an embodiment of the present invention.

The method shown in Fig. 5 can be performed by a behavior mimic device of a robot that imitates a behavior of a demonstrator according to an embodiment of the present invention shown in Fig.

Referring to FIG. 5, in step 510, an object region and a human body region are detected from sensed data according to a behavior of a viewer sensed by a sensor.

For example, step 510 includes a step of detecting an object zone from depth information in the sensed data (not shown) and a step of detecting a human body zone from color information, depth information and temperature information in the sensed data Lt; / RTI >

According to an embodiment, step 510 may utilize an RGBDT (color distance temperature) sensor to obtain sensed data on the behavior of the performer, wherein the RGBDT sensor includes a color sensor for sensing color information, a distance sensor , And a temperature sensor that senses temperature information.

In step 520, the object region detected in the color information in the sensed data is extracted to recognize the position and type of the object.

Step 520 may be a step of recognizing the position and type of an object according to color information using a deep neural network.

For example, in step 520, the location and type of an object can be recognized using the color information and the bounding box based on the object area. For the characteristics that are robust to object recognition performance and environment change, The object recognition may be performed using the object recognition method. Here, the depth neural network may be a model (or algorithm) for performing learning by collecting learning data on objects of a task to be demonstrated and learned.

In step 530, the object region and the human body region detected for a predetermined period of time and the object recognition result are stored and retained.

Step 530 may be a step of storing the position and type of the recognized object in the object zone and the position of the body part in the person's body zone.

In addition, step 530 may be a step of storing and maintaining relative distances and relative velocities between the object and the object or between the object and the body based on the position of the object and the position of the part of the body detected for a predetermined period of time.

The object region and the human body region according to the behavior of the performer stored and held at Step 540 and the sequence of the action based on the object recognition result are learned.

Step 540 can recognize a sequence of actions for each frame of the sensing data based on the object region and the human body region and the object recognition result using the Recurrent Neural Network (RNN).

For example, in step 540, since the behavior recognition should be performed based on the characteristics of the behavior of the performer for a predetermined period of time, the object area and the human body area stored in step 530 and the object recognition result are received, Can be learned.

Here, the Recurrent Neural Network (RNN) may be sequentially input by modifying the model to allow recursive input in a hidden layer of a neural network, or a sequence of features may be input It may be a neural network model considering temporal aspects that can effectively learn patterns from data given as input.

Thereafter, the robot is controlled to imitate according to the sequence of the actions learned in step 550. [

The apparatus described above may be implemented as a hardware component, a software component, and / or a combination of hardware components and software components. For example, the apparatus and components described in the embodiments may be implemented within a computer system, such as, for example, a processor, a controller, an arithmetic logic unit (ALU), a digital signal processor, a microcomputer, a field programmable array (FPA) A programmable logic unit (PLU), a microprocessor, or any other device capable of executing and responding to instructions. The processing device may execute an operating system (OS) and one or more software applications running on the operating system. The processing device may also access, store, manipulate, process, and generate data in response to execution of the software. For ease of understanding, the processing apparatus may be described as being used singly, but those skilled in the art will recognize that the processing apparatus may have a plurality of processing elements and / As shown in FIG. For example, the processing unit may comprise a plurality of processors or one processor and one controller. Other processing configurations are also possible, such as a parallel processor.

The software may include a computer program, code, instructions, or a combination of one or more of the foregoing, and may be configured to configure the processing device to operate as desired or to process it collectively or collectively Device can be commanded. The software and / or data may be in the form of any type of machine, component, physical device, virtual equipment, computer storage media, or device , Or may be permanently or temporarily embodied in a transmitted signal wave. The software may be distributed over a networked computer system and stored or executed in a distributed manner. The software and data may be stored on one or more computer readable recording media.

The method according to an embodiment may be implemented in the form of a program command that can be executed through various computer means and recorded in a computer-readable medium. The computer-readable medium may include program instructions, data files, data structures, and the like, alone or in combination. The program instructions to be recorded on the medium may be those specially designed and configured for the embodiments or may be available to those skilled in the art of computer software. Examples of computer-readable media include magnetic media such as hard disks, floppy disks and magnetic tape, optical media such as CDROMs and DVDs, magnetic optical media such as floppy disks, magnetooptical media, and hardware devices specifically configured to store and execute program instructions such as ROM, RAM, flash memory, and the like. Examples of program instructions include machine language code such as those produced by a compiler, as well as high-level language code that can be executed by a computer using an interpreter or the like. The hardware devices described above may be configured to operate as one or more software modules to perform the operations of the embodiments, and vice versa.

While the present invention has been particularly shown and described with reference to exemplary embodiments thereof, it is to be understood that the invention is not limited to the disclosed exemplary embodiments. For example, it is to be understood that the techniques described may be performed in a different order than the described methods, and / or that components of the described systems, structures, devices, circuits, Lt; / RTI > or equivalents, even if it is replaced or replaced.

Therefore, other implementations, other embodiments, and equivalents to the claims are also within the scope of the following claims.

Claims (17)

A detector for detecting an object region and a human body region from sensed data according to a behavior of a performer sensed by the sensor;
An object recognition unit for extracting the detected object zone from the color information in the sensed data and recognizing the position and type of the object;
An object storage unit for storing the object recognition result and the object area detected for a predetermined period of time; And
And a behavior recognition unit for learning a sequence of behaviors based on the object recognition result and the object region stored in the information storage unit and held by the performer in accordance with the behavior of the object region and the human body region,
The behavior mimic device of the robot. The method according to claim 1,
And controls to imitate according to the learned action sequence
Further comprising: a behavior mimic device of the robot. The method according to claim 1,
The detection unit
A behavior mimic device of a robot that uses the RGBDT (Color Distance Temperature) sensor to obtain the sensing data for the behavior of the demonstrator. The method of claim 3,
The detection unit
An object detection module that detects the object area from the depth information in the sensing data; And
A body detection module for detecting the human body region from the color information, the depth information, and the temperature information in the sensed data,
The behavior mimic device of the robot including the robot. 5. The method of claim 4,
The object detection module
Detecting a plane by applying a Random Access Consensus (RANSAC) algorithm to the sensed data, and removing the bottom region of the detected plane, and detecting the object region based on the depth information Imitation device. 5. The method of claim 4,
The body detection module
A skin color detection algorithm (skin color detection algorithm), and a skin area of a human using the temperature information, and detects the human body area using the depth information Behavior mimicry. The method according to claim 1,
The object recognition unit
And the location and type of the object according to the color information are recognized using a deep neural network. The method according to claim 1,
The information storage unit
And stores the position and type of the recognized object in the object zone and the position of the body part in the body zone of the person. 9. The method of claim 8,
The information storage unit
And stores and maintains relative distances and relative velocities between the object and the object or between the object and the body based on the position of the recognized object and the position of the body part detected for a predetermined period of time Imitation device. The method according to claim 1,
The behavior recognition unit
Wherein the robot learns the sequence of the behavior according to the behavior of the performer, which is demonstrated for a predetermined period of time. 11. The method of claim 10,
The behavior recognition unit
Characterized by recognizing the sequence of behavior for each frame of the sensing data based on the object region and the human body region using the Recurrent Neural Network (RNN) and the object recognition result, Behavior mimic device. A behavior mimicry method of a robot that imitates a behavior of a performer using a behavior mimic device of a robot,
Detecting an object region and a human body region from sensed data according to a behavior of a performer sensed by the sensor;
Extracting the detected object zone from the color information in the sensing data to recognize the location and type of the object;
Storing the object region and the body region of the human being detected for a predetermined period of time and storing the object recognition result;
Learning a sequence of behaviors based on the object region and the body region of the person according to the behavior of the performer stored and held, and the object recognition result; And
Controlling the robot to imitate according to the learned sequence of actions
Wherein the robot simulates the behavior of the robot. 13. The method of claim 12,
The step of detecting the object region and the human body region
Detecting the object zone from the depth information in the sensing data; And
Detecting the human body region from the color information, the depth information, and the temperature information in the sensed data
Wherein the robot simulates the behavior of the robot. 13. The method of claim 12,
Recognizing the location and type of the object
And recognizing a position and a type of the object according to the color information using a deep neural network. 13. The method of claim 12,
Storing and maintaining
A position and a type of the recognized object in the object zone and a position of a body part in the body zone of the person are stored. 13. The method of claim 12,
The step of learning the sequence of actions
Characterized by recognizing the sequence of behavior for each frame of the sensing data based on the object region and the human body region using the Recurrent Neural Network (RNN) and the object recognition result, How to mimic behavior. 17. A computer program stored in a computer-readable medium for performing the method of any one of claims 12 to 16.

Images