KR101986002B1 - Artificial agents and method for human intention understanding based on perception-action connected learning, recording medium for performing the method - Google Patents

Abstract

The behavior-aware link learning-based intention understanding device includes: an input unit for detecting joint information of user behavior observed every frame and object information of a user's surrounding environment; A pre-processing unit for pre-processing the joint information and the object information received from the input unit so as to enable artificial neural network processing; A behavior recognition processing unit for classifying the behavior information of the user based on the joint information and the object information output from the preprocessing unit; An object relation information processing unit for outputting an object candidate group related to a user's action by using the object information output from the preprocessing unit and the behavior information output from the behavior recognition processing unit; And an intention output unit for outputting the user's intention recognition result through the artificial neural network that receives the behavior information output from the behavior recognition processing unit and the object candidate group output from the object relationship information processing unit. Accordingly, the user's intention can be accurately predicted from the user behavior and the object information related to the behavior.

Description

BACKGROUND OF THE INVENTION 1. Field of the Invention The present invention relates to a behavior-aware connection learning intention understanding device, a method, and a recording medium for performing the method. 2. Description of the Related Art [0002]

BACKGROUND OF THE INVENTION 1. Field of the Invention The present invention relates to an apparatus, a method, and a recording medium for understanding a behavior-aware connection learning-based intention, and more particularly to an apparatus for predicting a user's intention from user behavior and object information related thereto .

In recent years, many services such as natural interface design based on user behavior, acquisition of life related information based on smart home, user health in healthcare, state and exercise management have been planned. However, It is often difficult to achieve product implementation because it can not guarantee reliable intent based on interaction.

Understanding intention is important for developing an artificial intelligence agent such as a robot that can provide various services to human beings. Over the years, we have made considerable progress in the development of artificial agents, but it is far from realizing some of the basic elements of cognition, such as emotion or intent perception. They are part of the mental capacity that naturally belongs to humans and makes human-human interaction unique.

In particular, the ability to understand other people's intentions, or "empathy," has been claimed to be the basis of human-human communication. Theory of Mind (Premack & Woodruff, 1978) argues that humans have the innate inherent ability to understand others' intentions and to empathize with others. This ability to enable humans to respond in a consistent and useful way also extends to other human domains, such as language understanding, learning, and emotional awareness.

Therefore, it is important to develop such mental abilities to develop artificial agents that behave and respond in a manner similar to those of intelligent humans, especially the ability to understand others' intentions.

However, in order to grasp the intention, it is necessary to first understand how human beings acquire this ability, what the physiological, biological basis of empathy, or the ability to understand others' intentions, what can be learned from this cognitive process, And the ability to simulate intentional recognition capability in the context of the present invention.

Currently, several computational models have been proposed to implement the ability to recognize intent in artificial agents. Some are based on the object's affordance, while others are based on behavioral predictions. However, the proposed models have similar behavior to explicit gestures, but there are limits to objects that do not take into account the objects related to motion.

KR 10-1605078 B1 KR 10-1592977 B1

 Yu, Zhibin, and Minho Lee, Human motion based intent recognition using a deep dynamic neural model, Robotics and Autonomous Systems, 2015  Kim, S., Kavuri, S., & Lee, M., Intention Recognition and Object Recommendation System using Deep Auto-encoder based Affordance Model, In the 1st International Conference on Human-Agent Interaction, 2013  Yu, Z., & Lee, M., Real-time human action classification using a dynamic neural model, Neural Networks, 69, 29-43, 2015

SUMMARY OF THE INVENTION Accordingly, the present invention has been made in view of the above problems, and it is an object of the present invention to provide a behavior-aware connection learning intention understanding device.

It is another object of the present invention to provide a method for understanding behavior-aware connection learning based intent.

Yet another object of the present invention is to provide a recording medium on which a computer program for performing the behavior-aware connection learning-based intention understanding method is recorded.

According to an embodiment of the present invention, there is provided an apparatus for understanding an action-aware connection learning intention according to an embodiment of the present invention includes an input unit for detecting joint information of user behavior observed per frame and object information of a user's surrounding environment; A pre-processing unit for pre-processing the joint information and the object information received from the input unit so as to enable artificial neural network processing; A behavior recognition processing unit for classifying the behavior information of the user based on the joint information and the object information output from the preprocessing unit; An object relation information processing unit for outputting an object candidate group related to a user's action by using the object information output from the preprocessing unit and the behavior information output from the behavior recognition processing unit; And an intention output unit for outputting the user's intention recognition result through the artificial neural network that receives the behavior information output from the behavior recognition processing unit and the object candidate group output from the object relationship information processing unit.

In an embodiment of the present invention, the input unit may collect two-dimensional or three-dimensional positional information of a user's main joint in real time based on at least one of a visible light ray zone image sensor and an active infrared ray pattern projection sensor.

In an embodiment of the present invention, the input unit may perform at least one of face recognition or tracking to avoid interference of behavior modeling information of each user when there are a plurality of users in the sensing area.

In the embodiment of the present invention, the pre-processing unit may include: a joint information preprocessing unit for normalizing and encoding the joint information received from the input unit and transmitting the joint information to the behavior recognition processing unit; And an object information preprocessing unit for extracting the object information from the image information received from the input unit and delivering the label of the object held by the user to the object relationship information processing unit.

In an embodiment of the present invention, the joint information preprocessing unit may include: a normalization unit for normalizing major joint information based on an absolute coordinate acquired in a sensing region of the input unit to a relative coordinate representation for each user; And an encoding unit for expressing the normalized relative coordinate expression in a neural network-friendly manner.

In an embodiment of the present invention, the encoding unit may use a self-organizing map (SOM).

In the embodiment of the present invention, the behavior recognition processing unit may model the joint information and the object information output from the preprocessing unit through the regression neural network to designate the recognition only node so that the behavior recognition can be performed.

In the embodiment of the present invention, the behavior recognition processor may extract the closest behavior by comparing the user behavior vectors with each other when learning a recognition-only node output corresponding to a given input in the real-time processing step and the test step .

In the embodiment of the present invention, the object relationship information processing unit may model the object relation through the self-coding network, on the object information output from the preprocessing unit and the behavior information output from the behavior recognition processing unit, can do.

In an embodiment of the present invention, the joint information of the user's actions may include a midline of the spine, neck, head, left shoulder, left elbow, left wrist, right shoulder, right elbow, right wrist, left hip, right hip and shoulder ) May be used.

According to another aspect of the present invention, there is provided a method of understanding an action-aware connection learning intention according to an exemplary embodiment of the present invention includes: detecting joint information of user behavior and object information of a user's environment; A pre-processing step of pre-processing the joint information and the object information so as to enable artificial neural network processing; A behavior recognition processing step of classifying the behavior information of the user based on the joint information and the object information output in the preprocessing step; An object relation information processing step of outputting an object candidate group related to a user's action using the object information output in the preprocessing step and the behavior information output in the behavior recognition processing step; And an intention output step of outputting the user's intention recognition result through the artificial neural network inputting the behavior information output from the behavior recognition processing step and the object candidate group output from the object relationship information processing step.

In the embodiment of the present invention, the step of detecting the object information of the user's surroundings may include the step of detecting two-dimensional or three-dimensional position information of the main joint of the user based on at least one of the visible light ray group image sensor and the active infrared ray pattern projection sensor It can be collected in real time.

In the embodiment of the present invention, the step of detecting object information of the user environment may include at least one of face recognition or tracking function to avoid interference of behavior modeling information of each user when there are a plurality of users in the sensing area can do.

In the embodiment of the present invention, the pre-processing step may include: normalizing the joint-based information on the basis of the absolute coordinates acquired in the sensing area into a relative coordinate representation per user; And an encoding step of representing the normalized relative coordinate representation in a neural network-friendly manner.

In an embodiment of the present invention, the encoding step may use a self-organizing map (SOM).

In the embodiment of the present invention, the preprocessing step extracts the object information from the received image information, and transmits the label of the object that the user holds by hand.

In the embodiment of the present invention, the behavior recognition processing step may model the joint information and the object information output from the preprocessing step through the regression neural network to designate the recognition only node so that the behavior recognition can be performed.

 In the embodiment of the present invention, the behavior recognizing process step may include extracting the closest behavior by comparing with the user behavior vectors when learning a recognition-only node output corresponding to a given input in the real-time processing step and the testing step have.

In the embodiment of the present invention, the object relationship information processing step may include modeling an object relation through the self-encoding network by the object information output from the preprocessing unit and the behavior information output from the behavior recognition processing unit, Can be output.

According to another aspect of the present invention, there is provided a computer-readable storage medium storing a computer program for performing a behavior-aware connection learning based understanding understanding method.

In order to overcome the limitations of the existing user behavior based intention recognition device, the behavior-aware connection learning based intention understanding method is combined with the object recognition model and suggested an appropriate learning method, so that a reliable user intention having higher flexibility and accuracy A recognition device is provided.

In addition, the technology proposed by the present invention can dramatically improve performance by using mutual information by combining a relationship model between objects having a static property and a user behavior recognition model having a dynamic property. Furthermore, by using the structure and learning method proposed in the present invention, both the user behavior recognition performance and the inter-object relationship model performance can be improved.

1 is a block diagram of a behavior-aware connection learning intention understanding device in accordance with an embodiment of the present invention.
2 is a diagram illustrating an OA-SMTRN model for intention recognition proposed in the present invention.
Figure 3 is a flow chart of a simplified implementation of the model proposed in the present invention.
Figure 4 is a diagram illustrating a biological model for behavioral modeling and intentional inference.
5 is a behavior module for intentional reasoning and object prediction based on behavior understanding.
6 is a simplified diagram of an automatic encoder.
7 is a diagram showing a learning process of the deep automatic encoder.
8 is a diagram showing a structure of a prediction module of OA-SMTRNN of FIG.
9 is a flowchart of a behavior-aware connection learning intention understanding method according to an embodiment of the present invention.

The following detailed description of the invention refers to the accompanying drawings, which illustrate, by way of illustration, specific embodiments in which the invention may be practiced. These embodiments are described in sufficient detail to enable those skilled in the art to practice the invention. It should be understood that the various embodiments of the present invention are different, but need not be mutually exclusive. For example, certain features, structures, and characteristics described herein may be implemented in other embodiments without departing from the spirit and scope of the invention in connection with an embodiment. It is also to be understood that the position or arrangement of the individual components within each disclosed embodiment may be varied without departing from the spirit and scope of the invention. The following detailed description is, therefore, not to be taken in a limiting sense, and the scope of the present invention is to be limited only by the appended claims, along with the full scope of equivalents to which such claims are entitled, if properly explained. In the drawings, like reference numerals refer to the same or similar functions throughout the several views.

Hereinafter, preferred embodiments of the present invention will be described in more detail with reference to the drawings.

1 is a block diagram of a behavior-aware connection learning intention understanding device in accordance with an embodiment of the present invention.

In order to overcome the limitations of the existing user behavior based intention recognizing apparatus, the device 10 for understanding the behavior-aware connection learning based on the present invention combines with the object recognition model and proposes an appropriate learning method, A user's intention recognizing device.

First, there are two cognitive processes to realize the ability to recognize intention: awareness of object rationality in the environment and prediction of human behavior. These two processes provide implicit and explicit information about the intent and interact at various levels. The present invention assumes that the implementation of the recognition and behavioral connection can be the key to implementing the function of recognizing human intention in the artificial agent.

& Stagge, 2003)이다. 이 모델은 행동 분류를 위해 슈퍼바이저된(supervised) MTRNN(SMTRNN)으로 확장되었다(Yu & Lee, 2015).On the other hand, various dynamic models have been developed to manage behavior classification problems, as well as extract intent signals from dynamic human behavior. HMM is a well-known model for analyzing and classifying human behavior (Gehrig, Kuehne, Woerner, & Schultz, 2009). The other is a repetitive neural network (RNN) -based model such as Multiple Timescale Recurrent Neural Network (MTRNN) (Yamashita & Tani, 2008)

& Stagge, 2003). This model has been extended to supervised MTRNN (SMTRNN) for behavioral classification (Yu & Lee, 2015).

The system proposed in the present invention integrates two independent processes based on the recognition-action connection into Object Augmented-SMTRNN (OA-SMTRNN) for intent classification. Determining the intention based solely on the rationality of a recognized object or a predicted sequence of actions is likely to cause errors because it represents several possible intentions.

Therefore, in order to accurately determine a specific intention, it is necessary to grasp the interaction between the recognition and the action. The present invention assumes that the recognition and the action together form a loop for improving the accuracy of determining the user's intention, The user's intention is predicted from the object information.

User behavior refers to the user's body movements over time, and dynamic signals must be handled to model them. In addition, the objects used during the user 's behavior are used alone, or there are candidates to be used together, and relational modeling that can derive them is required for user intention recognition.

The present invention implements recognition-action-based learning for understanding human intention in a newly proposed model called Object Augmented Multiple Timescale Recurrent Neural Network (hereinafter, OA-SMTRN). The model proposed in the present invention is an extension of the previous study (Kim, Kavuri, & Lee, 2013; Yu & Lee, 2015).

The device 10 according to the present invention recognizes the user's current behavior based on the user's main joint information and surrounding environment information, and detects the user's intention information by detecting related objects and candidate groups.

Referring to FIG. 1, an apparatus 10 according to the present invention includes an input unit 100, a preprocessing unit 300, and a processing unit 500. More specifically, the pre-processing unit 300 includes a joint information preprocessing unit 310 and an object information preprocessing unit 330. The processing unit 500 includes a behavior recognition processing unit 510, an object relation information processing unit 530, And an intention output unit 550.

The apparatus 10 of the present invention can be installed and executed with software (application) for understanding the behavior-aware connection learning intention. The apparatus 10 can be implemented by the input unit 100, the preprocessing unit 300, The configuration can be controlled by software for understanding the behavior-aware connection learning-based intention that is executed in the device 10. [

The device 10 may be a separate terminal or some module of the terminal. The configuration of the input unit 100, the preprocessing unit 300, and the processing unit 500 may be an integrated module or may be composed of one or more modules. However, conversely, each configuration may be a separate module.

The device 10 may be mobile or stationary. The device 10 may be in the form of a server or an engine and may be a device, an apparatus, a terminal, a user equipment (UE), a mobile station (MS) a wireless device, a handheld device, and the like.

The device 10 may execute or produce various software based on an operating system (OS), i.e., a system. The operating system is a system program for allowing software to use the hardware of a device. The operating system includes a mobile computer operating system such as Android OS, iOS, Windows Mobile OS, Sea OS, Symbian OS, Blackberry OS, MAC, AIX, and HP-UX.

The input unit 100 detects the joint information of the user's behavior and the object information of the user's surroundings, which are observed every frame, and transmits the detected object information to the preprocessing unit 300. The input unit 100 may acquire two-dimensional or three-dimensional positional information of the user's major joints in real time based on at least one of the visible light ray image sensor and the active infrared ray pattern projection sensor.

If there are a plurality of users in the sensing area, the input unit 100 may perform at least one of face recognition and tracking functions to avoid interference of the behavior modeling information of each user.

The joint information of the user's actions may be skeletal points such as midline, neck, head, left shoulder, left elbow, left wrist, right shoulder, right elbow, right wrist, left hip, right hip and shoulder have.

In addition, the input unit 100 detects and collects user peripheral object information in real time based on the visible light ray sensor image sensor information to grasp the user's peripheral object information.

In another embodiment, the input unit 100 may include an input unit for acquiring information on a user's main joint position and an input unit for acquiring information on a user's neighboring object.

The preprocessing unit 300 includes a joint information preprocessing unit 310 for normalizing and encoding the joint information received from the input unit 100 and transferring the joint information to the behavior recognition processing unit 510, And an object information preprocessing unit 330 for extracting the object information from the information and transferring the label of the object held by the user to the object relationship information processing unit 530.

The joint information preprocessing unit 310 may include a normalization unit for normalizing the joint information based on the absolute coordinates obtained in the sensing region of the input unit 100 to a relative coordinate representation for each user, and a normalized relative coordinate representation to a neural- As shown in FIG. The encoding unit may use a self-organizing map (SOM) or the like.

The processing unit 500 includes a behavior recognition processing unit 510 for classifying the behavior information of the user based on the joint information and the object information that are preprocessed and output in the preprocessing unit 300, An object relation information processor 530 for outputting object candidates related to the user's behavior using the object information output from the behavior recognition processor and the behavior information output from the behavior recognition processor, And an intention output unit 550 for outputting a user's intention recognition result through an artificial neural network that receives an object candidate group output from the object relationship information processing unit 530 as an input.

The behavior recognition processor 510 continuously recognizes the behavior through the regression neural network based on the preprocessed information. Specifically, the joint information and the object information output from the preprocessing unit 300 are modeled through a regression neural network to designate a recognition dedicated node so that the behavior recognition can be performed. The behavior recognition processor 510 extracts the closest behavior by comparing with user behavior vectors when learning a dedicated recognition node output corresponding to a given input in a real-time processing step and a test step.

The object relationship information processing unit 530 continuously derives an associated candidate object through the object relationship model based on the preprocessed information. Specifically, the object information output from the preprocessing unit 300 and the behavior information output from the behavior recognition processing unit 510 are modeled through the self-encoding network, and the object self-encoding result is output.

The intention output unit 550 outputs the user intention information by integrating the results of the behavior recognition processing unit 510 and the object relationship information processing unit 530. That is, the neural network or the regression neural network, which receives the outputs of the behavior recognition processing unit 510 and the object relationship information processing unit 530 as input, synthesizes behavior and object relationship information and outputs only the user's intention recognition result.

The processing unit 500 is implemented in an Object Augmented-SMTRNN (hereinafter, OA-SMTRNN) model in the present invention, and will be described in detail below.

2 is a diagram illustrating an OA-SMTRN model for intention recognition proposed in the present invention.

The right hand side of FIG. 2 is modeled by a deep auto-encoder (Hinton & Salakhutdinov, 2006), which is used to analyze the relationship information of objects. The code layer of the deep autocoder rectifies the information in the visible layer and represents the potential human intent information associated with the object selected by the intended user. The coded information is used as an additional input to the low-speed context layer of SMTRNN with a weight W _pa .

The SMTRNN shown on the left side of Figure 2 is used to analyze skeletal trajectories of human motion associated with a particular intent. Further, the present invention seeks to predict the selection of objects associated with the current behavior as well as classify the intent of the human being by the SMTRNN.

Thus, two classification node groups are defined for both human intention classification and object prediction. Using additional information obtained from the code layer of the deep automatic encoder, SMTRNN is a similar operation, but can efficiently understand the various intentions of using other objects.

In addition, the object predicted by SMTRNN can be reused as an additional visible input to the hidden layer of the deep automatic encoder via the W _ap connection. Let C be the number of neurons in the code layer of the deep autocoder. The first hidden layer of the deep autocoder includes H hidden neurons. The object prediction task of SMTRNN contains O types of objects. The intention classification task of SMTRNN has I intentions.

Then, the size of W _ap becomes O * H , and the size of W _pa is defined as (SIO) * C. Here, S is a low-speed context node number. Since W _pa is designed to prevent circuit shorts, there is no direct connection between the code layer and the classification outputs ( I and O ).

In Figure 2, the arrows indicate the flow of information, and the two arrows between the two modules describe the path for behavior-aware connection learning.

Figure 3 is a flow chart of a simplified implementation of the model proposed in the present invention.

Fig. 3 describes the execution flow of the proposed model in the period of time index t. The left and right portions of OA-SMTRNN in FIG. 2 are referred to as behavioral modules and recognition modules, respectively. If a person with the intention acts at time t, the action module, the left module of Figure 2, attempts to understand the meaning of human behavior and predicts the object label associated with the human intention.

Then, a deep autonomous encoder with predicted object labels, which is a recognition module, generates latent information at the code layer at time t, and the code layer is connected to the slow context unit to identify human intention as well as time t + 1 Lt; RTI ID = 0.0 > of < / RTI > A more detailed procedure of the computational model for behavior-aware connection learning is described below.

The behavioral module of the method proposed in the present invention is extended to object prediction based on understanding human behavior related to a specific human intention (Yu & Lee, 2015). The behavioral module of OA-SMTRNN in FIG. 2 uses three layers to model IPL and PMv, and regards it as a bottom-up model. The input signal at the input-output layer is human skeleton data and the slow context layer has another input path connected to the code layer of the deep automatic encoder for object behavior fluidity.

Prediction of behavior has the effect of constraining the corresponding possible objects. In the present invention, a special group of neurons in the low-speed context layer is defined as a classifier node for intentional reasoning based on rational information of behavior and objects. In the present invention, another output node is newly defined in the low-speed context layer for object label prediction. This model will predict "types of objects" that need to interact or interact with human behavioral intentions.

Figure 4 is a diagram illustrating a biological model for behavioral modeling and intentional inference.

The input-output layer is modeled by a self-organizing map (SOM) to receive and output the action sequence. The SOM algorithm can preserve the topology characteristics of the input space.

The main layer of the MTRNN, the context layer, is modeled using a continuous time repetitive neural network (CTRNN). CTRNN is a special type of RNN and is a dynamic system model of biological neural networks. The output of each neuron in the CTRNN is computed using the current input samples and the past history of the neural status. Thus, the CTRNN is suitable for predicting successive sensory motion sequences.

The back propagation time (BPTT) algorithm is used for learning. The error function is defined by the following Equation 1 using Kullback-Leibler divergence.

[Equation 1]

는 시간 단계 t에서의 i 번째 뉴런의 원하는 출력 값이고,

는 기존의 가중치와 초기 상태에서 i번째 뉴런의 예측값이다. 로봇과 관련된 MTRNN에 대한 이전 연구에서 시각 입력 및 모터 고유값을 포함한 포함한 두 개의 SOM 계층이 일반적이지만(Yamashita & Tani, 2008), 본 발명에서는 실제 로봇을 사용하지 않으므로, 고유값 비전(실험에서 골격 좌표) 입력을 단일 SOM 계층에 결합한다.Where O is the node of the input-output layer,

Is the desired output value of the i < th > neuron at time step t,

Is the predicted value of the ith neuron in the original weight and initial state. In previous research on MTRNNs related to robots, two SOM layers including visual input and motor eigenvalues are common (Yamashita & Tani, 2008), but since the present invention does not use actual robots, Coordinate) input into a single SOM layer.

The weight update rule is described by the following equation (2).

&Quot; (2) "

,

,

는 다음의 수학식 3과 같이 주어진다.Here, n is an iterative step and? Is a learning rate set to 0.0005 in the experiment. Partial differential

Is given by the following equation (3).

&Quot; (3) "

5 is a behavior module for intentional reasoning and object prediction based on behavior understanding.

Detailed structure of the behavior module of the OA-SMTRNN model The concept of the input / output hierarchy and the high-speed and low-speed context hierarchy of FIG. 5 is inherited from the MTRNN model. If time information is obtained at time step t, the SOM node is used for feature extraction.

I / O layer and low-speed context layer are not connected. The high-speed context layer functions as a connection portion connecting the input / output and low-speed context layers, and the nodes in the high-speed and context layers are completely connected. The final output of the SOM node is used to compute the prediction error using the BPTT algorithm.

According to the brain model defined in Yu & Lee, 2015 used in the present invention, a special type of low-speed context node called a classification node is defined. The classification node is part of the low-speed context layer and has the same time constant τ. The difference between classification nodes and other slow context nodes is that intent and object labels are used to learn classification nodes in slow context units.

Classification nodes must not only propagate behavior prediction errors, but also propagate intention reasoning and object prediction errors back to other nodes. All nodes, including classification nodes, operate synchronously, and when a test sequence is given, the classification node corresponding to the label is activated. The partial differential equation after considering the classification error is defined by Equation (4) below.

&Quot; (4) "

는 뉴런 출력과 이상 값의 차이고,

는 t 시간 단계에서 i번째 뉴런 상태이고,

는 뉴런 상태 업데이트 속도를 제어하는 상수이고,

는 크로네커 델타(i=k라면

　= 1, 그렇지 않으면 0)이다. O는 입출력 노드 집합을 나타내고,

및

는 각각 저속 컨텍스트 계층에서의 객체 예측 및 의도 분류에 사용되는 분류 노드를 나타낸다. Here, f '(x) is a derivative of the Sigmoid function,

Is the difference between the neuron output and the ideal value,

Is the i-th neuron state at time t,

Is a constant that controls the neuron status update rate,

Is a Kronecker delta (i = k if

= 1, otherwise 0). O represents an input / output node set,

And

Represent classification nodes used for object prediction and intention classification in the low-speed context layer, respectively.

At the input-output layer, a prediction signal of behavior is generated while the classification operation is performed in the low-speed context layer. However, the present invention has modified this equation to accommodate object prediction. The final output is calculated using the Sigmoid function instead of the Softmax activation function (Yu & Lee, 2015) used in the classification. This is done so that multiple objects can be used for an intent. The Softmax function supports only one category.

As noted above, user intent can also be inferred based on behavioral affordability. In the OA-SMTRNN proposed in the present invention, since the deep structured network is known to have a capability of capturing a high-dimensional relationship or a characteristic that can not be directly observed, behavioral fluidity is modeled using a deep automatic encoder, (Kim, et al., 2013).

The auto-encoder model typically consists of two parts: an encoder and a decoder. The encoder converts the original input signal into a relatively short code that can be viewed as a compression. The decoder corresponds to the encoder and reconstructs the original information in the encoded signal. In previous studies that refer to this model, the input of an automatic encoder is a vector representing behavioral fluidity information. The behavioral fluidity information is encoded and the associated object is decoded.

6 is a simplified diagram of an automatic encoder.

In general, the work of an automatic encoder can be considered insignificant because it unnecessarily compresses information and restores it. However, if an auto-encoder is built for a specific purpose, the auto-encoder can extract useful information from the signal and remove noise. The present invention requires an automatic encoder that analyzes the behavioral fluidity of objects associated with human intent and extracts meaningful latent information from the observed objects to aid in directed reasoning.

Because of its deep structure, it is difficult to learn automatic encoders. In the model of the present invention, the method proposed by Hinton and Salakhutdinov (Hinton & Salakhutdinov, 2006) is used. Building by Restricted Boltzmann Machine (RBM) initializes the encoder network by layer. The decoder network can be simply constructed by reversing the encoder and then tweaking it simply.

7 is a diagram showing a learning process of the deep automatic encoder.

In the fine tuning procedure of the automatic encoder network, an error back propagation is used, which is a powerful learning method for artificial neural networks.

After the automatic encoder network is established, the output vector generated in the code layer of this prediction module contains sufficient object behavioral fluidity potential information for reconstruction. When a person intends to do something, he usually uses the corresponding object. The potential code vectors can be used to integrate information about the co-occurrence of objects. Thus, code, which is object behavior fluidity information, has an inhibitory effect on irrelevant intentions and raises the intent of the intent.

At the top level, the object associated with the behavioral fluidic code is reconstructed. Even if the information of the object in the original vector is missing due to noise, the information can be released through the decoding process through the behavioral fluid code. This property can yield a prediction of the future state of the object even when the deep automatic encoder model is static. This is because the object selection occurs sequentially and missing information that is not observed at the current time can be predicted by behavioral fluidity and decoding.

8 is a diagram showing a structure of a prediction module of OA-SMTRNN of FIG.

OA-SMTRNN models can reconstruct related objects based on visual information, but in some cases, visible nodes due to noise do not get the correct information. In this case, the predicted object label from the behavior module of OA-SMTRNN in FIG. 2 may help this model to be robust against the noise problem.

The additional node that predicts the object label is connected to the hidden layer as the dynamic bias, and the weight between the predicted node and the hidden layer is learned. At the same time, the other weights are adjusted to the additional nodes to reconstitute the appropriate information. The detailed recognition-action cycle process is described below.

The structure of the model proposed in the present invention is shown in Fig. The behavior module shown on the left side of FIG. 2 can analyze the skeleton sequence for intent reasoning and is modeled on the basis of SMTRNN. The recognition module shown on the right of FIG. 2 is modeled using a deep automatic encoder to analyze and extract meaningful behavioral fluidity information. The object label is fed into the next deep automatic encoder predicted by the action module of OA-SMTRNN. As mentioned above, we have defined a new group of neurons to predict the object label associated with the behavior in the low-speed context layer.

Using the prediction result from the behavior module of OA-SMTRNN, the automatic encoder can be improved as shown in Equation (5) below.

&Quot; (5) "

는 숨겨진 뉴런 출력이고,

는 보이는 계층과 현재 숨겨진 계층 사이의 연결이며,

는 행동 모듈에서 인식 모듈로의 연결이고, b는 바이어스이다.here,

Is a hidden neuron output,

Is the link between the visible layer and the current hidden layer,

Is the connection from the action module to the recognition module, and b is the bias.

의 학습은 여전히 오류 역 전파 규칙을 따를 수 있다.As shown in the following Equation (6)

Learning can still follow the error back propagation rules.

&Quot; (6) "

는 자동 인코더에서 뉴런의 시냅스 이전 값이고

는 인간 행동 이해에 기반한 객체 예측 출력의 시냅스 이후 값이다.here,

Is the pre-synaptic value of the neuron in the auto-encoder

Is the post-synaptic value of the object prediction output based on human behavior understanding.

The predicted object label can then be viewed as the contextual bias of the recognition module implemented by the deep autocoder. Thereafter, the new input vector for the recognition module is: < EMI ID = 7.0 >

&Quot; (7) "

Where {,} denotes the vector connection, v _original is the original visible node of the deep automatic encoder, and y _a is the object prediction output from the behavior module of OA-SMTRNN.

The behavioral prediction signal is generated in the input-output layer with a small time constant, since the behavioral prediction signal is as fast as the original action sequence while the prediction node of the object and the object's prediction node are located in the low-speed context layer. Behavioral prediction also plays an essential role in learning. As already mentioned, mirror neurons are important in understanding the behavior of others and acquiring new skills through imitation.

The method of predicting the object label in the action module of OA-SMTRNN is placed as an additional visible node in the bottom layer of the deep automatic encoder. As a result, weights from the visible layer to the first hidden layer increase.

Thus, when a new sequence is obtained, the task module of OA-SMTRNN is used to estimate possible objects based on current and past task sequences. The additional information assumes that the recognition module can help to exclude unrelated objects and to reconstruct object labels in case of confusing or false object detection.

In the above, the improvement from behavior to recognition is explained. In the following, improvement from recognition to action is explained.

The primary goal of MTRNN is to predict dynamic signals. Thus, the MTRNN includes multiple CTRNN layers with different time constants at each layer. The information in the CTRNN layer with a small time constant (fast context) changes rapidly, but the slow context layer arranges the order of basic behavior information stored in the fast context layer.

Learn the behavioral fluidity model using the methods introduced in the previous model. The decoder portion is required to learn the depth automatic encoder, but the code layer output of the recognition module and the deep automatic encoder are considered as additional inputs to the slow context layer of the behavior module. The weights between the code and the slow context hierarchy are fully concatenated. The weight between the code and the slow context layer is updated to Equation (3) using the back propagation rule with the Kullback-Leibler divergence error of Equation (1).

Information stored in the code layer is related to intention (Kim, et al., 2013). They are trained with different weights of OA-SMTRNN. The behavioral fluidity model information for the behavioral model is indicated by an arrow in the code that makes the context hierarchy slow in Fig. 2, and enriches the functions necessary to accurately recognize human intentions.

The knowledge of the object behavior fluidity module is already abstracted and does not change rapidly. The output from the code layer of the recognition module is directly connected to the low-speed context layer. Originally, MTRNN and SMTRNN predicted action sequences based on the dynamics of latent states over time. However, by integrating this information, the intention classification function of OA-SMTRNN can simultaneously consider skeletal dynamics and object rationality in the integrated framework.

Since the object behavior fluidity awareness module is linked to the low-speed context layer, the information affects the input-output layer that predicts the behavior sequence. Thus, the recognition information helps the module to easily suppress unrelated behavior. If several intents share similar dynamic characteristics, the behavior prediction module of the lower layer provides a similar output. That is, although their characteristics are hardly distinguished, the encoded feature information of the behavioral fluidity recognition module can easily separate data by inserting feature dimensions related to object behavior fluidity.

The present invention expects that the information stored in the code layer will help OA-SMTRNN accurately predict the behavior sequence and improve the performance of the inference reasoning.

The present invention can provide a reliable user's intention recognizing device having flexibility and accuracy by combining a relationship model between objects having a static property and a user behavior recognition model having a dynamic property.

9 is a flowchart of a behavior-aware connection learning intention understanding method according to an embodiment of the present invention.

The behavior-aware connection learning intention understanding method according to this embodiment may proceed in substantially the same configuration as the device 10 of FIG. Therefore, the same constituent elements as those of the apparatus 10 of FIG. 1 are denoted by the same reference numerals, and repeated description is omitted.

Furthermore, the behavior-aware connection learning-based intention understanding method according to the present embodiment can be executed by software (application) for performing behavior-aware connection learning based intention understanding.

Referring to FIG. 9, the behavior-aware connection learning-based intention understanding method according to the present embodiment detects joint information of user behavior observed per frame and object information of the user's surrounding environment (step S10).

This can collect two-dimensional or three-dimensional positional information of the user's major joints in real time based on at least one of the visible light ray image sensor and the active infrared ray pattern projection sensor. If there are a large number of users in the sensing area of the sensor, at least one of face recognition or tracking can be performed to avoid interference of the behavior modeling information of each user.

The joint information of the user's actions may be a skeletal point such as midline, neck, head, left shoulder, left elbow, left wrist, right shoulder, right elbow, right wrist, left hip, right hip and shoulder have.

Also, in order to grasp the information of the surrounding object of the user, based on the visual sensor information of the visible ray,

When the joint information of the user's behavior and the object information of the user's surrounding environment are detected, the joint information and the object information are preprocessed so as to enable the artificial neural network processing (step S30).

In the preprocessing step (step S30), the process of pre-processing the joint information of the user behavior and the process of preprocessing the object information of the user's environment may be performed separately. In this case, the normalization step and the encoding step may be performed in each step .

Wherein the step of pre-processing the joint information of the user's action comprises: normalizing the joint-based information of the absolute coordinates based on the relative coordinates obtained by the user in the sensing area of the sensor, encoding the relative coordinate- can do. For example, the encoding step may use a self-organizing map (SOM).

The preprocessing of the object information of the user environment may extract the object information from the received image information, and may transmit the label of the object that the user holds by hand.

A behavior recognition processing step of classifying the behavior information of the user based on the joint information and the object information, which are pre-processed and output in the pre-processing step, is performed (step S50).

In the behavior recognition processing step (step S50), the joint information and the object information output from the preprocessing step are modeled through a regression neural network to designate a recognition dedicated node so that the behavior recognition can be performed. In addition, when learning a recognition-only node output corresponding to a given input in the real-time processing step and the testing step, the closest behavior is extracted by comparing with the user behavior vectors.

In step S70, the object relation information processing step is continuously performed to output the object candidate group related to the user's behavior using the object information, which has been pre-processed and output in the pre-processing step, and the behavior information output in the behavior recognition processing step.

The object information preprocessed and output in the pre-processing step and the behavior information output from the behavior recognition processing step are modeled through the self-encoding network, and the object self-encoding result is output.

An intention output step of finally outputting the user's intention recognition result through the artificial neural network inputting the behavior information output in the behavior recognition processing step and the object candidate group output from the object relationship information processing step is performed in step S90.

That is, only the user's intention recognition result is synthesized by integrating behavior and object relationship information into an artificial neural network or a regression neural network that receives the outputs of the behavior recognition processing step and the object relation information processing step as inputs.

As a result, reliable user intention results with flexibility and accuracy can be output, and can be used as a core underlying technology in all areas where it is necessary to recognize or predict a user's intention.

Such behavior-aware connection learning intention understanding methods can be implemented in an application or implemented in the form of program instructions that can be executed through various computer components and recorded on a computer-readable recording medium. The computer-readable recording medium may include program commands, data files, data structures, and the like, alone or in combination.

The program instructions recorded on the computer-readable recording medium may be ones that are specially designed and configured for the present invention and are known and available to those skilled in the art of computer software.

Examples of computer-readable recording media include magnetic media such as hard disks, floppy disks and magnetic tape, optical recording media such as CD-ROMs and DVDs, magneto-optical media such as floptical disks, media, and hardware devices specifically configured to store and execute program instructions such as ROM, RAM, flash memory, and the like.

Examples of program instructions include machine language code such as those generated by a compiler, as well as high-level language code that can be executed by a computer using an interpreter or the like. The hardware device may be configured to operate as one or more software modules for performing the processing according to the present invention, and vice versa.

It will be apparent to those skilled in the art that various modifications and variations can be made in the present invention without departing from the spirit or scope of the present invention as defined by the following claims. You will understand.

The present invention can be used as a core infrastructure technology in all areas where it is necessary to recognize or predict human intentions. For example, it can be a data analysis technology essential for many new growth businesses such as smart home and IOT, and it can secure competitiveness in various industries. Accordingly, the present invention is a core source technology that can be utilized in fields such as IT and Human Computer Interface (HCI) and signal processing, and can be applied to smart home, IoT, healthcare, security, and the like.

10: Behavior-aware connection learning-based intent understanding device
100: Input unit
300:
310: Joint information preprocessing section
330: object information preprocessing unit
500:
510: Behavior recognition processor
530: Object relation information processor
550: intention output section

Claims (20)

An input unit for detecting joint information of user behavior observed every frame and object information of a user's surrounding environment;
A joint information preprocessing unit for normalizing and encoding the joint information received from the input unit; and an artificial neural network processing unit for detecting a label of specific object information selected by a user in a current frame among a plurality of object information constituting the user peripheral environment A preprocessing unit configured to preprocess the object information;
A behavior recognition processing unit for classifying the behavior information of the user in the current frame based on the joint information and the specific object information output from the preprocessing unit;
An object relation modeling unit for modeling the object information through the auto-encoder network on the specific object information selected by the user in the current frame output from the preprocessing unit and the behavior information of the user in the current frame output from the behavior recognition processing unit, An object relationship information processor for estimating an object predicted to be selected by a user in a next frame among the plurality of object information and outputting the estimated object as an object candidate group; And
And an intention output unit for outputting a user's intention recognition result through an artificial neural network inputting behavior information output from the behavior recognition processing unit and an object candidate group output from the object relationship information processing unit,
The object relationship information processing unit,
Understanding the behavior-aware connection learning intention that learns the weight between the prediction node and the hidden layer using the object candidate group and reconstructs the missing object by rearranging the remaining weights excluding the weight between the prediction node and the hidden layer according to the learning result Device.
2. The apparatus according to claim 1,
Wherein the two-dimensional or three-dimensional position information of the user's major joints is collected in real time on the basis of information of at least one of the visible light ray image sensor and the active infrared ray pattern projection sensor.
3. The apparatus according to claim 2,
Wherein the at least one of the face recognition or tracking function is performed to avoid interference of the behavior modeling information of each user when there are a plurality of users in the sensing area.
delete The apparatus according to claim 1, wherein the joint information pre-
A normalization unit for normalizing major joint information based on an absolute coordinate obtained in a sensing region of the input unit to a relative coordinate representation for each user; And
And an encoding unit for expressing the normalized relative coordinate representation using a neural network input method using a self-organizing map (SOM).
delete The apparatus according to claim 1,
A behavior-aware connection learning based intention understanding device for modeling the joint information and the specific object information output from the preprocessing part through a regression neural network to designate a recognition only node so that behavior recognition can be performed.
8. The apparatus according to claim 7,
A behavior-aware connection learning intention understanding device for extracting the closest behavior compared with user behavior vectors when learning a recognition-only node output corresponding to a given input in a real-time processing step and a test step.
delete The method according to claim 1,
The joint information of the user's action includes at least one skeletal point of the mid-spine, neck, head, left shoulder, left elbow, left wrist, right shoulder, elbow right, right wrist, left hip, right hip and shoulder A behavior - aware connection learning - based intent understanding device used.
Behavior-aware Connected Learning Understanding the Intention Behavior-aware Connected Learning Performed by the Device In learning-based intention understanding methods,
The behavior-aware connection learning-based intention understanding device comprising: detecting joint information of user behavior observed every frame and object information of a user's surrounding environment;
Wherein the behavior-aware connection learning-based intention understanding device normalizes and encodes the joint information, detects a label of a specific object information selected by a user in a current frame among a plurality of object information items constituting the user's peripheral environment, A preprocessing step for preprocessing to enable the preliminary processing;
A behavior recognition processing step of classifying the behavior information of the user in the current frame based on the joint information and the specific object information output from the preprocessing step;
The behavior-aware connection learning intention understanding device may use the behavior information output from the behavior information recognition step and the specific object information output in the pre-processing step to select a user in the next frame of the plurality of object information An object relation information processing step of estimating a predicted object and outputting the estimated object as an object candidate group; And
The behavior-aware connection learning-based intention understanding device includes an intention to output a user's intention recognition result through an artificial neural network inputting behavior information output from the behavior recognition processing step and an object candidate group output from the object relationship information processing step Output stage,
In the object relation information processing step, the weight between the prediction node and the hidden layer is learned using the object candidate group, and the missing object is reconstructed by re-adjusting the remaining weights excluding the weight between the prediction node and the hidden layer according to the learning result Further comprising a behavior-aware connection learning-based intent understanding method.
12. The method of claim 11, wherein detecting the object information of the user environment comprises:
Based learning learning based on the knowledge of at least one of a visible light ray image sensor, an active infrared ray image sensor, and an active infrared ray pattern projection sensor to collect two-dimensional or three-dimensional position information of a user's major joint in real time.
13. The method of claim 12, wherein detecting the object information of the user environment comprises:
Wherein the at least one of the face recognition or tracking function is performed to avoid interference of the behavior modeling information of each user when there are a plurality of users in the sensing area.
12. The method according to claim 11,
Normalizing the major joint information based on the absolute coordinates obtained in the sensing region into a relative coordinate representation per user; And
And a coding step of representing the normalized relative coordinate representation using a neural network input method using a self-organizing map (SOM).
delete delete 12. The method according to claim 11,
A behavior-aware connection learning based intention understanding method for modeling the joint information and the object information output from the preprocessing step through a regression neural network to designate a recognition only node so that the behavior recognition can be performed.
18. The method according to claim 17,
A behavior-aware connection learning intention understanding method for extracting the closest behavior by comparing with user behavior vectors when learning a recognition-only node output corresponding to a given input in a real-time processing step and a testing step.
delete 12. A computer-readable recording medium on which a computer-readable recording medium for performing a behavior-aware connection learning based intention understanding method according to claim 11 is recorded.

Images