KR102095089B1 - Digital agent embedded mobile manipulator and method of operating thereof - Google Patents

Abstract

A digital agent mobile manipulator and a method of its operation are presented. The digital agent movement manipulator according to an embodiment includes: a head unit including a photographing unit acquiring image information and an angle control unit controlling an angle of rotation of the photographing unit; And a body part disposed under the head part and comprising a manipulator having a plurality of joints for work, and the angle control part of the head part can rotate the imaging part independently of the body part.

Description

DIGITAL AGENT EMBEDDED MOBILE MANIPULATOR AND METHOD OF OPERATING THEREOF

The following embodiments relate to a digital agent mobile manipulator and a method of operation thereof, and more specifically, a new mobile manipulator that integrates a social robot agent capable of omni-directional operation and emotion-based conversation. ) It is about the platform.

A remote robot having a mobile platform is an omni-directional mobile platform equipped with a manipulator and manipulator that performs various manipulations such as a human hand. Can be configured. In this case, the manipulator and the mobile platform need to be operated independently of each other, but in general, a user operating a remote robot using two arms (or two hands) has a problem that it is difficult to control both the manipulator and the mobile platform at the same time. .

Among the patents related to the existing mobile manipulator, Korean Patent Publication No. 10-2006-0113099 is an invention relating to a mobile manipulator that works and a vision robot that automatically detects and photographs the work. However, it is difficult to work efficiently in a small indoor space because the working robot and the robot shooting the environment are separated. In addition, Korean Patent Publication No. 10-2012-0047490 relates to a domestic assistant robot manipulator, and two articulated manipulators are attached on a wheel base. However, there are no cameras or robot face parts, and both manipulators can only work forward. Such a structure is more inefficient in a narrow indoor space because the entire body must be rotated to control the work.

Korean Patent Publication No. 10-2006-0113099

 K.-H. Han and J.-H. Kim, "Quantum-inspired evolutionary algorithm for a class of combinatorial optimization," IEEE Trans. Evol. Comput., Vol. 6, no. 6, pp. 580-593, 2002.  J. Yun, J. Lee, D. Han, J. Ju, and J. Kim, "Cost-efficient 3D face reconstruction from a single 2D image," in Proc. Int. Conf. Advanced Communications Tech., Pyeongchang, Republic of Korea, Feb. 2017.  W. Hwang, W. Kim, S. Suh, J.-J. Han, and J. Kim, "Face recognition using average example image," Electron. Lett., Vol. 50, no. 25, pp. 1921-1923, 2014.  H. Yoo, Y. Suh, and H. Kim, "Voice conversion in HTS using VTLN warping factor transform," in Proc. Korean Society Speech Sci., Seoul, Republic of Korea, Nov. 2016, pp. 141-142.

Embodiments describe a digital agent mobile manipulator and a method of operation thereof, and more specifically, a new mobile manipulator platform that integrates a social robot agent capable of omni-directional operation and at the same time capable of emotion-based conversation. to provide.

In the embodiments, the robot does not only perform a simple task, but creates a person's appearance and personality based on digital DNA to interact with the people around the robot, and implements the creature in the manipulator, thereby allowing the robot to communicate It is to provide a digital agent mobile manipulator that can interact with a user by inferring context information about the surrounding environment and reflecting it on a facial expression, and a method of operating the same.

In addition, embodiments are provided to construct a manipulator on a wheel base and attach a tablet robot face and a camera to take up a small space as a whole, thereby providing a digital agent movement manipulator capable of efficiently moving and working in an indoor environment and a method of operating the same. .

The digital agent movement manipulator according to an embodiment includes: a head unit including a photographing unit acquiring image information and an angle control unit controlling an angle of rotation of the photographing unit; And a body part disposed under the head part and comprising a manipulator having a plurality of joints for work, and the angle control part of the head part can rotate the imaging part independently of the body part.

It is disposed on the lower portion of the body, it may be made to further include a leg portion that is configured at least one or more wheels for movement.

The photographing unit of the head portion may be a camera that acquires the surrounding image and transmits it to a user through a communication module.

The head portion may further include a display module displaying a robot face and displaying an expression based on information previously described in a conversation.

The display module generates and implements a specific person or unique appearance or personality based on digital DNA (Deoxyribonucleic Acid), and deduces context information about the surrounding environment during conversation and reflects it on the user's face Can interact with.

The display module includes a display unit that displays an expression and a mouth shape during a conversation through a robot face; A microphone that recognizes external voice; A speaker that outputs voice; And a robot face, facial expression, to analyze voice recognized through the microphone, obtain context information through the recognized voice or an image obtained through the photographing unit, and output voice through the speaker and display through the display unit. It may include a control unit for determining the shape of the mouth.

And the display module is made of a tablet (tablet) PC, the digital agent mobile manipulator is implemented to occupy a small space as the body is formed, the manipulator is formed on the top of the wheel base, attached to the tablet PC indoors Efficient movement and work is possible in an environment or a narrow environment.

The angle control unit of the head may control a rotation angle using a motor having a pan and tilt function.

The body portion may include: a first rotating working motor that performs control for moving the manipulator to a working position; And a second rotating working motor which is arranged to be spaced apart from the first rotating working motor at a predetermined distance and works in opposition to the first rotating working motor, and includes the first rotating working motor and the second rotating working motor. Is made of a pair so that the rotation operation of the manipulator and the rotation of the head are made independently.

An end effector for working may be attached to the end of the manipulator.

The leg unit may further include an operation control unit that controls a movement direction and speed through the wheel according to a user's remote control, and controls a rotation angle of the angle control unit and an operation of the manipulator.

The display module is connected to a user terminal and receives the result of inferring the user's context information in real time to display an expression conforming to the digital DNA encoded in relation to the character, appearance, and voice of the AE robot or to output a voice, The user terminal can extract the selected features from the sensor data collected from the user's terminal and infer the situation information of the user in real time using a QEA-NN (Quantum-inspired Evolutionary Algorithm-based Neural Network). .

The user terminal is connected to the display module through execution of an application, and uses the learned neural network to infer real-time context information of the user related to location, user behavior, smart phone status information, and time information, and the user By selecting a specific digital DNA, information on personality, appearance, and voice encoded with the digital DNA can be transmitted to the display module.

The display module receives the selected digital DNA from the user terminal and expresses personality, appearance, and voice, and is based on personality DNA based on the Big 5 personality model, openness, conscientiousness, and extroversion to experience. 3D model using dimensionality reduction method through Principal Component Analysis (PCA) based on external DNA to determine personality by expressing with 5 characteristics of (extraversion), affinity, and neuroticism A face is generated, and a response generated from the user terminal based on the voice DNA can be output as a unique voice or a voice similar to a specific individual using a voice excitation signal and a vocal track spectrum model.

The digital agent mobile manipulator according to another embodiment is installed on the manipulator and the manipulator and is connected to a user terminal to receive the result of inferring the user's context information about the surrounding environment in real time, the personality, appearance and voice of the AE robot It may include a display module that outputs a voice or displays an expression conforming to an encoded digital DNA (Deoxyribonucleic Acid).

The user terminal is connected to the display module through execution of an application, extracts selected features from sensor data collected from the user's terminal, and uses QEA-NN (Quantum-inspired Evolutionary Algorithm-based Neural Network) The user's context information can be inferred in real time, and as the user selects a specific digital DNA, information on personality, appearance, and voice encoded with the digital DNA can be transmitted to the display module.

The display module receives the selected digital DNA from the user terminal and expresses personality, appearance, and voice, and is based on personality DNA based on the Big 5 personality model, openness, conscientiousness, and extroversion to experience. 3D model using dimensionality reduction method through Principal Component Analysis (PCA) based on external DNA to determine personality by expressing with 5 characteristics of (extraversion), affinity, and neuroticism A face is generated, and a response generated from the user terminal based on the voice DNA can be output as a unique voice or a voice similar to a specific individual using a voice excitation signal and a vocal track spectrum model.

According to the embodiments, the robot is not only a simple task, but a robot is generated by generating a person's appearance and personality based on digital DNA and interacting with people around the robot to implement the creature in a manipulator. It is possible to provide a digital agent mobile manipulator that can interact with a user by inferring context information about the surrounding environment in a conversation and reflecting it on an expression, and a method of operating the same.

In addition, according to embodiments, a manipulator is configured on a wheel base and a tablet robot face and a camera are attached to implement a space that occupies a small space, thereby providing a digital agent movement manipulator capable of efficient movement and operation in an indoor environment and a method of operating the same can do.

1 is a diagram schematically showing a digital agent movement manipulator according to an embodiment.
2 is a diagram for explaining QEA for learning algorithm optimization according to an embodiment.
3 is a diagram for explaining encoding of personality DNA using QEA-NN according to an embodiment.
4 is a diagram illustrating a linear combination of a 3D shape of a human face image and a target face used to reconstruct a texture according to an embodiment.
5 is a view showing a result of a 3D model of a face according to an embodiment.
6 shows a speech DNA-based speech synthesis system according to an embodiment.
7 is a diagram for describing context information recognition using QEA-NN according to an embodiment.
8 is a diagram illustrating a training result of QEA-NN for context information recognition according to an embodiment.
9 is a diagram schematically showing the structure of an AE robot according to an embodiment.
10 shows an example of a robot platform according to an embodiment.
11 is a diagram illustrating the strength of a gesture according to various characteristics of an agent according to an embodiment.
12 is a diagram illustrating the intensity of expression according to various situation information and personality of an agent according to an embodiment.
13 is a diagram illustrating an example of a snapshot of a video clip that performs expression for an anger scenario according to an embodiment.
14 is a diagram illustrating an example of a snapshot of a video clip performing expression for a greeting scenario according to an embodiment.

Hereinafter, embodiments will be described with reference to the accompanying drawings. However, the described embodiments may be modified in various other forms, and the scope of the present invention is not limited by the embodiments described below. In addition, various embodiments are provided to more fully describe the present invention to those skilled in the art. The shape and size of elements in the drawings may be exaggerated for a more clear description.

The following embodiments are a new mobile manipulator platform that integrates a social robot agent capable of omni-directional work and at the same time capable of emotion-based conversation.

The present invention relates to a mobile manipulator design and its function. The details of the invention are as follows.

Efficient movement and work must be possible in an indoor environment. To this end, a multi-joint manipulator capable of omni-directional operation is attached to the lower body of the wheel. In addition, to control the robot remotely, it recognizes the surroundings of the robot and attaches a camera to be transmitted to the user.

Robots do not just do simple tasks, they interact with people around them. Creates a person's appearance and personality based on digital DNA and embodies the creature in a manipulator. Accordingly, the robot makes a facial expression and acts when talking, infers context information about the surrounding environment, and also implements a function that is reflected in the facial expression during the conversation. To implement and express this, a tablet PC-based robot face is attached.

In order to behave effectively in an indoor space, the design of the robot should be as compact as possible. To this end, a manipulator is mounted on the wheel base, and a tablet robot face and a camera are attached thereon to occupy only a small space. However, when interacting and recognizing the surroundings, it is designed to move independently of the manipulator, so as not to be influenced by the manipulator at work.

1 is a diagram schematically showing a digital agent movement manipulator according to an embodiment.

Referring to FIG. 1, the digital agent movement manipulator 100 according to an embodiment may include a head part 120, a body part 110, and a leg part 130.

The head 120 may include a photographing unit and an angle control unit 123, and may further include a display module 121 according to an embodiment. In addition, the head unit 120 may further include a frame 122 for fixing the photographing unit and / or the display module 121.

The photographing unit may acquire image information. For example, the photographing unit may be a camera that acquires the surrounding image and transmits it to a user through a communication module.

The angle control unit 123 may control the rotation angle of the photographing unit. For example, the angle control unit 123 may control a rotation angle using a motor having a pan and tilt function.

In addition, the display module 121 may display a robot face and display an expression based on previously described information in a conversation. More specifically, the display module 121 generates and implements a specific person or a unique appearance or personality based on digital DNA (Deoxyribonucleic Acid), induces context information about the surrounding environment in a conversation, and reflects it in a facial expression. Can interact.

For example, the display module 121 may include a display unit, a microphone, a speaker, and a control unit. Meanwhile, a photographing unit may be included in the display module 121.

The display unit may display an expression and a mouth shape during a conversation through a robot face, the speaker may recognize external voice, and the speaker may output voice. In addition, the controller analyzes the voice recognized through the microphone, acquires context information through the recognized voice or the image acquired through the photographing unit, and displays the voice to be output through the speaker and the robot face, facial expression, and mouth shape to be displayed through the display unit. Can decide.

In particular, the display module 121 may be formed of a tablet PC. Accordingly, the digital agent movement manipulator is formed to occupy a small space as the body portion 110 is formed with the manipulator 111 formed on the top of the wheel base, and is attached to the tablet PC, thereby enabling efficient movement and work in an indoor environment or in a narrow environment. It is possible.

The display module 121 is connected to a user terminal and receives a result of inferring the user's context information in real time to display an expression conforming to the digital DNA encoded in relation to the character, appearance, and voice of the AE robot or to output a voice. You can.

Meanwhile, the user terminal can extract the selected features from the sensor data collected from the user's terminal and infer the user's context information in real time using a Quantum-inspired Evolutionary Algorithm-based Neural Network (QEA-NN).

More specifically, the user terminal is connected to the display module 121 through the execution of the application, and uses the learned neural network to infer the user's context information related to location, user behavior, smart phone status information, and time information in real time. You can. In addition, as the user selects a specific digital DNA, information on personality, appearance, and voice encoded with digital DNA may be transmitted to the display module 121.

The display module 121 may receive the selected digital DNA from the user terminal and express personality, appearance, and voice. In other words, based on personality DNA, personality is determined based on the Big 5 personality model with five characteristics: openness to experience, conscientiousness, extraversion, affinity, and neuroticism. Then, a 3D model face can be generated using a dimensional reduction method through Principal Component Analysis (PCA) based on external DNA, and the response generated from the user terminal based on the speech DNA can be generated with a voice excitation signal. The vocal track spectrum model can be used to output a unique voice or a voice similar to a specific individual.

Such a function may be implemented through the display module 121, particularly a tablet PC, but may also be implemented through a separate control system. For example, it may be implemented through a separate control system configured in the leg part 130 or the body part 110 of the digital agent movement manipulator, or may be implemented together with the motion control part of the leg part 130.

The body portion 110 is disposed under the head portion 120, and a manipulator 111 having a plurality of joints 112 for work may be configured. At this time, the angle control unit 123 of the head unit 120 may rotate the imaging unit independently of the body unit 110.

The body portion 110 may include a first rotating working motor 114 and a second rotating working motor 115. The first rotation work motor 114 may perform control for moving the manipulator 111 to a work position. In addition, the second rotational work motor 115 is arranged to be spaced apart from the first rotational work motor 114 by a predetermined distance, and may operate in opposition to each other in conjunction with the first rotational work motor 114. In this way, the first rotating working motor 114 and the second rotating working motor 115 are formed in a single pair so that the rotating operation of the manipulator 111 and the rotation of the head 120 are independently made.

In addition, an end effector 113 for working may be attached to the body 110 at the end of the manipulator 111.

The leg portion 130 is disposed under the body portion 110, and at least one wheel 131 for movement may be configured. For example, the at least one wheel 131 may be configured as an omni wheel.

In addition, the leg unit 130 further adjusts the movement direction and speed through the wheel 131 according to the user's remote control, and further includes an operation control unit that controls the rotation angle of the angle control unit 123 and the operation of the manipulator 111. It can contain.

The digital agent mobile manipulator according to another embodiment may include a manipulator and a display module.

The manipulator may perform the same function as that of the body 110 or the body 110 of the digital agent mobile manipulator 100 according to the above-described embodiment, and repeated description will be omitted.

The display module is installed on the manipulator, and connected to the user terminal, receives the result of inferring the user's situation information about the surrounding environment in real time, and expresses the expression following the digital DNA encoded in relation to the personality, appearance, and voice of the AE robot. You can display or output audio.

Also, the display module can receive the selected digital DNA from the user terminal and express personality, appearance, and voice. More specifically, personality is represented by five characteristics of openness, conscientiousness, extraversion, affinity, and neuroticism for experience according to the Big 5 personality model based on personality DNA. Is determined, and a 3D model face is generated using a dimensional reduction method through Principal Component Analysis (PCA) based on the external DNA, and the response generated from the user terminal based on the speech DNA is combined with a voice excitation signal. The vocal track spectrum model can be used to output a unique voice or a voice similar to a specific individual.

Here, the user terminal is connected to the display module through the execution of the application, extracts selected features from sensor data collected from the user's terminal, and uses QEA-NN (Quantum-inspired Evolutionary Algorithm-based Neural Network) in real time. The user's context information can be inferred. In addition, as the user selects a specific digital DNA, information on personality, appearance, and voice encoded with digital DNA may be transmitted to the display module.

In particular, the display module may be made of a tablet PC, and may perform the same function as the display module 121 of the digital agent mobile manipulator 100 according to one embodiment described above, and repeated descriptions will be omitted.

As described above, for example, the digital agent movement manipulator is equipped with a multi-joint manipulator on a mobile robot platform composed of a wheel, and an end effector for work may be attached to the end. A camera and a display module are attached to the top of the manipulator to acquire image information for work using a multi-joint manipulator. The camera and display module can control the angle of rotation using a pan and tilt motor.

In particular, two manipulator rotation operation motor pairs are used to independently rotate the manipulator rotation and the rotation of the camera and display modules. When the rotating working motor located at the bottom performs control to move the desired manipulator to the working position, the rotating working motor located at the upper part is controlled in the opposite direction by interlocking with the lower motor. Therefore, the rotation operation of the manipulator does not affect the viewing angles of the camera and display module located above the manipulator, and rotation angle control using a fan motor is independently performed.

In addition, voice is recognized through a microphone included in the attached display module, and dialogue is expressed through a speaker. In addition, the display expresses the faces, expressions, and mouth shapes of the digital creatures.

Therefore, various operations can be efficiently performed in a small indoor space through a mobile manipulator capable of omnidirectional operation. In addition, it is possible to recognize the environment by using a camera module that moves independently of the manipulator and perform intelligent tasks appropriate to it, and remote control is also possible by transmitting environmental information to the user. In the display module, it is possible to interact variously with people around it by displaying and recognizing and reproducing images according to the characteristics of the digital creatures implemented in the robot. Through this, various services can be expected to be provided.

According to the embodiments as described above, by providing a social robot platform that communicates on the basis of emotion and interaction with a person using an manipulator in an indoor environment, it can be applied to various fields such as a personal robot, a domestic help robot, and a social robot. have.

In the following, the configuration in which context information is inferred and reflected in the expression and interacts with the user will be described in more detail.

In order to interact emotionally with humans, the agent embedded (AE) system not only expresses human-like features externally, but also internally integrates human-like features such as personality, appearance, and voice. To this end, we propose a digital DNA concept to introduce human-like features to both the inside and outside of the AE robot. By encoding and decoding digital DNA, AE robots are provided with human-like features both inside and outside, allowing them to interact with humans. Using the proposed Quantum-inspired evolution algorithm-based neural network (QEA-NN), select key features from sensor data collected from the user's terminal (here, for example, a smartphone) and infer context information in real time do. The user's context information is dynamically reflected in the communication between the robot and the user. To demonstrate the effectiveness of the agent embedded system, an experiment was conducted using the humanoid robot Mybot developed by KAIST's Robot Intelligence Technology Lab.

HUMAN Robot Interaction (HRI) is one of the important research tasks in the field of social intelligence because robot systems continue to evolve beyond performing simple predefined motions for specific situations. Social intelligence has been proposed to be obtained from primates, especially human social life. Moreover, some researchers have argued that as a result of evolution, humans can use social intelligence to transfer their expertise from the social to the non-social. Therefore, social intelligence is regarded as a basic factor for developing artificial intelligence (AI) similar to humans as well as additional functions.

HRI is a research field to apply social intelligence to interaction with robots, and in this sense, social robots are being researched and developed. Social robots can be defined in a variety of ways, including robots that are socially evocative, socially located, social, socially intelligent, and socially interactive as described above. Social robots are practically applied in various fields such as interaction with children, elderly care, and treatment of autism. Understanding the meaning and intent of long-term interactions and human behavior is also being studied from the perspective of HRI.

With the continuous research on social robots and the development of human-like robots, social robots are able to express their feelings as well as simple conversation with people. Further research has been conducted to develop robots that mimic humans and perform tasks according to human actions. In addition, robots capable of imitating human personality have been developed, and robots exhibiting two types of personality have been reported.

The definition of social robots for socially interacting robots also includes the expression of such characteristics. As these developments continue and robots begin to emulate the characteristics of a particular person, humans can not only participate in intimate emotional interactions with robots, but also feel as if they are communicating long distances with real people, such as a deceased family member or entertainer .

In order to introduce human-like features into and out of the robot, a digital DNA (Digital Deoxyribonucleic Acid) concept for an agent inserted into the robot can be proposed, which is inspired by the concept of an artificial life gene. By encoding and decoding digital DNA, agent embedded (AE) robots can be provided with functions similar to humans, both internally and externally, and can take action or truly interact with humans.

When people communicate with each other, it is very important to be aware of the situation and properly reflect it in the interaction. If a robot can infer meaningful information about the user's situation as well as explicit information about its immediate environment, humans can interact with the robot emotionally.

Here, the term "context" is used to refer to the above information about the user's situation. To solve this problem, the QEA-NN (Quantum-inspired Evolutionary Algorithm-based Neural Network) optimization that selects key functions from the sensor data collected from the user's terminal (eg, a smart phone) and infers context information in real time. ). People have smart phones all day, so smart devices can be used to more accurately recognize user context information. The agent embedded can infer the user's context information using the sensor data collected from the smart phone and reflect it in the communication between the AE robot and the user.

The proposed AE system can be designed as a general system that can be applied to hardware platforms such as robots and smart devices. In fact, this system can be implemented using a 3D avatar as an agent for a smart phone and a humanoid robot. In one embodiment, the AE system may be implemented in a humanoid robot, and experimentally prove the operation of the system for a robot developed in KAIST's Robot Intelligence Technology Lab.

In the following, first, preliminary information for explaining QEA-NN is presented, and QEA-NN is applied twice in the system. It also explains the concept and context recognition module of digital DNA, shows how the system is designed and built on a robot platform, and the results of experiments with HRI applications.

2 is a diagram for explaining QEA for learning algorithm optimization according to an embodiment.

Referring to FIG. 2, it shows a QEA for learning algorithm optimization, where a neural network is used as a learning algorithm for QEA-NN. QEA-NN selects a core function as an input from all sets and can train an optimized neural network using QEA (Quantum-inspired Evolutionary Algorithm) (non-patent document 1). When multiple functions are provided as inputs to the neural network, QEA converges faster than the existing optimization algorithm and almost falls to the local minimum.

QEA-NN can use m sets of Q bits as follows, and can be expressed as the following equation.

[Equation 1]

는 다음과 같이 n 개의 Q 비트로 구성된다.Here, m is the population size for each generation t . Each Q bit element (individual),

Is composed of n Q bits as follows.

[Equation 2]

와 같이 표현되며, 이 행렬에서 n 은 j = 1, 2, ..., n 및 k = 1, 2, ..., m 특징의 개수이다.Where each Q bit is

It is expressed as follows, where n is the number of j = 1, 2, ..., n and k = 1, 2, ..., m features.

는 대응하는 k 번째 특징이 입력으로서 사용될 확률을 제공하고,

는 k 번째 특징이 입력으로 사용되지 않을 확률을 제공하므로,

이 된다. 각각의 Q 비트

는 관찰된 상태

로서 2진수를 갖는다. For each Q bit,

Gives the probability that the corresponding k- th feature will be used as input,

Gives the probability that the kth feature will not be used as input,

It becomes. Each Q bit

The observed state

As has a binary number.

에 대한 이진 개체

를 구성하고, m 개의 이진 개체 집합 X(t)는 각각 m 개의 Q 비트 개체 Q(t)에 해당하며, 다음 식과 같이 나타낼 수 있다.As shown below, if the value is 1, the corresponding k- th feature is used as the input of the neural network, and if the value is 0, it is not used. n observed states are corresponding Q bit entities

For binary objects

And m binary object sets X ( t ) each correspond to m Q bit objects Q ( t ), and may be expressed as follows.

[Equation 3]

, j = 1, 2, ..., n 및 k = 1, 2, ..., m는 이진수이다.here,

, j = 1, 2, ..., n and k = 1, 2, ..., m are binary.

The fitness function is defined as the sum of the error of the neural network and the number of features used as input as follows, and can be expressed as the following equation.

[Equation 4]

는 레이블이

인 경우 각 특징에 대한 생성에서 신경망으로부터 측정된 오차이며,

는 신경망의 출력이고, 트레이닝된 가중치 W 및 바이어스 b를 가지는

이며, X는 한 발생에서의 Q 비트 개체의 관측 상태이고,

은

-norm이다. here,

Has a label

Is the error measured from the neural network in the generation for each feature,

Is the output of the neural network, with trained weights W and bias b

Where X is the observed state of the Q bit entity in one occurrence,

silver

-norm.

는 해당 특성이 사용되거나 사용되지 않으면 각각 1 또는 0이므로, 모든 j = 1, 2, ..., n에 대한

의 합계는 값이 1 인

의 수와 동일하다. 즉,

는 신경망의 입력으로 사용되는 특성의 수이다. E 및

은 계수

에 따라 선형적으로 결합된다.As mentioned earlier, each observed state of the Q bit

Is 1 or 0, respectively, if that property is used or not, so for all j = 1, 2, ..., n

The sum of is 1

Is equal to the number of In other words,

Is the number of properties used as input to the neural network. E and

Silver modulus

According to linear combination.

The optimized Q bit used to select the characteristics to be used to ensure optimal classification accuracy in all processes of QEA-NN can be obtained by minimizing the fitness function.

Digital DNA represents a genetic expression that can define the properties of an agent. AE systems, including robots, can act like humans based on digital DNA. In this embodiment, the digital DNA includes information about characteristics expressed externally and internally by the agent. This allows the system to emotionally interact with humans like humans. The characteristics that have a great influence on emotional interaction are the appearance, voice, and personality of the agent expressed in conversation and behavior.

Because each characteristic has a different tendency, the digital DNA for each can be defined in different ways. Therefore, these characteristics can be extracted from a person to be encoded as digital DNA and then expressed as a characteristic by an agent acting like a human.

Definition 1. Digital DNA encoding the properties of a specific agent can be defined as the following equation.

[Equation 5]

Here, N is the number of DNA components encoded according to the type of encoded property. In this example, personal DNA (D _p ), external DNA (D _f ), and negative DNA (D _v ) are defined as examples.

Personality DNA ( D _p )

According to the Big 5 personality model, a person's personality can be represented by five characteristics: openness to experience, conscientiousness, extraversion, affinity, and neuroticism. These models can be used to generate personality DNA by assigning agent personality. By using DNA, the nature of each agent can be determined by how strong each property is.

Table 1 shows an example of log data collected from a smart phone.

[Table 1]

Since people follow a specific pattern when using a smartphone, according to people's personality, as shown in Table 1, a correlation between the log data of the smart phone and the personality of the user can be derived and applied to generate personality DNA.

The correlation between the log data of the smart phone and the personality of the user can be expressed as a weight in the neural network.

Table 2 shows examples of features extracted from smart phone log data.

[Table 2]

를 이용하여 산술 평균

, 표준 편차

, 엔트로피

및 프랙털 차원의 로그 데이터로부터 추출한 특징

는 다음 식과 같이 정의될 수 있다. The neural network includes features extracted as input from log data as listed in Table 2, and the ranking of each characteristic in the Big 5 personality model can be used as an output. Correlation integral

Arithmetic mean

, Standard Deviation

, Entropy

And features extracted from fractal-level log data

Can be defined as the following equation.

[Equation 6]

는 프랙탈(fractal) 차원의 특징이다. 상관 관계 차원을 사용하여 두 점을 연결하는 선(선의 길이가 배율 인수(scale factor) R보다 작음)의 수인 C를 계산하고, P는 맵의 모든 점의 수이며,

는 Heaviside 스텝 함수이며, 다음 식과 같이 나타낼 수 있다.Here, X _i is log data, L is the number of data of each source, m is the arithmetic mean of the data, M is the number of events equal to a specific location of each location data, p _j is the probability that the j th event will occur ( Example: probability of visiting the jth place).

Is a characteristic of the fractal dimension. Use the correlation dimension to calculate C, the number of lines connecting the two points (the length of the line is less than the scale factor R), P is the number of all points on the map,

Is a Heaviside step function and can be expressed as the following equation.

 [Equation 7]

Since not all log data affects people's personality, personality reasoning becomes more accurate when effective characteristics extracted from log data for input of neural networks are selectively applied. For this reason, it is possible to select a feature for each personality as an input of a neural network, and use QEA-NN to determine an optimized structure such as the number of nodes in each layer.

In this example, a database of 34 people was used. Their personality information was collected from five personality test surveys, and smartphone sensor data was collected for 50 days.

Table 3 shows the optimized results of Big 5 personality inference using QEA-NN.

[Table 3]

The characteristics selected as inputs in Table 3 were used to construct five personality characteristics for each personality DNA.

3 is a diagram for explaining encoding of personality DNA using QEA-NN according to an embodiment.

As shown in FIG. 3, each DNA can be expressed by the output of each neural network selected and trained by QEA-NN. Finally, the personality DNA (D _p ) expressing the personality of the big 5 can be defined as follows.

[Equation 8]

는 j = 1, 2, ..., n에 대해 QEA-NN을 사용하여 최적화된 신경망의 입력으로 채택된 특징 중 j 번째 특징의 실제 값이고, n은 선택된 특징의 수이며 N_p = n이다.
From here,

Is the actual value of the j th feature among the features adopted as the input of the optimized neural network using QEA-NN for j = 1, 2, ..., n , n is the number of features selected and N _p = n .

External DNA ( D _f )

People's appearance can be explained based on texture and shape. In the same way, the AE robot can have a unique face or look like a specific person, especially if the agent embedded in the robot has characteristics defined by appearance and expression. In this case, people have more emotional interactions with the AE robot and can feel as if the robot is communicating with a similar person.

4 is a diagram illustrating a linear combination of a 3D shape of a human face image and a target face used to reconstruct a texture according to an embodiment.

는 동일한 사람들에 대한 텍스처 및 3차원 형상 데이터베이스가 각각 선형으로 결합되는 방법을 나타내는 계수로서 사용될 수 있다. 텍스처 T와 형상 S의 특징을 미리 정의된 위치에 대해 추출하고, 각 위치에 대해, 평균값(텍스처에 대한

및 형상에 대한

)을 가산함으로써 대응하는 데이터베이스의 선형 조합으로서의 대상 얼굴을 나타낼 수 있다(비특허문헌 3).First, a face may be reconstructed as a 3D model using a dimensional reduction method such as Principal Component Analysis (PCA) (Non-Patent Document 2). As shown in FIG. 4, a coefficient representing a method in which a two-dimensional face image database is linearly combined.

Can be used as a coefficient indicating how each texture and three-dimensional shape database for the same people are linearly combined. Features of texture T and shape S are extracted for predefined locations, and for each location, the average value (for texture

And for shape

) Can be added to represent the target face as a linear combination of the corresponding database (Non-Patent Document 3).

[Equation 9]

Using the calculated coefficients, the external DNA (D _f ) of the agent's face can be defined as follows.

[Equation 10]

는 j = 1, 2, ..., n에 대한 j 번째 데이터에 대한 계수이고, n은 얼굴 재구성을 위해 사용된 이미지 데이터의 수이며, N_f = n이다. From here,

Is a coefficient for j- th data for j = 1, 2, ..., n , n is the number of image data used for facial reconstruction, and N _f = n .

5 is a view showing a result of a 3D model of a face according to an embodiment.

Referring to FIG. 5, a 3D model result (b) of the target face (a) may be represented, and more specifically, a 3D model of the face may be configured for the agent using external DNA. Accordingly, the agent looks like a person.

Another important role of appearance DNA is to enable the agent to express emotions with facial expressions. In addition to having a human-like face, the agent must be able to open his mouth while speaking or express different faces depending on his emotions. With this configuration, the user talking to the AE robot can feel closer as a companion. To achieve this, the coefficients of [Equation 9] described above can be used with different facial expressions with different person's face databases.

Negative DNA ( D _v )

The human voice also expresses the individuality of the individual. If the agent built into the system, including the robot, can make its own voice, people will feel more active and intimate communication with the agent.

The human voice can be divided into a voice excitation signal and a vocal track spectrum model, and each model can be parameterized as a feature. The extracted features are vectorized into components of negative DNA. After the negative DNA is used to train a speech synthesis system based on HMM (Hidden Markov Model), the human speech can be synthesized by the system (Non-Patent Document 4).

6 shows a speech DNA-based speech synthesis system according to an embodiment.

Referring to FIG. 6, when defining individual negative DNA for each agent, the agent can have a unique voice or a voice similar to a specific individual.

The negative DNA (D _v ) for the agent's voice can be defined as the following equation.

 [Equation 11]

는 j = 1, 2, ..., n ₁에 대한 음성 여기 신호로부터 추출된 특징 벡터이고,

는 j = 1, 2, ..., n ₂에 대한 보컬 트랙 스펙트럼 모델로부터 추출된 특징 벡터이며, n ₁과 n ₂는 해당 특징 벡터의 구성 요소 수이고, N_v = n ₁ + n ₂이다.From here,

Is a feature vector extracted from the speech excitation signal for j = 1, 2, ..., n ₁ ,

Is a feature vector extracted from the vocal track spectrum model for j = 1, 2, ..., n ₂ , n ₁ and n ₂ are the number of components of the feature vector, and N _v = n ₁ + n ₂ .

When people understand their personal situation and the situation of the person they are talking to while they are talking, they can interact in a more natural and emotional way. An AE system including a robot can feel as if a person is speaking and communicating emotionally if the AE robot is able to recognize and understand human context information. To this end, a module for recognizing information about the user's context information in real time can be constructed.

Recently, people live on their smartphones all day long. When the user's context information is recognized using various sensors mounted on the smart phone, meaningful information on the corresponding situation can be obtained in real time.

To this end, it is possible to provide a module that infers and recognizes the user's context information using a sensor embedded in the smart phone, and then applies it to human robot interaction. That is, a context recognition system for interaction with a user can be constructed. In this embodiment, it is possible to provide a neural network that uses the main characteristics of sensor data collected from a smart phone as an input and uses context information of a user as an output. QEA-NN can select the features used as input and optimize the network structure. The entire module can be implemented in a smart phone. Meanwhile, a smart phone is described in more detail as an example, but not only a smart phone, but also a mobile phone, a portable multimedia player (PMP), a mobile internet device (MID), a desktop, and a tablet computer (Tablet PC), a notebook (Note book), a netbook (Net book) and a terminal having various communication functions such as an information communication device may be used.

7 is a diagram for describing context information recognition using QEA-NN according to an embodiment.

There are several features that can be extracted from smartphone sensor data. The reasoning accuracy can be increased by using a user's context information recognition process that selects a valid feature from all data. To this end, it is possible to select a feature selected as the input of the recognition neural network, and construct a neural network having an optimal structure using QEA-NN as shown in FIG. 7. The context information recognition 700 using the QEA-NN may perform the context information recognition 730 using the QEA-NN 720 after the feature extraction 710.

For example, for feature extraction 710, 14 types of data can be collected from a smart phone, including 3-axis gyroscope, 3-axis direction, luminance, proximity, noise, longitude, and latitude GPS. .

, 분산

, 평균 제곱근

및 평균 절대 편차

의 4 가지 특성 세트인

가 각 센서 데이터 유형에 대해 다음 식과 같이 계산될 수 있다.And average

, Dispersion

, Mean square root

And mean absolute deviation

The four characteristic sets of

Can be calculated as the following equation for each sensor data type.

[Equation 12]

56 extracted features may be input to the QEA-NN 720, and a neural network input for context recognition may be selected. The optimized structure of the neural network can also be selected using QEA-NN.

8 is a diagram illustrating a training result of QEA-NN for context information recognition according to an embodiment.

로 신경망을 최적화하였다. 마지막으로, 31 개의 특징을 선택하여

정확도를 수행하였다.Referring to FIG. 8, the neural network was optimized for 20 generations using a fitness function,

Optimized the neural network. Finally, by selecting 31 features

Accuracy was performed.

Since the collected sensor data and predefined context labels were used in QEA-NN, we built a neural network for context information and trained optional features as input. Based on these characteristics, the recognized context information is divided into three classes according to each characteristic: location, user behavior, and smartphone status information, and each class has different labels. The trained (neural) network, as well as information about the features used, is stored in the smart phone, and the user's context information is inferred by the neural network while the sensor data is collected in real time.

For example, context information recognized as four classes may be represented. That is, the estimated location, user behavior, smart phone status information, and time information are context information finally recognized by the user. Using this information, you can create an agent's conversations and gestures, and perform emotional interactions with humans.

9 is a diagram schematically showing the structure of an AE robot according to an embodiment.

Robots are not only for interacting with humans, but also for performing tasks on behalf of humans. If the humanoid robot can communicate with humans emotionally, it will be able to truly communicate with humans. AE robots that interact with humans can be implemented by integrating digital DNA and agents that recognize the user's context information into the robot platform.

Referring to FIG. 9, since the AE robot according to an embodiment must have a built-in function for collecting sensor data from a smart phone, the smart phone system 910 and the robot system 920 are separately designed and integrated. The smart phone system implemented in the smart phone may recognize what the user 930 has said and generate a response transmitted to the robot system 920 together with the user context information inferred from the smart phone in real time. Then, the robot system 920 may speak sentences associated with facial expressions and gestures that follow the encoded digital DNA in relation to the appearance, voice, and personality of the AE robot according to the user's context information.

The context information recognition module of the smart phone system 910 is automatically started when the application 911 of the smart phone is started and may be connected to the AE robot. The context information recognition module may collect sensor data from a smart phone and infer real-time user context information related to location, user behavior, smart phone status information, and time information using a learned neural network. When the user 930 selects a specific digital DNA before communicating with the AE robot, information on personality, appearance, and voice encoded with the digital DNA may be transmitted to the robot system 920.

During the conversation, the user 930 may speak something to the AE robot through the microphone of the smart phone, and the voice recognition module may recognize the voice of the user 930. Then, a response such as a sentence may be generated in the dialogue generation module. The generated sentence and recognized context information can be transmitted to the AE robot.

The robotic system 920 can be implemented on an AE robotic platform and can be used to represent digital DNA and deliver generated responses to voices, facial expressions and gestures. When the selected digital DNA is transmitted from the smart phone, the corresponding personality, appearance and voice can be expressed. In addition, a 3D model face constructed based on external DNA may be generated. If the generated response is received during a conversation with the user 930, the AE robot can synthesize and reproduce the response with a voice representing speech DNA. Gestures 921 and facial expressions 922 that match the characteristics based on the personality DNA and the context information sent from the smart phone may be performed while the AE robot generates speech.

Meanwhile, a recurrent neural network with parameteric biases (RNNPB) to which a facial bias bias is applied may be used to express facial expressions in a temporal order. The intensity of facial expressions representing specific emotions is changed according to the situation information and personality of the AE robot. After determining the number of primitive facial models representing emotions in RNNPB, the intensity of the generated facial expression sequence is changed. Face model at time t reflects the emotions that change the face of the model at time t +1 and generates status information. In RNNPB, the face models at times t and t +1 are input and output, respectively. Context information can be used as context, and emotion can be used as parameter bias.

To select appropriate gestures based on personality and reaction, several categories of gestures have been defined, and each category is divided into specific gestures by size. And, the selection process can be performed as follows.

[Equation 13]

가

보다 큰 경우 1,

가

보다 작거나 같은 경우 0이 된다. e는 j = 1, 2, 3, 4, 5에 대한 성분을 e _j 로 갖는 성격 DNA로부터 디코딩된 성격 벡터의 표현이다. s는 천이 행렬 H와 e를 곱하여 계산된 각 제스처에 대한 스케일 요소를 요소로 가지며, k는 제스처 범주의 수이고, c는 모든 s 및 l = 1, 2, ..., k의 원소들 중에서 최대 요소의 인덱스인 선택된 제스처이다. 본 실시예에서는 행복, 슬픔, 놀람, 분노, 혐오감의 5 가지 감정을 바탕으로 k = 5로 제스처 카테고리를 정의한다.
Here, e _j indicates whether or not each personality characteristic is expressed, and since the value is binary,

end

Greater than 1,

end

If less than or equal to 0. e is an expression of a personality vector decoded from personality DNA having components for j = 1, 2, 3, 4, 5 as e _j . s has the scale factor for each gesture calculated by multiplying the transition matrix H and e as an element, k is the number of gesture categories, and c is among all s and l = 1, 2, ..., k elements The selected gesture that is the index of the largest element. In this embodiment, a gesture category is defined as k = 5 based on five emotions: happiness, sadness, surprise, anger, and disgust.

10 shows an example of a robot platform according to an embodiment.

The robot platform shown in FIG. 10 is a robot platform developed by Robot Intelligence Technology Lab, Mybot, which implements a situational human interaction AE system that includes digital DNA in Mybot. Meanwhile, the robot platform 100 may include a head portion 120, a body portion 110, and a leg portion 130, as described above. A human-robot interaction experiment was performed to demonstrate that the AE robot can express a given digital DNA by interacting with the user and showing various expressions or gestures in various scenarios. In the experiment, external DNA and personalized DNA were expressed in Mybot.

The system was implemented in three modules and was integrated to function as one system by sending and receiving data between each other on the network. The smart phone system of the AE system was implemented in an Android smart phone. The robot system was implemented on Mybot's motherboard. The laptop controls the robot's body, and the tablet PC controls the robot's face. Equipment specifications are listed in Table 4.

[Table 4]

Robot control: Since Mybot was developed using the ROS (Robot Operating System), the AE robot and the laptop's motherboard operate as ROS nodes in the same ROS environment. In this experiment, in addition to the hardware controller package, two modules were implemented in the ROS package for the experiment. The speech synthesis module synthesizes speech from the received sentence and plays it through the speaker. NAVER Corporation's Text-to-Speech (TTS) open API was used to synthesize speech and save the results as an MP3 file.

The gesture performance package selects a gesture suitable for the current situation information by reflecting personality among gestures previously stored by the passive learning process. The passive learning process is a method in which the robot records the trajectory of the motor as a series of joint values and the duration between each step when the instructor moves a part of the robot. This method saves the trajectory as a ROS Bag file. When the selected gesture is played, the AE robot generates a gesture with a gesture.

11 is a diagram illustrating the strength of a gesture according to various characteristics of an agent according to an embodiment. The strength of the gestures according to the various characteristics of the agents listed in Table 5 is illustrated in FIG. 10.

[Table 5]

3D face control: In addition to the ROS package, we developed a face expression module on the tablet PC. Since Mybot's face is expressed as a 3D model using the Unity program, the module was implemented in Windows on the tablet PC. The module receives messages from the ROS package for the AE system. When information on the external DNA is received, a corresponding face of the 3D model is generated on the display. If the signal sends a signal to Mybot to start voice audio playback during a conversation where the mouth is repeatedly opened and closed, the transmission of the signal informing the playback signal is repeated until the end signal is transmitted.

12 is a diagram illustrating the intensity of expression according to various situation information and personality of an agent according to an embodiment. After speaking, an appropriate expression is selected and displayed according to personality as shown in FIG. 12.

Dialogue in the smartphone: Since the AE system uses a smartphone for context recognition and voice interaction, a smartphone-related module was implemented using the Android OS. Using the implemented Android application, the user is connected to the AE robot Mybot. When the user touches the connect button, the selected digital DNA information is sent to the ROS package and a face is created. When the user speaks on the smartphone, the application recognizes it and generates an appropriate response. On the display of the smart phone, the user can check (confirm) the recognition result and the generated sentence. The generated response is sent to the ROS package, and then Mybot reacts. We used SYSTRAN International Incorporation's speech recognition engine and the Korean module YALLY Incorporation's dialog creation API.

A series of experiments was performed to demonstrate the AE system for HRI applications. During the interaction with the AE robot, since the interactive AE system was implemented, we observed that facial expressions and gestures were changed according to context information based on various digital DNA. In order to prove that the function of the AE system was successfully implemented in the robot Mybot, two cases of the scenario were pre-defined and observed how the AE robot responds according to the scenario.

Anger scenario: When the user tells Mybot, "Can you hear me?", The AE robot expressed anger and responded, "I hear a very faint sound." Using this trend, the agent's reaction was observed while the sentence was repeatedly spoken to Mybot. Despite the same scenario, the agent responded with different facial expressions and gestures depending on the context information and the agent's personality.

13 is a diagram illustrating an example of a snapshot of a video clip that performs expression for an anger scenario according to an embodiment.

As shown in FIG. 13, the agent exhibits a more intensely angry facial expression from (a) to (c). The agent of (c) with the strongest facial expression tended to be similar to a gesture because he put his arms on his back.

Greeting Scenario: An experiment on the greeting scenario was also conducted. The greeting scenario is that the digital DNA was expressed differently when the AE robot and the user first met and greeted each other. When an agent with high extroversion and open personality DNA was embedded in the robot, the AE robot tried to hug the user while greeting. Mybot, with an extroverted and open agent, raised both arms and showed a big smile with a welcoming look. Nervous Mybot prevented users from coming closer.

14 is a diagram illustrating an example of a snapshot of a video clip performing expression for a greeting scenario according to an embodiment.

As shown in Fig. 14, various responses displayed by Mybot appear in the greeting scenario.

As described above, in the present embodiment, a digital DNA concept that enables the AE system including a humanoid robot to express characteristics of a specific person can be provided. In particular, personality DNA, external DNA and negative DNA were encoded. AE system based on digital DNA was designed and implemented in the real robot hardware platform Mybot. Through the proposed digital DNA, the agent embedded robot Mybot can act like a human and participate in speaking with various appropriate gestures and expressions.

The device described above may be implemented with hardware components, software components, and / or combinations of hardware components and software components. For example, the devices and components described in the embodiments include, for example, a processor, a controller, an arithmetic logic unit (ALU), a digital signal processor (micro signal processor), a microcomputer, a field programmable array (FPA), It may be implemented using one or more general purpose computers or special purpose computers, such as a programmable logic unit (PLU), microprocessor, or any other device capable of executing and responding to instructions. The processing device may run an operating system (OS) and one or more software applications running on the operating system. In addition, the processing device may access, store, manipulate, process, and generate data in response to the execution of the software. For convenience of understanding, a processing device may be described as one being used, but a person having ordinary skill in the art, the processing device may include a plurality of processing elements and / or a plurality of types of processing elements. It can be seen that may include. For example, a processing device may include a plurality of processors or a processor and a controller. In addition, other processing configurations, such as parallel processors, are possible.

The software may include a computer program, code, instruction, or a combination of one or more of these, and configure the processing device to operate as desired, or process independently or collectively You can command the device. Software and / or data may be interpreted by a processing device, or to provide instructions or data to a processing device, of any type of machine, component, physical device, virtual equipment, computer storage medium or device. Can be embodied in The software may be distributed on networked computer systems, and stored or executed in a distributed manner. Software and data may be stored in one or more computer-readable recording media.

The method according to the embodiment may be implemented in the form of program instructions that can be executed through various computer means and recorded on a computer readable medium. The computer-readable medium may include program instructions, data files, data structures, or the like alone or in combination. The program instructions recorded on the medium may be specially designed and configured for the embodiments or may be known and usable by those skilled in computer software. Examples of computer-readable recording media include magnetic media such as hard disks, floppy disks, and magnetic tapes, optical media such as CD-ROMs, DVDs, and magnetic media such as floptical disks. -Hardware devices specially configured to store and execute program instructions such as magneto-optical media, and ROM, RAM, flash memory, and the like. Examples of program instructions include high-level language code that can be executed by a computer using an interpreter, etc., as well as machine language codes produced by a compiler.

As described above, although the embodiments have been described by a limited embodiment and drawings, those skilled in the art can make various modifications and variations from the above description. For example, the described techniques are performed in a different order than the described method, and / or the components of the described system, structure, device, circuit, etc. are combined or combined in a different form from the described method, or other components Alternatively, even if replaced or substituted by equivalents, appropriate results can be achieved.

Therefore, other implementations, other embodiments, and equivalents to the claims are also within the scope of the following claims.

Claims (17)

In the digital agent mobile manipulator,
A head unit including a photographing unit acquiring image information and an angle control unit controlling the rotation angle of the photographing unit; And
It is disposed on the lower portion of the head, the body portion is composed of a manipulator is formed a plurality of joints for work
Including,
The head portion,
A display module that displays the face of the robot and displays facial expressions based on the previously described information during the conversation
It includes,
The angle control part of the head rotates the imaging part independently of the body part,
The display module,
Creates and implements a specific person or unique appearance or personality based on digital DNA (Deoxyribonucleic Acid), induces context information about the surrounding environment during conversation, reflects it in facial expressions, and interacts with the user, It is connected to the user terminal and receives the result of inferring the user's context information in real time to display the facial expressions that follow the digital DNA encoded in relation to the character, appearance, and voice of the AE robot, or to output a voice,
The user terminal,
Extracting selected features from sensor data collected from the user's terminal and inferring the user's context information in real time using QEA-NN (Quantum-inspired Evolutionary Algorithm-based Neural Network)
Characterized by, a digital agent mobile manipulator. According to claim 1,
A leg portion disposed at the lower portion of the body portion and configured with at least one wheel for movement
Further comprising, a digital agent mobile manipulator. According to claim 1,
The photographing unit of the head,
A camera that acquires surrounding images and transmits them to the user through a communication module
Characterized by, a digital agent mobile manipulator. delete delete According to claim 1,
The display module,
A display unit for displaying facial expressions and mouth shapes during a conversation through a robot face;
A microphone that recognizes external voice;
A speaker that outputs voice; And
Analyzes the voice recognized through the microphone, acquires context information through the recognized voice or the image acquired through the photographing unit, and outputs the voice through the speaker and the robot face, facial expression, and mouth to be displayed through the display unit Control unit that determines shape
Including, digital agent mobile manipulator. According to claim 1,
The display module,
It consists of a tablet PC,
The digital agent mobile manipulator,
The body part formed with the manipulator is formed on the upper part of the wheel base, and is implemented to occupy a small space as the tablet PC is attached, so that efficient movement and work can be performed in an indoor environment or a narrow environment.
Characterized by, a digital agent mobile manipulator. According to claim 1,
The angle control unit of the head,
Controlling the angle of rotation using a motor with pan and tilt functions
Characterized by, a digital agent mobile manipulator. According to claim 1,
The body portion,
A first rotating working motor that performs control to move the manipulator to a working position; And
A second rotating working motor which is arranged to be spaced apart from the first rotating working motor and operates in opposition to each other in conjunction with the first rotating working motor.
Including,
The first rotating working motor and the second rotating working motor are formed in a pair so that the rotating operation of the manipulator and the rotation of the head are made independently.
Characterized by, a digital agent mobile manipulator. According to claim 1,
The body portion,
An end effector for work is attached to the end of the manipulator.
Characterized by, a digital agent mobile manipulator. According to claim 2,
The leg portion,
An operation control unit that controls a movement direction and speed through the wheel according to a user's remote control, and controls a rotation angle of the angle control unit and an operation of the manipulator.
Further comprising, a digital agent mobile manipulator. delete According to claim 1,
The user terminal,
It is connected to the display module through the execution of the application, and uses the learned neural network to infer the user's situation information in real time related to location, user behavior, smart phone status information and time information, and the user selects a specific digital DNA To transmit information about the personality, appearance and voice encoded by the digital DNA to the display module
Characterized by, a digital agent mobile manipulator. The method of claim 13,
The display module,
The selected digital DNA is transmitted from the user terminal to express personality, appearance and voice, and based on personality DNA, openness, conscientiousness, extraversion, and extrinsic to experience according to the Big 5 personality model It is characterized by representing five characteristics of agreeableness and neuroticism, and a three-dimensional model face is generated using a dimensional reduction method through principal component analysis (PCA) based on external DNA, Outputting the response generated from the user terminal based on the speech DNA as a unique voice or a voice similar to a specific individual using a voice excitation signal and a vocal track spectrum model
Characterized by, a digital agent mobile manipulator. In the digital agent mobile manipulator,
Manipulator; And
Digital DNA (Deoxyribonucleic Acid), which is installed in the manipulator, is connected to the user terminal and receives the result of inferring the user's context information about the surrounding environment in real time, and is encoded in relation to the character, appearance, and voice of the AE robot. Display module that displays facial expressions following) or outputs voice
Including, digital agent mobile manipulator. The method of claim 15,
The user terminal,
It is connected to the display module through the execution of an application, extracts selected features from sensor data collected from the user's terminal, and uses the QEA-NN (Quantum-inspired Evolutionary Algorithm-based Neural Network) in real time. Inferring contextual information, and transmitting information on personality, appearance, and voice encoded with the digital DNA to the display module as the user selects a specific digital DNA
Characterized by, a digital agent mobile manipulator. The method of claim 16,
The display module,
The selected digital DNA is transmitted from the user terminal to express personality, appearance and voice, and based on personality DNA, openness, conscientiousness, extraversion, and extrinsic to experience according to the Big 5 personality model It is characterized by representing five characteristics of agreeableness and neuroticism, and a three-dimensional model face is generated using a dimensional reduction method through principal component analysis (PCA) based on external DNA, Outputting the response generated from the user terminal based on the speech DNA as a unique voice or a voice similar to a specific individual using a voice excitation signal and a vocal track spectrum model
Characterized by, a digital agent mobile manipulator.

Images