KR102147835B1 - Apparatus for determining speech properties and motion properties of interactive robot and method thereof - Google Patents

Abstract

Disclosed is an apparatus and method for determining speech and gesture properties of an interactive robot. The method for determining the speech and gesture properties of the interactive robot performed by the apparatus for determining the speech and gesture properties of the interactive robot according to the present invention includes at least one of voice information and image information about a counterpart interacting with the interactive robot. Receiving conversation information, analyzing the conversation information of the other party, and extracting at least one of a linguistic attribute and a non-verbal attribute of the other party, contextual context information estimated from the conversation contents of the interactive robot and the other party, Determining at least one of a speech attribute and a gesture attribute of the interactive robot based on at least one of the verbal attribute and the non-verbal attribute, and at least one of the determined speech attribute and the gesture attribute of the interactive robot Based on, controlling the operation of the interactive robot.

Description

A device and method for determining speech and gesture properties of an interactive robot {APPARATUS FOR DETERMINING SPEECH PROPERTIES AND MOTION PROPERTIES OF INTERACTIVE ROBOT AND METHOD THEREOF}

The present invention relates to a technique for determining speech and gesture properties of an interactive robot, and more particularly, to a technique for determining speech properties and gesture properties of a robot in a dialogue situation between a human and a robot.

In addition to industrial robots that were mainly applied to assembly lines in the manufacturing process, today's robots have a wide variety of uses and applications, including rescue robots at disaster sites, robots for education and childcare, care robots, and robots for killing in battlefields.

Recently, in the robotics world, conversational robot technology that helps people to avoid feelings of isolation is receiving great attention. Dialogue robots can communicate not only with humans, but also with robots, making eye contact with people around them, giving the impression that they are actually participating in the conversation.

When a person has a conversation, he freely changes the size, speed, and gestures of his speech according to the context of the conversation or the surrounding situation. However, robots according to the prior art may repeat only preset speech and gestures. That is, in the prior art, since all robots perform the same type of speech and the same type of gesture, environmental adaptability and the robot's personality cannot be reflected.

Accordingly, there is a need to develop a technology that enables the counterpart to feel the robot more intimately and intelligently by determining the speech and gesture properties of the robot in a situation where a human and a robot communicate.

Korean Patent Publication No. 10-2017-0027706, published on March 10, 2017 (Name: Method and system for managing conversations of robots)

An object of the present invention is to change the speech attribute and gesture attribute of the interactive robot in consideration of the verbal attribute of the conversation partner and the personality of the interactive robot.

In addition, an object of the present invention is to change the ignition property and gesture property of the interactive robot according to the counterpart and the surrounding environment, so that the counterpart can feel the robot more intimately and intelligently.

The method for determining utterance and gesture properties of an interactive robot performed by the apparatus for determining utterance and gesture properties of an interactive robot according to the present invention for achieving the above object is, among voice information and image information for a counterpart interacting with the interactive robot. Receiving a counterpart's conversation information including at least one, analyzing the counterpart conversation information, and extracting at least one of a linguistic attribute and a non-verbal attribute of the counterpart, conversation content between the interactive robot and the counterpart Determining at least one of a speech attribute and a gesture attribute of the interactive robot, based on at least one of the contextual context information, the linguistic attribute, and the non-verbal attribute estimated as, and the determined speech attribute of the interactive robot, and And controlling the motion of the interactive robot based on at least one of the gesture properties.

At this time, the step of extracting at least one of a linguistic attribute and a non-verbal attribute for the counterpart includes the verbal attribute for the counterpart including at least one of the pitch, amplitude, and speed of the sound by analyzing the voice information. The non-verbal attribute for the counterpart may be extracted, including at least one of the counterpart's gesture use frequency and gesture size, by extracting an attribute or analyzing the image information.

In this case, the step of determining at least one of the speech attribute and the gesture attribute of the interactive robot may include the interactive robot based on at least one of basic setting information, the contextual context information, and the extracted linguistic attribute. You can set the robot's firing properties.

In this case, the extracting at least one of a linguistic attribute and a non-verbal attribute of the counterpart includes estimating the contextual context information from the conversation content with the counterpart, and the basic linguistic attribute and a preset time of the counterpart In the meantime, using the language attribute information of the counterpart, extracting the language attribute change information of the counterpart, and determining at least one of the speech attribute and the gesture attribute of the interactive robot, the context context And determining the speech attribute of the interactive robot by using the information and the language attribute change information of the counterpart.

In this case, the step of estimating the contextual context information may include any of a copy mode that simulates the language attribute of the other party, a basic setting mode, and an opposite mode corresponding to the language attribute of the other party based on the conversation content. With one, the contextual context information can be estimated.

In this case, the determining of at least one of the speech attribute and the gesture attribute of the interactive robot may include setting the gesture attribute of the interactive robot based on at least one of the basic setting information of the interactive robot and the extracted non-verbal attribute. I can.

In this case, the extracting at least one of a linguistic attribute and a non-verbal attribute for the counterpart may include extracting the skeleton information of the counterpart from the image information, and normalizing the joint coordinates of the extracted skeleton information. The step, using the normalized skeleton information, calculating a frequency of use of the gesture of the other party, and calculating a gesture size of the other party using the normalized skeleton information.

In this case, calculating the frequency of use of the counterpart's gesture may include determining whether or not the counterpart uses a gesture using the position change of the joint coordinates during a unit time period, and counting the frequency of use of the counterpart's gesture during a preset period. have.

In this case, in the calculating of the gesture size of the counterpart, the gesture size of the counterpart may be calculated by using a difference between the maximum value of the joint coordinates and the minimum value of the joint coordinates during a unit time.

In this case, determining at least one of the speech attribute and the gesture attribute of the interactive robot may include a sum of the non-verbal attribute of the counterpart to which the first weight is assigned and the basic gesture attribute of the interactive robot to which the second weight is assigned. It may further include determining a gesture property of the interactive robot by calculating.

In this case, the image information on the other party may be photographed from the viewpoint of the interactive robot.

In addition, an apparatus for determining speech and gesture properties of an interactive robot according to an embodiment of the present invention includes a counterpart conversation receiving input of counterpart conversation information including at least one of voice information and image information for a counterpart interacting with the interactive robot. An information input unit, a conversation attribute extraction unit of the other party that analyzes the conversation information of the other party and extracts at least one of a linguistic attribute and a non-verbal attribute of the other party, basic setting information of the interactive robot, and the interactive robot and the other party A speech and gesture determination unit that determines at least one of a speech attribute and a gesture attribute of the interactive robot based on at least one of contextual context information, the verbal attribute, and the non-verbal attribute estimated as conversation content, and the determined And an interactive robot control unit that controls an operation of the interactive robot based on at least one of a speech attribute and the gesture attribute of the interactive robot.

In this case, the conversation attribute extraction unit of the counterpart analyzes the voice information to extract the linguistic property of the counterpart including at least one of pitch, amplitude, and speed of the sound, or analyzes the video information The non-verbal attribute of the counterpart may be extracted, including at least one of the counterpart's gesture use frequency and gesture size.

At this time, the conversation attribute extraction unit of the counterpart estimates the contextual context information from the conversation contents with the counterpart, and uses the counterpart's basic linguistic attribute and the counterpart's linguistic attribute information for a preset time. By extracting the linguistic attribute change information, the speech and gesture determination unit may determine the speech attribute of the interactive robot using the contextual context information and the language attribute change information of the counterpart.

At this time, the utterance and gesture determination unit, based on the conversation content, one of a simulation mode that simulates the language attribute of the other party, a basic setting mode, and an opposite mode corresponding to the language attribute of the other party. Context information can be estimated.

At this time, the conversation attribute extraction unit of the counterpart extracts the counterpart's skeleton information from the image information, normalizes the joint coordinates of the extracted skeleton information, and uses the normalized skeleton information to make the counterpart's gesture The frequency of use may be calculated, and the gesture size of the counterpart may be calculated using the normalized skeleton information.

In this case, the conversation attribute extraction unit of the counterpart may determine whether or not the counterpart uses a gesture by using the position change of the joint coordinates during a unit time, and count the frequency of the counterpart's gesture use during a preset period.

In this case, the conversation attribute extraction unit of the counterpart may calculate the gesture size of the counterpart by using a difference between the maximum value of the joint coordinate and the minimum value of the joint coordinate during a unit time.

In this case, the speech and gesture determination unit determines the gesture attribute of the interactive robot by calculating a sum of the non-verbal attribute of the counterpart to which the first weight is assigned and the basic gesture attribute of the interactive robot to which the second weight is assigned. I can.

In this case, the image information on the other party may be photographed from the viewpoint of the interactive robot.

According to the present invention, it is possible to change the speech attribute and gesture attribute of the interactive robot in consideration of the verbal attribute of the conversation partner and the personality of the interactive robot.

In addition, according to the present invention, by changing the speech attribute and gesture attribute of the interactive robot according to the other party and the surrounding environment, the other party can feel the robot more intimately and intelligently.

1 is a diagram schematically illustrating an environment in which an apparatus for determining speech and gesture properties of an interactive robot according to an embodiment of the present invention is applied.
2 is a block diagram showing a configuration of an apparatus for determining speech and gesture properties of an interactive robot according to an embodiment of the present invention.
3 is a flowchart illustrating a method of determining speech and gesture properties of an interactive robot according to an embodiment of the present invention.
4 is a flowchart illustrating a method of determining ignition properties of an interactive robot according to an embodiment of the present invention.
FIG. 5 is a diagram illustrating a process of determining a speech attribute by an apparatus for determining speech and gesture attributes of an interactive robot according to an embodiment of the present invention.
6 is a flowchart illustrating a method of determining a gesture property of an interactive robot according to an embodiment of the present invention.
7 is a diagram for describing a process of determining a gesture attribute by an apparatus for determining a speech and gesture attribute of an interactive robot according to an embodiment of the present invention.
FIG. 8 is an exemplary diagram illustrating a process of extracting skeleton information in step S610 of FIG. 6.
9 is an exemplary diagram for explaining a process of calculating a gesture size in step S630 of FIG. 6.
10 is a block diagram showing a computer system according to an embodiment of the present invention.

In the present invention, various modifications may be made and various embodiments may be provided, and specific embodiments will be illustrated in the drawings and described in detail.

However, this is not intended to limit the present invention to a specific embodiment, it is to be understood to include all changes, equivalents, and substitutes included in the spirit and scope of the present invention.

The terms used in the present application are only used to describe specific embodiments, and are not intended to limit the present invention. Singular expressions include plural expressions unless the context clearly indicates otherwise. In the present application, terms such as "comprise" or "have" are intended to designate the presence of features, numbers, steps, actions, components, parts, or combinations thereof described in the specification, but one or more other features. It is to be understood that the presence or addition of elements or numbers, steps, actions, components, parts, or combinations thereof, does not preclude in advance.

Unless otherwise defined, all terms used herein, including technical or scientific terms, have the same meaning as commonly understood by one of ordinary skill in the art to which the present invention belongs. Terms as defined in a commonly used dictionary should be interpreted as having a meaning consistent with the meaning in the context of the related technology, and should not be interpreted as an ideal or excessively formal meaning unless explicitly defined in the present application. Does not.

Hereinafter, preferred embodiments of the present invention will be described in more detail with reference to the accompanying drawings. In describing the present invention, in order to facilitate an overall understanding, the same reference numerals are used for the same elements in the drawings, and duplicate descriptions for the same elements are omitted.

1 is a diagram schematically illustrating an environment in which an apparatus for determining speech and gesture properties of an interactive robot according to an embodiment of the present invention is applied.

As shown in FIG. 1, the interactive robot speech and gesture attribute determination system according to an embodiment of the present invention includes an interactive robot 100, an interactive robot speech and gesture attribute determination apparatus 200, and a counterpart 300. Can include.

The interactive robot 100 may use a microphone to recognize the spoken voice of the other party 300, generate a conversation response appropriate to the spoken voice of the other party 300, and output it through a text-to-speech (TTS). In addition, the interactive robot 100 may output a gesture suitable for a conversation response.

At this time, the interactive robot 100 continuously repeats the process of outputting a conversation response based on the speech property received from the apparatus 200 for determining speech and gesture properties of the interactive robot and a process of outputting a gesture based on the gesture property. , It is possible to conduct a conversation with the other party (300).

In particular, the interactive robot 100 may receive a motion control signal including at least one of a speech attribute and a gesture attribute of the interactive robot 100 from the apparatus 200 for determining speech and gesture attributes of the interactive robot. In addition, the interactive robot 100 may output a conversation response through the TTS according to the utterance property of the motion control signal, or may output a gesture by moving a body, an arm, etc. according to the gesture property of the motion control signal.

Here, the speech property of the interactive robot 100 may include at least one of pitch, amplitude, and speed, and the gesture property is a frequency of gesture execution and a gesture size ( Volume) may include at least one of.

In addition, the interactive robot 100 recognizes the voice of the other party 300 and generates a conversation response may be substantially the same as a typical chatbot or artificial intelligence speaker.

Next, the apparatus 200 for determining speech and gesture properties of the interactive robot considers at least one of basic setting information, context context information, linguistic property and non-verbal property of the interactive robot 100, and the interactive robot ( 100) utterance property and gesture property may be determined, and motion of the interactive robot may be controlled according to the determined property.

The apparatus 200 for determining utterance and gesture properties of the interactive robot may receive input of counterpart conversation information including at least one of voice information and image information for the counterpart 300.

Here, the image information on the counterpart 300 may be photographed from the viewpoint of the interactive robot 100. In addition, the apparatus 200 for determining speech and gesture properties of the interactive robot may receive the other party's conversation information from the interactive robot 100, or directly collect the other party's conversation information using a module such as a microphone or a camera.

In addition, when the other party 300 is implemented in the form of a communication capable device, the interactive robot speech and gesture property determination apparatus 200 includes at least one of voice information and image information from the other party 300 implemented in the device form. The conversation information of the other party can be input.

In addition, the apparatus 200 for determining speech and gesture properties of the interactive robot may analyze the conversation information of the other party and extract a verbal property and a non-verbal property for the other party.

The apparatus 200 for determining speech and gesture properties of the interactive robot may analyze the voice information of the other party and extract a linguistic property of the other party including at least one of pitch, amplitude, and speed of a sound. In addition, the apparatus 200 for determining speech and gesture properties of the interactive robot may analyze image information about the other party and extract a non-verbal property of the other party including at least one of a frequency and a gesture size of the other party's gesture use.

In addition, the apparatus 200 for determining speech and gesture properties of the interactive robot includes the extracted linguistic properties, non-verbal properties, basic setting information of the interactive robot 100, and a situation estimated by the conversation content between the interactive robot 100 and the counterpart 300. The speech attribute and the gesture attribute of the interactive robot 100 may be determined based on at least one of the context information.

In addition, the apparatus 200 for determining speech and gesture attributes of the interactive robot may control the interactive robot 100 by transmitting a motion control signal including the determined speech attribute and gesture attribute to the interactive robot 100.

For convenience of explanation, it has been described that the interactive robot 100 and the apparatus 200 for determining utterance and gesture properties of the interactive robot are implemented as separate devices, but the apparatus 200 for determining utterance and gesture properties of the interactive robot is an interactive robot. It can be implemented in a form mounted in (100).

Finally, the other party 300 communicates with the interactive robot 100 and may be a person or another interactive robot 100.

In addition, the other party 300 may be implemented as a device capable of communication, and a person may communicate with the interactive robot 100 using the other party 300 in the form of a device.

Hereinafter, a configuration of an apparatus for determining speech and gesture properties of an interactive robot according to an embodiment of the present invention will be described in more detail with reference to FIG. 2.

2 is a block diagram showing a configuration of an apparatus for determining speech and gesture properties of an interactive robot according to an embodiment of the present invention.

As shown in FIG. 2, the apparatus 200 for determining speech and gesture properties of the interactive robot includes a counterpart conversation information input unit 210, a conversation attribute extraction unit 220, a speech and gesture determination unit 230, and an interactive robot controller ( 240).

First, the counterpart conversation information input unit 210 receives counterpart conversation information for the counterpart 300 interacting with the interactive robot 100. Here, the counterpart conversation information may include at least one of audio information and video information of the counterpart 300.

In addition, the conversation attribute extraction unit 220 analyzes the conversation information of the other party and extracts at least one of a linguistic attribute and a non-verbal attribute of the other party.

The conversation attribute extracting unit 220 may analyze the voice information of the counterpart 300 to extract a linguistic attribute including at least one of pitch, amplitude, and speed of the counterpart.

In addition, the conversation attribute extraction unit 220 may analyze image information of the other party 300 to extract a non-verbal attribute including at least one of a frequency of use of a gesture and a gesture size of the other party.

Next, the speech and gesture determination unit 230 includes at least one of basic setting information of the interactive robot 100, contextual context information estimated from the conversation contents of the interactive robot 100 and the counterpart 300, a linguistic attribute, and a non-verbal attribute. At least one of a speech attribute and a gesture attribute of the interactive robot 100 may be determined based on one.

The speech and gesture determination unit 230 determines the speech and gesture properties of the interactive robot 100 based on basic setting information of the interactive robot 100, and thus speech and gestures in consideration of the personality of the interactive robot 100 Gestures can be output.

In addition, the speech and gesture determination unit 230 estimates contextual context information from the conversation content with the counterpart 300, extracts language attribute change information of the counterpart 300, and contextual context information and language attribute change information The ignition property of the interactive robot 100 may be set by using.

At this time, the speech and gesture determination unit 230 extracts the language attribute change information of the counterpart 300 by using the basic speech attribute of the counterpart 300 and the linguistic attribute information of the counterpart 300 for a preset time. I can.

In addition, the speech and gesture determination unit 230 may estimate contextual context information based on the content of the conversation with the counterpart 300. Here, the context context information may be any one of a copy mode that mimics the linguistic attribute of the counterpart 300, an opposite mode that corresponds to the linguistic attribute of the counterpart 300, and a default setting mode.

In addition, the speech and gesture determination unit 230 may calculate the gesture size of the counterpart 300 by using the skeleton information of the counterpart 300 and count the frequency of use of the gesture of the counterpart 300 during a preset period.

In this case, the speech and gesture determination unit 230 may extract skeleton information of the other party 300 from the image information, and normalize joint coordinates of the extracted skeleton information. In addition, the speech and gesture determination unit 230 may calculate a gesture size and a gesture use frequency of the other party 300 by using the normalized skeleton information.

In addition, when calculating the gesture size, the speech and gesture determination unit 230 may calculate the gesture size of the counterpart 300 by using the difference between the maximum value and the minimum value of the joint coordinate for a unit time.

In addition, when calculating the frequency of use of the gesture, the speech and gesture determination unit 230 may determine whether or not to use the gesture of the other party 300 by using the position change of the joint coordinates for a unit time, and count the frequency of use of the gesture. .

The speech and gesture determination unit 230 calculates at least one of a gesture size and a gesture use frequency of the other party 300, and then assigns a weight to the gesture size and the gesture use frequency of the other party 300, and the interactive robot 100 By assigning a weight to the basic gesture attribute of ), the gesture attribute of the interactive robot 100 may be determined.

Finally, the interactive robot control unit 240 may control the operation of the interactive robot based on the determined speech attribute and gesture attribute of the interactive robot.

The interactive robot controller 240 may control the interactive robot 100 to ignite according to the ignition property of the interactive robot including at least one of pitch, amplitude, and speed. . In addition, the interactive robot control unit 240 may control the interactive robot 100 to output a gesture according to a gesture property of the interactive robot including at least one of a gesture execution frequency (Frequency) and a gesture size (Volume). .

When the apparatus 200 for determining speech and gesture properties of the interactive robot is implemented as a device separate from the interactive robot 100, the apparatus 200 for determining speech and gesture properties of the interactive robot is used as the interactive robot 100. A motion control signal including at least one of the attributes may be transmitted.

On the other hand, when the interactive robot 100 includes the interactive robot's speech and gesture attribute determination device 200, the interactive robot's speech and gesture attribute determination device 200 utters according to the speech attribute, and The interactive robot 100 may be controlled to output a gesture according to an attribute.

Hereinafter, a method for determining utterance and gesture properties of the interactive robot performed by the apparatus for determining speech and gesture properties of the interactive robot according to an embodiment of the present invention will be described in more detail with reference to FIG. 3.

3 is a flowchart illustrating a method of determining speech and gesture properties of an interactive robot according to an embodiment of the present invention.

First, the apparatus 200 for determining speech and gesture properties of the interactive robot receives conversation information of the other party 300 (S310).

The apparatus 200 for determining speech and gesture properties of the interactive robot receives counterpart conversation information including audio information and image information for the counterpart 300.

In this case, the apparatus 200 for determining utterance and gesture properties of the interactive robot may receive counterpart conversation information from the interactive robot 100 or may receive counterpart conversation information from the counterpart 300 implemented in the form of a device capable of communication. In addition, the apparatus 200 for determining speech and gesture properties of the interactive robot may collect the other party's conversation information using the equipped microphone and camera, and the conversation information about the other party 300 interacting with the interactive robot 100 The method of receiving the input is not limited thereto.

In addition, the apparatus 200 for determining speech and gesture attributes of the interactive robot extracts a verbal attribute and a non-verbal attribute for the counterpart 300 (S320).

The apparatus 200 for determining speech and gesture attributes of the interactive robot may extract a linguistic attribute from the voice information input in step S310 and extract a non-verbal attribute from the image information. Here, the linguistic property may include at least one of pitch, amplitude, and speed of a sound of the counterpart 300's voice, and the non-verbal property is at least one of a gesture use frequency and a gesture size for a gesture of the counterpart 300 It can contain either.

Next, the apparatus 200 for determining speech and gesture properties of the interactive robot determines speech properties and gesture properties of the interactive robot 100 (S330).

The apparatus 200 for determining speech and gesture properties of the interactive robot includes verbal and non-verbal properties for the extracted counterpart 300, basic setting information of the interactive robot 100, and a conversation between the interactive robot 100 and the counterpart 300. Based on at least any one of contextual context information about the situation, the speech property and the gesture property of the interactive robot may be determined.

In particular, the apparatus 200 for determining speech and gesture properties of the interactive robot may determine the speech properties of the interactive robot 100 by combining at least two of a linguistic property, basic setting information of the interactive robot 100, and context context information. I can.

The apparatus 200 for determining the speech and gesture properties of the interactive robot determines the speech properties of the interactive robot 100 by reflecting basic setting information of the interactive robot 100, taking into account the personality of the interactive robot 100. The interactive robot 100 can be controlled to fire.

In addition, the apparatus 200 for determining speech and gesture properties of the interactive robot determines the speech properties of the interactive robot 100 by reflecting the contextual information estimated based on the conversation contents of the interactive robot 100 and the counterpart 300, The interactive robot 100 may be controlled to properly utter a conversation situation or conversation content.

Also, the apparatus 200 for determining speech and gesture properties of the interactive robot may determine a gesture property of the interactive robot 100 by combining the non-verbal property and basic setting information of the interactive robot 100.

The apparatus 200 for determining the speech and gesture properties of the interactive robot determines the gesture properties of the interactive robot 100 by reflecting the basic setting information of the interactive robot 100, taking into account the personality of the interactive robot 100. The interactive robot 100 may be controlled to output a gesture.

In addition, the apparatus 200 for determining the speech and gesture properties of the interactive robot determines the gesture properties of the interactive robot 100 by reflecting the frequency of use of the gesture and the size of the gesture, which are non-verbal properties of the other party 300, so that the other party 300 feels stable Alternatively, the interactive robot 100 may be controlled to output a gesture of a degree to which intimacy can be felt.

The process of determining the speech and gesture properties of the interactive robot 100 in step S330 will be described in more detail in steps S440 of FIG. 4 and S640 of FIG. 6 to be described later.

Finally, the apparatus 200 for determining speech and gesture properties of the interactive robot controls the operation of the interactive robot 100 (S340).

The apparatus 200 for determining speech and gesture properties of the interactive robot controls the interactive robot 100 to operate according to the speech and gesture properties of the interactive robot 100 determined in step S330.

At this time, when the interactive robot's speech and gesture attribute determination device 200 is implemented as a separate device from the interactive robot 100, it transmits a motion control signal including the speech attribute and the gesture attribute to the interactive robot 100, It is possible to control speech and gesture output of the interactive robot 100.

Hereinafter, a process of determining the speech property of the interactive robot by the apparatus for determining speech and gesture properties of the interactive robot according to an embodiment of the present invention will be described in more detail with reference to FIGS. 4 and 5.

4 is a flowchart showing a method of determining utterance properties of an interactive robot according to an embodiment of the present invention, and FIG. 5 is a device for determining speech and gesture properties of an interactive robot according to an embodiment of the present invention to determine utterance properties. It is a diagram for explaining the process of doing.

First, the apparatus 200 for determining speech and gesture properties of the interactive robot extracts the linguistic properties of the counterpart 300 (S410).

The apparatus 200 for determining speech and gesture properties of the interactive robot may extract a language property of the counterpart from voice information about the counterpart. In this case, the apparatus 200 for determining speech and gesture attributes of the interactive robot may extract a linguistic attribute including at least one of pitch, amplitude, and speed of a sound for the other party's speech.

In addition, the apparatus 200 for determining speech and gesture properties of the interactive robot extracts linguistic property change information of the counterpart 300 (S420).

The apparatus 200 for determining speech and gesture properties of the interactive robot may extract linguistic property change information by using the basic linguistic property of the counterpart 300 and the linguistic property information of the counterpart 300 for a preset time. .

The apparatus 200 for determining speech and gesture properties of the interactive robot may extract Pitch ^human_recent , Amplitude ^{human_recent,} and Speed ^human_recent , which are linguistic properties of the speech uttered by the other party 300 during the last M hours from the speech information. In addition, the apparatus 200 for determining speech and gesture properties of the interactive robot calculates the difference between the linguistic property of the last M time and the basic linguistic property of the counterpart 300, and the linguistic property change information as shown in Equation 1 below. Can be extracted.

[Equation 1]

Pitch ^human_recent -Pitch ^human_base

Amplitude change information = Amplitude ^human_recent -Amplitude ^human_base

Speed change information = Speed ^human_recent -Speed ^human_base

Here, Pitch ^human_base, Amplitude ^human_base, Speed ^human_base the default language attribute of the other side 300 refers to a language attribute of the entire audio of the other party 300, and may be extracted from the audio information.

Next, the apparatus 200 for determining speech and gesture properties of the interactive robot estimates contextual context information from the conversation content (S430).

The apparatus 200 for determining speech and gesture properties of the interactive robot estimates context context information, which is information about the conversation situation, based on the conversation contents of the interactive robot 100 and the counterpart 300. The apparatus 200 for determining utterance and gesture properties of the interactive robot may recognize the atmosphere of the conversation situation and estimate the context context information to one of three values of -1, 0, and +1 based on the recognized atmosphere information. .

Pseudocode 1 below shows an example of a pseudo code for a technique for estimating contextual information.

[Water code 1]

if Sentiment (robotic speech content) =='Neutral'

Context = 0

else if Sentiment(contents of other party's speech) == Sentiment(contents of robotic speech)

Context = 1

else

Context = -1

As shown in the capital code 1, when the interactive robot's speech and gesture attribute determination device 200 is in a neutral state in which emotions are not included in the speech atmosphere of the interactive robot 100, the interactive robot's speech and gestures The attribute determining device 200 may set the contextual context information to 0 and set the contextual context information to +1 when the utterance atmosphere of the counterpart 300 and the interactive robot 100 is the same.

In addition, the apparatus 200 for determining speech and gesture properties of the interactive robot may set the contextual context information to -1 when the speech atmosphere of the counterpart 300 and the interactive robot 100 is opposite.

For example, in a situation where the other party 300 is angry, and the interactive robot 100 utters a response that calms the other party 300, the utterance atmosphere of the other party 300 and the interactive robot 100 is opposite. The apparatus 200 for determining speech and gesture properties of the interactive robot may set contextual context information to -1.

Finally, the apparatus 200 for determining speech and gesture properties of the interactive robot determines speech properties of the interactive robot 100 (S440).

The apparatus 200 for determining speech and gesture properties of the interactive robot includes linguistic property change information of the counterpart 300 extracted in step S420, contextual context information estimated in step S430, and basic setting information of the interactive robot 100 Based on at least one of them, the ignition property of the interactive robot 100 is determined.

Here, the ignition property of the interactive robot 100 may include at least one of pitch, amplitude, and speed of sound.

The basic setting information of the interactive robot 100 means personality unique to the interactive robot 100 and may include Pitch ^robot_base , Amplitude ^robot_base , and Speed ^robot_base . In addition, basic setting information of the interactive robot 100 may be set by a robot manufacturer or a robot owner.

For example, when the interactive robot 100 is a female robot, the Pitch ^robot_base may be set high, and when the interactive robot 100 is a male robot, the Pitch ^robot_base may be set low. In this way, by modifying the basic setting information of the interactive robot 100, the personalities of the individual interactive robots 100 can be applied.

In addition, the apparatus 200 for determining speech and gesture properties of the interactive robot may determine the speech properties of the interactive robot 100 as shown in the following number code 2. Pseudocode 2 below shows an example of a pseudo code for a technique of determining pitch, amplitude, and speed of sound, which are the firing properties of the interactive robot 100.

[Water Code 2]

Pitch ^robot <- Pitch ^robot_base + Context * (Pitch ^human_recent -Pitch ^human_base )

Amplitude ^robot <- Amplitude ^robot_base + Context * (Amplitude ^human_recent -Amplitude ^human_base )

Speed ^robot <- Speed ^robot_base + Context * (Speed ^human_recent -Speed ^human_base )

Like the number code 2, the interactive robot's speech and gesture attribute determination apparatus 200 multiplies the linguistic attribute change information of the counterpart 300 by the contextual context information, and adds basic setting information of the interactive robot 100 The ignition property of the interactive robot 100 may be determined.

In the capital code 2, when the contextual context information (Context) is 0, the interactive robot's speech and gesture property determination apparatus 200 does not reflect the language property of the other party 300 and the preset interactive robot 100 The ignition property of the interactive robot 100 may be determined with the basic setting information of.

On the other hand, when the contextual context information (Context) is +1, the apparatus 200 for determining speech and gesture properties of the interactive robot may determine the speech properties of the interactive robot 100 by simulating the linguistic properties of the counterpart 300. .

For example, when the contextual context information (Context) is +1 and the other party 300 yells in a loud voice, the interactive robot 100 may simulate the loud voice of the other party 300 and set the volume of the voice to be large. The firing properties can be determined.

In addition, when the contextual context information (Context) is -1, the apparatus 200 for determining the speech and gesture properties of the interactive robot may determine the speech property of the interactive robot 100 as a property opposite to the linguistic property of the counterpart 300. have.

For example, when the contextual context information (Context) is -1 and the other party 300 speaks quickly with a loud voice, the apparatus 200 for determining the speech and gesture properties of the interactive robot may match the loud and fast voice of the other party 300 Conversely, the speech property of the interactive robot 100 may be determined so as to respond calmly by setting the volume of the voice to be small and setting the speed to be slow.

Hereinafter, a process of determining the gesture attribute of the interactive robot by the apparatus for determining the utterance and gesture attribute of the interactive robot according to an embodiment of the present invention will be described in more detail with reference to FIGS. 6 and 9.

6 is a flowchart showing a method of determining a gesture property of an interactive robot according to an embodiment of the present invention, and FIG. 7 is a flow chart showing a method of determining a gesture property of an interactive robot according to an embodiment of the present invention. It is a diagram for explaining the process of doing.

First, the apparatus 200 for determining speech and gesture properties of the interactive robot extracts skeleton information of the counterpart 300 and performs normalization (S610).

The apparatus 200 for determining speech and gesture properties of the interactive robot extracts skeleton information of the other party 300 from the image information. Here, the image information may be image information photographing the other party 300 from the viewpoint of the interactive robot 100.

In addition, as shown in FIG. 7, the apparatus 200 for determining speech and gesture properties of the interactive robot may first perform step S610 in order to extract non-verbal properties for the counterpart 300.

In step S610, the apparatus 200 for determining speech and gesture properties of the interactive robot may extract a skeleton using existing image recognition software, or extract skeleton information using a 3D sensor such as Kinect.

FIG. 8 is an exemplary diagram illustrating a process of extracting skeleton information in step S610 of FIG. 6.

As shown in FIG. 8, each joint point in the extracted skeleton information may be expressed as x, y coordinates (J ^raw _i = (x _i , y _i )), and the number of joint points (joints) in FIG. 8 Is 19 and i has a value of 1 to 19.

In addition, the interactive robot's speech and gesture property determination apparatus 200 may perform normalization (J _i = J ^raw _i -J _center ) based on the body's center point (J _center ) so that the joint coordinates have a value in a certain range. have. In this case, the center point of the body may mean a joint coordinate corresponding to the center point of the counterpart in the skeleton information.

Next, the apparatus 200 for determining speech and gesture properties of the interactive robot calculates the frequency of using the gesture of the other party 300 (S620).

The interactive robot's speech and gesture attribute determination apparatus 200 determines that the other party 300 has expressed a gesture when it is determined that there is a movement of more than a preset threshold value, and counts the number of times when there is a movement of more than a threshold value to make a gesture. You can calculate the frequency of use.

The following pseudo code 3 shows an example of a pseudo code for a technique of calculating the frequency of use of a gesture by the counterpart 300.

[Water Code 3]

time_slot = (T, TT _s ), (T- T _s , T-2* T _s ), ... (T-(N-1)* T _s , TN* T _s )

frequency = 0

for each time_slot i

motion_sum =

if motion_sum> threshold

frequency += 1

frequency = frequency / N

Here, T _s denotes a unit time, and N denotes the number of unit sections to measure the frequency of gesture use. For example, in the case of measuring the frequency of use of the gesture for the last minute, T _s may be 10 seconds and N may be set to 6.

Referring to FIG. 6 again, the apparatus 200 for determining speech and gesture properties of the interactive robot calculates a gesture size of the other party 300 (S630).

The gesture size of the counterpart 300 indicates how large the movement of the gesture expressed by the counterpart 300 is, and the apparatus 200 for uttering the interactive robot and determining the gesture property is between the minimum and maximum values for each of the joint coordinates x and y. Using the difference, the size of the gesture of the other party 300 may be calculated.

9 is an exemplary diagram for explaining a process of calculating a gesture size in step S630 of FIG. 6.

As in the first gesture 910 illustrated in FIG. 9, when the other party 300 expresses the gesture with a wide open arm, the gesture size value of the other party 300 increases. On the other hand, when the other party 300 expresses a small gesture in front of the body like the second gesture 920, the gesture size value of the other party 300 has a small value.

In FIG. 9, it has been described that a rectangular box is set using the maximum value of joint coordinates according to the gesture of the other party 300, and the gesture size of the other party 300 is calculated using the set size of the rectangular box, but the present invention is not limited thereto. , The gesture size may be calculated using a joint coordinate value according to the movement of the arm or hand of the counterpart 300.

The process of calculating the frequency of use of the gesture of the counterpart 300 and the size of the gesture in steps S620 and S630 is substantially the same as the process of extracting non-verbal attributes for the counterpart 300 in step S320 of FIG. For the sake of this, duplicate descriptions are omitted.

Finally, the apparatus 200 for determining speech and gesture properties of the interactive robot determines the gesture properties of the interactive robot 100 (S640).

The interactive robot's speech and gesture attribute determination apparatus 200 corresponds to the frequency of use of the gesture of the counterpart 300 calculated in step S620, the gesture size of the counterpart 300 calculated in step S630, and the personality of the interactive robot 100 Based on the basic setting information, a gesture property of the interactive robot 100 may be determined.

Here, the gesture attribute of the interactive robot 100 determined by the apparatus 200 for determining speech and gesture attributes of the interactive robot may include at least one of a gesture execution frequency and a gesture volume.

In addition, the apparatus 200 for determining speech and gesture properties of the interactive robot assigns a first weight to the non-verbal properties of the counterpart 300, such as a frequency ^human and a volume ^human , and the interactive robot 100 The gesture attribute of the interactive robot 100 may be determined by assigning a second weight to the basic gesture attributes (Frequency ^robot_base and Volume ^robot_base ) among the basic setting information.

The following pseudo code 4 shows an example of a pseudo code for a technique of determining the gesture property of the interactive robot 100 by assigning a weight to the non-verbal property of the counterpart 300 and the basic gesture property of the interactive robot 100 .

[Water Code 4]

Frequency ^robot <- w * Frequency ^human + (1-w) * Frequency ^robot_base

Volume ^robot <- w * Volume ^human + (1-w) * Volume ^robot_base

Here, w means a first weight, and 1-w means a second weight. And the weight may be set by the robot manufacturer or the robot owner.

In addition, the pseudo code 5 below shows an example of a pseudo code for a technique of generating a gesture motion to be output by the interactive robot 100 based on the determined gesture property of the interactive robot 100.

[Water Code 5]

for each word in speech_sentence

if random() <Frequency ^robot

gesture = get_gesture(word)

make_gesture(gesture, Volume ^robot )

Here, speech_sentence means a sentence to be spoken by the interactive robot 100.

The apparatus 200 for determining speech and gesture properties of the interactive robot may be set to output a gesture for all words of a sentence to be uttered by the interactive robot 100, and as shown in the capital code 5, the frequency of the gesture execution of the interactive robot 100 It is possible to determine whether to generate and output a gesture according to (Frequency ^robot ).

For example, when the frequency ^robot of the interactive robot 100 is 0.7, the interactive robot's speech and gesture attribute determination device 200 includes 10 words included in a sentence to be uttered by the interactive robot 100 You can create gestures for 7 words among them.

In addition, when the interactive robot 100 outputs the generated gesture, the interactive robot's speech and gesture attribute determination apparatus 200 generates a motion control signal that outputs a gesture in a size corresponding to the gesture size (Volume ^robot ). I can.

In the pseudo code 5, the get_gesture function, which is a function for selecting a gesture suitable for each word, and the make_gesture function, which is a function for expressing the selected gesture by the interactive robot 100, may be implemented in different ways for each robot manufacturer.

10 is a block diagram showing a computer system according to an embodiment of the present invention.

Referring to FIG. 10, an embodiment of the present invention may be implemented in a computer system 1000 such as a computer-readable recording medium. As shown in FIG. 10, the computer system 1000 includes one or more processors 1010, a memory 1030, a user interface input device 1040, and a user interface output device 1050 communicating with each other through a bus 1020. And a storage 1060. Further, the computer system 1000 may further include a network interface 1070 connected to the network 1080. The processor 1010 may be a central processing unit or a semiconductor device that executes processing instructions stored in the memory 1030 or the storage 1060. The memory 1030 and the storage 1060 may be various types of volatile or nonvolatile storage media. For example, the memory may include a ROM 1031 or a RAM 1032.

Accordingly, an embodiment of the present invention may be implemented in a computer-implemented method or a non-transitory computer-readable medium in which instructions executable in a computer are recorded. When computer-readable instructions are executed by a processor, the computer-readable instructions may perform a method according to at least one aspect of the present invention.

As described above, the apparatus and method for determining speech and gesture properties of an interactive robot according to the present invention are not limited to the configuration and method of the embodiments described above, but various modifications may be made to the embodiments. All or part of each of the embodiments may be selectively combined and configured.

100: interactive robot
200: Interactive robot speech and gesture property determination device
210: counterpart conversation information input unit
220: conversation attribute extraction unit
230: speech and gesture determination unit
240: interactive robot control unit
910: first gesture
920: second gesture
1000: computer system
1010: processor
1020: bus
1030: memory
1031: Rom
1032: RAM
1040: user interface input device
1050: user interface output device
1060: storage
1070: network interface
1080: network

Claims (20)

In the interactive robot speech and gesture attribute determination method performed by an interactive robot speech and gesture attribute determination device,
Receiving a counterpart conversation information including at least one of voice information and video information for a counterpart interacting with the interactive robot,
Analyzing the counterpart conversation information, extracting at least one of a verbal attribute and a non-verbal attribute for the counterpart,
Determining at least one of a speech attribute and a gesture attribute of the interactive robot based on at least one of contextual context information, the linguistic attribute, and the non-verbal attribute estimated as the conversation content between the interactive robot and the counterpart, And
Including the step of controlling the operation of the interactive robot based on at least one of the determined speech attribute and the gesture attribute of the interactive robot,
Extracting at least one of a linguistic attribute and a non-verbal attribute for the counterpart,
In order to determine the speech attribute and gesture attribute of the interactive robot, the speech information is analyzed to extract the linguistic attribute of the counterpart including any one of pitch, amplitude, and speed, or the image information is analyzed. By doing so, extracting the non-verbal attribute of the other party including at least one of a frequency of use of a gesture and a gesture size of the other party,
Extracting at least one of a linguistic attribute and a non-verbal attribute for the counterpart,
Extracting skeleton information of the counterpart from the image information,
Performing normalization on the joint coordinates of the extracted skeleton information,
Calculating the frequency of use of the counterpart's gesture using the normalized skeleton information, and
And calculating a gesture size of the counterpart by using the normalized skeleton information. delete The method of claim 1,
Determining at least one of a speech attribute and a gesture attribute of the interactive robot,
The method of determining speech and gesture properties of the interactive robot, characterized in that, based on at least one of the basic setting information of the interactive robot, the contextual context information, and the extracted linguistic attribute, the speech attribute of the interactive robot is set. The method of claim 1,
Extracting at least one of a linguistic attribute and a non-verbal attribute for the counterpart,
Estimating the contextual context information from the conversation content with the counterpart, and
And extracting change information of the counterpart's linguistic attribute by using the counterpart's basic linguistic attribute and the counterpart's linguistic attribute information for a preset time,
Determining at least one of a speech attribute and a gesture attribute of the interactive robot,
And determining the speech attribute of the interactive robot using the contextual context information and the language attribute change information of the counterpart. The method of claim 4,
Estimating the contextual context information,
Based on the conversation content, the contextual context information is estimated in one of a simulation mode that simulates the language attribute of the other party, a basic setting mode, and an opposite mode that is a mode corresponding to the language attribute of the other party. How to determine speech and gesture properties of interactive robots. The method of claim 1,
Determining at least one of a speech attribute and a gesture attribute of the interactive robot,
A method of determining speech and gesture properties of the interactive robot, comprising setting a gesture property of the interactive robot based on at least one of the basic setting information of the interactive robot and the extracted non-verbal property. delete The method of claim 1,
The step of calculating the frequency of use of the counterpart's gesture,
A method for determining speech and gesture properties of an interactive robot, comprising determining whether or not the counterpart uses a gesture by using a position change of the joint coordinate during a unit time, and counting the frequency of the counterpart's gesture use during a preset period. The method of claim 1,
The step of calculating the gesture size of the other party,
A method for determining speech and gesture properties of an interactive robot, characterized in that the gesture size of the counterpart is calculated using a difference between the maximum value of the joint coordinate and the minimum value of the joint coordinate during a unit time. The method of claim 1,
Determining at least one of a speech attribute and a gesture attribute of the interactive robot,
Comprising the sum of the non-verbal properties of the counterpart to which the first weight is assigned and the basic gesture properties of the interactive robot to which the second weight is assigned, and determining the gesture properties of the interactive robot. How to determine speech and gesture properties of interactive robots. The method of claim 1,
The video information for the other party,
A method of determining speech and gesture properties of an interactive robot, characterized in that photographed from the viewpoint of the interactive robot. A counterpart conversation information input unit for receiving counterparty conversation information including at least one of voice information and video information for the counterpart interacting with the interactive robot,
A conversation attribute extraction unit of the other party for analyzing the conversation information of the other party and extracting at least one of a linguistic attribute and a non-verbal attribute for the other party,
Based on at least one of the basic setting information of the interactive robot, contextual context information estimated from the conversation content of the interactive robot and the counterpart, the verbal attribute and the non-verbal attribute, among the speech attribute and gesture attribute of the interactive robot A speech and gesture determination unit that determines at least any one, and
An interactive robot control unit that controls an operation of the interactive robot based on at least one of the determined speech attribute and the gesture attribute of the interactive robot,
The conversation attribute extraction unit of the other party,
In order to determine the speech attribute and the gesture attribute of the interactive robot, the speech information is analyzed to extract the linguistic attribute of the counterpart including at least one of pitch, amplitude, and speed, or the image information And extracting the non-verbal attribute of the counterpart including at least one of the counterpart's gesture use frequency and gesture size by analyzing
The conversation attribute extraction unit of the other party,
Extracts the counterpart's skeleton information from the image information, normalizes the joint coordinates of the extracted skeleton information, calculates the counterpart's gesture usage frequency using the normalized skeleton information, and normalizes the skeleton An apparatus for determining speech and gesture properties of an interactive robot, characterized in that the gesture size of the counterpart is calculated using information. delete The method of claim 12,
The conversation attribute extraction unit of the other party,
By estimating the contextual context information from the contents of the conversation with the counterpart, extracting the counterpart's linguistic attribute change information using the counterpart's basic linguistic attribute and the counterpart's linguistic attribute information for a preset time, The speech and gesture attribute determination apparatus of an interactive robot, wherein the speech and gesture determination unit determines the speech attribute of the interactive robot using the contextual context information and the language attribute change information of the counterpart. The method of claim 14,
The speech and gesture determination unit,
Based on the conversation content, the contextual context information is estimated in one of a simulation mode that simulates the language attribute of the other party, a basic setting mode, and an opposite mode that is a mode corresponding to the language attribute of the other party. A device for determining speech and gesture properties of an interactive robot. delete The method of claim 12,
The conversation attribute extraction unit of the other party,
The apparatus for determining utterance and gesture properties of an interactive robot, characterized in that for determining whether or not the counterpart uses a gesture by using a position change of the joint coordinates during a unit time, and counting the frequency of the counterpart's gesture use during a preset period. The method of claim 12,
The conversation attribute extraction unit of the other party,
An apparatus for determining speech and gesture properties of an interactive robot, characterized in that for calculating a gesture size of the counterpart by using a difference between a maximum value of the joint coordinate and a minimum value of the joint coordinate during a unit time. The method of claim 12,
The speech and gesture determination unit,
The speech of the interactive robot, characterized in that the sum of the non-verbal properties of the counterpart to which the first weight is assigned and the basic gesture properties of the interactive robot to which the second weight is assigned is calculated to determine the gesture properties of the interactive robot. And a gesture property determination device. The method of claim 12,
The video information for the other party,
An apparatus for determining speech and gesture properties of an interactive robot, characterized in that photographed from a viewpoint of the interactive robot.

Images