WO2018107343A1 - Robot gesture generation - Google Patents

Robot gesture generation Download PDF

Info

Publication number
WO2018107343A1
WO2018107343A1 PCT/CN2016/109547 CN2016109547W WO2018107343A1 WO 2018107343 A1 WO2018107343 A1 WO 2018107343A1 CN 2016109547 W CN2016109547 W CN 2016109547W WO 2018107343 A1 WO2018107343 A1 WO 2018107343A1
Authority
WO
WIPO (PCT)
Prior art keywords
gesture
labanotation
robot
symbolic representation
concept
Prior art date
Application number
PCT/CN2016/109547
Other languages
French (fr)
Inventor
Katsushi Ikeuchi
Mona Soliman HABIB
Original Assignee
Microsoft Technology Licensing, Llc.
Priority date (The priority date is an assumption and is not a legal conclusion. Google has not performed a legal analysis and makes no representation as to the accuracy of the date listed.)
Filing date
Publication date
Application filed by Microsoft Technology Licensing, Llc. filed Critical Microsoft Technology Licensing, Llc.
Priority to CN201680091517.3A priority Critical patent/CN110062683B/en
Priority to US16/469,024 priority patent/US11443161B2/en
Priority to EP16923912.6A priority patent/EP3551393A4/en
Priority to PCT/CN2016/109547 priority patent/WO2018107343A1/en
Publication of WO2018107343A1 publication Critical patent/WO2018107343A1/en

Links

Images

Classifications

    • GPHYSICS
    • G06COMPUTING; CALCULATING OR COUNTING
    • G06NCOMPUTING ARRANGEMENTS BASED ON SPECIFIC COMPUTATIONAL MODELS
    • G06N3/00Computing arrangements based on biological models
    • G06N3/004Artificial life, i.e. computing arrangements simulating life
    • G06N3/008Artificial life, i.e. computing arrangements simulating life based on physical entities controlled by simulated intelligence so as to replicate intelligent life forms, e.g. based on robots replicating pets or humans in their appearance or behaviour
    • BPERFORMING OPERATIONS; TRANSPORTING
    • B25HAND TOOLS; PORTABLE POWER-DRIVEN TOOLS; MANIPULATORS
    • B25JMANIPULATORS; CHAMBERS PROVIDED WITH MANIPULATION DEVICES
    • B25J11/00Manipulators not otherwise provided for
    • B25J11/0005Manipulators having means for high-level communication with users, e.g. speech generator, face recognition means
    • BPERFORMING OPERATIONS; TRANSPORTING
    • B25HAND TOOLS; PORTABLE POWER-DRIVEN TOOLS; MANIPULATORS
    • B25JMANIPULATORS; CHAMBERS PROVIDED WITH MANIPULATION DEVICES
    • B25J9/00Programme-controlled manipulators
    • B25J9/16Programme controls
    • BPERFORMING OPERATIONS; TRANSPORTING
    • B25HAND TOOLS; PORTABLE POWER-DRIVEN TOOLS; MANIPULATORS
    • B25JMANIPULATORS; CHAMBERS PROVIDED WITH MANIPULATION DEVICES
    • B25J9/00Programme-controlled manipulators
    • B25J9/16Programme controls
    • B25J9/1628Programme controls characterised by the control loop
    • B25J9/163Programme controls characterised by the control loop learning, adaptive, model based, rule based expert control
    • BPERFORMING OPERATIONS; TRANSPORTING
    • B25HAND TOOLS; PORTABLE POWER-DRIVEN TOOLS; MANIPULATORS
    • B25JMANIPULATORS; CHAMBERS PROVIDED WITH MANIPULATION DEVICES
    • B25J9/00Programme-controlled manipulators
    • B25J9/16Programme controls
    • B25J9/1694Programme controls characterised by use of sensors other than normal servo-feedback from position, speed or acceleration sensors, perception control, multi-sensor controlled systems, sensor fusion
    • B25J9/1697Vision controlled systems
    • GPHYSICS
    • G06COMPUTING; CALCULATING OR COUNTING
    • G06NCOMPUTING ARRANGEMENTS BASED ON SPECIFIC COMPUTATIONAL MODELS
    • G06N5/00Computing arrangements using knowledge-based models
    • G06N5/02Knowledge representation; Symbolic representation
    • G06N5/022Knowledge engineering; Knowledge acquisition
    • GPHYSICS
    • G06COMPUTING; CALCULATING OR COUNTING
    • G06VIMAGE OR VIDEO RECOGNITION OR UNDERSTANDING
    • G06V40/00Recognition of biometric, human-related or animal-related patterns in image or video data
    • G06V40/20Movements or behaviour, e.g. gesture recognition
    • G06V40/28Recognition of hand or arm movements, e.g. recognition of deaf sign language
    • GPHYSICS
    • G10MUSICAL INSTRUMENTS; ACOUSTICS
    • G10LSPEECH ANALYSIS TECHNIQUES OR SPEECH SYNTHESIS; SPEECH RECOGNITION; SPEECH OR VOICE PROCESSING TECHNIQUES; SPEECH OR AUDIO CODING OR DECODING
    • G10L15/00Speech recognition
    • G10L15/22Procedures used during a speech recognition process, e.g. man-machine dialogue
    • GPHYSICS
    • G10MUSICAL INSTRUMENTS; ACOUSTICS
    • G10LSPEECH ANALYSIS TECHNIQUES OR SPEECH SYNTHESIS; SPEECH RECOGNITION; SPEECH OR VOICE PROCESSING TECHNIQUES; SPEECH OR AUDIO CODING OR DECODING
    • G10L15/00Speech recognition
    • G10L15/22Procedures used during a speech recognition process, e.g. man-machine dialogue
    • G10L2015/223Execution procedure of a spoken command
    • GPHYSICS
    • G10MUSICAL INSTRUMENTS; ACOUSTICS
    • G10LSPEECH ANALYSIS TECHNIQUES OR SPEECH SYNTHESIS; SPEECH RECOGNITION; SPEECH OR VOICE PROCESSING TECHNIQUES; SPEECH OR AUDIO CODING OR DECODING
    • G10L15/00Speech recognition
    • G10L15/22Procedures used during a speech recognition process, e.g. man-machine dialogue
    • G10L2015/225Feedback of the input speech

Definitions

  • robots Numerous types of robots have been used in a wide variety of applications. Traditionally, robots are mainly used in industrial applications. Recently, more and more robots have been utilized in the field of human-robot interaction, such as robots that are used in family service and medical service. With such trend, how to realize a more natural and smoother communication between a human being and a robot attracts more attention.
  • Embodiments of the subject matter described herein provide a method, apparatus and robot system for robot gesture generation. Generally speaking a concept corresponding to a utterance to be spoken by a robot is determined. After the concept is determined or selected, a symbolic representation of a gesture that corresponds to the determined concept is retrieved from a predetermined gesture library. Subsequently, the symbolic representation is provided to cause the robot to perform the gesture.
  • embodiments of the present disclosure provide a method and an apparatus for robot gesture generation, especially in a chatting robot system, which enable a generation of appropriate gesture along with or in synchronization with the spoken utterances. In such way, a more natural, comprehensive and effective communication may be achieved between human and robots.
  • FIG. 1 is a schematic diagram illustrating a robot system in which example implementations of the subject matter described herein can be implemented.
  • FIG. 2 illustrates a flowchart of a method for robot gesture generation according to embodiments of the present disclosure.
  • FIG. 3A illustrates a schematic diagram of a typical labanotation and the corresponding concept represented by the labanotation according to embodiments of the present disclosure.
  • FIG. 3B illustrates example symbols of body parts of a robot.
  • FIG. 4 illustrates a flowchart of a process for mapping a utterance to a vector according to embodiments of the present disclosure.
  • FIG. 5 is a block diagram of an apparatus suitable for implementing one or more implementations of the subject matter described herein.
  • the phrase “include (s) ” and its variants shall be interpreted as an open term meaning “including but not limited to. ”
  • the phrase “based on” shall be interpreted as “at least partially based on. ”
  • the term “a” is to be read as “one or more” unless otherwise specified.
  • the phrase “an embodiment” or “one embodiment” shall be interpreted as “at least one embodiment. ”
  • the phrase “another embodiment” shall be interpreted as “at least one other embodiment. ”
  • the phrases like “first” and “second” may refer to different or the same objects. Other definitions might also be included explicitly and implicitly in the following description.
  • FIG. 1 illustrates a schematic diagram of a robot system 1000 according to embodiments of the present disclosure.
  • the robot system 1000 generally includes a robot 100, a server 300, and an apparatus 10 according to embodiments of present application.
  • the apparatus 10 is used for generating appropriate gesture along with the spoken utterances, depending on the utterance to be spoken by the robot 100.
  • the robot 100 may be, for example, a chatting robot.
  • the server 300 is usually a device capable of processing the received message and providing the processed information and may be in the form of a cloud-based computer, for example, a chatting intelligence for the case of a chatting robot. That is, a user may have a “conversation” with the chatting intelligence 300 through, for example, an audio-visual-tactile intelligence.
  • the user may first make expressions or ask questions to the robot 100, and then expect to receive a response, such as an answer to the user’s question, from the robot 100.
  • the processor associated with the server 300 may perform some analyses based on the received expressions/questions, and prepare accordingly a response or an answer to the expression that is to be spoken by the robot 100.
  • some conventional chatting robots are not provided with any motion or gesture functions.
  • Some other conventional chatting robots may be able to provide some gestures along with the spoken utterances, these gestures, however, may not be appropriate/correct ones, and for most of cases, these generated gestures may just be random ones.
  • an appropriate gesture response to the user along with or in synchronization with the voice output from the robot 100 can be achieved.
  • a translation mechanism from spoken language to gestures can be realized.
  • the generated gesture in such a way would be more accurate and natural.
  • FIG. 2 illustrates a flow chart of a method 200 for robot gesture generation according to embodiments of the present disclosure.
  • the method 200 can be executed, for example, on the apparatus 10 as illustrated in FIG. 1.
  • the apparatus 10 as shown can be a client device or a cloud-based apparatus, or it can be part of the server 300 as illustrated in FIG. 1. It should be understood that the method 200 may also include additional actions not shown and/or omit the illustrated steps. Scope of the subject matter described herein is not limited in this aspect.
  • the method 200 will be described as below with reference to FIG. 1, FIG. 3 and FIG. 4.
  • one or more concepts C 1 , C 2 , ..., C N ) corresponding to a utterance to be spoken by a robot 100 may be determined.
  • the utterance that is to be spoken by the robot 100 may be a proactive or responsive vocal interaction in a conversation with the user. It may be a bit of spoken language from a word, a phrase, a plurality of words to a full utterance. It is to be noted that although the utterance to be spoken as illustrated in FIG. 1 is received from the server 300, in some other embodiments, the utterance can be instead obtained directly from the apparatus 10.
  • the user may first make an expression/question to the robot 100 and further to the server 300 by, for example, transmitting his/her voice via a microphone and his/her image via a camera integrated on the robot 100.
  • the server 300 receives the expression/question from the user, the processor (not shown) associated with the server 300 will perform analyses based on the received expressions, and prepare a utterance that is to be spoken by the robot 100 accordingly, as a response or an answer to that question.
  • the incoming expression/question from the user will be firstly checked locally. If it is included in a special set, or in other words, if it is one of special conversations, then an answer to this expression/question will be prepared in local. If not, the incoming expression/question will be further sent to the server 300 for a response.
  • a concept is generally defined as one representative extracted from a cluster of words.
  • the cluster of words may contain those words that share the same or a similar meaning within a context and thus are located in close proximity to one another in a vector space.
  • Typical concepts may include, for example, “Hello, ” “Good, ” ” Thanks, ” “Hungry, ” and so on. It is to be noted that scope of the subject matter described herein is not limited in the formats or contents included in the concepts. Users may define other particular concepts as needed.
  • the robot system 1000 as illustrated in FIG. 1 includes a gesture library 108 which is created to store a plurality of predetermined gestures and the corresponding concepts.
  • each concept that corresponds to a gesture in the gesture library 108 may be represented by a first vector.
  • determining the concept (C 1 , C 2 , ..., C N ) corresponding to the utterance may include obtaining a second vector representing the utterance, and then determining the similarity degrees of the second vector to the plurality of first vectors. In this way, based on the determined similarity degrees, the concept corresponding to the utterance can be selected from the plurality of concepts stored in the gesture library 108.
  • the received utterance or expression may be first given to a system, such as “distributed word embedding” system, to convert the utterance to a vector and thus map to a concept.
  • a system such as “distributed word embedding” system
  • the concept having a maximum similarity degree may be selected from the plurality of concepts, as the determined concept corresponding to the utterance.
  • FIG. 4 shows an example illustrating a process 400 of conversion from an utterance to a vector.
  • the utterance 410 that is to be spoken by the robot 100 is “I agree with you, ” and the “distributed word embedding” system may determine the best matched concept 420, for example to be “agree, ” and then the concept “Agree” will be mapped to a corresponding vector 430 for comparison with those vectors stored in the gesture library 108.
  • a symbolic representation (SR 1 , SR 2 , ..., SR N ) of a gesture that corresponds to the determined concept (C 1 , C 2 , ..., C N ) is retrieved from the predetermined gesture library 108.
  • the gesture library 108 may be previously created by using a so-called “learning from observation” technique.
  • a gesture of the robot 100 needs to be recorded first.
  • the gesture is usually performed by a performer (for example, a human) in front of a camera in association with the robot 100.
  • Some important poses from the performer then can be captured or “observed” by the camera and subsequently converted into symbolic representations (also called symbols for short) that correspond to the performed gesture.
  • retrieving the symbolic representation includes retrieving a labanotation (LA 1 , LA 2 , ..., LA N ) . Consequently, with the obtained labanotation, a relationship between the labanotation representing the performed gesture and a concept (e.g. labanotation-concept pairs) can be established and stored in the gesture library 108 for a future use.
  • Labanotation has been used for recording human dances.
  • Labanotation herein particularly defines orientations of at least one body part of the robot 100 with respect to a plurality of time slots.
  • the labanotation employed in the robot system 1000 is necessary and sufficient condition to describe human actions. This is because, watching the same dance by several dancers end up with the same labanotation, and on the other hand, from one labanotation, any dancer may perform/reconstruct the same dance.
  • labanotation is machine-independent (or hardware-independent) , and thereby can overcome the hardware difference.
  • a concise symbolic representation it is easy to transmit labanotation between a robot and the cloud computer through limited communication channels.
  • labanotation requires smaller memory than other types of representations.
  • orientations of the at least one body part of the robot 100 in the plurality of time slots 301 can be determined, and then symbols corresponding the orientations can be obtained. After that, the symbols in association with the corresponding time slots 301 as a part of the labanotation can be saved.
  • the at least one body part includes a plurality of body parts
  • the labanotation includes a first dimension corresponding to the plurality of time slots 301 and a second dimension corresponding to the plurality of body parts (that is, two-dimensional labanotation) .
  • FIG. 3A illustrated such a 2D labanotation representative of a concept of “Good” .
  • each of columns that is, the second dimension
  • Each row (that is, the first dimension) corresponds to the time slots 301 with a given duration.
  • a symbol represents to which direction that the body part is oriented at that time.
  • FIG. 3A the sample labanotation in FIG. 3A is merely shown for purpose of illustration without suggesting any limitation as to the scope of the subject matter described herein.
  • a more complicated labanotation with more body parts involved such as left elbow or right knee (labeled by dotted circles) as shown in FIG. 3B is also possible.
  • a plurality of labanotations of a plurality of gestures can be grouped into a cluster of labanotation, and for each cluster of labanotation, a representative labanotation can be determined.
  • similar labanotations indicating similar gestures can be clustered/grouped.
  • a cluster of labanotation/gestures (rather than a single labanotation/gesture) is represented by a representative labanotation. In this way, it is guaranteed that any concepts corresponding to a same cluster will give a same gesture.
  • the gesture library 108 created in such an alternative way described as above contains “multiple-to-one” concepts to labanotation matches. Accordingly, retrieving, from the predetermined gesture library 108, a symbolic representation (SR 1 , SR 2 , ..., SR N ) of a gesture that corresponds to the determined concept (C 1 , C 2 , ..., C N ) includes obtaining a second vector representing the utterance, and determining similarity degrees of the second vector to the plurality of first vectors, and then selecting, from the plurality of concepts, the concept corresponding to the utterance based on the similarity degrees.
  • a symbolic representation SR 1 , SR 2 , ..., SR N
  • the retrieving step further includes determining a cluster of symbolic representation to which the selected concept corresponds, for example based upon the similarity of labanotations, and then retrieving, from the cluster of symbolic representation, a representative symbolic representation stored in the gesture library 108.
  • the symbolic representation is provided to cause the robot 100 to perform the gesture.
  • causing the robot 100 to perform the gesture includes executing the labanotation along the first dimension to trigger the plurality of body parts to perform the gesture according to the respective orientations in the plurality of time slots 301.
  • the time passes from the bottom to the top (as labeled by the arrow) , and a specific combination of various symbols indicating the various orientations of multiple body parts at a given time slot 301 will be executed, so that the robot 100 can continuously perform the corresponding motion with respect to time.
  • the method 200 may further include a step of causing the robot 100 to speak the utterance in synchronization with the gesture. In this way, a more natural and smoother communication between a human and a robot can be achieved.
  • the gesture library 108 or the labanotation may be stored at a remote server 200, such as a cloud computer, as illustrated in FIG. 1.
  • a remote server 200 such as a cloud computer
  • the user may easily update gesture by editing/change the labanotation on the cloud computer, as required. This is quite helpful especially for avoiding user’s boredom due to the repeated and same gestures every day.
  • the robot 100 is allowed to perform different gestures every day.
  • gesture library 108 only some part of the gesture library 108 is stored at a remote server 200, while some other part of the gesture library 108 is stored in local.
  • specialized gestures as part of gesture library 108 can be designed using labanotation and saved locally, while for the random/general conversations, general gestures as the other part of the gesture library 108 will be stored at a remote server 200.
  • FIG. 5 is a block diagram of apparatus 10 suitable for implementing one or more implementations of the subject matter described herein.
  • the apparatus 10 may function as discussed above with reference to FIG. 1. It is to be understood that the apparatus 10 is not intended to suggest any limitation as to scope of use or functionality of the subject matter described herein, as various implementations may be implemented in diverse general-purpose or special-purpose computing environments.
  • the apparatus 10 includes at least one processing unit (also called processor) 102 and a memory 104.
  • the processing unit 102 executes computer-executable instructions and may be a real or a virtual processor. In a multi-processing system, multiple processing units execute computer-executable instructions to increase processing power.
  • the memory 104 may be volatile memory (e.g., registers, cache, RAM) , non-volatile memory (e.g., ROM, EEPROM, flash memory) , or some combination thereof.
  • the memory 104 is coupled to the processing unit 102 and storing instructions for execution by the processing unit 102. Those instructions, when executed by the processing unit 102, causes the apparatus 10 to: receive a utterance to be spoken by the robot 100; determine a concept (C 1 , C 2 , ..., C N ) corresponding to the received utterance; retrieve, from predetermined gesture library 108, a symbolic representation (SR 1 , SR 2 , ..., SR N ) of a gesture that is paired with the determined concept (C 1 , C 2 , ..., C N ) ; and cause the robot 100 to execute the symbolic representation to perform the gesture.
  • a symbolic representation SR 1 , SR 2 , ..., SR N
  • the apparatus 10 further includes one or more communication connections 510.
  • An interconnection mechanism such as a bus, controller, or network interconnects the components of the apparatus 10.
  • operating system software provides an operating environment for other software executing in the apparatus 10, and coordinates activities of the components of the apparatus 10.
  • the communication connection (s) 510 enables communication over a communication medium to another computing entity. Additionally, functionality of the components of the apparatus 10 may be implemented in a single computing machine or in multiple computing machines that are able to communicate over communication connections. Thus, the apparatus 10 may operate in a networked environment (for example, the robot system environment 1000) using logical connections to one or more other servers, network PCs, or another common network node.
  • communication media include wired or wireless networking techniques.
  • a method for robot gesture generation comprises: determining a concept corresponding to a utterance to be spoken by a robot; retrieving, from a predetermined gesture library, a symbolic representation of a gesture that corresponds to the determined concept; and providing the symbolic representation (SR 1 , SR 2 , ..., SR N ) to cause the robot to perform the gesture.
  • retrieving the symbolic representation comprises: retrieving a labanotation defining orientations of at least one body part of the robot with respect to a plurality of time slots.
  • the method further comprises creating the gesture library by: capturing a gesture of a performer; determining, from the captured gesture, orientations of the at least one body part of the performer in the plurality of time slots; generating symbols representing the orientations in association with the corresponding time slots as a part of the labanotation.
  • the method further comprises grouping a plurality of labanotations into a cluster of labanotation; and determining, from the cluster of labanotation, a representative labanotation.
  • At least a part of the gesture library is stored at a remote server.
  • the at least one body part includes a plurality of body parts
  • the labanotation includes a first dimension corresponding to the plurality of time slots and a second dimension corresponding to the plurality of body parts.
  • providing the symbolic representation to cause the robot to perform the gesture comprises: executing the labanotation along the first dimension to trigger the plurality of body parts to perform the gesture according to the respective orientations in the plurality of time slots.
  • the method further comprises: causing the robot to speak the utterance in synchronization with the gesture.
  • the predetermined gesture library includes a plurality of first vectors representing a plurality of concepts, respectively, and the retrieving, from a predetermined gesture library, a symbolic representation of a gesture that corresponds to the determined concept comprises: obtaining a second vector representing the utterance; determining similarity degrees of the second vector to the plurality of first vectors; and selecting, from the plurality of concepts, the concept corresponding to the utterance based on the similarity degrees.
  • the gesture library includes a plurality of first vectors representing a plurality of concepts, respectively, and the retrieving, from a predetermined gesture library (108) , a symbolic representation of a gesture that corresponds to the determined concept comprises: obtaining a second vector representing the utterance; determining similarity degrees of the second vector to the plurality of first vectors; and selecting, from the plurality of concepts, the concept corresponding to the utterance based on the similarity degrees; determining a cluster of symbolic representation to which the selected concept corresponds; and retrieving, from the cluster of symbolic representation, a representative symbolic representation.
  • an apparatus for robot gesture generation comprises: a processing unit; and a memory coupled to the processing unit and storing instructions for execution by the processing unit, the instructions, when executed by the processing unit, causing the apparatus to: determine a concept corresponding to a received utterance to be spoken by a robot; retrieve, from a predetermined gesture library, a symbolic representation of a gesture that corresponds to the determined concept; and provide the symbolic representation to cause the robot to perform the gesture.
  • retrieving the symbolic representation comprises: retrieving a labanotation defining orientations of at least one body part of the robot with respect to a plurality of time slots.
  • the apparatus is further configured to create the gesture library by: capturing a gesture of a performer; determining, from the captured gesture, orientations of the at least one body part of the robot in the plurality of time slots; generating symbols representing the orientations in association with the corresponding time slots as a part of the labanotation; and relating the labanotation to a concept.
  • the apparatus is further configured to group a plurality of labanotations into a cluster of labanotation; and determine, from the cluster of labanotation, a representative labanotation.
  • At least part of the gesture library is stored at a remote server.
  • the at least one body part includes a plurality of body parts
  • the labanotation includes a first dimension corresponding to the plurality of time slots and a second dimension corresponding to the plurality of body parts.
  • providing the symbolic representation to cause the robot to perform the gesture comprises: executing the labanotation along the first dimension to trigger the plurality of body parts to perform the gesture according to the respective orientations in the plurality of time slots.
  • the apparatus is further configured to: cause the robot to speak the utterance in synchronization with the gesture.
  • the predetermined gesture library includes a plurality of first vectors representing a plurality of concepts, respectively, and the retrieving, from a predetermined gesture library (108) , a symbolic representation of a gesture that corresponds to the determined concept comprises: obtaining a second vector representing the utterance; determining similarity degrees of the second vector to the plurality of first vectors; and selecting, from the plurality of concepts, the concept corresponding to the utterance based on the similarity degrees.
  • the gesture library includes a plurality of first vectors representing a plurality of concepts, respectively, and the retrieving, from a predetermined gesture library, a symbolic representation of a gesture that corresponds to the determined concept comprises: obtaining a second vector representing the utterance; determining similarity degrees of the second vector to the plurality of first vectors; and selecting, from the plurality of concepts, the concept corresponding to the utterance based on the similarity degrees; determining a cluster of symbolic representation to which the selected concept corresponds; and retrieving, from the cluster of symbolic representation, a representative symbolic representation.
  • a computer program product for storage of a solid-state drive.
  • the computer program product is tangibly stored on a non-transient computer-readable medium and comprising machine-executable instructions, the machine-executable instructions, when executed, causing a machine to: determine a concept corresponding to a received utterance to be spoken by the robot; retrieve, from predetermined gesture library, a symbolic representation of a gesture that corresponds to the determined concept; and provide the symbolic representation to cause the robot to perform the gesture.
  • retrieving the symbolic representation comprises: retrieving a labanotation defining orientations of at least one body part of the robot with respect to a plurality of time slots.
  • the computer program further comprises causing a machine to: create the gesture library by: capturing a gesture of a performer; determining, from the captured gesture, orientations of the at least one body part of the performer in the plurality of time slots; generating symbols representing the orientations in association with the corresponding time slots as a part of the labanotation; and relating the labanotation to a concept.
  • At least part of the gesture library is stored at a remote server.

Landscapes

  • Engineering & Computer Science (AREA)
  • Physics & Mathematics (AREA)
  • Theoretical Computer Science (AREA)
  • Robotics (AREA)
  • Health & Medical Sciences (AREA)
  • Computational Linguistics (AREA)
  • Mechanical Engineering (AREA)
  • Human Computer Interaction (AREA)
  • General Health & Medical Sciences (AREA)
  • General Physics & Mathematics (AREA)
  • General Engineering & Computer Science (AREA)
  • Audiology, Speech & Language Pathology (AREA)
  • Multimedia (AREA)
  • Evolutionary Computation (AREA)
  • Mathematical Physics (AREA)
  • Software Systems (AREA)
  • Data Mining & Analysis (AREA)
  • Computing Systems (AREA)
  • Artificial Intelligence (AREA)
  • Biomedical Technology (AREA)
  • Molecular Biology (AREA)
  • Biophysics (AREA)
  • Life Sciences & Earth Sciences (AREA)
  • Acoustics & Sound (AREA)
  • Computer Vision & Pattern Recognition (AREA)
  • Psychiatry (AREA)
  • Social Psychology (AREA)
  • Manipulator (AREA)
  • User Interface Of Digital Computer (AREA)

Abstract

A method and apparatus for robot gesture generation is described. Generally speaking, a concept corresponding to a utterance to be spoken by a robot is determined (204). After a concept is determined or selected, a symbolic representation of a gesture that corresponds to the determined concept is retrieved from a predetermined gesture library (206). Subsequently, the symbolic representation is provided to cause the robot to perform the gesture (208). In such way, a more natural, comprehensive and effective communication between human and robots may be achieved.

Description

ROBOT GESTURE GENERATION BACKGROUND
Numerous types of robots have been used in a wide variety of applications. Traditionally, robots are mainly used in industrial applications. Recently, more and more robots have been utilized in the field of human-robot interaction, such as robots that are used in family service and medical service. With such trend, how to realize a more natural and smoother communication between a human being and a robot attracts more attention.
SUMMARY
Embodiments of the subject matter described herein provide a method, apparatus and robot system for robot gesture generation. Generally speaking a concept corresponding to a utterance to be spoken by a robot is determined. After the concept is determined or selected, a symbolic representation of a gesture that corresponds to the determined concept is retrieved from a predetermined gesture library. Subsequently, the symbolic representation is provided to cause the robot to perform the gesture.
On the contrary to the conventional chatting robots, which may not be provided with any gesture responses along with the spoken utterances from chatting intelligence, or may only provide limited but not appropriate gesture function, embodiments of the present disclosure provide a method and an apparatus for robot gesture generation, especially in a chatting robot system, which enable a generation of appropriate gesture along with or in synchronization with the spoken utterances. In such way, a more natural, comprehensive and effective communication may be achieved between human and robots.
It is to be understood that the Summary is not intended to identify key or essential features of implementations of the subject matter described herein, nor is it intended to be used to limit the scope of the subject matter described herein. Other features of the subject matter described herein will become easily comprehensible through the description below.
DESCRIPTION OF DRAWINGS
The above and other objectives, features and advantages of the subject matter described herein will become more apparent through more detailed depiction of example embodiments of the subject matter described herein in conjunction with the accompanying drawings, wherein in the example embodiments of the subject matter described herein, same reference numerals usually represent same components.
FIG. 1 is a schematic diagram illustrating a robot system in which example implementations of the subject matter described herein can be implemented.
FIG. 2 illustrates a flowchart of a method for robot gesture generation according to embodiments of the present disclosure.
FIG. 3A illustrates a schematic diagram of a typical labanotation and the corresponding concept represented by the labanotation according to embodiments of the present disclosure.
FIG. 3B illustrates example symbols of body parts of a robot.
FIG. 4 illustrates a flowchart of a process for mapping a utterance to a vector according to embodiments of the present disclosure.
FIG. 5 is a block diagram of an apparatus suitable for implementing one or more implementations of the subject matter described herein.
Throughout the drawings, the same or similar reference symbols are used to indicate the same or similar elements.
DETAILED DESCRIPTION OF EMBODIEMTNS
Principles of the present disclosure will now be described with reference to several example embodiments shown in the drawings. Though example embodiments of the present disclosure are illustrated in the drawings, it is to be understood that the embodiments are described only to facilitate those skilled in the art in better understanding and thereby achieving the present disclosure, rather than to limit the scope  of the disclosure in any manner.
As used herein, the phrase “include (s) ” and its variants shall be interpreted as an open term meaning “including but not limited to. ” The phrase “based on” shall be interpreted as “at least partially based on. ” The term “a” is to be read as “one or more” unless otherwise specified. The phrase “an embodiment” or “one embodiment” shall be interpreted as “at least one embodiment. ” The phrase “another embodiment” shall be interpreted as “at least one other embodiment. ” The phrases like “first” and “second” may refer to different or the same objects. Other definitions might also be included explicitly and implicitly in the following description.
FIG. 1 illustrates a schematic diagram of a robot system 1000 according to embodiments of the present disclosure. As shown in FIG. 1, the robot system 1000 generally includes a robot 100, a server 300, and an apparatus 10 according to embodiments of present application. The apparatus 10 is used for generating appropriate gesture along with the spoken utterances, depending on the utterance to be spoken by the robot 100.
The robot 100 may be, for example, a chatting robot. The server 300 is usually a device capable of processing the received message and providing the processed information and may be in the form of a cloud-based computer, for example, a chatting intelligence for the case of a chatting robot. That is, a user may have a “conversation” with the chatting intelligence 300 through, for example, an audio-visual-tactile intelligence.
During a normal chatting process between a user (not shown) and the robot 100, the user may first make expressions or ask questions to the robot 100, and then expect to receive a response, such as an answer to the user’s question, from the robot 100.
Conventionally, after the server 300 receives the expressions/utterances from the user via a robot, the processor associated with the server 300 may perform some analyses based on the received expressions/questions, and prepare accordingly a response or an answer to the expression that is to be spoken by the robot 100. However, as mentioned above, some conventional chatting robots are not provided with any motion or gesture functions. Some other conventional chatting robots, on the other hand, may be  able to provide some gestures along with the spoken utterances, these gestures, however, may not be appropriate/correct ones, and for most of cases, these generated gestures may just be random ones.
According to various embodiments of the present application, an appropriate gesture response to the user along with or in synchronization with the voice output from the robot 100 can be achieved. In other words, a translation mechanism from spoken language to gestures can be realized. Furthermore, due to the fact that those gestures are generated based on the to-be-spoken utterances, or at least based on some concepts extracted from those utterances, the generated gesture in such a way would be more accurate and natural.
For the sake of discussions, example implementations of the subject matter described herein will be described with reference to the robot system 1000. However, it is to be understood that such a robot system 1000 is described merely for the purpose of illustration, without suggesting any limitations as to the scope of the subject matter described herein. For example, the ideas and principles are applicable to a stand-alone machine as well.
FIG. 2 illustrates a flow chart of a method 200 for robot gesture generation according to embodiments of the present disclosure. The method 200 can be executed, for example, on the apparatus 10 as illustrated in FIG. 1. The apparatus 10 as shown can be a client device or a cloud-based apparatus, or it can be part of the server 300 as illustrated in FIG. 1. It should be understood that the method 200 may also include additional actions not shown and/or omit the illustrated steps. Scope of the subject matter described herein is not limited in this aspect.
The method 200 will be described as below with reference to FIG. 1, FIG. 3 and FIG. 4. At 204, one or more concepts (C1, C2, ..., CN) corresponding to a utterance to be spoken by a robot 100 may be determined. The utterance that is to be spoken by the robot 100 may be a proactive or responsive vocal interaction in a conversation with the user. It may be a bit of spoken language from a word, a phrase, a plurality of words to a full utterance. It is to be noted that although the utterance to be spoken as illustrated in FIG. 1 is received from the server 300, in some other embodiments, the utterance can be  instead obtained directly from the apparatus 10.
As mentioned above, in some embodiments, the user may first make an expression/question to the robot 100 and further to the server 300 by, for example, transmitting his/her voice via a microphone and his/her image via a camera integrated on the robot 100. After the server 300 receives the expression/question from the user, the processor (not shown) associated with the server 300 will perform analyses based on the received expressions, and prepare a utterance that is to be spoken by the robot 100 accordingly, as a response or an answer to that question.
In some embodiments, the incoming expression/question from the user will be firstly checked locally. If it is included in a special set, or in other words, if it is one of special conversations, then an answer to this expression/question will be prepared in local. If not, the incoming expression/question will be further sent to the server 300 for a response.
In this context of the subject matter described herein, a concept is generally defined as one representative extracted from a cluster of words. The cluster of words may contain those words that share the same or a similar meaning within a context and thus are located in close proximity to one another in a vector space. Typical concepts may include, for example, “Hello, ” “Good, ” ” Thanks, ” “Hungry, ” and so on. It is to be noted that scope of the subject matter described herein is not limited in the formats or contents included in the concepts. Users may define other particular concepts as needed.
The robot system 1000 as illustrated in FIG. 1 includes a gesture library 108 which is created to store a plurality of predetermined gestures and the corresponding concepts. In some embodiments, each concept that corresponds to a gesture in the gesture library 108 may be represented by a first vector. In this case, determining the concept (C1, C2, ..., CN) corresponding to the utterance may include obtaining a second vector representing the utterance, and then determining the similarity degrees of the second vector to the plurality of first vectors. In this way, based on the determined similarity degrees, the concept corresponding to the utterance can be selected from the plurality of concepts stored in the gesture library 108.
In an example embodiment, the received utterance or expression may be first  given to a system, such as “distributed word embedding” system, to convert the utterance to a vector and thus map to a concept. Next, by examining the similarity degree between the vector converted from the utterance and each of those vectors from the gesture library 108, the concept having a maximum similarity degree may be selected from the plurality of concepts, as the determined concept corresponding to the utterance.
FIG. 4 shows an example illustrating a process 400 of conversion from an utterance to a vector. As shown in FIG. 4, the utterance 410 that is to be spoken by the robot 100 is “I agree with you, ” and the “distributed word embedding” system may determine the best matched concept 420, for example to be “agree, ” and then the concept “Agree” will be mapped to a corresponding vector 430 for comparison with those vectors stored in the gesture library 108. It is to be understood that although this example only describes a word-based matching mechanism (that is, based on a word-level similarity) , the concept matching based on other levels of similarities, such as a phrase-level similarity, a sentence-level similarity, a document-level similarity and even a semantic-level similarity, are also possible.
Continuing to refer to FIG. 1, at 206, a symbolic representation (SR1, SR2, ..., SRN) of a gesture that corresponds to the determined concept (C1, C2, ..., CN) is retrieved from the predetermined gesture library 108.
In some embodiments, the gesture library 108 may be previously created by using a so-called “learning from observation” technique. In particular, a gesture of the robot 100 needs to be recorded first. The gesture is usually performed by a performer (for example, a human) in front of a camera in association with the robot 100. Some important poses from the performer then can be captured or “observed” by the camera and subsequently converted into symbolic representations (also called symbols for short) that correspond to the performed gesture. In some embodiments, retrieving the symbolic representation (SR1, SR2, ..., SRN) includes retrieving a labanotation (LA1, LA2, ..., LAN) . Consequently, with the obtained labanotation, a relationship between the labanotation representing the performed gesture and a concept (e.g. labanotation-concept pairs) can be established and stored in the gesture library 108 for a future use.
Labanotation has been used for recording human dances. Labanotation herein  particularly defines orientations of at least one body part of the robot 100 with respect to a plurality of time slots. According to the inventor’s observation, the labanotation employed in the robot system 1000 is necessary and sufficient condition to describe human actions. This is because, watching the same dance by several dancers end up with the same labanotation, and on the other hand, from one labanotation, any dancer may perform/reconstruct the same dance.
Furthermore, labanotation is machine-independent (or hardware-independent) , and thereby can overcome the hardware difference. In addition, as a concise symbolic representation, it is easy to transmit labanotation between a robot and the cloud computer through limited communication channels. Also, labanotation requires smaller memory than other types of representations.
In some embodiments, through a continuously captured/recorded gesture, orientations of the at least one body part of the robot 100 in the plurality of time slots 301 can be determined, and then symbols corresponding the orientations can be obtained. After that, the symbols in association with the corresponding time slots 301 as a part of the labanotation can be saved.
In some embodiments, the at least one body part includes a plurality of body parts, and the labanotation includes a first dimension corresponding to the plurality of time slots 301 and a second dimension corresponding to the plurality of body parts (that is, two-dimensional labanotation) . FIG. 3A illustrated such a 2D labanotation representative of a concept of “Good” . In this labanotation, each of columns (that is, the second dimension) corresponds to one specific body part, such as left hand, left arm, support, right arm, right hand, head. Each row (that is, the first dimension) corresponds to the time slots 301 with a given duration. Further, a symbol represents to which direction that the body part is oriented at that time.
It is to be noted that the sample labanotation in FIG. 3A is merely shown for purpose of illustration without suggesting any limitation as to the scope of the subject matter described herein. In other words, a more complicated labanotation with more body parts involved, such as left elbow or right knee (labeled by dotted circles) as shown in FIG. 3B is also possible.
Alternatively or additionally, for the previously created the gesture library 108, a plurality of labanotations of a plurality of gestures can be grouped into a cluster of labanotation, and for each cluster of labanotation, a representative labanotation can be determined. In this case, among multiple labanotations, similar labanotations indicating similar gestures can be clustered/grouped. As a result, a cluster of labanotation/gestures (rather than a single labanotation/gesture) is represented by a representative labanotation. In this way, it is guaranteed that any concepts corresponding to a same cluster will give a same gesture.
Compared to the gesture library 108 containing “one-to-one” concept-labanotation pairs, the gesture library 108 created in such an alternative way described as above contains “multiple-to-one” concepts to labanotation matches. Accordingly, retrieving, from the predetermined gesture library 108, a symbolic representation (SR1, SR2, ..., SRN) of a gesture that corresponds to the determined concept (C1, C2, ..., CN) includes obtaining a second vector representing the utterance, and determining similarity degrees of the second vector to the plurality of first vectors, and then selecting, from the plurality of concepts, the concept corresponding to the utterance based on the similarity degrees. With the selected concept, the retrieving step further includes determining a cluster of symbolic representation to which the selected concept corresponds, for example based upon the similarity of labanotations, and then retrieving, from the cluster of symbolic representation, a representative symbolic representation stored in the gesture library 108.
At 208, the symbolic representation is provided to cause the robot 100 to perform the gesture. In some embodiments, causing the robot 100 to perform the gesture includes executing the labanotation along the first dimension to trigger the plurality of body parts to perform the gesture according to the respective orientations in the plurality of time slots 301.
As shown in FIG. 3A, when executing the labanotation as shown in FIG. 3A, the time passes from the bottom to the top (as labeled by the arrow) , and a specific combination of various symbols indicating the various orientations of multiple body parts at a given time slot 301 will be executed, so that the robot 100 can continuously perform  the corresponding motion with respect to time.
Alternatively or additionally, the method 200 may further include a step of causing the robot 100 to speak the utterance in synchronization with the gesture. In this way, a more natural and smoother communication between a human and a robot can be achieved.
In some embodiments, the gesture library 108 or the labanotation may be stored at a remote server 200, such as a cloud computer, as illustrated in FIG. 1. In this case, the user may easily update gesture by editing/change the labanotation on the cloud computer, as required. This is quite helpful especially for avoiding user’s boredom due to the repeated and same gestures every day. In other words, by changing the labanotation, the robot 100 is allowed to perform different gestures every day. Of course, it is also possible to store the gesture library 108 or the labanotation in local, for example, in the apparatus 10.
In some embodiments, only some part of the gesture library 108 is stored at a remote server 200, while some other part of the gesture library 108 is stored in local. As mentioned above, for some specialized conversations, specialized gestures as part of gesture library 108 can be designed using labanotation and saved locally, while for the random/general conversations, general gestures as the other part of the gesture library 108 will be stored at a remote server 200.
FIG. 5 is a block diagram of apparatus 10 suitable for implementing one or more implementations of the subject matter described herein. For example, the apparatus 10 may function as discussed above with reference to FIG. 1. It is to be understood that the apparatus 10 is not intended to suggest any limitation as to scope of use or functionality of the subject matter described herein, as various implementations may be implemented in diverse general-purpose or special-purpose computing environments.
As shown, the apparatus 10 includes at least one processing unit (also called processor) 102 and a memory 104. The processing unit 102 executes computer-executable instructions and may be a real or a virtual processor. In a multi-processing system, multiple processing units execute computer-executable instructions to increase processing power. The memory 104 may be volatile memory (e.g., registers, cache,  RAM) , non-volatile memory (e.g., ROM, EEPROM, flash memory) , or some combination thereof.
In accordance with implementations of the subject matter described herein, the memory 104 is coupled to the processing unit 102 and storing instructions for execution by the processing unit 102. Those instructions, when executed by the processing unit 102, causes the apparatus 10 to: receive a utterance to be spoken by the robot 100; determine a concept (C1, C2, ..., CN) corresponding to the received utterance; retrieve, from predetermined gesture library 108, a symbolic representation (SR1, SR2, ..., SRN) of a gesture that is paired with the determined concept (C1, C2, ..., CN) ; and cause the robot 100 to execute the symbolic representation to perform the gesture.
In the example shown in FIG. 4, the apparatus 10 further includes one or more communication connections 510. An interconnection mechanism (not shown) such as a bus, controller, or network interconnects the components of the apparatus 10. Typically, operating system software (not shown) provides an operating environment for other software executing in the apparatus 10, and coordinates activities of the components of the apparatus 10.
The communication connection (s) 510 enables communication over a communication medium to another computing entity. Additionally, functionality of the components of the apparatus 10 may be implemented in a single computing machine or in multiple computing machines that are able to communicate over communication connections. Thus, the apparatus 10 may operate in a networked environment (for example, the robot system environment 1000) using logical connections to one or more other servers, network PCs, or another common network node. By way of example, and not limitation, communication media include wired or wireless networking techniques.
Hereinafter, some example implementations of the subject matter described herein will be enumerate.
In some embodiments, a method for robot gesture generation is provided. The method comprises: determining a concept corresponding to a utterance to be spoken by a robot; retrieving, from a predetermined gesture library, a symbolic representation of a gesture that corresponds to the determined concept; and providing the symbolic  representation (SR1, SR2, ..., SRN) to cause the robot to perform the gesture.
In some embodiments, retrieving the symbolic representation comprises: retrieving a labanotation defining orientations of at least one body part of the robot with respect to a plurality of time slots.
In some embodiments, the method further comprises creating the gesture library by: capturing a gesture of a performer; determining, from the captured gesture, orientations of the at least one body part of the performer in the plurality of time slots; generating symbols representing the orientations in association with the corresponding time slots as a part of the labanotation.
In some embodiments, the method further comprises grouping a plurality of labanotations into a cluster of labanotation; and determining, from the cluster of labanotation, a representative labanotation.
In some embodiments, at least a part of the gesture library is stored at a remote server.
In some embodiments, the at least one body part includes a plurality of body parts, and the labanotation includes a first dimension corresponding to the plurality of time slots and a second dimension corresponding to the plurality of body parts.
In some embodiments, providing the symbolic representation to cause the robot to perform the gesture comprises: executing the labanotation along the first dimension to trigger the plurality of body parts to perform the gesture according to the respective orientations in the plurality of time slots.
In some embodiments, the method further comprises: causing the robot to speak the utterance in synchronization with the gesture.
In some embodiments, the predetermined gesture library includes a plurality of first vectors representing a plurality of concepts, respectively, and the retrieving, from a predetermined gesture library, a symbolic representation of a gesture that corresponds to the determined concept comprises: obtaining a second vector representing the utterance; determining similarity degrees of the second vector to the plurality of first vectors; and selecting, from the plurality of concepts, the concept corresponding to the utterance based  on the similarity degrees.
In some embodiments, the gesture library includes a plurality of first vectors representing a plurality of concepts, respectively, and the retrieving, from a predetermined gesture library (108) , a symbolic representation of a gesture that corresponds to the determined concept comprises: obtaining a second vector representing the utterance; determining similarity degrees of the second vector to the plurality of first vectors; and selecting, from the plurality of concepts, the concept corresponding to the utterance based on the similarity degrees; determining a cluster of symbolic representation to which the selected concept corresponds; and retrieving, from the cluster of symbolic representation, a representative symbolic representation.
In some embodiments, an apparatus for robot gesture generation is provided. The apparatus comprises: a processing unit; and a memory coupled to the processing unit and storing instructions for execution by the processing unit, the instructions, when executed by the processing unit, causing the apparatus to: determine a concept corresponding to a received utterance to be spoken by a robot; retrieve, from a predetermined gesture library, a symbolic representation of a gesture that corresponds to the determined concept; and provide the symbolic representation to cause the robot to perform the gesture.
In some embodiments, retrieving the symbolic representation comprises: retrieving a labanotation defining orientations of at least one body part of the robot with respect to a plurality of time slots.
In some embodiments, the apparatus is further configured to create the gesture library by: capturing a gesture of a performer; determining, from the captured gesture, orientations of the at least one body part of the robot in the plurality of time slots; generating symbols representing the orientations in association with the corresponding time slots as a part of the labanotation; and relating the labanotation to a concept.
In some embodiments, the apparatus is further configured to group a plurality of labanotations into a cluster of labanotation; and determine, from the cluster of labanotation, a representative labanotation.
In some embodiments, at least part of the gesture library is stored at a remote server.
In some embodiments, the at least one body part includes a plurality of body parts, and the labanotation includes a first dimension corresponding to the plurality of time slots and a second dimension corresponding to the plurality of body parts.
In some embodiments, providing the symbolic representation to cause the robot to perform the gesture comprises: executing the labanotation along the first dimension to trigger the plurality of body parts to perform the gesture according to the respective orientations in the plurality of time slots.
In some embodiments, the apparatus is further configured to: cause the robot to speak the utterance in synchronization with the gesture.
In some embodiments, the predetermined gesture library includes a plurality of first vectors representing a plurality of concepts, respectively, and the retrieving, from a predetermined gesture library (108) , a symbolic representation of a gesture that corresponds to the determined concept comprises: obtaining a second vector representing the utterance; determining similarity degrees of the second vector to the plurality of first vectors; and selecting, from the plurality of concepts, the concept corresponding to the utterance based on the similarity degrees.
In some embodiments, the gesture library includes a plurality of first vectors representing a plurality of concepts, respectively, and the retrieving, from a predetermined gesture library, a symbolic representation of a gesture that corresponds to the determined concept comprises: obtaining a second vector representing the utterance; determining similarity degrees of the second vector to the plurality of first vectors; and selecting, from the plurality of concepts, the concept corresponding to the utterance based on the similarity degrees; determining a cluster of symbolic representation to which the selected concept corresponds; and retrieving, from the cluster of symbolic representation, a representative symbolic representation.
In some embodiments, a computer program product for storage of a solid-state drive is provided. The computer program product is tangibly stored on a non-transient  computer-readable medium and comprising machine-executable instructions, the machine-executable instructions, when executed, causing a machine to: determine a concept corresponding to a received utterance to be spoken by the robot; retrieve, from predetermined gesture library, a symbolic representation of a gesture that corresponds to the determined concept; and provide the symbolic representation to cause the robot to perform the gesture.
In some embodiments, retrieving the symbolic representation comprises: retrieving a labanotation defining orientations of at least one body part of the robot with respect to a plurality of time slots.
In some embodiments, the computer program further comprises causing a machine to: create the gesture library by: capturing a gesture of a performer; determining, from the captured gesture, orientations of the at least one body part of the performer in the plurality of time slots; generating symbols representing the orientations in association with the corresponding time slots as a part of the labanotation; and relating the labanotation to a concept.
In some embodiments, at least part of the gesture library is stored at a remote server.
It should be appreciated that the above detailed embodiments of the present disclosure are only to exemplify or explain principles of the present disclosure and not to limit the present disclosure. Therefore, any modifications, equivalent alternatives and improvement, etc. without departing from the spirit and scope of the present disclosure shall be included in the scope of protection of the present disclosure. Meanwhile, appended claims of the present disclosure aim to cover all the variations and modifications falling under the scope and boundary of the claims or equivalents of the scope and boundary.

Claims (20)

  1. A method for robot gesture generation (100) , comprising:
    determining a concept (C1, C2, ..., CN) corresponding to an utterance to be spoken by a robot (100) ;
    retrieving, from a predetermined gesture library (108) , a symbolic representation (SR1, SR2, ..., SRN) of a gesture that corresponds to the determined concept (C1, C2, ..., CN) ;
    providing the symbolic representation (SR1, SR2, ..., SRN) to cause the robot (100) to perform the gesture.
  2. The method of claim 1, wherein retrieving the symbolic representation (SR1, SR2, ..., SRN) comprises:
    retrieving a labanotation (LA1, LA2, ..., LAN) defining orientations of at least one body part of the robot (100) with respect to a plurality of time slots (301) .
  3. The method of claim 2, further comprising creating the gesture library (108) by:
    capturing a gesture of a performer;
    determining, from the captured gesture, orientations of the at least one body part of the performer in the plurality of time slots (301) ;
    generating symbols representing the orientations in association with the corresponding time slots (301) as a part of the labanotation; and
    relating the labanotation to a concept.
  4. The method of claim 3, further comprising:
    grouping a plurality of labanotations into a cluster of labanotation; and
    determining, from the cluster of labanotation, a representative labanotation.
  5. The method of claim 1, wherein at least a part of the gesture library (108) is stored at a remote server (200) .
  6. The method of claim 2, wherein the at least one body part includes a plurality of body parts, and the labanotation includes a first dimension corresponding to the plurality of time slots (301) and a second dimension corresponding to the plurality of body parts.
  7. The method of claim 6, wherein providing the symbolic representation to cause the robot (100) to perform the gesture comprises:
    executing the labanotation along the first dimension to trigger the plurality of body parts to perform the gesture according to the respective orientations in the plurality of time slots (301) .
  8. The method of claim 1, further comprising:
    causing the robot (100) to speak the utterance in synchronization with the gesture.
  9. The method of claim 1, wherein the gesture library (108) includes a plurality of first vectors representing a plurality of concepts, respectively, and the retrieving, from a predetermined gesture library (108) , a symbolic representation (SR1, SR2, ..., SRN) of a gesture that corresponds to the determined concept (C1, C2, ..., CN) comprises:
    obtaining a second vector representing the utterance;
    determining similarity degrees of the second vector to the plurality of first vectors; and
    selecting, from the plurality of concepts, the concept corresponding to the utterance based on the similarity degrees.
  10. The method of claim 4, wherein the gesture library (108) includes a plurality of first vectors representing a plurality of concepts, respectively, and the retrieving, from a predetermined gesture library (108) , a symbolic representation (SR1, SR2, ..., SRN) of a gesture that corresponds to the determined concept (C1, C2, ..., CN) comprises:
    obtaining a second vector representing the utterance;
    determining similarity degrees of the second vector to the plurality of first vectors; and
    selecting, from the plurality of concepts, the concept corresponding to the utterance based on the similarity degrees;
    determining a cluster of symbolic representation to which the selected concept corresponds; and
    retrieving, from the cluster of symbolic representation, a representative symbolic representation.
  11. An apparatus (10) for robot gesture generation (100) , comprising:
    a processing unit (102) ; and
    a memory (104) coupled to the processing unit (102) and storing instructions for execution by the processing unit, the instructions, when executed by the processing unit (102) , causing the apparatus to:
    determine a concept (C1, C2, ..., CN) corresponding to a received utterance to be spoken by a robot (100) ;
    retrieve, from a predetermined gesture library (108) , a symbolic representation (SR1, SR2, ..., SRN) of a gesture that corresponds to the determined concept (C1, C2, ..., CN) ; and
    provide the symbolic representation (SR1, SR2, ..., SRN) to causing the robot (100) to perform the gesture.
  12. The apparatus (10) of claim 11, wherein retrieving the symbolic representation (SR1, SR2, ..., SRN) comprises:
    retrieving a labanotation (LA1, LA2, ..., LAN) defining orientations of at least one body part of the robot (100) with respect to a plurality of time slots (301) .
  13. The apparatus (10) of claim 12, wherein the apparatus is further configured to create the gesture library (108) by:
    capturing a gesture of a performer;
    determining, from the captured gesture, orientations of the at least one body part of the performer in the plurality of time slots (301) ;
    generating symbols representing the orientations in association with the corresponding time slots (301) as a part of the labanotation; and
    relating the labanotation to a concept.
  14. The apparatus (10) of claim 13, wherein the apparatus is further configured to
    group a plurality of labanotations into a cluster of labanotation; and
    determine, from the cluster of labanotation, a representative labanotation.
  15. The apparatus (10) of claim 11, wherein at least part of the gesture library (108) are stored at a remote server (200) .
  16. The apparatus (10) of claim 12, wherein the at least one body part includes a plurality of body parts, and the labanotation includes a first dimension corresponding to the plurality of time slots (301) and a second dimension corresponding to the plurality of body parts.
  17. The apparatus (10) of claim 16, wherein providing the symbolic representation to cause the robot (100) to perform the gesture comprises:
    executing the labanotation along the first dimension to trigger the plurality of body parts to perform the gesture according to the respective orientations in the plurality of time slots (301) .
  18. The apparatus (10) of claim 11, wherein the predetermined gesture library (108) includes a plurality of first vectors representing a plurality of concepts, respectively, and the retrieving, from a predetermined gesture library (108) , a symbolic representation (SR1, SR2, ..., SRN) of a gesture that corresponds to the determined concept (C1, C2, ..., CN) comprises:
    obtaining a second vector representing the utterance;
    determining similarity degrees of second vector to the plurality of first vectors; and
    selecting, from the plurality of concepts, the concept corresponding to the utterance based on the similarity degrees.
  19. The apparatus of claim 11, wherein the gesture library (108) includes a plurality of first vectors representing a plurality of concepts, respectively, and the retrieving, from a predetermined gesture library (108) , a symbolic representation (SR1, SR2, ..., SRN) of a gesture that corresponds to the determined concept (C1, C2, ..., CN) comprises:
    obtaining a second vector representing the utterance;
    determining similarity degrees of second vector to the plurality of first vectors; and
    selecting, from the plurality of concepts, the concept corresponding to the utterance based on the similarity degrees;
    determining a cluster of symbolic representation to which the selected concept corresponds; and
    retrieving, from the cluster of symbolic representation, a representative symbolic representation.
  20. A computer program product for storage of a solid-state drive, the computer program product being tangibly stored on a non-transient computer-readable medium and comprising machine-executable instructions, the machine-executable instructions, when executed, causing a machine to:
    determine a concept (C1, C2, ..., CN) corresponding to a received utterance to be spoken by the robot (100) ;
    retrieve, from predetermined gesture library (108) , a symbolic representation (SR1, SR2, ..., SRN) of a gesture that corresponds to the determined concept (C1, C2, ..., CN) ; and
    provide the symbolic representation (SR1, SR2, ..., SRN) to cause the robot (100) to perform the gesture.
PCT/CN2016/109547 2016-12-12 2016-12-12 Robot gesture generation WO2018107343A1 (en)

Priority Applications (4)

Application Number Priority Date Filing Date Title
CN201680091517.3A CN110062683B (en) 2016-12-12 2016-12-12 Robot gesture generation method, device and computer readable medium
US16/469,024 US11443161B2 (en) 2016-12-12 2016-12-12 Robot gesture generation
EP16923912.6A EP3551393A4 (en) 2016-12-12 2016-12-12 Robot gesture generation
PCT/CN2016/109547 WO2018107343A1 (en) 2016-12-12 2016-12-12 Robot gesture generation

Applications Claiming Priority (1)

Application Number Priority Date Filing Date Title
PCT/CN2016/109547 WO2018107343A1 (en) 2016-12-12 2016-12-12 Robot gesture generation

Publications (1)

Publication Number Publication Date
WO2018107343A1 true WO2018107343A1 (en) 2018-06-21

Family

ID=62557899

Family Applications (1)

Application Number Title Priority Date Filing Date
PCT/CN2016/109547 WO2018107343A1 (en) 2016-12-12 2016-12-12 Robot gesture generation

Country Status (4)

Country Link
US (1) US11443161B2 (en)
EP (1) EP3551393A4 (en)
CN (1) CN110062683B (en)
WO (1) WO2018107343A1 (en)

Families Citing this family (1)

* Cited by examiner, † Cited by third party
Publication number Priority date Publication date Assignee Title
US11270717B2 (en) * 2019-05-08 2022-03-08 Microsoft Technology Licensing, Llc Noise reduction in robot human communication

Citations (4)

* Cited by examiner, † Cited by third party
Publication number Priority date Publication date Assignee Title
CN102596516A (en) * 2009-07-10 2012-07-18 奥尔德巴伦机器人公司 System and method for generating contextual behaviours of a mobile robot
CN103119644A (en) * 2010-07-23 2013-05-22 奥尔德巴伦机器人公司 Humanoid robot equipped with a natural dialogue interface, method for controlling the robot and corresponding program
WO2015158887A2 (en) 2014-04-17 2015-10-22 Aldebaran Robotics Method of performing multi-modal dialogue between a humanoid robot and user, computer program product and humanoid robot for implementing said method
CN105930785A (en) * 2016-04-15 2016-09-07 丁盛 Intelligent concealed-type interaction system

Family Cites Families (12)

* Cited by examiner, † Cited by third party
Publication number Priority date Publication date Assignee Title
US7136818B1 (en) 2002-05-16 2006-11-14 At&T Corp. System and method of providing conversational visual prosody for talking heads
ATE524784T1 (en) 2005-09-30 2011-09-15 Irobot Corp COMPANION ROBOTS FOR PERSONAL INTERACTION
US8224652B2 (en) 2008-09-26 2012-07-17 Microsoft Corporation Speech and text driven HMM-based body animation synthesis
US8751215B2 (en) * 2010-06-04 2014-06-10 Microsoft Corporation Machine based sign language interpreter
CN102506518A (en) 2011-10-11 2012-06-20 北京卡林新能源技术有限公司 Integrated heat-supplying air-conditioning control mechanism of solar ground source heat pump
FR2989209B1 (en) 2012-04-04 2015-01-23 Aldebaran Robotics ROBOT FOR INTEGRATING NATURAL DIALOGUES WITH A USER IN HIS BEHAVIOR, METHODS OF PROGRAMMING AND USING THE SAME
US9302393B1 (en) 2014-04-15 2016-04-05 Alan Rosen Intelligent auditory humanoid robot and computerized verbalization system programmed to perform auditory and verbal artificial intelligence processes
US9868212B1 (en) * 2016-02-18 2018-01-16 X Development Llc Methods and apparatus for determining the pose of an object based on point cloud data
DE112017003651T5 (en) * 2016-07-20 2019-04-04 Groove X, Inc. Autonomous robot that understands body contact
US11250844B2 (en) * 2017-04-12 2022-02-15 Soundhound, Inc. Managing agent engagement in a man-machine dialog
US11279041B2 (en) * 2018-10-12 2022-03-22 Dream Face Technologies, Inc. Socially assistive robot
US11270717B2 (en) * 2019-05-08 2022-03-08 Microsoft Technology Licensing, Llc Noise reduction in robot human communication

Patent Citations (4)

* Cited by examiner, † Cited by third party
Publication number Priority date Publication date Assignee Title
CN102596516A (en) * 2009-07-10 2012-07-18 奥尔德巴伦机器人公司 System and method for generating contextual behaviours of a mobile robot
CN103119644A (en) * 2010-07-23 2013-05-22 奥尔德巴伦机器人公司 Humanoid robot equipped with a natural dialogue interface, method for controlling the robot and corresponding program
WO2015158887A2 (en) 2014-04-17 2015-10-22 Aldebaran Robotics Method of performing multi-modal dialogue between a humanoid robot and user, computer program product and humanoid robot for implementing said method
CN105930785A (en) * 2016-04-15 2016-09-07 丁盛 Intelligent concealed-type interaction system

Non-Patent Citations (1)

* Cited by examiner, † Cited by third party
Title
See also references of EP3551393A4

Also Published As

Publication number Publication date
CN110062683A (en) 2019-07-26
US20190385040A1 (en) 2019-12-19
CN110062683B (en) 2023-04-28
US11443161B2 (en) 2022-09-13
EP3551393A1 (en) 2019-10-16
EP3551393A4 (en) 2020-08-12

Similar Documents

Publication Publication Date Title
CN108319599B (en) Man-machine conversation method and device
Gulcehre et al. Pointing the unknown words
US11487953B2 (en) Method and apparatus with natural language processing
US20240054767A1 (en) Multi-modal Model Training Method, Apparatus and Device, and Storage Medium
US20160092438A1 (en) Machine translation apparatus, machine translation method and program product for machine translation
US20180121801A1 (en) Method and device for classifying questions based on artificial intelligence
US20170004824A1 (en) Speech recognition apparatus, speech recognition method, and electronic device
JP2018026127A (en) Translation method, translation device, and computer program
US20190251174A1 (en) Machine translation method and apparatus
WO2018006472A1 (en) Knowledge graph-based human-robot interaction method and system
US11113335B2 (en) Dialogue system and computer program therefor
JP2007234024A (en) Method and apparatus for bilingual word alignment, method and apparatus for training bilingual word alignment model
US9442720B2 (en) Adding on-the-fly comments to code
JP2019049604A (en) Instruction statement estimation system and instruction statement estimation method
WO2020101743A1 (en) N-best softmax smoothing for minimum bayes risk training of attention based sequence-to-sequence models
US10713019B2 (en) Developer and runtime environments supporting multi-input modalities
CN112465144A (en) Multi-modal demonstration intention generation method and device based on limited knowledge
WO2022252890A1 (en) Interaction object driving and phoneme processing methods and apparatus, device and storage medium
US11443161B2 (en) Robot gesture generation
WO2019225028A1 (en) Translation device, system, method, program, and learning method
WO2023226767A1 (en) Model training method and apparatus, and speech meaning understanding method and apparatus
JP7029351B2 (en) How to generate OOS text and the device that does it
JP2021039727A (en) Text processing method, device, electronic apparatus, and computer-readable storage medium
CN110991155B (en) Text correction method, device and medium
JP6633556B2 (en) Acoustic model learning device, speech recognition device, acoustic model learning method, speech recognition method, and program

Legal Events

Date Code Title Description
121 Ep: the epo has been informed by wipo that ep was designated in this application

Ref document number: 16923912

Country of ref document: EP

Kind code of ref document: A1

NENP Non-entry into the national phase

Ref country code: DE

WWE Wipo information: entry into national phase

Ref document number: 2016923912

Country of ref document: EP