WO2024237300A1 - 制御システム - Google Patents

制御システム Download PDF

Info

Publication number
WO2024237300A1
WO2024237300A1 PCT/JP2024/018061 JP2024018061W WO2024237300A1 WO 2024237300 A1 WO2024237300 A1 WO 2024237300A1 JP 2024018061 W JP2024018061 W JP 2024018061W WO 2024237300 A1 WO2024237300 A1 WO 2024237300A1
Authority
WO
WIPO (PCT)
Prior art keywords
unit
weight
control
determination unit
operation target
Prior art date
Legal status (The legal status is an assumption and is not a legal conclusion. Google has not performed a legal analysis and makes no representation as to the accuracy of the status listed.)
Ceased
Application number
PCT/JP2024/018061
Other languages
English (en)
French (fr)
Japanese (ja)
Inventor
雅之 平田
壮留 藏富
翔平 吉田
Current Assignee (The listed assignees may be inaccurate. Google has not performed a legal analysis and makes no representation or warranty as to the accuracy of the list.)
University of Osaka NUC
Original Assignee
Osaka University NUC
Priority date (The priority date is an assumption and is not a legal conclusion. Google has not performed a legal analysis and makes no representation as to the accuracy of the date listed.)
Filing date
Publication date
Application filed by Osaka University NUC filed Critical Osaka University NUC
Priority to JP2025520626A priority Critical patent/JPWO2024237300A1/ja
Priority to CN202480032672.2A priority patent/CN121127816A/zh
Publication of WO2024237300A1 publication Critical patent/WO2024237300A1/ja
Anticipated expiration legal-status Critical
Ceased legal-status Critical Current

Links

Images

Classifications

    • G—PHYSICS
    • G06—COMPUTING OR CALCULATING; COUNTING
    • G06F—ELECTRIC DIGITAL DATA PROCESSING
    • G06F3/00—Input arrangements for transferring data to be processed into a form capable of being handled by the computer; Output arrangements for transferring data from processing unit to output unit, e.g. interface arrangements
    • G06F3/01—Input arrangements or combined input and output arrangements for interaction between user and computer
    • G—PHYSICS
    • G06—COMPUTING OR CALCULATING; COUNTING
    • G06F—ELECTRIC DIGITAL DATA PROCESSING
    • G06F3/00—Input arrangements for transferring data to be processed into a form capable of being handled by the computer; Output arrangements for transferring data from processing unit to output unit, e.g. interface arrangements
    • G06F3/01—Input arrangements or combined input and output arrangements for interaction between user and computer
    • G06F3/048—Interaction techniques based on graphical user interfaces [GUI]
    • B—PERFORMING OPERATIONS; TRANSPORTING
    • B25—HAND TOOLS; PORTABLE POWER-DRIVEN TOOLS; MANIPULATORS
    • B25J—MANIPULATORS; CHAMBERS PROVIDED WITH MANIPULATION DEVICES
    • B25J13/00—Controls for manipulators

Definitions

  • the present invention relates to a control system.
  • Patent Document 1 discloses a machine system that reflects, in addition to intended operation based on the user's intention, autonomous operation based on the machine's own judgment of the situation when determining the machine's operation. The machine system of Patent Document 1 can reduce the hassle and burden on the user when operating a machine using a BMI.
  • the mechanical system of Patent Document 1 uses the BMI to determine a user-intended action, which is an action intended by the user.
  • the mechanical system of Patent Document 1 also determines the autonomous action of the robot based on the robot's motion information and information about the robot's surroundings acquired via sensors (e.g., obstacle detection information). Then, when the user-intended action and the autonomous action do not match, the mechanical system determines the action that the robot should perform according to predetermined rule information.
  • the mechanical system of Patent Document 1 makes the weighting coefficient for the autonomous action larger than the weighting coefficient for the user-intended action, calculates a weighted linear sum of the user-intended action and the autonomous action, and determines the action to be performed by the robot.
  • the inconvenience and burden on the user is reduced when operating a machine using BMI.
  • the present invention was made in consideration of the above problems, and its purpose is to provide a control system that is less likely to cause the controlled object to behave in a manner different from the user's intention.
  • a control system controls an object to be operated.
  • the control system includes a detection unit, a first determination unit, a second determination unit, and a weight determination unit.
  • the detection unit detects a detection object related to the autonomous control of the object to be operated.
  • the first determination unit determines the next action of the object to be operated by voluntary control based on the brain waves of an operator who operates the object to be operated.
  • the second determination unit determines the next action of the object to be operated by autonomous control based on the detection result by the detection unit.
  • the weight determination unit determines a first weight for the next action of the object to be operated by voluntary control and a second weight for the next action of the object to be operated by autonomous control based on the detection result by the detection unit.
  • the weight determination unit changes the first weight and the second weight according to the detection result by the detection unit.
  • control system further includes an intention detection unit.
  • the intention detection unit detects the intention of the operator based on the brain waves of the operator.
  • the weight determination unit determines the first weight and the second weight based on the detection result by the detection unit and the intention of the operator.
  • control system further includes an instruction unit.
  • the instruction unit instructs the operation target to perform a next action of the operation target based on the next action determined by the first determination unit, the next action determined by the second determination unit, the first weight, and the second weight.
  • control system further includes a storage unit.
  • the storage unit stores change conditions for the first weight and the second weight.
  • the change conditions are adjusted based on an evaluation function that includes, as parameters, control performance and operability acquired during training of the control system.
  • the control performance indicates a value that reflects the amount of deviation between the behavior intended by the operator who operates the operation object during training and the actual behavior of the operation object.
  • the operability indicates a value that reflects the ease of operation of the operation object evaluated by the operator.
  • the instruction unit instructs the operation target to take a next action based on the next action determined by the first determination unit at an early stage of the operation target's action.
  • the instruction unit instructs the operation target to take a next action based on the next action determined by the second determination unit at the final stage of the operation target's action.
  • the autonomous control provides more precise control of the target object than the voluntary control.
  • the detection unit detects an object.
  • the weight determination unit dynamically changes the first weight and the second weight according to the distance between the operation target and the object.
  • control system further includes an instruction unit.
  • the instruction unit instructs the operation target to take a next action of the operation target based on the next action determined by the first determination unit and the next action determined by the second determination unit.
  • the second determination unit determines the next action of the operation target by autonomous control based on the detection result by the detection unit, the first weight, and the second weight.
  • control system further includes a storage unit.
  • the storage unit stores change conditions for the first weight and the second weight.
  • the change conditions are adjusted based on an evaluation function that includes, as parameters, control performance and operability acquired during training of the control system.
  • the control performance indicates a value that reflects the amount of deviation between the behavior predicted by the operator who will operate the operation object during training and the behavior of the operation object determined by the second determination unit.
  • the operability indicates a value that reflects the ease of operation of the operation object evaluated by the operator.
  • the behavior of the operation target is an utterance by the operation target.
  • the detection unit detects an utterance by a speech partner to which the operation target speaks.
  • the second determination unit determines at least one candidate for the content of the next utterance of the operation target based on the detection result by the detection unit, the first weight, and the second weight.
  • the candidate for the content of the next utterance determined by the second determination unit can incorporate content of utterance by voluntary control.
  • the first determination unit determines the content of the next utterance based on the brain waves.
  • the instruction unit instructs the operation target on the content of the utterance by the operation target.
  • control system makes it less likely that the controlled object will behave differently than intended by the user.
  • FIG. 1 illustrates a control system according to an embodiment of the present invention.
  • 1 is a block diagram showing a configuration of a robot control system which is an example of a control system according to an embodiment of the present invention.
  • FIG. 1 is a diagram showing a portion of a robot arm included in a robot control system.
  • FIG. 2 is a diagram showing the configuration of a drive mechanism of a robot arm included in the robot control system.
  • 2 is a block diagram showing a configuration of a drive unit included in the robot control system and a configuration of a drive mechanism of a robot arm.
  • FIG. 1A is a plan view showing a gripping unit of a robot arm included in a robot control system, and FIG.
  • FIG. 1B is a block diagram showing the configuration of a control unit and a robot arm included in the robot control system.
  • FIG. 2 is a block diagram showing a configuration of an intention detection unit included in the robot control system.
  • FIG. 2 is a block diagram showing hardware of a control unit included in the robot control system.
  • 1A is a side view showing a robot arm included in a robot control system
  • FIG. 1B is a plan view showing a robot arm included in the robot control system.
  • 13 is a flowchart illustrating an example of a process executed by the robot control system.
  • 1 is a block diagram showing a configuration of a communication system which is an example of a control system according to an embodiment of the present invention.
  • FIG. 2 shows a two-dimensional avatar controlled by a communication system.
  • FIG. 13 is a diagram showing a voluntary movement determination unit included in the communication system.
  • FIG. 1 is a diagram illustrating an autonomous operation decision unit included in the communication system.
  • FIG. 1 is a diagram showing an avatar system included in a communication system.
  • FIG. 2 is a block diagram showing a configuration of an intention detection unit included in the communication system.
  • 2 is a block diagram showing hardware of a control unit included in the communication system.
  • FIG. 11 is a diagram showing computer programs and data stored in a second memory unit of a control unit included in the communication system.
  • FIG. 13A to 13C are diagrams showing screens displayed on a display unit when a two-dimensional avatar is made to speak using a fixed phrase.
  • FIGS. 13A to 13C are diagrams showing screens displayed on a display unit when a candidate is selected from among candidates for the next speech content of a two-dimensional avatar.
  • 5A and 5B are diagrams showing a screen displayed on a display unit when a user inputs an arbitrary character.
  • 5A and 5B are diagrams showing a screen displayed on a display unit when a user inputs an arbitrary character.
  • FIG. 13 is a diagram showing a screen displayed on the display unit when deciding whether or not to confirm the next speech content of the two-dimensional avatar.
  • 4 is a flowchart showing a process executed by an intention detection unit and a control unit included in the communication system.
  • FIG. 1 is a diagram showing a control system 100 according to this embodiment.
  • the control system 100 controls the operation of the operation target 2. More specifically, the control system 100 has a function of autonomously controlling the operation target 2.
  • the control system 100 further has a function of voluntarily controlling the operation target 2 based on the brain waves of the user P.
  • the control system 100 may include a BMI (Brain Machine Interface).
  • the user P is an operator who operates the operation target 2.
  • the control system 100 according to this embodiment includes the operation target 2, a detection unit 3, a control unit 4, an intention detection unit 5, and an brain wave detection device 6.
  • autonomous control may be referred to as "autonomous control”.
  • voluntarily control may be referred to as "voluntary control”.
  • the operation target 2 is not particularly limited as long as it is capable of voluntary and autonomous control.
  • the operation target 2 may be a robot arm 2A described later with reference to Figures 2 to 10, or a two-dimensional avatar 2B described later with reference to Figures 11 to 24.
  • the operation target 2 may be a humanoid robot, an avatar robot, an electric wheelchair, a running robot, a drone, or a mouse (user interface device).
  • the brainwave detection device 6 detects brainwaves from the brain B of the user P. For example, the brainwave detection device 6 detects brainwaves every 0.1 or 0.2 seconds. In this embodiment, the brainwave detection device 6 is placed inside the skull of the user P and detects intracranial electroencephalography (iEEG). Specifically, the brainwave detection device 6 has electrodes 61 and a wireless communication unit 62.
  • iEEG intracranial electroencephalography
  • the electrode 61 may be, for example, a brain surface electrode type. In this case, the electrode 61 has a grid-like dish electrode. The grid-like dish electrode is placed on the surface of the brain B to detect cortical electroencephalograms. Alternatively, the electrode 61 may be an insertion needle electrode type. In this case, the electrode 61 has an array of tiny needle electrodes. The needle electrodes are inserted into the brain B.
  • the wireless communication unit 62 performs wireless communication with the intention detection unit 5.
  • the wireless communication unit 62 transmits a wireless signal to the intention detection unit 5.
  • the wireless signal indicates the brain waves detected by the electrodes 61.
  • an electrical signal indicating the brain waves is output from the electrodes 61 to the wireless communication unit 62.
  • the wireless communication unit 62 converts the electrical signal into a wireless signal and transmits it to the intention detection unit 5.
  • the wireless communication unit 62 and the intention detection unit 5 may perform wireless communication in accordance with a short-range wireless communication standard such as Bluetooth (registered trademark).
  • the brain wave detection device 6 may further include a signal amplifier.
  • the signal amplifier amplifies the electrical signal output from the electrode 61 and transmits it to the wireless communication unit 62. Specifically, the signal amplifier amplifies the weak current output from the electrode 61 and inputs it to the wireless communication unit 62.
  • the detection unit 3 detects a detection target related to the autonomous control of the operation target 2.
  • the detection unit 3 may have an imaging device 31A as described later with reference to Figures 2 to 10.
  • the detection unit 3 may have an imaging device 31B and a sound collector 32B as described later with reference to Figures 11 to 24.
  • the intention detection unit 5 detects the intention of the user P based on the brain waves of the user P.
  • the control unit 4 controls the operation of the operation target 2 based on the result of detection by the detection unit 3 and the brain waves of the user P.
  • the control unit 4 controls the operation of the operation target 2 based on the result of detection by the detection unit 3 and the intention of the user P detected by the intention detection unit 5.
  • the control unit 4 includes an environment recognition unit 41 and a harmony control unit 42.
  • the environment recognition unit 41 recognizes the environment around the operation target 2 based on the detection result by the detection unit 3. For example, the environment recognition unit 41 may determine the type of object ST or the center of gravity position of the object ST as the environment around the operation target 2, as described later with reference to Figures 2 to 10. Alternatively, the environment recognition unit 41 may determine the content (meaning) of the speech of the speaking partner or the emotions of the speaking partner as the environment around the operation target 2, as described later with reference to Figures 11 to 24.
  • the harmonic control unit 42 controls the operation target 2 by harmonizing voluntary control and autonomous control.
  • the harmonic control unit 42 determines the next action of the operation target 2 by voluntary control based on the brain waves of the user P.
  • the harmonic control unit 42 also determines the next action of the operation target 2 by autonomous control based on the detection result by the detection unit 3. Then, the harmonic control unit 42 harmonizes the next action of the operation target 2 by voluntary control and the next action of the operation target 2 by autonomous control to determine the next action of the operation target 2.
  • the harmony control unit 42 determines the next action of the operation target 2 by voluntary control based on the intention of the user P detected by the intention detection unit 5.
  • the harmony control unit 42 also determines the next action of the operation target 2 by autonomous control based on the environment around the operation target 2 recognized by the environment recognition unit 41. More specifically, the harmony control unit 42 includes a voluntary action determination unit 43, an autonomous action determination unit 44, a weight determination unit 45, and an instruction unit 46.
  • the voluntary movement determination unit 43 determines the next action of the operation target 2 under voluntary control based on the brain waves of the user P. In this embodiment, the voluntary movement determination unit 43 determines the next action of the operation target 2 under voluntary control based on the intention of the user P detected by the intention detection unit 5. Hereinafter, the next action of the operation target 2 under voluntary control may be referred to as a "voluntary movement.”
  • the voluntary movement determination unit 43 is an example of a "first determination unit.”
  • the autonomous operation decision unit 44 decides the next action of the operation target 2 by autonomous control based on the detection result by the detection unit 3. In this embodiment, the autonomous operation decision unit 44 decides the next action of the operation target 2 by autonomous control based on the environment around the operation target 2 recognized by the environment recognition unit 41. Hereinafter, the next action of the operation target 2 by autonomous control may be referred to as "autonomous operation”.
  • the autonomous operation decision unit 44 is an example of a "second decision unit”.
  • the weight determination unit 45 determines the weight for the voluntary movement and the weight for the autonomous movement based on the detection result by the detection unit 3. In this embodiment, the weight determination unit 45 determines the weight for the voluntary movement and the weight for the autonomous movement based on the environment around the operation target 2 recognized by the environment recognition unit 41.
  • the weighting for voluntary movement may be referred to as the "first weighting.”
  • the weighting for autonomous movement may be referred to as the "second weighting.”
  • the weighting of voluntary control relative to autonomous control increases.
  • the weighting of voluntary movement relative to autonomous movement increases.
  • the weighting of autonomous control relative to voluntary control increases.
  • the weighting of autonomous movement relative to voluntary movement increases.
  • the weight determination unit 45 changes the first weight and the second weight according to the detection result by the detection unit 3.
  • the weight determination unit 45 changes the first weight and the second weight based on the environment around the operation target 2 recognized by the environment recognition unit 41. Therefore, the values of the first weight and the second weight change according to changes in the environment around the operation target 2.
  • the instruction unit 46 instructs the operation target 2 to perform the next action.
  • the instruction unit 46 outputs a command (signal) to cause the operation target 2 to execute the next action.
  • the action that the instruction unit 46 instructs the operation target 2 to perform indicates an action in which the autonomous action and the voluntary action are harmonized by the first weighting and the second weighting.
  • the first weight and the second weight can be changed in response to changes in the environment around the operation target 2.
  • the proportion of voluntary movement and the proportion of autonomous movement can be changed in response to changes in the environment around the operation target 2.
  • the movement of the operation target 2 can be controlled by harmonizing voluntary control and autonomous control.
  • the instruction unit 46 does not instruct the operation target 2 to perform an autonomous action, but rather instructs the operation target 2 to perform an action (behavior) that harmonizes the voluntary action and the autonomous action. Therefore, even when the voluntary action and the autonomous action are different, the voluntary action can be reflected in the action instructed to the operation target 2. Therefore, the action of the operation target 2 is less likely to be an action different from the action intended by the user P.
  • the instruction unit 46 when the voluntary action indicates a lateral movement of the wrist 25 of the robot arm 2A and the autonomous action indicates a forward movement of the wrist 25 of the robot arm 2A, the instruction unit 46 does not instruct the robot arm 2A to move the wrist 25 forward, but rather instructs the robot arm 2A to perform an action that harmonizes the voluntary action and the autonomous action by the first weight and the second weight. For example, the instruction unit 46 instructs the robot arm 2A to move the wrist 25 diagonally forward. Therefore, the voluntary movement (lateral movement of the wrist 25) is reflected in the next action of the robot arm 2A.
  • the weight determination unit 45 determines the first weight and the second weight based on the detection result by the detection unit 3 and the brain waves of the user P. More specifically, the weight determination unit 45 determines the first weight and the second weight based on the environment around the operation target 2 recognized by the environment recognition unit 41 and the intention of the user P detected by the intention detection unit 5.
  • the weight determination unit 45 changes the first weight and the second weight according to the environment around the operation target 2 recognized by the environment recognition unit 41 and the intention of the user P detected by the intention detection unit 5. Therefore, the values of the first weight and the second weight change according to changes in the environment around the operation target 2. Furthermore, the values of the first weight and the second weight change according to changes in the intention of the user P. Therefore, the intention of the user P can be better reflected in the values of the first weight and the second weight. As a result, it becomes even less likely that the operation target 2 will perform an action different from the action intended by the user P.
  • Robot control system 100A A robot control system 100A, which is an example of the control system 100 of this embodiment, will be described below with reference to Figures 1 to 10. However, matters different from those described with reference to Figure 1 will be described, and descriptions of matters that are the same as those described with reference to Figure 1 will be omitted as appropriate.
  • FIG. 2 is a block diagram showing the configuration of a robot control system 100A, which is an example of the control system 100 of this embodiment.
  • FIG. 3 is a diagram showing a portion of a robot arm 2A included in the robot control system 100A.
  • the robot control system 100A is an example of a "control system 100.”
  • the operation target 2 of the robot control system 100A is the robot arm 2A.
  • the robot control system 100A includes a robot arm 2A, a detection unit 3A, a control unit 4A, an intention detection unit 5A, and an brain wave detection device 6.
  • the robot control system 100A has a function of autonomously controlling the robot arm 2A.
  • the robot control system 100A further has a function of voluntarily controlling the robot arm 2A based on the brain waves of a user P.
  • the user P is an operator who operates the robot arm 2A.
  • autonomous control can control the operation target 2 more precisely than voluntary control.
  • precise control of the operation target 2 includes controlling the operation target 2 with high accuracy and making the operation target 2 perform complex operations.
  • Complex operations include linking a plurality of operations.
  • the operation of making the robot arm 2A grasp the object ST requires precise control. Therefore, the operation of making the robot arm 2A grasp the object ST is an operation that can be performed by autonomous control, and is difficult to execute by voluntary control.
  • the operation of making the grasping portion 26 grasp the object ST is a complex operation, and is difficult to control by voluntary control (brain waves).
  • the detection unit 3A detects a detection target related to the autonomous control of the robot arm 2A.
  • the detection unit 3A has an imaging device 31A.
  • the imaging device 31A captures an image of the detection target and transmits imaging data indicating the captured image to the control unit 4A.
  • the control unit 4A determines the next action of the robot arm 2A under autonomous control based on the image captured by the imaging device 31A.
  • the image may be, for example, a two-dimensional image.
  • the imaging device 31A and the control unit 4A may communicate with each other.
  • the imaging device 31A and the control unit 4A may communicate with each other via a LAN (Local Area Network) or a wireless LAN.
  • the imaging device 31A and the control unit 4A may communicate with each other via a public line network such as the Internet.
  • the imaging device 31A captures an image of the surroundings of the robot arm 2A to detect the detection target.
  • the imaging device 31A may capture a moving image of the surroundings of the robot arm 2A, or may capture a still image of the surroundings of the robot arm 2A.
  • the imaging device 31A captures an image in front of the robot arm 2A (see Figure 3).
  • the imaging device 31A is controlled by the control unit 4A.
  • Detection targets of the imaging device 31A include, for example, the object ST and the robot arm 2A (see FIG. 3). That is, the imaging device 31A may detect the object ST and the robot arm 2A.
  • the control unit 4A controls the operation of the robot arm 2A to cause the robot arm 2A to grasp the object ST.
  • the control unit 4A acquires information indicating the relationship between the current state of the robot arm 2A and the position of the object ST from the image (imaging data) captured by the imaging device 31A.
  • the control unit 4A determines the next action of the robot arm 2A under autonomous control based on the information acquired from the imaging data.
  • the control unit 4A may identify the position of the wrist 25 (see Figure 3) of the robot arm 2A and the position of the object ST based on the imaging data, and calculate the distance between the wrist 25 (see Figure 3) of the robot arm 2A and the object ST.
  • the control unit 4A determines the next action (autonomous operation) of the robot arm 2A under autonomous control based on the distance between the wrist 25 of the robot arm 2A and the object ST.
  • the imaging device 31A may capture a moving image that indicates distance information to the subject.
  • the imaging device 31A may capture a still image that indicates distance information to the subject.
  • the imaging device 31A may be a stereo camera.
  • the imaging device 31A has two cameras arranged at a distance from each other, and calculates the distance from the imaging device 31A to the subject based on the principle of triangulation.
  • the imaging device 31A then generates an image showing the distance to the subject and transmits it to the control unit 4A.
  • the imaging device 31A may be a TOF (Time Of Flight) camera.
  • the imaging device 31A has a light source that irradiates the subject with infrared light, and a camera, and captures the infrared light reflected from the subject with the camera.
  • the imaging device 31A calculates the distance from the imaging device 31A to the subject based on the time it takes for the infrared light emitted from the light source to return to the camera.
  • the imaging device 31A then generates an image showing the distance to the subject and transmits it to the control unit 4A.
  • the image showing the distance to the subject may, for example, be colored to represent the distance.
  • the imaging device 31A may generate an image in which the closer the distance from the imaging device 31A, the bluer the subject, and the farther the distance, the redder the subject.
  • the intention detection unit 5A detects the intention of the user P based on the brain waves of the user P. More specifically, the intention detection unit 5A detects the next action of the robot arm 2A intended by the user P based on the brain waves detected by the brain wave detection device 6. Specifically, the intention detection unit 5A acquires the brain waves of the user P from a wireless signal transmitted from the wireless communication unit 62 of the brain wave detection device 6. The intention detection unit 5A then analyzes the acquired brain waves to detect the next action of the robot arm 2A intended by the user P.
  • the next action of the robot arm 2A intended by the user P may be referred to as the "intended action”.
  • the control unit 4A controls the robot arm 2A based on the output of the imaging device 31A and the output of the intention detection unit 5A.
  • the output of the imaging device 31A indicates the detection result by the imaging device 31A. That is, the output of the imaging device 31A indicates the image (imaging data) captured by the imaging device 31A.
  • the output of the intention detection unit 5A indicates the intended movement.
  • the control unit 4A and the intention detection unit 5A may communicate with each other.
  • the control unit 4A and the intention detection unit 5A may communicate with each other via a LAN or wireless LAN.
  • the control unit 4A and the intention detection unit 5A may communicate with each other via a public line network such as the Internet.
  • the robot arm 2A has a drive mechanism 27 and a drive unit 28.
  • the drive unit 28 drives the drive mechanism 27.
  • the drive mechanism 27 is driven to operate the robot arm 2A.
  • the control unit 4A controls the drive unit 28 to operate the robot arm 2A.
  • the control unit 4A includes an environment recognition unit 41A and a harmony control unit 42A.
  • the harmony control unit 42A includes a voluntary movement determination unit 43A, an autonomous movement determination unit 44A, a weight determination unit 45A, and an instruction unit 46A.
  • the voluntary movement determination unit 43A determines the next action (voluntary movement) of the robot arm 2A under voluntary control based on the brain waves of the user P. In more detail, the voluntary movement determination unit 43A determines the next action (voluntary movement) of the robot arm 2A under voluntary control based on the intended movement detected by the intention detection unit 5A.
  • the intention detection unit 5A estimates a velocity vector Vpp of the wrist 25 (see FIG. 3) of the robot arm 2A as the intended movement.
  • the velocity vector Vpp estimated by the intention detection unit 5A may be referred to as the "intended velocity Vpp".
  • the voluntary movement determination unit 43A includes a first velocity determination unit 431.
  • the first velocity determination unit 431 determines the velocity vector Vp of the wrist 25 (see FIG. 3) of the robot arm 2A based on the intended velocity Vpp.
  • the velocity vector Vp determined by the voluntary movement determination unit 43A (first velocity determination unit 431) may be referred to as the "velocity Vp by voluntary control”.
  • the first velocity determination unit 431 may correct the intended velocity Vpp to determine the velocity Vp by voluntary control.
  • the autonomous operation decision unit 44A decides the next action (autonomous operation) of the robot arm 2A by autonomous control based on the detection result by the imaging device 31A. In other words, the autonomous operation decision unit 44A decides the autonomous operation based on the image (imaging data) captured by the imaging device 31A. In more detail, the autonomous operation decision unit 44A decides the autonomous operation based on the environment around the robot arm 2A recognized by the environment recognition unit 41A.
  • the environment recognition unit 41A identifies the robot arm 2A and the object ST included in the image captured by the imaging device 31A, and identifies the current state of the robot arm 2A and the position of the object ST.
  • the autonomous operation determination unit 44A determines the autonomous operation based on the current state of the robot arm 2A and the position of the object ST.
  • the environment recognition unit 41A identifies the three-dimensional position of the wrist 25 (see Figure 3) of the robot arm 2A and the three-dimensional position of the object ST from an image captured by the imaging device 31A.
  • the environment recognition unit 41A then calculates the distance between the wrist 25 (see Figure 3) and the object ST.
  • the autonomous operation determination unit 44A determines the autonomous operation based on the distance between the wrist 25 (see Figure 3) and the object ST.
  • the autonomous operation includes an operation of moving the wrist 25 (see Figure 3) closer to the object ST.
  • the autonomous operation determination unit 44A includes a second speed determination unit 441.
  • the second speed determination unit 441 determines a speed vector Va of the wrist 25 (see FIG. 3) of the robot arm 2A based on the distance between the wrist 25 (see FIG. 3) of the robot arm 2A and the object ST.
  • the speed vector Va determined by the autonomous operation determination unit 44A may be referred to as the "speed Va by autonomous control.”
  • the autonomous operation further includes an operation of grasping an object ST with the grasping unit 26 (see Figure 3) of the robot arm 2A.
  • the intention detection unit 5A further detects an intention to grasp as an intended operation.
  • the intention to grasp indicates an intention to grasp the object ST.
  • the autonomous operation determination unit 44A determines an operation of making the grasping unit 26 (see Figure 3) grasp the object ST as the autonomous operation.
  • the environment recognition unit 41A identifies the three-dimensional shape of the object ST from the image captured by the imaging device 31A. Furthermore, the environment recognition unit 41A determines the position of the center of gravity of the object ST based on the three-dimensional shape of the object ST.
  • the autonomous operation determination unit 44A determines the autonomous operation of the gripping unit 26 (see FIG. 3) of the robot arm 2A based on the three-dimensional shape and center of gravity position of the object ST. In more detail, the autonomous operation determination unit 44A determines the position at which the gripping unit 26 grips the object ST based on the center of gravity position of the object ST. Then, the autonomous operation determination unit 44A determines the operation of the gripping unit 26 when gripping the object ST based on the three-dimensional shape of the object ST.
  • the environment recognition unit 41A identifies the type of object ST based on the three-dimensional shape of the object ST.
  • the autonomous operation determination unit 44A adjusts the pressure with which the gripping unit 26 grips the object ST based on the type of object ST.
  • the weight determination unit 45A determines a weight for voluntary movement (first weight) and a weight for autonomous movement (second weight) based on the detection result by the imaging device 31A. That is, the weight determination unit 45A determines a weight for voluntary movement (first weight) and a weight for autonomous movement (second weight) based on the image captured by the imaging device 31A. In detail, the weight determination unit 45A determines the first weight and the second weight based on the distance between the wrist 25 (see FIG. 3) of the robot arm 2A and the object ST. In the robot control system 100A described with reference to FIG. 2 to FIG. 10, the first weight indicates a weight for the speed Vp by voluntary control. Also, the second weight indicates a weight for the speed Va by autonomous control. As already described, the distance between the wrist 25 (see FIG. 3) of the robot arm 2A and the object ST is calculated by the environment recognition unit 41A.
  • the weight determination unit 45A changes the values of the first weight and the second weight according to the detection result by the imaging device 31A. Specifically, the weight determination unit 45A dynamically changes the values of the first weight and the second weight according to the distance between the wrist portion 25 (see FIG. 3) of the robot arm 2A and the object ST.
  • the weight determination unit 45A determines the first weight and the second weight based on the distance between the wrist 25 (see Figure 3) of the robot arm 2A and the object ST, and the intended velocity Vpp.
  • the intended velocity Vpp indicates the velocity vector of the wrist 25 (see Figure 3) estimated based on the brain waves of the user P.
  • the weight determination unit 45A may determine the values of the first weight and the second weight according to the distance between the wrist 25 (see FIG. 3) of the robot arm 2A and the object ST, and then correct the values of the first weight and the second weight based on the intended velocity Vpp. For example, the weight determination unit 45A may correct the values of the first weight and the second weight by multiplying each of the values of the first weight and the second weight by a coefficient.
  • the coefficient may indicate a value that changes according to the value of the intended velocity Vpp, or may indicate a constant value.
  • the weight determination unit 45A may increase the value of the first weight and decrease the value of the second weight as the intended velocity Vpp increases.
  • the instruction unit 46A instructs the robot arm 2A to perform the next action. Specifically, the instruction unit 46A controls the drive unit 28 of the robot arm 2A to drive the drive mechanism 27. As a result, the robot arm 2A operates. More specifically, the instruction unit 46A controls the next action of the robot arm 2A based on the voluntary action determined by the voluntary action determination unit 43A, the autonomous action determined by the autonomous action determination unit 44A, the first weight, and the second weight. As a result, the voluntary control and autonomous control of the robot arm 2A are harmonized.
  • the instruction unit 46A changes the proportion of the voluntary movement using the first weighting, changes the proportion of the autonomous movement using the second weighting, and calculates the linear sum of the voluntary movement and the autonomous movement after the proportions have been changed. Then, the instruction unit 46A determines the next action to be performed by the robot arm 2A based on the calculated linear sum.
  • the instruction unit 46A includes a posture determination unit 461.
  • the instruction unit 46A applies a first weight to the velocity Vp under voluntary control to change the velocity Vp under voluntary control, and applies a second weight to the velocity Va under autonomous control to change the velocity Va under autonomous control.
  • the instruction unit 46A then calculates the linear sum of the velocity Vp and the velocity Va after being changed by the first and second weights.
  • the posture determination unit 461 determines the next posture of the robot arm 2A based on the calculated linear sum, and controls the drive unit 28 based on the determined posture. As a result, the drive mechanism 27 operates, and the posture of the robot arm 2A becomes the determined posture.
  • Figure 4 is a diagram showing the configuration of the drive mechanism 27 of the robot arm 2A included in the robot control system 100A.
  • Figure 5 is a block diagram showing the configuration of the drive unit 28 included in the robot control system 100A and the configuration of the drive mechanism 27 of the robot arm 2A.
  • the imaging device 31A may be supported by the robot arm 2A.
  • the imaging device 31A In the robot control system 100A described with reference to FIGS. 2 to 10, the imaging device 31A is fixed to the upper part of the robot arm 2A.
  • the line of sight direction VA of the imaging device 31A may be set diagonally downward.
  • the angle ⁇ between the line of sight direction VA of the imaging device 31A and the horizontal plane is, for example, 50°.
  • the robot arm 2A has a shoulder 21, an upper arm 22, an elbow 23, a forearm 24, a wrist 25, and a grip 26.
  • the drive mechanism 27 includes a first joint drive mechanism 27a to a seventh joint drive mechanism 27g.
  • the robot arm 2A has seven degrees of freedom.
  • the first joint drive mechanism 27a to the seventh joint drive mechanism 27g may each include an electric motor capable of forward and reverse rotation.
  • the electric motor capable of forward and reverse rotation may be a stepping motor.
  • the shoulder 21 has a first joint drive mechanism 27a, a second joint drive mechanism 27b, and a third joint drive mechanism 27c.
  • the elbow 23 has a fourth joint drive mechanism 27d.
  • the forearm 24 has a fifth joint drive mechanism 27e.
  • the wrist 25 has a sixth joint drive mechanism 27f and a seventh joint drive mechanism 27g.
  • Figures 3 and 4 show the robot arm 2A in a position in which the upper arm 22 extends vertically downward from the shoulder 21, the elbow 23 is bent at a right angle, and the forearm 24 and wrist 25 extend horizontally.
  • the first joint drive mechanism 27a can freely swing in a rotational direction around an axis extending in the Y-axis direction.
  • the Y-axis direction is included in the horizontal direction.
  • the first joint drive mechanism 27a is connected to the second joint drive mechanism 27b.
  • the drive force of the first joint drive mechanism 27a is transmitted to the second joint drive mechanism 27b. Therefore, the second joint drive mechanism 27b is swung by the first joint drive mechanism 27a around the axis of the first joint drive mechanism 27a.
  • the second joint drive mechanism 27b is disposed on the -Y side with respect to the first joint drive mechanism 27a.
  • the second joint drive mechanism 27b is freely oscillating in a rotational direction about an axis extending in a direction perpendicular to the axis of the first joint drive mechanism 27a.
  • the axis of the second joint drive mechanism 27b extends in the X-axis direction.
  • the axis of the second joint drive mechanism 27b extends in the horizontal direction.
  • the second joint drive mechanism 27b is connected to the third joint drive mechanism 27c.
  • the drive force of the second joint drive mechanism 27b is transmitted to the third joint drive mechanism 27c. Therefore, the third joint drive mechanism 27c is oscillated by the second joint drive mechanism 27b about the axis of the second joint drive mechanism 27b.
  • the third joint drive mechanism 27c is disposed on the -Z side relative to the second joint drive mechanism 27b.
  • the third joint drive mechanism 27c is free to swing in a rotational direction about an axis extending in a direction perpendicular to the axis of the second joint drive mechanism 27b.
  • the axis of the third joint drive mechanism 27c extends in the Z-axis direction.
  • the axis of the third joint drive mechanism 27c extends in the vertical direction.
  • the third joint drive mechanism 27c is connected to the fourth joint drive mechanism 27d.
  • the drive force of the third joint drive mechanism 27c is transmitted to the fourth joint drive mechanism 27d. Therefore, the fourth joint drive mechanism 27d is swung by the third joint drive mechanism 27c about the axis of the third joint drive mechanism 27c.
  • the fourth joint drive mechanism 27d is disposed on the -Z side relative to the third joint drive mechanism 27c.
  • the fourth joint drive mechanism 27d is free to swing in a rotational direction about an axis extending in a direction perpendicular to the axis of the third joint drive mechanism 27c.
  • the axis of the fourth joint drive mechanism 27d extends in the Y-axis direction. In other words, the axis of the fourth joint drive mechanism 27d extends in the horizontal direction.
  • the fourth joint drive mechanism 27d is connected to the fifth joint drive mechanism 27e.
  • the drive force of the fourth joint drive mechanism 27d is transmitted to the fifth joint drive mechanism 27e. Therefore, the fifth joint drive mechanism 27e is swung by the fourth joint drive mechanism 27d about the axis of the fourth joint drive mechanism 27d.
  • the fifth joint drive mechanism 27e is disposed on the +X side relative to the fourth joint drive mechanism 27d.
  • the fifth joint drive mechanism 27e is free to swing in a rotational direction about an axis extending in a direction perpendicular to the axis of the fourth joint drive mechanism 27d.
  • the axis of the fifth joint drive mechanism 27e extends in the X-axis direction.
  • the axis of the fifth joint drive mechanism 27e extends in the horizontal direction.
  • the fifth joint drive mechanism 27e is connected to the sixth joint drive mechanism 27f.
  • the drive force of the fifth joint drive mechanism 27e is transmitted to the sixth joint drive mechanism 27f. Therefore, the sixth joint drive mechanism 27f is swung by the fifth joint drive mechanism 27e about the axis of the fifth joint drive mechanism 27e.
  • the sixth joint drive mechanism 27f is disposed on the +X side relative to the fifth joint drive mechanism 27e.
  • the sixth joint drive mechanism 27f is freely oscillating in a rotational direction about an axis extending in the same direction as the axis of the fifth joint drive mechanism 27e.
  • the axis of the sixth joint drive mechanism 27f extends in the X-axis direction.
  • the axis of the sixth joint drive mechanism 27f extends in the horizontal direction.
  • the sixth joint drive mechanism 27f is connected to the seventh joint drive mechanism 27g.
  • the drive force of the sixth joint drive mechanism 27f is transmitted to the seventh joint drive mechanism 27g. Therefore, the seventh joint drive mechanism 27g is oscillated by the sixth joint drive mechanism 27f about the axis of the sixth joint drive mechanism 27f.
  • the seventh joint drive mechanism 27g is disposed on the +X side relative to the sixth joint drive mechanism 27f.
  • the seventh joint drive mechanism 27g is free to swing in a rotational direction about an axis extending in a direction perpendicular to the axis of the sixth joint drive mechanism 27f.
  • the axis of the seventh joint drive mechanism 27g extends in the Z-axis direction.
  • the axis of the seventh joint drive mechanism 27g extends in the vertical direction.
  • the seventh joint drive mechanism 27g is connected to the gripper 26.
  • the drive force of the seventh joint drive mechanism 27g is transmitted to the gripper 26. Therefore, the gripper 26 is swung by the seventh joint drive mechanism 27g about the axis of the seventh joint drive mechanism 27g.
  • the drive unit 28 includes a first drive unit 28a to a seventh drive unit 28g.
  • the first drive unit 28a drives the first joint drive mechanism 27a.
  • the second drive unit 28b to the seventh drive unit 28g drive the second joint drive mechanism 27b to the seventh joint drive mechanism 27g, respectively.
  • the first drive unit 28a to the seventh drive unit 28g include, for example, motor drivers.
  • the harmonic control unit 42A controls the first drive unit 28a to drive the first joint drive mechanism 27a. Similarly, the harmonic control unit 42A controls the second drive unit 28b to the seventh drive unit 28g to drive the second joint drive mechanism 27b to the seventh joint drive mechanism 27g, respectively.
  • the harmonic control unit 42A controls the swing angle ⁇ c1 (rotation angle) of the first joint drive mechanism 27a. More specifically, the harmonic control unit 42A controls the swing angle ⁇ c1 (rotation angle) of the second joint drive mechanism 27b around the axis of the first joint drive mechanism 27a.
  • the harmonic control unit 42A controls the swing angles ⁇ c2 to ⁇ c7 of the second joint drive mechanism 27b to the seventh joint drive mechanism 27g, respectively.
  • the harmonic control unit 42A controls the swing angle ⁇ c2 (rotation angle) of the third joint drive mechanism 27c centered on the axis of the second joint drive mechanism 27b.
  • the harmonic control unit 42A controls the swing angle ⁇ c3 (rotation angle) of the fourth joint drive mechanism 27d centered on the axis of the third joint drive mechanism 27c.
  • the harmonic control unit 42A controls the swing angle ⁇ c4 (rotation angle) of the fifth joint drive mechanism 27e centered on the axis of the fourth joint drive mechanism 27d.
  • the harmonic control unit 42A controls the swing angle ⁇ c5 (rotation angle) of the sixth joint drive mechanism 27f centered on the axis of the fifth joint drive mechanism 27e.
  • the harmonic control unit 42A controls the swing angle ⁇ c6 (rotation angle) of the seventh joint drive mechanism 27g around the axis of the sixth joint drive mechanism 27f.
  • the harmonic control unit 42A controls the swing angle ⁇ c7 (rotation angle) of the gripper 26 around the axis of the seventh joint drive mechanism 27g.
  • Fig. 6(a) is a plan view showing the gripper 26 of the robot arm 2A included in the robot control system 100A.
  • Fig. 6(b) is a block diagram showing the configuration of the control unit 4A and robot arm 2A included in the robot control system 100A.
  • the gripping portion 26 may have a first gripping member 26a, a second gripping member 26b, a first gear 26c, a second gear 26d, and a support member 26e.
  • the support member 26e supports the first gear 26c and the second gear 26d so that they can rotate freely. More specifically, the first gear 26c and the second gear 26d are supported by the support member 26e with their teeth meshing with each other. Therefore, the first gear 26c and the second gear 26d rotate in opposite directions.
  • the first gripping member 26a is supported by a support member 26e so as to be able to swing freely, and protrudes from the support member 26e.
  • the driving force of the first gear 26c is transmitted to the base end of the first gripping member 26a.
  • the second gripping member 26b is supported by a support member 26e so as to be able to swing freely, and protrudes from the support member 26e on the same side as the first gripping member 26a.
  • the driving force of the second gear 26d is transmitted to the base end of the second gripping member 26b.
  • the first gripping member 26a and the second gripping member 26b face each other.
  • the first gear 26c and the second gear 26d swing, the first gripping member 26a and the second gripping member 26b open and close.
  • the opening of the first gripping member 26a and the second gripping member 26b indicates that the first gripping member 26a and the second gripping member 26b rotate in directions away from each other around their respective base ends.
  • the closing of the first gripping member 26a and the second gripping member 26b indicates that the first gripping member 26a and the second gripping member 26b rotate in directions toward each other around their respective base ends.
  • the gripper 26 has a gripper drive mechanism 27h.
  • the drive unit 28 includes an eighth drive unit 28h.
  • the eighth drive unit 28h drives the gripper drive mechanism 27h.
  • the gripper drive mechanism 27h includes an electric motor capable of forward and reverse rotation.
  • the electric motor capable of forward and reverse rotation may be a stepping motor.
  • the eighth drive unit 28h includes, for example, a motor driver.
  • the support member 26e may be box-shaped. In this case, the gripper drive mechanism 27h may be housed inside the support member 26e.
  • the harmonic control unit 42A controls the eighth drive unit 28h to operate the gripping unit 26.
  • the harmonic control unit 42A controls the eighth drive unit 28h to close the first gripping member 26a and the second gripping member 26b.
  • the object ST sandwiched between the first gripping member 26a and the second gripping member 26b is gripped by the first gripping member 26a and the second gripping member 26b.
  • the harmonic control unit 42A controls the eighth drive unit 28h to open the first gripping member 26a and the second gripping member 26b.
  • the environment recognition unit 41A identifies the three-dimensional shape of the object ST
  • the autonomous operation determination unit 44A determines the operation of the gripping unit 26 when gripping the object ST based on the three-dimensional shape of the object ST. Therefore, the harmonic control unit 42A can control the swing angle (rotation angle) of the first gripping member 26a and the second gripping member 26b in accordance with the three-dimensional shape of the object ST. In other words, the harmonic control unit 42A can control the opening angle between the first gripping member 26a and the second gripping member 26b in accordance with the three-dimensional shape of the object ST. As a result, the object ST can be gripped with a more appropriate pressure.
  • the environment recognition unit 41A identifies the type of object ST
  • the autonomous operation determination unit 44A determines the operation of the gripping unit 26 when gripping the object ST based on the three-dimensional shape of the object ST and the type of object ST. Therefore, the harmonic control unit 42A can control the swing angle (rotation angle) of the first gripping member 26a and the second gripping member 26b in accordance with the three-dimensional shape of the object ST and the type of object ST. In other words, the harmonic control unit 42A can control the opening angle between the first gripping member 26a and the second gripping member 26b in accordance with the three-dimensional shape and type of the object ST. As a result, the object ST can be gripped with a more appropriate pressure.
  • FIG. 7 is a block diagram showing the configuration of the intention detection unit 5A included in the robot control system 100A.
  • the intention detection unit 5A has a first storage unit 51, a first processing unit 52, and a wireless communication unit 53.
  • the intention detection unit 5A may be, for example, a personal computer.
  • the wireless communication unit 53 receives a wireless signal indicating the brain waves of the user P from the brain wave detection device 6. Specifically, the wireless communication unit 53 performs wireless communication with the wireless communication unit 62 of the brain wave detection device 6 described with reference to FIG. 1. More specifically, the wireless communication unit 53 performs wireless communication in accordance with the same wireless communication standard as the wireless communication unit 62 of the brain wave detection device 6. The wireless communication unit 53 converts the wireless signal indicating the brain waves into an electrical signal indicating the brain waves and inputs it to the first processing unit 52.
  • the first memory unit 51 stores various computer programs and various data.
  • the first memory unit 51 has, for example, a semiconductor memory.
  • the semiconductor memory includes, for example, a RAM (Random Access Memory) and a ROM (Read Only Memory).
  • the semiconductor memory may include a flash memory, an EPROM (Erasable Programmable Read Only Memory), or an EEPROM (Electrically Erasable Programmable Read Only Memory), and may also include a VRAM (Video RAM).
  • the first storage unit 51 may include a hard disk drive (HDD) or a solid state drive (SSD), or may include removable media.
  • HDD hard disk drive
  • SSD solid state drive
  • the first processing unit 52 executes various processes such as numerical calculations and information processing by executing the computer programs stored in the first storage unit 51.
  • the first processing unit 52 may include, for example, a CPU (Central Processing Unit) or an MPU (Micro Processing Unit).
  • the first processing unit 52 may further include a GPU (Graphics Processing Unit) or an NPU (Neural Network Processing Unit).
  • the first processing unit 52 may include a quantum computer.
  • the first processing unit 52 obtains the intention (intended action) of the user P from the brain waves by multivariate analysis.
  • the multivariate analysis includes analysis using a machine learning model.
  • the first storage unit 51 stores the first machine learning model D1.
  • the first machine learning model D1 is a computer program that analyzes brain waves to estimate the intention of the user P.
  • the first machine learning model D1 is trained to analyze brain waves to estimate the intended action.
  • the first machine learning model D1 is trained to output a velocity vector (intended velocity Vpp) of the wrist 25 of the robot arm 2A through regression analysis.
  • the intended velocity Vpp (velocity vector) includes an X-axis component Vppx, a Y-axis component Vppy, and a Z-axis component Vppz.
  • the first machine learning model D1 is also trained to detect a grasping intent through regression analysis.
  • the first machine learning model D1 may perform analysis using support vector regression.
  • the first machine learning model D1 may perform preprocessing before the analysis using support vector regression.
  • the preprocessing includes a process of extracting features from the EEG.
  • the preprocessing may be partial least squares regression (PLS regression) or independent component analysis (ICA).
  • the intended velocity Vpp can be calculated by the following formula (1):
  • iEEG indicates intracranial electroencephalogram
  • W1 indicates the weight of pre-processing
  • W2 indicates the weight of support vector regression.
  • the first processing unit 52 converts the electrical signal indicative of the brain waves into data indicative of the brain waves and inputs it to the first machine learning model D1.
  • the first machine learning model D1 analyzes the brain waves to estimate the intention of the user P.
  • the first machine learning model D1 analyzes the brain waves to estimate the intended action.
  • the first processing unit 52 outputs data indicative of the intended action estimated by the first machine learning model D1 to the control unit 4A.
  • the first processing unit 52 (intention detection unit 5A) outputs a signal indicative of the intended velocity Vpp to the control unit 4A as data indicative of the intended action.
  • the first processing unit 52 outputs a signal indicative of a grasping intention to the control unit 4A as data indicative of the intended action.
  • the first processing unit 52 performs band-pass filtering on the electrical signal representing the brain waves, and inputs the brain wave data after band-pass filtering to the first machine learning model D1.
  • the band-pass filtering extracts signals in a frequency band related to movement from the electrical signal representing the brain waves.
  • the frequency band related to movement is, for example, a frequency band between 60 Hz and 200 Hz.
  • the machine learning algorithm for constructing the first machine learning model D1 is not limited to support vector regression.
  • the machine learning algorithm for constructing the first machine learning model D1 may be, for example, a decision tree, a nearest neighbor method, a naive Bayes classifier, or a neural network. Therefore, the first machine learning model D1 includes a decision tree, a nearest neighbor method, a naive Bayes classifier, a support vector machine, or a neural network. Backpropagation may be used for machine learning.
  • the neural network includes an input layer, one or more intermediate layers, and an output layer.
  • the neural network includes at least one of a deep neural network (DNN), a recurrent neural network (RNN), or a convolutional neural network (CNN), and performs deep learning.
  • the deep neural network includes an input layer, multiple intermediate layers, and an output layer.
  • FIG. 8 is a block diagram showing the hardware of the control unit 4A included in the robot control system 100A.
  • the control unit 4A has a second storage unit 4a, a second processing unit 4b, and an input unit 4c.
  • the control unit 4A may be, for example, a personal computer.
  • the second storage unit 4a stores various computer programs and various data.
  • the second storage unit 4a has, for example, a semiconductor memory.
  • the semiconductor memory includes, for example, a RAM and a ROM.
  • the semiconductor memory may include a flash memory, an EPROM, or an EEPROM, or may include a VRAM.
  • the second storage unit 4a may further include an HDD or SSD, and may include a removable medium.
  • the second storage unit 4a stores a robot operating system (ROS) D11.
  • the robot operating system D11 is middleware (computer program) for operating the robot.
  • the second processing unit 4b executes the robot operating system D11 to control the robot arm 2A.
  • the second processing unit 4b executes various processes such as numerical calculations and information processing by executing the computer programs stored in the second storage unit 4a.
  • the second processing unit 4b may include, for example, a CPU or an MPU.
  • the second processing unit 4b may further include a GPU or may further include an NPU.
  • the second processing unit 4b may include a quantum computer.
  • the second processing unit 4b executes the computer program stored in the second storage unit 4a to function as the environment recognition unit 41A, the voluntary movement determination unit 43A, the autonomous movement determination unit 44A, the weight determination unit 45A, and the instruction unit 46A described with reference to FIG. 2.
  • the second processing unit 4b determines a voluntary movement based on the intended movement. Specifically, when functioning as the voluntary movement determination unit 43A (first velocity determination unit 431), the second processing unit 4b determines the velocity vector Vp (velocity Vp due to voluntary control) of the wrist 25 of the robot arm 2A based on the intended velocity Vpp.
  • the velocity Vp due to voluntary control includes a component Vpx in the X-axis direction, a component Vpy in the Y-axis direction, and a component Vpz in the Z-axis direction.
  • the second processing unit 4b When functioning as the environment recognition unit 41A, the second processing unit 4b identifies the current position of the wrist 25 of the robot arm 2A and the position of the object ST based on the image captured by the imaging device 31A, and calculates the distance between the wrist 25 and the object ST.
  • the second processing unit 4b determines the autonomous operation based on the distance between the wrist 25 and the object ST.
  • the second processing unit 4b when the second processing unit 4b functions as the autonomous operation decision unit 44A (second speed decision unit 441), it calculates the autonomously controlled speed Va (speed vector Va of the wrist 25) based on the distance between the wrist 25 and the object ST.
  • the autonomously controlled speed Va includes a component Vax in the X-axis direction, a component Vay in the Y-axis direction, and a component Vaz in the Z-axis direction.
  • the second processing unit 4b identifies the three-dimensional shape of the object ST based on the image captured by the imaging device 31A. Then, the second processing unit 4b identifies the center of gravity position of the object ST and the type of the object ST based on the three-dimensional shape of the object ST.
  • the second storage unit 4a stores the second machine learning model D12.
  • the second machine learning model D12 is incorporated into the robot operating system D11.
  • the second machine learning model D12 is trained to analyze an image captured by the imaging device 31A, identify the current position of the wrist 25 of the robot arm 2A and the position of the object ST, and calculate the distance between the wrist 25 of the robot arm 2A and the object ST.
  • the second machine learning model D12 is trained to analyze an image captured by the imaging device 31A, and estimate the three-dimensional shape of the object ST, the center of gravity position of the object ST, and the type of the object ST.
  • the second processing unit 4b inputs data indicating the image captured by the imaging device 31A to the second machine learning model D12. As a result, information indicating the distance between the wrist 25 of the robot arm 2A and the object ST, the three-dimensional shape of the object ST, the center of gravity position of the object ST, and the type of the object ST is output from the second machine learning model D12.
  • the machine learning algorithm for constructing the second machine learning model D12 is not particularly limited, and may be, for example, a decision tree, a nearest neighbor method, a naive Bayes classifier, a support vector machine, or a neural network. Therefore, the second machine learning model D12 includes a decision tree, a nearest neighbor method, a naive Bayes classifier, a support vector machine, or a neural network. Backpropagation may be used for machine learning.
  • the second processing unit 4b determines the first weight and the second weight based on the detection result by the imaging device 31A.
  • the second processing unit 4b determines the first weight and the second weight based on the distance between the wrist unit 25 (see Figure 3) and the object ST and the intended velocity Vpp.
  • the second storage unit 4a further stores weight change conditions D13. Specifically, the weight change conditions D13 are incorporated into the robot operating system D11. The second processing unit 4b determines the values of the first weight and the second weight by referring to the weight change conditions D13.
  • the weight change condition D13 indicates a condition for changing the values of the first weight and the second weight. Specifically, the weight change condition D13 indicates a condition for changing the values of the first weight and the second weight according to the detection result by the imaging device 31A. In the robot control system 100A described with reference to Figures 2 to 10, the weight change condition D13 indicates a condition for changing the values of the first weight and the second weight according to the detection result by the imaging device 31A and the intended action.
  • the input unit 4c is a user interface device operated by the worker.
  • the input unit 4c typically has a keyboard and a mouse.
  • the input unit 4c may have a touch sensor.
  • the worker can adjust the weight change condition D13 by operating the input unit 4c. In other words, the worker can adjust the change conditions for each value of the first weight and the second weight by operating the input unit 4c.
  • the weight change condition D13 indicates a condition for changing each value of the first weight and the second weight depending on the distance between the wrist 25 of the robot arm 2A and the object ST and the intended speed Vpp.
  • the weight change condition D13 indicates the relationship between the distance between the wrist 25 and the object ST, the intended speed Vpp, and each value of the first weight and the second weight.
  • the weight change condition D13 indicates the relationship between the distance between the wrist 25 and the object ST, the intended velocity Vpp, and the weight Wp (first weight) for the voluntary movement.
  • the first weight (Wp) is the weight for the speed Vp due to voluntary control
  • the second weight (1-Wp) is the weight for the speed Va due to autonomous control.
  • the weight change condition D13 may prescribe the first weight (Wp) so that the robot arm 2A behaves by voluntary control in the initial stage of the behavior of the robot arm 2A.
  • the weight change condition D13 may prescribe the first weight (Wp) so that the robot arm 2A behaves by voluntary control when the distance between the object ST and the wrist 25 of the robot arm 2A is 1 m or more.
  • the weight change condition D13 may prescribe the value of the first weight (Wp) to be "1" or approximately "1" when the distance between the object ST and the wrist 25 of the robot arm 2A is 1 m or more.
  • the second processing unit 4b controls the robot arm 2A by the intended operation, so that the behavior of the robot arm 2A becomes the behavior intended by the user P.
  • the behavior of the robot arm 2A becomes the behavior intended by the user P.
  • the weight change condition D13 may also define the first weight (Wp) so that the robot arm 2A acts by autonomous control in the final stage of the behavior of the robot arm 2A.
  • the weight change condition D13 may define the first weight (Wp) so that the robot arm 2A acts by autonomous control when the distance between the object ST and the wrist 25 of the robot arm 2A is 30 cm or less.
  • the weight change condition D13 may define the value of the first weight (Wp) to be "0" or approximately "0" when the distance between the object ST and the wrist 25 of the robot arm 2A is 30 cm or less.
  • the relationship between the distance between the wrist 25 and the object ST and the weight Wp may be continuous or stepwise. That is, the value of the weight Wp may change continuously or stepwise in response to a change in the distance between the wrist 25 and the object ST.
  • the weight change condition D13 may be a condition for continuously reducing the value of the first weight (Wp) during the course of the action of the robot arm 2A, or a condition for reducing it stepwise.
  • the weight change condition D13 may be a condition for continuously or stepwise reducing the value of the first weight (Wp) as the distance between the object ST and the wrist 25 of the robot arm 2A becomes smaller, when the distance between the object ST and the wrist 25 of the robot arm 2A is smaller than 1 m and larger than 30 cm.
  • the weight change condition D13 may indicate a condition for assigning a coefficient to the first weight (Wp) determined based on the distance between the object ST and the wrist 25 of the robot arm 2A when the intended speed Vpp is equal to or greater than a predetermined speed.
  • the coefficient indicates a value greater than "1".
  • the coefficient may be a constant value or a value that changes according to the intended speed Vpp.
  • the coefficient may change continuously or in stages according to the intended speed Vpp.
  • the second processing unit 4b When functioning as the instruction unit 46A, the second processing unit 4b changes the proportion of the intended movement using the first weight, changes the proportion of the autonomous movement using the second weight, and calculates the linear sum of the intended movement and the autonomous movement after the proportion has been changed. Then, the second processing unit 4b determines the next action to be executed by the robot arm 2A based on the calculated linear sum.
  • the second processing unit 4b when the second processing unit 4b functions as the instruction unit 46A, it changes the velocity Vp under voluntary control using a first weight (Wp) and changes the velocity Va under autonomous control using a second weight (1-Wp).
  • the second processing unit 4b then calculates the linear sum of the changed velocity Vp and velocity Va to determine the velocity vector Vcmd of the wrist 25 of the robot arm 2A (see Figure 3).
  • the velocity vector Vcmd is a command value.
  • the velocity vector Vcmd may be referred to as the "command velocity Vcmd".
  • the command velocity Vcmd can be obtained based on the following equation (3):
  • the command velocity Vcmd includes an X-axis component Vcmd_x, a Y-axis component Vcmd_y, and a Z-axis component Vcmd_z.
  • Vcmd Vp ⁇ Wp+Va ⁇ (1-Wp)...(3)
  • the second processing unit 4b controls the robot arm 2A based on the command velocity Vcmd. Specifically, the second processing unit 4b executes the robot operating system D11 and controls the drive mechanism 27 (first joint drive mechanism 27a to seventh joint drive mechanism 27g) described with reference to Figures 3 to 5 so that the wrist 25 moves at the command velocity Vcmd.
  • the second processing unit 4b integrates the X-axis component Vcmd_x, the Y-axis component Vcmd_y, and the Z-axis component Vcmd_z included in the command velocity Vcmd, respectively, to determine the next three-dimensional position (Xo, Yo, Zo) of the wrist 25 of the robot arm 2A, and controls the driving mechanism 27 described with reference to FIGS. 3 to 5 so that the wrist 25 of the robot arm 2A moves to the next three-dimensional position (Xo, Yo, Zo).
  • the next three-dimensional position (Xo, Yo, Zo) can be obtained by the following equations (4) to (6), respectively.
  • the three-dimensional coordinates may be set with the shoulder 21 of the robot arm 2A as the origin (0, 0, 0).
  • the next three-dimensional position may be referred to as the "target position”.
  • the robot operating system D11 incorporates a control formula that calculates the target angles ⁇ 1 to ⁇ 7 of the first joint driving mechanism 27a to the seventh joint driving mechanism 27g based on inverse kinematics.
  • the second processing unit 4b calculates the target angles ⁇ 1 to ⁇ 7 of the first joint driving mechanism 27a to the seventh joint driving mechanism 27g from the target positions (Xo, Yo, Zo).
  • the second processing unit 4b controls the first joint driving mechanism 27a to the seventh joint driving mechanism 27g so that the swing angles ⁇ c1 to ⁇ c7 of the first joint driving mechanism 27a to the seventh joint driving mechanism 27g become the target angles ⁇ 1 to ⁇ 7.
  • the robot operating system D11 may incorporate the following formulas (7) to (19) as control formulas. Specifically, when Yo is equal to or less than 0, the second processing unit 4b (robot operating system D11) calculates the target angles ⁇ 1 to ⁇ 7 from formulas (7) to (14). When Yo is greater than 0, the second processing unit 4b (robot operating system D11) calculates the target angles ⁇ 1 to ⁇ 7 from formulas (7) to (9) and formulas (15) to (19).
  • Figure 9(a) is a side view showing the robot arm 2A included in the robot control system 100A
  • Figure 9(b) is a plan view showing the robot arm 2A included in the robot control system 100A.
  • Fig. 10 is a flowchart showing an example of processing executed by the robot control system 100A.
  • Fig. 10 shows an example of processing for causing the gripping portion 26 of the robot arm 2A to grip an object ST.
  • the processing executed by the robot control system 100A may include steps S1 to S10.
  • the process shown in FIG. 10 starts, for example, in response to detection of the brain waves of user P by the brain wave detection device 6.
  • the intention detection unit 5A (first processing unit 52) estimates an intended action from the brain waves of user P (step S1).
  • the intention detection unit 5A determines whether the intended motion is an intention to grasp (step S2). If it is determined that the intended motion is not an intention to grasp (No in step S2), the intended motion indicates the intended speed Vpp. In this case, the intention detection unit 5A (first processing unit 52) transmits a signal indicating the intended speed Vpp to the control unit 4A. If it is determined that the intended motion is an intention to grasp (Yes in step S2), the intention detection unit 5A (first processing unit 52) transmits a signal indicating the intention to grasp to the control unit 4A.
  • the second processing unit 4b (voluntary movement determination unit 43A) acquires information indicating the intended velocity Vpp from the intention detection unit 5A (No in step S2), it determines a voluntary movement (step S3). In more detail, the second processing unit 4b (voluntary movement determination unit 43A) calculates a velocity vector Vp due to voluntary control from the intended velocity Vpp.
  • the second processing unit 4b acquires information indicating the intended speed Vpp from the intention detection unit 5A (No in step S2), it determines the first weight (Wp) and calculates the second weight (1-Wp) (step S4).
  • the second processing unit 4b calculates the distance between the object ST and the wrist 25 of the robot arm 2A based on the image captured by the imaging device 31A. Then, the second processing unit 4b (weight determination unit 45A) determines the first weight (Wp) and calculates the second weight (1-Wp) based on the distance between the object ST and the wrist 25 of the robot arm 2A and the intended velocity Vpp.
  • the second processing unit 4b determines the autonomous operation based on the distance between the object ST and the wrist 25 of the robot arm 2A (step S5).
  • the second processing unit 4b autonomous operation determination unit 44A
  • the autonomous control speed Va calculates the autonomous control speed Va.
  • the second processing unit 4b instruction unit 46A calculates the command speed Vcmd based on the voluntary control speed Vp, the autonomous control speed Va, the first weight (Wp), and the second weight (1-Wp) (step S6).
  • the second processing unit 4b After calculating the command velocity Vcmd, the second processing unit 4b (instruction unit 46A) integrates the command velocity Vcmd to calculate the target position (Xo, Yo, Zo) (step S7). Then, the second processing unit 4b (instruction unit 46A) calculates the target angles ⁇ 1 to ⁇ 7 of the first joint drive mechanism 27a to the seventh joint drive mechanism 27g based on the target positions (Xo, Yo, Zo) (step S8).
  • the second processing unit 4b (instruction unit 46A) calculates the target angles ⁇ 1 to ⁇ 7, it drives the first joint driving mechanism 27a to the seventh joint driving mechanism 27g so that the swing angles ⁇ c1 to ⁇ c7 of the first joint driving mechanism 27a to the seventh joint driving mechanism 27g become the target angles ⁇ 1 to ⁇ 7.
  • the wrist portion 25 of the robot arm 2A moves to the target position (Xo, Yo, Zo) (step S9).
  • the process returns to step S1.
  • the intention detection unit 5A (first processing unit 52) estimates the intended movement every 0.1 or 0.2 seconds.
  • the second processing unit 4b determines the value of the first weight (Wp) to be "1" or approximately "1".
  • the command velocity Vcmd becomes equal to or approximately equal to the velocity Vp by voluntary control. Therefore, the wrist 25 moves as intended by the user P.
  • the second processing unit 4b reduces the value of the first weight (Wp) as the distance between the object ST and the wrist 25 of the robot arm 2A becomes smaller.
  • Wp first weight
  • the second processing unit 4b determines the value of the first weight (Wp) to be "0" or approximately "0".
  • the command speed Vcmd becomes equal to or approximately equal to the speed Va by the autonomous control. Therefore, the wrist 25 moves by the autonomous control.
  • the second processing unit 4b determines that the intended action is an intention to grasp (Yes in step S2), it controls the gripping unit 26 based on the image captured by the imaging device 31A to cause the gripping unit 26 to grasp the object ST (step S10). As a result, the process shown in FIG. 10 ends.
  • the second processing unit 4b (environment recognition unit 41A) identifies the three-dimensional shape, center of gravity position, and type of the object ST based on the image captured by the imaging device 31A. Then, the second processing unit 4b (autonomous operation determination unit 44A) determines the autonomous operation based on the three-dimensional shape, center of gravity position, and type of the object ST of the object ST. The second processing unit 4b (instruction unit 46A) controls the gripping drive mechanism 27h based on the determined autonomous operation to operate the first gripping member 26a and the second gripping member 26b. In other words, the second processing unit 4b operates the gripping unit 26 by autonomous control. As a result, the object ST is gripped by the first gripping member 26a and the second gripping member 26b.
  • the robot arm 2A when the robot arm 2A is made to grasp the object ST, the gripping unit 26 is operated by autonomous control. Therefore, the robot arm 2A can grasp the object ST with greater precision than when the gripping unit 26 is operated by voluntary control.
  • the weight change condition D13 is adjusted based on an evaluation function.
  • the evaluation function includes the control performance and operability acquired during training of the robot control system 100A (control system 100) as parameters.
  • the control performance indicates a value reflecting the amount of deviation between the action intended by the operator operating the robot arm 2A (operation target 2) during training of the robot control system 100A (control system 100) and the actual action of the robot arm 2A (operation target 2).
  • the operability indicates a value reflecting the ease of operation of the robot arm 2A (operation target 2) evaluated by the operator.
  • the weight change condition D13 may be adjusted using the following equation (20) as an evaluation function.
  • Evaluation function a ⁇ f (operation feeling) ⁇ b ⁇ g (error) (20)
  • f (operation feel) is a function that includes the operation feel as a parameter.
  • g (error) is a function that includes the deviation (error) between the action intended by the operator operating the robot arm 2A (operation target 2) and the actual action of the robot arm 2A (operation target 2) as a parameter.
  • f (operation feel) may be a function whose value increases as the value of the operation feel increases.
  • g (error) may be a function whose value increases as the deviation (error) increases.
  • the coefficients "a” and "b” may be arbitrarily determined by the operator.
  • coefficients "a” and “b” may be determined based on the following formulas (21) and (22).
  • the training of the robot control system 100A indicates training of voluntary control.
  • the training of voluntary control is performed after training of the first machine learning model D1.
  • the worker causes the brainwave detection device 6 to detect the worker's brainwaves, causes the intention detection unit 5A (intention detection unit 5) to detect the worker's intention, and causes the control unit 4A (control unit 4) to operate the robot arm 2A (operation target 2) based on the worker's intention.
  • the control unit 4A controls the robot arm 2A (operation target 2) by voluntary control.
  • the control unit 4A (control unit 4) operates the robot arm 2A (operation target 2) based on the intended operation.
  • the worker determines the difference (error) between the motion of the robot arm 2A (operation target 2) intended by the worker and the actual motion of the robot arm 2A (operation target 2).
  • the worker then operates the input unit 4c to adjust the weight change condition D13 so that the difference (error) between the motion of the robot arm 2A (operation target 2) intended by the worker and the actual motion of the robot arm 2A (operation target 2) becomes smaller.
  • the worker adjusts the change condition of the weight Wp so as to reduce the error.
  • the worker when training the robot control system 100A (control system 100), the worker evaluates the ease of operation (operational feel) of the robot arm 2A (operation target 2) using a scale such as a 5-point scale. Alternatively, the worker may evaluate the ease of operation (operational feel) of the robot arm 2A (operation target 2) using a visual continuous scale (visual analog scale). The worker then operates the input unit 4c to adjust the weight change condition D13 so that the evaluation is higher. In other words, the worker adjusts the change condition of the weight Wp so that the evaluation is higher.
  • the operator refers to formula (20) and operates the input unit 4c to adjust the weight change condition D13 (the change condition of the weight Wp) so that the value of the evaluation function becomes as large as possible.
  • the operator adjusts the weight change condition D13 (the change condition of the weight Wp) so that the value of the evaluation function becomes a locally optimal solution.
  • the robot control system 100A which is an example of the control system 100 of this embodiment, has been described with reference to Figures 2 to 10.
  • voluntary control and autonomous control can be harmonised.
  • the movement of the robot arm 2A (operation target 2) is less likely to differ from the intention of the user P.
  • the robot arm 2A is made to grasp an object ST
  • the wrist portion 25 is from the object ST
  • the larger the value of the weight Wp so that voluntary control becomes dominant and the robot arm 2A moves as intended by the user P.
  • the closer the wrist portion 25 is to the object ST the smaller the value of the weight Wp becomes, and the proportion of voluntary control gradually decreases.
  • the proportion of voluntary control is not 0, so if the movement of the robot arm 2A (operation target 2) deviates from the intention of the user P, the user P can correct the movement of the robot arm 2A (operation target 2) by brain waves.
  • the intention of the user P can be reflected in the weight Wp. Therefore, the movement of the robot arm 2A (operation target 2) can be more reliably corrected by the brain waves of the user P.
  • the gripper 26 moves close to the object ST (for example, within 30 cm of the object ST)
  • autonomous control becomes dominant. As a result, the robot arm 2A can be controlled with higher precision to grip the object ST.
  • the change conditions of the weight Wp can be adjusted so that the evaluation function becomes as large as possible based on the control performance and the feeling of operation.
  • the first processing unit 52 acquires the intention of the user P from the electroencephalogram by multivariate analysis, but the first processing unit 52 may also acquire the intention of the user P by image recognition by converting the electroencephalogram signal into an image.
  • the first storage unit 51 may store a machine learning model for image recognition trained to estimate the intention of the user P from an image of an electroencephalogram, and the first processing unit 52 may acquire the intention of the user P by inputting the image of an electroencephalogram into the machine learning model for image recognition.
  • FIG. 11 is a block diagram showing the configuration of a communication system 100B, which is an example of the control system 100 of this embodiment.
  • FIG. 12 is a diagram showing a two-dimensional avatar 2B controlled by the communication system 100B.
  • the operation target 2 of the communication system 100B is the two-dimensional avatar 2B.
  • the communication system 100B controls the behavior of the two-dimensional avatar 2B (operation target 2).
  • the behavior of the two-dimensional avatar 2B (operation target 2) controlled by the communication system 100B includes speech by the two-dimensional avatar 2B and body language by the two-dimensional avatar 2B.
  • the communication system 100B includes a two-dimensional avatar 2B, a detection unit 3B, a control unit 4B, an intention detection unit 5B, and an brain wave detection device 6.
  • the communication system 100B has a function of controlling the speech content of the two-dimensional avatar 2B based on the brain waves of the user P. In other words, the communication system 100B performs voluntary control. Specifically, the communication system 100B may include a BMI.
  • the user P is the operator who operates the two-dimensional avatar 2B.
  • the communication system 100B has a function of autonomously generating candidates for the speech content of the two-dimensional avatar 2B. In other words, the communication system 100B performs autonomous control.
  • the communication system 100B which will be described with reference to Figures 11 to 24, further has a function of autonomously determining the body language of the two-dimensional avatar 2B.
  • Detection unit 3B detects a detection target related to the autonomous control of two-dimensional avatar 2B. Specifically, detection unit 3B has an imaging device 31B and a sound collector 32B.
  • the imaging device 31B captures an image of the face of the person with whom the two-dimensional avatar 2B is speaking.
  • the imaging device 31B may include, for example, a video camera or a webcam.
  • the detection targets related to the autonomous control of the operation target 2 include the facial expression of the person with whom the speech is being spoken.
  • the imaging device 31B transmits the captured video image (imaging data) to the control unit 4B. Specifically, the imaging device 31B transmits imaging data (video image) showing the face of the person being spoken to the control unit 4B.
  • the sound collector 32B collects sound and generates sound data.
  • the sound collector 32B has, for example, a microphone. Specifically, the sound collector 32B collects sound emitted by the speaking partner of the two-dimensional avatar 2B. In other words, the sound collector 32B detects speech by the speaking partner.
  • detection targets related to the autonomous control of the operation target 2 include speech by the speaking partner.
  • the sound collector 32B transmits the voice data to the control unit 4B.
  • the voice data represents the voice collected by the sound collector 32B. Specifically, the voice data represents the speech of the speaking partner.
  • the detection unit 3B (imaging device 31B and sound collector 32B) and the control unit 4B may communicate with each other.
  • the detection unit 3B (imaging device 31B and sound collector 32B) and the control unit 4B may communicate with each other via a LAN or wireless LAN.
  • the detection unit 3B (imaging device 31B and sound collector 32B) and the control unit 4B may communicate with each other via a public line network such as the Internet.
  • the control unit 4B controls the speech of the two-dimensional avatar 2B based on the output of the imaging device 31B, the output of the sound collector 32B, and the output of the intention detection unit 5B. Specifically, the control unit 4B determines the content of the next speech of the two-dimensional avatar 2B.
  • the next speech of the two-dimensional avatar 2B is an example of the "next action of the operation target 2.”
  • the control unit 4B autonomously generates a sentence ASE indicating a candidate for the next speech content of the two-dimensional avatar 2B (autonomous control). That is, the control unit 4B generates the sentence ASE by autonomous control.
  • the sentence ASE includes a gap. That is, the control unit 4B generates a sentence ASE with a gap by autonomous control.
  • the control unit 4B determines the remaining part of the next speech content of the two-dimensional avatar 2B (the speech content of the gap part of the sentence ASE) based on the input from the intention detection unit 5B. In other words, the control unit 4B determines the remaining part of the next speech content of the two-dimensional avatar 2B based on the brain waves of the user P. As a result, a sentence CSE indicating the next speech content of the two-dimensional avatar 2B is completed.
  • the control unit 4B causes the two-dimensional avatar 2B to speak based on the completed sentence CSE.
  • the control unit 4B determines one of the multiple candidates based on the input from the intention detection unit 5B. In other words, the control unit 4B determines one of the multiple candidates based on the brain waves of the user P.
  • the intention detection unit 5B communicates wirelessly with the brainwave detection device 6 and acquires the brainwaves of user P from the brainwave detection device 6.
  • the intention detection unit 5B then causes the control unit 4B to determine the remaining part of the next utterance content of the two-dimensional avatar 2B (the utterance content of the blank part of the sentence ASE) based on the brainwaves of user P.
  • part of the next utterance content of the two-dimensional avatar 2B is controlled arbitrarily. Therefore, the next utterance content of the two-dimensional avatar 2B can include the content intended by user P.
  • the intention detection unit 5B causes the control unit 4B to select one of the multiple candidates based on the brain waves of the user P.
  • the candidate for the next speech content of the two-dimensional avatar 2B is arbitrarily determined.
  • the control unit 4B includes an environmental recognition unit 41B and a harmony control unit 42B.
  • the environmental recognition unit 41B recognizes the environment around the two-dimensional avatar 2B based on the detection result by the detection unit 3B.
  • the environmental recognition unit 41B recognizes the meaning of the speech content of the speaking partner and the emotions of the speaking partner as the environment around the two-dimensional avatar 2B.
  • the environmental recognition unit 41B includes a voice processing unit 411, a meaning determination unit 412, and a first emotion determination unit 413.
  • the voice processing unit 411 extracts the contents of the speech of the speaking partner based on the voice data received from the sound collector 32B.
  • the meaning determination unit 412 identifies the meaning of the contents of the speech extracted by the voice processing unit 411.
  • the meaning determination unit 412 determines whether or not the reply to the speech content of the speech partner includes content that should be entered arbitrarily, based on the meaning of the speech content of the speech partner.
  • the first emotion determination unit 413 analyzes the imaging data (moving image showing the face of the speaking partner) received from the imaging device 31B and determines the emotion of the speaking partner from the facial expression of the speaking partner. For example, the first emotion determination unit 413 may determine whether the speaking partner is calm, troubled, angry, etc. from the facial expression of the speaking partner.
  • the harmony control unit 42B autonomously determines candidates for the next speech content of the two-dimensional avatar 2B (sentence ASE) based on the output (image data) of the imaging device 31B and the output (audio data) of the sound collector 32B.
  • the harmony control unit 42B also voluntarily determines the speech content of the blank part of the sentence ASE (part of the next speech content of the two-dimensional avatar 2B) based on the brain waves of the user P.
  • the harmony control unit 42B then harmonizes the candidate speech content (sentence ASE with blank) determined by the autonomous control with the part of the speech content determined by the voluntary control (sentence content of the blank part) to complete the sentence CSE and control the speech of the two-dimensional avatar 2B.
  • the autonomous operation determination unit 44B autonomously determines candidates for the next utterance content (sentence ASE) of the two-dimensional avatar 2B based on the output of the detection unit 3B (autonomous control). Specifically, the autonomous operation determination unit 44B autonomously determines candidates for the next utterance content of the two-dimensional avatar 2B based on the output of the environment recognition unit 41B. In other words, the autonomous operation determination unit 44B generates candidates for the next utterance content of the two-dimensional avatar 2B by autonomous control based on the meaning of the utterance content of the speaking partner and the feelings of the speaking partner.
  • the voluntary movement determination unit 43B determines part of the next speech content of the two-dimensional avatar 2B based on the brain waves of the user P (voluntary control). Specifically, the voluntary movement determination unit 43B determines the speech content of the blank part of the sentence ASE based on the input from the intention detection unit 5B. As a result, part of the next speech content of the two-dimensional avatar 2B is determined voluntarily. Furthermore, the voluntary movement determination unit 43B determines (selects) one of multiple candidates for speech content based on the brain waves of the user P (voluntary control). Specifically, the voluntary movement determination unit 43B determines (selects) one of multiple candidates for speech content based on the input from the intention detection unit 5B.
  • the weight determination unit 45B determines a weight for voluntary control and a weight for autonomous control based on the output of the environment recognition unit 41B. That is, the weight determination unit 45B determines a weight for voluntary control and a weight for autonomous control based on the meaning of the speech content of the speech partner recognized by the environment recognition unit 41B and the emotions of the speech partner.
  • the weight for voluntary control may be referred to as the "first weight.”
  • the weight for autonomous control may be referred to as the "second weight.”
  • the first weight determines the proportion of the next speech content of the two-dimensional avatar 2B that is generated by voluntary control.
  • the second weight determines the proportion of the next speech content of the two-dimensional avatar 2B that is generated by autonomous control.
  • the first weight determines the proportion of the number of characters in the next speech content of the two-dimensional avatar 2B that is generated by voluntary control.
  • the first weight determines the number of blank spaces contained in the sentence ASE.
  • the second weight determines the proportion of the number of characters in the next speech content of the two-dimensional avatar 2B that is generated by autonomous control.
  • the second weight determines the number of characters in the part of the sentence ASE that is not blank spaces.
  • the weight determination unit 45B dynamically changes the values of the first weight and the second weight according to the meaning of the content of the speech of the speech partner. Specifically, the weight determination unit 45B may decrease the first weight and increase the second weight as the proportion of the part that can be estimated in the reply to the speech of the speech partner increases.
  • speech content determined by voluntary control may be referred to as the "speech portion determined by voluntary control.”
  • speech content determined by autonomous control may be referred to as the "speech portion determined by autonomous control.”
  • the weight determination unit 45B further dynamically changes the values of the first weight and the second weight while creating the speech portion by voluntary control (the speech content of the blank part of the sentence ASE) based on the output of the intention detection unit 5B.
  • the weight determination unit 45B dynamically changes the values of the first weight and the second weight when a part of the speech portion by voluntary control (the speech content of the blank part of the sentence ASE) is determined by the output of the intention detection unit 5B.
  • the weight determination unit 45B decreases the first weight and increases the second weight.
  • a new sentence ASE with blanks is generated.
  • the amount of speech portion by autonomous control increases and the number of blanks decreases.
  • the instruction unit 46B instructs the two-dimensional avatar 2B to take the next action. Specifically, the instruction unit 46B harmonizes the speech portion produced by voluntary control with the speech portion produced by autonomous control to instruct the two-dimensional avatar 2B on the next speech content. More specifically, the instruction unit 46B generates a sentence CSE indicating the next speech content of the two-dimensional avatar 2B by incorporating characters indicating the speech portion produced by voluntary control into the blank portion of the sentence ASE generated by autonomous control. The instruction unit 46B then causes the two-dimensional avatar 2B to speak based on the completed sentence CSE. As a result, the action instructed by the instruction unit 46B to the two-dimensional avatar 2B is an action in which the autonomous operation and the voluntary operation are harmonized by the first weight and the second weight. In other words, the voluntary control and the autonomous control of the next speech of the two-dimensional avatar 2B are harmonized.
  • the instruction unit 46B instructs the two-dimensional avatar 2B on the next speech content based on the standard phrase TSE. In other words, the instruction unit 46B causes the two-dimensional avatar 2B to speak the standard phrase TSE.
  • FIG. 13 is a diagram showing the voluntary movement determination unit 43B included in the communication system 100B.
  • the voluntary movement determination unit 43B includes a character input unit 432 and a sentence selection unit 434.
  • the character input unit 432 determines the characters to be input into the blanks of the sentence ASE based on the output of the intention detection unit 5B. In other words, the character input unit 432 determines the characters to be input into the blanks of the sentence ASE based on the brain waves of the user P.
  • the sentence selection unit 434 selects (determines) one of the multiple sentence ASEs based on the output of the intention detection unit 5B.
  • the sentence selection unit 434 also selects (determines) one of multiple standard phrase TSEs (multiple candidates for speech content) based on the output of the intention detection unit 5B.
  • FIG. 14 is a diagram showing the autonomous operation decision unit 44B included in the communication system 100B.
  • the autonomous operation decision unit 44B includes a sentence prediction unit 442, a second emotion determination unit 444, and a body language decision unit 446.
  • the sentence prediction unit 442 determines the speech portion through autonomous control based on the output of the environment recognition unit 41B. Specifically, the sentence prediction unit 442 autonomously determines candidates for the next speech content of the two-dimensional avatar 2B based on the meaning of the speech content of the speech partner and the facial expression (emotion) of the speech partner.
  • the sentence prediction unit 442 determines the next speech content (the speech portion by autonomous control) of the two-dimensional avatar 2B by autonomous control based on the meaning of the speech content of the speech partner determined by the meaning determination unit 412 shown in FIG. 11 and the emotion of the speech partner determined by the first emotion determination unit 413 shown in FIG. 11.
  • the sentence prediction unit 442 determines at least one sentence ASE indicating a candidate for the next speech content of the two-dimensional avatar 2B based on the meaning of the speech content of the speaking partner, the emotion of the speaking partner, and the first weight and second weight determined by the weight determination unit 45B.
  • the sentence ASE indicating the speech content generated by autonomous control includes a gap.
  • the sentence prediction unit 442 generates a sentence ASE with a gap. The gap is a portion into which the speech content by voluntary control is incorporated.
  • the sentence prediction unit 442 determines the proportion of the gap portion (the speech portion by voluntary control) and the proportion of the portion where characters are generated autonomously (the speech portion by autonomous control) based on the first weight and the second weight, and generates a sentence ASE with a gap based on these proportions.
  • the number of characters in the blank portion of the entire sentence ASE will be greater than the number of characters generated autonomously.
  • the value of the first weight is less than the value of the second weight, the number of characters in the blank portion of the entire sentence ASE will be less than the number of characters generated autonomously.
  • the greater the value of the first weight the greater the number of characters in the blank portion of the entire sentence ASE.
  • the smaller the value of the first weight the fewer the number of characters in the blank portion of the entire sentence ASE.
  • the sentence prediction unit 442 leaves blank any portion that cannot be predicted as a reply to the speaking partner. Furthermore, the sentence prediction unit 442 may assign a value indicating likelihood to each character that is generated autonomously. Here, likelihood indicates the degree to which a character is suitable as a reply to the speaking partner. The sentence prediction unit 442 may leave blank characters with lower likelihood values based on the first weight and the second weight.
  • the second emotion determination unit 444 analyzes the next speech content of the two-dimensional avatar 2B to determine the emotion of the user P. Specifically, the second emotion determination unit 444 analyzes the text CSE generated by the instruction unit 46B or the template TSE determined by optional control to determine the emotion of the user P.
  • the body language determination unit 446 determines the body language of the two-dimensional avatar 2B. Specifically, the body language determination unit 446 determines the body language of the two-dimensional avatar 2B depending on the emotion of the user P determined by the second emotion determination unit 444. For example, the body language determination unit 446 may determine whether the user P is calmly replying to the person he is speaking to, whether he is troubled by the content of the speech of the other party, or whether he is angry at the content of the speech of the other party. The body language determination unit 446 determines different body language depending on whether the user P is calmly replying to the person he is speaking to, whether he is troubled by the content of the speech of the other party, or whether he is angry at the content of the speech of the other party.
  • the instruction unit 46B controls the body language of the two-dimensional avatar 2B based on the body language determined by the body language determination unit 446.
  • FIG. 15 is a diagram showing the avatar system 7 included in the communication system 100B.
  • the communication system 100B further includes an avatar system 7.
  • the avatar system 7 is a system that displays a two-dimensional avatar 2B and causes the two-dimensional avatar 2B to speak.
  • the avatar system 7 has a communication unit 71, an avatar display unit 72, an audio output unit 73, a memory unit 74, and a system processing unit 75.
  • the communication unit 71 is controlled by the system processing unit 75 and communicates with the control unit 4B. Specifically, the communication unit 71 receives data indicating the content to be spoken by the two-dimensional avatar 2B from the control unit 4B. More specifically, the communication unit 71 receives data indicating a sentence CSE generated by the harmony control unit 42B, or data indicating a standard phrase TSE determined by the harmony control unit 42B. The sentence CSE and the standard phrase TSE indicate the next speech content of the two-dimensional avatar 2B. The communication unit 71 further receives data indicating body language from the control unit 4B. The communication unit 71 may communicate with the control unit 4B, for example, via a LAN or wireless LAN. Alternatively, the communication unit 71 may communicate with the control unit 4B via a public line network such as the Internet line. The communication unit 71 may include, for example, a LAN board or a LAN module.
  • the avatar display unit 72 is controlled by the system processing unit 75 to display the two-dimensional avatar 2B (see FIG. 12).
  • the avatar display unit 72 includes, for example, a liquid crystal display or an organic EL (electroluminescence) display.
  • the system processing unit 75 generates an image of the two-dimensional avatar 2B and causes the avatar display unit 72 to display the two-dimensional avatar 2B.
  • the audio output unit 73 outputs audio under the control of the system processing unit 75. Specifically, the system processing unit 75 causes the audio output unit 73 to output the next utterance of the two-dimensional avatar 2B based on the data received from the control unit 4B. More specifically, the system processing unit 75 generates audio data from data indicating a text CSE or a standard phrase TSE, and causes the audio output unit 73 to output audio. As a result, the two-dimensional avatar 2B speaks.
  • the audio output unit 73 has, for example, a speaker.
  • the system processing unit 75 causes the two-dimensional avatar 2B to use the body language determined by the body language determination unit 446 described with reference to FIG. 14 when the two-dimensional avatar 2B next speaks. Specifically, the system processing unit 75 generates an image of the two-dimensional avatar 2B using the body language determined by the body language determination unit 446 described with reference to FIG. 14.
  • the storage unit 74 stores various computer programs and various data.
  • the storage unit 74 has, for example, a semiconductor memory.
  • the semiconductor memory includes, for example, a RAM and a ROM.
  • the semiconductor memory may include a flash memory, an EPROM, or an EEPROM, or may include a VRAM.
  • the storage unit 74 may include an HDD or an SSD, or may include a removable medium.
  • the system processing unit 75 controls the operation of each part of the avatar system 7 by executing the computer programs stored in the memory unit 74.
  • the system processing unit 75 may include, for example, a CPU or an MPU.
  • the system processing unit 75 may further include a GPU.
  • FIG. 16 is a block diagram showing the configuration of the intention detection unit 5B included in the communication system 100B.
  • the intention detection unit 5B has a first memory unit 51, a first processing unit 52, and a wireless communication unit 53, similar to the intention detection unit 5A described with reference to FIG. 7.
  • the first storage unit 51 stores the first machine learning model D2.
  • the first machine learning model D2 is a computer program that analyzes brain waves to estimate the intention of the user P. Specifically, the first machine learning model D2 is trained to analyze brain waves and estimate input to the control unit 4B.
  • the first processing unit 52 outputs data indicating the input content estimated by the first machine learning model D2 to the control unit 4B.
  • the first processing unit 52 performs band-pass filtering on the electrical signal indicating the brain waves, and inputs the brain wave data after band-pass filtering to the first machine learning model D2.
  • the first machine learning model D2 may perform analysis using support vector regression.
  • the machine learning algorithm for constructing the first machine learning model D2 is not limited to support vector regression.
  • the machine learning algorithm for constructing the first machine learning model D2 may be, for example, a decision tree, a nearest neighbor method, a naive Bayes classifier, or a neural network. Backpropagation may be used for machine learning.
  • Fig. 17 is a block diagram showing the hardware of the control unit 4B included in the communication system 100B.
  • the control unit 4B has a second storage unit 81, a display unit 82, a communication unit 84, and a second processing unit 85.
  • Fig. 18 is a diagram showing computer programs and data stored in the second storage unit 81 of the control unit 4B included in the communication system 100B.
  • the control unit 4B may be, for example, a personal computer.
  • the second storage unit 81 stores various computer programs and various data.
  • the second storage unit 81 has, for example, a semiconductor memory.
  • the semiconductor memory includes, for example, a RAM and a ROM.
  • the semiconductor memory may include a flash memory, an EPROM, or an EEPROM, or may include a VRAM.
  • the second storage unit 81 may further include an HDD or SSD, and may include removable media.
  • the display unit 82 is controlled by the second processing unit 85 to display various screens GA.
  • the display unit 82 includes, for example, a liquid crystal display or an organic EL display.
  • the various screens GA include a first screen GA1 to a sixth screen GA6, which will be described with reference to Figures 19 to 23.
  • the communication unit 84 is controlled by the second processing unit 85 and communicates with the communication unit 71 (see FIG. 15) of the avatar system 7.
  • the communication unit 84 transmits text data indicating the next utterance of the two-dimensional avatar 2B to the avatar system 7.
  • the communication unit 84 also transmits data indicating the body language to be performed by the two-dimensional avatar 2B when the two-dimensional avatar 2B next speaks to the avatar system 7.
  • the communication unit 84 may include, for example, a LAN board or a LAN module.
  • the second processing unit 85 executes a computer program stored in the second storage unit 81 to perform various processes such as numerical calculations and information processing.
  • the second processing unit 85 may include, for example, a CPU or an MPU.
  • the second processing unit 85 may further include a GPU, or may further include an NPU.
  • the second processing unit 85 may include a quantum computer.
  • the second processing unit 85 executes the computer programs stored in the second memory unit 81, thereby functioning as the voice processing unit 411, meaning determination unit 412, first emotion determination unit 413, weight determination unit 45B, instruction unit 46B, character input unit 432, sentence selection unit 434, sentence prediction unit 442, second emotion determination unit 444, and body language determination unit 446 described with reference to Figures 11, 13, and 14.
  • the second memory unit 81 stores the second machine learning model D21 to the seventh machine learning model D26.
  • the second machine learning model D21 is trained to analyze voice data and extract the contents of the speech.
  • the second machine learning model D21 may be trained to convert voice data into a multidimensional vector.
  • the second processing unit 85 inputs the voice data (voice of the speaking partner) received from the detection unit 3B (sound collector 32B) to the second machine learning model D21.
  • the detection unit 3B sound collector 32B
  • data indicating the contents of the speaking partner's speech is output from the second machine learning model D21 to the second processing unit 85.
  • the third machine learning model D22 is trained to determine the meaning of the speech content of the speaking partner.
  • the third machine learning model D22 may be trained to determine meaning from a multidimensional vector.
  • the third machine learning model D22 is trained to determine whether or not a reply to the speech content of the speaking partner by a fixed phrase TSE is possible from the meaning of the speech content of the speaking partner.
  • data indicating the meaning of the speech content of the expected speaking partner and whether or not a reply by a fixed phrase TSE is possible may be used.
  • the training data for the third machine learning model D22 may include data that combines the meaning of the speech content of the expected speaking partner and data indicating whether or not a reply by a fixed phrase TSE is possible.
  • the second processing unit 85 When functioning as the meaning determination unit 412, the second processing unit 85 inputs data indicating the utterance content extracted by the second machine learning model D21 to the third machine learning model D22. As a result, data indicating the meaning of the utterance content of the other party and a determination result indicating whether or not it is possible to reply to the utterance content of the other party with a fixed phrase TSE are output from the third machine learning model D22 to the second processing unit 85.
  • the fourth machine learning model D23 is trained to analyze imaging data and determine the emotion of the speaking partner from the facial expression of the speaking partner.
  • the second processing unit 85 inputs imaging data (moving images showing the speaking partner's face) received from the detection unit 3B (imaging device 31B) to the fourth machine learning model D23.
  • data indicating the emotion of the speaking partner is output from the fourth machine learning model D23 to the second processing unit 85.
  • the fifth machine learning model D24 is trained to analyze the meaning of the speech content of the speaking partner and the emotions of the speaking partner to determine the values of the first weight and the second weight. For example, training data to be used when training the fifth machine learning model D24 may be created based on the meaning of the expected speech content of the speaking partner, the expected emotions of the speaking partner, the number of characters of the speech content expected as a reply to the speaking partner, and the number of characters to be arbitrarily input in the reply speech content.
  • the second processing unit 85 When functioning as the weight determination unit 45B, the second processing unit 85 inputs data indicating the meaning of the spoken content determined by the third machine learning model D22 and data indicating the emotions of the speaking partner determined by the fourth machine learning model D23 to the fifth machine learning model D24. As a result, data indicating the values of the first weight and the second weight are output from the fifth machine learning model D24 to the second processing unit 85.
  • the fifth machine learning model D24 is trained to decrease the first weight and increase the second weight when the data input from the intention detection unit 5B indicates that the speech content of a part of the gap in the sentence ASE is to be determined.
  • the second processing unit 85 inputs the data input from the intention detection unit 5B to the fifth machine learning model D24 while creating the speech part (the speech content of the gap part in the sentence ASE) by voluntary control.
  • the values of the first weight and the second weight change dynamically.
  • the fifth machine learning model D24 may be trained to determine the value of the first weight by analyzing the meaning of the speech content of the speaking partner and the emotion of the speaking partner.
  • the sixth machine learning model D25 is trained to predict a reply to a speaking partner from the meaning of the speech content of the speaking partner and the emotions of the speaking partner.
  • training data is used that combines the meaning of the expected speech content of the speaking partner, the expected emotions of the speaking partner, the expected next speech content of the two-dimensional avatar 2B (the content of the reply to the speaking partner), and the values of the first weight and the second weight determined according to the content of the reply.
  • the second processing unit 85 When functioning as the sentence prediction unit 442, the second processing unit 85 inputs data indicating the meaning of the spoken content determined by the third machine learning model D22, data indicating the emotion of the speaking partner determined by the fourth machine learning model D23, and the values of the first weight and second weight determined by the fifth machine learning model D24 to the sixth machine learning model D25. As a result, candidates for the next spoken content of the two-dimensional avatar 2B (data indicating the sentence ASE) are output from the sixth machine learning model D25 to the second processing unit 85.
  • the seventh machine learning model D26 is trained to determine the emotion of the user P from the next utterance content of the two-dimensional avatar 2B.
  • the second processing unit 85 inputs data indicating the next utterance content of the two-dimensional avatar 2B to the seventh machine learning model D26.
  • data indicating the emotion of the user P is output from the seventh machine learning model D26 to the second processing unit 85.
  • the machine learning algorithm for constructing the second machine learning model D21 to the seventh machine learning model D26 is not particularly limited, and may be, for example, a decision tree, a nearest neighbor method, a naive Bayes classifier, a support vector machine, or a neural network. Backpropagation may be used for machine learning.
  • the machine learning algorithm for constructing the second machine learning model D21 may be a Transformer.
  • the Transformer is a deep learning model.
  • the control unit 4B will be further described with reference to Figures 17 and 18.
  • the second storage unit 81 further stores fixed phrase data D27 and body language data D28.
  • the fixed phrase data D27 indicates various fixed phrases TSE. Specifically, the fixed phrase data D27 may indicate "Good morning,” “Hello,” “Good evening,” “Welcome,” “What are you looking for,” and the like.
  • the third machine learning model D22 determines that a reply can be made to the speech content of the other party using a fixed phrase TSE
  • the second processing unit 85 refers to the fixed phrase data D27 and causes the display unit 82 to display a list of fixed phrases TSE.
  • the body language data D28 indicates various types of body language.
  • the various types of body language include body language corresponding to various emotions.
  • the body language data D28 may indicate body language suitable for calmly replying to the other person, body language suitable for being troubled by the content of the other person's speech, and body language suitable for being angry at the content of the other person's speech.
  • the second processing unit 85 determines the body language expressing the user P's emotions from the body language data D28 based on the output from the seventh machine learning model D26.
  • Figs. 19(a) to 19(c) are diagrams showing the screen GA displayed on the display unit 82 when the two-dimensional avatar 2B is made to speak using a fixed phrase TSE.
  • Figs. 19(a) and 19(b) show the screen GA displayed on the display unit 82 when a fixed phrase TSE is selected.
  • Fig. 19(c) shows the screen GA displayed on the display unit 82 when deciding whether or not to confirm the next speech content of the two-dimensional avatar 2B.
  • the second processing unit 85 refers to the template data D27 and causes the display unit 82 to display a screen GA (first screen GA1) showing a list of template TSEs.
  • FIG. 19(a) illustrates the first screen GA1 on which five template sentences TSE1 to TSE5 are arranged vertically. Note that "x" represents a letter.
  • the second processing unit 85 further displays a first cursor CA1 on the first screen GA1.
  • the first cursor CA1 is a cursor for selecting a sentence.
  • the first cursor CA1 may surround one of the multiple standard phrases TSE.
  • FIG. 19(a) illustrates an example of the first cursor CA1 selecting standard phrase TSE1 of the five standard phrases TSE1 to TSE5. In this case, the first cursor CA1 may surround the standard phrase TSE1.
  • the second processing unit 85 moves the first cursor CA1 in the direction in which the standard phrases TSE are arranged.
  • the second processing unit 85 moves the first cursor CA1 vertically.
  • the standard phrases TSE selected by the first cursor CA1 change over time.
  • the intention detection unit 5B first processing unit 52
  • the control unit 4B second processing unit 85
  • the user P thinks about the first action in his/her mind at the timing when the standard phrase TSE with the content that he/she wants the two-dimensional avatar 2B to utter is selected by the first cursor CA1.
  • the first action indicates an action of the user P.
  • the first action that the user P thinks of is not particularly limited as long as it is an action that can be detected from brain waves.
  • the first action may be an action of the user P raising his/her hand, or an action of the user P moving his/her finger.
  • the first action may be determined arbitrarily. Data that associates the first action with the stopping of the first cursor CA1 is prestored in the first storage unit 51 (see FIG. 16) of the intention detection unit 5B.
  • the intention detection unit 5B (first processing unit 52) generates a command to stop the movement of the first cursor CA1 and sends it to the control unit 4B.
  • the second processing unit 85 stops the first cursor CA1.
  • the second processing unit 85 stops the first cursor CA1 it causes the second screen GA2 to be displayed on the display unit 82.
  • the second screen GA2 displays the template TSE selected based on the user P's brain waves.
  • the second screen GA2 further displays a first button SB1, a second button SB2, and a second cursor CA2.
  • Figure 19(c) illustrates an example in which a template TSE3 is selected based on the user P's brain waves.
  • the first button SB1 is a soft button for confirming the next speech content of the two-dimensional avatar 2B.
  • the second button SB2 is a soft button for canceling the selection of the standard text TSE. In other words, the second button SB2 is a cancel button.
  • the second cursor CA2 may surround one of the first button SB1 and the second button SB2.
  • the second processing unit 85 moves the second cursor CA2 in the direction in which the first button SB1 and the second button SB2 are aligned. As a result, the soft button selected by the second cursor CA2 changes over time between the first button SB1 and the second button SB2.
  • the intention detection unit 5B (first processing unit 52) and the control unit 4B (second processing unit 85) press the first button SB1 based on the user P's brain waves while the second screen GA2 is being displayed.
  • the user P determines the next speech content of the two-dimensional avatar 2B to be the content of the standard text TSE displayed on the second screen GA2, the user P thinks about the second action in his/her mind at the timing when the first button SB1 is selected by the second cursor CA2.
  • the second action indicates an action of user P.
  • the second action that user P thinks of is not particularly limited as long as it is an action that can be detected from brain waves.
  • the second action can be determined arbitrarily.
  • the second action may be the same as the first action, or may be a different action from the first action.
  • Data associating the second action with the stopping of the second cursor CA2 is prestored in the first storage unit 51 (see FIG. 16) of the intention detection unit 5B.
  • the intention detection unit 5B first processing unit 52
  • the second processing unit 85 presses the first button SB1 to confirm the content of the standard text TSE.
  • the second processing unit 85 presses the soft button selected by the second cursor CA2.
  • the intention detection unit 5B (first processing unit 52) and the control unit 4B (second processing unit 85) press the second button SB2 based on the user P's brain waves while the second screen GA2 is being displayed.
  • the second processing unit 85 presses the second button SB2 to cancel the selection of the standard phrase TSE.
  • the second processing unit 85 presses the second button SB2 it causes the display unit 82 to display the first screen GA1 again.
  • Figs. 20(a) to 20(c) are diagrams showing screens GA displayed on the display unit 82 when selecting one of the candidates (multiple sentence ASE) for the next utterance content of the two-dimensional avatar 2B.
  • Figs. 21(a), 21(b), 22(a), and 22(b) are diagrams showing screens GA displayed on the display unit 82 when the user P inputs any characters.
  • Fig. 23 is a diagram showing screen GA displayed on the display unit 82 when deciding whether or not to determine the next utterance content of the two-dimensional avatar 2B.
  • FIG. 20(a) illustrates an example of the third screen GA3 in which five candidates (sentence ASE11 to sentence ASE15) are arranged vertically.
  • the sentence ASEs contain blank spaces.
  • the parts indicated by "circles” are blank spaces. Note that "x" represents a letter.
  • the second processing unit 85 further displays a first cursor CA1 on the third screen GA3.
  • the first cursor CA1 in the third screen GA3 selects a candidate for speech content (sentence ASE).
  • the first cursor CA1 may surround one candidate (sentence ASE) among multiple candidates (sentence ASE).
  • the second processing unit 85 moves the first cursor CA1 in the direction in which the candidates for speech content (sentence ASE) are lined up.
  • the candidates (sentence ASE) selected by the first cursor CA1 change over time.
  • the intention detection unit 5B (first processing unit 52) and the control unit 4B (second processing unit 85) select one candidate (sentence ASE) from among multiple candidates (sentence ASE) based on the user P's brain waves while the third screen GA3 is being displayed.
  • the second processing unit 85 causes the display unit 82 to display the fourth screen GA4.
  • the fourth screen GA4 displays candidates for speech content (sentence ASE) selected based on the user P's brain waves while the third screen GA3 is displayed.
  • the fourth screen GA4 further displays a first button SB1, a second button SB2, and a second cursor CA2, similar to the second screen GA2 described with reference to FIG. 19(c).
  • FIG. 20(c) illustrates an example in which sentence ASE13 is selected based on the user P's brain waves.
  • the candidate speech content (sentence ASE) is finalized.
  • the selection of the candidate speech content (sentence ASE) is canceled.
  • the second processing unit 85 causes the display unit 82 to display the third screen GA3 again.
  • the second processing unit 85 causes the display unit 82 to display the fifth screen GA5, as shown in FIG. 21(a).
  • the fifth screen GA5 displays the Japanese syllabary JS and the third cursor CA3.
  • the third cursor CA3 is a cursor that selects one of the multiple rows included in the 50-sound table JS.
  • the third cursor CA3 may surround one of the multiple rows included in the 50-sound table JS.
  • Figure 21 (a) illustrates an example of the third cursor CA3 that selects the "A" row in the 50-sound table JS. In this case, the third cursor CA3 may surround the "A" row.
  • the second processing unit 85 moves the third cursor CA3 in the direction in which multiple rows included in the Japanese syllabary JS are arranged. For example, the second processing unit 85 moves the third cursor CA3 horizontally. As a result, the row selected by the third cursor CA3 changes over time.
  • the intention detection unit 5B (first processing unit 52) and the control unit 4B (second processing unit 85) select one of the multiple rows included in the Japanese syllabary JS based on the brain waves of the user P. For example, when the user P selects the "na" row, he or she thinks about the third action at the time the "na" row is selected by the third cursor CA3.
  • the third action indicates an action of user P. There are no particular limitations on the third action that user P thinks of, so long as it is an action that can be detected from brain waves.
  • the third action can be determined arbitrarily.
  • the third action may be the same as the first or second action, or may be a different action from the first and second actions.
  • the first storage unit 51 (see FIG. 16) of the intention detection unit 5B prestores data associating the third action with the stopping of the third cursor CA3.
  • the intention detection unit 5B first processing unit 52
  • the intention detection unit 5B (first processing unit 52) generates a command to stop the movement of the third cursor CA3 and transmits it to the control unit 4B.
  • the second processing unit 85 stops the third cursor CA3.
  • the fourth cursor CA4 is a cursor that selects each character included in the 50-sound table JS one by one.
  • the fourth cursor CA4 may encircle one of the multiple characters included in the 50-sound table JS.
  • FIG. 22(a) illustrates an example of the fourth cursor CA4 selecting the character "na”. In this case, the fourth cursor CA4 may encircle the character "na”.
  • the second processing unit 85 moves the fourth cursor CA4 in the direction in which multiple rows (columns) included in the Japanese syllabary JS are arranged.
  • the second processing unit 85 moves the fourth cursor CA4 vertically.
  • the second processing unit 85 moves the fourth cursor CA4 so as to sequentially select each character included in the row selected based on the brain waves of the user P. For example, when the "na" row is selected based on the brain waves of the user P, each character included in the "na" row is sequentially selected by the fourth cursor CA4.
  • the intention detection unit 5B first processing unit 52
  • the control unit 4B second processing unit 85
  • the fourth movement indicates a movement of user P.
  • the fourth movement can be determined arbitrarily.
  • the fourth movement may be the same as the first movement, the second movement, or the third movement, or may be a movement different from any of the first movement, the second movement, and the third movement.
  • the first storage unit 51 of the intention detection unit 5B prestores data associating the fourth action with the stopping of the fourth cursor CA4.
  • the intention detection unit 5B first processing unit 52
  • the intention detection unit 5B first processing unit 52
  • the second processing unit 85 stops the fourth cursor CA4.
  • the second processing unit 85 inserts the characters selected by the fourth cursor CA4 when the fourth cursor CA4 is stopped into the blanks of the candidate speech content (sentence ASE) confirmed on the fourth screen GA4 (see FIG. 20(c)). After that, the intention detection unit 5B and the control unit 4B repeat the processes described with reference to FIG. 21(a) to FIG. 22(b) until all blanks included in the candidate speech content (sentence ASE) are filled with characters.
  • the second processing unit 85 causes the display unit 82 to display the sixth screen GA6, as shown in FIG. 23.
  • the sixth screen GA6 displays the completed sentence CSE.
  • the sixth screen GA6 further displays a first button SB1, a second button SB2, and a second cursor CA2, similar to the second screen GA2 described with reference to FIG. 19(c).
  • the next speech content (text CSE) of the two-dimensional avatar 2B is confirmed.
  • the second button SB2 is pressed based on the user P's brain waves while the sixth screen GA6 is displayed, the next speech content (text CSE) of the two-dimensional avatar 2B is canceled.
  • the second processing unit 85 causes the display unit 82 to display the third screen GA3 (see FIG. 20(a)) again.
  • the second processing unit 85 may cause the display unit 82 to display the fifth screen GA5 (see FIG. 21(a)) again.
  • Fig. 24 is a flowchart showing the processing executed by the intention detection unit 5B and the control unit 4B included in the communication system 100B.
  • the processing shown in Fig. 24 starts when the speaking partner speaks. In other words, the processing shown in Fig. 24 starts when the sound collector 32B collects the voice (utterance) of the speaking partner.
  • the second processing unit 85 of the control unit 4B acquires the contents of the speech of the other party through the voice processing unit 411 (step S41).
  • the second processing unit 85 inputs the voice data received from the sound collector 32B to the second machine learning model D21.
  • data indicating the contents of the speech of the other party is output from the second machine learning model D21 to the second processing unit 85.
  • the second processing unit 85 acquires the content of the speech of the other party, it acquires the meaning of the content of the speech by the meaning determination unit 412. Then, the second processing unit 85 judges whether or not a reply using a fixed phrase TSE is possible based on the meaning of the content of the speech (step S42).
  • the second processing unit 85 inputs data indicating the content of the speech extracted by the second machine learning model D21 to the third machine learning model D22. As a result, the result of estimating the meaning of the content of the speech of the other party is output from the third machine learning model D22 to the second processing unit 85. Furthermore, a judgment result indicating whether or not a reply using a fixed phrase TSE is possible to the content of the speech of the other party is output from the third machine learning model D22 to the second processing unit 85.
  • the second processing unit 85 causes the display unit 82 to display the first screen GA1 by referring to the standard phrase data D27, as described with reference to FIG. 19(a) (step S43).
  • one of the multiple standard phrases TSE is selected by optional control (step S44).
  • the intention detection unit 5B first processing unit 52
  • the control unit 4B second processing unit 85
  • the second processing unit 85 uses the second emotion determination unit 444 to determine the emotion of the user P from the finalized template TSE (step S45).
  • the second processing unit 85 inputs data indicating the next utterance content of the two-dimensional avatar 2B to the seventh machine learning model D26.
  • data indicating the emotion of the user P is output from the seventh machine learning model D26 to the second processing unit 85.
  • the second processing unit 85 inputs the finalized template TSE by optional control to the seventh machine learning model D26 to determine the emotion of the user P.
  • the second processing unit 85 determines the emotion of the user P, it refers to the body language data D28 to determine the body language expressing the emotion of the user P (step S46).
  • the second processing unit 85 When the second processing unit 85 has determined the next speech content and body language of the two-dimensional avatar 2B, it transmits data indicating the next speech content of the two-dimensional avatar 2B and data indicating the body language to the avatar system 7 via the communication unit 84 (step S47). As a result, the process shown in FIG. 24 ends.
  • the second processing unit 85 determines the emotion of the speaking partner based on the facial expression of the speaking partner shown in the video received from the imaging device 31B using the first emotion determination unit 413 (step S51). In detail, the second processing unit 85 inputs the imaging data received from the imaging device 31B (video showing the speaking partner's face) to the fourth machine learning model D23. As a result, data indicating the emotion of the speaking partner is output from the fourth machine learning model D23 to the second processing unit 85.
  • the second processing unit 85 determines the emotion of the speaking partner, it calculates each value of the first weight and the second weight (step S52).
  • the second processing unit 85 inputs data indicating the meaning of the utterance content estimated in step S42 and data indicating the emotion of the speaking partner determined in step S51 to the fifth machine learning model D24.
  • data indicating each value of the first weight and the second weight is output from the fifth machine learning model D24 to the second processing unit 85.
  • the second processing unit 85 After calculating the values of the first weight and the second weight, the second processing unit 85 predicts a reply to the speaking partner from the meaning of the speaking partner's content and the speaking partner's emotions, and displays a list of candidates for the next speaking content of the two-dimensional avatar 2B on the display unit 82 (step S53). In other words, the second processing unit 85 causes the third screen GA3 to be displayed on the display unit 82, as described with reference to FIG. 20(a).
  • the second processing unit 85 inputs data indicating the meaning of the utterance content estimated in step S42, data indicating the emotion of the speaking partner determined in step S51, and the first and second weighting values calculated in step S52 to the sixth machine learning model D25.
  • candidates for the next utterance content of the two-dimensional avatar 2B (sentence ASE with a blank) are output from the sixth machine learning model D25 to the second processing unit 85.
  • the second processing unit 85 causes the display unit 82 to display a list of the candidates for the utterance content output from the sixth machine learning model D25.
  • one of the multiple candidates (sentence ASE) is selected by optional control (step S54).
  • the intention detection unit 5B first processing unit 52
  • the control unit 4B second processing unit 85
  • step S55 characters are input into the blank spaces included in the candidate speech content by optional control (step S55).
  • the intention detection unit 5B first processing unit 52
  • the control unit 4B second processing unit 85
  • the next speech content of the two-dimensional avatar 2B is determined.
  • the second processing unit 85 determines the emotion of the user P from the next speech content (text CSE) of the two-dimensional avatar 2B using the second emotion determination unit 444 (step S45).
  • the second processing unit 85 determines the emotion of the user P, it refers to the body language data D28 to determine the body language expressing the emotion of the user P (step S46).
  • the second processing unit 85 When the second processing unit 85 has determined the next speech content and body language of the two-dimensional avatar 2B, it transmits data indicating the next speech content of the two-dimensional avatar 2B and data indicating the body language to the avatar system 7 via the communication unit 84 (step S47). As a result, the process shown in FIG. 24 ends.
  • the fifth machine learning model D24 is an example of a "change condition.” In other words, the first weight and the second weight are changed by the fifth machine learning model D24.
  • the fifth machine learning model D24 (the change conditions for the first weight and the second weight) is adjusted based on the evaluation function. Adjusting the fifth machine learning model D24 indicates retraining the fifth machine learning model D24. When retraining the fifth machine learning model D24, the training data adjusted based on the evaluation function is input to the fifth machine learning model D24.
  • the training data for the fifth machine learning model D24 is created based on the meaning of the expected speech content of the interlocutor, the expected emotions of the interlocutor, the number of characters in the expected speech content as a reply to the interlocutor, and the number of characters to be input arbitrarily in the reply speech content.
  • the number of characters to be input arbitrarily in the reply speech content corresponds to the number of "blanks" included in the sentence ASE indicating candidates for the next speech content of the two-dimensional avatar 2B.
  • the "number of blanks" to be input as training data to the fifth machine learning model D24 is adjusted based on an evaluation function.
  • the evaluation function includes, as parameters, the control performance and operability acquired during training of the communication system 100B.
  • the control performance indicates a value reflecting the amount of deviation between the number of blank spaces predicted by the operator operating the two-dimensional avatar 2B during training of the communication system 100B and the number of blank spaces included in the candidate speech content determined by the sentence prediction unit 442 (second processing unit 85).
  • the control performance indicates a value reflecting the amount of deviation between the number of blank spaces predicted by the operator to be included in the next utterance content and the number of blank spaces included in the candidate speech content displayed on the third screen GA3.
  • the control performance may indicate the difference between the number of blank spaces predicted by the operator and the actual number of blank spaces.
  • the operability indicates a value reflecting the ease of operation of the spoken content as evaluated by the worker.
  • the operability may be evaluated using a scale such as a 5-point scale.
  • the operability may be evaluated using a visual analog scale. For example, the worker may rate the operability low when the actual number of blank spaces differs from the predicted number. Also, the worker may rate the operability high when the actual number of blank spaces differs from the predicted number but the worker feels that the actual number of blank spaces makes it easier to complete the intended sentence.
  • the number of voids included in the training data may be adjusted using the following equation (24) as an evaluation function.
  • Evaluation function a f (operation feeling) ⁇ b g (error) (24)
  • f (operation feel) is a function that includes the operation feel as a parameter.
  • g (error) is a function that includes the deviation (error) between the number of voids predicted by the operator and the actual number of voids as a parameter.
  • f (operation feel) may be a function whose value increases as the value of the operation feel increases.
  • g (error) may be a function whose value increases as the deviation (error) increases.
  • the coefficients "a” and "b” may be arbitrarily determined by the operator.
  • the worker refers to equation (24) and adjusts the data (number of blanks) included in the training data so that the value of the evaluation function becomes as large as possible.
  • the change conditions for the first and second weights are adjusted so that the value of the evaluation function becomes as large as possible.
  • the worker adjusts the data (number of blanks) included in the training data so that the value of the evaluation function becomes a locally optimal solution.
  • the change conditions for the first and second weights (fifth machine learning model D24) are adjusted so that the value of the evaluation function becomes a locally optimal solution.
  • the communication system 100B which is an example of the control system 100 of this embodiment, has been described with reference to Figures 11 to 24.
  • voluntary control and autonomous control can be harmonized.
  • the next utterance content of the two-dimensional avatar 2B (the action of the operation target 2) is less likely to differ from the intention of the user P.
  • harmonizing voluntary control and autonomous control it becomes easier to create a sentence CSE using voluntary control.
  • the burden on the user P can be reduced.
  • the intention of the user P can be reflected in each value of the first weight and the second weight. As a result, the burden on the user P can be further reduced.
  • the change conditions for the first and second weights can be adjusted based on the control performance and the operation feel so that the evaluation function is as large as possible.
  • the proportion of the speech portion due to voluntary control and the proportion of the speech portion due to autonomous control are adjusted so that the evaluation function is as large as possible. Therefore, it is possible to harmonize the creation of a sentence that is closer to the intention of the user P as a candidate for the next speech content of the two-dimensional avatar 2B (operation target 2) with the ease of operation of the speech content by the user P. This improves the convenience of the communication system 100B.
  • At least one candidate for the next utterance content of the two-dimensional avatar 2B is determined based on the speech content of the speech partner, the emotion of the speech partner, the first weight, and the second weight.
  • the sentence prediction unit 442 may determine at least one candidate for the next utterance content of the two-dimensional avatar 2B based on the speech content of the speech partner, the first weight, and the second weight. In other words, the sentence prediction unit 442 may determine the content of the reply to the speech partner using only the speech content of the speech partner out of the speech content of the speech partner and the emotion of the speech partner.
  • the present invention is useful for systems that control objects that can be autonomously or voluntarily controlled.

Landscapes

  • Engineering & Computer Science (AREA)
  • General Engineering & Computer Science (AREA)
  • Theoretical Computer Science (AREA)
  • Human Computer Interaction (AREA)
  • Physics & Mathematics (AREA)
  • General Physics & Mathematics (AREA)
  • Manipulator (AREA)
PCT/JP2024/018061 2023-05-17 2024-05-16 制御システム Ceased WO2024237300A1 (ja)

Priority Applications (2)

Application Number Priority Date Filing Date Title
JP2025520626A JPWO2024237300A1 (https=) 2023-05-17 2024-05-16
CN202480032672.2A CN121127816A (zh) 2023-05-17 2024-05-16 控制系统

Applications Claiming Priority (2)

Application Number Priority Date Filing Date Title
JP2023081911 2023-05-17
JP2023-081911 2023-05-17

Publications (1)

Publication Number Publication Date
WO2024237300A1 true WO2024237300A1 (ja) 2024-11-21

Family

ID=93519768

Family Applications (1)

Application Number Title Priority Date Filing Date
PCT/JP2024/018061 Ceased WO2024237300A1 (ja) 2023-05-17 2024-05-16 制御システム

Country Status (3)

Country Link
JP (1) JPWO2024237300A1 (https=)
CN (1) CN121127816A (https=)
WO (1) WO2024237300A1 (https=)

Citations (2)

* Cited by examiner, † Cited by third party
Publication number Priority date Publication date Assignee Title
JP2010237859A (ja) * 2009-03-30 2010-10-21 Honda Motor Co Ltd 機械および機械システム
JP2022059140A (ja) * 2020-10-01 2022-04-13 株式会社Agama-X 情報処理装置及びプログラム

Patent Citations (2)

* Cited by examiner, † Cited by third party
Publication number Priority date Publication date Assignee Title
JP2010237859A (ja) * 2009-03-30 2010-10-21 Honda Motor Co Ltd 機械および機械システム
JP2022059140A (ja) * 2020-10-01 2022-04-13 株式会社Agama-X 情報処理装置及びプログラム

Also Published As

Publication number Publication date
JPWO2024237300A1 (https=) 2024-11-21
CN121127816A (zh) 2025-12-12

Similar Documents

Publication Publication Date Title
Chen et al. Real-time multi-modal human–robot collaboration using gestures and speech
Xue et al. Progress and prospects of multimodal fusion methods in physical human–robot interaction: A review
Mahmud et al. Interface for human machine interaction for assistant devices: A review
JP7520123B2 (ja) 人間－ロボット混在製造プロセスにおける自動異常検出のためのシステムおよび方法
EP3743892B1 (en) Visualization of reconstructed handstate information
Poirier et al. Voice control interface prototype for assistive robots for people living with upper limb disabilities
Oz et al. American sign language word recognition with a sensory glove using artificial neural networks
US7340100B2 (en) Posture recognition apparatus and autonomous robot
US10603797B2 (en) Machine learning device, robot system, and machine learning method for learning motion of robot engaged in task performed by human and robot in cooperate with each other
US11833691B2 (en) Hybrid robotic motion planning system using machine learning and parametric trajectories
US11945117B2 (en) Anticipating user and object poses through task-based extrapolation for robot-human collision avoidance
JP2005199403A (ja) 情動認識装置及び方法、ロボット装置の情動認識方法、ロボット装置の学習方法、並びにロボット装置
US12440996B2 (en) Communication robot, communication robot control method, and program
CN120395905B (zh) 基于深度学习的人形机器人体感控制系统及方法
CN120095830B (zh) 机器人控制方法、计算机设备及计算机可读存储介质
CN110634505A (zh) 声音期间检测装置、声音期间检测方法、存储介质、声音认识装置以及机器人
Wang et al. Deep-learning-based object classification of tactile robot hand for smart factory: D. Wang et al.
US11712804B2 (en) Systems and methods for adaptive robotic motion control
US11554480B2 (en) Systems and methods for implementing miniaturized cycloidal gears
WO2024237300A1 (ja) 制御システム
JP2004066367A (ja) 行動パターン生成装置、行動パターン生成方法、及び行動パターン生成プログラム
CN120872157A (zh) 基于多模态情感感知的动作生成方法、装置、设备及介质
US11724390B2 (en) Systems and methods for automated preloading of actuators
MB et al. Real-Time Sign Language Translator with IoT and Flex Sensors for Multilingual Support
Shree et al. A Virtual Assistor for Impaired People by using Gestures and Voice

Legal Events

Date Code Title Description
121 Ep: the epo has been informed by wipo that ep was designated in this application

Ref document number: 24807250

Country of ref document: EP

Kind code of ref document: A1

ENP Entry into the national phase

Ref document number: 2025520626

Country of ref document: JP

Kind code of ref document: A

WWE Wipo information: entry into national phase

Ref document number: 2025520626

Country of ref document: JP

NENP Non-entry into the national phase

Ref country code: DE