JP7237891B2 - LEARNING EXECUTION DEVICE, PROGRAM, AND LEARNING EXECUTION METHOD - Google Patents

Description

The present invention relates to a learning execution device, program, and learning execution method.

In the field of CG (Computer Graphics), simulation techniques based on muscle contraction have been known (see, for example, Non-Patent Documents 1 to 6). In conventional simulation methods, a so-called Hill-type model, a so-called CPG (Central Pattern Generator), and the like have been used.
[Prior art documents]
[Non-Patent Literature]
[Non-Patent Document 1] Thomas Geitenbeek, Michiel van de Panne, AF vds Flexible muscle-based locomotion for bipedal creatures. ACM Transactions on Graphics, (206), 2013.
[Non-Patent Document 2] Jack M. Wang, Samuel R. Hmner, SLVK Optimizing locomotion controllers using biologically-based actuators and objectives. ACM Trans. Graph, 31(4), 2012.
[Non-Patent Document 3] Yoonsang Lee, Moon Seok Park, TKJL Locomotion control for many-muscle humanoids. ACM Transactions on Graphics, 33(6), 2014.
[Non-Patent Document 4] Sehee Min, Jungdam Won, SLJPJL Softcon: simulation and control of soft-bodied animals with biomimetic actuators. ACM Transactions on Graphics, 38(6):208:1-208:12, 2019.
[Non-Patent Document 5] Cecila Laschi, Matteo Cianchetti, BML m. MFPD Soft robot arm inspired by the octopus. Advanced Robotics, 26(7):709-727, 2012.
[Non-Patent Document 6] Jungdam Won, Jongho Park, KKJL How to train your dragon: Example-guided control of flapping flight. ACM Transactions on Graphics, 36(4):1:1-1:12, 2017.

According to a first aspect of the present invention, a learning execution device is provided. The learning execution device stores a muscle model that operates the muscle by contracting muscle fibers connected to the motor neurons included in the motor units according to the firing pattern of a plurality of interneurons to which the motor units are connected respectively. You may have a department. The learning execution device may include a motion setting unit that sets a target motion of the muscle model. The learning execution device is a learning execution unit that learns firing patterns, and implements a target motion by executing learning that rewards firing patterns in which the motion of the muscle model is closer to the target motion, among a plurality of firing patterns. A learning execution unit that learns the firing pattern to be used may be provided.

The learning execution unit operates the muscle model according to each of the plurality of firing patterns, generates a plurality of firing patterns based on the firing pattern in which the motion of the muscle model is closer to the target motion, and generates the plurality of firing patterns. and repeatedly generating a plurality of firing patterns based on the firing pattern in which the motion of the muscle model is closer to the target motion, thereby learning the firing pattern that realizes the target motion. you can The learning execution unit operates the muscle model according to each of the plurality of randomly generated firing patterns, generates a plurality of firing patterns based on the firing pattern in which the motion of the muscle model is closer to the target motion, The target motion is realized by operating the muscle model according to each of the plurality of firing patterns and repeatedly generating a plurality of firing patterns based on the firing pattern in which the motion of the muscle model is closer to the target motion. Firing patterns may be learned. The learning execution unit operates the muscle model according to each of the plurality of firing patterns generated based on the learned firing pattern, and operates the muscle model based on the firing pattern closer to the target motion. Generating a firing pattern, operating the muscle model according to each of the plurality of firing patterns, and repeatedly generating a plurality of firing patterns based on the firing pattern in which the motion of the muscle model is closer to the target motion, A firing pattern that achieves the above target operation may be learned. The learning execution unit may cause the motor unit that contracted the muscle fiber to grow when the muscle model is operated based on the firing pattern. The muscle model may include a fast-twitch motor unit and a slow-twitch motor unit. When the muscle model is operated based on the firing pattern, the learning execution unit performs the fast-twitch motor unit. Units and motor units of the slow twitch muscle may be grown according to different criteria. The information storage unit stores a first parameter indicating whether the motor unit is fast-twitch or slow-twitch, a second parameter indicating contractile energy, a maximum value of the second parameter, and A third parameter indicating self-resilience and the maximum value of the third parameter may be stored, and the learning execution unit stores the first parameter, the second parameter, the maximum value of the second parameter, the Learning using three parameters and the maximum value of the third parameter may be performed. The learning execution unit subtracts a predetermined value from the second parameter each time the motor unit contracts, and recovers the second parameter over time while the third parameter is not 0. you can The information storage unit stores a fourth parameter indicating an amount of energy consumed each time the motor unit contracts when the motor unit is a fast-twitch muscle, and the learning execution unit stores the first parameter, Learning may be performed using the second parameter, the maximum value of the second parameter, the third parameter, the maximum value of the third parameter, and the fourth parameter. The learning execution unit subtracts the value of the fourth parameter from the second parameter each time the motor unit contracts if the motor unit is a fast-twitch muscle, and if the motor unit is a slow-twitch muscle, , a value other than the value of the fourth parameter may be subtracted from the second parameter each time the motor unit contracts. When the learning execution unit determines that the muscle fiber has recovered after determining that the muscle fiber has been damaged, if the motor unit is a fast-twitch muscle, the maximum value of the second parameter and the The value of the fourth parameter may be increased, and the maximum value of the third parameter may be increased if the motor unit is a slow twitch muscle. After determining that the muscle fiber is damaged, the learning execution unit determines that the muscle fiber has recovered, and if the motor unit is a fast-twitch muscle, the maximum value of the third parameter is not increased. you can After determining that the muscle fiber has been damaged, the learning execution unit determines that the muscle fiber has recovered, and if the motor unit is a slow twitch muscle, the maximum value of the second parameter is not increased. you can The learning execution unit may determine that the muscle fiber is damaged when the second parameter becomes zero. The information storage unit may store, for the motor unit, a fifth parameter related to the use of the motor unit, and the learning execution unit controls the level of the motor unit as the fifth parameter increases. is improved, the higher the level of the motor unit, the more difficult it is to increase the maximum value of the second parameter and the value of the fourth parameter when the motor unit is fast-twitch, and the motor unit is slow-twitch. It may be difficult to increase the maximum value of the third parameter in some cases. After contracting one motor unit, the learning execution unit may learn the firing pattern by preventing the one motor unit from contracting until a predetermined refractory period elapses. The learning execution unit may learn the firing pattern by shortening the refractory period as the temperature of the motor unit increases. When the motion of the muscle model that is operated according to the plurality of time-series firing patterns achieves the target motion, the learning execution unit traces back a predetermined time from the firing pattern in which the target motion was achieved. The learning may be performed by updating the firing patterns in the state of

According to a second aspect of the present invention, a learning execution device is provided. The learning execution device provides, for each of a plurality of muscle fibers included in a muscle, a first parameter indicating whether the muscle fiber is fast-twitch or slow-twitch, a second parameter indicating contractile energy, and You may provide the information storage part which stores the maximum value of a 2nd parameter, the 3rd parameter which shows self-healing power, and the maximum value of a 3rd parameter. The learning execution device executes learning using the first parameter, the second parameter, the maximum value of the second parameter, the third parameter, and the maximum value of the third parameter, thereby forming a muscle model. A learning execution unit for learning may be provided.

The information storage unit may store a fourth parameter indicating an amount of energy consumed each time the muscle fiber contracts when the muscle fiber is a fast-twitch muscle fiber, and the learning execution unit stores the first Learning using the parameter, the second parameter, the maximum value of the second parameter, the third parameter, the maximum value of the third parameter, and the fourth parameter may be performed. The learning execution unit subtracts a predetermined value from the second parameter each time the muscle fiber contracts, and restores the second parameter over time while the third parameter is not 0. , when it is determined that the muscle fiber has recovered after determining that the muscle fiber is damaged, and if the muscle fiber is a fast-twitch, the maximum value of the second parameter and the value of the fourth parameter and if the muscle fiber is slow twitch, the muscle model may be learned by increasing the maximum value of the third parameter. The learning execution unit subtracts the value of the fourth parameter from the second parameter each time the muscle fiber contracts when the muscle fiber is a fast-twitch muscle fiber, and subtracts the value of the fourth parameter from the second parameter each time the muscle fiber contracts. , a value other than the value of the fourth parameter may be subtracted from the second parameter each time the muscle fiber contracts. After determining that the muscle fiber is damaged, the learning execution unit determines that the muscle fiber has recovered, and if the muscle fiber is a fast-twitch muscle fiber, does not increase the maximum value of the third parameter. you can After determining that the muscle fiber has been damaged, the learning execution unit determines that the muscle fiber has recovered, and if the muscle fiber is slow twitch, the maximum value of the second parameter is not increased. you can

According to a third aspect of the present invention, there is provided a program for causing a computer to function as the learning execution device.

According to a fourth aspect of the present invention, there is provided a computer-implemented learning execution method. In the learning execution method, according to the firing pattern of multiple interneurons, each of which is connected to a motor unit, the muscle fibers connected to the motor neurons included in the motor unit are contracted to move the muscle, thereby achieving the target motion of the muscle model. An operation setting step for setting may be provided. The learning execution method includes a learning execution step of learning a firing pattern that realizes a target motion by performing learning that rewards a firing pattern, among a plurality of firing patterns, in which the motion of the muscle model is closer to the target motion. good.

According to a fifth aspect of the present invention, there is provided a computer-implemented learning execution method. In the learning execution method, for each of a plurality of muscle fibers included in a muscle, a first parameter indicating whether the muscle fiber is fast-twitch or slow-twitch, a second parameter indicating energy that can be contracted, A storage step may be provided for storing a maximum value of the second parameter, a third parameter indicative of self-healing power, and the maximum value of the third parameter. The learning execution method includes a learning execution step of learning a muscle model by executing learning using a first parameter, a second parameter, a maximum value of the second parameter, a third parameter, and a maximum value of the third parameter. may be provided.

It should be noted that the above summary of the invention does not list all the necessary features of the invention. Subcombinations of these feature groups can also be inventions.

An example of the learning execution device 100 is shown schematically. An example of a muscle model 300 is shown schematically. An example firing pattern 400 is schematically shown. An example of the functional configuration of the learning execution device 100 is shown schematically. A specific example of a muscle model 300 is shown schematically. 1 schematically shows an example of a hardware configuration of a computer 1200 functioning as a learning execution device 100. FIG.

Hereinafter, the present invention will be described through embodiments of the invention, but the following embodiments do not limit the invention according to the claims. Also, not all combinations of features described in the embodiments are essential for the solution of the invention.

FIG. 1 schematically shows an example of a learning execution device 100. As shown in FIG. The learning execution device 100 performs learning for causing a muscle model, which is a model of muscle motion, to perform a target motion.

A muscle model corresponds, for example, to some muscles of a person. A muscle model may correspond to all muscles of a person. The muscle model is not limited to humans, and may correspond to any organism having muscles. Also, the muscle model may correspond to a CG character or the like.

For example, the learning execution device 100 according to the present embodiment contracts muscle fibers connected to motor neurons included in motor units according to firing patterns of a plurality of interneurons each connected to a motor unit, thereby contracting muscles. Stores the muscle model to operate. Interneurons are sometimes called interneurons. Motor neurons are sometimes called motor neurons. Motor units are sometimes called motor units.

The learning execution device 100 learns a firing pattern that allows a muscle model to achieve a target motion. The learning execution device 100 learns firing patterns by, for example, performing learning that rewards firing patterns in which the motion of the muscle model is close to the target motion among a plurality of randomly generated firing patterns.

Hill-type models, CPG, and the like are known as simulation methods based on conventional muscle contraction. In the conventional method, the parameters were set manually to simulate motion. In the conventional method, it was necessary to manually set the parameters when trying to make the muscle model perform different motions. On the other hand, according to the learning execution device 100 according to the present embodiment, it is possible to automatically learn a firing pattern that can realize a target motion, so it is possible to eliminate the need to set parameters individually for each type of motion. .

The learning execution device 100 may cause the muscles of the muscle model to grow when the muscle model is operated based on the firing pattern while learning is progressing. For example, when the muscle model is operated based on the firing pattern, the learning execution device 100 grows motor units that contract muscle fibers. In the conventional method, depending on the parameter settings, there were cases where movements different from the actual movements of the muscles were realized. On the other hand, the learning execution device 100 according to the present embodiment can realize more realistic movements by also considering muscle growth.

The learning execution device 100 may be applied to various fields. The learning execution device 100, for example, learns a firing pattern that causes a CG character to perform an arbitrary action, and generates a CG animation of the character performing the arbitrary action.

Conventionally, it was necessary to create an animation for a character to perform an arbitrary motion. By learning the firing patterns of interneurons to perform, it is possible to generate CG animation of characters that automatically perform arbitrary actions. For example, if a dance motion is set as a target motion, a CG animation of a character performing the dance can be automatically generated. According to the learning execution device 100 according to the present embodiment, it is possible to realize realistic movements by learning firing patterns from interneurons and realizing the same control system movements as those of actual living things.

Further, in the conventional technique, for example, when a character with a three-headed body is to perform a dance movement of an eight-headed human, there are cases where the movements do not correspond to each other, resulting in an unnatural movement. rice field. On the other hand, according to the learning execution device 100 according to the present embodiment, by executing learning in consideration of the muscle structure and growth of the three-headed character, natural movements are realized for the three-headed character. can be made

The learning execution device 100 displays, for example, the generated CG animation on a display provided in the learning execution device 100. FIG. Further, the learning execution device 100 may cause the communication terminal 200 to display the generated CG animation by transmitting it to the communication terminal 200 via the network 20, for example.

The communication terminal 200 may be a PC (Personal Computer), a tablet terminal, a smart phone, or the like. Learning execution device 100 and communication terminal 200 may communicate via network 20 . Network 20 may include the Internet. The network 20 may include a LAN (Local Area Network). Network 20 may include a mobile communication network. The mobile communication network complies with any of the 3G (3rd Generation) communication method, the LTE (Long Term Evolution) communication method, the 5G (5th Generation) communication method, and the communication method after the 6G (6th Generation) communication method. good too.

Also, the learning execution device 100 may be applied to, for example, the field of rehabilitation. The learning execution device 100 registers, for example, a muscle model of a person who performs walking rehabilitation, and also registers walking as a target motion. Then, the learning of the interneuron firing pattern is advanced, and the movements and muscle growth until the animal is able to walk are recorded. As a result, it is possible to search for an appropriate motion until walking becomes possible.

Also, the learning execution device 100 may be applied to, for example, the field of sports science. The learning execution device 100 registers, for example, a muscle model of an athlete, and also registers an ideal form or the like as a target motion. Then, the learning of the interneuron firing pattern is advanced, and the movement and muscle growth until the ideal form is acquired are recorded. This allows you to explore training methods.

Note that learning execution device 100 may apply muscle growth to a muscle model other than a muscle model that operates muscles according to the firing pattern of interneurons. For example, the learning execution device 100 applies muscle growth to a muscle model based on a Hill-type model. Also, for example, the learning execution device 100 applies muscle growth to a muscle model using CPG. Also, for example, the learning execution device 100 applies muscle growth to a muscle model using DQN. The learning execution device 100 may also apply muscle growth to any existing model.

FIG. 2 schematically shows an example of muscle model 300 . Muscle model 300 includes a plurality of interneurons 320 within spinal cord 310 and a plurality of motor units 330 connected to each of the plurality of interneurons 320 . A single motor unit 330 includes a motor neuron 340 and a muscle fiber 350 connected to the motor neuron 340 . A plurality of muscle fibers 350 are connected to one motor neuron 340 .

FIG. 3 schematically shows an example firing pattern 400 . Firing pattern 400 shows a time series of on 402 and off 404 of a plurality of interneurons 320 . By applying firing pattern 400 to muscle model 300 , signals are input to each motor unit 330 in time series from interneuron 320 , and muscle fibers 350 of motor unit 330 contract according to ON 402 . As a result, various muscle movements are realized.

FIG. 4 schematically shows an example of the functional configuration of the learning execution device 100. As shown in FIG. The learning execution device 100 includes an information storage unit 102 , an input reception unit 104 , a data reception unit 106 , an operation setting unit 108 , a learning execution unit 110 and a display control unit 112 .

The information storage unit 102 stores various information. The information storage unit 102 may store muscle models. The information storage unit 102 operates muscles by contracting muscle fibers 350 connected to motor neurons 340 included in the motor units 330 according to firing patterns of a plurality of interneurons 320 each connected to the motor units 330. A muscle model may be stored.

The information storage unit 102 may store relevant parameters for each of the plurality of motor units 330 included in the muscle model 300 . The information storage unit 102 may store a type parameter indicating whether the motor unit 330 is fast-twitch or slow-twitch. A type parameter may be an example of a first parameter.

The information storage unit 102 may store HP, which is a parameter indicating contractile energy. HP may be an example of the second parameter. The information storage unit 102 may store MAXHP indicating the maximum value of HP.

The information storage unit 102 may store MP, which is a parameter indicating self-resilience. MP may be an example of the third parameter. The information storage unit 102 may store MAXMP indicating the maximum value of MP.

The information storage unit 102 may store a fourth parameter indicating the amount of energy consumed each time the muscle fiber 350 contracts when the muscle fiber 350 is fast-twitch. In this example, the information storage unit 102 stores DIAM indicating the diameter of the muscle fiber 350, which is an example of the fourth parameter. Information store 102 may store EXP, a parameter associated with the use of motor unit 330 . EXP, for example, may be a parameter that increases each time motor unit 330 is used. EXP, for example, may be a parameter related to the number of times motor unit 330 has been used. EXP may simply be the number of times motor unit 330 is used. EXP may be an example of a fifth parameter.

The input reception unit 104 receives various inputs. The input reception unit 104 may receive input via an input device included in the learning execution device 100 .

A data receiving unit 106 receives various data via the network 20 . The data receiving unit 106 receives, for example, the muscle model 300 from the communication terminal 200 and stores it in the information storage unit 102 . The data receiving unit 106 also receives parameters of the motor unit 330 from the communication terminal 200 and stores them in the information storage unit 102, for example.

A motion setting unit 108 sets a target motion of the muscle model. The motion setting unit 108 may set the target motion of the muscle model 300 according to the input received by the input receiving unit 104, for example. The motion setting section 108 may set the target motion of the muscle model 300 according to the setting instruction received by the data receiving section 106 from the communication terminal 200 .

The learning executing unit 110 executes learning. The learning execution unit 110 may learn firing patterns. The learning execution unit 110 may learn the firing pattern that realizes the target motion by learning that rewards the firing pattern in which the motion of the muscle model 300 is closer to the target motion among the multiple firing patterns. The learning execution unit 110 uses, for example, reinforcement learning. The learning execution unit 110 may use DQN (Deep Q-Network). The learning execution unit 110 may use GA (Genetic Algorithm). The learning execution unit 110 may use any other learning method.

For example, when learning an firing pattern for realizing a certain target action, the learning execution unit 110 first randomly generates a plurality of firing patterns. The learning execution unit 110 operates the muscle model 300 according to each of a plurality of randomly generated firing patterns, and generates a plurality of firing patterns based on the firing pattern in which the motion of the muscle model 300 is closer to the target motion. The learning execution unit 110 operates the muscle model 300 according to each of the generated firing patterns, and generates the firing patterns based on the firing pattern in which the motion of the muscle model 300 is closer to the target motion. The learning execution unit 110 may learn the firing pattern that realizes the target motion by repeating these steps.

The learning executing unit 110 may generate a plurality of firing patterns based on learned firing patterns. For example, the learning execution unit 110 learns the firing pattern for the target motion of keeping the knee bent at 20 degrees, the firing pattern learned for the target motion of keeping the knee bent at 60 degrees, A plurality of firing patterns are generated based on the firing patterns learned for the target motion of keeping the knee bent at 90 degrees. As a result, for example, it is possible to easily prepare a plurality of firing patterns for the target motion of keeping the knee bent at an arbitrary angle, and compared with the case of randomly generating the firing patterns, the total cost is reduced. time can be shortened.

During learning, the learning execution unit 110 may cause the motor unit 330 that contracts the muscle fiber 350 to grow when the muscle model 300 is operated based on the firing pattern.

The muscle model 300 may include a fast-twitch motor unit 330 and a slow-twitch motor unit 330 . The learning execution unit 110 may cause the fast-twitch motor unit 330 and the slow-twitch motor unit 330 to grow according to different criteria when the muscle model 300 is operated based on the firing pattern.

The learning execution unit 110 may subtract a predetermined value from the HP each time the motor unit 330 contracts, and may restore the HP over time while the MP is not 0. If the motor unit 330 is a fast-twitch muscle, the learning execution unit 110 may subtract DIAM from HP each time the motor unit 330 contracts. If the motor unit 330 is a slow-twitch muscle, the learning execution unit 110 may subtract 1 from HP each time the motor unit 330 contracts. Note that the learning execution unit 110 may also subtract a value other than DIAM from HP each time the motor unit 330 contracts when the motor unit 330 is a fast-twitch muscle. Further, when the motor unit 330 is a slow-twitch muscle, the learning execution unit 110 may subtract a value other than 1, such as the value of DIAM, from HP each time the motor unit 330 contracts. The learning execution unit 110 may restore HP over time while the MP is not 0. The learning execution unit 110 does not need to recover HP when MP becomes 0. The learning execution unit 110 may recover MP over time.

When the learning execution unit 110 determines that the muscle fiber 350 has recovered after determining that the muscle fiber 350 has been damaged, if the motor unit 330 is a fast-twitch muscle, the learning execution unit 110 increases MAXHP and DIAM to restore the motor unit. MAXMP may be increased if 330 is slow twitch.

When the learning execution unit 110 determines that the muscle fiber 350 has recovered after determining that the muscle fiber 350 has been damaged, and if the motor unit 330 is a fast-twitch muscle, MAXMP does not need to be increased. When the learning execution unit 110 determines that the muscle fiber 350 has recovered after determining that the muscle fiber 350 has been damaged and the motor unit 330 is a slow twitch muscle, MAXHP does not need to be increased. For example, the learning execution unit 110 may determine that the muscle fiber 350 is damaged when the HP becomes 0, and when the HP becomes MAXHP or when the HP becomes higher than a predetermined threshold, It may be determined that the muscle fiber 350 has recovered.

The learning execution unit 110 may improve the level of the motor unit 330 as the EXP increases. For example, the learning execution unit 110 registers an EXP value determined for each level, and improves the level of the motor unit 330 when the EXP value exceeds the EXP value corresponding to the level. More EXP values may be registered for higher levels.

The higher the level of the motor unit 330, the learning execution unit 110 makes it more difficult to increase MAXHP and DIAM when the motor unit 330 is fast-twitch, and makes it more difficult to increase MAXMP when the motor unit 330 is slow-twitch. you can

After contracting the muscle fiber 350 of the motor unit 330, the learning execution unit 110 may prevent the muscle fiber 350 from contracting until a predetermined refractory period elapses. The information storage unit 102 may store the temperature of each of the plurality of motor units 330 . The learning execution unit 110 may increase the temperature of the motor unit 330 as the motor unit 330 is used, and may decrease the temperature of the motor unit 330 over time if the motor unit 330 is not used. good. The learning execution unit 110 may shorten the refractory period as the temperature of the motor unit 330 is higher.

When the motion of the muscle model that is operated according to a plurality of time-series firing patterns achieves the target motion, the learning execution unit 110 performs firing in a state that is a predetermined time before the firing pattern in the state in which the target motion was achieved. Learning may be performed by updating the pattern. Since there are various time delays, such as refractory periods and the law of inertia, from the time the firing pattern is generated until the muscle actually moves, it may not be desirable to update the firing pattern at the instant of reward. be. On the other hand, according to the learning execution unit 110, since the firing pattern in the state that is a predetermined time before the firing pattern in the state in which the target motion is achieved is updated, the learning accuracy can be improved.

The predetermined time may be arbitrarily set or changeable. The learning execution unit 110 may use different times for fast-twitch and slow-twitch. For example, if the motor unit 330 is a fast-twitch muscle, the learning execution unit 110 updates the firing pattern in the state 20 ms before the target motion was achieved. The firing pattern in the state 40 ms before the firing pattern in the state of achieving the action may be updated.

The learning execution unit 110 may generate display data using the learned firing pattern. The learning execution unit 110, for example, generates a CG animation in which an arbitrary character is moved according to the firing pattern. The learning execution unit 110 may generate display data that displays data related to the movement of the muscle model 300 and muscle growth from the start of learning of the muscle model 300 until the target movement is achieved. The learning execution unit 110 may generate display data that displays data related to the movement of the muscle model 300 and muscle growth from the start of learning of the muscle model 300 until the ideal form is achieved. good.

The display control unit 112 controls various displays related to learning results by the learning execution unit 110 . The display control unit 112 causes, for example, the display data generated by the learning execution unit 110 to be displayed on the display provided in the learning execution device 100 . The display control unit 112 may transmit the display data generated by the learning execution unit 110 to the communication terminal 200 via the network 20 and display it on the display provided in the communication terminal 200 .

The information storage unit 102 may store muscle models according to known models. The information storage unit 102 stores, for example, a muscle model based on a Hill type model. The information storage unit 102 may store a muscle model using CPG. The information storage unit 102 may store a muscle model using DQN.

The information storage unit 102 stores a type parameter, HP, MAXHP, MP, MAXMP, and DIAM for each of a plurality of muscle fibers included in a muscle of a muscle model according to a known model. you can

Each time the muscle fiber contracts, the learning execution unit 110 subtracts DIAM from HP if the muscle fiber is fast-twitch, subtracts 1 from HP if the muscle fiber is slow-twitch, and MP is not 0. During the period, HP is recovered over time, and after determining that the muscle fiber is damaged, when the muscle fiber is determined to have recovered, MAXHP and DIAM are increased if the muscle fiber is fast-twitch. and if the muscle fiber is slow twitch, MAXMP may be increased.

The learning execution unit 110 does not need to recover HP when MP becomes 0. The learning execution unit 110 may recover MP over time. Learning execution unit 110 does not need to increase MAXMP if the muscle fiber is a fast-twitch muscle fiber when HP has recovered after determining that the muscle fiber is damaged. When the learning execution unit 110 recovers HP after determining that the muscle fiber is damaged, if the muscle fiber is slow twitch, MAXHP does not need to be increased.

The learning execution unit 110 may improve the level of muscle fibers as the EXP increases. For example, the learning execution unit 110 registers an EXP value determined for each level, and improves the muscle fiber level when the EXP value exceeds the EXP value corresponding to the level. More EXP values may be registered for higher levels. The higher the muscle fiber level, the more difficult the learning execution unit 110 increases the values of MAXHP and DIAM when the muscle fiber is fast-twitch, and the more difficult it is to increase MAXMP when the muscle fiber is slow-twitch. good.

After contracting the muscle fiber, the learning execution unit 110 may prevent the muscle fiber from contracting until a predetermined refractory period elapses. The information storage unit 102 may store the temperature of each of a plurality of muscle fibers. The learning execution unit 110 may increase the temperature of the muscle fiber as the muscle fiber is used, and may decrease the temperature of the muscle fiber over time if the muscle fiber is not used. The learning execution unit 110 may shorten the refractory period as the temperature of the muscle fibers increases.

FIG. 5 schematically shows a concrete example of muscle model 300 . FIG. 5 illustrates a muscle model 300 of muscles 360 corresponding to tendons 372, knees 374, and bones 376 of a human. As noted above, within spinal cord 310 are a plurality of interneurons 320 . The spinal cord 310 can also be viewed as a learner. Each of the plurality of interneurons 320 can have two states: firing and non-firing. Motor unit 330 includes motor neurons 340 and muscle fibers 350 connected to motor neurons 340 . A plurality of muscle fibers 350 are connected to one motor neuron 340 . Motor neurons 340 may be of two types: fast-twitch and slow-twitch. Motor neurons 340 may be fast-twitch if large in size and slow-twitch if small in size. A motor neuron 340 is, for example, fast-twitch if its size is greater than a threshold and slow-twitch if its size is less than the threshold. Muscle fibers 350 may be of two types, fast-twitch fibers and slow-twitch fibers. A muscle 360 is a collection of muscle fibers 350 . In this example, the learning execution unit 110 controls a simple motion with a firing pattern for a knee joint to which two muscles (extensor and flexor) are connected as one model. There may be two types of motor units 330, fast-twitch and slow-twitch, and the learning execution unit 110 may cause the fast-twitch and slow-twitch to grow differently.

In order to use firing patterns to control muscles, it is necessary to calculate neuronal action potentials. When the learning execution unit 110 fires the interneuron 320, for example, the Hodgkin-Huxley model (A.L. Hodgkin, A. A quantitative description of membrane current and its application to conduction and excitation in nerve, from the physiological laboratory. University of Cambridge, pp. 500-544, 1952.) Action potentials may be calculated. The calculated action potentials are distributed to the connected motor neurons 340 according to Kirchhoff's laws.

The learning execution unit 110 may use an extended Hill-type model, and may use the sum of the action potentials of the firing motor neurons 340 as the input signal for the muscle model. In the muscle model, the contractile force of the muscle is calculated, and according to the laws of physics, the contractile force of the muscle is converted into the torque of the knee joint, and the knee is moved to change the joint angle. The learning execution unit 110 may output the exercise result as a joint angle. The learning executor 110 may reward the firing pattern when the joint angle achieves the target angle.

As noted above, the learning executor 110 may use an extended Hill-type model. A conventional Hill-type model consists of a muscle contractile element (CE), a parallel elastic element (PEE) arranged parallel to the CE, and a series elastic element (SEE) arranged in series. The extended model adds a damping factor to the tendon force calculation in the traditional Hill-type model to mitigate muscle spasm due to spring constant. In the Hill-type model, muscle fibers acquire current from motor neurons and convert it to force using PEE, SEE, and CE.

lopt is the length optimized for maximal force of the CE, A is the muscle activity ratio, and lce is the length of the CE. Several equations have been proposed to approximate this function. For example, the Rosen and Kuo model (Deshpande, P.-H. K. A. D. Contribution of passive properties of muscle-tendon units to the metacarpophalangeal joint torque of the index finger. IEEE, pp. 288-294, 2010.) may be applied. .

Vce is the contraction velocity of CE and Vmax is the maximum contraction velocity of CE. The formula for Fpe, which is the force generated by PE, is as follows.

Kpe is the spring constant of the PE, lpe is the length of the PE, lpe_rest is the equilibrium length of the PE, dpe is the damping coefficient of the PE, and Vpe is the terminal velocity of the PE. The formula for Fse, the force of SEE, is:

kse is the spring constant of the SEE, lse is the length of the SEE, lse_rest is the equilibrium length of the SEE, dse is the damping coefficient of the SEE, and Vse is the terminal velocity of the SEE.

When a motor unit 330 receives an action potential, it causes muscle contraction. The time it takes for the contraction to reach the peak force is called the contraction time. The slow-twitch motor unit 330 has a longer contraction time and a smaller maximum contraction force. A fast-twitch motor unit 330 has a short contraction time and a high peak contraction force. A single muscle is composed of multiple fast-twitch motor units 330 and multiple slow-twitch and motor units 330 . Therefore, a Hill-type model consisting of these motor units 330 is adopted.

N is the number of fast-twitch motor units 330, M is the number of slow-twitch motor units 330, Fce_f_i is the contractile force of the i-th fast-twitch muscle, and Fce_s_j is the contractile force of the j-th slow-twitch muscle. is.

The biological properties of the slow-twitch motor unit 330 and the fast-twitch motor unit 330 are modeled by the muscle model according to this embodiment. In the growth model of a motor unit 330 composed of motor neurons 340 and muscle fibers 350, every muscle fiber 350 has an energy value (HP) available for muscle contraction. With each contraction, the value of fast-twitch HP may be decreased by a value equal to the diameter of muscle fiber 350 . Also, the value of the slow twitch HP may be decreased by one with each contraction. Continuous muscle contraction reduces HP, and when HP reaches 0, muscle tear occurs. Once a muscle rupture occurs, the muscle fiber 350 cannot contract again, even if an electrical signal is received from the interneuron 320 without recovery. On the other hand, if the interval between signals from interneurons 320 is large enough, muscle fibers 350 can heal spontaneously. In this model, the muscle fibers 350 recover over time as long as the MP indicating the self-healing power is not 0. With these, the learning execution unit 110 can learn various firing patterns.

A motor unit 330 gains EXP each time it is used and promotes growth. In this model, we define LV to represent the level of growth. Slow-twitch motor units and fast-twitch motor units have different growth rules. For fast-twitch motor units 330, the MAXHP and muscle fiber 350 diameter parameters are increased. The rules are based on biological growth rules.

Fast-twitch motor units 330 have satellite cells surrounding the muscle fibers. When a muscle tear occurs, the satellite cells divide and the size of the fast-twitch muscle fiber 350 increases. Thicker muscle fibers 350 are stronger, but require more HP.

Slow-twitch muscle fibers 350 do not increase in size, but increase in self-healing power. According to the biological growth rule, the number of capillaries surrounding the slow-twitch muscle fibers 350 increases, thus increasing the amount of oxygen transported to the slow-twitch muscle fibers 350 .

This model may further include SP, which is a parameter indicating fatigue of the muscle fibers 350 . Only in slow-twitch muscle fibers 350, the SP value decreases with growth. That is, slow-twitch muscle fibers can be used longer.

Table 1 shows a description of each parameter and Table 2 shows an example of the algorithm.

Learning executor 110 may use Q-learning to learn firing patterns of interneurons 320 . The learning process may be based on multi-agent system learning with each interneuron 320 as an agent. Each agent monitors its environment. The environment may be defined as the connectivity of interneurons 320 and motor neurons 340 and the parameters of motor units 330 . During learning, initial connection settings are not changed, but motor unit parameters may be changeable.

Each interneuron 320 is connected to a plurality of motor neurons 340 and connects to either fast or slow twitch muscles. Whether the muscle fiber 350 of the motor unit 330 is fast-twitch or slow-twitch depends on the size of the connected motor neuron. This principle comes from biology. Agents can share state information among themselves. This is equivalent to information sharing through myelin connections.

The combination of state and action in Q-learning is Qi=(si:ai), where Si indicates the state of each agent.

M is the sum of motor neurons 340 connected to an interneuron 320 and O is the sum of motor neurons 340 connected to other interneurons 320 . Each agent performs an action ai such as fire (1) or not fire (0).

Each interneuron 320 monitors all the parameters of each motor unit to which it is connected, as well as the sum of the motor unit energies held by other interneurons 320 with which it shares information, and determines whether to fire. decide. Based on the firing of interneurons 320, using the Hodgkin Huxley model, the electrical signals of the connected motor units are computed and used to compute the input signal. The angles calculated from muscle contraction are then fed back to the agent using an extended Hill-type model.

Rewards may be of two types: immediate and delayed. As an immediate reward, receive an rgoal each time the knee joint achieves the target angle. As long as the knee joint continues to achieve the target angle, the agent continues to receive the reward.

As a delayed reward, the total remaining HP of all agents is distributed equally to all agents as a reward at the end of the episode. This contributes to coordinated behavior that produces efficient movement.

FIG. 6 schematically shows an example of a hardware configuration of a computer 1200 functioning as the learning execution device 100. As shown in FIG. Programs installed on the computer 1200 cause the computer 1200 to function as one or more "parts" of the apparatus of the present embodiments, or cause the computer 1200 to operate or perform operations associated with the apparatus of the present invention. Multiple "units" can be executed and/or the computer 1200 can be caused to execute the process or steps of the process according to the present invention. Such programs may be executed by CPU 1212 to cause computer 1200 to perform certain operations associated with some or all of the blocks in the flowcharts and block diagrams described herein.

Computer 1200 according to this embodiment includes CPU 1212 , RAM 1214 , and graphics controller 1216 , which are interconnected by host controller 1210 . Computer 1200 also includes input/output units such as communication interface 1222 , storage device 1224 , DVD drive, and IC card drive, which are connected to host controller 1210 via input/output controller 1220 . The DVD drive may be a DVD-ROM drive, a DVD-RAM drive, and the like. Storage devices 1224 may be hard disk drives, solid state drives, and the like. Computer 1200 also includes legacy input/output units, such as ROM 1230 and keyboard, which are connected to input/output controller 1220 via input/output chip 1240 .

The CPU 1212 operates according to programs stored in the ROM 1230 and RAM 1214, thereby controlling each unit. Graphics controller 1216 retrieves image data generated by CPU 1212 into a frame buffer or the like provided in RAM 1214 or itself, and causes the image data to be displayed on display device 1218 .

Communication interface 1222 communicates with other electronic devices over a network. Storage device 1224 stores programs and data used by CPU 1212 within computer 1200 . The DVD drive reads programs or data from a DVD-ROM or the like and provides them to the storage device 1224 . The IC card drive reads programs and data from IC cards and/or writes programs and data to IC cards.

ROM 1230 stores therein programs that are dependent on the hardware of computer 1200, such as a boot program that is executed by computer 1200 upon activation. Input/output chip 1240 may also connect various input/output units to input/output controller 1220 via USB ports, parallel ports, serial ports, keyboard ports, mouse ports, and the like.

The program is provided by a computer-readable storage medium such as DVD-ROM or IC card. The program is read from a computer-readable storage medium, installed in storage device 1224 , RAM 1214 , or ROM 1230 , which are also examples of computer-readable storage media, and executed by CPU 1212 . The information processing described within these programs is read by computer 1200 to provide coordination between the programs and the various types of hardware resources described above. An apparatus or method may be configured by implementing information operations or processing according to the use of computer 1200 .

For example, when communication is performed between the computer 1200 and an external device, the CPU 1212 executes a communication program loaded into the RAM 1214 and sends communication processing to the communication interface 1222 based on the processing described in the communication program. you can command. The communication interface 1222 reads transmission data stored in a transmission buffer area provided in a recording medium such as a RAM 1214, a storage device 1224, a DVD-ROM, or an IC card under the control of the CPU 1212, and transmits the read transmission data. Data is transmitted to the network, or received data received from the network is written in a receive buffer area or the like provided on the recording medium.

In addition, the CPU 1212 causes the RAM 1214 to read all or necessary portions of files or databases stored in an external recording medium such as a storage device 1224, a DVD drive (DVD-ROM), an IC card, etc. Various types of processing may be performed on the data. CPU 1212 may then write back the processed data to an external recording medium.

Various types of information, such as various types of programs, data, tables, and databases, may be stored on recording media and subjected to information processing. CPU 1212 performs various types of operations on data read from RAM 1214, information processing, conditional decisions, conditional branching, unconditional branching, and information retrieval, which are described throughout this disclosure and are specified by instruction sequences of programs. Various types of processing may be performed, including /replace, etc., and the results written back to RAM 1214 . In addition, the CPU 1212 may search for information in a file in a recording medium, a database, or the like. For example, when a plurality of entries each having an attribute value of a first attribute associated with an attribute value of a second attribute are stored in the recording medium, the CPU 1212 selects the first attribute from among the plurality of entries. search for an entry that matches the specified condition of the attribute value of the attribute, read the attribute value of the second attribute stored in the entry, and thereby determine the first attribute that satisfies the predetermined condition An attribute value of the associated second attribute may be obtained.

The programs or software modules described above may be stored in a computer-readable storage medium on or near computer 1200 . Also, a recording medium such as a hard disk or RAM provided in a server system connected to a dedicated communication network or the Internet can be used as a computer-readable storage medium, whereby the program can be transferred to the computer 1200 via the network. offer.

The blocks in the flowcharts and block diagrams in this embodiment may represent steps in the process in which the operations are performed or "parts" of the apparatus responsible for performing the operations. Certain steps and "sections" may be provided with dedicated circuitry, programmable circuitry provided with computer readable instructions stored on a computer readable storage medium, and/or computer readable instructions provided with computer readable instructions stored on a computer readable storage medium. It may be implemented by a processor. Dedicated circuitry may include digital and/or analog hardware circuitry, and may include integrated circuits (ICs) and/or discrete circuitry. Programmable circuits, such as Field Programmable Gate Arrays (FPGAs), Programmable Logic Arrays (PLAs), etc., perform AND, OR, EXCLUSIVE OR, NOT AND, NOT OR, and other logical operations. , flip-flops, registers, and memory elements.

A computer-readable storage medium may comprise any tangible device capable of storing instructions to be executed by a suitable device, such that a computer-readable storage medium having instructions stored thereon may be illustrated in flowchart or block diagram form. It will comprise an article of manufacture containing instructions that can be executed to create means for performing specified operations. Examples of computer-readable storage media may include electronic storage media, magnetic storage media, optical storage media, electromagnetic storage media, semiconductor storage media, and the like. More specific examples of computer readable storage media include floppy disks, diskettes, hard disks, random access memory (RAM), read only memory (ROM), erasable programmable read only memory (EPROM or flash memory) , electrically erasable programmable read only memory (EEPROM), static random access memory (SRAM), compact disc read only memory (CD-ROM), digital versatile disc (DVD), Blu-ray disc, memory stick , integrated circuit cards, and the like.

The computer readable instructions may be assembler instructions, Instruction Set Architecture (ISA) instructions, machine instructions, machine dependent instructions, microcode, firmware instructions, state setting data, or object oriented programming such as Smalltalk, JAVA, C++, etc. language, and any combination of one or more programming languages, including conventional procedural programming languages, such as the "C" programming language or similar programming languages. good.

Computer readable instructions are used to produce means for a processor of a general purpose computer, special purpose computer, or other programmable data processing apparatus, or programmable circuits to perform the operations specified in the flowchart or block diagrams. A general purpose computer, special purpose computer, or other programmable data processor, locally or over a wide area network (WAN) such as the Internet, etc., to execute such computer readable instructions. It may be provided in the processor of the device or in a programmable circuit. Examples of processors include computer processors, processing units, microprocessors, digital signal processors, controllers, microcontrollers, and the like.

Although the present invention has been described above using the embodiments, the technical scope of the present invention is not limited to the scope described in the above embodiments. It is obvious to those skilled in the art that various modifications or improvements can be made to the above embodiments. It is clear from the description of the scope of the claims that forms with such modifications or improvements can also be included in the technical scope of the present invention.

The execution order of each process such as actions, procedures, steps, and stages in the devices, systems, programs, and methods shown in the claims, the specification, and the drawings is etc., and it should be noted that they can be implemented in any order unless the output of the previous process is used in the subsequent process. Regarding the operation flow in the claims, the specification, and the drawings, even if the description is made using "first," "next," etc. for convenience, it means that it is essential to carry out in this order. not a thing

Although the present invention has been described above using the embodiments, the technical scope of the present invention is not limited to the scope described in the above embodiments. It is obvious to those skilled in the art that various modifications or improvements can be made to the above embodiments. It is clear from the description of the scope of the claims that forms with such modifications or improvements can also be included in the technical scope of the present invention.

The execution order of each process such as actions, procedures, steps, and stages in the devices, systems, programs, and methods shown in the claims, the specification, and the drawings is etc., and it should be noted that they can be implemented in any order unless the output of the previous process is used in the subsequent process. Regarding the operation flow in the claims, the specification, and the drawings, even if the description is made using "first," "next," etc. for convenience, it means that it is essential to carry out in this order. not a thing

20 network, 100 learning execution device, 102 information storage unit, 104 input reception unit, 106 data reception unit, 108 operation setting unit, 110 learning execution unit, 112 display control unit, 200 communication terminal, 300 muscle model, 310 spinal cord, 320 interneurons, 330 motor units, 340 motor neurons, 350 muscle fibers, 360 muscles, 372 tendons, 374 knees, 376 bones, 400 firing patterns, 402 on, 404 off, 1200 computer, 1210 host controller, 1212 CPU, 1214 RAM, 1216 graphic controller, 1218 display device, 1220 input/output controller, 1222 communication interface, 1224 storage device, 1230 ROM, 1240 input/output chip

Claims (7)

an information storage unit that stores a muscle model that operates a muscle by contracting muscle fibers connected to motor neurons included in the motor unit according to the firing pattern of a plurality of interneurons, each of which is connected to a motor unit;
a motion setting unit that sets a target motion of the muscle model;
A learning execution unit that learns the firing pattern, and realizes the target motion by executing learning that rewards the firing pattern, among a plurality of firing patterns, in which the motion of the muscle model is closer to the target motion. a learning execution unit that learns firing patterns;
The learning execution unit executes multi-agent system learning using the plurality of interneurons as agents,
Learning execution device. In the learning execution unit, each of the plurality of interneurons calculates the sum of all parameters of each connected motor unit and the energy of motor units held by other interneurons sharing information. 2. The learning execution device of claim 1, which executes the multi-agent system learning to monitor and determine whether to fire. The learning execution unit operates the muscle model according to each of the plurality of firing patterns, generates a plurality of firing patterns based on the firing pattern in which the motion of the muscle model is closer to the target motion, and generates a plurality of firing patterns. and learning a firing pattern that realizes the target motion by repeatedly generating a plurality of firing patterns based on the firing pattern in which the motion of the muscle model is closer to the target motion. 3. The learning execution device according to claim 1 or 2 . The learning execution unit operates the muscle model according to each of the plurality of randomly generated firing patterns, generates a plurality of firing patterns based on the firing pattern in which the motion of the muscle model is closer to the target motion, The target motion is realized by operating the muscle model according to each of the plurality of firing patterns and repeatedly generating a plurality of firing patterns based on the firing pattern in which the motion of the muscle model is closer to the target motion. 4. The learning execution device according to claim 3 , which learns firing patterns. The learning execution unit operates the muscle model according to each of the plurality of firing patterns generated based on the learned firing pattern, and operates the muscle model to perform a plurality of firing patterns based on the firing pattern closer to the target motion. Generating a firing pattern, operating the muscle model according to each of the plurality of firing patterns, and repeatedly generating a plurality of firing patterns based on the firing pattern in which the motion of the muscle model is closer to the target motion, 4. The learning execution device according to claim 3 , which learns firing patterns for realizing said target motion. A program for causing a computer to function as the learning execution device according to any one of claims 1 to 5 . A computer implemented learning method comprising:
Motion setting for setting a target motion of a muscle model to move a muscle by contracting a muscle fiber connected to a motor neuron included in the motor unit according to the firing pattern of a plurality of interneurons each connected to the motor unit. a step;
a learning execution step of learning a firing pattern that realizes the target motion by executing learning that rewards a firing pattern, from among a plurality of firing patterns, in which the motion of the muscle model is closer to the target motion;
The learning execution step executes multi-agent system learning using the plurality of interneurons as agents.
learning execution method.

Images