JP6431785B2 - Psychological state feedback method, system, and program - Google Patents

Description

  The present invention relates to a technology for feeding back a psychological state for improving exercise.

  In order to properly acquire basic sports activities such as pitching in baseball, swinging and putting in golf, putting, etc., `` variation '' and `` rhythm '' that are the points of exercise are visualized and audible with images and sounds, and exercise A technique for easily grasping the knack has been proposed (for example, Non-Patent Document 1).

Even if you can grasp the knack of exercise, it is not enough, and it is also important to maintain a better psychological state in order to perform as usual or higher. Extreme strength and tension is known to reduce performance. (For example, Non-Patent Document 2)
There are also known methods for quantifying tension / relaxation, for example by calculating the low frequency component (LF: 0.05-0.15 Hz) and the high frequency component (HF: 0.15-0.4 Hz) of heart rate variability. , HF is evaluated as relaxation degree, and LF / HF is evaluated as tension degree (for example, Non-Patent Document 3).

http://www.kecl.ntt.co.jp/openhouse/2014/exhibition/25/index.html Psychophysiology, 47 (2010), 1109-1118. Wiley Periodicals, Inc. “Psychological, muscular and kinematic factors mediate performance under pressure” September 2014 | Volume5 | Article1040 | 1-19 “Ahealthy heart is not a metronome: an integrative review of the heart ’s anatomy and heart rate variability” (http://www.frontiersin.org/)

However, it is difficult to determine what psychological state is preferable, and a mechanism for determining whether the current psychological state is preferable or not is desired.
The present invention has been made by paying attention to the above circumstances, and its purpose is to provide a psychological state feedback method, system for feeding back the current psychological state with respect to a reference psychological state (correct psychological state) to the subject, and To provide a program.

(1) In order to achieve the above object, the psychological state feedback method according to the embodiment determines the psychological state of the primary subject based at least on primary biological information acquired from the primary subject during a predetermined exercise. Or storing and determining the psychological state of the secondary subject based at least on the secondary biological information acquired from the secondary subject during a predetermined exercise, and the psychological state of the primary subject and the secondary subject comparing the psychological state of the subject, according to the comparison result of the psychological state of the secondary subject and psychological state of the primary subject, the secondary when it indicates that the psychological state is good The secondary information is exaggerated by correcting the biometric information to a better value, or the secondary biometric information is reduced or exaggerated or reduced by correcting to a worse value when the psychological state is not indicative of good. Know biological information visually or auditorily And it outputs a feedback signal which is converted to a format.

(2) In addition, feedback method of psychological state of (1), said primary body information and said secondary biometric information comprises at least cardiac potential data, the feedback signals, cardiac potential of the exaggerated secondary biometric information The data or the electrocardiographic data of the reduced secondary biological information is converted into an acoustic signal, an image signal, a video signal, or a combination thereof.

( 3 ) The psychological state feedback system according to the embodiment executes the psychological state feedback method described in (1) or (2) above.
( 4 ) The psychological state feedback program according to the embodiment causes the computer to execute the determination, comparison, and output described in (1) or (2) above.

  That is, according to the present invention, it is possible to provide a psychological state feedback method, system, and program for feeding back a determination result of a current psychological state with respect to a reference psychological state (correct psychological state).

  According to the psychological state feedback method of (1) above, it is substantially the same as the psychological state (primary index) based on electrocardiographic data acquired from an advanced person or a professional (reference player) during a predetermined exercise. From the comparison result (difference from correct exercise) with the psychological state (secondary index) based on the electrocardiographic data acquired from the aspiring candidates during the predetermined exercise, the psychological state of the advanced candidate and the professional psychological state (Good or bad) is judged, and feedback that is exaggerated or reduced according to the judgment result can be made to support the improvement of the movement of the improvement candidate. The same effect can be obtained by the system (4) and the program (5).

FIG. 1 is a diagram illustrating an example of a feedback system of a psychological state for improving exercise. FIG. 2 is a diagram illustrating an example of the sensing unit of the psychological state determination device. FIG. 3 is a diagram illustrating an example of a state determination unit of the psychological state determination device. FIG. 4 is a diagram illustrating an example of a feedback device of the psychological state feedback system. FIG. 5 is a flowchart illustrating an example of psychological state determination processing by the psychological state determination device. FIG. 6 is a flowchart illustrating an example of psychological state feedback processing by the feedback device. FIG. 7 is a block diagram illustrating an example of the configuration of the synthesized sound / image generation unit. FIG. 8 is a block diagram illustrating an example of the configuration of the operation timing verification unit. FIG. 9 is a block diagram illustrating another example of the configuration of the synthesized sound / image generation unit. FIG. 10 is a block diagram illustrating an example of the configuration of the audible device.

Embodiments according to the present invention will be described below with reference to the drawings.
In the present embodiment, description will be given of determination of a mental state during exercise by the psychological state determination device 130 constituting the feedback system 100 and feedback of a determination result by the feedback device 140. As an example of exercise, a baseball pitching operation will be described. The exercise using the feedback system 100 of the present embodiment is not limited to baseball, and the feedback system 100 described in the present embodiment is limited only for improving exercise and demonstrating high performance. It can also be used for rehabilitation and technology transfer.

FIG. 1 is a diagram illustrating an example of a feedback system of a psychological state for improving exercise.
The feedback system 100 includes a psychological state determination device 130 and a feedback device 140. The psychological state determination device 130 includes a sensing unit 131 and a state determination unit 132.

  Each of the psychological state determination device 130 and the feedback device 140 includes, for example, a processor (for example, hardware including one or more central processing units (CPUs)) or a memory (for example, random-access memory (RAM) or ROM (read). -only memory)), a general-purpose or dedicated computer may be configured to execute a predetermined program. This program may be installed in the computer, or may be recorded in the computer in advance. In addition, some or all of the processing units may be configured using an electronic circuit that realizes a processing function independently instead of an electronic circuit (circuitry) that realizes a functional configuration by reading a program like a CPU. . The output device 200 includes a display device such as a liquid crystal display, for example.

  The sensing unit 131 senses a cardiac potential signal from at least one cardiac potential sensor 120 attached to the primary subject and the secondary subject. The sensing unit 131 may sense a signal obtained from the chest, sense a signal obtained from the wrist instead of a signal obtained from the chest, or obtain a signal obtained from the chest and the wrist. It is also possible to sense the obtained signal and use either sensing result from the sensing results of both. The subject (operation subject) to which the electrocardiographic sensor 120 is attached is not limited to a person, and may be an animal other than a person as long as it is an operation subject. The electrocardiogram sensor 120 is composed of electrodes and the like, and each electrocardiogram sensor 120 outputs an electrocardiogram signal (biological information) corresponding to the muscle activity level. For example, the plurality of electrocardiographic sensors 120 are wearable sensors attached to the wear, and each electrocardiographic sensor 120 is attached to the wear so as to correspond to the breasts of the primary and secondary subjects. ing. As a result, the cardiac potential sensor 120 detects cardiac potential signals (such as primary biological information from the primary subject and secondary biological information from the secondary subject) from the breasts and the like of the primary and secondary subjects. ) Can be output, for example, a cardiac signal can be output over a certain period from the start of the throwing operation to the end of the pitching operation (that is, during a predetermined movement). Alternatively, the electrocardiographic sensor 120 may be attached to a wristband, a hair band, or the like, and a heart signal corresponding to the mounting position of the wrist band, the hair band, or the like may be acquired. The primary biological information is not limited to the above-described electrocardiographic signal, and at least one of the electrocardiographic signal, pulse wave, breathing, sweating, brain wave, eye movement, and pupil diameter is used as the primary biological information. It can be.

FIG. 2 is a diagram illustrating an example of the sensing unit of the psychological state determination device.
A plurality of electrocardiographic signals from the electrocardiographic sensor 120 are input to the sensing unit 131. As shown in FIG. 2, the sensing unit 131 includes a signal amplifier 131a and the like. The sensing unit 131 senses the cardiac potential signal amplified by the signal amplifier 131a, and acquires, for example, cardiac potential data over a certain period from the start of the pitching operation to the end of the pitching operation.

  For example, the electrocardiographic sensor 120 is attached to the primary subject (the primary subject wears the wear with the electrocardiographic sensor 120 attached), and the sensing unit 131 acquires electrocardiographic data (primary biological information) over a certain period of time. To do. Similarly, the electrocardiographic sensor 120 is attached to the secondary subject (the secondary subject wears the wear on which the electrocardiographic sensor 120 is attached), and the sensing unit 131 detects the electrocardiographic data (secondary living body) over a certain period. Information). In addition, the primary subject and the secondary subject may wear the same wear, and if the primary subject and the secondary subject have different physiques, different wear may be worn. In this case, the attachment position of the electrocardiographic sensor 120 in each wear is substantially the same corresponding position.

  For example, the primary examinee is a senior or a professional pitcher, and the secondary examinee is a person who wants to improve baseball pitching skills (aspiring candidates). The electrocardiogram data from one advanced pitcher or professional pitcher may be acquired and used as the electrocardiogram data from the primary subject, or the electrocardiogram data from 100 senior pitchers or professional pitchers may be acquired to obtain the primary subject. The electrocardiographic data from For example, the electrocardiogram data from 100 advanced players or professional pitchers may be statistically processed to calculate ideal electrocardiogram data, and the calculated electrocardiogram data may be used as electrocardiogram data from the primary subject.

  Alternatively, electrocardiographic data from 100 throwing operations of the same person may be acquired and used as electrocardiographic data from the primary subject. For example, the electrocardiographic data from 100 pitching motions of a senior or professional pitcher may be acquired and used as the electrocardiographic data from the primary subject, or the primary subject and the secondary subject The electrocardiogram data from 100 pitching motions (for example, pitching motions during practice) of the same candidate (skilled candidates) is acquired and used as the cardiac potential data from the primary subject. It is also possible to acquire electrocardiographic data from a single pitching operation (for example, a game pitching operation) and use it as electrocardiographic data from a secondary subject. In this case, for example, a method may be considered in which a candidate who wants to improve becomes a primary subject when favorable, and becomes a secondary subject when not well (during malfunction). Alternatively, the electrocardiographic data from 100 pitching operations may be statistically processed to calculate ideal electrocardiographic data, and the calculated electrocardiographic data may be used as electrocardiographic data from the primary subject.

FIG. 3 is a diagram illustrating an example of the state determination unit 132 of the psychological state determination device 130.
The state determination unit 132 includes a determination unit 132a, a psychological state storage unit 132b, a switching unit 132c, and a comparator 132d.

  For example, the determiner 132a determines the psychological state of the primary subject based on the electrocardiographic data from the primary subject, and the switch 132c receives input from the operator (information input instruction from the primary subject). The psychological state of the primary subject is sent to the psychological state storage unit 132b, and the psychological state storage unit 132b stores the psychological state of the primary subject as the reference psychological state. The determiner 132a determines the psychological state of the secondary subject based on the electrocardiographic data from the secondary subject, and the switch 132c receives input from the operator (information input from the secondary subject). Based on the instruction), the psychological state of the secondary subject is sent to the comparator 132d. The comparator 132d compares the psychological state of the primary subject with the psychological state of the secondary subject and outputs a comparison result (difference). For example, the psychological state of the secondary subject with respect to the psychological state of the primary subject (the psychological state is good or bad) is output. The comparison result may include electrocardiographic data of the primary subject and the secondary subject, a change amount of a value indicating a psychological state, and the like.

FIG. 4 is a diagram illustrating an example of the feedback device 140 of the psychological state feedback system 100.
The feedback device 140 includes a signal generation unit 141 and a synthesized sound / image generation unit 142.

  The signal generation unit 141 receives a signal indicating a psychological state (a psychological state is good or bad) output from the state determination unit 132, electrocardiographic data, a change amount of a value indicating the psychological state, or the like. The signal generation unit 141 determines whether the psychological state is good based on the received psychological state, electrocardiographic data, a change amount of a value indicating the psychological state, or the like. When the received signal indicates that the psychological state is good, the signal generation unit 141 outputs a signal that exaggerates feedback. If the received signal does not indicate that the psychological state is good, the signal generation unit 141 outputs a signal for reducing feedback.

  The signal that exaggerates the feedback may be, for example, a signal that makes the received determination result a better determination, or may be a signal that indicates the level of exaggeration or the exaggerated physical quantity. The signal that exaggerates the feedback may include an audio signal or an image (video) signal for exaggerating the feedback. The signal that reduces the feedback may be, for example, a signal that makes the received determination result worse, or may be a signal that indicates the level of the degree of reduction or the physical quantity that has been reduced. Further, the signal for reducing the feedback may include an acoustic signal or an image (video) signal for reducing the feedback.

  In the above example, the signal generation unit 141 outputs a signal so as to change the feedback depending on whether the psychological state is good or bad, but the determination result of the psychological state may be 3 or more stages. In that case, the signal generation unit 141 outputs a signal that exaggerates or reduces the feedback step by step.

  The synthesized sound / image generation unit 142 receives a signal that exaggerates or reduces feedback from the signal generation unit 141. The synthesized sound / image generating unit 142 exaggerates or reduces at least one of an acoustic signal such as electrocardiographic data, sound, voice, and the like, an image signal, a video signal, and these synthesized signals according to the received signal, Output to the external output device 200. The synthesized sound / image generation unit 142 receives an image or a sound signal from the outside as needed, and can synthesize these images (or images) or the sound signal. The output device 200 includes a display device such as a liquid crystal display device, a speaker, an earphone, and the like.

FIG. 5 is a flowchart illustrating an example of psychological state determination processing performed by the psychological state determination device 130.
For example, the sensing unit 131 acquires primary biological information (cardiac potential data) from the primary subject in advance (ST11), and the determiner 132a determines a psychological state from the primary biological information of the primary subject ( ST12), the psychological state storage unit 132b stores the psychological state of the primary subject (ST13). For example, a psychological state based on primary biometric information from one primary subject may be stored, or primary biometric information from a plurality of primary subjects is statistically analyzed and stored as a single psychological state. Alternatively, it may be stored as a psychological state for a plurality of persons based on primary biometric information from a plurality of primary subjects.

  Subsequently, the sensing unit 131 acquires secondary biological information (cardiac potential data) from the secondary subject (ST14), and the determiner 132a determines the psychological state from the primary biological information of the primary subject ( ST15), the comparator 132d compares the psychological state from the primary subject with the psychological state from the secondary subject (ST16), and outputs the comparison result (ST17). For example, based on the change characteristic of the psychological state of the primary subject and the change characteristic of the psychological state of the secondary subject, the psychological state of the primary subject and the psychological state of the secondary subject are associated and the difference Is detected. Various methods can be applied to the difference detection method. The comparator 132d outputs whether the psychological state of the secondary subject is good or bad with respect to the psychological state of the primary subject. For example, in parallel with the acquisition of electrocardiographic data from primary and secondary subjects, acquire myoelectric data from primary and secondary subjects and use the characteristics of changes in myoelectric data Thus, the electrocardiographic data from the primary subject and the secondary subject can be associated with each other. The myoelectric potential data from the secondary subject during the pitching motion that is substantially the same as the myoelectric potential data from the primary subject during the pitching motion includes similar feature quantities at specific motion timings. “Specific operation timing” means the timing (time or time interval) of a specific operation event (for example, a foot lift or a ball release) in an operation (for example, a pitching operation) composed of a series of operation events. Extract feature values obtained at specific operation timings from time-series primary subject myoelectric potential data and time-series secondary subject myoelectric potential data. Based on the feature amount extracted from the myoelectric potential data of the subject, the myoelectric potential data and cardiac potential data of the primary subject are associated with the myoelectric potential data and cardiac potential data of the secondary subject, Comparison of potential data and detection of difference in cardiac potential data are possible.

  For example, compare the psychological state of one primary subject with the psychological state of a secondary subject, and compare the quality of the secondary subject's psychological state with respect to the psychological state of one primary subject Can be output as In addition, the psychological state of the primary subject and the psychological state of the secondary subject are compared, and the quality of the secondary subject's psychological state with respect to the psychological state of each primary subject is output. You can also The state determination unit may output the cardiac potential of the secondary subject to the feedback device 140 together with the psychological state.

  Further, the first psychological state of the secondary subject is determined based on the electrocardiographic data acquired from the secondary subject during the predetermined exercise at the first time (5 defensives) during the game performance, The second psychological state of the secondary subject is determined based on the electrocardiographic data acquired from the secondary subject during the predetermined exercise at the second time (7 times defensive) during the game performance, and the first It is also possible to compare the psychological state of the subject and the second psychological state, and output a change in the psychological state of the secondary subject over time (for example, fatigue) as a comparison result.

  In this embodiment, the case where the psychological state storage unit 132b holds the psychological state serving as a reference (correct answer) has been described. However, the psychological state serving as a reference is received from a server or the like, and the received psychological state is used. Also good.

FIG. 6 is a flowchart showing an example of psychological state feedback processing by the feedback device 140.
First, the signal generation unit 141 receives a determination result (step ST21). The signal generation unit 141 determines whether the psychological state of the secondary subject is good or bad from the determination result (good or bad psychological state, electrocardiographic data, amount of change in the value indicating the psychological state, etc.) (step ST22). When the received signal indicates that the psychological state is good, the signal generation unit 141 outputs a signal that exaggerates feedback to the synthesized sound / image generation unit 142. When the received signal does not indicate that the psychological state is good, the signal generation unit 141 outputs a signal for reducing feedback to the synthesized sound / image generation unit 142.

  The synthesized sound / image generation unit 142 receives a signal that exaggerates or reduces feedback from the signal generation unit 141. When the synthesized sound / image generating unit 142 receives a signal that exaggerates feedback, the synthesized sound / image generating unit 142 exaggerates a signal including at least one of an acoustic signal such as an electrocardiogram, sound, and sound, an image signal, a video signal, and a synthesized signal thereof. (Step ST23), and outputs a feedback signal to the external output device 200. When the synthesized sound / image generating unit 142 receives a signal for reducing feedback, the synthesized sound / image generating unit 142 reduces an acoustic signal such as an electrocardiogram, sound, and sound, an image signal, a video signal, and a signal including at least one of these synthesized signals. Then, a feedback signal is output to the external output device 200 (step ST24).

  When exaggerating the feedback, the synthesized sound / image generating unit 142 may output the electrocardiogram after correcting the electrocardiogram to a better value, and the difference between the electrocardiogram after the amendment to the good value and the actual electrocardiogram. May be output, a sound signal having a louder volume than the reference sound may be output, or a sound signal in which another sound is superimposed on the reference sound may be output. Further, the synthesized sound / image generation unit 142 may output an audio signal that gives up or encourages the secondary subject. The synthesized sound / image generation unit 142 may output an image signal of a predetermined color, may output an image signal of a symbol such as “◯” or “◎”, and outputs an image signal of a smile Alternatively, an image signal including words such as giving up or encouraging the secondary subject may be output, or an image signal or video signal acquired from the outside may be processed and output. Furthermore, the synthesized sound / image generation unit 142 may be a synthesized signal that is a combination of at least two of the above-described sound signals, acoustic signals such as audio signals, image signals, and video signals.

  When the feedback is reduced, the synthesized sound / image generation unit 142 may output the electrocardiogram corrected to a worse value, and output the difference between the electrocardiogram corrected to the bad value and the actual electrocardiogram. Alternatively, a sound signal having a lower volume than that of the reference sound may be output, and the sound signal may be omitted for silence. Further, the synthesized sound / image generation unit 142 may output an audio signal such as “I am one step further” or “I am not yet” according to the psychological state of the secondary subject. The synthesized sound / image generation unit 142 may output an image signal of a predetermined color, may output an image signal of a symbol such as “Δ” or “x”, and outputs an image signal of a crying face Alternatively, an image signal including words according to the level of the secondary subject may be output, or an image signal acquired from the outside may be processed and output. Furthermore, the synthesized sound / image generating unit 142 may be a synthesized signal that is a combination of at least two of the above-described sound signals, audio signals such as audio signals, image signals, and video signals.

  Hereinafter, the sensing unit 131 and the state determination unit 132 will be summarized. Here, not only the sensing, determination, and comparison of the psychological state, but also the sensing, determination, and comparison of the motion state will be described.

1. Sensing unit 131
The sensing unit 131 measures biological information (motion signal and psychological signal) from one or more objects. It does not matter what means the biometric information is acquired.

<Motion signal sensing>
The sensing unit 131 measures a myoelectric potential signal (biological electrode) from one or a plurality of muscles. Furthermore, the sensing unit 131 can also measure an acceleration signal (acceleration sensor), a pressure signal (pressure sensor), and a joint angle signal (bending sensor) at one or more parts.

<Psychological signal sensing>
The sensing unit 131 can also measure autonomic nerve activity in order to determine a psychological state, for example. For example, the sensing unit 131 measures a cardiac potential signal (biological electrode) and a pulse wave (photosensor). Thereby, autonomic nerve activity (excitement / relaxation) can be detected.

  The sensing unit 131 can also measure other physiological reactions. For example, the sensing unit 131 measures respiration, sweating, brain waves, eye movement, and pupil diameter. As a result, emotions (excitement / relaxation), emotions (surprise or preference), attention, intention, etc. can be detected.

  Possible measurement targets for baseball equipment include helmet, hat (electroencephalogram), sunglasses (eye movement (gaze direction, microsaccade), pupil diameter), gloves (sweat, pulse wave, pressure, acceleration), underwear (myoelectricity) , Electrocardiogram, breathing, acceleration), shoes (sweat, pulse wave, pressure, acceleration).

2. State determination unit 132
<(Exercise) determiner 132a>
The determiner 132a determines the following exercise state from the measured myoelectric potential signal or the like.
The determiner 132a determines an exercise pattern (knack or force). For example, the determiner 132a determines a muscle activity pattern for a certain movement (change in acceleration or joint angle), a mutual timing of the muscle activity, and the like.
The determiner 132a determines a posture pattern (balance). For example, the determiner 132a determines an activity state or activity timing of a muscle combination with respect to a change in plantar pressure. For example, the determiner 132a determines the pressure change of the sole and how to use the muscles when skiing.
The determiner 132a determines the degree of muscle fatigue. For example, the determiner 132a performs determination by performing frequency analysis on one or a plurality of myoelectric potential signals (when fatigued, the power of muscle activity shifts to a low frequency band). For example, this determination result can be used for determination at the time of a player change.

<(Motion) Comparator 132d>
The comparator 132d compares the information obtained from one or more primary subjects with the information obtained from the secondary subjects in order to determine changes in exercise status and good / bad.
The comparator 132d compares the exercise state of the primary subject, which is a specific object, with the exercise state of the secondary subject. In advance, the determiner 132a determines the exercise state of a primary subject such as a specific advanced player or first-class player (example), and the comparator 132d determines the exercise state of the primary subject and the secondary subject. Compare the exercise state.
The comparator 132d compares the exercise state obtained from the primary subject with a certain group with the exercise state obtained from the secondary subject. For example, the determiner 132a determines in advance a movement state (an average pattern, a pattern characterized by data mining, etc.) obtained from a certain group of primary subjects, and the comparator 132d The movement state and the movement state of the secondary subject are compared.
Further, the comparator 132d sets itself (secondary subject) as the primary subject, and compares the exercise state of the primary subject with the exercise state of the secondary subject. The comparator 132d compares the normal (average value when throwing 100 balls in practice or the like) and the current (actual such as game) exercise state.

  Further, the comparator 132d may compare (determine) the relationship of the exercise state between the partners. For example, the determiner 132a determines the motion state (for example, the degree of strength) of the pitcher and the batter, and the comparator 132d compares the motion state of the pitcher and the batter. The comparison result can be used to judge the state of compatibility and compatibility.

<(Psychological state) determiner 132a>
The determiner 132a includes an autonomic state (balance of sympathetic and parasympathetic activities = tension / relaxation) by performing frequency analysis of heart rate and blood flow fluctuation (fluctuation) obtained from an electrocardiogram signal or a pulse wave. The psychological state can be determined. (For example, Non-Patent Document 3). To quantify tension / relaxation, for example, calculate the low frequency component (LF: 0.05-0.15 Hz) and the high frequency component (HF: 0.15-0.4 Hz) of the heart rate variability. LF / HF can be evaluated as tension.

In addition, the determiner 132a can also determine the state of emotion (excitement / relaxation), emotion (surprise / preference), attention, intention, etc. based on breathing, sweating, brain waves, eye movement, pupil diameter, etc. ) Comparator 132d>
The comparator 132d compares the information obtained from one or more primary subjects with the information obtained from the secondary subjects in order to determine changes in exercise status and good / bad.
The comparator 132d compares the autonomic state obtained from the primary subject with a certain group with the autonomic state obtained from the secondary subject. For example, the determiner 132a determines in advance an autonomic state (average value, a value characterized by data mining, etc.) obtained from a group of primary subjects, and the comparator 132d The autonomic state of the subject and the autonomic state of the secondary subject are compared.
The comparator 132d sets itself (secondary subject) as the primary subject, and compares the autonomic state of the primary subject with the autonomic state of the secondary subject. The comparator 132d compares the autonomic nerve state in the first situation (usually during practice, for example) and the autonomic nerve state in the second situation (currently, for example, during a game).
The comparator 132d compares (determines) the psychological relationship between partners. For example, the determiner 132a determines the psychological state of the pitcher and the batter, and the comparator 132d compares the psychological state of the pitcher and the batter. The comparison result can be used to judge the state of compatibility and compatibility.

<(Exercise state and psychological state) comparator 132d>
The comparator 132d compares (determines) “relationship between the autonomic nerve state and the motor state” by combining the motor state and the autonomic nerve state. For example, if you are nervous, you can see the relationship that exercise is disturbed.
Next, the determination of the motion state and the state of the autonomic nerve will be described in detail.

As Example 1, we will explain the comprehensive monitoring of professional pitchers in actual battles.
1. The pitcher A wears compression inner wear (for example, Hitoe (registered trademark)) equipped with a myoelectric potential sensor (wearable sensor) on both upper and lower body. Compression inner wear (for example, Hitoe (registered trademark)) equipped with an electrocardiographic sensor 120 (wearable sensor) is worn on the upper body. In order to measure the myoelectric potential in the upper body wear, for example, myoelectric potential sensors are arranged at a plurality of locations corresponding to the wearer's arm, shoulder, chest circumference, and waist. Further, in order to measure the electrocardiogram in the upper body wear, for example, the electrocardiogram sensor 120 is arranged at a location corresponding to the wearer's heart circumference. The lower body wear includes muscles at multiple locations corresponding to the wearer's buttocks to measure the myoelectric potential of the wearer's buttocks (or other muscles of the lower limbs) in order to measure the fatigue of the lower body. A potential sensor is arranged.

2. The state determination unit 132 can estimate the degree of muscle fatigue by frequency analysis of the myoelectric potentials of the upper and lower bodies. On the other hand, the state determination unit 132 can estimate the activity balance between the pitcher's sympathetic nerve and the parasympathetic nerve from the analysis of the heart rate variability based on the electrocardiogram.

3. The myoelectric sensor, the electrocardiographic sensor 120, and the sensing unit 131 continuously measure the myoelectric potential and electrocardiographic potential of the pitcher A from normal practice and games, and long-term data in various situations as listed below. Save up.

3.1 Unpractical situation where batter is not standing in bullpen 3.2 Situation where a batter is played (practice)
3.3 Situation where a batter is played (match)
The following data acquired from the above situation is input to the state determination unit 132 that functions as a biological state estimator. The state determination unit 132 sequentially updates a model for estimating the latest future pitching performance with “3.1”, “3.2”, and “3.3” as inputs.

3.4.1 Objective muscle fatigue estimated from myoelectric potential 3.4.2 Motor chain efficiency estimated from whole body muscle activity pattern 3.4.2 Autonomic nervous activity balance estimated from heart rate variability 3 4.4 Performance indicators such as pitching speed and control variations <User use case 1: assuming actual battle scene>
The information in the above 3.4.1 to 3.4.4 is in a state where the secondary subject (the person whose data is being measured) and surrounding people can be monitored at all times. That is, the feedback device 140 outputs the information of the above 3.4.1 to 3.4.4.

  4). First half of the game: One of the challenges for pitcher A is that the performance immediately after the start of the game (rise) is unstable. From long-term data measurements, it is known that the mental state estimated from the electrocardiogram frequently falls into an overstress state. In addition, it has been found that there is a correlation between overstressing and a decrease in throwing performance. For this reason, when such a situation occurs during the match, the pitcher coach rarely disturbs the performance by taking action such as “getting a gap”.

  5. Mid game: When fatigue is increased, it is inferred from the monitored EMG data, but the muscle state inferred from muscle activity is inferior (for example, accumulation of fatigue) The leaders decide to continue.

  6). The second half of the game: If both the muscle fatigue and mental aspects of the condition are estimated to be degraded, and the muscle activity pattern throughout the body also shows a pattern specific to fatigue, the pitcher A must be released before being struck by the batter. Make a decision.

7). The use as described above not only makes it possible to change the player at a better timing, but also reduces the risk of injury due to a reduction in the chance of being forced.
<User usage example 2: Overcoming problems in the game through practice>
8). Pitcher A has the problem of not being able to demonstrate the performance as usual in the game (although it may be influenced by mental), the surrounding people speculate that there may be a physical cause )

  9. In order to overcome the above-mentioned problems, the muscle activity pattern of the whole body, the autonomic nerve activity balance, and the pitching performance index at the time of pitching in each of the preceding items 4.1 to 4.3 are compared.

  10. As a result, pitcher A is throwing in a very relaxed state only when throwing with a bullpen, and it is found that there is a great change in the mental aspect in a battle scene with a batter, particularly in a battle scene in a game. Furthermore, it is possible to pick up game situations that cause mental variations.

  11. Pitcher A receives the feedback of “10.” and discusses with the leader how to deal with the situation of the game (organization of the situation, how to hold feelings, etc.). Furthermore, in practice, it can be expected that the instability of performance at the time of the game will be improved by performing actual battle practice that reproduces the situation.

As example 2, heart rate variability analysis as prevention of overtraining will be described.
12 The young baseball player A (pitcher) must perform high-intensity training in parallel with many matches during the season. If you want to play a game with accumulated fatigue, you will not only be able to perform well, but also increase the risk of injury. Therefore, a mechanism that fluidly determines whether or not to climb the day can be useful by monitoring the condition of the player on the day.

  13. Overtraining syndrome is detected from heart rate variability data obtained from electrocardiograms when waking up.

The young baseball player A continuously collects heart rate variability and myoelectric potential data by measuring the electrocardiogram and the myoelectric potential on a daily basis from the time of getting up. It has been reported that athletes who fall into overtraining have increased heart rate and decreased heart rate variability when waking up. It has also been reported that the degree of muscle fatigue estimated from myoelectric potential is related to the degree of fatigue estimated from heart rate variability. Together with these and the subjective fatigue level of the person, the conditioning coach determines whether or not pitcher A can climb on the morning of the day.
At the same time, it is determined that pitcher B, which is estimated to be in the best condition in the same measurement, is to be pitched in place of young baseball player A.

FIG. 7 is a block diagram illustrating an example of the configuration of the synthesized sound / image generation unit.
First, an outline will be described. In this embodiment, a superimposed image of a plurality of images of one or more operation subjects obtained at a specific operation timing and myoelectric potential data of the operation subjects obtained at a specific operation timing, such as a plurality of myoelectric potentials, etc. Are combined with the muscle activity degree image derived from the biometric information (in other words, derived from a plurality of myoelectric potentials), and a composite image obtained thereby is displayed. The composite image is derived from “multiple” images and myoelectric potential. Therefore, it is possible to provide visual feedback of “appropriateness of motion and accompanying muscle activity” that is difficult to judge from “single” images and information derived from only myoelectric potential.

  The “operation subject” may be a person or a person other than a person (for example, an animal). An example of a single “acting subject” is one actuating subject that is an object of motion feedback. Examples of multiple `` acting subjects '' are two acting subjects consisting of an acting subject acting as a model and an actuating subject subject to feedback, and an act acting as a model and its behavior. A plurality of action subjects (including a feedback target action subject) or a plurality of action subjects other than a model action subject (including a feedback target action subject), etc. .

  The plurality of images may be taken in a plurality of operation processes of one or a plurality of operation subjects, or may be taken in a single operation process of a plurality of operation subjects. These images may be images of the whole body of the motion subject, or may be images of a part of the motion subject (for example, the upper body or legs).

  The “plural myoelectric potentials” may be obtained by a plurality of motion processes by one or a plurality of motion subjects, or may be obtained at a plurality of measurement points of a single or a plurality of motion subjects. It may be obtained by both of them, or may be obtained by a single operation process of a plurality of operation subjects. The “plural myoelectric potentials” are, for example, myoelectric potentials of one or more muscles that are involved (contributing) to the motion of the motion subject represented in the “plural images” (for example, main muscles involved in the motion of the motion subject) Of myoelectric potential and antagonistic myoelectric potential of the corresponding antagonist muscle).

  The “muscle activity level image” is, for example, an image representing a value obtained from a relationship between a plurality of muscle activity levels, an image representing a variation in a plurality of muscle activity levels, an image representing a relative value of a plurality of muscle activity levels, or a plurality of At least one of the images representing the variation in the relative value of the muscle activity level. However, each of the plurality of muscle activity levels corresponds to each of the plurality of myoelectric potentials. “Muscle activity” is a myoelectric potential or a function value thereof. An example of the function value of the myoelectric potential is a function value representing the magnitude of the myoelectric potential, for example, a broad monotonically increasing function value (for example, monotonically increasing function value) of the absolute amplitude value of the myoelectric potential. For example, the absolute amplitude value of the myoelectric potential or the mean square amplitude value in a predetermined time interval can be set as the “muscle activity level”. As the function value of the myoelectric potential, a value representing the magnitude of the myoelectric potential that has been smoothed or removed from the DC component may be used. Alternatively, the “muscle activity image” may be derived from a plurality of values obtained by converting myoelectric potential or muscle activity into the frequency domain (frequency spectrum or the like). With the images as described above, it is possible to visually feedback the appropriateness of the muscle activity accompanying the movement.

  The “variation” of the plurality of muscle activity levels may be a variation in a plurality of muscle activity levels derived from a plurality of myoelectric potentials obtained in a trial process of a plurality of times at the same measurement location of a single motion subject. In addition, it may be a variation in a plurality of muscle activities derived from a plurality of myoelectric potentials obtained at a plurality of measurement locations of a single action subject, or a plurality of obtained at the same measurement location of a plurality of action subjects. It may be a variation in a plurality of muscle activities derived from a myoelectric potential, or may be a variation in a plurality of muscle activities derived from a plurality of myoelectric potentials obtained at a plurality of measurement locations of a plurality of motion subjects. Good. An example of “variation” is “statistical variation” such as variance and standard deviation.

  The relative value of a plurality of muscle activity levels may be a relative value between a plurality of muscle activity levels derived from a plurality of myoelectric potentials obtained at the same measurement location of a single motion subject, or a single motion subject. It may be a relative value between a plurality of muscle activity levels derived from a plurality of myoelectric potentials obtained at a plurality of measurement points, or a plurality of myoelectric potentials obtained at the same measurement point of a plurality of motion subjects. It may be a relative value between a plurality of muscle activity levels derived from, or a relative value between a plurality of muscle activity levels derived from a plurality of myoelectric potentials obtained at a plurality of measurement points of a plurality of motion subjects. There may be. Examples of the “relative value” are “difference” with positive / negative distinction, “difference” with no positive / negative distinction, “square error”, or “ratio”, or a function value of any of these.

  The “image representing variation” is an image representing the magnitude of variation. In the example of “image representing variation”, (F-1) the vertical axis and the horizontal axis are time axes, and coordinates corresponding to each time have a “color tone” point corresponding to the magnitude of the variation at that time. Or (F-2) a time series graph in which the horizontal axis is the time axis, the vertical axis is the axis representing the magnitude of variation at each time, and the magnitude of variation corresponding to each time is plotted (F-3) A histogram in which the horizontal axis is the axis representing the magnitude of variation, the vertical axis is the axis representing the frequency of variation, and the frequency of each variation is represented by a bar graph. An “image representing a relative value” is an image representing a relative value or its size. An example of “an image representing a relative value” is an image obtained by replacing the “variation magnitude” in the above examples (F-1) to (F-3) with “relative value” or “relative value magnitude”. Etc.

  The “multiple myoelectric potentials” include the myoelectric potentials of the main muscles of the motion and the myoelectric potentials of the antagonistic muscles corresponding to the main muscles, and the “muscle activity image” indicates the myogenic activity corresponding to the myoelectric potentials of the main muscles. An image representing a value obtained from the relationship between the degree of muscle activity and the muscle potential corresponding to the myoelectric potential of the antagonistic muscle, the muscle activity corresponding to the muscle potential of the main muscle and the muscle corresponding to the muscle potential of the antagonistic muscle An image showing a relative value between the activity levels, or an image showing a variation in relative values between the muscle activity level corresponding to the myoelectric potential of the main muscle and the muscle activity level corresponding to the myoelectric potential of the antagonist muscle. At least one of them may be included. This image can represent the “power” of the subject. In particular, when this "relative value" is a value obtained by dividing "muscle activity corresponding to myoelectric potential of antagonistic muscle" by "muscle activity corresponding to myoelectric potential of main muscle" (divided value), Activity level (ratio of antagonistic muscle activity to main muscle activity) can be quantified, and "force" can be expressed more clearly. The “antagonist muscle corresponding to the main muscle” means a muscle that relaxes in an operation in which the main muscle contracts. For example, in the operation of bending the wrist, the wrist flexor muscles contract as the main muscles and the extensor muscles relax as the antagonistic muscles. Conversely, in the action of stretching the wrist, the extensor is contracted as the main muscle and the wrist flexor is relaxed as the antimuscle. When bending the wrist, it is desirable that the wrist flexor muscle that is the main muscle is active and the activity of the extensor that is the antagonist muscle is suppressed. However, if the antagonistic muscle is excessively active, it becomes a so-called “force” state and inhibits exercise. In the above-mentioned “muscle activity level image”, the relationship between the main muscle activity and the antagonistic muscle activity associated with the motion can be presented, and this strength level can be fed back.

  The “muscle activity image” may include an image derived from “activity start time” of a plurality of muscles obtained from a plurality of myoelectric potentials (for example, a plurality of muscles involved in a motion of a certain motion subject or a series of motions). . The activity start time means the time when the muscle starts the muscle activity. Information on the start timing of muscle activity (for example, variation in the start timing of muscle activity, relative value, and appropriateness) can be expressed by this muscle activity level image. For example, the muscle activity level image may include an image representing a variation in “activity start time” of a plurality of muscles, an image representing a relative value, or an image representing a variation in relative value. In addition, an image derived from an order relationship at the time of starting the activity of a plurality of muscles may be included. The relative value of the “activity start time” may be a relative value of another “activity start time” with respect to any of the “activity start time” of the plurality of muscles, It may be a relative value of “activity start time”. An example of an image derived from an order relationship between the activity start points of a plurality of muscles is (F-4) an image in which the color points corresponding to each muscle are plotted at the position of the time axis representing the activity start time of the muscles, (F-5) An image in which identifiers (numbers, etc.) corresponding to each muscle are plotted on the position of the time axis representing the activity start time of the muscle, or (F-6) the order relationship between the activity start times of a plurality of muscles. Images that differ in “color” and “brightness” between appropriate and inappropriate cases, and (F-7) The closer the order relationship at the start of activity of multiple muscles is to the closer the appropriate order relationship, An image that approaches a specific “color” or “brightness”, or (F-8) a figure that indicates how close the order relationship at the start of activity of multiple muscles is to an appropriate order relationship, or a figure that indicates the size An image including It should be noted that whether or not the order relationship at the start of activity of a plurality of muscles is appropriate depends on the operation. For example, in a ball release operation of a pitching exercise, an order relationship in which muscle activity is started in the order of a muscle that moves the shoulder, a muscle that moves the elbow, a muscle that moves the wrist, and a muscle that moves the finger is appropriate. Such an appropriate order relationship may be set in advance according to the operation. As described above, for example, it is possible to feed back whether a plurality of muscles involved in a predetermined action have started activities in an appropriate order.

  The muscle activity level image may include an image derived from a plurality of “integrated feature amounts”. However, each of the “integrated feature amounts” is a feature amount derived from both a plurality of images obtained at a specific operation timing and each of a plurality of myoelectric potentials obtained at a specific operation timing. is there. For example, each integrated feature amount is derived from both an image obtained at a specific operation timing and a myoelectric potential of a muscle involved in the operation appearing in the image. This makes it possible to visualize and correlate the correlation between the posture and the muscle activity in the motion event.

  The “specific operation timing” means the timing (time or time interval) of a specific operation event (for example, a foot lift or a ball release) in an operation (for example, pitching exercise) composed of a series of operation events. In the embodiment, based on the feature amount derived from the action subject image, the action subject myoelectric potential, or both, the action subject's “multiple” obtained from the action subject time-series image at a specific action timing. The “image” is extracted, and “the plurality of myoelectric potentials” of the motion subject obtained at a specific motion timing is extracted from the time series myoelectric potentials of the motion subject. In particular, it is possible to accurately extract “a plurality of images” and “a plurality of myoelectric potentials” at a “specific operation timing” based on feature amounts derived from both images and myoelectric potentials.

  In addition, “multiple electromyograms”, “cardiac activity images”, “cardiac activity”, “variation” of multiple cardiac activities, and relative values of multiple cardiac activities are also “multiple myoelectric potentials”. , “Muscle activity level image”, “muscle activity level”, “variation” of a plurality of muscle activity levels, and the above description regarding relative values of a plurality of muscle activity levels.

  For example, the “plural electrocardiograms” may be obtained in a plurality of operation processes by one or a plurality of action subjects, or may be obtained at measurement points of one or more action subjects. It may be obtained by both of them, or may be obtained by a single operation process of a plurality of operation subjects. The “plural electrocardiograms” include, for example, one or a plurality of electrocardiograms that participate (contribute) to the action of the subject of action represented in “plural images”. When the output of the signal generation unit 141 is an exaggerated or reduced electrocardiogram signal, the exaggerated or reduced electrocardiogram signal may be input to the visualization device 11 as an electrocardiogram signal.

  The “cardiac activity level image” is, for example, an image representing a value obtained from a relationship between a plurality of cardiac activity levels, an image representing a plurality of cardiac activity variation, an image representing a relative value of a plurality of cardiac activity levels, or a plurality of At least one of the images representing the variation in the relative value of the heart activity level. However, each of the plurality of cardiac activity levels corresponds to each of the plurality of electrocardiograms. “Cardiac activity” is a cardiac potential or a function value thereof. An example of the function value of the electrocardiogram is a heart rate, and the heart rate in a predetermined time interval can be referred to as “cardiac activity”. Alternatively, the “cardiac activity image” may be derived from a plurality of values obtained by converting the cardiac activity into the frequency domain (frequency spectrum or the like). With the above image, the state of the heart activity accompanying the motion can be visually fed back.

  The “variation” of the plurality of cardiac activities may be a variation in a plurality of cardiac activities derived from a plurality of electrocardiograms obtained in a trial process of a plurality of times at the same measurement location of a single action subject. In addition, it may be a variation in a plurality of cardiac activities derived from a plurality of electrocardiograms obtained at a single measurement location of an action subject, or a plurality of electrocardiograms obtained at the same measurement location of a plurality of action subjects. May be a plurality of cardiac activity variability derived from the above, or may be a plurality of cardiac activity variability derived from a plurality of electrocardiograms obtained at measurement points of a plurality of motion subjects. An example of “variation” is “statistical variation” such as variance and standard deviation.

  The relative value of a plurality of cardiac activity levels may be a relative value between a plurality of muscle activity levels derived from a plurality of cardiac potentials obtained at the same measurement location of a single motion subject, or a single motion subject. It may be a relative value between a plurality of cardiac activities derived from a plurality of electrocardiograms obtained at a plurality of measurement locations, or may be derived from a plurality of electrocardiograms obtained at the same measurement location of a plurality of action subjects. It may be a relative value between a plurality of cardiac activity levels, or may be a relative value between a plurality of cardiac activity levels derived from a plurality of electrocardiograms obtained at measurement points of a plurality of motion subjects. . Examples of the “relative value” are “difference” having a positive / negative distinction, “difference” or “ratio” having no positive / negative distinction, or a function value of any of these.

    Hereinafter, each embodiment will be described in detail with reference to the drawings.

[First Embodiment]
A first embodiment will be described.
<Configuration>
As illustrated in FIG. 7, the synthesized sound / image generation unit 142 includes, for example, the visualization device 11. The output device 200 includes an image presentation device 12. Note that a part of the configuration of the visualization device 11 may be included in the signal generation unit 141. The visualization device 11 includes an operation timing detection unit 111, a feature amount calculation unit 112, a feature amount storage unit 113, an operation timing collation unit 114, a collation result storage unit 115, a superimposed image creation unit 116, and muscle activity (or cardiac activity). An image generation unit 117 and a synthesis unit 118 are included. As illustrated in FIG. 8, the motion timing collation unit 114 includes a feature amount extraction unit 1141, a similarity calculation unit 1142, an operation timing detection unit 1143, an image extraction unit 1144, and a muscle activity (or cardiac activity) extraction unit 1145. And an end determination unit 1146. The visualization device 11 is a general-purpose device including, for example, a processor (for example, hardware including one or more CPUs (central processing units)) and a memory (for example, RAM (random-access memory) and ROM (read-only memory)). Alternatively, a dedicated computer, a mobile terminal such as a smartphone or a tablet is configured to execute a predetermined program. This program may be installed in the computer, or may be recorded in the computer in advance. In addition, some or all of the processing units may be configured using an electronic circuit that realizes a processing function independently instead of an electronic circuit (circuitry) that realizes a functional configuration by reading a program like a CPU. . The image presentation device 12 is a display device such as a liquid crystal display.

<Pre-processing>
In the pre-processing, a feature amount corresponding to an operation event at a specific operation timing is acquired. First, the motion timing detection unit 111 has a time-series image (video) obtained by photographing a series of motions of a specific motion subject (for example, a primary subject) with a camera and a series of muscle activities (or minds) of the motion subject. Timing designation information for designating at least one of a myoelectric signal representing a time-series myoelectric potential (or a cardiac potential signal representing a cardiac potential) obtained by measuring (activity) and one or more operation timings. Entered. The video and myoelectric potential signal (or cardiac potential signal) may be obtained in advance and stored in a storage device (not shown), or may be obtained in real time.

  The myoelectric potential signal is, for example, an electrode (for example, a myoelectric potential sensor) at one or a plurality of measurement points (for example, skin surface) in the vicinity of one or a plurality of muscles (main muscles, antagonistic muscles, etc.) involved in the motion of the subject. Is a time-series signal obtained by measuring the action potential generated from each muscle. For example, a myoelectric signal representing a time series of myoelectric potential of one system is obtained from one measurement location. For example, the myoelectric potential signal of the main muscle is obtained based on the potential of the electrode at the measurement location near the main muscle, and the myoelectric potential signal of the antagonist muscle is obtained based on the potential of the electrode at the measurement location near the antagonist muscle.

  The cardiac potential signal is measured by a cardiac potential sensor composed of electrodes or the like. Each cardiac potential sensor outputs a cardiac potential signal (biological information) corresponding to the activity level of the heart. For example, the electrocardiogram sensor is attached to the wear so as to correspond to the body (chest and the like) of the primary subject and the secondary subject. For example, an electrocardiogram signal representing one system of time series electrocardiogram is obtained from one measurement location.

  The timing designation information is information that designates, for example, a time or a time interval representing a specific operation timing arbitrarily selected by the user. These times and time intervals may be relative times or absolute times. For example, the pitching motion includes a plurality of motion events such as “(a) leg raising”, “(b) stepping out”, “(c) grounding”, and “(d) ball release”. At this time, if the time (or time interval) indicating the operation timing of “(d) ball release” is designated, “(d) ball with respect to the start of the pitching motion (at the start of“ (a) leg raising ”) You may specify the relative time at the start of “Release” (or at the start and end), or the absolute time at the start of “(d) Ball Release” (or at the start and end) May be specified. Further, the timing designation information may designate a time or time interval of operation timings of two or more. For example, the timing designation information may designate the operation timing time or time interval of “(b) stepping” and “(d) ball release”. When the operation timing is a time interval, this time interval may be specified by specifying a specific time. This “specific time” is a reference time for specifying the time interval that is the operation timing, and examples thereof include the start time and the center time of the time interval that is the operation timing. The timing designation information may be an operation timing of a primary subject, for example, an advanced person or a professional (reference player) who is performing a predetermined exercise.

  The motion timing detection unit 111 extracts the motion timing image and / or myoelectric signal (or cardiac potential signal) specified by the timing designation information from the input time-series image and / or time-series myoelectric signal. To do. The operation timing detection unit 111 may extract only the image and / or myoelectric potential signal (or electrocardiographic signal) of the time or time interval specified by the timing designation information, and further before that time or time interval. Images and / or myoelectric potential signals (or electrocardiographic signals) of the time interval or the subsequent time interval may be extracted. Alternatively, the operation timing detection unit 111 may extract an image and / or myoelectric potential signal (or cardiac potential signal) in a time interval narrower than the time interval specified by the timing designation information. For example, when the timing designation information designates the start time and end time of “(b) stepping” in the pitching operation, the operation timing detection unit 111 displays the images from the start time to the end time of “(b) stepping” and Only the myoelectric potential signal (or electrocardiographic signal) may be extracted, and further, the image and / or myoelectric potential signal (or electrocardiographic potential) before and after the start of “(b) stepping”. Signal) may be extracted. Or you may extract only the image and / or myoelectric potential signal (or electrocardiogram signal) of a time interval narrower than the time interval from the start time to the end time of “(b) stepping”. The extracted image and / or myoelectric potential signal (or electrocardiographic signal) is sent to the feature amount calculation unit 112. When images and / or myoelectric potential signals (or cardiac potential signals) are extracted for a plurality of operation timings, the extracted images and / or myoelectric potential signals (or cardiac potential signals) and the corresponding operation timings are extracted. Is output in association with the information for identifying.

  The feature amount calculation unit 112 calculates a feature amount (feature amount of motion timing) derived from the myoelectric potential represented by the input image and / or myoelectric potential signal, or the input image and / or cardiac potential signal is A feature amount (for example, heart rate) derived from the expressed cardiac potential is calculated and output. When the input image and / or myoelectric potential signal (or electrocardiographic signal) corresponds to a plurality of operation timings (for example, “(b) stepping” and “(d) ball release” operation timings) The amount calculation unit 112 calculates and outputs a feature amount corresponding to each operation timing. This feature amount may be derived from an image, may be derived from myoelectric potential (or cardiac potential), or may be derived from both an image and myoelectric potential (or cardiac potential). It may be a thing. The feature amount derived from both the image and the myoelectric potential (or cardiac potential) may include the feature amount derived from the image and the feature amount derived from the myoelectric potential (or cardiac potential). It may be derived from a function value (for example, a relative value) between a corresponding value and a value corresponding to myoelectric potential (or cardiac potential). This is a feature amount derived from a value corresponding to an image of a motion part (upper body, leg, etc.) involving a muscle in the vicinity of a myoelectric potential measurement point and a value corresponding to a myoelectric potential obtained at the measurement point. May be. The “value corresponding to myoelectric potential” is myoelectric potential or a function value of myoelectric potential. The “value corresponding to the myoelectric potential” may be the same as or different from the “muscle activity”. The “value corresponding to the cardiac potential” is a cardiac potential or a function value of the cardiac potential. The “value corresponding to the cardiac potential” may be the same as or different from the “cardiac activity”. Each feature amount is exemplified below.

《Examples of features derived from motion timing images》
As the feature amount derived from the image, there is the aforementioned MHI. Examples of feature values by MHI are given below.

  However, H (x, y, t) represents the feature quantity (MHI pixel value) of the coordinates (x, y) at time t. d (x, y, t) is a motion detection function, and d (x, y, t) = 1 indicates that the pixel value of the coordinates (x, y) of the image has changed at time t. A larger t represents a new time. ρ is a positive attenuation. max (α, β) returns α when α ≧ β, and returns β when α <β. The initial value of H (x, y, t) is 0, for example. For example, when the operation timing is a specific time t, H (x, y, t) at all coordinates (x, y) at this time t (where x = x1,..., X2, y = y1 ,..., Y2, x1 <x2, y1 <y2) as features. For example, in a baseball pitching motion, H () at time t corresponding to four motion events “(a) leg raising”, “(b) stepping out”, “(c) grounding”, and “(d) ball release”, respectively. A set of x1, y1, t),..., H (x2, y2, t) is a feature quantity. For example, when the operation timing is a specific time interval [t1, t2], all coordinates (x, y) in this time interval [t1, t2] (where x = x1,..., X2, y = y1 ,..., Y2, x1 <x2, y1 <y2), and H (x, y, t) (where t = t1,..., T2) is an element. Alternatively, a value H (x, y, T) (average of H (x, y, t) at all coordinates (x, y) in the time interval [t1, t2] for each predetermined time interval T. However, a set having x = x1,..., X2, y = y1,..., Y2, x1 <x2, y1 <y2) as elements may be used as the feature amount. Or it is good also considering what converted H (x, y, t) and H (x, y, T) into the frequency domain in a certain time section.

<< Example 1 of feature amount derived from myoelectric potential (or cardiac potential) at operation timing >>
When there is a characteristic change in muscle activity (or electrocardiogram) at the operation timing (for example, a large peak appears in upper limb muscle activity by “(d) ball release”), myoelectric signal (or electrocardiographic signal) A feature amount may be a time change amount of a value corresponding to the myoelectric potential (or cardiac potential) represented by or a set having a time change amount of a value corresponding to the myoelectric potential (or cardiac potential) as an element. For example, the amount of time change of the magnitude of the myoelectric potential represented by the myoelectric potential signal, or a set having the time change of the magnitude of the myoelectric potential as an element, or the heart rate represented by the electrocardiographic signal may be used as the feature quantity. Specifically, for example, the myoelectric potential signals Vi (t) and Vi (t−1) obtained at I measurement points (where I is an integer equal to or greater than 1) have a specific time t. (Where i is the channel i = 1,..., I corresponding to the measurement location), the myoelectric potential magnitude Mi (t) indicated by the myoelectric potential signals Vi (t) and Vi (t−1), respectively. ), Mi (t−1) time variation ΔMi (t) = Mi (t) −Mi (t−1) or time variation ΔM1 (t),..., ΔMI (t). It may be an amount. For example, when the operation timing is a specific time section [t1, t2], a set of time change amounts ΔMi (t) = Mi (t) −Mi (t−1) in the time section [t1, t2] is obtained. A feature amount may be used, or a set of time change amounts ΔMi (T) obtained by averaging the time change amounts ΔMi (t) for each predetermined time interval T may be used as the feature amount. Alternatively, a time change amount of a value obtained by converting a value corresponding to a myoelectric potential (or cardiac potential) represented by a myoelectric potential signal (or a cardiac potential signal) into a frequency domain, or a temporal change of a value converted into such a frequency domain. A set having a quantity as an element may be used as a feature quantity.

<< Example 2 of feature amount derived from myoelectric potential (or cardiac potential) at operation timing >>
A value corresponding to the myoelectric potential (or cardiac potential) represented by the input myoelectric potential signal (or cardiac potential signal), or a set having a value corresponding to the myoelectric potential (or cardiac potential) as an element may be used as the feature amount. . For example, the magnitude of the myoelectric potential represented by the myoelectric potential signal, a set having the magnitude of the myoelectric potential as an element, or the heart rate represented by the electrocardiographic signal may be used as the feature amount. Specifically, for example, the operation timing is a specific time t, and the myoelectric potential magnitude Mi (t) or M1 (t), which is indicated by the myoelectric potential signal Vi (t) obtained at the I measurement points, .., MI (t) may be a feature amount. For example, when the operation timing is a specific time interval [t1, t2], a set of myoelectric potential magnitudes Mi (t) in the time interval [t1, t2] may be used as the feature amount. A set of Mi (T) obtained by averaging the size Mi (t) for each predetermined time interval T may be used as the feature amount. Alternatively, the value corresponding to the myoelectric potential (or cardiac potential) represented by the myoelectric potential signal (or cardiac potential signal) is converted into the frequency domain, or the set including such a value converted into the frequency domain as an element is characterized. It may be an amount.

<< Example 1 of Feature Value Derived from Both Motion Timing Image and Myoelectric Potential (or Cardiac Potential) >>
A relative value between a feature amount derived from an image and a time change amount of a value corresponding to a myoelectric potential (or cardiac potential) represented by a myoelectric potential signal (or cardiac potential signal), or a set having such a relative value as an element It is good also as a feature-value. For example, the relative value between H (x, y, t) and ΔMi (t) described above, the relative value between H (x, y, T) and ΔMi (T), or such relative value as an element. It is good also considering the set to be a feature-value. Specifically, for example, the relative value (for example, H (x) of H (x, y, t) and ΔMi (t) of a single or plural coordinates of a motion part involving a muscle near the measurement part of channel i. x, y, t) / ΔMi (t)) and relative values of such H (x, y, T) and ΔMi (T) (for example, H (x, y, T) / ΔMi (T) ) Or a set having such relative values as elements may be used as the feature amount. Further, as described above, relative values of H (x, y, T) and Mi (T) obtained by time averaging for each time interval T, or H (x, y, T) and Mi (T). Or a set having such a relative value as an element may be used as the feature amount. Further, in these examples, a feature amount obtained by replacing H (x, y, t), ΔMi (t), H (x, y, T), and ΔMi (T) with values in the frequency domain may be used. Alternatively, the relative value between the feature amount derived from the image and the time change amount of the value corresponding to the myoelectric potential (or cardiac potential) represented by the myoelectric potential signal (or cardiac potential signal) converted into the frequency domain, or A set having a relative value as an element may be used as the feature amount.

<< Example 2 of Feature Value Derived from Both Motion Timing Image and Myoelectric Potential or Cardiac Potential >>
A relative value between a feature value derived from an image and a value corresponding to the myoelectric potential (or electrocardiogram) represented by the myoelectric potential signal (or electrocardiogram signal) or a set having such a relative value as an element can be used as the feature value. Good. For example, the above-described set having H (x, y, t) and ΔMi (t) as elements, the set having H (x, y, T) and ΔMi (T) as elements, and H (x, y, t), ΔMi (t), and Mi (t) as elements and a set having H (x, y, T), ΔMi (T), and Mi (T) as elements Also good. Furthermore, a set obtained by adding the feature amount of the feature amount example 1 derived from both the action timing image and the myoelectric potential (or electrocardiogram) to such a set may be used as the feature amount. Further, in these examples, a feature amount obtained by replacing H (x, y, t), ΔMi (t), H (x, y, T), and ΔMi (T) with values in the frequency domain may be used.
The feature quantity of the operation timing obtained as described above is stored in the feature quantity storage unit 113.

<Image generation processing>
A time-series image obtained by photographing a series of motions (for example, pitching motion) of a specific motion subject with the camera and a series of muscle activities (or heart activity) of the motion subject are displayed in the motion timing verification unit 114. A myoelectric signal (or electrocardiogram signal) representing a time-series myoelectric potential (or electrocardiogram) obtained by measurement is input. These images and myoelectric potential signals (or electrocardiographic signals) were obtained by a single action subject repeatedly performing the above-described series of actions a plurality of times (obtained in a plurality of action processes by a single action subject). Or obtained by performing the above-described series of actions one or more times in sequence (obtained in a plurality of action processes by a plurality of action subjects). Thing). The plurality of myoelectric potentials (or cardiac potentials) may be obtained at a single measurement location of the operation subject, or the myoelectric potential may be obtained at a plurality of measurement locations. The operation timing collating unit 114 uses the feature amount read from the feature amount storage unit 113, and obtains from the input image and myoelectric potential signal (or cardiac potential signal) from the time-series image of the operation subject at a specific operation timing. A plurality of images of the action subject are extracted, and a plurality of action subject myoelectric potentials (or cardiac potentials) obtained at a specific action timing are extracted from the time series myoelectric potential (or cardiac potential) of the action subject. The operation timing collation unit 114 obtains muscle activity corresponding to each of the plurality of extracted myoelectric potentials (or cardiac potentials), and stores the plurality of images and muscle activity (or cardiac activity) in the collation result storage unit 115. To do. Hereinafter, details of this processing will be exemplified.

≪Example of operation timing verification details≫
FIG. 8 is a block diagram illustrating an example of the configuration of the operation timing verification unit 114.
The feature amount extraction unit 1141 of the operation timing verification unit 114 indicates a time-series image and a time-series myoelectric potential in a time interval [ts, te] (where ts <te) where the operation timing verification process is unprocessed. At least one of myoelectric potential signals (or cardiac potential signals) is input. The time interval [ts, te] is a time interval in which a series of operations (for example, a pitching operation) is performed, or a time interval assumed to be a time interval. The determination of the time interval [ts, te] may be performed based on, for example, an input from the user, may be performed by dividing every predetermined time, or may be input time series It may be performed based on information representing a time interval added to the image. For a plurality of times or time intervals belonging to the time interval [ts, te], the feature quantity extraction unit 1141 has a myoelectric potential (or an image and / or myoelectric potential signal (or cardiac potential signal) represented by the time or time interval. The feature quantity derived from the electrocardiogram is calculated and output. However, the feature amount extraction unit 1141 calculates the feature amount by the same method as the feature amount calculation unit 112 described above. The feature amount is sent to the similarity calculation unit 1142.

  The similarity calculation unit 1142 calculates, for each time or each time interval, the similarity between the obtained feature amount at each time or each time interval and the feature value of the operation timing read from the feature amount storage unit 113. Output. When the feature amount storage unit 113 stores a plurality of feature amounts at the operation timing, the similarity calculation unit 1142 calculates and outputs the similarity for each feature amount at the operation timing. The similarity is sent to the operation timing detection unit 1143.

  The operation timing detection unit 1143 detects a time or a time interval in which the similarity is a peak within the time interval [ts, te]. When the similarity of a plurality of operation timings is sent to the operation timing detection unit 1143, the time or time interval that is the peak for the similarity of each operation timing is detected. In addition, you may impose the restriction that the similarity of a peak exceeds a predetermined threshold value. The operation timing detection unit 1143 outputs the detected time or time interval as a time or time interval representing the operation timing. The time or time interval representing the detected motion timing is sent to the image extraction unit 1144 and the muscle activity extraction unit 1145.

  The image extraction unit 1144 receives a time-series image in the time interval [ts, te] and a time or time interval representing the operation timing. The image extraction unit 1144 extracts an image (an image obtained at a specific operation timing) in a time or time interval representing the operation timing, and stores it in the matching result storage unit 115 (FIG. 7).

  The muscle activity level (or cardiac activity level) extracting unit 1145 represents a myoelectric signal (or cardiac potential signal) representing a time-series myoelectric potential (or cardiac potential) and an operation timing in the time interval [ts, te]. A time or time interval is entered. The muscle activity level (or cardiac activity level) extraction unit 1145 extracts a myoelectric potential signal at a time or time interval representing motion timing, and outputs a myoelectric potential or cardiac potential (or electrocardiographic potential) represented by the extracted myoelectric potential signal (or electrocardiographic signal). A value corresponding to an action subject's myoelectric potential or cardiac potential obtained at a specific action timing is calculated as a muscle activity level (or cardiac activity level) and stored in the verification result storage unit 115. Examples of “muscle activity (or cardiac activity)” are as described above. For example, an absolute amplitude value or a mean square amplitude value of myoelectric potential can be used as the muscle activity.

  The end determination unit 1146 determines whether an unprocessed time interval [ts, te] exists in the time-series image and myoelectric potential signal (or electrocardiographic signal) input to the operation timing verification unit 114. Here, if there is an unprocessed time interval [ts, te], the process returns to the beginning of the operation timing verification process. On the other hand, when there is no unprocessed time section [ts, te], the process of the operation timing verification process is terminated.

As described above, the collation result storage unit 115 stores a plurality of motion subject images obtained at specific motion timings and muscle activity levels corresponding to motion subject myoelectric potentials obtained at the motion timings ( << End of Example of Details of Operation Timing Matching >>
When all the images and the muscle activity levels at the operation timing are stored in the collation result storage unit 115, the next step is to generate a composite image.

  The superimposed image creation unit 116 (FIG. 7) reads out a plurality of images of the operation subject stored in the collation result storage unit 115 and superimposes these images (one or a plurality of operation subjects obtained at a specific operation timing). Are generated and output). The superimposed image is sent to the synthesis unit 118. For example, when the output of the signal generation unit 141 is an image (or video) signal to be superimposed, the output of the signal generation unit 141 may be input to the synthesis unit 118 as an image (or video) signal. For example, when the output of the signal generation unit 141 is character data to be superimposed on an image or the like, the output of the signal generation unit 141 may be input to the superimposed image creation unit 116.

  The muscle activity level (or cardiac activity level) image generation unit 117 reads at least a plurality of muscle activity levels (or cardiac activity levels) stored in the verification result storage unit 115, and the plurality of myoelectric potential levels (or cardiac potentials). A muscle activity level image (or a heart activity level image) derived from (degree) is generated and output. Specific examples of muscle activity level images (or cardiac activity level images) are shown below.

≪Example 1 of muscle activity level image≫
The muscle activity level image (or cardiac activity level image) includes, for example, an image representing a value obtained from a relationship between a plurality of muscle activity levels (or a plurality of cardiac activity levels). At this time, the value obtained from the relationship between the plurality of muscle activity levels (or the plurality of cardiac activity levels) is the value obtained from the relationship between the plurality of muscle activity levels (or the plurality of cardiac activity levels) measured at a plurality of times. Including. For example, the muscle activity level image (or cardiac activity level image) includes (1-a) an image representing variations in a plurality of muscle activity levels (or cardiac activity levels), and (1-b) a plurality of muscle activity levels (or heart rates). At least one of an image representing a relative value of (activity level) and (1-c) an image representing variation of relative values of a plurality of muscle activity levels (or cardiac activity levels). Examples of images representing relative values and their variations are the aforementioned contour plot, time series graph, and histogram.

  (1-a) An image representing a variation in a plurality of muscle activity levels (or cardiac activity levels) is, for example, a muscle activity level at a specific motion timing (or a sequence of motions performed a plurality of times by the same motion subject (or It is an image showing the variation of (heart activity). Such an image may be obtained for each measurement point of myoelectric potential. Another example is an image representing variations in a plurality of muscle activity levels corresponding to myoelectric potentials obtained at a plurality of measurement locations of the same action subject. Another example is an image representing variation in muscle activity (or cardiac activity) at a specific motion timing in a series of motions performed by a plurality of motion subjects. Another example is an image representing a variation in a plurality of muscle activity levels (or cardiac activity levels) corresponding to myoelectric potentials (or cardiac potentials) obtained at measurement points of a plurality of motion subjects. The meaning of “variation” is as described above, and is, for example, variance or standard deviation.

  (1-b) An image representing a relative value of a plurality of muscle activity levels (or cardiac activity levels) is, for example, a muscle activity level (or a specific activity timing) in a series of motions performed a plurality of times by the same motion subject (or It is an image showing the relative value of (heart activity). Specifically, for example, an image representing a relative value between a muscle activity level (or cardiac activity level) in a first motion and a muscle activity level (or cardiac activity level) in a motion performed after the second time. . Such an image may be obtained for each measurement point of myoelectric potential. Examples of relative values are as described above, and are “difference” with positive / negative distinction, “difference” without positive / negative distinction, “square error”, or “ratio”, or any of these function values. is there. Another example is an image representing a relative value of a plurality of muscle activity levels corresponding to a myoelectric potential obtained at a measurement location of the same action subject. Specifically, for example, the image represents a relative value between the muscle activity level corresponding to the myoelectric potential of the main muscle and the muscle activity level corresponding to the myoelectric potential of the antagonist muscle. This image can represent “force”. In particular, in the image representing the value obtained by dividing the muscle activity corresponding to the myoelectric potential of the antagonist muscle by the muscle activity corresponding to the myoelectric potential of the main muscle, the “force” can be clearly expressed. Another example is an image representing a relative value of muscle activity (or cardiac activity) at a specific motion timing in a series of motions performed by a plurality of motion subjects. Another example is an image representing a relative value of a plurality of muscle activity levels (or cardiac activity levels) corresponding to myoelectric potentials (or cardiac potentials) obtained at measurement points of a plurality of motion subjects.

  (1-c) An example of an image representing a variation in relative values of a plurality of muscle activity levels (or cardiac activity levels) is the above-described (1-b) relative value of a plurality of muscle activity levels (or cardiac activity levels). It is an image showing the variation of a specific example. As a specific example, it is an image representing a variation in relative value between the muscle activity corresponding to the myoelectric potential of the main muscle and the muscle activity corresponding to the myoelectric potential of the antagonist muscle. More specifically, for example, the image represents a variation in a value obtained by dividing the muscle activity corresponding to the myoelectric potential of the antagonist muscle by the muscle activity corresponding to the myoelectric potential of the main muscle. In these specific examples, variation in “force” can be expressed.

≪Example 2 of muscle activity image≫
The muscle activity (or cardiac activity) image generation unit 117 further reads out a plurality of images of the subject of action (a plurality of images obtained at a specific operation timing) stored in the matching result storage unit 115, An integrated feature amount derived from both each of the images and each of a plurality of myoelectric potentials (or cardiac potentials) obtained at a specific operation timing is obtained, and a muscle activity image derived from the plurality of integrated feature amounts ( Alternatively, a heart activity level image) may be generated. An example of the integrated feature amount is “a feature amount derived from both an image and a myoelectric potential (or cardiac potential)” exemplified as the feature amount calculated by the feature amount calculating unit 112 described above. For example, the integrated feature value corresponds to a value corresponding to an image of an action site involving a muscle in the vicinity of a measurement location of myoelectric potential (or cardiac potential) and a myoelectric potential (or cardiac potential) obtained at the measurement location. This is a feature amount derived from the muscle activity level (or the heart activity level). The feature amount generated by the feature amount calculation unit 112 and the feature amount generated by the muscle activity (or heart activity) image generation unit 117 may be the same as or different from each other. Examples of muscle activity level images (or cardiac activity level images) derived from a plurality of integrated feature values include an image representing a variation in a plurality of integrated feature values, an image representing a relative value of a plurality of integrated feature values, and a plurality of integrated features. It is an image showing the dispersion | variation in the relative value of quantity. Examples of such muscle activity level images (or cardiac activity level images) are the aforementioned contour plot, time series graph, and histogram.

≪Example 3 of muscle activity level image≫
In addition, any one of the images in Example 1 of the above-described muscle activity image (or cardiac activity image) and any of the images in Example 2 of the muscle activity image (or cardiac activity image) may be generated. .
The synthesizer 118 synthesizes the input superimposed image and muscle activity level image (or cardiac activity level image) to generate and output a synthesized image. It may be a composite image in which the superimposed image and the muscle activity level image (or cardiac activity level image) are superimposed, or may be a composite image in which these are not superimposed. When superimposing the superimposed image and the muscle activity level image (or cardiac activity level image), the muscle activity level image (or cardiac activity) at or near the position of the muscle corresponding to the muscle activity level image (or cardiac activity level image). It is desirable to place an image). Thereby, it becomes easy to grasp the spatial positional relationship between the motion and the muscle activity. The image presentation device 12 displays this composite image. As a result, an image of an action subject at a specific action timing and muscle activity (or cardiac activity) can be displayed simultaneously.

FIG. 9 is a block diagram illustrating another example of the configuration of the synthesized sound / image generation unit.
First, an outline will be described. In this embodiment, an acoustic signal representing myoelectric potential data, for example, information derived from biological information such as a plurality of myoelectric potentials (in other words, derived from a plurality of myoelectric potentials) is output. . “Information obtained from a relationship between a plurality of values” is, for example, a statistical value or a relative value regarding the relationship between a plurality of values. Thus, by using the acoustic signal that directly represents “information obtained from the relationship between a plurality of values derived from a plurality of myoelectric potentials”, information based on the relationship between a plurality of muscle activities can be clearly presented.

  The “plural myoelectric potentials” may be obtained by a plurality of motion processes by one or a plurality of motion subjects, or may be obtained at a plurality of measurement points of a single or a plurality of motion subjects. It may be obtained by both of them, or may be obtained by a single operation process of a plurality of operation subjects. For example, each of “plural myoelectric potentials” is a time series signal. The “moving subject” may be a person or an animal other than a person. An example of a single “acting subject” is a single actuating subject who is trained in motion. Examples of multiple “acting subjects” include two acting subjects consisting of an acting subject acting as an example and an acting subject performing an action according to the action, an acting subject acting as an example, and an action according to the action. For example, a plurality of action subjects composed of a plurality of action subjects to be performed, or a plurality of action subjects other than the action subject serving as a model.

  “A plurality of values derived from a plurality of myoelectric potentials” may be time-domain values or frequency-domain values. “Information obtained from a relationship between a plurality of values derived from a plurality of myoelectric potentials” includes, for example, variations in the plurality of values, relative values of the plurality of values, variations in relative values of the plurality of values, and a plurality of muscles. This is the relationship (for example, order relationship) between the start times of the activities of a plurality of muscles obtained from the potential.

  The “acoustic signal” is, for example, a time series signal, a relationship between a plurality of muscle activity levels, a plurality of muscle activity level variations, a plurality of muscle activity level relative values, or a plurality of muscle activity level relative value variations, Represents at least one of the following. However, each of the plurality of muscle activity levels corresponds to each of the plurality of myoelectric potentials. “Muscle activity” is a myoelectric potential or a function value thereof. An example of the function value of the myoelectric potential is a function value representing the magnitude of the myoelectric potential, for example, a broad monotonically increasing function value (for example, monotonically increasing function value) of the absolute amplitude value of the myoelectric potential. For example, the absolute amplitude value of the myoelectric potential or the mean square amplitude value in a predetermined time interval can be set as the “muscle activity level”. As the function value of the myoelectric potential, a value representing the magnitude of the myoelectric potential that has been smoothed or removed from the DC component may be used.

  The “variation” of a plurality of muscle activity levels may be a variation in a plurality of muscle activity levels derived from a plurality of myoelectric potentials obtained at the same measurement location of a single motion subject, or a plurality of a single motion subject. It may be a variation in a plurality of muscle activities derived from a plurality of myoelectric potentials obtained at a plurality of muscle potentials, or a plurality of muscles derived from a plurality of myoelectric potentials obtained at the same measurement site of a plurality of motion subjects It may be a variation in activity level, or may be a variation in a plurality of muscle activity levels derived from a plurality of myoelectric potentials obtained at a plurality of measurement locations of a plurality of motion subjects. An example of “variation” is “statistical variation” such as variance and standard deviation.

  The relative value of multiple muscle activities is the multiple muscle activities derived from multiple myoelectric potentials (myoelectric potentials obtained in the process of repeating multiple actions) obtained at the same measurement location of a single motion subject. May be a relative value between a plurality of muscle activity levels derived from a plurality of myoelectric potentials obtained at a plurality of measurement points of a single action subject, or a plurality of It may be a relative value between a plurality of muscle activity levels derived from a plurality of myoelectric potentials obtained at the same measurement location of the motion subject, or a plurality of values obtained at a plurality of measurement locations of a plurality of motion subjects. It may be a relative value between a plurality of muscle activity levels derived from myoelectric potential. Examples of the “relative value” are “difference” with positive / negative distinction, “difference” with no positive / negative distinction, “square error”, or “ratio”, or a function value of any of these.

  The “acoustic signal representing variation” is an acoustic signal representing the magnitude of variation. An example of “acoustic signal representing variation” is (F-1) an acoustic signal obtained by mapping the magnitude of variation to acoustic features (eg, fundamental frequency, tone color, loudness, etc.). Another example of the “acoustic signal representing variation” is (F-2) a multi-channel acoustic signal (for example, left and right acoustic signals to be reproduced in stereo) having a volume ratio that varies depending on the magnitude of variation. The “acoustic signal representing a relative value” is an acoustic signal representing a relative value or its magnitude. In the example of “acoustic signal representing relative value”, “variation magnitude” in the above examples (F-1) and (F-2) is replaced with “relative value” or “relative value magnitude”. An acoustic signal or the like.

  The “multiple myoelectric potentials” include the myoelectric potentials of the main muscles and the myoelectric potentials of the antagonistic muscles corresponding to the main muscles. Corresponds to the relationship between the muscle activity corresponding to the potential, the relative value between the muscle activity corresponding to the main muscle potential and the muscle activity corresponding to the antagonist muscle potential, or the main muscle potential It is also possible to represent at least one of a relative value variation between the muscle activity level to be measured and the muscle activity level corresponding to the myoelectric potential of the antagonist muscle. This “acoustic signal” can represent the “force” of the operation subject. In particular, when this "relative value" is a value obtained by dividing "muscle activity corresponding to myoelectric potential of antagonistic muscle" by "muscle activity corresponding to myoelectric potential of main muscle" (divided value), Activity level (ratio of antagonistic muscle activity to main muscle activity) can be quantified, and "force" can be expressed more clearly. The “antagonist muscle corresponding to the main muscle” means a muscle that relaxes in an operation in which the main muscle contracts. For example, in the operation of bending the wrist, the wrist flexor muscles contract as the main muscles and the extensor muscles relax as the antagonistic muscles. Conversely, in the action of stretching the wrist, the extensor is contracted as the main muscle and the wrist flexor is relaxed as the antimuscle. When bending the wrist, it is desirable that the wrist flexor muscle that is the main muscle is active and the activity of the extensor that is the antagonist muscle is suppressed. However, if the antagonistic muscle is excessively active, it becomes a so-called “force” state and inhibits exercise. By using the relative value of the main muscle activity and antagonistic muscle activity that accompanies the movement, this degree of force can be fed back.

  The “acoustic signal” may represent a relationship at the start of activity of a plurality of muscles obtained from a plurality of myoelectric potentials (for example, a plurality of muscles involved in a motion of a certain subject or a series of motions). The activity start time means the time when the muscle starts the muscle activity. By this “acoustic signal”, information on the start timing of muscle activity (for example, variation in the start timing of muscle activity, relative value, and appropriateness) can be expressed. For example, the “acoustic signal” may represent a variation in “activity start time” of a plurality of muscles, may represent a relative value, may represent a variation in a relative value, or may represent a plurality of muscle activities. The order relationship at the start time may be expressed. The relative value of the “activity start time” may be a relative value of another “activity start time” with respect to any of the “activity start time” of the plurality of muscles, It may be a relative value of “activity start time”. An example of the acoustic signal representing the order relationship at the time of starting the activity of the plurality of muscles is (F-3) an acoustic signal obtained by synthesizing a plurality of sub time-series signals having different acoustic characteristics. However, each of the plurality of sub time series signals corresponds to each of the plurality of lines, and the order relationship of the start points of the plurality of synthesized sub time series signals is the order at the start of the activities of the plurality of lines. Corresponds to the relationship. An example of the sub time series signal is a signal having an arbitrary waveform, for example, a signal representing a sine wave. In addition, (F-4) an acoustic signal having different acoustic characteristics depending on whether the order relationship of the activity start points of the plurality of muscles is appropriate or inappropriate, or (F-5) the activity start time of the plurality of muscles. An acoustic signal closer to a specific acoustic feature may be used as the order relationship becomes closer to an appropriate order relationship. It should be noted that whether or not the order relationship at the start of activity of a plurality of muscles is appropriate depends on the operation. For example, in the ball release operation of the pitching exercise, an order relationship in which muscle activity is started in the order of a muscle that moves the shoulder, a muscle that moves the elbow, a muscle that moves the wrist, and a muscle that moves the finger is appropriate. Such an appropriate order relationship may be set in advance according to the operation. Moreover, you may add the sub time series signal corresponding to the time of the muscular activity start of several muscles (antagonist muscle) which are not concerned with operation | movement or a series of operation | movement. As described above, for example, it is possible to make audible whether a plurality of muscles involved in a predetermined action have started activities in an appropriate order or whether a plurality of muscles unnecessary for the predetermined action have not been activated.

Hereinafter, each embodiment will be described in detail with reference to the drawings.
[Second Embodiment]
In the second embodiment, the relationship between the main muscle activity and the antagonist muscle activity is made audible for the combination of the main muscle and the antagonist muscle involved in the desired motion, and the strength of the motion is fed back.

<Configuration>
As illustrated in FIG. 9, the synthesized sound / image generation unit 142 includes, for example, a muscle activity audible device 21, and the output device 200 includes an acoustic signal presentation device 22. Note that a part of the audible device 21 may be included in the signal generation unit 141. The muscle activity audible device 21 of this embodiment includes a muscle activity level extraction unit 211, a relationship-derived information extraction unit 212, and an acoustic signal synthesis unit 213.

FIG. 10 is a block diagram showing an example of the configuration of the audible device 21.
As illustrated in FIG. 10, the muscle activity level extraction unit 211 includes muscle activity level calculation units 2111-n and 2112-n (where n = 1,..., N, where N is an integer equal to or greater than 1). Have The relationship-derived information extraction unit 212 includes a muscle activity modulation unit 2121-n (where n = 1,..., N). The acoustic signal synthesis unit 213 includes an acoustic signal generation unit 2131-n (where n = 1,..., N) and a synthesis unit 2132. Note that when N = 1, the acoustic signal synthesis unit 213 may not include the synthesis unit 2132. The audible device 21 of the present embodiment includes, for example, a processor (for example, hardware including one or more central processing units (CPUs)) and a memory (for example, random-access memory (RAM) and read-only memory (ROM)). A general-purpose or dedicated computer provided with a) executes a predetermined program. This program may be installed in the computer, or may be recorded in the computer in advance. In addition, some or all of the processing units may be configured using an electronic circuit that realizes a processing function independently instead of an electronic circuit (circuitry) that realizes a functional configuration by reading a program like a CPU. . The acoustic signal presentation device 22 is one or a plurality of speakers or headphones that output an acoustic signal as sound.

<Processing>
Hereinafter, the process of the activity auralization apparatus 21 of this form is demonstrated.
≪Process of muscle activity extraction unit 211≫
A myoelectric potential signal representing the myoelectric potential of the main muscle (hereinafter referred to as “main myoelectric potential signal”) is input to the muscle activity calculating unit 2111-n, and the muscle activity calculating unit 2112-n corresponds to the main muscle. A myoelectric potential signal indicating the myoelectric potential of the antagonistic muscle (hereinafter referred to as “antagonistic myoelectric potential signal”) is input. The set of the muscle activity level calculation unit 2111-n and the muscle activity level calculation unit 2112-n corresponds to a set of main muscles and antagonist muscles involved in a certain motion (hereinafter referred to as “muscle set”). A set of a main muscle potential signal and an antagonistic muscle potential signal is input. These myoelectric potential signals are generated from each muscle, for example, by attaching electrodes to a total of N sets of measurement points (for example, the surface of the main muscle and the corresponding antagonist muscle in the vicinity of the main muscle). It is a time series signal obtained by measuring an action potential. The main muscle potential signal is obtained based on the potential of the electrode at the measurement site near the main muscle, and the antagonistic electron signal is obtained based on the potential of the electrode at the measurement site near the antagonistic muscle.

  The muscle activity level calculation unit 2111-n obtains and outputs a muscle activity level corresponding to the myoelectric potential represented by the input main muscle potential signal (hereinafter, “main muscle activity level”), and outputs the muscle activity level calculation unit 2112-n. Obtains and outputs a muscle activity level (hereinafter referred to as “antagonistic muscle activity level”) corresponding to the myoelectric potential represented by the input antagonistic myoelectric potential signal. Examples of the “muscle activity level” are as described above. For example, the absolute amplitude value or the mean square amplitude value of the myoelectric potential can be used as the muscle activity level. When the input main muscle potential signal and antagonist muscle potential signal are time series signals, the main muscle activity level and antagonist muscle activity level are also time series.

<< Processing of Relationship Origin Information Extraction Unit 212 >>
The active muscle activity level output from the muscle activity level calculation unit 2111-n and the antagonistic muscle activity level output from the muscle activity level calculation unit 2112-n are input to the muscle activity level modulation unit 2121-n. The muscle activity modulation unit 2121-n corresponds to the relative value between the input main muscle activity and the antagonist muscle activity (that is, the muscle activity corresponding to the main muscle myoelectric potential and the myoelectric potential of the antagonist muscle). Relative value between muscle activity levels) is obtained as a “force index”. Examples of “relative value” are as described above. The muscle activity level modulation unit 2121-n obtains, for example, a value obtained by dividing the antagonist muscle activity level by the main muscle activity level as a “force index”. When the main muscle activity level and the antagonistic muscle activity level are in time series, the muscle activity modulation unit 2121-n obtains, for example, a “strength index” at each time, and obtains and outputs the “strength index” time series. To do.

<< Processing of Acoustic Signal Synthesizer 213 >>
The “force index” output from the muscle activity modulation unit 2121-n is input to the acoustic signal generation unit 2131-n. The acoustic signal generation unit 2131-n obtains and outputs an acoustic signal representing the input “force index”. The acoustic signal representing the “force index” is an acoustic signal representing the “force index” or its magnitude. An example of such an acoustic signal is an acoustic signal obtained by replacing the “variation magnitude” in the above examples (F-1) and (F-2) with a “force index” or “a magnitude index”. Signal. When the “force index” is time series, the acoustic signal is also time series. However, it is desirable to change the acoustic characteristics (basic frequency, timbre, etc.) for each set of muscles so that the difference between the muscle sets can be recognized. In other words, it is desirable that the acoustic features of the acoustic signals output from the acoustic signal generation units 2131-1 to 2131 -N are different from each other. For example, an acoustic signal in which a “force index” or its size is mapped to a first acoustic feature, and a difference between muscle pairs is represented by a second acoustic feature (however, the second acoustic feature is different from the first acoustic feature). It may be. Alternatively, a “force index” or a difference in size thereof may be expressed by a difference in acoustic characteristics, and a difference in muscle pairs may be expressed by a difference in channel of an acoustic signal (that is, a difference in speakers outputting an acoustic signal). .

  The acoustic signals output from the acoustic signal generation units 2131-1 to 2131 -N are input to the synthesis unit 2132. The synthesizer 2132 outputs an acoustic signal obtained by synthesizing them. When outputting an N-channel acoustic signal, an acoustic signal obtained by synchronizing the acoustic signals output from the acoustic signal generation units 2131-1 to 2131 -N is output. When N = 1, the synthesis unit 2132 outputs the input acoustic signal as it is. When the output of the signal generation unit 141 is a superposed acoustic signal, it may be input to the synthesis unit 2132 as an acoustic signal. As described above, when the acoustic signal synthesis unit 213 does not include the synthesis unit 2132, the acoustic signal synthesis unit 213 outputs the acoustic signal output from the acoustic signal generation unit 2131-1.

  The acoustic signal output as described above is input to the acoustic signal presentation device 22. The acoustic signal presentation device 22 outputs the input acoustic signal as a sound. In the case of an N-channel acoustic signal, the acoustic signal presentation device 22 outputs the N-channel acoustic signal as sound (for example, stereo reproduction).

<Features of this embodiment>
As described above, the audible device 21 of the present embodiment uses a plurality of myoelectric potentials including the myoelectric potential of the main muscle and the myoelectric potential of the antagonistic muscle corresponding to the main muscle, and the muscle activity level corresponding to the myoelectric potential of the main muscle. And an acoustic signal representing a relative value between the muscle activity corresponding to the myoelectric potential of the antagonistic muscle. Thereby, the degree of “strength” can be presented audibly. In particular, by outputting an acoustic signal representing the value of the muscle activity corresponding to the myoelectric potential of the antagonist muscle divided by the muscle activity corresponding to the myoelectric potential of the main muscle, the degree of “strength” is presented more clearly. it can. In the case of N ≧ 2, the degree of “strengthening” in a plurality of muscle pairs and the correlation between them can be presented audibly. As described above, the user can audibly grasp the degree of “strengthening” corresponding to a plurality of muscle pairs as a difference in how the sound is heard.

  The present invention is not limited to the embodiment described above. For example, the various processes described above are not only executed in time series according to the description, but may also be executed in parallel or individually as required by the processing capability of the apparatus that executes the processes. Needless to say, other modifications are possible without departing from the spirit of the present invention.

  When the above configuration is realized by a computer, the processing contents of the functions that each device should have are described by a program. By executing this program on a computer, the above processing functions are realized on the computer. The program describing the processing contents can be recorded on a computer-readable recording medium. An example of a computer-readable recording medium is a non-transitory recording medium. Examples of such a recording medium are a magnetic recording device, an optical disk, a magneto-optical recording medium, a semiconductor memory, and the like.

  This program is distributed, for example, by selling, transferring, or lending a portable recording medium such as a DVD or CD-ROM in which the program is recorded. Furthermore, the program may be distributed by storing the program in a storage device of the server computer and transferring the program from the server computer to another computer via a network.

  A computer that executes such a program first stores, for example, a program recorded on a portable recording medium or a program transferred from a server computer in its own storage device. When executing the process, this computer reads a program stored in its own recording device and executes a process according to the read program. As another execution form of the program, the computer may read the program directly from the portable recording medium and execute processing according to the program, and each time the program is transferred from the server computer to the computer. The processing according to the received program may be executed sequentially. The above-described processing may be executed by a so-called ASP (Application Service Provider) type service that realizes a processing function only by an execution instruction and result acquisition without transferring a program from the server computer to the computer. Good.

In the above embodiment, the processing functions of the apparatus are realized by executing a predetermined program on a computer. However, at least a part of these processing functions may be realized by hardware.
In addition, various modifications can be made without departing from the scope of the present invention.
In short, the present invention is not limited to the above-described embodiment as it is, and can be embodied by modifying the constituent elements without departing from the scope of the invention in the implementation stage. Further, various inventions can be formed by appropriately combining a plurality of constituent elements disclosed in the embodiment. For example, some components may be deleted from all the components shown in the embodiment. Furthermore, you may combine suitably the component covering different embodiment.

DESCRIPTION OF SYMBOLS 100 ... Feedback system 120 for exercise improvement ... Electrocardiogram sensor 130 ... Psychological state determination apparatus 131 ... Sensing part 132 ... State determination part 132a ... Determinator 132b ... Psychological state memory | storage part 132c ... Switch 132d ... Comparator 140 ... Feedback Device 141 ... Signal generation unit 142 ... Synthetic sound / image generation unit 200 ... Output device

Claims (4)

Determine or store the psychological state of the primary subject based at least on the primary biological information acquired from the primary subject during the predetermined exercise, and store the secondary biological information acquired from the secondary subject during the predetermined exercise. Determine the psychological state of the secondary subject based at least on
Comparing the psychological state of the primary subject and the psychological state of the secondary subject;
According to the comparison result of the psychological state of the primary subject and the psychological state of the secondary subject, the secondary biological information is corrected to a better value when the psychological state is good. Or by exaggerating the secondary biometric information by correcting to a worse value when the psychological state is not indicative of good ,
Exaggerated or dwarf the secondary biological information visually or audibly perceptible feedback method psychological state of outputting the feedback signal converted into the format. The primary biological information and the secondary biological information include at least electrocardiographic data,
The feedback signal is obtained by converting the exaggerated cardiac potential data of the secondary biological information or the reduced cardiac potential data of the secondary biological information into an acoustic signal, an image signal, a video signal, or a combination thereof. The psychological state feedback method according to claim 1.   A psychological state feedback system that executes the psychological state feedback method according to claim 1.   A psychological feedback program for causing a computer to execute the determination, comparison, and output according to claim 1.