JP5574407B2 - Facial motion estimation apparatus and facial motion estimation method - Google Patents

Description

  The present invention relates to a facial motion estimation device and a facial motion estimation method, and more particularly, to a facial motion estimation device and a facial motion estimation method for easily perceiving facial motion.

  Conventionally, facial expressions in dialogue and the like have given additional information to a person and have played an important role in human relationships as one measure for acquiring the person's emotions.

  Therefore, for example, there is a technique related to facial expression recognition of a person photographed by a camera or the like, and a technique for recognizing a facial expression based on a bioelectric potential signal (myoelectricity) obtained from an electrode or the like installed at a predetermined part of the face. Is also disclosed (for example, Patent Documents 1 and 2).

  For example, Patent Document 1 discloses a technique for detecting biological information and exercise information of a user, determining a user situation based on the detected biological information and exercise information, and determining a facial expression or action of a display character corresponding to the user situation. Has been.

  Further, in Patent Document 2, an electrode is disposed at either end of the laughing muscle and zygomatic muscle, and the person is determined based on the myoelectricity of the frequency distribution of 20 to 100 Hz of the laughing muscle and zygomatic muscle detected thereby. A technique for detecting whether a person is laughing or not is shown.

JP 2009-39157 A JP 2006-340986 A

  However, when estimating a person's facial expression from an image taken with a camera or the like, at least the front face must be projected on the screen, for example, when the face crawls or faces the camera In some cases, the facial expression and movement cannot be measured.

  In addition, for example, when measuring a bioelectric potential signal using an electrode to detect a facial expression of a laughing face, conventionally, an electrode is placed on the laughing muscle inside the face and the living body generated by the movement of the laughing muscle Although a potential signal is acquired, this arrangement is not preferable because it is placed in front of the face, which makes it look strange and disturbs the expression itself. Absent.

  Also, for example, as shown in Patent Document 2, when acquiring a biopotential signal from the auricle using headphones or the like, there is a distance from the muscle, so noise and attenuation when propagating through the skin For example, the degree of smile cannot be accurately estimated.

  Furthermore, conventionally, for example, for a person whose facial expression cannot be made well due to facial paralysis or the like, the signal inside the face obtained by the bioelectric potential signal is clearly presented to the person or the like by light or sound rather than by muscle movement. Thus, there has been no wearable device that can be easily seen by others.

  The present invention has been made in view of the above-described problems, and an object of the present invention is to provide a facial motion estimation apparatus and a facial motion estimation method for easily perceiving facial motion.

  In order to solve the above-described problems, the present invention employs means for solving the problems having the following characteristics.

According to a first aspect of the present invention, there is provided a facial motion estimation device for estimating a facial motion based on a bioelectric potential signal obtained from a predetermined face area, for obtaining the bioelectric potential signal for the predetermined face area. Separation by assuming a plurality of electrodes, filter means for filtering each signal obtained from the plurality of electrodes in a predetermined band, and independence of signals preset with respect to the signal obtained by the filter means The facial expression and the degree of the facial expression are estimated from the signal processing means, the machine learning analysis means for performing signal analysis on the signal obtained by the signal processing means by machine learning, and the analysis result obtained by the machine learning analysis means. A facial expression estimation means; and a presentation means for making a preset presentation corresponding to the facial expression and degree of the face, and the predetermined area of the face is: Serial plurality of the wearer's face wheels muscle electrodes, zygomaticus minor muscle, zygomaticus major muscle, among risorius, the left and right sides of the face can be acquired biopotential signals mixed transmitted through the skin of at least two muscles A region on the facial nerve cheek muscle branch , and the signal processing means arranges two pairs of electrodes vertically on the left and right sides of the region on the facial nerve cheek muscle branch on the left and right sides of the face. 4 channel biopotential signals are acquired, and independent components are extracted from the acquired combinations of the 4 channel biopotential signals.

According to the first aspect of the present invention, the facial motion can be easily quantified and perceived. In addition, signals to a plurality of muscles necessary for forming a facial expression can be obtained efficiently and with high time accuracy without depending on the movement of one muscle. In addition, it is not necessary to determine from which part the biopotential signal is acquired, and independent components can be extracted by focusing on only a plurality of biopotential signals acquired from a plurality of electrodes.

According to a second aspect of the present invention, the machine learning analysis means performs analysis using a preset neural network that outputs the type and degree of facial expression corresponding to the input of the bioelectric potential signal of the four channels. It is characterized by performing.

According to the second aspect of the present invention, by using a neural network as machine learning, the facial motion estimation device can be used without limitation for any user with different facial expressions.

The invention described in claim 3 is characterized in that the presenting means presents with a plurality of lights or sounds, and presents the plurality of lights or sounds according to the facial expression and degree of the face. To do.

According to the third aspect of the present invention, the wearer, the person facing the person, the surrounding person, and the like can easily grasp the expression by light or sound. As a result, it is possible to accurately grasp changes in facial expression that are performed unconsciously. Furthermore, since the light and sound are changed according to the type and degree of the facial expression, it is possible to confirm detailed changes in facial expression with high accuracy.

According to a fourth aspect of the present invention, in the facial motion estimation method using a facial motion estimation device that estimates facial motion based on a bioelectric potential signal obtained from a predetermined region of the face, the facial motion estimation device includes a plurality of facial motion estimation devices. A signal acquisition step for acquiring the biopotential signal for a predetermined region of the face using the electrodes of the filter, and a filter step of filtering each signal from the plurality of electrodes obtained by the signal acquisition step in a predetermined band And a signal processing stage for sorting by assuming independence of a preset signal with respect to the signal obtained by the filter stage, and signal analysis by machine learning for the signal obtained by the signal processing stage A machine learning analysis stage, a facial expression estimation stage for estimating a facial expression and degree from the analysis result obtained by the machine learning analysis stage, and A predetermined presentation area corresponding to the facial expression and degree of the facial expression, and the predetermined region of the face includes a ring muscle, a small zygomatic muscle, and a large zygomatic muscle of a wearer of the plurality of electrodes A region on the facial nerve buccal muscles on the left and right sides of the face from which the mixed bioelectric potential signal transmitted through the skin from at least two muscles of the laughing muscle can be obtained, and the signal processing step comprises of the left and right portions of the facial nerve buccinator branch on the side area, on the left and right by placing two pairs of electrodes longitudinally obtains biopotential signals of four channels, acquired biological potential of the 4-channel An independent component is extracted from the combination of signals.

According to the fourth aspect of the present invention, the facial motion can be easily quantified and perceived. In addition, signals to a plurality of muscles necessary for forming a facial expression can be obtained efficiently and with high time accuracy without depending on the movement of one muscle. In addition, it is not necessary to determine from which part the biopotential signal is acquired, and independent components can be extracted by focusing on only a plurality of biopotential signals acquired from a plurality of electrodes.

According to a fifth aspect of the present invention, in the machine learning analysis stage, analysis is performed using a preset neural network that outputs the type and degree of facial expression corresponding to the input of the bioelectric potential signal of the four channels. It is characterized by performing.

According to the invention described in claim 5 , by using a neural network as machine learning, it can be used without limitation for any user with different facial expressions.

In the invention described in claim 6 , in the presenting step, the presenting means is used to present with a plurality of lights or sounds, and the plurality of lights or sounds are changed according to the facial expression and degree. It is characterized by presenting.

According to the sixth aspect of the present invention, the wearer, the person facing the person, the surrounding person, and the like can easily grasp the facial expression by light or sound. As a result, it is possible to accurately grasp changes in facial expression that are performed unconsciously. Furthermore, since the light and sound are changed according to the type and degree of the facial expression, it is possible to confirm detailed changes in facial expression with high accuracy.

  According to the present invention, facial movement can be easily perceived.

It is a figure for demonstrating the arrangement position of an electrode. It is a figure which shows an example of the external form of the facial motion estimation apparatus in this embodiment. It is a figure for demonstrating the modification of a facial motion estimation apparatus. It is a figure which shows an example of a function structure of the facial motion estimation apparatus in this embodiment. It is a figure for demonstrating the outline | summary of the signal processing in a facial motion estimation apparatus. It is a figure which shows an example of a signal processing module. It is a figure which shows an example of the block diagram of ICA. It is a figure which shows the example of a production | generation of the basic data in the facial expression estimation in this embodiment. It is FIG. (1) which shows an example of the experimental result to which this embodiment is applied. It is FIG. (2) which shows an example of the experimental result to which this embodiment is applied. It is FIG. (3) which shows an example of the experimental result to which this embodiment is applied.

<About the present invention>
The present invention identifies facial expressions, the contents of actions and their degrees based on bioelectrical signals (BES) related to the movement of facial muscles. Specifically, the conventional arrangement of electrodes on the front face of the face impedes facial expression and facial impression, so in the present invention, the biopotential signal on the facial nerve cheek muscle branch on the side of the face By acquiring, the contents of facial expressions and actions and their degree are identified.

  In the measurement of the bioelectric potential signal on the side of the face, it is necessary to remove the influence of the neck movement and the masseter muscle, but in the present invention, signals of two or more channels obtained using a plurality of electrodes are used. By using independent component analysis and neural network, etc., facial expressions and actions (for example, laughter, fear, neutral (true face), anger, crying, yawning, etc.) and the degree of facial expressions and actions (for example, It is possible to determine the degree of laughing, the size of the yawning mouth, etc.) and present it by the presenting means.

  In addition, the present invention accumulates the facial expressions and actions described above for a predetermined period and statistically uses the accumulated data, for example, for each wearer of the apparatus, for example, one day, one week, one month. It is possible to easily and accurately acquire a change in facial expression that the wearer is not aware of, such as how much he / she laughs within a predetermined period.

  In addition, in the present invention, in order to clearly present a signal inside the face to the person or the like by light or sound, for example, by having a light display function by a light emitting element or the like and a sound output function by a speaker or the like, it is obtained from a biopotential signal Depending on the facial expression and the type of action determined from the information to be displayed, the contents of the facial expression can be obtained even if the subject does not appear to change the facial expression by optical display or sound output. Can be clearly understood.

  Furthermore, the present invention, for example, by providing the present invention as a wearable device, enables immediate and reliable visual and acoustic feedback to the wearer and the surrounding person, so that it is convenient for the user with an inexpensive configuration. It is possible to provide a facial motion estimation device with improved performance.

  Hereinafter, embodiments in which the facial motion estimation device and the facial motion estimation method according to the present invention are suitably implemented will be described with reference to the drawings.

<Operation method of the present invention>
First, an operation method according to the present invention will be described. The present invention relates to a facial motion estimation method using a wearable device configured to identify a real-time emotional facial expression or motion from biopotential signals obtained from a plurality of electrodes and continuously present its contents. It is.

  Therefore, in the present invention, an electrode for acquiring a bioelectric potential signal is arranged at a predetermined position near the temple on the facial nerve cheek muscle branch. Note that a plurality of electrodes are used so that at least two channels (one channel is two electrodes) or more can be acquired in total on the left and right. The signals to be acquired are preferably two channels on the left and right, and a total of four channels. This makes it possible to recognize not only the presence / absence of laughter, but also the degree of smile, other facial expressions such as an angry face, and actions such as yawning.

  FIG. 1 is a diagram for explaining the arrangement positions of the electrodes. As shown in FIG. 1 (a), a facial expression of the wearer 10 is formed by a plurality of muscles, and among them, the ring muscle 11 and the small cheekbone, which are considered to contribute particularly to human facial expressions. A bioelectric potential signal generated from the movement of one or more muscles among the muscle 12, the great zygomatic muscle 13, and the laughing muscle 14 is acquired. Note that two or more biopotential signals to be acquired are preferable. As a result, it is possible to efficiently and highly accurately acquire signals to a plurality of muscles necessary for forming a facial expression without depending on the movement of one muscle.

  Further, in the present embodiment, the position where the plurality of electrodes are installed is selected as a place on the face of the wearer 10 with as little displacement as possible to generate a facial expression, and a part capable of acquiring a bioelectric potential signal capable of estimating the facial expression Select as That is, in this embodiment, as shown in FIG. 1B, a plurality of electrodes 16 are arranged in a predetermined region 15 near the temple.

  A biopotential signal is acquired by a potential difference between a pair of electrodes, that is, a signal acquired by two electrodes. Therefore, as shown in FIG. 1B, the electrodes 16-1 and 16-2 constitute one biopotential signal acquisition unit (one channel), and the electrodes 16-3 and 16-4 constitute one biopotential. A signal acquisition unit (one channel) is configured. In the present embodiment, the biopotential signal acquisition unit acquires biopotential signals of a total of 4 channels on both the left and right sides, two on each of the left and right sides. The number of electrodes installed in the present invention is not limited to this.

  As shown in FIG. 1B described above, the position of the electrode 16 is set in the vicinity of the temple, so that the person facing the person does not feel strange and the expression of the wearer 10 is inhibited by the electrode. Without this, it is possible to obtain a mixed signal of ripple-shaped electrode signals transmitted from the face ring muscle 11, the small zygomatic muscle 12, the large zygomatic muscle 13, and the laughing muscle 14 through the skin or the like.

  Further, in the present embodiment, the position of the electrode obtained from a plurality of arranged electrodes and the phase difference of the signal can specify the muscle signal where the acquired signal is directly, such as noise. It is possible to acquire a mixed signal for effectively estimating the facial expression of the facial ring muscle 11, the small zygomatic muscle 12, the large zygomatic muscle 13, and the laughing muscle 14 without being influenced by the above.

  Note that the number of biopotential signal channels acquired in the present embodiment may be plural, and is not particularly limited in the present invention. However, if the number is too large, it may interfere with the face when worn, so about 2 to 8 channels. 4 channels are particularly preferable.

  Further, from the viewpoint of realizing an arrangement that does not interfere with the face described above, one electrode preferably has a diameter of about 2 to 10 mm, for example, and a distance between the electrodes forming the pair is preferably about 20 mm, for example. Moreover, although it is preferable that each electrode 16-1 to 16-4 is arrange | positioned substantially linearly vertically with respect to the face, in this invention, it does not restrict | limit in particular.

<External form of facial motion estimation device>
Next, an example of the external form (interface) of the facial motion estimation apparatus in the present embodiment will be described. FIG. 2 is a diagram illustrating an example of an external form of the facial motion estimation device according to the present embodiment. Here, FIG. 2A is a diagram for explaining a state of wearing by the wearable facial motion estimation device 20 and the wearer 10, and FIG. 2B is an end portion of the facial motion estimation device 20. It is a figure which shows the mode of formation of.

  As shown in FIGS. 2A to 2C, the facial motion estimation apparatus 20 shown in FIG. 2 has an arcuate shape on the back of the head, and the left and right end portions are near the left and right temples. The end of the facial motion estimation device 20 is formed vertically long so as to be installed at the position of the electrode 16 described above, and a plurality of electrodes 16 are provided in pairs there.

  That is, as shown in FIGS. 2 (A)-(a) and 2 (B)-(a), the electrode 16 described above is disposed on the inner side of the end portion of the facial motion estimation device 20 mounted near the temple. Is provided. These electrodes may be only one of the left and right, or both. In the case of one of the left and right, for example, when rehabilitating a patient who is paralyzed on either the left or right side, an electrode is installed only on the paralyzed side so that a signal on one side is received. It can be used in any case.

  Further, as shown in FIGS. 2 (A)-(a) and 2 (B)-(b), outside the end portion of the facial motion estimation device 20, light composed of a plurality of light emitting elements such as LEDs. The presenting means 21 and the sound presenting means 22 such as a speaker which is a voice output function are configured.

  The light presenting means 21 is provided with LEDs (LEDs 21-1, 21-2 in FIGS. 2 (B)-(b)) made of a plurality of different colors or the same color, or a combination thereof. For example, the number or position of light to be lit is changed, blinked, or blinked based on preset conditions according to the facial expression analyzed by the bioelectric potential signal from the electrode 16, the content of the action, and the degree thereof. Can be presented at different intervals. Similarly, the sound presenting means 22 can output the sound by changing the loudness, sound type, voice, etc. according to the analyzed facial expression, the content of the action, and its degree.

  Thereby, for example, by attaching the facial motion estimation device 20 to a patient or an elderly person whose facial function is reduced or stopped, the facial expression obtained by analyzing the biopotential signal from the electrode 16, the content of the motion, The degree can be easily transmitted to other people according to light and sound.

  As described above, the facial motion estimation device 20 according to the present embodiment allows the wearing type in a form that does not disturb the facial expression as much as possible in consideration of the position of the electrode 16.

  When the facial motion estimation device 20 is worn, it is preferable that the contact surface with the side surface of the face is fixed at a predetermined position using a gel-like adhesive or the like. Further, the facial motion estimation device 20 is made of a flexible material, for example, a material such as resin, because the shape of the head for each wearer 10 is different. In the present embodiment, a general adjustment mechanism may be provided so that the length of the arc portion can be adjusted according to the size of the head.

<Modification>
Here, FIG. 3 is a diagram for explaining a modification of the facial motion estimation apparatus. As a modification, for example, as shown in FIG. 3A, the facial motion estimation device 30-1 is engaged with the uneven shape of the ear so as to cover the entire ear of the wearer, or hooked to the base of the ear. Then, it is fixed and attached by hooking the locking part. In addition, the facial motion estimation device 30-1 is provided with a plurality of electrodes that are installed in the vicinity of the temple in a fixed state. Further, the facial motion estimation device 30-1 includes a light presentation unit 31-1 including a plurality of light emitting elements such as LEDs, a speaker, and the like. The sound presenting means 32-1 is arranged on the front side of the face (a position where it can be seen by the person facing you). Thereby, it is possible to easily grasp the contents presented from the person to be faced or the mirror.

  Further, the facial motion estimation device 30-2 in FIG. 3B is also locked to the ear in the same manner as the facial motion estimation device 30-1 described above, and the light presentation means 31-1 including the plurality of light emitting elements described above. Or sound presenting means 32-1 such as a speaker is arranged on the side surface. In addition, the facial motion estimation device 30-2 is provided with a plurality of electrodes 16 in the vicinity of the temple, and the processing in this embodiment is performed. Thereby, the content of the facial expression presented from the third party who is not facing can be confirmed.

Note that the facial motion estimation devices 30-1 and 30-2 shown in FIGS. 3 (A) and 3 (B) are combined with one or more of the light presentation means 31 and / or the sound presentation means 32, respectively. Can be installed. Also,
In addition, the facial motion estimation device according to the present embodiment incorporates, for example, a solar battery, a rechargeable lithium battery, and the like, and each process is executed based on electric power supplied therefrom.

<Facial motion estimation device: functional configuration example>
Next, an example of a functional configuration of the facial motion estimation device in the present embodiment will be described with reference to the drawings. FIG. 4 is a diagram illustrating an example of a functional configuration of the facial motion estimation device according to the present embodiment. In the following description, the facial motion estimation device 20 shown in FIG. 2 is used as an example of the facial motion estimation device.

  The facial motion estimation device 20 shown in FIG. 4 includes an electrode signal input unit 41, a storage unit 42, a filter unit 43, a signal processing unit 44, a machine learning analysis unit 45, a facial expression estimation unit 46, and a presentation unit 47. And a transmission / reception means 48 and a control means 49. The presenting means 47 specifically includes a light presenting means 47-1 and a sound presenting means 47-2.

  The electrode signal input means 41 receives a signal of each electrode from a plurality of electrodes on the left and right sides or one side of the face. In addition, an electrode shows an electrical potential difference detection electrode etc., for example, and is comprised by a pair of electrodes as mentioned above. Note that each of the pair of electrodes in the present embodiment includes both a single electrode or a plurality of electrodes. That is, the case where each of a pair of electrodes is constituted by handling them electrically even if they are provided separately and independently is included.

  The accumulating unit 42 accumulates a signal (biopotential signal) from the electrode obtained by the electrode signal input unit 41 together with time information, and performs a temporal or statistical analysis using the machine learning analysis unit 45 or the like. To guess and present the pattern and degree of facial expressions and actions.

  The storage means 42 is the result of processing by the filter means 43, the result of signal processing by the signal processing means 33, the result of analysis by the machine learning analysis means 45, the content presented by the light presentation means 46, Various kinds of information necessary for implementation in the present embodiment are accumulated from other external devices acquired by the transmission / reception means 47 via the communication network. The storage unit 42 also stores the above-described various information acquired by the transmission / reception unit 47. Furthermore, the storage means 42 can read out various data stored as required.

  The filter means 43 performs a smoothing process to remove noise and the like contained in the biopotential signal obtained from the electrode signal input means 41 and the storage means 42. Specifically, the filter means 43 performs a band pass filter (BPF), a notch filter (BEF), etc. using the acquired bioelectric potential signal, and performs signal filtering in a predetermined band. For example, a band of about 5 to 400 Hz is used as the predetermined band.

  The signal processing unit 44 extracts a significant signal from the signal obtained from the filter unit 43 by using independent component analysis (ICA). That is, the signal processing unit 44 separates the signal obtained by the filter unit 43 by assuming independence of a preset signal. Details of the independent component analysis in the signal processing means 44 will be described later.

  The machine learning analysis unit 45 performs machine learning based on the significant signal obtained by the signal processing unit 44. Specifically, the machine learning analysis unit 45 discriminates the above-described significant signal using machine learning and converts it into a signal capable of estimating a desired facial expression. The machine learning analysis unit 45 can statistically analyze the above-described analysis based on the biopotential signal stored together with the time information by the storage unit 42.

  Here, as an example of machine learning in the machine learning analysis unit 45, for example, a neuron network (Artificial Neural Network; ANN) can be used. In the present embodiment, machine learning can be performed by a technique such as multiple regression analysis or a support vector machine, for example, other than the ANN described above. A detailed description of the machine learning analysis unit 45 will be described later.

  The facial expression estimation means 46 estimates the facial expression and motion of the wearer 10 based on the signal that can be estimated by the machine learning analysis means 45.

  Specifically, the facial expression estimation means 46 determines which of a plurality of preset facial expressions and actions, and information about their degree, according to the preset amplitude and frequency of the signal, waveform pattern, etc. Select if applicable and make an estimate. In this embodiment, the signal amplitude, frequency, waveform pattern and a plurality of facial expressions, movements, and degrees are generated as correspondence tables and stored in the storage means 42, etc. The content of the corresponding data can be easily estimated using.

  Thus, the facial expression estimation means 46 can estimate the degree of a single facial expression such as “laughing face”, for example, and can realize detailed facial expression recognition.

  Furthermore, the facial expression estimation means 46 first wears the facial motion estimation device 20 for each wearer, and then performs various facial expressions such as a smile or an angry face in a predetermined time (for example, about 2 to 4 seconds). The above-described table (basic data) can be updated by intentionally performing an operation such as yawning and using the biopotential signal from each electrode 16 obtained at that time. Accordingly, facial expression recognition can be performed with high accuracy even when the laughing method differs for each wearer. In the present embodiment, in addition to the above-described contents, for example, the recognition process for the intra-oral masticatory movement can be performed using a technique similar to the technique described above.

  The presentation unit 47 performs a predetermined presentation based on the facial expression estimation result obtained by the facial expression estimation unit 46. Specifically, the presenting means 47 presents the light to the outside by the light presenting means 47-1 based on the facial expression estimation result. In addition, as the light presentation means 47-1, the color corresponding to the facial expression estimated by the facial expression estimation means 46 among a plurality of LEDs of the same color or different colors, or one or a plurality of LEDs among the plurality of LEDs, for example. Is displayed or blinked at predetermined time intervals. Moreover, the light presentation means 47-1 can make the blinking interval faster or slower depending on the facial expression, action, and degree thereof.

  As a result, for example, if the user is laughing loudly with a loud mouth, it can be flashed quickly, and if it is about smiling, it can be presented.

  Furthermore, the presentation unit 47 can present a sound corresponding to the facial expression estimated by the sound presentation unit 47-1 based on the facial expression estimation result. This sound may be a human voice. In other words, when you are laughing, you can output a laughing voice, and you can change the size of the laughing voice according to the degree. "", "Ahaha ...", etc. can be output corresponding to the degree. Moreover, the presentation means 47 can also present simultaneously combining both light and an audio | voice.

  The transmission / reception means 48 transmits the data stored by the storage means 42 using wired or wireless to an external device connected via a communication network or the like, inputs various necessary data in the present embodiment, and performs various executions. The result can be output to an external device.

  The control means 49 controls the entire components of the facial motion estimation device 20. Specifically, for example, the control unit 49 performs electrode signal input processing, filter processing, signal processing, machine learning analysis processing, facial expression estimation processing, presentation processing, transmission / reception processing to realize facial motion presentation in the present embodiment. Each control is performed.

<Processing contents in signal processing means 44>
Next, specific processing contents in the signal processing means 44 described above will be described. FIG. 5 is a diagram for explaining an outline of signal processing in the facial motion estimation apparatus. As shown in FIG. 5, the facial motion estimation device 20 acquires a biopotential signal (BES) from the electrode 16 installed on the side of the face of the wearer 10, and outputs a signal from the filter means 43 to the acquired signal. Smoothing processing (filtering) is performed, processing is performed by the signal processing unit 44, and the content is presented as an optical signal or sound signal based on a method preset by the presentation unit 47.

  FIG. 6 is a diagram illustrating an example of the signal processing module. In FIG. 6, in order to classify facial expressions, biopotential signals (BES) are obtained from a plurality of electrodes installed on the side of the face. An independent component is analyzed using independent component analysis (ICA) as signal processing means 44 from a plurality of signals (Mixed Signal) acquired from the plurality of electrodes.

  After performing the independent component analysis, for example, smoothing processing (filtering) is performed on the input signal so that processing in the neuron network (ANN) is performed with high accuracy using the filter unit 43 as shown in FIG. I do. That is, the filter means 43 is used to separate independent components (IC).

  Further, the result of the independent component analysis is learned by the ANN which is the machine learning analysis means 45 as shown in FIG. 6 through an appropriate time window (for example, about 50 to 200 ms), and further, the expression and the content of the action ( Pattern) and degree are output.

  That is, in this embodiment, a biopotential signal (BES) is obtained by using a plurality of electrodes on the side of the filtered face, and facial expression estimation is performed by the signal processing module described above.

<About the contents of independent component analysis>
Next, the contents of the independent component analysis (ICA) described above will be described. FIG. 7 is a diagram illustrating an example of a block diagram of ICA. As shown in FIG. 7, for example, values s ₁ (t), s ₂ (t),..., S _m (t) obtained in an unknown environment are linearly mixed conversion means (Linear). (mixer) A generates sample data x ₁ (t), x ₂ (t),..., x _m (t), and further, linear mixed inverse transformation means (Linear) corresponding to the generated sample data X demixer) W is generated to generate independent components (IC) y ₁ (t), y ₂ (t),..., y _m (t).

<Example of basic data generation in facial expression estimation>
Here, in the present embodiment, as described above, the face motion estimation device 20 is first worn for each wearer, and then a smiling face, an angry face, etc. in a predetermined time (for example, about 2 to 4 seconds). It is also possible to intentionally perform various facial expressions, yawning and the like, and to generate basic data using bioelectric potential signals from the respective electrodes 16 obtained at that time.

  FIG. 8 is a diagram illustrating an example of generation of basic data in facial expression estimation in the present embodiment. In FIG. 8, as an example of facial expression estimation, a method for detecting a smile in real time in the present embodiment will be described.

  Real-time smile detection uses ICA and learned ANN, and a function displayed on the facial motion estimation device 20 is implemented.

  The facial motion estimation device 20 samples, for example, myoelectric potential signals (EMG) as bioelectric potential signals from a predetermined region on the side of the face set in advance, and analyzes them as predetermined blocks. The sampling is appropriately performed, for example, every 0.5 seconds, but is not limited to this in the present invention.

  In addition, the recognition of immediacy (real-time) in the present embodiment is performed by first generating the ICA matrix and learning the ANN, so that the estimation in the facial motion estimation device can be performed for the expression and motion of each wearer. It can be done regardless of the difference.

  Here, the training signal input to the facial motion estimation device 20 can perform, for example, predetermined expressions “neutral (neutral)”, “biting”, “biting and smiling”, “smile”, and the like.

  In the present embodiment, by generating these data, the facial motion estimation device 20 as an accurate emotion display device in real time can be provided. Expressions and actions of the face are not limited to those described above. For example, “hide the eyebrows”, “grind the eyebrows”, “get angry”, “yawn”, and “recognize masticatory movements in the oral cavity”, etc. For example, a part of the training set can be used.

<Experimental results>
Next, the experimental results to which this embodiment is applied will be described. 9 to 11 are diagrams (part 1) to (part 3) illustrating examples of experimental results to which the present embodiment is applied.

  In this experiment, biopotential signals are acquired using electrodes from the sides of the face on two subjects. Here, the sampling frequency of the signal used in the experiment is about 1024 Hz, but may be about 800 to 1200 Hz in the present embodiment. In this experiment, a total of 16 seconds (each facial expression is about 4 seconds if there are 4 types of facial expressions) is used to test both training and ANN. The facial expressions trained for facial expression estimation are, for example, “neutral (natural)”, “biting”, “biting smile”, or “smile”.

  First, FIG. 9 shows the experimental results for the “laughing” facial expression. The recognition rate when using one's own training data set (Training set), test data set (Test set) (data excluding data used during training), for example, subject 1 (Subject 1), subject 2 (Subject 2) It is shown in each. If this embodiment is provided, it turns out that the recognition rate of a facial expression can be acquired at a high level.

  Further, FIG. 10 shows the total recognition results for the training data set and the test data set for the three facial expressions of “laughter”, “anger”, and “neutral” as in FIG. 9 described above. The same results as those in FIG. 9 were obtained for these three facial expressions.

  In addition, when the data learned by the subject 1 is applied to the subject 2, the opposite recognition rate is two graphs (S.1 Trained ANN & S.2 Test set, S.2 Trained) shown on the right side of FIG. ANN & S.1 Test set), both of which indicate low recognition rates. That is, as shown in FIG. 9, in the recognition of facial expressions, the machine learning necessary in advance needs to be calibrated using the wearable facial motion estimation device actually used by the person, The facial motion estimation apparatus according to the present embodiment is convenient because it can easily perform machine learning, and can be said to be an optimal device for facial expression recognition and the like.

  Here, in this embodiment, in order to improve the estimation accuracy after training of the neuron network (ANN), independent component analysis (ICA) for the bioelectric potential signal of the face is implemented.

  First, when four facial potential signals were directly given to the neuron network (ANN) as a test data set to estimate the facial expression, the correct answer rate was only 76.52%. Therefore, as a result of facial expression estimation using component analysis (IC) in the present embodiment, it is possible to obtain a highly accurate classification result of 96.95% in average by the neuron network (ANN) in the test set. It was.

  Furthermore, in order to investigate the individual properties and independent component analysis (ICA) of the myoelectric potential signal (EMG), which is a biopotential signal, the above-described experiment was conducted on a neuron network (ANN) trained with other data. Although we were able to classify biopotential signals, the classification of facial expressions in other data was not very good.

Furthermore, for example, the results shown in FIG. 10 were obtained when facial expressions were estimated between “Frowning the eyebrows (Flow)”, “Smile” (Smile), and “Neither” (Neither). .
Further, in the present embodiment, an accurate average ratio (%) of classification for one subject is obtained for different numbers of independent components (independent components) that can be separated for classification of facial expressions. Can do.

  Here, as shown in FIG. 11, according to the experimental results, the probability that accurate data can be acquired can be increased as the number of independent components (IC) used increases. That is, according to the present embodiment, the accuracy of classification can be increased by increasing the number of ICs. According to FIG. 11, the accuracy of the IC is low in one case, and a very accurate classification is obtained in the four cases. Therefore, in this embodiment, facial expression estimation can be performed with high accuracy by using four electrode pairs that can be comfortably attached to the side of the face of the wearer.

  In addition, according to the present embodiment, there are noises and the like created by jaw muscles caused by biting, but by supplying a good input vector to the ANN for the biological signal of the face, the ICA method described above The result can be obtained more accurately.

  In this embodiment, a new approach to facial expression analysis is provided by using ANN for identification based on facial expression patterns instead of identifying signals belonging to muscles.

  Furthermore, the ANN used to estimate facial expressions can overcome weaknesses in independent component analysis (ICA) regarding uncertainty, regardless of the order of the input channels and their amplitude.

  As described above, according to the present invention, facial motion can be easily perceived. Specifically, it is shown that the present invention can analyze the facial expression and motion and the degree thereof using an independent component analysis or a neural network (ANN) using a biopotential signal. It was.

  Also, instead of placing electrodes directly on muscles that form facial expressions such as smiles, it is possible to place facial electrodes on the cheek muscle branches of the facial nerve and detect facial expressions such as smiles from signals obtained from them. It was further shown that there is. Electrodes can also be achieved with a small number of electrodes placed on the side of the face.

  Further, according to the present invention, discrimination based on facial expression patterns is possible, and facial expression estimation can be performed with high accuracy using an adaptive mechanism such as a neural network.

  Furthermore, by applying the present invention, it is possible to realize long-time measurement using, for example, a wearable smile measuring device, application to biofeedback such as smile training, and demonstration with a smile are necessary. In art that expresses using the body such as dance, it is possible to provide technology that performs feedback control of physical activity in real time, and human support technology such as light presentation and entertainment related to physical movement measurement, such as sports science and exercise Can be applied to.

  Furthermore, realization of alternative presentation means for people with difficulty in facial expression recognition, training equipment for physical movement such as dance and ballet, and exercise, application to exercise support, new wearable facial expression estimation device that is compact and configurable It can also be used as welfare equipment or rehabilitation equipment for application, muscle activity presentation, etc.

  The preferred embodiments of the present invention have been described in detail above, but the present invention is not limited to such specific embodiments, and various modifications, within the scope of the gist of the present invention described in the claims, It can be changed.

DESCRIPTION OF SYMBOLS 10 Wearer 11 Face ring muscle 12 Small zygomatic muscle 13 Large zygomatic muscle 14 Laugh muscle 15 Predetermined region 16 Electrode 20,30 Facial motion estimation apparatus 21,31 Light presentation means 22,32 Sound presentation means 41 Electrode signal input means 42 Accumulation means 43 Filter means 44 Signal processing means 45 Machine learning analysis means 46 Facial expression estimation means 47 Presentation means 48 Transmission / reception means 49 Control means

Claims (6)

In a facial motion estimation device that estimates facial motion based on a biopotential signal obtained from a predetermined region of the face,
A plurality of electrodes for obtaining the biopotential signal for a predetermined region of the face;
Filter means for filtering each signal obtained from the plurality of electrodes in a predetermined band;
Signal processing means for sorting by assuming independence of a preset signal with respect to the signal obtained by the filter means;
Machine learning analysis means for performing signal analysis by machine learning on the signal obtained by the signal processing means;
A facial expression estimation means for estimating the facial expression and degree from the analysis result obtained by the machine learning analysis means;
Presenting means for making a preset presentation corresponding to the facial expression and degree of the face,
The predetermined region of the face can acquire a mixed bioelectric potential signal transmitted through the skin from at least two muscles among the ring muscle, small zygomatic muscle, large zygomatic muscle, and laughing muscle of the wearer of the plurality of electrodes. The area on the facial nerve cheek muscle branch on the left and right sides of the face ,
The signal processing means obtains a 4- channel biopotential signal by arranging two pairs of electrodes vertically on the left and right sides of the region on the facial nerve buccal muscle branch on the left and right sides of the face , A facial motion estimation apparatus, wherein an independent component is extracted from the acquired combination of the bioelectric potential signals of the four channels . The machine learning analysis means includes
2. The facial motion estimation apparatus according to claim 1, wherein analysis is performed using a preset neural network that outputs the type and degree of facial expression corresponding to the input of the four-channel biopotential signal. . The presenting means presents with a plurality of lights or sounds,
The facial motion estimation apparatus according to claim 1, wherein the plurality of lights or sounds are changed and presented in accordance with the facial expression and degree. In a facial motion estimation method using a facial motion estimation device that estimates facial motion based on a biopotential signal obtained from a predetermined region of the face,
The facial motion estimation device comprises:
A signal acquisition step for acquiring the biopotential signal for a predetermined region of the face using a plurality of electrodes;
A filter step of filtering each signal from the plurality of electrodes obtained by the signal acquisition step in a predetermined band;
A signal processing step of separating by assuming a predetermined signal independence with respect to the signal obtained by the filtering step;
A machine learning analysis stage for performing signal analysis by machine learning on the signal obtained by the signal processing stage;
A facial expression estimation stage for estimating the facial expression and degree from the analysis result obtained by the machine learning analysis stage,
A presentation stage that is presented by a preset presentation means corresponding to the facial expression and degree of the face,
The predetermined region of the face can acquire a mixed bioelectric potential signal transmitted through the skin from at least two muscles among the ring muscle, small zygomatic muscle, large zygomatic muscle, and laughing muscle of the wearer of the plurality of electrodes. The area on the facial nerve cheek muscle branch on the left and right sides of the face ,
In the signal processing step, for a region on the facial nerve buccal muscle branch on the left and right sides of the face, a pair of electrodes are arranged vertically on the left and right to obtain a 4- channel biopotential signal, An independent component is extracted from a combination of the acquired bioelectric potential signals of the four channels . The machine learning analysis stage includes
5. The facial motion estimation method according to claim 4, wherein analysis is performed using a preset neural network that outputs the type and degree of facial expression corresponding to the input of the four-channel biopotential signal. . In the presenting step, a plurality of lights or sounds are presented using the presenting means,
6. The facial motion estimation method according to claim 4 or 5, wherein the plurality of lights or sounds are changed and presented according to the facial expression and degree.