JP7082444B2 - Machine learning equipment and machine learning programs - Google Patents

Description

The present invention relates to a machine learning device and a machine learning program for accumulating features used for individual identification, emotion recognition, and physical condition estimation.

With the increase in computing power, it has become possible to identify individuals and recognize emotions from faces and voices. In addition, with the spread of wearable devices (smart watches, activity meters), it is possible to acquire the biological conditions of a person such as body movement, body temperature, and heart rate. Along with this, by identifying an individual and recognizing the emotion of the individual, a service that facilitates dialogue between a person and a robot and provides advice from the aspect of health and exercise management is provided.

Machine learning is used to identify individuals and realize emotion recognition. Machine learning has two processes: learning that a computer finds a pattern from the data of things (training data), and guessing what the thing is from the data of unknown things after learning. Labels indicating things are given to the training data, and labels such as personal identification information are given for personal identification and labels such as joy and anger are given for emotion recognition. Normally, the guess is made after learning, but in order to improve the accuracy of the guess, the learning may be repeated (re-learning).

The technique described in Patent Document 1 is one of machine learning, and aims to improve the accuracy of individual identification and emotion recognition by recognizing the individuality of a user.

Japanese Unexamined Patent Publication No. 2001-83984

In the invention described in Patent Document 1, in order to improve the emotion recognition accuracy and, by extension, the individual identification accuracy, the user is inquired about the correctness of the recognized emotion (see FIG. 15 and paragraph 0063 of Patent Document 1). .. The timing of making this inquiry is after recognizing the emotion (see step 102 and paragraph 0036 of FIG. 7 of Patent Document 1), and is repeatedly executed.
However, a person's emotions cannot always be recognized reliably even by the person himself / herself, and it is very troublesome for the user to repeatedly answer the emotions, and it is desirable to reduce the number of inquiries as much as possible. In addition, there is no description about recognition of physical condition.

Therefore, it is an object of the present invention to provide a machine learning device and a machine learning program capable of improving the accuracy of individual identification, emotion recognition, and physical condition estimation.

In order to solve the above-mentioned problems, the present invention is a machine learning device in which a camera and a microphone are connected, and a face feature that calculates a facial feature amount from the face data of a user of the machine learning device taken by the camera. The amount calculation unit, the voice feature amount calculation unit that calculates the voice feature amount from the voice data of the user collected by the microphone, the face feature amount or the voice feature amount, the user's personal identification label, and the user. , A storage unit that stores a label database that stores records including personal identification information of the user, a calibration function unit that updates the label database, and a face that is close to the face feature amount calculated by the face feature amount calculation unit. A record including a face personal identification unit that outputs a user's personal identification label by searching a record containing a feature amount in the label database and a voice feature amount close to the voice feature amount calculated by the voice feature amount calculation unit. Is provided with a voice personal identification unit that outputs the user's personal identification label by searching the label database, and the calibration function unit outputs the user's personal identification label by the face personal identification unit. When the difference between the face feature amount calculated by the face feature amount calculation unit and the face feature amount of the user in the label database becomes larger than a predetermined value, the user in the label database The face feature amount is updated to the face feature amount calculated by the face feature amount calculation unit, and the voice feature calculated by the voice feature amount calculation unit when the voice personal identification unit outputs the user's personal identification label. When the difference between the amount and the voice feature amount of the user in the label database becomes larger than a predetermined value, the voice feature amount calculation unit calculates the voice feature amount of the user in the label database. It is a machine learning device characterized by updating to the voice features.

INDUSTRIAL APPLICABILITY According to the present invention, it is possible to provide a machine learning device and a machine learning program capable of improving the accuracy of individual identification, emotion recognition, and physical condition estimation.

It is a figure which shows the structure of the software function of the machine learning apparatus which concerns on this embodiment. It is a figure which illustrates the functional structure of the machine learning apparatus which concerns on this embodiment. It is a figure which shows that the emotion is expressed by using the three axes of comfort degree, arousal degree, and fatigue degree. It is a figure which showed emotion by two axes of comfort degree and arousal degree. It is a figure which illustrates the data structure of the label DB which concerns on this embodiment. It is a figure which illustrates the data structure of the temporary label DB which concerns on this embodiment. It is a figure which illustrates the data structure of the label history DB which concerns on this embodiment. It is a flowchart of the whole machine learning process which concerns on this embodiment. It is a flowchart of face personal label processing, voice personal label processing, and face emotion label processing in the label DB update process which concerns on this embodiment. It is a flowchart of the voice emotion label processing, the heartbeat label processing, and the label history DB update processing in the label DB update processing which concerns on this embodiment.

≪Software configuration≫
Hereinafter, embodiments for carrying out the present invention will be described with reference to each figure. FIG. 1 is a diagram showing a configuration of software functions of the machine learning device 800 according to the present embodiment. The software functions include config setting 801 and result output function 803, sensing data reception function 804, signal processing function 805, machine learning function 807, feature amount calculation function 808, personal / emotion estimation function 809, physical condition estimation function 810, and failure diagnosis function. 811 includes calibration function 812.

Sensors 820 such as a camera, a microphone, and a heart rate sensor are connected to the machine learning device 800 by using a cable, short-range wireless communication, or the like, and a setting DB (Database) 802, a label DB 56, a temporary label DB 57, and a label history DB 58 are connected. And prepare.
The config setting 801 is a function for setting parameters related to the hardware environment of a computer such as a sensor 820 that senses an external environment including people and things and a machine learning device 800. The setting DB 802 is a database in which the parameters set by using the config setting 801 are stored. The result output function 803 is a function that outputs the results estimated by the individual / emotion estimation function 809 and the physical condition estimation function 810, which will be described later.

The sensing data receiving function 804 is a function of receiving data from the sensor 820. The signal processing function 805 is a function for processing data such as images and sounds received from the sensor 820, and performs Fourier transform, image processing, filtering, and the like.
The machine learning function 807 is a general-purpose machine learning function using various machine learning models such as neural networks, genetic programming, and support vector machines. The feature amount calculation function 808 processes the data received by the sensing data reception function 804 with the signal processing function 805, calculates the feature amount, and causes the machine learning function 807 to input the feature amount. Voice features include voice pitch and pitch, and facial features include the positional relationship between the end points of the eyes and mouth.

The individual / emotion estimation function 809 uses the machine learning function 807 to identify an individual or infer emotions from the sensing data of a person (user of the machine learning device 800). The physical condition estimation function 810 estimates the physical condition from human sensing data using the machine learning function 807. The failure diagnosis function 811 diagnoses the cause of failure of the device or predicts the failure by using the machine learning function 807 from the sensing data of the device. The results estimated / diagnosed by the individual / emotion estimation function 809, the physical condition estimation function 810, and the failure diagnosis function 811 are output using the result output function 803.

The calibration function 812 is the feature amount calculated by the feature amount calculation function 808, and the result of estimation / diagnosis by the individual / emotion estimation function 809, the physical condition estimation function 810, and the failure diagnosis function 811 (label of personal identification information and anger / joy). Update relationships with emotional labels such as).
The label DB 56 includes a relationship between a label used for individual identification and emotion recognition (guessing) and a feature amount. The label that is the estimated result is temporarily stored in the temporary label DB 57. The label history DB 58 stores the history of the inferred label in order to monitor the secular variation of emotions and physical condition. Since the above three DBs contain personal information, they need to be stored safely.
The machine learning device 800 capable of improving the accuracy of individual identification, emotion recognition, and physical condition estimation described below is realized by using the above software function.

≪Functional configuration of machine learning device≫
FIG. 2 is a diagram illustrating a functional configuration of the machine learning device 800 according to the present embodiment. The machine learning device 800 includes an input / output unit 822, a control unit 823, a storage unit 824, a hypervisor 825, a real-time OS826, a secure OS827, a rich OS828, a feature amount calculation function 808, a machine learning function 807, a calibration function 812, and an individual. It includes an emotion / physical condition estimation unit 52, a label DB 56, a temporary label DB 57, and a label history BD 58. Further, a camera 21, a microphone 22, and a heart rate sensor 23 are connected to the machine learning device 800.

The input / output unit 822 is composed of a communication network card and the like, and transmits / receives data to / from a device / device such as a camera 21 or a touch panel (not shown). The sensing data receiving function 804 receives data from the camera 21, the microphone 22, and the heart rate sensor 23 via the input / output unit 822. The storage unit 824 is composed of a RAM (Random Access Memory), a ROM (Read Only Memory), and a hard disk, and stores various programs, and stores a label DB 56, a temporary label DB 57, and a label history DB 58. The control unit 823 is composed of a CPU (Central Processing Unit), and operates the machine learning device 800 by executing a program of various functions and other programs stored in FIG. 1 stored in the storage unit 824.

The hypervisor 825 is a program that virtually operates a plurality of computers (OS (Operating System)) on one computer. The real-time OS 826 is an OS that operates on the hypervisor 825 and operates an application program for which a processing time guarantee is required. The secure OS 827 is an OS that operates on the hypervisor 825, operates an application program that requires high security, and stores data. The rich OS 828 is an OS on which an application program that requires an advanced GUI (Graphical User Interface) operates.

The feature amount calculation function 808 includes a face feature amount calculation unit 31, a voice feature amount calculation unit 32, and a heart rate acquisition unit 33, and operates on a real-time OS 826 because real-time processing is required. The machine learning function 807 includes a face emotion analysis unit 41, a face individual identification unit 42, a voice emotion analysis unit 43, a voice individual identification unit 44, and a heart rate analysis unit 45, and operates on a real-time OS 826 because real-time processing is required. do.
In order to handle personal information, the calibration function 812 operates on the secure OS 827, and the label DB 56, the temporary label DB 57, and the label history BD 58 are stored on the secure OS 827.

The facial feature amount calculation unit 31 calculates the facial feature amount of the user from the video data taken by the camera 21. The feature amount calculated by the face feature amount calculation unit 31 is output to the face emotion analysis unit 41 and the face individual identification unit 42.
The voice feature amount calculation unit 32 calculates the feature amount of the user's voice collected by the microphone 22. The feature amount calculated by the voice feature amount calculation unit 32 is output to the voice emotion analysis unit 43 and the voice individual identification unit 44.
The heart rate acquisition unit 33 acquires the heart rate of the user detected by the heart rate sensor 23 and outputs it to the heart rate analysis unit 45.

The facial emotion analysis unit 41 analyzes (estimates) the user's emotion from the feature amount calculated by the facial feature amount calculation unit 31, and outputs an emotion label. In addition to the emotion label of emotions, the emotion label includes a numerical parameter indicating the coordinates in FIG. 3A, which will be described later, or an emotion label shown in FIG. 3B.
3A and 3B are diagrams for explaining the emotion type according to the present embodiment. FIG. 3A shows that emotions are expressed using three axes of comfort, arousal, and fatigue. For example, the label "emotion: 1, -2, 0" is an emotion label indicating a state in which the comfort level is 1, the arousal level is -2, and the fatigue level is 0. FIG. 3B shows emotions on the two axes of comfort and arousal. The label "emotion: 1, 2" is an emotion label indicating a state in which the comfort level is 1 and the arousal level is 2. Instead of the two numbers, labels such as excitement and alert shown in FIG. 3B may be used.

Returning to the explanation of FIG. 2, the face emotion analysis unit 41 searches the label DB 56 for a record having a feature amount closest to the feature amount output by the face feature amount calculation unit 31, and analyzes (estimates) the emotion. ..
FIG. 4 is a diagram illustrating the data structure of the label DB 56 according to the present embodiment. The label DB 56 is a database in a table format, and one row (record) represents the related information of one label. The label DB 56 is composed of columns (attributes) of a type 561, a feature amount 562, a label 563, an individual ID 564, and an update date and time 565.

Type 561 indicates the type of the record. Types include facial emotions including facial features and emotion labels, face individuals including facial features and personal identification labels, voice emotions including voice features and emotion labels, and voice individuals including voice features and personal identification labels. There is a heartbeat that includes the heart rate.
The feature amount 562 is a feature amount representing the record. The format and contents of the feature amount 562 vary depending on the type 561. For example, if the type 561 is a facial emotion, the feature amount 562 is in the form of a feature amount calculated by the facial feature amount calculation unit 31. For example, if the type 561 is a heart rate, the feature amount 562 is the heart rate acquired by the heart rate acquisition unit 33.

Label 563 indicates the label of the record. The label format varies depending on the type 561. For example, if the type 561 is a facial emotion, the label 563 is a label (emotion label) expressing emotions using the two parameters of comfort and arousal described in FIG. 3B. If the type 561 is a face individual, the label 563 is a label (individual identification label) indicating personal identification information such as "individual 164". There are other physical condition labels such as "Physical condition: Poor".

The personal ID 564 indicates whether the record is a specific individual or a general, and if it is a specific individual, it is the personal identification information, and if it is a general, it is "000". If the individual ID 564 is a specific individual, the feature amount 562 is a feature amount representing the feature amount of the individual. If the individual ID 564 is "000" (general), the feature amount 562 indicates an average feature amount that does not depend on the individual.
The update date and time 565 indicates the date and time when the record was created.

Returning to the explanation of FIG. 2, the face emotion analysis unit 41 searches the label DB 56 for a record in which the type 561 is a face emotion and the feature amount 562 is the closest to the feature amount output by the face feature amount calculation unit 31. Then, by outputting the label 563 of the record, the emotion is estimated, and the emotion label 563 is output to the temporary label DB 57 and the individual / emotion / physical condition estimation unit 52. The details of this process will be described in detail later with reference to FIG. 7.

FIG. 5 is a diagram illustrating the data structure of the temporary label DB 57 according to the present embodiment. The temporary label DB 57 is a table format database. One line is a record including a label output by the face emotion analysis unit 41, the face individual identification unit 42, the voice emotion analysis unit 43, the voice individual identification unit 44, or the heart rate analysis unit 45. The temporary label DB 57 is composed of columns of type 571, feature amount 572, label 573, accuracy 574, and acquisition date and time 575.

The type 571 indicates the type of the record, similarly to the type 561 of the label DB 56.
The feature amount 572 indicates the feature amount of the record. For example, if the record is a record output by the face emotion analysis unit 41, the feature amount is the feature amount output by the face feature amount calculation unit 31.
The label 573 indicates the label of the record in the same manner as the label 563 of the label DB 56.
The accuracy 574 indicates the certainty of the label 573 of the record, and the closer the feature amount 572 and the feature amount 562 of the searched record of the label DB 56 are, the larger the value becomes.
The acquisition date and time 575 is the date and time when the record was created.

Returning to the explanation of FIG. 2, the face individual identification unit 42 searches for the label DB 56 from the feature amount calculated by the face feature amount calculation unit 31, identifies (estimates) the individual, and assigns the personal identification label to the temporary label DB 57. And output to the individual / emotion / physical condition estimation unit 52.
The voice emotion analysis unit 43 searches for the label DB 56 from the feature amount calculated by the voice feature amount calculation unit 32, analyzes (estimates) the emotion, and uses the emotion label as the temporary label DB 57 and the individual / emotion / physical condition estimation unit 52. Output to.

The voice individual identification unit 44 searches for the label DB 56 from the feature amount calculated by the voice feature amount calculation unit 32, identifies (guesses) the individual, and sets the personal identification label as the temporary label DB 57 and the individual / emotion / physical condition estimation unit. Output to 52.
The heart rate analysis unit 45 searches the label DB 56 from the heart rate output by the heart rate acquisition unit 33, analyzes (estimates) emotions and physical condition, and temporarily labels the emotion label and physical condition label with the personal / emotion / physical condition DB 57. It is output to the estimation unit 52.

The calibration function 812 acquires a record from the temporary label DB 57 and updates the label DB 56 and the label history DB 58. By adding a personal label (personal ID 564 is personal identification information), the accuracy of sentiment analysis can be improved. The processing of the calibration function 812 will be described in detail later with reference to FIGS. 8 and 9.
FIG. 6 is a diagram illustrating the data structure of the label history DB 58 according to the present embodiment. In the label history DB 58, the detected emotions and physical conditions are classified and recorded as a history. The label history DB 58 is a table-type database, and one row (record) indicates the time when one emotion appears. The label history DB 58 is composed of columns of an individual ID 581, a label 582, an acquisition date and time 583, and an appearance time 584.

The personal ID 581 is personal identification information in which the emotion of the record is inferred. Label 582 is the emotion label of the record. The acquisition date and time 583 indicates the date and time when the emotion of the label 582 was estimated. The appearance time 584 indicates the cumulative time in which the emotion of the record is estimated.
The individual / emotion / physical condition estimation unit 52 is realized by combining the individual / emotion estimation function 809 (see FIG. 1) and the physical condition estimation function 810, and is realized by the face emotion analysis unit 41, the face individual identification unit 42, and the voice emotion. Individuals are comprehensively identified from the labels output by the analysis unit 43, the voice individual identification unit 44, and the heart rate analysis unit 45, and emotions and physical conditions are analyzed. The individual identified by the individual / emotion / physical condition estimation unit 52 is referred to in the label DB update process (see FIGS. 8 and 9) described later.

≪Machine learning process≫
FIG. 7 is a flowchart of the entire machine learning process according to the present embodiment. With reference to FIG. 7, the processing of the feature amount calculation function 808, the machine learning function 807, and the calibration function 812 will be described. The three series of processes of the series of processes of steps S111 to S114, the series of processes of steps S121 to S124, and the series of processes of steps S131 to S132 are processed asynchronously and independently. For example, the series of processes of steps S111 to S114 may be executed several times while the series of processes of steps S131 to S132 is executed once.

In step S111, the face feature amount calculation unit 31 acquires the user's face image from the camera 21.
In step S112, the face feature amount calculation unit 31 calculates the face feature amount from the face image and outputs it to the face emotion analysis unit 41 and the face individual identification unit 42. As feature quantities, there is a relative positional relationship between points such as both ends of the eye and the midpoint and the upper and lower points, both ends of the mouth and the middle point and the upper and lower points, the tip of the nose, the nostrils, and both ends of the eyebrows and the middle point.

In step S113, the facial emotion analysis unit 41 analyzes (estimates) emotions from facial features and outputs emotion labels to the temporary label DB 57 and the individual / emotion / physical condition estimation unit 52. Specifically, the face emotion analysis unit 41 records in the label DB 56 (see FIG. 4) that the type 561 is a face emotion and has a feature amount 562 closest to the feature amount output by the face feature amount calculation unit 31. The emotion is analyzed by searching and outputting the label 563 of the record.

However, if the individual / emotion / physical condition estimation unit 52 has already identified the individual user, the facial emotion analysis unit 41 analyzes the emotion as follows and outputs an emotion label. (1) The personal ID 564 searches for the closest feature amount 562 in the record that is the identification information of the individual, and calculates the distance from the feature amount output by the face feature amount calculation unit 31. Next, (2) the record having the closest feature amount 562 among the records having the personal ID 564 of "000" is searched, and the distance from the feature amount output by the face feature amount calculation unit 31 is calculated. If the record of (1) exists and the distance of (1) is smaller than a predetermined multiple of the distance calculated in (2), the label 563 of the record of (1) is output, otherwise (2). The label 563 of the record of is output.

To output the label to the temporary label DB 57, a record is added to the temporary label DB 57, the type 571 is the facial emotion, the feature amount 572 is the feature amount output by the face feature amount calculation unit 31, the label 573 is the label 563, and the accuracy. 574 is the closeness of the feature amount, and the acquisition date and time 575 is updated to the current time.

In step S114, the face individual identification unit 42 identifies (estimates) an individual from the facial features, and outputs the individual identification label to the temporary label DB 57 and the individual / emotion / physical condition estimation unit 52. Specifically, the face individual identification unit 42 searches the label DB 56 for a record in which the type 561 is a face individual and the feature amount 562 is close to the feature amount output by the face feature amount calculation unit 31. Then, the individual is identified by outputting the label 563 of the record.

If there is no record having the feature amount 562 of a predetermined proximity, the face personal identification unit 42 newly adds a record to the label DB 56 as an unregistered user. The type 561 of the record is a face individual, the feature amount 562 is the feature amount output by the face feature amount calculation unit 31, the label 563 is a new personal identification label, the personal ID 564 is the new personal identification information, and the update date and time 565 is the current time. be. If the individual has been identified and the feature amount 562 of the record of the individual on the label DB 56 and the feature amount output by the face feature amount calculation unit 31 are close to each other, the individual identification has already been identified. Output the label.
The processes of steps S113 and S114 are processed asynchronously and independently.

The processing of steps S121 to S124 is the same as that of steps S111 to S114 in which the camera 21 is replaced with the microphone 22 and the face is replaced with voice. Speech features include formaton frequency and peak value by cepstrum analysis, frequency power distribution by Fourier transform, and speaking speed by extracting phonemes.
In step S131, the heart rate acquisition unit 33 acquires the user's heart rate from the heart rate sensor 23.
In step S132, the heart rate analysis unit 45 analyzes (estimates) emotions and physical conditions from the heart rate by searching for a record having a similar heart rate in the label DB 56, and sets the emotion label or the physical condition label as the temporary label DB 57 and the individual. -Output to the emotion / physical condition estimation unit 52.

When the temporary label DB 57 is updated in steps S113, S114, S123, S124 or S132, the processing of the calibration function 812 after step S151 starts. The individual / emotion / physical condition estimation unit 52 refers to the output label to identify the user, estimate the emotion, and estimate the physical condition. The identified user's personal identification information is referred to in step S151 and the label DB update process (step S152) described later.
In step S151, the calibration function 812 determines whether the identified individual indicated on the most recently added personal identification label is the same as the already identified individual, and if the same (Y), proceeds to step S154. If they are not the same (N), the process proceeds to step S152.

In step S152, the calibration function 812 performs the label DB update process. The details of this process will be described in detail later with reference to FIGS. 8 and 9.
In step S153, the calibration function 812 deletes all the records on the temporary label DB 57.
In step S154, the calibration function 812 determines whether a predetermined time has elapsed since the previous label DB update process, and if it has not elapsed (N), the calibration function 812 is set to steps S111, S121, and S131, which are the first processes. If it returns and has passed (Y), the process proceeds to step S152.

≪Label DB update process≫
FIG. 8 is a flowchart of a face personal label process, a voice personal label process, and a face emotion label process in the label DB update process according to the present embodiment. FIG. 9 is a flowchart of the voice emotion label process, the heartbeat label process, and the label history DB update process in the label DB update process according to the present embodiment. With reference to FIGS. 8 and 9, a process corresponding to step S152 in which the calibration function 812 updates the label DB 56 will be described.

≪Label DB update process: Face individual and voice individual≫
In step S212, the calibration function 812 determines whether the accuracy of the individual face is reduced in the record of the temporary label DB 57 (see FIG. 5). Specifically, the calibration function 812 searches the record of the temporary label DB 57 for the record of the individual face of the type 571, and determines whether the average value of the accuracy 574 is lower than the predetermined value. The calibration function 812 may determine that the accuracy has decreased if the record with the accuracy 574 lower than the predetermined value is higher than the predetermined ratio.

In step S213, the calibration function 812 proceeds to step S214 when it is determined that the accuracy has decreased (Y), and proceeds to step S215 when it is determined that the accuracy has not decreased (N).
In step S214, the calibration function 812 calculates a new feature amount and updates the record of the personal identification label of the label DB 56. The record of the personal identification label of the label DB 56 is a personal identification label in which the label 563 corresponds to the identified individual, and a record in which the type 561 is a face individual.

The calibration function 812 searches for a record of an individual face of type 571 in the record of the temporary label DB 57, calculates a new feature amount from the feature amount 572 of the record, and features the record of the personal identification label of the label DB 56. Update 562. In order to calculate a new feature amount, the average feature amount of the feature amount 572 extracted from the temporary label DB 57 may be calculated, or a representative feature amount is selected from the place where the feature amount 572 is most densely packed. You may. Further, the feature amount 562 of the label DB 56 may be updated with a new feature amount, or the feature amount may be updated with a feature amount intermediate between the new feature amount and the feature amount 562 of the label DB 56. The calibration function 812 updates the update date and time 565 to the current time.

In step S215, the calibration function 812 determines whether the record of the personal identification label of the label DB 56 has passed a predetermined period from the previous update. Specifically, the calibration function 812 determines whether the update date and time 565 of the record in which the label 563 is a specified individual in the record of the label DB 56 is an individual identification label and the type 561 is a face individual is not before a predetermined period. ..

In step S216, the calibration function 812 proceeds to step S217 when it is determined that the predetermined period has passed (Y), and proceeds to step S222 when it is determined that the predetermined period has not passed (N).
In step S217, the calibration function 812 updates the record of the personal identification label of the label DB 56. The method of updating is the same as in step S214.
Steps S222 to S227 are similar processes in which the face of steps S212 to S217 is replaced with voice.

≪Label DB update process: facial emotions and voice emotions≫
In step S231, the calibration function 812 searches the temporary label DB 57 for records having similar facial features and features that are continuous for a predetermined time. Specifically, the calibration function 812 searches for records of type 571 of facial emotions in the records of the temporary label DB 57, and searches for records of which the feature amount 572 is close and the acquisition date and time 575 is continuous for a predetermined time. do.

In step S232, the calibration function 812 proceeds to step S233 if there is a record of the search result of step S231 (Y), and if such a record does not exist, it goes to step T1 (step S241 of FIG. 9). move on.
In step S233, the calibration function 812 searches the label DB 56 for a record of facial emotions including a feature amount 562 that is close to the feature amount 572 of the search result record. Specifically, the calibration function 812 searches the records of the label DB 56 for records in which the type 561 is facial emotion and the feature amount 562 is closer to the feature amount 572 in the search result than a predetermined value.

In step S234, the calibration function 812 proceeds to step S235 if a close record exists as a result of the search in step S233 (Y), and if such a record does not exist, step T1 (step S241 in FIG. 9). ).
In step S235, if the personal ID 564 of the record of the label DB 56, which is the search in step S233, is the identification information of the identified individual, the calibration function 812 considers that the personal ID 564 has been registered in the label DB 56, and steps T1 (step S241 in FIG. 9). ). If the personal ID 564 is not the identified personal identification information, the calibration function 812 registers the label in the label DB 56 after confirming with the user whether the content of the label is correct.

Specifically, the calibration function 812 indicates whether the emotion shown at the record label 573, which is the search result of step S233, was close to the emotion at the time of acquisition date / time 575, or what kind of emotion it was if it was not. Display the figure on the touch panel (not shown) and make an inquiry. Next, the calibration function 812 adds a record, sets the type 561 as facial emotion, the feature amount 562 as the average value of the feature amount 572 in the search result record of step S233, and labels 563 as the result of the inquiry. In addition, the personal ID 564 is updated to the identification information of the identified individual, and the update date and time 565 is updated to the current time. Instead of the average value of the feature amount 572, the representative feature amount described in step S214 may be used.

Steps S241 to S245 are similar processes in which the facial emotions of steps S231 to S235 are replaced with voice emotions.
≪Label DB update process: Heart rate≫
In step S251, the calibration function 812 searches for a record of heart rate of type 571 in the record of the temporary label DB 57, and the acquisition date and time is a record in which the heart rate of the feature amount 572 is higher than a predetermined value or lower than a predetermined value. 575 searches for consecutive records for a given time.

In step S252, the calibration function 812 proceeds to step S253 if the search result record of step S251 exists (Y), and proceeds to step S261 if it does not exist (N).
In step S253, the calibration function 812 is in a state of emotion excitement (see FIG. 3B) higher than the predetermined heart rate from the emotion label at the time of acquisition date / time 575 in the record of the search result in step S251, or the predetermined heart rate. Determine if emotions are classified as calm below the number. The emotion label has the acquisition date and time 575 at the time of acquisition date and time 575 in the search result record of step S251, and the type 571 can be acquired from the label 573 of the record of facial emotion or voice emotion.

In step S254, the calibration function 812 proceeds to step S255 if it is classified (Y), and proceeds to step S256 if it is not classified (N).
In step S255, the calibration function 812 indicates whether the emotion at the time of acquisition date and time 575 was close to the emotion shown in the facial emotion or voice emotion in step S253, and what kind of emotion it was if it was not. Present the figure and inquire. Next, the calibration function 812 adds a record to the label DB 56, sets the type 561 to the heartbeat, sets the feature amount 562 to the average value of the feature amount 572 of the record which is the search result of step S251, and queries the label 563. In the label of, the personal ID 564 is updated to the identification information of the identified individual, and the update date and time 565 is updated to the current time. If the label is registered in the label DB 56 as a label for individuals (there is a record in which the feature amount 562 is close to the average value of the feature amount 572 and the personal ID 564 is the identification information of the identified individual), this process is not necessary. ..

In step S256, the calibration function 812 asks the user if he / she was in poor physical condition around the acquisition date / time 575.
In step S257, the calibration function 812 proceeds to step S261 if the physical condition is not bad (N) as a result of the inquiry, and proceeds to step S258 if it is bad (Y).
In step S258, the calibration function 812 adds a record of the poor physical condition label to the label DB 56. Specifically, the calibration function 812 adds a record to the label DB 56, sets the type 561 to the heartbeat, sets the feature amount 562 to the average value of the feature amount 572 included in the search result record of step S251, and sets the label 563 to "physical condition". In ": Bad", the personal ID 564 is updated to the identified personal identification information, and the update date and time 565 is updated to the current time.

≪History DB update process≫
In step S261, the calibration function 812 classifies the detected emotions and stores the detected cumulative time in the label history DB 58 (see FIG. 6). Emotions are classified according to where they are located in the region within a predetermined distance from the origin in the graph of FIG. 3B or in the other four quadrant regions divided into excitement / tranquility and comfort / discomfort. Specifically, the calibration function 812 labels "emotion: 0,0" if the type 571 is a facial emotion or a voice emotion and the two values of the emotion label on the label 573 are smaller than the predetermined values. If it is analyzed as an excitement and comfortable emotion, it is classified by labeling it as "emotion: 1,1". The same is true for other quadrants. The cumulative time is calculated for each emotion in the five areas, and a record is added, with the acquisition date and time 575 as the detected time. The personal ID 581 of the record is the identification information of the specified individual, the label 582 is the emotion label shown above, the acquisition date and time 583 is the current time, and the appearance time 584 is the cumulative time described above.

≪Variation example≫
In the above embodiment, the label DB update process of the calibration function 812 is executed when a predetermined time has elapsed (step S154 in FIG. 7). When the facial emotions acquire close features that are continuous for a predetermined time (step 231 in FIG. 8), and the voice emotions acquire close features that are continuous for a predetermined time (step 241 in FIG. 9), the predetermined heart rate. The label DB update process may be started when a higher heart rate is continuous (step S251 in FIG. 9) or when a heart rate lower than a predetermined heart rate is continuous (step S251). When an inquiry regarding emotions and physical condition occurs (steps S235, S245, S255, S256), more accurate emotions and physical condition can be obtained by inquiring when the occurrence is detected.

In the above embodiment, a camera, a microphone, and a heart rate sensor are used to acquire the biological state. Not limited to this, a biological state may be acquired by using a sensor that acquires body temperature or a three-dimensional acceleration sensor. For example, a wearable device worn on an arm or leg can be used to obtain body temperature, blood pressure, and tremor in addition to heart rate. In addition, it is possible to detect the direction of the face and the direction of the line of sight from the feature amount of the face acquired by the camera, the amount of change in the direction of the face and the direction of the line of sight, and the amount of body tremor (magnitude, frequency, time of tremor). The characteristic amount of poor physical condition including seizures can be obtained from, and the individual / emotion / physical condition estimation unit 52 can infer the poor physical condition.

If the machine learning device 800 is mounted on a car, the car is stopped or decelerated when it is estimated that the car is in poor physical condition, or when an abnormal heartbeat or body tremor is obtained from the feature amount of the poor physical condition. And can prevent accidents. Further, in step S261, in addition to the emotional history, the records of poor physical condition are aggregated and stored in the label history DB 58, so that the secular change of the physical condition can be grasped.
In addition, as other biological state information, body weight, body fat percentage, conversation amount, sleep time, ultraviolet exposure amount, respiratory rate, exercise amount, vocalization amount and the like may be used. In addition, emotions and physical conditions may be estimated using environmental information such as temperature, humidity, amount of light, illuminance, amount of ultraviolet rays, atmospheric pressure, and noise, and changes over time may be monitored.

When the calibration function 812 determines that the individual (user) specified in step S151 is different from the individual specified before that, the calibration function 812 has executed the label DB update process. Instead of determining that the individual is different by updating the temporary label DB 57 only once, the label DB update process may be started after it is determined that the individual is different for a predetermined time. By doing so, it is possible to prevent the label DB update process from being unnecessarily and frequently executed even if the individual identification becomes unstable due to the appearance of the camera or the voice of another person.

≪Effect≫
In the above embodiment, when the feature amount for identifying an individual changes from the feature amount on the label DB 56 (step S212, step S222 in FIG. 8) or when a certain period of time has elapsed (step S215, step S225), the feature amount is characterized. The amount is being updated. This makes it possible to prevent an individual from becoming unidentifiable even if secular variation occurs. In addition, it is possible to identify an individual with higher accuracy by using the characteristic amount of individual identification that has changed over time.

In the above embodiment, the emotions appearing in the facial features and the voice features appear continuously for a predetermined time, not immediately after the emotions appear (steps S231 and 9 in FIG. 8). The user is inquired in step S241) and registered in the label DB 56. Since the emotions are continuous for a predetermined time, the user can easily recognize the emotions, obtain accurate emotions, and register them in the label DB 56. As a result, the individual / emotion / physical condition estimation unit 52 can infer emotions more accurately.

Further, before making an inquiry, it is confirmed that the record is not registered in the label DB 56 as a specific personal record (step S235, step S245). Therefore, the number of inquiries to the user can be reduced, and the complexity of answering can be reduced.
By registering a record including the feature amount 562 of a specific individual and the emotional label 563 inquiring to the individual, it is possible to create a label DB 56 that matches the emotional expression of each user. It enables more accurate emotion estimation.
From the label history DB 58 (see FIG. 6), it is possible to grasp the secular variation of the user's emotion over a long period of time. For example, if the appearance time 584 of the label 582 "emotion: -1, -1" is on the increase, it means that the user is in a depressed state, and there is a possibility that a mental problem has occurred. It is possible to warn the user that there is.

21 Camera 22 Mike 23 Heart rate sensor 31 Face feature amount calculation unit 32 Voice feature amount calculation unit 33 Heart rate acquisition unit 41 Face emotion analysis unit 42 Face individual identification unit 43 Voice emotion analysis unit 44 Voice individual identification unit 45 Heart rate analysis unit 52 Individual・ Emotion / physical condition estimation part 56 Label DB
57 Temporary label DB
58 Label history DB
800 Machine learning device 807 Machine learning function 808 Feature calculation function 812 Calibration function

Claims (3)

A machine learning device with a camera and microphone connected
A face feature amount calculation unit that calculates a face feature amount from the face data of a user of the machine learning device taken by the camera, and a face feature amount calculation unit.
A voice feature amount calculation unit that calculates a voice feature amount from the voice data of the user collected by the microphone, and a voice feature amount calculation unit.
A storage unit that stores a label database that stores a record containing the face feature amount or the voice feature amount, the user's personal identification label, and the user's personal identification information.
The calibration function unit that updates the label database,
A face personal identification unit that outputs a user's personal identification label by searching the label database for a record containing a face feature amount close to the face feature amount calculated by the face feature amount calculation unit.
It is provided with a voice personal identification unit that outputs a user's personal identification label by searching the label database for a record containing a voice feature amount close to the voice feature amount calculated by the voice feature amount calculation unit.
The calibration function unit is
The difference between the face feature amount calculated by the face feature amount calculation unit when the face individual identification unit outputs the user's personal identification label and the face feature amount of the user in the label database is predetermined. When it becomes larger than the value, the face feature amount of the user in the label database is updated to the face feature amount calculated by the face feature amount calculation unit.
The difference between the voice feature amount calculated by the voice feature amount calculation unit when the voice individual identification unit outputs the user's personal identification label and the voice feature amount of the user in the label database is predetermined. When it becomes larger than the value, the voice feature amount of the user in the label database is updated to the voice feature amount calculated by the voice feature amount calculation unit.
The label database further stores a record containing the face feature amount or the voice feature amount, the emotion label, and the user's personal identification information.
The calibration function unit is
The label is a record in which the face feature amount calculated by the face feature amount calculation unit is close to a predetermined value for a predetermined time, and the face feature amount close to the face feature amount and the user's personal identification information are included. Includes the facial feature amount calculated by the facial feature amount calculation unit when it does not exist in the database, the personal identification information of the user, and the emotional label of the emotion as a result obtained by inquiring the user about the emotion. Add the record to the label database and
The label is a record in which the voice feature amount calculated by the voice feature amount calculation unit is close to a predetermined value for a predetermined time, and the voice feature amount close to the voice feature amount and the user's personal identification information are included. Includes the voice feature amount calculated by the voice feature amount calculation unit when it does not exist in the database, the personal identification information of the user, and the emotion label of the result obtained by inquiring the user about the emotion. A machine learning device characterized by adding records to the label database . A heart rate sensor, blood pressure sensor or 3D accelerometer is connected,
The label database further stores records containing heart rate, blood pressure or body tremor, emotional or physical condition labels, as well as the user's personally identifiable information.
A face emotion analysis unit that outputs a user's emotion label by searching the label database for a record containing a face feature amount close to the face feature amount calculated by the face feature amount calculation unit.
A voice emotion analysis unit that outputs a user's emotion label by searching the label database for a record containing a voice feature amount close to the voice feature amount calculated by the voice feature amount calculation unit is further provided.
The calibration function unit is
When the heart rate sensor is connected, the heart rate measured by the heart rate sensor is out of the predetermined range for a predetermined time, and the emotion label output by the facial emotion analysis unit or the voice emotion analysis unit is displayed. When the predetermined conditions are not met, a record including the heart rate, the personal identification information of the user, and a physical condition label indicating the physical condition obtained by inquiring the user about the physical condition is stored in the label database. Add and
When the blood pressure sensor is connected, the blood pressure measured by the blood pressure sensor for a predetermined time is out of the predetermined range, and the emotion label output by the facial emotion analysis unit or the voice emotion analysis unit is predetermined. When the condition of is not satisfied, a record including the blood pressure, the personal identification information of the user, and the physical condition label indicating the physical condition obtained by inquiring the user about the physical condition is added to the label database. ,
When the three-dimensional acceleration sensor is connected, the amount of body tremor measured by the three-dimensional acceleration sensor during a predetermined time is out of the predetermined range, and the facial emotion analysis unit or the voice emotion analysis unit is used. When the emotional label output by is not satisfied with the predetermined conditions, the amount of tremor of the body, the personal identification information of the user, and the physical condition label indicating the physical condition obtained by inquiring the user about the physical condition. The machine learning apparatus according to claim 1 , wherein a record including the above is added to the label database. The facial feature amount calculation procedure for calculating the facial feature amount from the user's face data taken by the camera, and the facial feature amount calculation procedure.
A voice feature calculation procedure for calculating a voice feature from the user's voice data collected by the microphone, and a voice feature calculation procedure.
A procedure for operating a label database that stores a record containing the face feature amount or the voice feature amount, the user's personal identification label, and the user's personal identification information.
A face individual identification procedure that outputs a user's personal identification label by searching the label database for a record containing a face feature amount close to the face feature amount calculated by the face feature amount calculation procedure.
A voice personal identification procedure for outputting a user's personal identification label by searching the label database for a record containing a voice feature amount close to the voice feature amount calculated by the voice feature amount calculation procedure.
The difference between the face feature amount calculated by the face feature amount calculation procedure when the user's personal identification label is output in the face individual identification procedure and the face feature amount of the user in the label database is predetermined. When the value becomes larger than the value, the procedure for updating the facial feature amount of the user in the label database to the facial feature amount calculated by the facial feature amount calculation procedure, and
The difference between the voice feature amount calculated by the voice feature amount calculation procedure when the user's personal identification label is output in the voice individual identification procedure and the voice feature amount of the user in the label database is predetermined. When the value becomes larger than the value, the procedure for updating the voice feature amount of the user in the label database to the voice feature amount calculated by the voice feature amount calculation procedure and the procedure.
Is a machine learning program that lets a computer execute
The label database further stores a record containing the face feature amount or the voice feature amount, the emotion label, and the user's personal identification information, and further stores the record.
The label is a record in which the facial feature amount calculated by the facial feature amount calculation procedure is close to a predetermined value for a predetermined time, and the facial feature amount close to the facial feature amount and the personal identification information of the user are included. Includes the facial feature amount calculated by the facial feature amount calculation procedure when it does not exist in the database, the personal identification information of the user, and the emotional label of the emotion obtained by inquiring the user about the emotion. The procedure for adding a record to the label database and
The label is a record in which the voice feature amount calculated by the voice feature amount calculation procedure is close to a predetermined value for a predetermined time, and the voice feature amount close to the voice feature amount and the user's personal identification information are included. Includes the voice feature amount calculated by the voice feature amount calculation procedure when it does not exist in the database, the personal identification information of the user, and the emotion label of the result obtained by inquiring the user about the emotion. The procedure for adding a record to the label database
A machine learning program that causes the computer to execute.

Images