JP2018007792A - Expression recognition diagnosis support device - Google Patents

Abstract

PROBLEM TO BE SOLVED: To provide an expression recognition diagnosis support device for supporting diagnosis of a relatively early symptom of dementia.SOLUTION: The expression recognition diagnosis support device includes: a face expression pattern storage part for previously storing a plurality of moving image patterns; a question pattern storage part for previously storing a plurality of question patterns; a camera part capable of photographing the face of a user performing response according to a moving image pattern and a question pattern; a face image processing part for extracting face information; a learning control part for performing machine learning by using the face information of the user face information storage part, and face information of a healthy person face information storage part or face information of a slight dementia face information storage part on the basis of preset teacher information, and storing a weighting coefficient of each part of an updated face; and a determination processing part for calculating a prescribed distance between the face information extracted by the face image processing part and the face information stored in the healthy person face information storage part or the face information stored in the slight dementia face information storage part to determine which a user belongs to either a healthy person group or a slight dementia group.SELECTED DRAWING: Figure 2

Description

  The present invention relates to a facial expression recognition diagnosis support apparatus for supporting diagnosis of relatively early symptoms of dementia.

  In recent years, interest in prevention and early detection of dementia has become very high. Dementia is listed as one of the three major causative diseases that require nursing care. In 2012, there were approximately 4.62 million people with dementia (about 1 in 7 people over the age of 65). However, it is expected to increase with the aging of the population in the future, and is expected to reach about 7 million in 2025 (about 1 in 5 people over 65).

  There are several types of dementia such as Alzheimer-type dementia, cerebrovascular dementia, frontotemporal dementia, and Lewy body dementia, but the most common type of dementia is Alzheimer-type dementia. It is symptom.

  Dementia is different from normal people in daily attitudes and behaviors. For example, a healthy person can fully recognize minute positional changes such as the corners of the mouth, the corners of the eyes, and the eyebrows due to the movement of the opponent's facial muscles and minute facial expression changes that appear due to eye movements. Further, it is known that the ability decreases as the obstacles occur in recognizing the facial expression change of the other party (Non-patent Document 1). In addition, as the dementia progresses, the ability decline becomes more prominent.

  On the other hand, it has been shown that mild cognitive impairment (MCI), which is a precursor of dementia, may be expected to recover to some extent due to the adaptability of the brain (Japan) Neurological Society, Dementia Disease Treatment Guidelines 2010, p75).

  In addition, Alzheimer-type dementia often begins with cranial nerve degeneration 10 years or more before the onset of the disease, and if it can be diagnosed in relatively early symptoms, the progression of dementia is prevented or the onset is prevented. It has been shown that the possibility of being able to be increased (Ministry of Health, Labor and Welfare, Dementia Prevention and Support Manual (revised version), March 2009).

  By the way, in the diagnosis of a person's medical condition including dementia, an inquiry and a visual inspection have been conventionally performed, and there is a diagnosis support system that supports these.

  For example, the dementia diagnosis support system disclosed in Patent Document 1 asks a patient about the patient's attributes, schedule, news, etc., and recognizes the patient based on the correct answer rate or the number of correct answers. The disease level is determined, and the result is notified to the computer terminal used by the doctor. The doctor can make a detailed inquiry by looking at the notified result.

  The system for predicting dementia disclosed in Patent Document 2 determines whether or not a dementia possibility determination criterion is satisfied based on input information input by a subject under a promise for periodic input. And, when applicable, notify you of the possibility of dementia. The subject can use the system for predicting the possibility of dementia based on regular input.

  The facial expression diagnosis support apparatus disclosed in Patent Document 3 captures the facial expression of a person to be diagnosed with a digital camera, extracts feature points of the facial image, and quantifies it as diagnostic data. It calculates using the diagnosis matrix which digitizes the knowledge of the house, and displays the result on the display as the diagnosis result. The diagnostician can easily determine the degree of the psychogenic disease of the diagnosed person from the diagnosis result.

JP 2007-282929 A Japanese Patent Laying-Open No. 2015-138488 JP 2006-305260 A

D. McCade et al. : Review of emotion recognition in mild cognitive impulse, Dimension Geriatr Cognition, 32, 257-266, 2011

  However, for mild cognitive impairment, although some recovery can be expected due to the adaptability of the brain, it is difficult to say that a means for supporting diagnosis of relatively early symptoms of dementia has been established. is the current situation.

  In addition, the diagnosis of dementia needs to properly capture the symptom, but in the diagnosis support performed by the conventional dementia diagnosis support system etc., for example, the facial features of the user, etc., fine features that are different from normal subjects Even if appears, it is unclear whether it can be properly understood as a symptom of dementia.

  The present invention has been made in view of such problems, and an object thereof is to provide a facial expression recognition diagnosis support apparatus for supporting diagnosis of relatively early symptoms of dementia.

In order to solve the above problems, the present invention
A facial expression pattern storage unit that pre-stores a plurality of moving image patterns obtained by stereoscopically photographing a human face at 120 frames or more per second;
A question pattern storage unit that stores a plurality of question patterns for the user in advance;
An operation unit that can accept input operations from the user,
A display control unit that controls the information display unit based on preset setting information and input information received by the operation unit;
Based on a command from the display control unit, an information display unit that displays a moving image pattern stored in the facial expression pattern storage unit and a question pattern stored in the question pattern storage unit;
A plurality of image sensors that can be photographed at 120 frames per second or more, a camera unit arranged so that a user's face that responds according to a moving image pattern and a question pattern displayed on the information display unit can be stereoscopically photographed,
A face image processing unit that extracts face information about each part of the face from a video imaged by the camera unit;
A user face information storage unit for storing the face information extracted by the face image processing unit;
A healthy person face information storage unit that stores face information belonging to the healthy person group in advance;
A mild cognitive impairment face information storage unit that stores face information belonging to the mild cognitive impairment group in advance;
Based on preset teacher information, the face information stored in the user face information storage unit and the face information stored in the healthy person face information storage unit or stored in the mild cognitive impairment face information storage unit A learning control unit that performs machine learning using the acquired face information and stores a weight coefficient of each part of the updated face;
The learning control between the face information extracted by the face image processing unit and the face information stored in the healthy person face information storage unit or the face information stored in the mild cognitive impairment person face information storage unit A determination processing unit that calculates a predetermined distance using the weighting factor stored in the unit, and determines whether the user belongs to a group of healthy persons or a group of mild cognitive impairments based on the distance;
A determination information storage unit for storing the determination result in the determination processing unit;
It is characterized by providing.

  According to the present invention, diagnosis support for relatively early symptoms of dementia can be easily performed.

1 is an overall view showing an overall configuration of the present invention. It is a block diagram which shows the functional structure of this invention. It is explanatory drawing which showed an example of the display in the information display part of this invention. It is the figure which showed the example of the question pattern of this invention in tabular form. It is explanatory drawing which showed an example of the vector information of each part of the face in a face image. It is a conceptual diagram regarding the distance D of the vector information of this invention. It is explanatory drawing which showed the example which mounted this invention in the portable terminal device.

  Hereinafter, embodiments of the present invention will be described in detail with reference to the drawings. The present invention is not limited to the following description, and can be appropriately changed without departing from the gist of the present invention.

First, the configuration of the facial expression recognition diagnosis support apparatus 100 according to the present invention will be described. FIG. 1 is an overall view showing the overall configuration of the present invention 100, and FIG. 2 is a block diagram showing the functional configuration of the present invention 100.
As shown in FIG. 1, the facial expression recognition diagnosis support apparatus 100 includes a main body unit 1, an information display unit 2, a camera unit 3, and an operation unit 4.

The camera unit 3 is composed of two or more image sensors capable of stereoscopic shooting at 120 frames or more per second. The image sensor is a CCD image sensor, a CMOS image sensor, or the like.
This camera unit 3 is used to cause a user 5 to respond according to a moving image pattern or a question pattern displayed on the information display unit 2 and to stereoscopically photograph the facial expression of the user 5 at that time. The camera unit 3 has a frame rate of 120 frames per second or more, preferably 240 frames per second, so that even a fine expression appearing on the face of the user 5 can be captured appropriately.
The camera unit 3 is provided with two or more image sensors at positions close to the information display unit 2 so that the face of the user 5 viewing the information display unit 2 can be stereoscopically photographed at a front angle. .

  The main unit 1 includes a processing device such as a CPU, a main storage device such as a ROM and a RAM, an auxiliary storage device such as a hard disk and a flash memory, and the like as physical configurations. Various programs are executed based on information and input signals from an input device such as the operation unit 4 to perform necessary control processing.

  On the other hand, as shown in FIG. 2, the body unit 1 has a functional configuration such as a facial expression pattern storage unit 101, a question pattern storage unit 102, a display control unit 103, a face image processing unit 111, and a determination processing unit 112. A judgment information storage unit 113, a user face information storage unit 121, a healthy person face information storage unit 122, a mild cognitive impairment person face information storage unit 123, and a learning control unit 131.

The facial expression pattern storage unit 101 includes a magnetic disk device such as a hard disk, a non-volatile memory device such as a flash memory, and the like.
The facial expression pattern storage unit 101 stores in advance a plurality of moving image patterns of a human face that are presented as a stereoscopic image to the user via the information display unit 2.
This moving image pattern is photographed at a frame rate of 120 frames per second or more, preferably 240 frames per second so that fine expressions appearing on the human face can be appropriately presented. In addition, stereoscopic imaging is performed with two or more cameras so that a human face can be displayed as a stereoscopic video.
In addition, there are a plurality of moving image patterns, such as patterns based on facial expressions and emotions appearing on a person's face, patterns based on a person's sex and age group, patterns based on composition and background settings, etc. It is preferable to adopt a pattern that is easy to imitate, act, or have a natural expression while looking at this moving image pattern.

The question pattern storage unit 102 includes a magnetic disk device such as a hard disk, a non-volatile memory device such as a flash memory, and the like.
The question pattern storage unit 102 stores in advance a plurality of question patterns that are presented to the user via the information display unit 2.
This question pattern includes not only character information presented by character display but also moving image information presented by image display as required.
In addition, there are multiple patterns of this question pattern. There are patterns based on themes, patterns based on the question format, etc., but the response of the user 5 is different from that of the healthy person, and the characteristics seen in mild cognitive impairment and dementia are likely to appear. Is preferably adopted.

FIG. 4 is a diagram showing an example of a question pattern according to the present invention in a table format.
In the example shown in this figure, there are four themes for the question pattern. One is discrimination of facial expressions, for collecting direct answers of the user 5. However, it does not make the user 5 aware of the correct answer or incorrect answer, and is designed so as not to give the user 5 a psychological burden.
The other three are facial expressions imitation, facial expressions, and natural facial expressions for collecting facial expressions of the user 5. The user 5 considers the expression on the screen so that the user can easily imitate, work, or have a natural expression.

  The display control unit 103 stores the moving image pattern of the human face stored in the facial expression pattern storage unit 101 and the question pattern storage unit 102 based on preset setting information, an input signal from the operation unit 4, and the like. The display control for displaying the question pattern thus displayed on the information display unit 2 is performed.

The face image processing unit 111 extracts face information that appropriately captures minute movements of each part such as a mouth corner, an eye corner, an eyebrow, and an eyeball from the face image 21 of the user 5 photographed by the camera unit 3.
This face information includes a velocity vector in addition to position vector information and time information of each part of the face of the user 5.

  FIG. 5 is a diagram showing an example of vector information of each part of the face in the face image 21. As shown in this figure, the face coordinate system 22 is an orthogonal coordinate system composed of three axes of the X axis, the Y axis, and the Z axis with the position of the nose apex of the face image 21 as a reference point. Position vector information 23 and time information such as mouth corners, corners of the eyes, eyebrows, and eyeballs, which are representative parts constituting the facial expression, are extracted as face information.

The user face information storage unit 121 includes a magnetic disk device such as a hard disk, a nonvolatile memory device such as a flash memory, and the like.
The user face information storage unit 121 stores the face information of the user 5 extracted by the face image processing unit 111.

The healthy person face information storage unit 122 includes a magnetic disk device such as a hard disk, a non-volatile memory device such as a flash memory, or the like.
In this healthy person face information storage unit 122, vector information and time information extracted for each part from a large number of facial images collected from a large number of healthy persons are stored as face information belonging to the healthy person group.
The face information stored in the healthy person face information storage unit 122 has the healthy person respond according to a moving image pattern or a question pattern presented on the information display unit 2, and the facial image photographed by the camera unit 3 at this time Vector information and time information are extracted for each part such as the corner of the mouth, the corner of the eye, the eyebrows, and the eyeballs, and accumulated, and calculated with the face information of the user 5 and other face information extracted by the face image processing unit 111. be able to.
This face information includes a velocity vector in addition to position vector information and time information of each part of the face.

The mildly cognitively impaired person face information storage unit 123 includes a magnetic disk device such as a hard disk, a non-volatile memory device such as a flash memory, and the like.
In this mild cognitive impairment face information storage unit 123, vector information and time information extracted for each part from a large number of facial images collected from a large number of mild cognitive impairment persons as face information belonging to the mild cognitive impairment group. It is remembered.
The face information stored in the mild cognitively impaired person face information storage unit 123 has a mild cognitively impaired person respond in advance according to a moving image pattern or a question pattern presented on the information display unit 2. Vector information and time information are extracted and accumulated for each part such as the mouth corner, the corner of the eye, the eyebrows, and the eyeballs from the photographed face image. The face information of the user 5 extracted by the face image processing unit 111 and other information are collected. It can be calculated with face information.
This face information includes a velocity vector in addition to position vector information and time information of each part of the face.

  The determination processing unit 112 includes the face information of the user 5 extracted by the face image processing unit 111, the face information belonging to the healthy person group stored in the healthy person face information storage part 122, or the face information storage part 123 with mild cognitive impairment. Between the face information belonging to the mild cognitive impairment group stored in the table, using the weighting coefficient of each part of the face determined by the learning control unit 131, and based on the comparison result of the distance Then, it is determined whether the user 5 belongs to the normal group or the mild cognitive impairment group.

  The determination can be performed as a discriminant function for determining whether the velocity vector of each part of the face of the user 5 belongs to a normal group or a mild cognitive impairment group at a certain time.

  For example, when the face information of the user 5 is input to the discriminant function, each of the distance between the average pattern of the face information belonging to the healthy person group and the average pattern of the face information belonging to the mild cognitive impairment group Is calculated, and it is possible to obtain a determination result as to whether or not the similarity is high based on the value obtained as the reciprocal of the shorter one of the distances.

  In addition to the Euclidean distance corresponding to the geometric distance, the distance includes a correlation distance defined as a numerical value obtained by subtracting the correlation coefficient between 1 and the velocity vector, a Mahalanobis distance considering dispersion, The Canberra distance to be evaluated can be used.

  The distance can be calculated on the assumption that each part of the face is equal, but in the present invention, the distance is calculated by weighting each part of the face with a weighting coefficient.

For example, the Euclidean distance D between the face information of the user 5 at time t and the face information belonging to the healthy subject group at time t ′ can be expressed by the following equation.
Where t0 and t′0 are times when the movement of the facial part n is started, and wn is a weighting factor in the facial part n, xn, yn, zn and x′n, y′n, z′n. Are the position vectors of the facial part n, Δxn, Δyn, Δzn and Δx′n, Δy′n, Δz′n are the velocity vectors of the facial part n, respectively.

The determination information storage unit 113 is configured by a magnetic disk device such as a hard disk, a nonvolatile memory device such as a flash memory, or the like.
The determination information storage unit 113 stores the result determined by the determination processing unit 112.

  The learning control unit 131 uses the face information stored in the user face information storage unit 121 and the face information stored in the normal person face information storage unit 122 or the mild cognitively impaired person face information based on preset teacher information. Machine learning is performed using the face information stored in the storage unit 123, and the weighting coefficient of each part of the updated face is stored.

  When the face information of the user 5 belongs to the healthy person group, the weight coefficient of each part of the face has a small distance from the face information of the mild cognitive impairment group and a large distance from the face information of the healthy person group. If, on the other hand, the user 5's face information belongs to the mild cognitive impairment group, the distance from the healthy person's face information is small, and the mild cognitive impairment The optimal weighting coefficient can be determined by machine learning based on teacher information set in advance so that the distance from the face information of the person group increases.

  If the distance is calculated using the optimum weight coefficient, the accuracy of the determination can be improved.

  Moreover, it is preferable that the learning control unit 131 is provided with a deep learning function such as a neural network having a multilayer structure of two or more layers in order to improve the accuracy of determination.

  When the user 5 has been diagnosed by a method such as fMRI, CT, or blood test in the past, in order to reflect the result, machine learning is performed based on teacher information corresponding to the detected value, etc. The accuracy can also be improved.

The information display unit 2 is configured by a flat panel display such as a liquid crystal display or a plasma display.
Based on the output signal from the display control unit 103, the information display unit 2 displays the moving image pattern stored in the facial expression pattern storage unit 101 and the question pattern stored in the question pattern storage unit 102 on the screen.
The information display unit 2 displays the frame rate at 120 frames per second or more, preferably 240 frames per second so that the user 5 can appropriately present a fine expression appearing on the human face.
Note that since the moving image pattern stored in the facial expression pattern storage unit 101 is a three-dimensional image, a three-dimensional image can be displayed without using 3D glasses.

FIG. 3 is a diagram showing an example of display on the information display unit 2. In this figure, a moving image pattern of a human face is displayed at a central portion of the screen at a high frame rate and a stereoscopic image, a question pattern 11 for the user 5 is displayed at the top of the screen, and answer options for the question pattern 11 are displayed at the bottom of the screen. Thirteen characters are displayed.
In this figure, two image scanners 12 are installed at the left and right corners of the information display unit 2, but the camera unit 3 may be installed in the information display unit 2 in this way.

The operation unit 4 includes operation keys, buttons, a touch pad, and the like.
The operation unit 4 accepts an input operation, for example, when the user 5 selects an answer to the question pattern presented on the information display unit 2 from among answer options.

  In addition to the above-described embodiment, for example, in the configuration of the main body unit 1, a user face information storage unit 121, a healthy person face information storage unit 122, a mild cognitive impairment person face information storage unit 123 that processes enormous data, The learning control unit 131, the determination information storage unit 113, and the face image processing unit 111 are mounted on a computer system that can be connected to the Internet and communicated, such as a cloud or a server system, and the information display unit 2, the camera unit 3, By mounting the remaining components such as the face information pattern storage unit 101, the question pattern 102, the display control unit 103, and the determination processing unit 112 of the operation unit 4 and the main body unit 1 on a mobile terminal device, a large number of devices that can be connected to the Internet It can also be made available to the user 5. FIG. 7 shows an example in which the present invention is implemented in a mobile terminal device.

  As described above, according to the embodiment of the present invention, the user 5 does not use the 3D glasses, and the 3D image of the moving face pattern of the human face displayed on the information display unit 2 at 120 frames or more per second and easily displayed. While viewing the display, it is possible to answer reasonably according to question patterns related to themes such as facial expression identification, facial expression imitation, facial expression creation, and natural facial expression. At this time, the camera unit 3 performs stereoscopic shooting at 120 frames or more per second. Face information that appropriately captures minute movements of each part is extracted from the face image of the user 5 and whether the user 5 belongs to a group of normal subjects or a group of mild cognitive impairment based on the face information is extracted. Since it can be determined, diagnosis support for relatively early symptoms of dementia can be easily performed.

  Further, according to the embodiment of the present invention, in the determination processing unit 112, the face information of the user 5 extracted by the face image processing unit 111 and the faces belonging to the healthy person group stored in the healthy person face information storage part 122. The distance is calculated using the weight coefficient of each part of the face determined by the learning control unit 131 between the information or the face information belonging to the mild cognitive impairment group stored in the face information storage unit 123 of the mild cognitive impairment In addition, since it is possible to determine whether the user 5 belongs to the normal group or the mild cognitive impairment group based on the comparison result of the distance, machine learning is performed based on preset teacher information. The accuracy of the determination can be improved by determining the optimum weighting coefficient for each part of the face.

  Further, according to another embodiment of the present invention, a large number of users 5 can easily use for self-diagnosis from the Internet. In addition, since a doctor at a remote location can confirm the determination result held by the user 5 via the Internet, it can be used as a remote diagnosis.

  The present invention can be applied to early diagnosis of mild cognitive impairment or dementia.

  The present invention is not limited to the embodiments described in the specification, and other examples and modifications are included in the field to which the present invention belongs from the embodiments and ideas described in the present specification. It is obvious to

DESCRIPTION OF SYMBOLS 100 Facial expression recognition diagnosis support apparatus 1 Main body part 2 Information display part 3 Camera part 4 Operation part 5 User 11 Question pattern 12 Camera part 13 Answer candidate display 21 User's face image 22 Facial coordinate system 23 Face part position vector 31 Diagnosis support Mobile terminal device 32 Question pattern 33 Camera unit 34 Answer candidate display 101 Facial expression pattern storage unit 102 Question pattern storage unit 103 Display control unit 111 Face image processing unit 112 Determination processing unit 113 Determination information storage unit 121 User face information storage unit 122 Healthy Person face information storage unit 123 Mild cognitive impairment person face information storage part 131 Learning control part

Claims (6)

A facial expression pattern storage unit that pre-stores a plurality of moving image patterns obtained by stereoscopically photographing a human face at 120 frames or more per second;
A question pattern storage unit that stores a plurality of question patterns for the user in advance;
An operation unit that can accept input operations from the user,
A display control unit that controls the information display unit based on preset setting information and input information received by the operation unit;
Based on a command from the display control unit, an information display unit that displays a moving image pattern stored in the facial expression pattern storage unit and a question pattern stored in the question pattern storage unit;
A plurality of image sensors that can be photographed at 120 frames per second or more, a camera unit arranged so that a user's face that responds according to a moving image pattern and a question pattern displayed on the information display unit can be stereoscopically photographed,
A face image processing unit that extracts face information about each part of the face from a video imaged by the camera unit;
A user face information storage unit for storing the face information extracted by the face image processing unit;
A healthy person face information storage unit that stores face information belonging to the healthy person group in advance;
A mild cognitive impairment face information storage unit that stores face information belonging to the mild cognitive impairment group in advance;
Based on preset teacher information, the face information stored in the user face information storage unit and the face information stored in the healthy person face information storage unit or stored in the mild cognitive impairment face information storage unit A learning control unit that performs machine learning using the acquired face information and stores a weight coefficient of each part of the updated face;
The learning control between the face information extracted by the face image processing unit and the face information stored in the healthy person face information storage unit or the face information stored in the mild cognitive impairment person face information storage unit A determination processing unit that calculates a predetermined distance using the weighting factor stored in the unit, and determines whether the user belongs to a group of healthy persons or a group of mild cognitive impairments based on the distance;
A determination information storage unit for storing the determination result in the determination processing unit;
A facial expression recognition diagnosis support apparatus comprising:   The facial expression recognition diagnosis support apparatus according to claim 1, wherein a frame rate at the time of photographing by the camera unit is 240 frames per second. The face information extracted by the face image processing unit includes vector information and time information of each part of the user's face,
The face information stored in the healthy person face information storage unit includes vector information and time information of each part of the face of a person belonging to the healthy person group,
The face information stored in the mild cognitive impairment face information storage unit includes vector information and time information of each part of the face of a person belonging to the mild cognitive impairment group. The facial expression recognition diagnosis support device described in 1.   The determination processing unit 112 includes the face information of the user extracted by the face image processing unit, the face information stored in the normal person face information storage unit, or the face stored in the mild cognitive impairment face information storage unit. The Euclidean distance, the correlation distance, the Mahalanobis distance or the Canberra distance is calculated using the weighting factor stored in the learning control unit between the information, and based on the comparison result of the distance, The facial expression recognition diagnosis support apparatus according to claim 1, wherein it is determined which of the mild cognitive impairment group it belongs to.   When the face information of the user belongs to a healthy person group, the learning control unit performs machine learning so that the distance from the face information of the mild cognitive impairment group is small and the distance from the face information of the healthy person group is large. On the other hand, when the user's face information belongs to the mild cognitive impairment group, machine learning is performed so that the distance from the healthy person's face information is small and the distance from the mild cognitive group is large. The facial expression recognition diagnosis support apparatus according to claim 1, wherein an optimum weighting coefficient is determined. The facial expression recognition diagnosis support apparatus according to claim 1, wherein the facial expression recognition diagnosis support apparatus is mounted on a portable terminal device.