WO2023054295A1 - Information processing device, information processing method, and program - Google Patents

Abstract

Provided are an information processing device and the like for determining a type of facial nerve paralysis by using a face image.　The information processing device comprises: an image acquisition unit for acquiring a face image of a user; a feature amount acquisition unit for acquiring a feature amount of the face image on the basis of the face image; and a type determination unit for determining whether paralysis of the user is peripheral paralysis or central paralysis on the basis of the feature amount. The information processing device comprises a paralysis determination unit for determining presence/absence of facial nerve paralysis on the basis of the face image, and the feature amount acquisition unit acquires the feature amount when the paralysis determination unit determines that there is paralysis.

Description

Information processing device, information processing method and program

The present invention relates to an information processing device, an information processing method, and a program.

A determination device has been proposed that determines the possibility of a stroke based on the presence or absence of facial nerve palsy determined based on the user's facial image, the results of a speech test, and the results of an interview (Patent Document 1). Early detection and treatment of stroke can greatly improve patient prognosis.

Japanese Patent Application Laid-Open No. 2020-199072

There are two types of facial paralysis: central paralysis and peripheral paralysis. Among these, when central palsy has occurred, there is a high possibility that an intracerebral disease such as stroke has developed, and prompt treatment is required.

However, in the determination device of Patent Document 1, only the presence or absence of facial paralysis is determined using the facial image. In addition to photographing a face image, an utterance test and an interview are performed to determine whether or not there is a stroke, which is troublesome for the user and takes time for the examination.

In one aspect, an object is to provide an information processing device or the like that determines the type of facial paralysis using a face image.

The information processing apparatus includes an image acquisition unit that acquires a face image of a user, a feature acquisition unit that acquires a feature amount of the face image based on the face image, and a peripheral paralysis of the user based on the feature amount. and a type determination unit that determines whether the condition is a paralysis or a central paralysis.

In one aspect, it is possible to provide an information processing device or the like that determines the type of facial paralysis using a face image.

FIG. 4 is an explanatory diagram for explaining an outline of a determination process; It is an explanatory view explaining the composition of an information processor. FIG. 4 is an explanatory diagram for explaining feature points of a face image; FIG. 4 is an explanatory diagram for explaining a record layout of a feature amount DB; It is an explanatory view explaining a decision model. 4 is a flowchart for explaining the flow of processing of a program; FIG. 10 is a flowchart for explaining the flow of processing of a subroutine for calculating a feature amount relating to the forehead; FIG. FIG. 11 is a flowchart for explaining the flow of processing of a subroutine for feature amount calculation regarding eyebrows; FIG. FIG. 10 is a flowchart for explaining the flow of processing of a subroutine for feature amount calculation relating to between eyes and eyebrows; FIG. FIG. 11 is a flow chart for explaining the flow of processing of a subroutine for feature amount calculation relating to a law line; FIG. FIG. 11 is a flow chart for explaining the flow of processing of a subroutine for calculating a feature amount related to mouth corners; FIG. FIG. 10 is a flowchart for explaining the flow of processing of a subroutine for calculating a feature amount relating to the mouth; FIG. FIG. 11 is a flowchart for explaining the flow of processing of a subroutine for feature amount calculation regarding contours; FIG. FIG. 10 is an explanatory diagram for explaining an example of a screen; It is an explanatory view explaining an example of a screen of a modification. FIG. 11 is an explanatory diagram for explaining the configuration of an information processing device according to a third embodiment; FIG. 11 is a functional block diagram of an information processing device according to a fourth embodiment;

[Embodiment 1]
There are two types of facial paralysis, central paralysis and peripheral paralysis. It is known that the majority of facial paralysis is peripheral paralysis, and the incidence of central paralysis is less than 1% of all facial paralysis.

In central paralysis, paralysis occurs mainly around the cheeks and mouth. The main cause of central paralysis is so-called stroke, such as cerebral infarction, cerebral hemorrhage, subarachnoid hemorrhage, and transient ischemic attack. It is known that the earlier the discovery and treatment of these diseases, the better the patient's prognosis. Treatment for central palsy is primarily performed in neurology and neurosurgery.

In peripheral paralysis, paralysis may occur over the entire face, from the upper part of the face such as the forehead and eyebrows to the lower part of the face such as the mouth. In addition, symptoms often differ on the left and right sides of the face. The main causes of peripheral paralysis are swelling of the facial nerve and neuritis caused by herpes virus. Early treatment is desirable for peripheral paralysis, but the urgency is lower than for central paralysis. Treatment for peripheral palsy is mainly performed by an otolaryngologist.

Therefore, it is important for patients with facial nerve palsy to accurately distinguish between central and peripheral palsy and to visit the appropriate department at the appropriate time. The Yanagihara method, the House-Brackmann method, the Sunnybrook method, and the like have been conventionally used in clinical practice when doctors examine patients with facial nerve palsy.

However, even if a general patient sees the symptoms of facial nerve palsy, it is difficult to distinguish whether it is central or peripheral palsy and select an appropriate department. For example, when a patient who has developed central palsy consults an otolaryngologist, it is necessary to move to a neurosurgery or the like, which delays the start of treatment. On the other hand, if many patients with peripheral palsy visit a neurosurgeon, the neurosurgery department may become overcrowded, delaying the treatment of patients who truly need urgent treatment.

In the present embodiment, an information processing device 20 (see FIG. 2) and the like that assists general patients to consult an appropriate clinical department and receive appropriate medical care will be described.

Fig. 1 is an explanatory diagram explaining the outline of the determination process. The user makes a plurality of specified facial expressions and takes a facial image with each facial expression. In FIG. 1, "No. 1" is "rest", "No. 2" is "wrinkles on the forehead", "No. 3" is "slightly closed eyes", and "No. 4" is "strongly closed eyes". Give an example. These facial expressions are four of the ten facial expressions conventionally used in the Yanagihara method.

In the following explanation, the case of using these four types of facial expressions will be explained as an example. These facial expressions are examples and are not intended to be limiting. For example, more than 5 or less than 3 facial expressions may be used. Expressions other than those specified by the Yanagihara method may be used.

The presence or absence of facial paralysis is determined based on multiple facial images. For example, if a plurality of facial images change according to a designated facial expression, it is determined that there is no facial paralysis. A method for determining the presence or absence of facial nerve palsy based on a facial image is known, for example, from Patent Document 1, and therefore detailed description thereof will be omitted.

If it is determined that there is no facial paralysis, the user is notified that there is no facial paralysis. For example, a patient who has had a stroke and is worried about the risk of another stroke can be reassured by a notification that he or she does not have facial paralysis.

When it is determined that there is facial nerve palsy, feature values are calculated for each of the multiple evaluation parts such as "forehead" and "eyebrows". Examples of evaluation parts and feature amounts will be described later. It is determined whether the facial paralysis is peripheral paralysis or central paralysis by combining the feature values of a plurality of evaluation sites.

If it is determined that you have peripheral paralysis, you will be notified that you are recommended to see an otolaryngologist. A list showing addresses, telephone numbers, consultation hours, URLs (Uniform Resource Locators), etc. of nearby otolaryngologists may be notified.

If it is determined that you have central palsy, you will be notified that it is recommended that you urgently consult a neurologist or neurosurgeon. A list showing addresses, telephone numbers, consultation hours, URLs, etc. of neurological departments or neurosurgeons in the vicinity may be notified. Contacts such as pre-registered family members, helpers or primary care physicians may be notified.

Improving the prognosis of patients while reducing waste of medical resources by promptly determining central palsy, which occurs less frequently than facial nerve palsy but is highly urgent for treatment, and recommending an appropriate clinical department. can be achieved. It is of great social significance that the number of patients suffering from aftereffects due to delays in starting treatment for central paralysis has decreased.

FIG. 2 is an explanatory diagram for explaining the configuration of the information processing device 20. As shown in FIG. The information processing device 20 includes a control unit 21, a main storage device 22, an auxiliary storage device 23, a communication unit 24, a display unit 25, an input unit 26, a speaker 27, an imaging unit 28, and a bus.

The control unit 21 is an arithmetic control device that executes the program of the present embodiment. One or a plurality of CPUs (Central Processing Units), GPUs (Graphics Processing Units), multi-core CPUs, or the like is used for the control unit 21 . The control unit 21 is connected to each hardware unit forming the information processing apparatus 20 via a bus.

The main storage device 22 is a storage device such as SRAM (Static Random Access Memory), DRAM (Dynamic Random Access Memory), flash memory, or the like. The main storage device 22 temporarily stores information necessary during the process performed by the control unit 21 and the program being executed by the control unit 21 .

The auxiliary storage device 23 is a storage device such as SRAM, flash memory, hard disk, or magnetic tape. The auxiliary storage device 23 stores a feature DB (database) 41, a determination model 46, programs to be executed by the control unit 21, and various data necessary for executing the programs. The feature amount DB 41 and the judgment model 46 may be stored in an external large-capacity storage device connected to the auxiliary storage device 23 .

The communication unit 24 is an interface that performs communication between the information processing device 20 and the network. The display unit 25 is, for example, a liquid crystal display panel or an organic EL (electro-luminescence) panel. The input unit 26 is, for example, a keyboard, mouse and microphone. The display unit 25 and the input unit 26 may be laminated to form a touch panel.

The imaging unit 28 is, for example, a general camera. The imaging unit 28 may be, for example, an infrared camera. The photographing unit 28 may be a stereoscopic photographing camera configured with two cameras. The imaging unit 28 may be a 3D TOF (3 Dimension Time of Flight) sensor. When using a camera for stereoscopic photography or a 3D TOF sensor, the photographing unit 28 can photograph a three-dimensional face image.

The display unit 25, the input unit 26, the speaker 27 and the imaging unit 28 may be external devices. The photographing unit 28 is desirably arranged at a position where the user's face can be photographed while the user is looking at the display unit 25 .

For example, the information processing device 20 is a general-purpose information device used by a user, such as a smartphone, personal computer, tablet, or smart speaker. The information processing device 20 may be dedicated hardware prepared for the determination method of the present embodiment. Information processing device 20 may be incorporated in, for example, a pet-type robot or a care robot.

The information processing device 20 operates on a large-scale computer connected via a network to so-called IoT devices such as a display unit 25, an input unit 26, a speaker 27, and a photographing unit 28 placed near the user. It may be configured by a combination with a virtual machine, a cloud computing system, or the like.

The functions of the information processing device 20 may be implemented by cooperating with a plurality of pieces of hardware. For example, the determination of the presence or absence of facial nerve palsy described using FIG. 1, the feature amount calculation for each part, and the determination of the type of paralysis may be performed by different hardware.

An overview of the determination procedure using this embodiment will be described. The control unit 21 displays the image captured through the image capturing unit 28 on the display unit 25 . The control unit 21 may display a guide indicating an appropriate face position and the like in the image to guide the user to the appropriate position and distance to the photographing unit 28 . If the imaging unit 28 has sufficiently high pixels, the control unit 21 may cut out an appropriate range from the image captured by the imaging unit 28 and display it on the display unit 25 .

The control unit 21 outputs from the speaker 27 a voice commanding a "rest" facial expression such as "open your eyes and close your mouth". The control unit 21 may display on the display unit 25 an illustration, characters, or the like explaining an appropriate facial expression. The user makes an instructed facial expression while confirming his/her own face displayed on the display unit 25 . The control unit 21 takes a face image. The control unit 21 calculates the feature amount of each part from the photographed face image and records it in the feature amount DB 41 .

The control unit 21 outputs a voice commanding the next facial expression from the speaker 27, such as "Look up and wrinkle your forehead." The user makes the indicated facial expression. The control unit 21 takes a face image. The control unit 21 calculates feature amounts and records them in the feature amount DB 41 .

The control unit 21 repeats the above processing, captures a face image with a predetermined expression, and records the feature amount. Note that the control unit 21 may output an instruction to return to the initial "rest" facial expression between each facial expression. The control unit 21 may capture a face image and record the feature amount each time the user returns to the "rest" expression.

Table 1 is a table outlining the method of distinguishing between central paralysis and peripheral paralysis.

The bilateral innervation area is the area where nerves from the left and right cerebral cortices reach, and is generally the area above the eyes. The unilateral innervation area is an area to which nerves from the cerebral cortex on the left and right opposite sides reach, and is generally the area below the eye.

In healthy people, paralysis does not occur in either the bilateral or unilateral areas of control. Patients with central palsy develop paralysis in the unilateral area of supremacy, but not in the bilateral area of supremacy. Patients with peripheral palsy develop paralysis in both unilateral and bilateral territorial tracts. In Table 1, "N/A (Not Applicable)" is indicated because the symptom of paralysis in the bilateral ruling area and no paralysis in the unilateral ruling area does not occur clinically.

Table 2 is a table showing the parts belonging to the bilateral dominant region and the unilateral dominant region, respectively, and an overview of the feature values of each part. Table 2 shows examples of feature amounts that can be calculated based on captured images of the four types of facial expressions described using FIG. A description of each feature amount will be given later.

In the bilateral dominant region, the control unit 21 determines that there is no paralysis when the amount of change in feature amount when changing facial expressions between different facial expressions is greater than a predetermined threshold. The control unit 21 may score the amount of change in the feature amount for each part of the bilateral areas of control according to a predetermined procedure, and determine the presence or absence of paralysis in the bilateral areas of control based on the total score or the like.

In the unilateral dominant region, the control unit 21 determines the presence or absence of paralysis based on the combination of the difference between the left and right feature amounts and the amount of change in the feature amount when the facial expression is changed between different facial expressions. For example, when the difference between the left and right feature amounts does not increase even when the facial expression is changed, the control unit 21 determines that there is no paralysis. The control unit 21 scores the difference between the left and right feature amounts and the amount of change in the feature amount for each part of the unilateral dominant area according to a predetermined procedure, and determines the presence or absence of paralysis in the unilateral dominant area based on the total score or the like. You can judge.

FIG. 3 is an explanatory diagram explaining the feature points 49 of the face image. Each feature point 49 is indicated by a black circle in FIG. Characteristic points 49 that are particularly used in the following description are denoted by alphabetical symbols such as characteristic point 49a and characteristic point 49b. Extraction of the feature points 49 can be realized by a known method such as OpenPose, and therefore detailed description is omitted.

In the following description, FIG. 3 will be described as an example of a so-called mirror image format in which the user's right hand side is displayed on the right side of FIG. 3 and the user's left hand side is displayed on the left side of FIG. Accordingly, the right side of FIG. 3 corresponds to the user's right eye.

Using Fig. 3, an overview of the method of calculating the feature amount will be explained. The control unit 21 calculates the scale of the face image based on the feature points 49 described using FIG. 3 and the size of the user's face registered in advance. The control unit 21 may calculate the scale of the face image based on the average face size of the user. The average face size may be defined for each user's gender or age, for example. In the following description, the case where the coordinates of each feature point 49 are determined by the coordinates corresponding to the actual size of the user's face will be described as an example.

　The outline of the method for calculating each feature amount is explained. The control unit 21 performs edge detection on the forehead portion of the face image. Since edge detection is an image processing technique that has been commonly used in the past, detailed description thereof will be omitted. The number of horizontal line-shaped edges detected by edge detection is the number of wrinkles on the forehead.

The control unit 21 calculates the depth of wrinkles based on the difference in brightness between the edge portion and its surroundings. Note that the relationship between the difference in brightness and the depth of wrinkles is measured in advance and incorporated into the program of the present embodiment. If the photographing unit 28 has a function of photographing a three-dimensional shape, the control unit 21 can directly read the depth of wrinkles from the face image.

The eyebrow angle means the angle between the left and right eyebrows and the midline of the face. The midline of the face is a straight line passing through the bridge of the nose. Specifically, the control unit 21 calculates a straight line L1 obtained by linearly approximating a plurality of feature points 49a indicating the bridge of the nose. The control unit 21 calculates a straight line L2 connecting the feature points 49b and 49c indicating both ends of the user's right eyebrow. The control unit 21 calculates the angle θa formed by L1 and L2. θa is the angle of the right eyebrow.

The control unit 21 calculates an arc that approximates the array of all feature points 49 related to the right eyebrow. The reciprocal of the radius of the calculated arc is the curvature of the right eyebrow. The control unit 21 similarly calculates the angle and curvature of the left eyebrow.

The control unit 21 calculates the area of the area surrounded by the line connecting the feature point 49 regarding the user's left eyebrow and the feature point 49 connecting the upper edge of the right eye. The ends of the two curves are interpolated by straight lines, for example. The portion for calculating the area is indicated by left-down hatching in FIG. The calculated area is the area between the left eyebrow and the left eye. The control unit 21 similarly calculates the area between the right eyebrow and the right eye.

The control unit 21 performs edge detection on the portion from the left side of the nose to the vicinity of the left corner of the mouth in the face image. The length of the sharpest edge detected by edge detection is the length of the left statute. The control unit 21 calculates the depth of the left law line. Since the depth calculation method is the same as that for the depth of wrinkles on the forehead, the description is omitted. The control unit 21 similarly calculates the length and depth of the right normal line.

The control unit 21 calculates a straight line L3 that connects the feature points 49d and 49e indicating the right and left corners of the user's mouth. The control unit 21 calculates a straight line L4 connecting the representative point of the left eye and the representative point of the right eye. The representative point is, for example, the center of gravity of each eye. The representative point may be the inner corner or the outer corner of each eye. The control unit 21 calculates the angle formed by L3 and L4. The calculated angle is the angle of the corner of the mouth.

The control unit 21 calculates the area of the area surrounded by the curve or polygonal line that connects the characteristic point 49 regarding the lower edge of the upper lip and the characteristic point 49 regarding the upper edge of the lower lip. A portion for calculating the area is indicated by hatching sloping downward to the right in FIG. The calculated area is the area of the mouth. If the user's mouth is tightly closed, the mouth area is zero.

The control unit 21 calculates an arc approximating the arrangement of all feature points 49 related to the contour of the right side of the face. The reciprocal of the calculated radius of the arc is the curvature of the right contour. The control unit 21 similarly calculates the curvature of the contour on the left side.

Through the above procedure, the control unit 21 can calculate the feature values shown in Table 2. The control unit 21 calculates a feature amount for each face image obtained by photographing a plurality of facial expressions described using FIG.

FIG. 4 is an explanatory diagram for explaining the record layout of the feature DB 41. FIG. The feature amount DB 41 is a DB that records the feature amount for each part calculated based on each face image. The feature amount DB 41 has a date field and a feature amount field. The feature amount field has facial expression fields corresponding to the respective facial expressions described using FIG.

Each facial expression field, such as the rest field, has a part field indicating a part of the face, such as a forehead field, an eyebrow field, a field between the eyebrows and the eyes, a normal line field, a corner of the mouth field, a mouth field, and an outline field.

The forehead field has a wrinkle number field and a wrinkle depth field. The eyebrow field has a left field and a right field. The left and right fields have angle and curvature fields respectively. The eyebrow-to-eye field has a left area field and a right area field.

The law line field has a right field and a left field. The right field and left field each have a depth field and a length field. The mouth corner field has an angle field. The mouth field has an area field. The contour field has a right field and a left field.

The date field records the date when the type of paralysis was determined. The number of wrinkles field records the number of wrinkles on the forehead. A representative value of the depth of wrinkles is recorded in the wrinkle depth field. The depth of each wrinkle may be recorded in the wrinkle depth field.

The angles formed by the left and right eyebrows and the bridge of the nose are recorded in the angle field within the eyebrow field. Similarly, the curvature field records the curvature of the left and right eyebrows. The area between the right eyebrow and the upper edge of the right eye is recorded in the right area field in the eyebrow-to-eye field. Similarly, the left area field records the area between the left eyebrow and the upper edge of the left eye.

In the depth field within the law line field, the depth of each left and right law line is recorded. Similarly, the length field records the length of each of the left and right legal lines. The angle formed by a line connecting the left and right corners of the mouth and a line connecting the centers of the left and right eyes is recorded in the mouth corner field.

The area of the region surrounded by the upper edge of the lower lip and the lower edge of the upper lip is recorded in the mouth field. Left and right fields within the contour field record the curvature of the left and right contour lines. Note that the feature amount DB 41 may have fields for recording face images corresponding to respective facial expressions.

FIG. 5 is an explanatory diagram for explaining the judgment model 46. FIG. The judgment model 46 accepts a series of feature values recorded in one record in FIG. is a model that outputs

The judgment model 46 is a logic-based program that outputs judgment results according to a known judgment method such as the Yanagihara method. The user's attending physician or the like may be able to appropriately adjust the determination parameters and the like of the determination model 46 in accordance with the user's symptoms such as pre-existing diseases. The judgment model 46 may be a model generated by machine learning using training data recording combinations of multiple sets of feature values and judgment results.

If the information processing device 20 is dedicated hardware, the determination model 46 may be configured in hardware using FPGA (Field Programmable Gate Array) or ASIC (Application Specific Integrated Circuit).

Note that the determination model 46 may output only one of the probability of "peripheral paralysis" or the probability of "central paralysis". The judgment model 46 may output a judgment result indicating either "peripheral paralysis" or "central paralysis".

FIG. 6 is a flowchart explaining the flow of program processing. Control unit 21 outputs a voice commanding a facial expression such as "Please open your eyes and close your mouth" from speaker 27 to instruct the user on the facial expression to be captured next (step S501). Through step S501, the control section 21 implements the function of the facial expression instruction section.

The control unit 21 controls the photographing unit 28 to photograph a face image and stores it in the auxiliary storage device 23 (step S502). By step S502, the control unit 21 realizes the function of the image acquiring unit that acquires the user's face image. The control unit 21 determines whether or not shooting of a predetermined facial expression has been completed (step S503). If it is determined that the processing has not ended (NO in step S503), the control unit 21 returns to step S501.

When it is determined that the process has ended (YES in step S503), the control unit 21 determines whether or not there is facial paralysis based on the series of photographed face images (step S504). As described above, the method for determining the presence or absence of facial nerve palsy based on a facial image is known, for example, from Patent Document 1, and therefore detailed description thereof is omitted. By step S504, the control unit 21 realizes the function of the paralysis determination unit.

When it is determined that there is no facial paralysis (NO in step S504), the control unit 21 outputs a message such as "no facial paralysis" from the speaker 27 or the display unit 25 (step S505). The control unit 21 terminates the processing.

When determining that there is facial paralysis (YES in step S504), the control unit 21 selects one facial image from the facial images captured in step S502 (step S511). The control unit 21 extracts the feature points 49 described using FIG. 3 from the face image (step S512).

The control unit 21 selects one part described using Table 2 (step S513). The control unit 21 starts a subroutine for feature amount calculation (step S514). The feature amount calculation subroutine is a subroutine for calculating the feature amount related to the part selected in step S513. The processing flow of the feature amount calculation subroutine for each part will be described later. By executing the feature amount calculation subroutine, the control unit 21 realizes the function of the feature amount acquisition unit.

The control unit 21 records the calculated feature amount in the feature amount DB 41 (step S515). The control unit 21 realizes the function of the feature amount recording unit in step S515. The control unit 21 determines whether or not the calculation of the feature amount for the predetermined part has ended (step S516). If it is determined that the processing has not ended (NO in step S516), the control unit 21 returns to step S513.

If it is determined that processing has been completed (YES in step S516), the control unit 21 determines whether processing of all face images captured in step S502 has been completed (step S517). If it is determined that the processing has not ended (NO in step S517), the control unit 21 returns to step S511.

If it is determined to have ended (YES in step S517), the control unit 21 uses the determination model 46 described with reference to FIG. A certain probability is obtained (step S518).

The control unit 21 determines whether the probability of central paralysis is high (step S519). It should be noted that in step S519, it is determined that there is central paralysis when the probability of having central paralysis exceeds a predetermined determination threshold value. By step S519, the control unit 21 realizes the function of the type determination unit that determines whether the user has peripheral paralysis or central paralysis.

A specific description will be given by taking as an example the case where the determination threshold in step S519 is set to 20%. When the output of the determination model 46 has a 30% probability of central paralysis and a 70% probability of peripheral paralysis, the control unit 21 determines that there is central paralysis in step S519 (YES in step S519). . By appropriately setting the determination threshold, the risk of overlooking central paralysis can be reduced.

If it is determined that the patient has central palsy (YES in step S519), the control unit 21 simply displays the determination result, for example, "Due to suspicion of stroke paralysis, it is recommended to see a neurosurgeon." The indicated message is output by voice or text (step S520).

The control unit 21 outputs a message indicating the basis of the judgment and the judgment result, for example, "There is paralysis of the law line, and central facial nerve palsy, which is a symptom of stroke, was suspected with a probability of 70%." Or you can output it as a character. The control unit 21 may superimpose an indicator indicating the location where it is determined that there is paralysis on the face image indicating the grounds for the determination. By showing the reason for the determination, the user can realize the necessity of prompt medical examination.

The control unit 21 may display a list of nearby neurological departments or neurosurgeons on the display unit 25. The control unit 21 may display a search button for searching neurology or neurosurgery on the display unit 25 .

The control unit 21 may automatically notify a pre-registered contact such as a family member, helper, or primary care doctor that the user has developed central palsy. The control unit 21 may receive an instruction from the user as to whether or not to notify the contact. If the information processing device 20 is a device having a call function such as a smart phone, the control unit 21 may display an emergency call button for making an emergency call such as 119, for example.

If it is determined that the patient does not have central palsy (NO in step S519), the control unit 21 may say, for example, "Because of suspicion of paralysis of the facial nerve, which is different from cerebral apoplexy, a visit to an otolaryngologist is recommended." A message that briefly indicates the determination result is output by voice or text (step S521). After completing step S520 or step S521, the control unit 21 ends the process. Through steps S520 and S521, the control unit 21 realizes the function of a type output unit that outputs whether the user has peripheral paralysis or central paralysis.

FIG. 7 is a flow chart for explaining the processing flow of the subroutine for calculating feature values relating to the forehead. The feature amount calculation subroutine is a subroutine for calculating the feature amount related to the part selected in step S513. When "amount" is selected in step S513, the control unit 21 starts a subroutine described using FIG. 7 in step S514.

The control unit 21 performs edge detection on the forehead portion of the face image (step S531). As described above, the edges detected as horizontal lines are wrinkles on the forehead. The control unit 21 calculates the number of wrinkles based on the edge detection result (step S532). Control unit 21 calculates the depth of wrinkles, for example, based on the difference in brightness between the edge portion and its surroundings (step S533). The control unit 21 terminates the processing.

In step S533, for example, the control unit 21 calculates the depth of all detected forehead wrinkles. The control unit 21 may further calculate a representative value such as an average value or maximum value of depth. Control unit 21 may calculate the depth of a typical wrinkle, such as the longest wrinkle or the central wrinkle, among the detected forehead wrinkles.

FIG. 8 is a flow chart for explaining the processing flow of the subroutine for feature amount calculation regarding eyebrows. When "amount" is selected in step S513, the control unit 21 starts a subroutine described using FIG. 8 in step S514.

The control unit 21 performs linear approximation of the plurality of feature points 49 indicating the bridge of the nose to calculate the median line (step S541). The control unit 21 calculates a straight line connecting the characteristic points 49 indicating both ends of one eyebrow (step S542). The control unit 21 calculates the angle between the straight line calculated in step S541 and the straight line calculated in step S542 (step S543).

The control unit 21 calculates an arc that approximates the arrangement of the plurality of feature points 49 indicating one eyebrow (step S544). Through step S544, the coordinates of the center point of the arc approximating one eyebrow and the radius are calculated. The control unit 21 calculates the curvature of the eyebrow, which is the reciprocal of the radius (step S545). The control unit 21 terminates the processing.

FIG. 9 is a flow chart for explaining the processing flow of a subroutine for feature amount calculation between eyebrows and eyes. When "Between eyebrows and eyes" is selected in step S513, the control unit 21 activates a subroutine described using FIG. 9 in step S514.

The control unit 21 acquires coordinates of a plurality of characteristic points 49 indicating one eyebrow (step S551). The control unit 21 acquires coordinates of a plurality of feature points 49 indicating the upper edge of the eye on the same side (step S552). The control unit 21 calculates the area of the area surrounded by straight lines or curved lines connecting the feature points 49 acquired in steps S551 and S552 (step S553). The control unit 21 terminates the processing.

FIG. 10 is a flow chart for explaining the processing flow of the subroutine for feature amount calculation related to the law line. When the control unit 21 selects "regulation line" in step S513, the control unit 21 starts a subroutine described using FIG. 10 in step S514.

The control unit 21 performs edge detection on the part of the face image from one side of the nose to the vicinity of the corner of the mouth on the same side (step S561). The control unit 21 selects an edge corresponding to the law line from the detected edges (step S562). Specifically, the control unit 21 selects the sharpest edge or the longest edge from among the edges that are farther from the midline toward the lower side of the face.

The control unit 21 calculates the length of the law line (step S563). Control unit 21 calculates the depth of the law line, for example, based on the difference in brightness between the edge portion and its surroundings (step S564). The control unit 21 terminates the processing.

FIG. 11 is a flowchart for explaining the processing flow of a subroutine for feature amount calculation related to mouth corners. When "corner of the mouth" is selected in step S513, the control unit 21 starts a subroutine described using FIG. 11 in step S514.

The control unit 21 calculates straight lines connecting the characteristic points 49 indicating the left and right corners of the mouth (step S571). The control unit 21 calculates representative points for each of the left and right eyes. For example, the control unit 21 calculates the center of gravity of a plurality of feature points 49 corresponding to the edge of one eye and uses it as a representative point. The control unit 21 may use, for example, the feature point 49 indicating the inner corner or the outer corner of the eye as the representative point. The control unit 21 calculates a straight line connecting the representative points of the left and right eyes (step S572).

The control unit 21 calculates the angle formed by the straight line calculated in step S571 and the straight line calculated in step S572 (step S573). The control unit 21 terminates the process.

FIG. 12 is a flow chart for explaining the processing flow of the subroutine for calculating feature values relating to the mouth. When "mouth" is selected in step S513, the control unit 21 starts a subroutine described using FIG. 12 in step S514.

The control unit 21 acquires coordinates of a plurality of feature points 49 corresponding to the lower edge of the upper lip (step S581). The control unit 21 acquires coordinates of a plurality of feature points 49 corresponding to the upper edge of the lower lip (step S582). The control unit 21 calculates the area of the area surrounded by straight lines or curved lines connecting the feature points 49 acquired in steps S581 and S582 (step S583). The control unit 21 terminates the processing.

FIG. 13 is a flow chart for explaining the processing flow of a subroutine for contour feature amount calculation. When "outline" is selected in step S513, the control unit 21 starts a subroutine described using FIG. 13 in step S514.

The control unit 21 calculates an arc approximating the arrangement of the plurality of characteristic points 49 representing one of the left and right contours (step S591). Through step S591, the coordinates of the center point of the arc that approximates one contour and the radius are calculated. The control unit 21 calculates the curvature of the contour, which is the reciprocal of the radius (step S592). The control unit 21 terminates the processing.

FIG. 14 is an explanatory diagram for explaining a screen example. FIG. 14 shows an example of a screen on which the control unit 21 notifies the user that there is a symptom of central paralysis in step S520 of the flowchart explained using FIG.

On the screen shown in FIG. 14, a face image column 71, a result column 72, a medical institution contact button 73, and a family contact button 74 are displayed. The result column 72 displays sentences indicating the locations where it was determined that there was paralysis and the probability of having central paralysis.

The user's face image is displayed in the face image column 71 . In the face image field 71, an index 76 is displayed to indicate the location where it is determined that there is paralysis.

When the user selects the medical institution contact button 73, the control unit 21 displays the pre-registered contact information for medical institutions such as neurosurgery. The control unit 21 may input the user's address and a keyword such as "neurosurgery" into a known search engine and display the search results. If the information processing device 20 is a device such as a smart phone that has a call function, the control unit 21 may initiate a call to a pre-registered medical institution.

When the user selects the family contact button 74, the control unit 21 sends a notice to the pre-registered family members that the user has developed central nervous system paralysis. For the notification, e-mail, SMS (Short Message Service), or message addressed to SNS (Social Network Service) can be used, for example. If the information processing device 20 is a device such as a smart phone that has a call function, the control unit 21 may initiate a call to a family member's phone number registered in advance.

According to the present embodiment, it is possible to provide the information processing apparatus 20 that determines whether the facial paralysis that the user has developed is peripheral paralysis or central paralysis. According to the present embodiment, it is possible to provide the information processing apparatus 20 that recommends a visit to an appropriate department according to the type of paralysis that the user has developed. It is possible to provide the information processing apparatus 20 that contributes to a significant improvement in the user's prognosis by allowing the user to promptly visit an appropriate clinical department.

According to the present embodiment, it is possible to provide the information processing device 20 that determines the type of facial paralysis only from the user's face image. Since the operation is simple, it is possible to provide the information processing apparatus 20 that even elderly people can easily use.

According to the present embodiment, it is possible to provide the information processing device 20 that notifies the user when the user has not developed facial paralysis. A patient who feels uneasy about the onset of facial paralysis can use the information processing apparatus according to the present embodiment as appropriate to confirm that the onset does not occur, and feel relieved.

The information processing apparatus 20 of the present embodiment may be used, for example, by firefighters in charge of emergency transportation. By determining the type of paralysis that the patient to be transported has developed, the transport destination can be appropriately selected.

Note that the control unit 21 may acquire each feature amount described using Table 2 using a model generated so as to output the feature amount when a face image is input. Such models are generated by machine learning, for example, using training data recording combinations of face images and feature quantities. The model for outputting the feature amount and the determination model 46 are integrated into a model, and when a face image is input, the type of paralysis may be output.

[Modification]
This modification relates to screen display for users who are undergoing rehabilitation after developing facial paralysis. Descriptions of parts common to the first embodiment are omitted.

　Whether it is central paralysis or peripheral paralysis, long-term rehabilitation may be required during the recovery period after acute treatment is completed. In this modified example, the control unit 21 displays the effect of rehabilitation based on the feature amount recorded in the feature amount DB 41 in time series.

FIG. 15 is an explanatory diagram for explaining a screen example of the modified example. A face image field 71 and a result field 72 are displayed on the screen shown in FIG. The control unit 21 compares the feature amount recorded in the past with the latest feature amount for the portion of the law line where the facial nerve palsy has developed. The control unit 21 displays the change in the user's state between the past and the present in the result column 72 based on the difference in feature amount.

According to this modification, the information processing apparatus 20 that increases the user's motivation for rehabilitation can be provided by objectively judging the effect of rehabilitation and displaying it in the result column 72 .

[Embodiment 2]
The present embodiment relates to an information processing apparatus 20 that determines the type of paralysis using facial images in which features of each part are likely to emerge. Descriptions of parts common to the first embodiment are omitted.

Table 3 shows the relationship between each part and the facial expressions that are susceptible to the type of paralysis.

For example, with regard to “legal lines”, “corners of the mouth”, “mouth” and “contour”, the two facial expressions of “rest” and “strongly closed eyes” are influenced by the type of central or peripheral paralysis. easy to receive. Therefore, even if the facial expressions of "wrinkles on the forehead" and "slightly closed eyes" are not calculated for these parts, the determination result is not significantly affected.

An overview of the processing performed by the control unit 21 in this embodiment will be described. In step S513 of the program described using FIG. 6, the control unit 21 selects only parts suitable for using the facial expression of the face image selected in step S511. For example, if a face image with a "quiet" expression is selected in step S511, the control unit 21 selects the "forehead", "legal lines", "corners of the mouth", "mouth" and "contour" in step S513. Select sequentially.

Similarly, if a face image with a facial expression of "wrinkle the forehead" is selected in step S511, the control unit 21 sequentially selects the "eyebrows" and "between the eyebrows and the eyes" in step S513.

According to the present embodiment, an information processing apparatus 20 is provided that performs appropriate determination with a small amount of calculation by not calculating feature amounts that have little effect on the determination of central paralysis or peripheral paralysis. can.

[Embodiment 3]
The present embodiment relates to a mode of realizing the information processing apparatus 20 of the present embodiment by operating a general-purpose computer 90 and a program 97 in combination. Descriptions of parts common to the first embodiment are omitted.

FIG. 16 is an explanatory diagram for explaining the configuration of the information processing device 20 according to the third embodiment. The computer 90 includes a control section 21, a main storage device 22, an auxiliary storage device 23, a communication section 24, a display section 25, an input section 26, a speaker 27, an imaging section 28, a reading section 29 and a bus. The computer 90 is an information device such as a general-purpose personal computer, tablet, smart phone, or server computer.

The program 97 is recorded on a portable recording medium 96. The control unit 21 reads the program 97 via the reading unit 29 and stores it in the auxiliary storage device 23 . Control unit 21 may also read program 97 stored in semiconductor memory 98 such as a flash memory installed in computer 90 . Furthermore, the control unit 21 may download the program 97 from another server computer (not shown) connected via the communication unit 24 and a network (not shown) and store it in the auxiliary storage device 23 .

The program 97 is installed as a control program of the computer 90, loaded into the main storage device 22 and executed. As described above, the computer 90 functions as the information processing apparatus 20 described above. Program 97 is an example of a program product.

[Embodiment 4]
FIG. 17 is a functional block diagram of the information processing device 20 according to the fourth embodiment. The information processing apparatus 20 includes an image acquisition section 81 , a feature amount acquisition section 82 and a type determination section 83 .

The image acquisition unit 81 acquires the user's face image. The feature quantity acquisition unit 82 acquires the feature quantity of the face image based on the face image. The type determination unit 83 determines whether the user has peripheral paralysis or central paralysis based on the feature amount.

The technical features (constituent elements) described in each embodiment can be combined with each other, and new technical features can be formed by combining them.
The embodiments disclosed this time are illustrative in all respects and should be considered not restrictive. The scope of the present invention is not defined by the above-described meaning, but is indicated by the scope of claims, and is intended to include all modifications within the meaning and scope equivalent to the scope of claims.

(Appendix 1)
an image acquisition unit that acquires a face image of a user;
a feature amount acquisition unit that acquires a feature amount of the face image based on the face image;
and a type determination unit that determines whether the user has peripheral paralysis or central paralysis based on the feature amount.

(Appendix 2)
Based on the facial image, comprising a paralysis determination unit that determines the presence or absence of facial nerve paralysis,
The information processing apparatus according to appendix 1, wherein the feature amount acquisition unit acquires the feature amount when the paralysis determination unit determines that there is paralysis.

(Appendix 3)
A facial expression instruction unit that instructs the user to express a facial expression,
The information processing apparatus according to appendix 1 or appendix 2, wherein the image acquisition unit acquires the face image of the user after the facial expression instruction unit issues an instruction.

(Appendix 4)
The facial expression instruction unit instructs a facial expression of wrinkling the forehead,
The information processing apparatus according to appendix 3, wherein the feature amount acquisition unit acquires the number and depth of wrinkles on the forehead.

(Appendix 5)
The facial expression instruction unit instructs a facial expression of wrinkling the forehead, a slightly closed eye expression, and a strongly closed eye expression,
The information processing apparatus according to appendix 3 or appendix 4, wherein the feature amount acquisition unit acquires angles and curvatures of left and right eyebrows for each image.

(Appendix 6)
The facial expression instruction unit instructs a facial expression of wrinkling the forehead, a slightly closed eye expression, and a strongly closed eye expression,
According to any one of appendices 3 to 5, the feature amount acquisition unit acquires an area between the left eyebrow and the eye and an area between the right eyebrow and the eye for each image. Information processing equipment.

(Appendix 7)
The information processing apparatus according to any one of appendices 1 to 6, further comprising a type output unit that outputs whether the user has peripheral palsy or central palsy.

(Appendix 8)
When the type determination unit determines that it is peripheral paralysis, output information about otolaryngology,
The information processing apparatus according to any one of appendices 1 to 7, wherein information relating to neurology or neurosurgery is output when the type determination unit determines that the patient has central palsy.

(Appendix 9)
The information processing apparatus according to any one of appendices 1 to 8, wherein a predetermined contact is notified when the type determination unit determines that the patient has central paralysis.

(Appendix 10)
A feature amount recording unit that records the feature amount in time series,
The information according to any one of Supplements 1 to 9, wherein the type determination unit determines whether the user has peripheral paralysis or central paralysis based on time-series changes in the feature amount. processing equipment.

(Appendix 11)
A feature amount recording unit that records the feature amount in time series,
The information processing apparatus according to any one of appendices 1 to 9, wherein the user's state change is output based on a time-series change in the feature amount.

(Appendix 12)
The information processing apparatus according to any one of appendices 1 to 11, wherein the image obtaining unit obtains the face image of the user as a three-dimensional image.

(Appendix 13)
Get the user's facial image,
acquiring a feature amount of the face image based on the face image;
An information processing method in which a computer executes a process of determining whether the user has peripheral palsy or central palsy based on the feature amount.

(Appendix 14)
Get the user's facial image,
acquiring a feature amount of the face image based on the face image;
A program that causes a computer to execute a process of determining whether the user has peripheral paralysis or central paralysis based on the feature amount.

20 information processing device 21 control unit 22 main storage device 23 auxiliary storage device 24 communication unit 25 display unit 26 input unit 27 speaker 28 photographing unit 29 reading unit 41 feature amount DB
46 judgment model 49 feature point 71 face image column 72 result column 73 medical institution contact button 74 family contact button 76 index 81 image acquisition unit 82 feature amount acquisition unit 83 type determination unit 90 computer 96 portable recording medium 97 program 98 semiconductor memory

Claims (14)

1. an image acquisition unit that acquires a face image of a user;
   a feature amount acquisition unit that acquires a feature amount of the face image based on the face image;
   and a type determination unit that determines whether the user has peripheral paralysis or central paralysis based on the feature amount.
2. Based on the facial image, comprising a paralysis determination unit that determines the presence or absence of facial nerve paralysis,
   The information processing apparatus according to claim 1, wherein the feature amount acquisition unit acquires the feature amount when the paralysis determination unit determines that there is paralysis.
3. A facial expression instruction unit that instructs the user to express a facial expression,
   The information processing apparatus according to claim 1, wherein the image obtaining section obtains the face image of the user after the facial expression instruction section gives an instruction.
4. The facial expression instruction unit instructs a facial expression of wrinkling the forehead,
   The information processing apparatus according to claim 3, wherein the feature quantity acquisition unit acquires the number and depth of wrinkles on the forehead.
5. The facial expression instruction unit instructs a facial expression of wrinkling the forehead, a slightly closed eye expression, and a strongly closed eye expression,
   The information processing apparatus according to claim 3, wherein the feature amount acquisition unit acquires angles and curvatures of left and right eyebrows for each image.
6. The facial expression instruction unit instructs a facial expression of wrinkling the forehead, a slightly closed eye expression, and a strongly closed eye expression,
   The information processing apparatus according to claim 3, wherein the feature amount acquisition unit acquires an area between the left eyebrow and the eye and an area between the right eyebrow and the eye for each image.
7. The information processing apparatus according to claim 1, further comprising a type output unit that outputs whether the user has peripheral palsy or central palsy.
8. When the type determination unit determines that it is peripheral paralysis, output information about otolaryngology,
   The information processing apparatus according to claim 1, wherein information relating to neurology or neurosurgery is output when the type determination unit determines that the patient has central palsy.
9. The information processing apparatus according to claim 1, wherein when the type determination unit determines that the patient has central palsy, a predetermined contact is notified.
10. A feature amount recording unit that records the feature amount in time series,
    10. The type determination unit according to any one of claims 1 to 9, wherein the type determination unit determines whether the user has peripheral paralysis or central paralysis based on time-series changes in the feature amount. information processing equipment.
11. A feature amount recording unit that records the feature amount in time series,
    The information processing apparatus according to any one of claims 1 to 9, wherein the state change of the user is output based on the time-series change of the feature quantity.
12. The information processing apparatus according to claim 1, wherein the image acquisition unit acquires the face image of the user as a three-dimensional image.
13. Get the user's facial image,
    acquiring a feature amount of the face image based on the face image;
    An information processing method in which a computer executes a process of determining whether the user has peripheral palsy or central palsy based on the feature amount.
14. Get the user's facial image,
    acquiring a feature amount of the face image based on the face image;
    A program that causes a computer to execute a process of determining whether the user has peripheral paralysis or central paralysis based on the feature amount.

Images