KR101580038B1 - Computerized objective measurement of facial motion for facial palsy functional grading - Google Patents

Abstract

According to the present invention, the motion of the facial muscles is acquired using the two image capturing devices, the mobility of the facial muscles is analyzed from the dynamic viewpoint using the motion distance and speed of the facial muscles, By correcting the image by tracking the movement of the entire face, accurate measurement can be performed without restricting the muscle movement of the facial region to be measured.

Description

[0001] The present invention relates to a method for quantifying facial nerve paralysis using dynamic motion measurement,

The present invention relates to a method for quantifying facial nerve paralysis, and more particularly, to a method for objectively quantifying the degree of facial paralysis by analyzing the movement speed of various facial muscles through computer image tracking technology.

The facial nerve is a unique nerve that passes through the longest and narrowest bone tissue in our body, and nerve palsy is a relatively common disease. Facial paralysis is the most common cause of Bell's palsy, with a good prognosis but the most common cause of facial paralysis in the United States is 15% of the common cause, such as inflammatory diseases such as herpes zoster, , Brain tumor, cerebral hemorrhage, and facial trauma. Unilateral facial paralysis can be a very lethal cause for sophisticated social life, and is a very serious disease with intangible loss, such as mental pain, rather than just a few muscle disorders. Therefore, if the prognosis is considered to be bad, aggressive treatment and adequate rehabilitation training should be recommended early, but there is a tendency to consider the disease very lightly in Korea.

Quantifying the degree of facial paralysis is important in planning a treatment plan or determining the prognosis. The facial nerve is responsible for 23 pairs of facial muscle movements, so it needs local evaluation as well as overall balance of both sides.

In 1985, the House-Brackmann facial paralysis rating system proposed by AAO-HNS (House-Bracm

(SFGS), which has a disadvantage of measuring only a part of the facial muscles and not being able to reflect changes such as synkinesis, although it is commonly used throughout the world. The localized assessment methods are presented, but like the HB-FGS, there is a limit to the subjective measurement method.

In the case of the electononeurography (ENoG), which is an important criterion for determining the prognosis of facial palsy or surgery, even if the correlation with the clinical stage is recognized, it is often difficult to accurately match the subjective tests including HB-FGS .

Recently, a variety of facial measurement techniques have been tried, including ultra-high speed imaging at over 2000fps, various facial muscle change measurement techniques using various 3D rendering techniques, and real-time facial modeling using illumination. However, these methods require special camera equipment, Or a special lighting device is needed, so that it can be applied to simple modeling. Furthermore, very fast imaging is necessary to measure changes in facial muscles such as synkinesis, but the methods presented today are not likely to adequately reflect this.

Therefore, it is very urgent to have a simple diagnostic system which is more convenient and economical in the medical environment such as telemedicine / home medical care that will be developed in the future, and raises the accuracy to the level similar to the 3D rendering.

In addition, the existing methods of quantifying facial paralysis evaluate the degree of facial paralysis only in a static state. On the other hand, patients actually complain about the unnaturalness of dynamic facial muscles in the treatment process, and even if they are judged as normal in the existing evaluation method, It is easy to see the case of the remaining. This is because the conventional facial nerve evaluation method emphasizes the concept of position of momentum and does not reflect pure motility while evaluating the nerve conduction and function as the muscle's momentum.

The present invention provides a method of quantifying facial palsy which is objective and has small error between examinations by measuring the mobility of facial muscles through image processing using a webcam and a computer.

This method of quantifying facial paralysis can be used as a method of providing information for diagnosis and treatment of facial paralysis.

SUMMARY OF THE INVENTION The present invention has been made to solve the above-mentioned problems, and it is an object of the present invention to provide a method and apparatus for acquiring moving images of a face using an image acquisition device and analyzing the motility of the facial muscles using the movement distance and speed of facial muscles, And corrects the image by tracking the movement of the entire face.

That is, a method for quantifying facial nerve paralysis according to an aspect of the present invention is a method for quantifying facial nerve paralysis using two image capturing apparatuses and an information processing apparatus connected to the two image capturing apparatuses, Acquiring a moving image of a face marked with a marker using an acquisition device; Transmitting the obtained moving image to the information processing apparatus; A step of tracking the facial contour using the moving image obtained by one of the two image capturing apparatuses by the information processing apparatus and correcting the moving image obtained by the other one of the two image capturing apparatuses; And calculating the facial motility through the corrected moving image by the information processing apparatus.

Calculating the facial motility includes: measuring a moving distance of the marker by the information processing apparatus; And calculating the moving speed using the measured moving distance by the information processing apparatus.

The step of measuring the movement distance of the marker may include: measuring a movement distance of the marker for a predetermined time; And calculating a ratio between the right and left sides of the face. The calculating of the moving speed may be performed by dividing the moving distance measured by the photographing speed of the image capturing apparatus.

Wherein the correcting step comprises: correcting unique information associated with the image acquiring device; Correcting information on the focal length; And correcting the rotation and the movement of the face, and is preferably performed using the following equations (1) and (2).

[Equation 1]

&Quot; (2) "

The calculation result of the facial movement can be displayed through the information processing apparatus.

A method for quantifying facial nerve paralysis according to another aspect of the present invention is a method for quantifying facial nerve paralysis using an image acquiring apparatus and an information processing apparatus connected to the image acquiring apparatus, Acquiring a moving image of the face; Transmitting the obtained moving image to the information processing apparatus; Measuring a moving distance of the marker by the information processing apparatus; And calculating the moving speed using the measured moving distance by the information processing apparatus.

The method may further include displaying the still image before and after the movement of the face, the movement distance of the marker, and the calculated movement speed through the display unit of the information processing apparatus.

According to the present invention, it is possible to obtain the result of quantification of facial paralysis which is very objective and has small error between examinations. In addition, the result of the quantification of facial paralysis according to the present invention reflects the subjective discomfort of the patient, and has the effect of being able to achieve an infinite gradation rather than a simple grade of about six stages.

1 is a diagram illustrating a measurement system for quantifying facial paralysis according to an embodiment of the present invention.
Figure 2 shows the location of the periocular markers used in the facial palsy quantification system according to an embodiment of the present invention.
3 is a flowchart of a method for quantifying facial palsy according to an embodiment of the present invention.
FIG. 4 shows a screen showing measurement examples of a facial palsy quantification system according to an embodiment of the present invention and an algorithm of motion calculation.
FIG. 5 is a screen display example of a facial paralysis quantification system according to an embodiment of the present invention, and shows a result of automatically quantifying facial paralysis on a screen.
FIG. 6 is a graph showing measurement results of facial motion of a healthy person for use as a comparative example.
7 is a graph comparing results of HB-FGS, facial nerve conduction test (ENoG), and facial paralysis according to the embodiment of the present invention, which are the conventional facial paralysis evaluation methods.
8 is a graph comparing measured results of HB-FGS, ENoG, and the facial paralysis method according to the embodiment of the present invention at the onset of facial paralysis and on the 14th day.

Hereinafter, a method for quantifying facial nerve paralysis through dynamic motion measurement according to an embodiment of the present invention will be described with reference to the drawings. In the following description of the present invention, a detailed description of known functions and configurations incorporated herein will be omitted when it may make the subject matter of the present invention rather unclear.

1 is a view showing a measurement system (hereinafter referred to as a "facial palsy quantification system") for quantifying facial paralysis according to an embodiment of the present invention.

1, the system for quantifying facial palsy according to an embodiment of the present invention includes two webcams 110 and 120 and a webcam 110 and 120 disposed on the front face 10 of a patient to be measured, And a computer (not shown) that receives and processes the measured moving image. A support for supporting a patient's face for seating, a support for supporting the face of the patient, an alignment unit for aligning the face of the patient and the webcam, and the like, but these components are generally understood in the art A detailed description thereof will be omitted.

One of the two webcams 110 and 120 is used to track the contours of the face, tracking the shaking of the entire head, and the other 120 is used to record the movement of the facial muscles.

When measuring facial motion with a webcam, head fixation is essential to obtain certain geometric data. However, excessive hair fixation may act as a limiting factor for facial muscle movements. Accordingly, in the facial paralysis quantification system according to the embodiment of the present invention, two webcams are used to perform the facial contour so that the effect of the measurement points is least affected even if the mouth is opened or interrupted without interfering with the measurement of the facial muscle motion Track. Specifically, in the embodiment shown in Fig. 1, the same two webcams 20 cm in front of the face portion were vertically arranged.

Figure 2 shows the location of the periocular markers used in the facial palsy quantification system according to an embodiment of the present invention.

The positions of the markers to be displayed on the face were selected from the face measurement points provided by the Korea Standards Research Institute (http://www.kriss.re.kr) to reflect the largest movement around the eyeball, In the experiment, water-soluble red paint was left on the selected spot with a 2 mm diameter tubular paint. However, the position and the number of the markers displayed on the face may be varied as needed, and there is no particular limitation on how to display them.

Now, a method of quantifying facial palsy using the system as described above and an experimental example practiced through the method will be described in detail.

3 is a flowchart of a method for quantifying facial palsy according to an embodiment of the present invention.

As shown in FIG. 3, first, a marker is displayed on the face of a patient to be diagnosed (S310).

Next, the moving picture is acquired using the two webcams as described above with respect to the motion of the face marked with the marker (S320).

The acquisition of moving images using a digital camera can show objective differences in image results depending on the surrounding illuminance. Therefore, in order to maintain constant illuminance, four LED lights with visible light wavelengths were illuminated in a straight line at a distance of 20 cm from the room without external illumination. The protocol for facial muscle motility recording recorded the movement of the muscles around the eye and the muscles of the mouth in the cases of "eye closure", "E", "O" After training the test protocol to be repeated 10 times or more, the final test was performed under the same conditions.

The obtained moving image is transmitted to a computer and corrected (S330).

Calibration of the camera and analysis program is performed by printing a 1 mm diameter black dot on a transparent film of A4 size in the form of a lattice of 0.5 cm intervals and attaching it to the front of the test tube and calculating the pixel ratio of the image obtained through the camera And assigned to the analysis program. If the theoretical point X, Y, Z is the coordinate in the real space, the transformed u and v are regarded as the pixel coordinates of the image obtained from the camera. The matrix A is used to correct the camera-specific information with the camera matrix, and the correction for the focal length is made through this matrix. And [R | t] is a factor to correct the rotation and movement. It is a factor that reflects the effect of external camera position rather than camera specific information. Therefore, when the above matrix information is obtained through the correction, and analysis is performed through the obtained images, it is possible to minimize the distortion of the two-dimensional projection of the real three-dimensional information. Specifically, correction of the image can be performed using the following equations (1) and (2).

Next, in order to analyze facial motility, the moving distance and speed of the marker are calculated (S340).

Specific methods for analyzing facial motility are as follows.

Using the open CV method, the moving distance was measured for a certain time using the binary image processing method and the moving pixel quantification method, and the ratio was calculated from the left and right sides of the face. The ratio of both sides obtained here is expressed as the value of the ratio of the unsteady side to the normal side, and when the final value is '1', it is read normally. The reason for doing this is to minimize the error when taking the marker. This method is the same as ENoG. Thus, the value obtained by dividing the value obtained by ENoG by 100 is obtained.

First, using the following equation (3), the difference image obtained to measure the moving distance per unit time is converted into a binary image.

Next, as shown in Equation (4), the moving distance of each pixel is extracted by subtracting the point (i, j) at which the marker is marked from the difference image,

The moving speed Vmax of the marker point is calculated by dividing the previously measured moving distance by the imaging speed of 30 fps, as shown in the following equation (5).

FIG. 4 shows a screen showing measurement examples of a facial palsy quantification system according to an embodiment of the present invention and an algorithm of motion calculation.

As shown in Fig. 4, the moving distance of the markers displayed around both eyes is obtained, and the velocity of the movement is calculated to display the two values.

In the screen example of FIG. 4, the distance and velocity values are displayed for the right and left eyes in the upper right portion, and the portion for displaying the mobility of the upper muscles in the lower right portion.

FIG. 5 shows another example of screen display of the facial paralysis quantification system according to the embodiment of the present invention, which automatically displays the result of quantification of facial paralysis on the screen.

In FIG. 5, a photograph showing the distance between the eye muscle markers and the photographs before and after the movement of the facial muscles is displayed on the screen, and the distances and velocities before and after the movement of the facial muscles are displayed.

Now, a detailed experimental example of the method of quantifying facial palsy according to an embodiment of the present invention will be described in detail.

This study was conducted prospectively in the form of patient - control study with the permission of the clinical trial screening committee (GIRBA2304) for patients who visited the tertiary hospital in Incheon from November 2009 to March 2010. Subjects participating in the study were the normal control group (y = 44 ± 9.68, n = 10; male / female ratio = 4/6) and facial paralysis group (y = 45 ± 10.51; n = 15; male / female ratio = , right- / left-side lesions = 7/8).

Patients with facial paralysis were Bell's palsy and were admitted to the hospital within 3 days after the onset of the illness. On the 7th day after the visit, facial muscle motility test and ENoG using the webcam according to the embodiment of the present invention were performed. In the control group, the same test was performed by voluntary participation in healthy adults considering facial paralysis group and age.

In facial paralysis patients, facial asymmetry due to photosensitivity or trauma, wrinkles, facial asymmetry, skin allergies due to water paint, skin pigmentation, facial changes due to diabetes or heart disease, difficulty in contraction and relaxation of vaginal facial muscles Patients were excluded from the selection process.

Image acquisition is a webcam ^{(PLEOMAX ® PWC-2200, Samsung} , Seoul, Korea) and WindowsO Movie Maker was used ^{(Microsoft ®, Redmond, WA,} US), the acquired video is IBM-compatible computer (Core ™ 2 Quad CPU Q6600 @ 2.40 GHz, Intel ^® , Santa Clara, Calif., USA) with 2 GB of DDR3 RAM, Samsung, Seoul, Korea. Windows ^® XP SP3 (Microsoft ^® , Redmond, WA, USA) was used as the operating system of the computer. The camera settings for recording are 800x600pixel size with B / W mode, 60Hz flicker, Gamma = 0.18, Contrast = 1.08, Brightness = 16, White balance = auto, USB bandwidth = auto, Frame rate = 1fps, Time limit = 3600s Saved as '* .avi' file. (Microsoft ^® , Redmond, WA, USA), Matlab ^® v6.5 (MathWorks Inc, MA, US) and Excel 2007 (Microsoft ^® , Redmond, WA, USA) was used.

Based on the image thus obtained, measurement results of the normal control group and the facial paralysis group were obtained using the algorithm according to the facial palsy quantification method according to the embodiment of the present invention described above with reference to FIG.

Statistical analysis was performed using linear regression and R-square, Pearson's r, p-value (95% confidence interval), Unpaired t-test with F-score using Prism ^® 4 for windows (Graphpad software Inc., test for variances (95% confidence interval).

FIG. 6 is a graph showing measurement results of facial motion of a healthy person for use as a comparative example.

In FIG. 6A, ERL and ELL represent eye lengthening distance (ERL), eye left length (ELL), pixel] of the left and right eyeballs, ERV and ELV represent eye lining speed (ELV); pixel / s]. FIG. 6B and 6C show the distribution of coefficient of variation of motion distance and velocity.

ERL and ELL, ERV and ELV were 65.19 ± 2.23, 70.78 ± 1.85 and 157.87 ± 23.04 and 164.95 ± 23.60 respectively. (P = 1.39E-15). There was no significant difference between the left and right eyes due to high deviation (p = 0.24).

(Y = 0.6021x - 0.6322, R² = 0.5258) for the distance, and the velocity (y) for the distance (Y = 0.8721x - 0.3575, R² = 0.7249) for the second and third quadrants.

7 is a graph comparing results of HB-FGS, facial nerve conduction test (ENoG), and facial paralysis according to the embodiment of the present invention, which are the conventional facial paralysis evaluation methods.

In order to compare with the facial paralysis method according to the embodiment of the present invention, the House-Brackmann grading method and the facial nerve conduction test (ENoG), which are most widely used as the conventional facial nerve evaluation method, were performed.

 HB-FGS was applied to all patients according to the opinions of one otorhinolaryngologist and three otolaryngologists who had experience of facial paralysis treatment. The results are shown in [Table 1].

Patient Autologist R1 R2 R3 ENoG This zero IV III IV IV 75 Shin 0 V V V V 100 Song 0 IV V IV IV 83.3 Kim, Young-Kyung IV IV IV V 56.2 Kim 0 Kyu IV VI V V 21.1 One ear V V V V 93.3 Night V V V V 91.7 Choi Wan Wan II II II II 27.3 Zhao County V IV IV IV 42.9 Kim 0 weeks II II II II 7.7 0 Wan II III III III 40.9 Choi Jin II II II II 66.6 Ball 0 V VI V V 93.8 Choi Young Young IV III III III 40.0 Hong-hee Hong III IV IV IV 81.3

Electrononeuronography and needle electromyography were performed to objectively assess the damage of the acute facial nerve itself. The needle electromyography was performed by inserting the fine needle electrode into the nasalis muscles and the oculomotor muscles of both sides and measuring the dislocation during the resting period and the vasoconstriction period.

As shown in FIG. 7, the result of ENoG showed a high correlation with HB-FGS (FIG. 7A), and the measurement result of the facial palsy quantification method according to the embodiment of the present invention also showed a high correlation with HB-FGS (Fig. 7C).

HB-FGS is classified into six stages. Therefore, even if the HB-FGS correction value is proportional to ENoG, the correlation coefficient between the eye clipping distance and the velocity is better when the eye clipping distance and the velocity left / right ratio (maximum value = 1) And this was the same on day 14 as well as on day 7.

8 is a graph comparing measured results of HB-FGS, ENoG, and the facial paralysis method according to the embodiment of the present invention at the onset of facial paralysis and on the 14th day.

As shown in FIG. 8, the method of quantifying facial palsy according to an embodiment of the present invention shows better accuracy than the HB-FGS method.

Facial expressions are completed by cooperative movement of facial muscles. In this process, incoordination of multiple muscle movements is perceived as inconvenience to the patient even though facial expression is the same. The methods of evaluating the existing facial paralysis by image analysis use a head restraint device and may limit the movement of some facial muscles. However, the facial paralysis evaluation system according to the embodiment of the present invention is not limited to the motion of the entire face The ability to track and calibrate it is advantageous because it allows precise measurements without restricting the muscle movements of the facial area to be fully measured. In addition, when the protocol used is used, the measurement time is less than 5 minutes and the error between the measured values is small.

While the present invention has been described in connection with what is presently considered to be the most practical and preferred embodiment, it is to be understood that the invention is not limited to the disclosed embodiments, but is capable of various modifications within the scope of the invention. Therefore, the scope of the present invention should not be limited by the illustrated embodiments, but should be determined by the scope of the appended claims and equivalents thereof.

Claims (9)

delete delete A method for quantifying facial nerve paralysis using two image acquisition devices and an information processing device connected to the two image acquisition devices,
Obtaining a moving image of a face displaying a marker using the two image acquisition devices;
Transmitting the obtained moving image to the information processing apparatus;
A step of tracking the facial contour using the moving image obtained by one of the two image capturing apparatuses by the information processing apparatus and correcting the moving image obtained by the other one of the two image capturing apparatuses; And
And calculating facial motility through the corrected moving image by the information processing apparatus,
Wherein the step of calculating the facial movement comprises:
Measuring a moving distance of the marker moved according to the movement of the face by the information processing apparatus; And
And calculating the moving speed using the measured moving distance by the information processing apparatus,
Wherein the step of measuring the movement distance of the marker comprises:
Measuring a moving distance of the marker for a predetermined time; And
And calculating a ratio from the left to the right of the facial plane.
The method of claim 3,
Wherein the calculating the moving speed comprises:
And calculating the moving distance by dividing the measured moving distance by the imaging speed of the image acquiring device.
A method for quantifying facial nerve paralysis using two image acquisition devices and an information processing device connected to the two image acquisition devices,
Obtaining a moving image of a face displaying a marker using the two image acquisition devices;
Transmitting the obtained moving image to the information processing apparatus;
A step of tracking the facial contour using the moving image obtained by one of the two image capturing apparatuses by the information processing apparatus and correcting the moving image obtained by the other one of the two image capturing apparatuses; And
And calculating facial motility through the corrected moving image by the information processing apparatus,
Wherein the correcting comprises:
And correcting the unique information associated with the image acquisition device,
The unique information associated with the image acquisition device may include:
And information about a focal length of the image acquisition device or information about rotation and movement of the facial image.
6. The method of claim 5,
Wherein the correcting comprises:
A method for quantifying facial nerve palsy performed using the following equations (1) and (2).
[Equation 1]

&Quot; (2) "

The method according to claim 3 or 5,
And displaying the calculation result of the facial movement through the information processing apparatus.
A method for quantifying facial nerve paralysis using an image acquisition device and an information processing device connected to the image acquisition device,
Obtaining a moving image of a face displaying a marker using the image acquiring device;
Transmitting the obtained moving image to the information processing apparatus;
A step of correcting the moving image acquired by the image acquiring device by tracking the facial contour using the moving image acquired by the image acquiring device by the information processing device; And
And calculating facial motility through the corrected moving image by the information processing apparatus,
Wherein the step of calculating the facial movement comprises:
Measuring a moving distance of the marker moved according to the movement of the face by the information processing apparatus; And
And calculating the moving speed using the measured moving distance by the information processing apparatus,
Wherein the step of measuring the movement distance of the marker comprises:
Measuring a moving distance of the marker for a predetermined time; And
And calculating a ratio from the left to the right of the facial plane.
9. The method of claim 8,
Displaying the still image before and after the movement of the face, the movement distance of the marker, and the calculated movement speed through the display unit of the information processing apparatus
≪ / RTI > further comprising the step of quantifying facial nerve paralysis.

Images