JP2023070542A - Diagnostic equipment of extrapyramidal symptom and diagnostic program of extrapyramidal symptom - Google Patents

Abstract

To provide a technology capable of properly diagnosing an extrapyramidal symptom.SOLUTION: Diagnostic equipment of extrapyramidal symptom comprises a control part. The control part executes to acquire a dynamic image obtained by taking a diagnosis target person, to acquire image data in a time series from the dynamic image, to obtain multiple feature points of the target person from the image data, to obtain locational information of each feature point in the time series, to obtain feature quantity of each feature point based on the change of the locational information with a time course of each feature point obtained in the time series, to calculate the severity of an extrapyramidal symptom to calculate the estimated severity of the extrapyramidal symptom when the feature quantity of the feature point is inputted, and by inputting the feature quantity obtained from the dynamic image obtained by taking the target person in a learnt diagnostic model of a machine learning-processed diagnostic model using training data, and to output the calculated severity of the extrapyramidal symptom.SELECTED DRAWING: Figure 1

Description

The present invention relates to an extrapyramidal symptom diagnostic device and an extrapyramidal symptom diagnostic program.

Extrapyramidal symptoms are a general term for symptoms caused by abnormalities in the extrapyramidal pathways other than the pyramidal pathways, among the neural pathways that issue commands to peripheral movements from the motor area of the cerebral cortex. Known extrapyramidal symptoms include tremor, drug-induced parkinsonism, dyskinesia, dystonia, akathisia, and parkinsonian symptoms. For example, dyskinesias may manifest symptoms such as repetitive pursed lips, mouth muzzling, mouth thrusting, teeth clenching, and wrinkled eyes when closed.

WO2018/179419 Japanese Patent Application Laid-Open No. 2014-168700

In order to determine the treatment policy for extrapyramidal symptoms, it is necessary to properly diagnose what kind of symptoms are present and to what extent. appears repeatedly, resulting in similar symptoms, making clear distinction difficult. Moreover, these symptoms may appear in combination. For example, when dyskinesia symptoms and dystonia symptoms appear, it is important to appropriately diagnose the severity of these symptoms, since the medication policy may be determined depending on which symptom is more severe. However, the severity of these symptoms is generally determined by a doctor's findings, and there is a problem that, for example, it is very difficult to make a diagnosis based on a fixed standard during a long-term treatment process.

An object of the disclosed technique is to provide a technique for appropriately diagnosing extrapyramidal symptoms.

One aspect of the disclosed technique is exemplified by the configuration of the following extrapyramidal symptom diagnosis device. That is, the present extrapyramidal symptom diagnosis device
Acquiring image data in chronological order from moving images of a person to be diagnosed;
Obtaining a plurality of feature points of the subject from the image data, and obtaining position information of each feature point in time series;
Obtaining a feature amount of each feature point based on a change in position information of each feature point over time obtained in time series;
When the feature amount of each feature point is input, a trained diagnostic model that has been subjected to machine learning processing using teacher data so as to calculate the estimated degree of extrapyramidal symptoms is added to the subject calculating the degree of the extrapyramidal symptom by inputting the feature amount obtained from the moving image photographed;
outputting the calculated degree of extrapyramidal symptoms;
A control unit for executing

In the extrapyramidal symptom diagnosis device, the feature value is the mean value, median value, standard deviation, variance, minimum value, maximum value, skewness, and kurtosis obtained for the amount of change in the positional information over time. , the sum,
At least one of the sum of squares, the number of crossing over the average value, and the average power spectrum may be used.

In the extrapyramidal symptom diagnostic apparatus, the change with time of the position information at each feature point may be a change in position relative to a reference position of the face of the subject.

In the apparatus for diagnosing extrapyramidal symptoms, the reference part of the face is the nose, and the feature point is a part of the subject's face that is a specific part including lips and peripheral parts of the lips. may

In the extrapyramidal symptom diagnosis device, the control unit
When calculating the degree of the extrapyramidal symptom, obtaining the feature point that has a high degree of influence on the calculation result;
displaying the feature points having the high degree of influence superimposed on the image;
may also be performed.

One aspect of the disclosed technology is exemplified by the following extrapyramidal symptom diagnosis program. That is, this extrapyramidal symptom diagnostic program is
Acquiring a moving image of a person to be diagnosed;
obtaining image data in time series from the moving image;
Obtaining a plurality of feature points of the subject from the image data, and obtaining position information of each feature point in time series;
Obtaining a feature amount of each feature point based on a change in position information of each feature point over time obtained in time series;
When the feature amount of each feature point is input, a trained diagnostic model that has been subjected to machine learning processing using teacher data so as to calculate the estimated degree of extrapyramidal symptoms is added to the subject calculating the degree of the extrapyramidal symptom by inputting the feature amount obtained from the moving image photographed;
outputting the calculated degree of extrapyramidal symptoms;
is executed by the control unit.

Furthermore, one aspect of the disclosed technique is exemplified by a computer-readable recording medium recording this extrapyramidal symptom diagnosis program. The function can be provided by causing a computer to read and execute the program of this recording medium.

Here, a computer-readable recording medium is a recording medium that can store information such as data and programs by electrical, magnetic, optical, mechanical, or chemical action and can be read by a computer, etc. Say. Examples of such recording media that can be removed from a computer or the like include flexible discs, magneto-optical discs, CDs (Compact Discs), CD-R/Ws, DVDs (Digital Versatile Discs), Blu-ray discs, (registered trademark) Disc), DAT, 8 mm tape, and memory cards such as flash memory. In addition, there are a hard disk, a ROM (read only memory), and the like as a recording medium fixed to a computer or the like.

According to the technique of the present disclosure, it is possible to appropriately diagnose extrapyramidal symptoms.

1 is a schematic configuration diagram of an extrapyramidal symptom diagnosis device according to the present embodiment; FIG. FIG. 4 is an explanatory diagram of processing for extracting feature points; It is a figure which shows the temporal displacement of a feature point. 4 is a graph showing changes in position of feature points over time; FIG. 4 is an explanatory diagram of power spectrum; An example in which 10 feature points having a high degree of influence are obtained and superimposed on an image is shown. A control unit of a diagnostic device is a diagram showing a process of performing machine learning according to a program. FIG. 4 is a diagram showing processing executed by a control unit of the diagnostic device according to an extrapyramidal symptom diagnostic program;

Hereinafter, an extrapyramidal symptom diagnosis device according to one embodiment will be described with reference to the drawings. The configuration of the following embodiment is an example, and the present extrapyramidal symptom diagnosis device is not limited to the configuration of the embodiment.

"Device configuration"
FIG. 1 is a schematic configuration diagram of an extrapyramidal symptom diagnostic device (hereinafter also simply referred to as a diagnostic device) 10 according to this embodiment. The diagnostic apparatus 10 inputs an image obtained by photographing the facial expression of the patient 20 to a diagnostic model, and calculates a diagnostic result of extrapyramidal symptoms. This diagnostic model captures the face of the patient 20 for a predetermined period of time (for example, several tens of seconds), extracts a plurality of facial feature points from the captured image, and obtains the displacement of each feature point in chronological order. Deep learning (machine learning) was performed using determined labels as teacher data. As a result, the extrapyramidal symptom diagnostic apparatus 10 of the present embodiment mechanically diagnoses the facial expression of the patient 20 with AI, for example, the presence or absence of symptoms of tremor, dyskinesia, or dystonia, and the degree of the symptoms It enables appropriate diagnosis of extrapyramidal symptoms.

《Diagnostic device》
The diagnostic device 10 is a computer having a control unit 12, a memory 13, an input/output IF (interface) 14, a communication IF 15, and an imaging device 16, which are interconnected by a connection bus 11. FIG. The control unit 12 processes input information and outputs processing results to control the entire apparatus. The control unit 12 is also called a CPU (Central Processing Unit) or an MPU (Micro-processing unit). The control unit 12 is not limited to a single processor, and may have a multiprocessor configuration. Also, a multi-core configuration having a plurality of cores in a single chip connected by a single socket may be used.

The memory 13 includes a main memory and an auxiliary memory. The main storage device is used as a work area for the control unit 12, a storage area for temporarily storing information processed by the control unit 12, and a buffer area for communication data. The main storage device is a storage medium for the control unit 12 to cache programs and data, and to develop work areas. The main memory includes, for example, RAM (Random Access Memory), ROM (Read Only Memory), and flash memory. The auxiliary storage device is a storage medium that stores programs executed by the control unit 12, data used for information processing, operation setting information, data used as diagnostic models, and the like. Auxiliary storage devices are, for example, HDDs (Hard-disk Drives), SSDs (Solid State Drives), EPROMs (Erasable Programmable ROMs), flash memories, USB memories, memory cards, and the like. In addition, the auxiliary storage device stores images to be diagnosed (captured images) and diagnosis results.

The input/output IF 14 is an interface for inputting/outputting data with devices connected to the diagnostic device 10 . The input/output IF 14 performs data input/output with devices such as a disk drive that reads data from a storage medium such as a CD or DVD, an operation unit, a display device, and a photographing device 16, for example. The operation unit is an input unit such as a mouse, a keyboard, a touch panel, or the like, through which information to the diagnostic apparatus 10 is input by operator's operation. The display device is an output unit that displays and outputs information such as processing results to the operator.

The communication IF 15 is an interface (communication module) that communicates with other devices via a communication line (network), and is also called a CCU (Communication Control Unit). For example, the communication IF 15 may be configured to receive information from, or transmit information to, a computer or imaging device located at a location other than the diagnostic device 10, such as the patient's 20 home.

The imaging device 16 captures the face of the patient 20 for a predetermined period of time to acquire moving image data, and inputs the moving image data to the control unit 12 via the input/output IF 14 . The imaging device 16 is a so-called camera, but may be a three-dimensional sensor or a ToF (Time of Flight) sensor as long as it can acquire moving image data. Alternatively, the imaging device 16 may be a stereo camera capable of obtaining three-dimensional information by imaging the face of the patient 20 with two horizontally spaced cameras.

The imaging device 16 is arranged, for example, to face the patient 20 so that the vertical and horizontal directions of the space in which the patient is present coincide with the vertical and horizontal directions of the captured image. Therefore, for example, when the patient 20 tilts his/her face with respect to the vertical direction, the tilt can be identified from the image.

A plurality of each of the components 11 to 16 of the diagnostic device 10 may be provided, or some components may be omitted. For example, the diagnostic apparatus 10 may be configured to acquire an image of the patient 20 captured by an external imaging device via the input/output IF 14 and the communication IF 15 without the imaging device 16 .

In the diagnostic apparatus 10 of the present embodiment, the control unit 12 executes an application program, whereby the control unit 12 includes a feature point determination unit 111, a preprocessing unit 112, a feature amount calculation unit 113, a machine learning unit 114, a diagnostic It functions as each processing unit such as the unit 115 and the result output unit 116 . That is, the control unit 12 can also be used as each processing unit according to the software to be executed. However, part or all of each of the above processing units is DSP (Digital Signal Processor), ASIC (Application Specific Integrated Circuit), FPGA (Field-Programmable Gate Array) such as dedicated LSI (large scale integration), logic circuit, etc. It may be formed by hardware such as a digital circuit. Further, at least part of each processing unit may include an analog circuit. The control unit 12 may have a configuration in which one processor functions as a plurality of processing units, or may have a configuration in which a plurality of processors function as one processing unit.

The feature point determining unit 111 acquires a moving image shot by the shooting device 16 from the shooting device 16, or acquires a moving image shot by another device via the input/output IF 14 or the communication IF 15, Acquire image data in time series from moving images. For example, when a moving image is composed by photographing a plurality of frames per second, each frame is obtained in photographing order and used as image data. The frame rate of moving images is not particularly limited, but is, for example, 15 fps to 120 fps, preferably 25 fps to 90 fps, and more preferably 30 fps to 60 fps. The frame rate of moving images in this embodiment is 30 fps. If the frame rate of the acquired moving image is different from the rate of image data input to the diagnostic model (the number of images per second), the number of image data can be reduced by thinning frames, or the frames can be interpolated. Adjustments may be made by increasing the number of image data by

Further, the feature point determination unit 111 extracts feature points from each image data. FIG. 2 is an explanatory diagram of processing for extracting feature points. Reference numeral 40 in FIG. 2 indicates an example of an image acquired as image data. The feature point determining unit 111 identifies a human face 41 from the image 40 by pattern matching, and identifies parts such as eyes 42 , eyebrows 43 , nose 44 and lips 45 from this face 41 . Then, the feature point determination unit 111 obtains a predetermined location in each part as a feature point. For example, for the nose 44, the tip of the nose (located on the frontmost side of the nose) 401, the location 402 located on the ridge of the nose, the location 403 located on the outline of the nose, and the location 404 located on the outline of the nostrils. etc. are used as feature points. In the case of the lip 45, the left and right ends 405 of the lip, the portion 406 located on the outline of the upper lip, the portion 407 located on the outline of the lower lip, the portion 408 located around the lip, etc. can be used as feature points. mentioned. In addition, cheeks, chin, and portions located on the outline of the face (face line) may be used as feature points. In FIG. 2, only some of the feature points are labeled, but dots similar to the feature points 401 to 408 indicate the feature points. Note that some feature points are omitted in FIG. 2, and in this embodiment, for example, 468 feature points are defined. Note that the positions and number of feature points are merely examples, and are not limited to these.

The preprocessing unit 112 performs preprocessing such as scaling, acquisition of feature point position information, and complementation of missing feature points for each image data. For example, if the size of the face shown in the image data differs from patient to patient, the magnitude of displacement when the feature point is displaced cannot be evaluated correctly. The length LW in the left-right direction, the distance LE between the center of the left eye and the center of the right eye, etc., are scaled so that the lengths of predetermined parts of the face of the patient 20 have prescribed sizes.

The preprocessing unit 112 also determines coordinates in each image data and acquires position information of each feature point. For example, the reference part of the face, in principle, the tip of the nose 401, is the origin, the vertical direction is the Y axis, the horizontal direction is the X axis, and the depth direction is the Z axis. That is, the change in the position of each part of the face relative to the tip of the nose (reference part) 401 is obtained. The reference part of the face is not limited to the nose, and may be the center between the right eye and the left eye.

Note that if the image capturing device 16 is a device that captures a two-dimensional image, the position information acquired from the image data is the position on the Y axis and the position on the X axis. However, when the human face is facing the imaging device 16, the tip of the nose is located on the imaging device 16 side (front side), and parts such as the eyes and cheeks are located further back than the tip of the nose. It is a curved surface that gradually moves to the back side as it goes to the face line side. For this reason, three-dimensional coordinates on a general curved surface of the face (hereinafter also referred to as a defined curved surface) are defined, and each feature point is mapped on this defined curved surface based on the XY coordinates of each feature source, and the mapped position is estimated as the Z coordinate of the feature point.

Furthermore, when the patient 20 faces left, the position of each feature point in the depth direction changes depending on the orientation of the face, such that the left side of the face is located on the far side. , and the Z coordinate of each feature point is estimated according to the face orientation. Note that the Z-coordinate estimation method is not limited to this, and other estimation methods may be used.

Moreover, when a stereo camera is used as the imaging device 16, the preprocessing unit 112 may obtain the three-dimensional coordinates of each feature point from the stereo images captured by the stereo camera. Furthermore, when a three-dimensional scanner or ToF sensor is used as the imaging device 16, the preprocessing unit 112 may obtain the three-dimensional coordinates of each feature point from the measurement results of the three-dimensional scanner or ToF sensor.

Depending on the direction of the face, such as when the patient 20 faces sideways or when the patient 20 faces downward, the back side of the face may not be captured by the imaging device 16 placed in front, and may be partially captured. Feature point data may be missing. For this reason, the preprocessing unit 112 obtains the orientation of the face of the patient 20 in each image data, and according to this orientation, the positions of the feature points that are not shown in the image are determined as the positions on the prescribed curved surface in the same manner as in the estimation method described above. and complement the location information.

The feature amount calculation unit 113 obtains the feature amount of each feature point based on the change in the position information of each feature point over time obtained in time series by the preprocessing unit 112 . Here, the feature amount is, for example, the mean value, median value, standard deviation, variance, minimum value, maximum value, skewness, kurtosis, sum, square sum, At least one of the number of times across the average value and the average power spectrum can be mentioned.

FIG. 3 is a diagram showing temporal displacement of feature points, and FIG. 4 is a graph showing changes in position of feature points over time. FIG. 3 shows a case where a feature point 451 included in the lip 45 moves downward in the Y-axis direction as time elapses from timing t1 to timing t4. In FIG. 4 , the horizontal axis represents time, the vertical axis represents the Y coordinate value, and a solid line 50 indicates the change in position of the feature point 451 . The feature amount calculation unit 113 divides the value of the Y coordinate that changes over time, that is, the change in the distance from the tip of the nose (origin) into predetermined time intervals (also called windows) tw, and obtains the values for each window tw. Statistics such as mean, median, standard deviation, variance, minimum value, maximum value, skewness, kurtosis, summation, sum of squares, number of crossings over the mean value, and power spectrum density average is calculated as a feature quantity. Similarly, feature quantities are calculated for the X-coordinate value and Z-coordinate value of each feature point. That is, when the above 12 statistical values are obtained for each of the three coordinate values (X, Y, Z coordinate values) of 468 feature points, 16848 feature amounts are obtained for each feature point.

Extrapyramidal symptoms are characterized by appearance, such as ``repeated lip pursedness'', ``mouth mumbling'', and ``mouth sticking out''. In order to capture this feature, the standard deviation, the variance, the number of crossing over the average value, and the average power spectrum are particularly effective among the above feature quantities. In addition, the number of times crossing the average value is the number of times the coordinate value changes from a value lower than the average to a value higher than the average, or from a value higher than the average to a value lower than the average with the passage of time. , the number of times is increased, and the number of times during the entire period is integrated.

FIG. 5 is an explanatory diagram of a power spectrum. A graph 51 in FIG. 5 shows the temporal transition of random signals, with time on the horizontal axis and signal intensity on the vertical axis. A graph 52 is called a power spectrum of this signal, in which the horizontal axis indicates the frequency and the vertical axis indicates the intensity of each frequency component. In the case of random signals, the distribution of the power spectrum is also random as shown in graph 52 .

On the other hand, graph 53 shows the time transition of a signal obtained by adding two periodic signals to a random signal. Graph 54 shows the power spectrum of this signal. As shown in graph 54, in the case of a periodic signal, a sharp peak appears in the power spectrum at a location corresponding to the frequency of the periodic signal. In such a power spectrum, when the frequency k is 0, 1, .

In this way, in the average power spectrum, in the case of a signal having many specific frequency components, F(k) at that frequency becomes very large, and when it is squared, it becomes a larger value.
As a result, it is presumed that the average power spectrum of a signal containing a specific frequency component due to repetitive motion such as "mouth-mugging" will be larger than that of a signal that does not. Therefore, by obtaining the average power spectrum, the periodicity of the displacement can be captured.

The machine learning unit 114 performs machine learning on the learning video based on the feature amount of each feature point obtained by the feature point determination unit 111 and the preprocessing unit 112 and the teacher label, and generates a learned diagnostic model. do. The diagnostic model may partially include a neural network trained by deep learning, or may consist entirely of the neural network. Here, the teacher label is, for example, a doctor's evaluation of the degree of extrapyramidal symptoms in the learning video. The degree of extrapyramidal symptoms in this embodiment is obtained by evaluating the level of symptoms such as tremor, dyskinesia, and dystonia on a five-grade scale of 0 to 4, for example. Level 0 indicates a state in which no symptoms are observed, and a higher level indicates a state in which symptoms are more pronounced.

The diagnosis unit 115 calculates the degree of extrapyramidal symptoms by inputting the feature amount of each feature point obtained by the feature point determination unit 111 and the preprocessing unit 112 into the diagnostic model for the moving image of the patient 20 . do.

The result output unit 116 outputs the degree of the extrapyramidal symptom calculated by the diagnosis unit 115 by displaying it on the display unit or the like. The output of the degree of extrapyramidal symptoms is not limited to display, and may be printed, transmitted to another device, or recorded on a recording medium.

Further, when calculating the degree of extrapyramidal symptoms, the result output unit 116 may obtain and output a feature point having a high degree of influence on the calculation result. FIG. 6 shows an example in which ten feature points 450 having a high degree of influence are obtained and superimposed on an image. This makes it possible to indicate which part of the face affected the evaluation result. The result output unit 116 outputs, for example, a feature variable (each feature value) highly correlated with the degree of extrapyramidal symptoms, which is the target variable of the diagnostic model, for example, a feature having a predetermined or more correlation coefficient with the target variable. A quantity value is obtained from a data table, a function, or the like, and used as a reference value. Then, the result output unit 116 compares each feature amount at each feature point determined by the feature point determination unit 111 with each reference value, and determines that the more feature amounts that reach the reference value, the higher the degree of influence. A predetermined number of feature points are obtained by ranking in descending order of . Specifically, standard values were defined for each symptom and level, such as the value of the feature value that can be taken when the dyskinesia level is 2 and the value of the feature value that can be taken when the dyskinesia level is 3. Define data tables, functions, etc., and set reference values according to diagnostic results. In addition, in this embodiment, since 12 feature quantities such as standard deviation, the number of crossing over the average value, and the average power spectrum are used, a reference value is set for each.

In addition to the "feature amount reaching the reference value", the "feature amount close to the reference value" may be obtained. For example, when the feature amount is "close to the reference value", the feature amount is within a predetermined range from the reference value. Then, for each feature point, the result output unit 116 counts feature amounts that are close to the reference value among the plurality of feature amounts, and determines the degree of influence. Without being limited to this, the result output unit 116 may determine the closeness between the feature amount and the reference value in multiple stages by setting multiple ranges for comparison with the feature amount. For example, multiple ranges centered on the reference value are defined, such as a predetermined range centered on the reference value as the first range, a second range that is wider than the first range, and a third range that is wider than the second range. , the closeness between the feature point and the reference value is determined according to the range in which the feature amount is included. In this case, the result output unit 116, for example, gives the highest weighting factor when the feature amount is included in the first range, and gives the second highest weighting factor when the feature amount exceeds the first range and is included in the second range. Assign a high weighting factor, and assign a third highest weighting factor when the feature exceeds the second range and is included in the third range. do. And the result output unit 11
6 may be obtained as an influence degree by accumulating weight coefficients of a plurality of feature amounts at each feature point. In this case, the size of each range and the value of the weighting factor can be appropriately determined for each feature quantity to be compared. For example, for the standard deviation, variance, number of times across the average value, or average power spectrum, the range for determining closeness may be set wider than in the case of other feature quantities, and the weighting factor to be added may be set to A large value may be set. In addition, for the degree of extrapyramidal symptoms, methods for obtaining highly correlated feature values include, for example, SHAP (Lundberg, Scott M., and Su-In Lee. "A unified approach to interpreting model predictions. " Advances in Neural Information Processing Systems. 2017. https://arxiv.org/abs/1705.07874) can be used. Further, the method for obtaining the degree of influence is not limited to the above method, and other methods may be used for calculation.

《Machine learning processing》
FIG. 7 is a diagram showing a process in which the control unit 12 of the diagnostic device 10 performs machine learning according to a program. When instructed to start machine learning, the control unit 12 executes the process of FIG. 7 .

In step S<b>10 , the control unit 12 acquires learning data from an auxiliary storage device or a storage medium connected to the input/output IF 14 . The learning data includes, for example, moving images of the patient 20 captured in advance for machine learning, and teacher labels obtained by evaluating the degree of extrapyramidal symptoms of the patient 20 by a doctor or the like.

At step S20, the control unit 12 extracts feature points from the moving image acquired at step S10. Extrapyramidal symptoms tend to appear as periodic movements around the eyes and mouth. It is desirable to include specific sites including the periphery as feature points.

In step S30, the control unit 12 performs preprocessing on the feature points obtained in step S20, such as scaling, acquisition of feature point position information, and complementation of missing feature points.

In step S40, the control unit 12 acquires the position information of the feature points obtained in step S30, and obtains the feature amount of each feature point based on the change in the position information of each feature point over time. In this embodiment, the amount of change in position information for each feature point is defined as the mean value, median value, standard deviation, variance, minimum value, maximum value, skewness, kurtosis, sum, sum of squares, and mean value. 12 feature quantities such as the number of times and the average power spectrum are obtained.

In step S50, the control unit 12 performs machine learning based on the feature amount obtained in step S40 and the teacher label obtained in step S10 to generate a diagnostic model.

In step S60, the control unit 12 determines whether or not learning has been completed for all learning data. If the determination is negative, the process returns to step S10.

<<Diagnostic processing for extrapyramidal symptoms>>
FIG. 8 is a diagram showing processing executed by the control unit 12 of the diagnostic device 10 according to the extrapyramidal symptom diagnostic program. When instructed to start diagnosis, the control unit 12 executes the process of FIG. 8 .

In step S<b>110 , the control unit 12 acquires a moving image of the patient 20 from the imaging device 16 , an auxiliary storage device, or a storage medium connected to the input/output IF 14 .

At step S120, the control unit 12 extracts feature points from the moving image acquired at step S110. The feature points extracted here are the same as those obtained in step S20 of the machine learning process shown in FIG.

In step S130, the control unit 12 performs preprocessing on the feature points obtained in step S120, such as scaling, acquisition of feature point position information, and complementation of missing feature points.

In step S140, the control unit 12 acquires the positional information of the feature points obtained in step S130, and obtains the feature amount of each feature point based on the change in the positional information of each feature point over time. The feature values calculated here are the same as those obtained in step S40 of the machine learning process shown in FIG. There are 12 types of feature quantities, such as sum, number of crossing over the average value, and average power spectrum.

In step S<b>150 , the control unit 12 inputs the feature amount obtained in step S<b>140 to the diagnostic model, and calculates the degree of extrapyramidal symptoms of the patient 20 .

In step S160, the control unit 12 outputs the degree of extrapyramidal symptoms calculated in step S150 as a diagnosis result. The control unit 12 also obtains feature points that have a high degree of influence on the diagnosis result, and superimposes them on the image for display as shown in FIG.

As described above, the diagnostic apparatus 10 of the present embodiment extracts feature points from a moving image of the patient 20, obtains feature amounts based on changes in the positions of the feature points over time, and uses the feature amounts as Based on this, AI calculates the degree of extrapyramidal symptoms. As a result, the diagnosis apparatus 10 mechanically diagnoses according to a certain standard, so that it is possible to appropriately evaluate the progress of symptoms over a long period of time, the results of treatment, and the like.

In addition, since the diagnostic apparatus 10 of the present embodiment superimposes the feature points that have a high degree of influence on the diagnostic result on the image, it is possible to determine where the extrapyramidal symptoms occur, for example, around the eyes, mouth, and so on. It can be presented to the patient 20 or the doctor whether it is around the neck, the cheek, the right side, the left side, and so on. Further, since it is presented which movement of the part resulted in the relevant diagnosis result, the doctor and the patient 20 can accurately grasp the diagnosis result, and the reliability of the diagnostic apparatus 10 can be improved.

"others"
The above embodiment is an example and is not limited to the configurations illustrated in the specification and drawings. For example, although the diagnostic device 10 described above has a function of performing machine learning and a function of performing diagnosis, the device for performing machine learning and the device for performing diagnosis may be configured separately. Further, the diagnostic apparatus 10 may be a server that acquires a moving image via a network and returns the result of diagnosis based on this moving image.

10: extrapyramidal symptom diagnostic device (diagnostic device)
11: Connection bus 111: Feature point determination unit 112: Preprocessing unit 113: Feature amount calculation unit 114: Machine learning unit 115: Diagnosis unit 116: Result output unit 12: Control unit 12: Above 13: Memory 16: Imaging device 20 : Patient 40: Image 401: Nose 401-408, 450, 451: Feature points

The feature point determining unit 111 acquires a moving image shot by the shooting device 16 from the shooting device 16, or acquires a moving image shot by another device via the input/output IF 14 or the communication IF 15, Acquire image data in time series from moving images. For example, when a moving image is composed by photographing a plurality of frames per second, each frame is obtained in photographing order and used as image data. The frame rate of moving images is not particularly limited, but is, for example, 15 fps to 120 fps, preferably 25 fps to 90 fps, and more preferably 30 fps to 60 fps. The frame rate of moving images in this embodiment is 30 fps. If the frame rate of the acquired moving image is different from the rate of image data input to the diagnostic model (the number of images per second), the number of image data can be reduced by thinning frames, or the frames can be interpolated. Adjustments may be made by increasing the number of image data by

The preprocessing unit 112 also determines coordinates in each image data and acquires position information of each feature point. For example, the reference part of the face, in this example , the tip of the nose 401, is the origin, the vertical direction is the Y axis, the horizontal direction is the X axis, and the depth direction is the Z axis. That is, the change in the position of each part of the face relative to the tip of the nose (reference part) 401 is obtained. The reference part of the face is not limited to the nose, and may be the center between the right eye and the left eye.

Claims (6)

Acquiring image data in chronological order from moving images of a person to be diagnosed;
Obtaining a plurality of feature points of the subject from the image data, and obtaining position information of each feature point in time series;
Obtaining a feature amount of each feature point based on a change in position information of each feature point over time obtained in time series;
When the feature amount of each feature point is input, a trained diagnostic model that has been subjected to machine learning processing using teacher data so as to calculate the estimated degree of extrapyramidal symptoms is added to the subject calculating the degree of the extrapyramidal symptom by inputting the feature amount obtained from the moving image photographed;
outputting the calculated degree of extrapyramidal symptoms;
An extrapyramidal symptom diagnostic device comprising a control unit for executing The feature quantity is the mean value, median value, standard deviation, variance, minimum value, maximum value, skewness, kurtosis, sum, sum of squares, and mean value obtained for the amount of change in the position information over time. 2. The extrapyramidal symptom diagnostic device according to claim 1, wherein at least one of the number of straddlings and the average power spectrum is selected. 3. The extrapyramidal symptom diagnosis apparatus according to claim 1, wherein the change over time of the positional information of each feature point is a relative positional change with reference to a reference portion of the subject's face. 4. The extrapyramidal tract according to claim 3, wherein the reference part of the face is the nose, and the feature point is a specific part of the subject's face, including the lips and the periphery of the lips. Symptom diagnosis device. The control unit
When calculating the degree of the extrapyramidal symptom, obtaining the feature point that has a high degree of influence on the calculation result;
superimposing and displaying the feature points having the high degree of influence on the image of the subject;
The extrapyramidal symptom diagnosis device according to any one of claims 1 to 4, further comprising: Acquiring image data in chronological order from moving images of a person to be diagnosed;
Obtaining a plurality of feature points of the subject from the image data, and obtaining position information of each feature point in time series;
Obtaining a feature amount of each feature point based on a change in position information of each feature point over time obtained in time series;
When the feature amount of each feature point is input, a trained diagnostic model that has been subjected to machine learning processing using teacher data so as to calculate the estimated degree of extrapyramidal symptoms is added to the subject calculating the degree of the extrapyramidal symptom by inputting the feature amount obtained from the moving image photographed;
outputting the calculated degree of extrapyramidal symptoms;
extrapyramidal symptom diagnosis program for causing the control unit to execute

Images