WO2022158737A1

WO2022158737A1 - Disease diagnosis device and method

Info

Publication number: WO2022158737A1
Application number: PCT/KR2021/019901
Authority: WO
Inventors: 이민식; 한경민; 이태호; 박상영
Original assignee: 한양대학교 에리카산학협력단
Priority date: 2021-01-21
Filing date: 2021-12-27
Publication date: 2022-07-28

Abstract

The present disclosure relates to a disease diagnosis device and method, and, more particularly, can provide a disease diagnosis device and method, which can comprehensively diagnose diseases from a plurality of pieces of image data obtained by capturing diagnostic protocol images, during the diagnosis of visually distinguishable diseases. Specifically, provided are the disease diagnosis device and method, which can diagnose diseases by integrating respective pieces of feature information extracted from the plurality of pieces of image data through an analysis module.

Description

Disease diagnosis apparatus and method

The present embodiments provide an apparatus and method for diagnosing a disease.

With the rapid aging of the world, the problem of supporting the elderly has become a global issue. The United Nations (UN) defines the elderly over 65 years of age as the standard of aging. Meanwhile, Korea's population aging is progressing at a faster pace. According to data released by the National Statistical Office, about 14.3% (about 7.07 million people) of Korea's total population in 2018 belong to the elderly, and it is predicted that about 20.8% (about 10.22 million people) will fall into the category of the elderly in 2026. . As such, as the number of patients with degenerative diseases such as Alzheimer's disease, Parkinson's disease, and stroke due to aging increases, the number of patients with degenerative diseases in which the cognitive ability of the brain is lowered has become an important social problem.

In addition, due to the rapid technological development of big data, machine learning, and artificial intelligence in recent years, research and development of diagnostic technology linked with artificial intelligence is increasing in the health care technology field as well. For example, interest in self-diagnosis services that can be used directly by general consumers without a doctor's prescription through the connection of smartphones and artificial intelligence with health care technology is increasing. Smart home health care, smart home health care, Access such as silver health care is required. For example, geriatric diseases can be determined through direct diagnosis between a doctor and a patient. Specifically, in the diagnosis of geriatric disease, the doctor instructs the patient to test various protocols, and at this time, the doctor and the patient can There are physical limitations that must be faced and the problem of imbalance in medical supply.

Accordingly, there is a need for a technology capable of comprehensively diagnosing a disease from a plurality of image data obtained by a diagnostic protocol image when diagnosing a disease that can be visually identified in order to provide a smooth medical service.

Against this background, the present embodiments provide an apparatus and method for diagnosing a disease capable of comprehensively diagnosing a disease from a plurality of image data from which a diagnostic protocol image is captured when diagnosing a visually distinguishable disease.

In order to achieve the above object, in one aspect, in the present embodiment, in an apparatus for diagnosing a disease, a plurality of image data obtained by continuously photographing a user's posture is collected, A reference information acquisition unit for acquiring reference information, a characteristic information extraction unit for extracting characteristic information for each landmark by applying an analysis module to landmark coordinate information for reference information extracted from each image data, and characteristic information for each landmark Provided is a disease diagnosis apparatus comprising a disease diagnosis unit for generating fusion characteristic information by concatenating and diagnosing a disease based on the integrated characteristic information.

In another aspect, in the present embodiment, in a method for diagnosing a disease, a plurality of image data obtained by continuously photographing a user's posture is collected, and reference information is obtained for obtaining preset reference information on a specific body part for each collected image data. Step, a feature information extraction step of extracting feature information for each landmark by applying an analysis module to landmark coordinate information for reference information extracted from each image data, and a fusion by concatenating feature information for each landmark ) generating characteristic information and providing a disease diagnosis method comprising a disease diagnosis step of diagnosing a disease based on the integrated characteristic information.

According to the present embodiments, it is possible to provide an apparatus and method for diagnosing a disease that can comprehensively diagnose a disease from a plurality of video data from which a diagnostic protocol image is captured when diagnosing a visually distinguishable disease.

1 is a diagram illustrating a configuration of an apparatus for diagnosing a disease according to an embodiment of the present disclosure.

FIG. 2 is a flowchart illustrating an operation of diagnosing a disease of the apparatus for diagnosing a disease according to an embodiment of the present disclosure.

3 is a flowchart illustrating an operation of applying an analysis module of an apparatus for diagnosing a disease according to an embodiment of the present disclosure.

4 is a diagram illustrating an example for describing a frame autonomy module of an apparatus for diagnosing a disease according to an embodiment of the present disclosure.

5 is a diagram illustrating an example of extracting characteristic information of an apparatus for diagnosing a disease according to an embodiment of the present disclosure.

6 is a diagram illustrating an example of generating integrated feature information in an apparatus for diagnosing a disease according to an embodiment of the present disclosure.

7 is a diagram illustrating an example of generating integrated feature information in an apparatus for diagnosing a disease according to another embodiment of the present disclosure.

8 is a flowchart of a disease diagnosis method according to an embodiment of the present disclosure;

9 is a block diagram of an apparatus for diagnosing a disease according to an embodiment of the present disclosure.

The present disclosure relates to an apparatus and method for diagnosing a disease.

Hereinafter, some embodiments of the present disclosure will be described in detail with reference to exemplary drawings. In adding reference numerals to the components of each drawing, it should be noted that the same components are given the same reference numerals as much as possible even though they are indicated on different drawings. In addition, in describing the present disclosure, if it is determined that a detailed description of a related known configuration or function may obscure the gist of the present disclosure, the detailed description thereof will be omitted.

In addition, in describing the components of the present disclosure, terms such as first, second, A, B, (a), (b), etc. may be used. These terms are only for distinguishing the elements from other elements, and the essence, order, or order of the elements are not limited by the terms. When it is described that a component is “connected”, “coupled” or “connected” to another component, the component may be directly connected or connected to the other component, but another component is between each component. It should be understood that elements may be “connected,” “coupled,” or “connected.”

The disease in the present specification describes, as an example, a geriatric disease in which the cognitive ability of the brain is lowered, such as Alzheimer's disease, Parkinson's disease, stroke, etc., but is not limited thereto as long as it is a visually distinguishable disease.

Hereinafter, the present disclosure will be described in detail with reference to the accompanying drawings.

Referring to FIG. 1 , an apparatus 100 for diagnosing a disease according to an embodiment of the present disclosure collects a plurality of image data continuously photographing a user's posture, and relates to a specific body part preset for each collected image data. A reference information acquisition unit 110 for acquiring reference information, a characteristic information extraction unit 120 for extracting characteristic information for each landmark by applying an analysis module to landmark coordinate information for reference information extracted from each image data, and A disease diagnosis apparatus 100 including a disease diagnosis unit 130 for generating fusion characteristic information by concatenating characteristic information for each landmark and diagnosing a disease based on the integrated characteristic information is provided.

The reference information obtaining unit 110 according to an embodiment may collect a plurality of image data obtained by continuously photographing a user's posture, and may obtain preset reference information regarding a specific body part for each of the collected image data. As an example, the reference information obtaining unit 110 may collect a plurality of image data obtained by continuously photographing a posture performed by a user according to a protocol for disease examination. For example, the reference information obtaining unit 110 may continuously capture a plurality of image data of a posture of a user performing a still posture test, an eye movement test, a thumb and index finger repeated exercise test, a simple walking test, or a daily walking test, etc. can be collected.

Also, as an example, the reference information acquisition unit 110 may set a specific body part including at least one of a user's face, pupil, body, and hand according to a diagnosis purpose or a diagnosis situation. In addition, the reference information acquisition unit 110 may acquire reference information regarding a specific body part preset for each collected image data. That is, the preset specific body part may be variously changed. For example, the reference information acquisition unit 110 may acquire reference information about a face and reference information about a body, respectively, from image data of a user who has performed a still posture test. Also, the reference information obtaining unit 110 may obtain reference information about a face and reference information about a pupil from image data of a user who has performed an eye movement test.

The feature information extractor 120 according to an embodiment may extract feature information for each landmark by applying an analysis module to landmark coordinate information for reference information extracted from each image data. For example, the feature information extractor 120 may extract landmark coordinate information for reference information through a coordinate extractor from each image data according to a protocol. For example, the coordinate extractor may be implemented as a face landmark coordinate extractor, a pupil landmark extractor, a body landmark extractor, a hand landmark extractor, or the like for a specific body part. In addition, the landmark coordinate information may be coordinate information for a face joint, a pupil joint, a Time Up and Go (TUG) joint, a Maintenance of Wakefulness Test (MWT) joint, a hand joint, and the like.

Also, as an example, the feature information extraction unit 120 may independently apply each extracted landmark coordinate information to an analysis module to extract feature information for each landmark. For example, the analysis module may include a first neural network module that generates joint point information based on landmark coordinate information and a second neural network module that combines time axis information based on joint point information within the same frame. have. As a specific example, the first neural network module may be based on a Graph Convolution Network (GCN), and the second neural network module may be based on a Convolutional Neural Network (CNN). Details of the analysis module will be described later with reference to FIG. 3 .

As another example, the analysis module may further include a frame autonomy module that adjusts the frame size of the landmark coordinate information to be the same. As a specific example, the frame autonomy module may select a central frame based on a preset number of frames, and combine a plurality of peripheral frames based on the selected central frame to generate a central frame set. In addition, the frame autonomy module may adjust the frame size of landmark coordinate information by concatenating a plurality of generated central frame sets. Accordingly, the analysis module may use the input data of the same size by adjusting the extracted landmark coordinate information to the same frame size through the frame autonomy module. Details of the frame autonomy module will be described later with reference to FIG. 4 .

The disease diagnosis unit 130 according to an embodiment may generate fusion characteristic information by concatenating characteristic information for each landmark, and diagnose a disease based on the integrated characteristic information. For example, the disease diagnosis unit 130 generates integrated characteristic information by concatenating characteristic information for each landmark, and inputs it into a fully-connected layer to determine whether or not a disease occurs, its type and severity. At least one can be diagnosed. In addition, the disease diagnosis unit 130 calculates the average value of the characteristic information for each landmark and the maximum value of the characteristic information for each landmark, and concatenates the calculated average value and the maximum value to generate Fusion characteristic information. may be Details on the generation of the integrated feature information will be described later with reference to FIGS. 6 and 7 .

The disease diagnosis unit 130 according to an exemplary embodiment may measure the accuracy of the disease diagnosis apparatus 100 by using the pre-labeled geriatric disease type and severity value. For example, the disease diagnosis unit 130 may measure the accuracy of the disease diagnosis apparatus 100 by comparing a diagnosis result and an actual result value for image data not used for learning.

Referring to FIG. 2 , the reference information acquisition unit 110 of the apparatus for diagnosing a disease according to an embodiment of the present disclosure may collect a plurality of image data ( S210 ). As an example, the reference information acquisition unit 110 may collect a plurality of image data obtained by continuously photographing a posture performed by a user according to a protocol for testing a user for a disease. For example, the reference information acquisition unit 110 may collect a plurality of image data obtained by photographing the user's posture according to 12 diagnostic protocols except for the voice test. In this case, the image data may be individually photographed from various angles, and an image number and type corresponding to each image data may be labeled and used. Specifically, the plurality of image data is a static posture test, eye movement test, thumb and index finger repetition exercise test, simple walking test, straight gait test, forehead wrinkle test, eye strain test, smile test, mouth open test, and tongue out test , it is possible to collect a plurality of image data continuously photographed from four directions (up, down, left and right) angles of the posture of the user performing the arm gathering test and finger/nose repetition exercise test. This has been described as an example of the type of image data, and is not limited thereto as long as it is suitable for diagnostic purposes.

The reference information acquisition unit 110 of the apparatus for diagnosing diseases according to an embodiment may acquire reference information for each image data ( S220 ). For example, the reference information acquisition unit 110 may acquire reference information regarding a specific body part preset according to a diagnosis purpose or a diagnosis situation for each collected image data. For example, the reference information acquisition unit 110 may set a specific body part including at least one of a user's face, pupil, body, and hand according to a diagnosis purpose or a diagnosis situation. In addition, the reference information acquisition unit 110 may acquire preset reference information on a specific body part from a plurality of image data according to 12 diagnostic protocols. As a specific example, the reference information obtaining unit 110 may obtain reference information about the face from image data of a smile test performed by the user.

The feature information extraction unit 120 of the apparatus for diagnosing a disease according to an embodiment may extract landmark coordinate information with respect to the reference information ( S230 ). For example, the feature information extractor 120 may extract landmark coordinate information for reference information through a coordinate extractor from each image data according to a protocol. For example, a coordinate extractor can extract landmark coordinates by learning a heat map to detect landmarks within an image by utilizing convolutional neural networks (CNNs). However, if the coordinates of the landmark can be extracted from the image data, the coordinate extractor is not limited thereto. As a specific example, the feature information extraction unit 120 may extract landmark coordinate information for the face by detecting key points of the face, such as the tip of the nose, eyebrows, corners of the eyes, and the mouth, from the image data taken from the smile test.

The feature information extraction unit 120 of the apparatus for diagnosing a disease according to an embodiment may extract feature information for each landmark ( S240 ). For example, the feature information extractor 120 may extract feature information for each landmark by applying landmark coordinate information for the extracted reference information to the analysis module. For example, the feature information extraction unit 120 may extract feature information for each landmark by applying each landmark coordinate information to an analysis module that extracts a feature vector. Here, the feature information may mean a feature vector. Also, before application of the analysis module, the feature information extractor 120 may apply landmark coordinate information having a different data size according to the frame length to the frame autonomy module to adjust the data size to the same. That is, the landmark coordinate information of which the frame size is adjusted may be used as input data of the analysis module. For a specific example, the feature information extraction unit 120 applies coordinate information about a face joint, a pupil joint, a Time Up and Go (TUG) joint, a Maintenance of Wakefulness Test (MWT) joint, a hand joint, etc. to the analysis module to land Characteristic information for each mark can be extracted. The characteristic information for each landmark may include facial characteristic information, body characteristic information, pupil characteristic information, or hand characteristic information extracted from each coordinate information. Details of the analysis module will be described later with reference to FIG. 3 .

According to an embodiment, the disease diagnosis unit 130 of the disease diagnosis apparatus may generate integrated characteristic information (S250). For example, the disease diagnosis unit 130 may generate fusion characteristic information by concatenating characteristic information for each landmark. For example, the disease diagnosis unit 130 may calculate the average value and the maximum value of the characteristic information for each channel from the extracted characteristic information for each landmark, and connect the calculated average value and the maximum value to generate the integrated characteristic information. That is, the disease diagnosis unit 130 may generate one piece of integrated characteristic information by integrating characteristic information for each landmark.

According to an embodiment, the disease diagnosis unit 130 of the disease diagnosis apparatus may diagnose a disease based on the integrated characteristic information ( S260 ). For example, the disease diagnosis unit 130 may diagnose at least one of the presence, absence, type, and severity of a disease by inputting the integrated characteristic information into a fully-connected layer. For example, the disease diagnosis unit 130 inputs the integrated characteristic information into a fully connected layer used for object classification purposes to determine the presence or absence of geriatric disease, the type (ex, dementia, Parkinson's disease, cerebrovascular disease, etc.) and severity (mild, severe, severe) can be diagnosed.

Referring to FIG. 3 , the feature information extraction unit 120 of the apparatus for diagnosing a disease according to an embodiment of the present disclosure may extract landmark coordinate information with respect to reference information ( S230 ). Then, the feature information extraction unit 120 may apply each landmark coordinate information for the extracted reference information to the analysis module (S300). As an example, the feature information extraction unit 120 may extract feature information for each landmark through an analysis module configured to further include a frame autonomy module or a fully connected layer based on the first neural network module and the second neural network module. . For example, the analysis module is a joint-frame-based deep learning network module and may be a hierarchical network module in which joint-level and frame-level dependencies are sequentially connected. That is, the analysis module may be a Sematic Guided Neural Network (SGN), but is not limited thereto.

The feature information extraction unit 120 of the apparatus for diagnosing disease according to an embodiment may adjust the frame size of the landmark coordinate information to be the same through the frame autonomy module (S310). For example, the feature information extractor 120 may input landmark coordinate information of the same size to the analysis module through the frame autonomy module. For example, the frame autonomy module may select a central frame based on a preset number of frames and combine a plurality of peripheral frames based on the central frame to generate a central frame set. In addition, the frame autonomy module may adjust the frame size of landmark coordinate information by connecting the generated central frame set.

The feature information extraction unit 120 of the disease diagnosis apparatus according to an embodiment may generate joint point information based on landmark coordinate information through the first neural network module (S320). As an example, the feature information extractor 120 may generate joint point information through a first neural network module based on a graph convolution network (GCN). For example, the first neural network module may extract characteristic information of a joint level as a joint-level module for utilizing joint correlation. Specifically, the first neural network module extracts joint-level feature information by learning the graph connection between nodes (joints) in a frame based on the meaning (ex, connection weight aspect) and dynamics of the joint type based on the graph convolutional neural network. can do.

The feature information extraction unit 120 of the disease diagnosis apparatus according to an embodiment may extract feature information by combining time axis information based on joint point information through the second neural network module (S330). As an example, the feature information extraction unit 120 may extract feature information combining time axis information by connecting a second neural network module based on a convolutional neural network (CNN) behind the first neural network. For example, the second neural network module may extract frame-level feature information as a frame-level module for utilizing the correlation between frames. Specifically, the second neural network module may extract frame-level feature information by performing a Spatial MaxPooling (SMP) operation on all features of joints within the same frame. In addition, the extracted frame-level feature information can be combined with the built-in frame index information to learn feature information for classification through two temporal convolutional neural network layers. That is, the second neural network module may extract the frame-level feature information by applying the extracted joint-level feature information to spatial pooling, temporal pooling, convolution layer, and the like. Here, the frame-level feature information may be feature information combining time axis information included in the frame index based on joint point information.

The feature information extraction unit 120 of the apparatus for diagnosing disease according to an embodiment may extract feature information for each landmark adjusted to a desired size through a fully-connected layer (S340). For example, the feature information extraction unit 120 may convert a matrix of feature vectors extracted for each landmark to a desired size by applying it to a fully-connected layer including an activation function such as softmax or sigmoid. have.

The feature information extraction unit 120 of the apparatus for diagnosing a disease according to an embodiment may extract feature information for each landmark ( S240 ). For example, the feature information extraction unit 120 may extract feature information for each landmark by applying an analysis module for each landmark coordinate information, respectively.

Referring to FIG. 4 , content in which the frame autonomy module of the apparatus for diagnosing a disease according to an embodiment of the present disclosure adjusts the frame size of landmark coordinate information to be the same may be described. For example, the frame autonomy module may generate a central frame set by combining a plurality of peripheral frames based on a central frame selected based on a preset number of frames. In addition, the frame autonomy module may adjust the frame size of landmark coordinate information by connecting the generated central frame set.

For example, the frame autonomy module may select the central frame 420 based on the preset number of frames C in the landmark coordinate information 410 for reference information extracted from each image data. Specifically, the total size of the landmark coordinate information 410 for the extracted reference information may be 2NХF. Here, N may be the number of joint points, 2N may mean the number of joint points in a two-dimensional space, and F may mean the total number of frames of image data.

For example, the frame autonomy module may generate the central frame set 440 by combining a plurality of peripheral frames 430 based on the selected central frame 420 . Specifically, the size of each generated central frame set may be 2NХF'. Here, F' may mean the number of peripheral frames 430 coupled with respect to the central frame 420 .

For example, the frame autonomy module may sequentially concatenate the generated central frame set 440 to adjust the frame size of landmark coordinate information identically. Accordingly, the scaled landmark coordinate information 450 may be used as an input of an analysis module for extracting feature information. Specifically, the size of the scaled landmark coordinate information 450 may be 2NХF'C. Here, C may mean the number of center frames 420 selected as the preset number of frames. Accordingly, the feature information extraction unit 120 of the disease diagnosis apparatus may adjust the number of frames to the same number through the frame autonomy module even if the plurality of image data is configured with different frame numbers.

Referring to FIG. 5 , content in which the feature information extraction unit 120 of the apparatus for diagnosing a disease according to an embodiment of the present disclosure extracts feature information may be described. For example, the reference information acquisition unit 110 may acquire preset reference information regarding a specific body position for each of a plurality of image data obtained by continuously photographing a user's posture according to a diagnosis protocol. For example, the reference information acquisition unit 110 may acquire reference information on the first body part and reference information on the second body part from image data 1 of a user who has performed a test according to a diagnosis protocol. In addition, Table 1 describes an example of test contents and respective reference information according to image data, but is not limited thereto.

As an example, the feature information extraction unit 120 may extract landmark coordinate information with respect to the reference information through the coordinate extraction module. For example, the feature information extraction unit 120 extracts landmark coordinate information for each body part by using the reference information about the first body part and the reference information about the second body part obtained from the image data 1 can do. That is, the feature information extraction unit 120 may extract at least one preset landmark coordinate information of a specific body part from the image data 1 through the coordinate extraction module. Accordingly, if the image data is different, the type and number of specific body parts from which landmark coordinate information is extracted may be changed.

For example, the feature information extraction unit 120 may extract feature information for each landmark by applying the extracted respective landmark coordinate information to the analysis module. For example, the feature information extraction unit 120 applies the first body part landmark coordinate information and the second body part landmark coordinate information to the analysis module, respectively, to obtain the first body part feature information and the second body part feature information. can be extracted. Also, for example, the feature information extraction unit 120 adjusts each landmark coordinate information to the same size through the frame autonomy module, and applies the joint-level first neural network module to the joint point for the landmark. information can be generated. Then, the feature information extraction unit 120 applies the frame-level second neural network module to the joint point information for the landmark, combines the time axis information, and finally extracts the feature information of the desired size through the fully connected layer. can That is, the feature information extraction unit 120 may extract feature information for each landmark from each landmark coordinate information through the analysis module.

Also, as an example, the disease diagnosis unit 130 may connect feature information for each landmark extracted from a plurality of image data to finally generate one piece of integrated feature information. Details on the generation of the integrated feature information will be described later with reference to FIGS. 6 and 7 .

Referring to FIG. 6 , the generation of the integrated feature information by the disease diagnosis unit 130 of the disease diagnosis apparatus according to an embodiment of the present disclosure may be described. For example, the disease diagnosis unit 130 may generate the integrated characteristic information 620 by concatenating the characteristic information 610 for each landmark extracted from each image data. For example, the disease diagnosis unit 130 may generate the integrated characteristic information 620 by connecting the characteristic information 610 for each landmark extracted from each image data in a manner of arranging in a line. As a specific example, if the feature information 610 for each landmark is a CХ1 type feature vector and there are a total of N landmarks, the disease diagnosis unit 130 integrates the feature information 620 that is a NCХ1 type feature vector. can create

Also, as an example, the disease diagnosis unit 130 may input the generated integrated characteristic information 620 into the fully connected layer 630 to diagnose at least one of the presence, absence, type, and severity of the disease. For example, the fully connected layer 630 includes a plurality of nodes, and each of the plurality of nodes may perform a predetermined operation on an input. Also, the fully connected layer 630 may output a result by applying a weight to the operation. Specifically, the output of the disease diagnosis unit 130 may indicate that the information included in the plurality of image data continuously photographing the user's posture may be at least one of the probability of occurrence of the corresponding disease, the probability of occurrence by disease type, or the occurrence probability by severity. have.

Referring to FIG. 7 , the generation of the integrated feature information by the disease diagnosis unit 130 of the disease diagnosis apparatus according to another embodiment of the present disclosure may be described. For example, the disease diagnosis unit 130 may calculate an average value 710 and a maximum value 720 by using the characteristic information 610 for each landmark extracted from each image data. In addition, the disease diagnosis unit 130 may generate the integrated feature information 730 by concatenating the average value 710 and the maximum value 720 . For example, the disease diagnosis unit 130 may generate the integrated feature information 730 by connecting the calculated average values 710 and the maximum values 720 in a manner that is arranged in a line. As a specific example, if the characteristic information 610 for each landmark is a CХ1 type feature vector and there are a total of N landmarks, the disease diagnosis unit 130 sets the average value 710 and the maximum value 710 that is a CХ1 type characteristic vector. Each value 720 may be calculated. Accordingly, the disease diagnosis unit may generate the integrated feature information 730 that is a 2CХ1 type feature vector.

Also, as an example, the disease diagnosis unit 130 may input the generated integrated characteristic information 730 into the fully connected layer 630 to diagnose at least one of the presence, absence, type, and severity of the disease. Details on the fully connected layer 630 are the same as described above with reference to FIG. 6 . However, in the fully connected layer 630 , the weight size of the fully connected layer 630 may be differently applied as the shape of the input feature vector changes.

Hereinafter, a method for diagnosing a disease that can be performed by the apparatus for diagnosing a disease described with reference to FIGS. 1 to 7 will be described.

8 is a flowchart of a method for diagnosing a disease according to an embodiment of the present disclosure.

Referring to FIG. 8 , the method of diagnosing a disease according to the present disclosure may include a step of acquiring reference information ( S810 ). For example, the apparatus for diagnosing a disease may collect a plurality of image data obtained by continuously photographing a user's posture, and may obtain preset reference information regarding a specific body part for each collected image data. For example, the disease diagnosis apparatus may collect a plurality of image data obtained by continuously capturing a posture of a user performed according to a disease test protocol.

Also, as an example, the disease diagnosis apparatus may set a specific body part including at least one of a user's face, pupil, body, and hand according to a diagnosis purpose or a diagnosis situation. In addition, the disease diagnosis apparatus may acquire preset reference information on a specific body part for each collected image data. That is, the preset specific body part may be variously changed.

The disease diagnosis method of the present disclosure may include a feature information extraction step (S820). For example, the disease diagnosis apparatus may extract characteristic information for each landmark by applying an analysis module to landmark coordinate information for reference information extracted from each image data. For example, the disease diagnosis apparatus may extract landmark coordinate information for reference information through a coordinate extractor from each image data according to a protocol. Specifically, the coordinate extractor may be a facial landmark coordinate extractor, a pupil landmark extractor, a body landmark extractor, or a hand landmark extractor. In addition, the landmark coordinate information may be coordinate information for a face joint, a pupil joint, a Time Up and Go (TUG) joint, a Maintenance of Wakefulness Test (MWT) joint, a hand joint, and the like.

Also, as an example, the apparatus for diagnosing the disease may extract characteristic information for each landmark by independently applying the extracted coordinate information of each landmark to the analysis module. For example, the analysis module may include a first neural network module that generates joint point information based on landmark coordinate information and a second neural network module that combines time axis information based on joint point information within the same frame. have. Specifically, the first neural network module may be based on a Graph Convolution Network (GCN), and the second neural network module may be based on a Convolutional Neural Network (CNN).

As another example, the analysis module may further include a frame autonomy module that adjusts the frame size of the landmark coordinate information to be the same. Specifically, the frame autonomy module may select a central frame based on a preset number of frames, and combine a plurality of peripheral frames based on the selected central frame to generate a central frame set. In addition, the frame autonomy module may adjust the frame size of landmark coordinate information by concatenating a plurality of generated central frame sets. Accordingly, the analysis module may adjust the extracted landmark coordinate information to the same frame size through the frame autonomy module and use it as input data.

The disease diagnosis method of the present disclosure may include a disease diagnosis step (S830). For example, the apparatus for diagnosing a disease may generate fusion characteristic information by concatenating characteristic information for each landmark, and diagnose a disease based on the integrated characteristic information. For example, the disease diagnosis apparatus generates integrated characteristic information by concatenating characteristic information for each landmark, and inputs it into a fully-connected layer to determine at least one of the presence, absence, type, and severity of disease. can be diagnosed Also, the disease diagnosis apparatus may calculate the average value of the characteristic information for each landmark and the maximum value of the characteristic information for each landmark, and concatenate the calculated average value and the maximum value to generate fusion characteristic information.

9 is a block diagram of an apparatus for diagnosing a disease according to an exemplary embodiment.

Referring to FIG. 9 , the disease diagnosis apparatus 100 according to an embodiment includes a communication interface 910 and a processor 920 . The disease diagnosis apparatus 1100 may further include a memory 930 . Each component, the communication interface 910 , the processor 920 , and the memory 930 may be connected to each other through a communication bus. For example, a communication bus may include circuitry that connects components to each other and transfers communications (eg, control messages and/or data) between components.

The communication interface 910 may acquire a plurality of images of the user's posture. Also, the communication interface 1110 may communicate with an external device through wireless communication or wired communication.

The processor 920 may perform the at least one method described above with reference to FIGS. 1 to 8 or an algorithm corresponding to the at least one method. The processor 920 may be a hardware-implemented data processing device having a circuit having a physical structure for executing desired operations. For example, desired operations may include code or instructions included in a program. For example, a data processing device implemented as hardware includes a microprocessor, a central processing unit, a processor core, a multi-core processor, and a multiprocessor. , a Neural Processing Unit (NPU), an Application-Specific Integrated Circuit (ASIC), and a Field Programmable Gate Array (FPGA).

Also, the processor 920 may execute a program and control the disease diagnosis apparatus 100 . The program code executed by the processor 920 may be stored in the memory 930 .

Information on the artificial intelligence model including the neural network according to the embodiment of the present disclosure may be stored in the internal memory of the processor 920 or stored in an external memory, that is, the memory 930 . For example, the memory 930 may store a plurality of images obtained by photographing the user's posture acquired through the communication interface 910 . The memory 930 may store an artificial intelligence model including a neural network. Also, the memory 930 may store various types of information generated in a process of the processor 920 and output information extracted by the processor 920 . The output information may be a neural network operation result or a neural network test result. The memory 930 may store a neural network learning result. The neural network learning result may be obtained from the disease diagnosis apparatus 100 or from an external apparatus. The neural network learning result may include a weight and a bias value. In addition, the memory 930 may store various data and programs. The memory 930 may include a volatile memory or a non-volatile memory. The memory 930 may include a mass storage medium such as a hard disk to store various data.

In the above, even though it has been described that all components constituting the embodiment of the present disclosure are combined or operated as one, the present disclosure is not necessarily limited to this embodiment. That is, within the scope of the present disclosure, all the components may operate by selectively combining one or more. In addition, although all the components may be implemented as one independent hardware, some or all of the components are selectively combined to perform some or all of the functions of one or a plurality of pieces of hardware. may be implemented as a computer program having a The computer-readable medium may include program instructions, data files, data structures, etc. alone or in combination. The program instructions recorded on the medium may be specially designed and configured for the embodiment, or may be known and available to those skilled in the art of computer software. Examples of the computer-readable recording medium include magnetic media such as hard disks, floppy disks and magnetic tapes, optical media such as CD-ROMs and DVDs, and magnetic such as floppy disks. - includes magneto-optical media, and hardware devices specially configured to store and execute program instructions, such as ROM, RAM, flash memory, and the like. Examples of program instructions include not only machine language codes such as those generated by a compiler, but also high-level language codes that can be executed by a computer using an interpreter or the like. The hardware devices described above may be configured to operate as one or more software modules to perform the operations of the embodiments, and vice versa.

In addition, terms such as "comprises", "comprises" or "have" described above mean that the corresponding component may be embedded, unless otherwise stated, excluding other components. Rather, it should be construed as being able to include other components further. All terms, including technical or scientific terms, have the same meaning as commonly understood by one of ordinary skill in the art to which this disclosure belongs, unless otherwise defined. Commonly used terms, such as those defined in the dictionary, should be interpreted as being consistent with the meaning of the context of the related art, and should not be interpreted in an ideal or excessively formal meaning unless explicitly defined in the present disclosure.

The above description is merely illustrative of the technical idea of the present disclosure, and various modifications and variations will be possible without departing from the essential characteristics of the present disclosure by those of ordinary skill in the art to which the present disclosure belongs. Accordingly, the embodiments disclosed in the present disclosure are for explanation rather than limiting the technical spirit of the present disclosure, and the scope of the technical spirit of the present disclosure is not limited by these embodiments. The protection scope of the present disclosure should be interpreted by the following claims, and all technical ideas within the scope equivalent thereto should be construed as being included in the scope of the present disclosure.

CROSS-REFERENCE TO RELATED APPLICATIONCROSS-REFERENCE TO RELATED APPLICATION

This patent application is based on U.S. Patent Law 119 for Patent Application No. 10-2021-0008800 filed in Korea on January 21, 2021 and Patent Application No. 10-2021-0142543 filed in Korea on October 25, 2021 Priority is claimed under section (a) (35 U.S.C § 119(a)), the entire contents of which are incorporated herein by reference. In addition, if this patent application claims priority for countries other than the United States for the same reason as above, all contents thereof are incorporated into this patent application by reference.

Claims

a reference information acquisition unit that collects a plurality of image data obtained by continuously photographing a user's posture, and acquires preset reference information on a specific body part for each collected image data;

a feature information extracting unit for extracting feature information for each landmark by applying an analysis module to the landmark coordinate information for the reference information extracted from each image data; and

and a disease diagnosis unit for generating fusion characteristic information by concatenating the characteristic information for each landmark, and diagnosing a disease based on the integrated characteristic information.
The method of claim 1,

The reference information obtaining unit,

The apparatus for diagnosing a disease, characterized in that the specific body part is set including at least one of the user's face, pupil, body, and hand according to a diagnosis purpose or a diagnosis situation.
The method of claim 1,

The analysis module,

Disease diagnosis, characterized in that it comprises a first neural network module for generating joint point information based on the landmark coordinate information and a second neural network module for combining time axis information based on the joint point information in the same frame Device.
4. The method of claim 3,

The first neural network module,

Based on Graph Convolution Network (GCN),

The second neural network module,

A disease diagnosis device, characterized in that it is based on a convolutional neural network (CNN).
The method of claim 1,

The analysis module,

The apparatus for diagnosing a disease further comprising a frame autonomy module that equalizes the frame size of the landmark coordinate information.
6. The method of claim 5,

The frame autonomy module,

Select a central frame based on a preset number of frames, combine a plurality of peripheral frames based on the central frame to create a central frame set, and concatenate the central frame set to frame the landmark coordinate information A disease diagnosis device, characterized in that the size is adjusted.
The method of claim 1,

The disease diagnosis unit,

The apparatus for diagnosing a disease, characterized in that by inputting the integrated characteristic information into a fully-connected layer, at least one of the occurrence, type, and severity of the disease is diagnosed.
8. The method of claim 7,

The disease diagnosis unit,

Calculating the average value of the characteristic information for each landmark and the maximum value of the characteristic information for each landmark, and concatenating the average value and the maximum value to generate the Fusion characteristic information Device.
a reference information acquisition step of collecting a plurality of image data obtained by continuously photographing a user's posture, and acquiring reference information on a specific body part preset for each of the collected image data;

a feature information extraction step of extracting feature information for each landmark by applying an analysis module to the landmark coordinate information for the reference information extracted from each image data; and

and a disease diagnosis step of generating fusion characteristic information by concatenating the characteristic information for each landmark, and diagnosing a disease based on the integrated characteristic information.
10. The method of claim 9,

The step of obtaining the reference information is

The method for diagnosing a disease, characterized in that the specific body part is set including at least one of the user's face, pupil, body, and hand according to a diagnosis purpose or a diagnosis situation.
10. The method of claim 9,

The analysis module,

Disease diagnosis, characterized in that it comprises a first neural network module for generating joint point information based on the landmark coordinate information and a second neural network module for combining time axis information based on the joint point information in the same frame Way.
10. The method of claim 9,

The analysis module,

The disease diagnosis method according to claim 1, further comprising a frame autonomy module configured to equalize the frame size of the landmark coordinate information.
13. The method of claim 12,

The frame autonomy module,

Select a central frame based on a preset number of frames, combine a plurality of peripheral frames based on the central frame to create a central frame set, and concatenate the central frame set to frame the landmark coordinate information A method for diagnosing a disease, characterized in that the size is adjusted.
10. The method of claim 9,

The disease diagnosis step is

The method for diagnosing a disease, characterized in that inputting the integrated characteristic information into a fully-connected layer to diagnose at least one of the presence or absence, type, and severity of the disease.
15. The method of claim 14,

The disease diagnosis step is

Calculating the average value of the characteristic information for each landmark and the maximum value of the characteristic information for each landmark, and concatenating the average value and the maximum value to generate the Fusion characteristic information Way.