KR20220095856A - Neuropsychiatric diseases and derma skin diseases diagnosing device and method thereof - Google Patents

Abstract

Disclosed are an apparatus and a method for diagnosing a neuropsychiatric disease and a skin disease. In accordance with one embodiment, a diagnosis assistance method for diagnosing a neuropsychiatric disease of a user by using an eye movement of the user, includes the following steps of: displaying an indicator for acquiring eye information of the user and moving the indicator; acquiring the eye information of the user; and acquiring diagnosis assistance information about the neuropsychiatric disease based on an output result outputted by inputting the eye information of the user into a first neural network model. The diagnosis assistance information includes at least one of information about whether the user has at least one neuropsychiatric disease, and information about a probability that the user has the at least one neuropsychiatric disease, and the at least one neuropsychiatric disease includes at least one of dementia, Alzheimer, and Parkinsonism. Therefore, the present invention is capable of promoting convenience for a user and an expert and increasing the accuracy of determination.

Description

Apparatus and method for diagnosing neuropsychiatric and skin diseases

The present invention relates to an apparatus and method for diagnosing neuropsychiatric and skin diseases.

In the case of the diagnosis of Alzheimer's, a type of conventional neuropsychiatric disease, it is made by diagnostic tests by offline experts such as visual acuity, contrast sensitivity, color vision, visual field loss, motion perception, depth perception and stereopsis, and ocular motor function. come.

Even in the case of the conventional skin disease diagnosis, the diagnosis through the naked eye by an offline expert was predominant.

In the case of the above-mentioned treatment performed in a hospital, as telemedicine is newly emerging, attempts are being made to be made online gradually.

One object of the present invention is to implement an online diagnosis method for a neuropsychiatric disease and a skin disease, which is made by the judgment of an existing offline expert.

One object of the present invention is to implement the above-described method so that not only the judgment of an expert is used, but also judgment or judgment assistance can be provided based on additionally transmitted information online.

According to an embodiment, a diagnostic assistance method for diagnosing a user's neurological disease using the user's eye movement includes: displaying an indicator for acquiring the user's eye information and moving the indicator; obtaining the user's eye information; and inputting the user's eye information into a first neural network model and obtaining diagnostic auxiliary information on the neurological disease based on an output result, wherein the diagnostic auxiliary information includes the user's at least one neurological disease. It includes at least one of information on whether or not the user has the at least one neurological disease or information on the probability that the user has the at least one neurological disease, and the at least one neurological disease includes at least one of dementia, Alzheimer's disease, or Parkinson's disease. can

According to an embodiment of the present invention, it is possible to provide convenience to users and experts by remotely judging or assisting in judging a user's neuropsychiatric disease and skin disease based on information transmitted online.

According to an embodiment of the present invention, the determination and assistance of the determination may be determined according to a predetermined process, and the time for the determination and the accuracy of the determination may be more accurate than that of an expert.

1 is a diagram illustrating a page for acquiring an image on a web page according to an embodiment.
2 is a diagram illustrating augmentation of image data using a plurality of filters according to an exemplary embodiment.
3 is a diagram illustrating a case in which a skin disease image is synthesized with a normal image for data enhancement according to an embodiment.
4 is a block diagram illustrating a method for tracking eye movement according to an embodiment.
5 is a schematic diagram of an eyeball for obtaining the center of the eyeball according to an embodiment.
6 is a schematic diagram of an eyeball for obtaining the center of the eyeball according to an embodiment.
7 is a diagram illustrating a change in a pupil size according to an exemplary embodiment.
8 is a diagram illustrating a test method that can be used for diagnosing a neuropsychiatric disorder according to an exemplary embodiment.
9 is a block diagram for driving neuropsychiatric disease diagnosis in a background mode according to an exemplary embodiment.
10 is a diagram illustrating a minimum separation distance between a diagnostic apparatus and an eyeball according to an exemplary embodiment.
11 is a diagram illustrating a maximum separation distance between a diagnostic apparatus and an eyeball according to an exemplary embodiment. 11A is a diagram showing the specificity of gaze movement required for diagnosing a neuropsychiatric disorder, and FIG. 11B is a diagram illustrating the device to calculate the maximum distance in consideration of the pixel of the device in order to capture the specificity of the gaze movement. to be.
12 is a diagram illustrating an indicator inverted in a diagnostic apparatus, according to an exemplary embodiment.
13 is a schematic diagram illustrating a serial structure of a neural network for outputting a neuropsychiatric disorder diagnosis result, according to an embodiment.
14 is a schematic diagram illustrating a parallel structure of a neural network for outputting a neuropsychiatric disease diagnosis result, according to an embodiment.
15 is a schematic diagram illustrating the use of a plurality of neural networks to output a neuropsychiatric disease diagnosis result, according to an embodiment.
16 is a diagram illustrating a diagnosis result of a neuropsychiatric disorder displayed on an output unit of a diagnosis apparatus according to an exemplary embodiment.
17 is a diagram illustrating a diagnosis result of a skin disease displayed on an output unit of a diagnosis apparatus according to an exemplary embodiment.
18 is a view showing a range in which an external drug for a skin disease can be applied in a photographed skin disease image, according to an embodiment.
19 is a view showing that the amount of ointment applied to a skin disease site is different for each area, according to an embodiment.
20 is a diagram illustrating a method of configuring a network using the ensemble technique of an artificial intelligence neural network, according to an embodiment.

The above-mentioned objects, features and advantages of the present invention will become more apparent from the following detailed description taken in conjunction with the accompanying drawings. However, since the present invention may have various changes and may have various embodiments, specific embodiments will be exemplified in the drawings and described in detail below.

In the drawings, the thicknesses of layers and regions are exaggerated for clarity, and also, an element or layer may be referred to as “on” or “on” another component or layer. What is referred to includes all cases in which another layer or other component is interposed in the middle as well as directly on top of another component or layer. Throughout the specification, like reference numerals refer to like elements in principle. In addition, components having the same function within the scope of the same idea shown in the drawings of each embodiment will be described using the same reference numerals.

If it is determined that a detailed description of a known function or configuration related to the present invention may unnecessarily obscure the gist of the present invention, the detailed description thereof will be omitted. In addition, numbers (eg, first, second, etc.) used in the description process of the present specification are only identification symbols for distinguishing one component from other components.

In addition, the suffixes "module" and "part" for components used in the following description are given or mixed in consideration of only the ease of writing the specification, and do not have distinct meanings or roles by themselves.

One. Diagnostic devices and diagnostic methods in general

Hereinafter, an apparatus for diagnosing a neuropsychiatric disease and a skin disease and a method therefor will be described.

Here, diagnosing the disease may be diagnosing the type of disease, but is not limited thereto, and an expert such as a doctor provides diagnostic auxiliary information for diagnosing the disease, or a drug and treatment regimen depending on whether the disease is diagnosed Any provision of prescribing information for a diagnosis can be expressed as a diagnosis.

1.1. diagnostic device

Hereinafter, a diagnostic device for diagnosing a neuropsychiatric disease or a skin disease may be a smartphone, computer, smart TV, or tablet, but is not limited thereto, and in the case of a device including a processor capable of processing an electronic operation It may be included in the aforementioned diagnostic device.

Also, the diagnostic device may be a server.

In addition, the aforementioned diagnostic apparatus may further include a photographing unit.

Here, the photographing unit may refer to an electronic device capable of converting an image photographed in an electronic form, such as a camera sensor, a digital camera, or a scanner.

Also, the aforementioned diagnostic device may include a motion detection sensor such as an accelerometer or gyroscope sensor.

1.2. Diagnostic method

An artificial intelligence model may be used as a method for diagnosing a neuropsychiatric disease or a skin disease.

Here, the AI model may be machine learning or deep learning, and specifically, it may be a convolutional neural network (CNN), a recurrent neural network (RNN), LSTM, STN, GRU, or the like.

Specifically, the skin region may be detected through the skin segmentation module for ease of learning and diagnosis of the AI model.

According to an embodiment, the corresponding learning may apply residual attention learning.

However, the present invention is not limited thereto, and a predetermined algorithm may be used to diagnose a neuropsychiatric disease or a skin disease.

Here, the algorithm may be configured in the form of a command that can be processed by the control unit of the diagnosis apparatus.

1.3. Data collection method

1 is a diagram illustrating a page for acquiring an image on a web page according to an exemplary embodiment.

Referring to FIG. 1 , a method of considering related words together in order to acquire image data for training an artificial intelligence neural network may be provided.

According to an embodiment, it is possible to increase efficiency and increase accuracy by automating image data acquisition by determining related words and media in a page searched for by a specific keyword on the web.

According to an embodiment, the specific keyword may be a disease name to be diagnosed, and the medium may be a news medium.

According to one embodiment, the disease name may be a skin disease such as herpes zoster, athlete's foot, acne.

In an embodiment, as shown in the figure of FIG. 1 , an image and a keyword for the image may be included in the web page. In this case, words (eg, athlete's foot, psoriasis) included in the web page may be extracted, and related images may be derived from the web using the word included in the web page as a keyword. The derived related images may be used as image data for training the artificial intelligence neural network.

Also, in another embodiment, image data for training an artificial intelligence neural network may be obtained without considering related words. For example, the image data may be acquired by an external device or a user input. In addition, an image similar to the reference image may be obtained as image data for training the artificial intelligence neural network.

1.4. Data Augmentation Methods

2 is a diagram illustrating augmentation of image data using a plurality of filters according to an exemplary embodiment.

Referring to FIG. 2 , the number of one acquired image data may be increased to a plurality of image data based on a plurality of filters.

According to an embodiment, one image data obtained through methods such as enlargement, reduction, cropping, inversion, rotation, etc. of an image as well as a method of augmenting image data using the plurality of filters is converted into a plurality of image data. may increase the number.

3 is a diagram illustrating a case in which a skin disease image is synthesized with a normal image for data enhancement according to an embodiment.

Referring to FIG. 3 , by synthesizing an image representing a target disease with an image in which a disease does not exist for data augmentation, the number of image data sets for training a neural network may be increased. For example, when the number of skin disease images is insufficient, training of the neural network may be insufficiently performed, and thus the diagnosis accuracy of the target disease may be lowered. However, when a skin disease image is generated by synthesizing an image representing a target disease with an image without a disease, and learning of the neural network is performed using the generated skin disease images, as the neural network is learned with a large number of images, Diagnosis accuracy can be increased.

According to an embodiment, the image synthesis may be to synthesize the input normal image and the disease image according to an algorithm.

According to an embodiment, the image synthesis may be to synthesize a normal image and a disease image through a learned artificial intelligence neural network (GAN).

According to an embodiment, the synthesizing method may synthesize an image in which a target disease is included in an image in which the disease does not exist. In another embodiment, the synthesis method may use a filter.

2. Diagnosis of neuropsychiatric disorders

Hereinafter, a diagnosis method for diagnosing a neuropsychiatric disorder, an environment for diagnosing the disease, and result values derived therefrom will be described.

Here, the neuropsychiatric disease may include degenerative brain diseases such as Alzheimer's dementia, cerebrovascular dementia or Parkinson's, and furthermore, neurotransmission circuit abnormalities and neurotransmission such as depression, manic depression, panic disorder, schizophrenia, schizophrenia, delusional disorder, etc. Diseases caused by material abnormalities may also be included.

Here, the neuropsychiatric disorder may include an intellectual disorder, a communication disorder, an autistic disorder, an attention deficit/hyperactivity disorder (ADHD), and a specific learning disorder indicated by a neurodevelopmental disorder.

2.1. How to diagnose neuropsychiatric disorders

Hereinafter, a plurality of methods for diagnosing a neuropsychiatric disorder will be described.

Each of the methods described below may independently diagnose a neuropsychiatric disorder, but a neuropsychiatric disorder may be diagnosed based on at least one or more diagnostic methods.

The results of disease types and prescriptions according to the diagnosis of neuropsychiatric disorders through the following device may be provided to the user by synthesizing the additional diagnosis results of the specialist.

2.1.1. Diagnosis using eye movements

According to an embodiment, the neuropsychiatric disorder may be diagnosed based on eye movement.

Specifically, the movement of the eyeball may follow a designated indicator such as a saccadic movement, an anti-saccadic movement, a step function shape, a step shape, and a spiral shape.

According to an embodiment, factors such as latency and accuracy may be considered in the diagnosis using the eye movement in Alzheimer's dementia.

According to an embodiment, in the diagnosis of Parkinson's disease using the movement of the eyeball, a tremble appearing in the movement may be considered.

2.1.1.1. How to track eye movements

4 is a block diagram illustrating a method for tracking eye movement according to an embodiment.

Referring to FIG. 4 , a method for tracking eye movement may be provided.

Acquiring the landmark of the face may include recognizing the user's face, extracting the landmark of the face, specifying the eye position, and extracting the pupil position and the central position of the eyeball.

According to an embodiment, in the step of recognizing the user's face, the diagnostic apparatus may recognize the user's face by extracting the feature of the object of interest through SVM-based Histogram of Oriented Gredients (HoG).

According to an embodiment, in the step of extracting the landmark of the face, the diagnosis apparatus may be performed through a regression tree-based machine learning algorithm.

According to an embodiment, in the step of specifying the eye position, the diagnosis apparatus may be performed by a Convex Hull algorithm.

According to an embodiment, in the step of extracting the pupil position and the central position of the eyeball, the diagnosis apparatus may be performed using an adaptive threshold technique.

5 and 6 are schematic diagrams of an eyeball for obtaining the center of the eyeball according to an embodiment.

Specifically, in order to obtain the center of the eyeball, the boundary line of the iris may be recognized by the contrast between the white of the pupil and the iris.

Referring to FIG. 5 , two points of the boundary line of the iris forming a straight line together with the first landmark and the second landmark may be obtained. The average value of the obtained points can be obtained as the center of the eyeball.

Referring to FIG. 6 , a tangent line may be obtained with respect to any two points of the boundary line of the iris obtained as described above, and a vertical line may be drawn at the positions of any two points of the boundary line of the iris of the corresponding tangent line. The point of intersection of the vertical lines drawn in this way may be the central position of the pupil.

According to an embodiment, the step of tracking the eye movement may be performed by comparing the difference between the initial eye center position and the eye center position over time.

According to an embodiment, the step of tracking the eye movement may be performed by comparing the position of the display unit of the diagnosis apparatus with the position of the pupil.

2.1.1.2. Diagnosis method using tracked eye movements

The previously obtained tracked eye movements may be obtained in the form of coordinates on a display in which an indicator is shown.

However, the present invention is not limited thereto, and the movement of the eyeball may be acquired without directly specifying the position of the eyeball by using a change in RGB value in addition to the coordinate value.

Hereinafter, it will be described that the tracked eye movement is obtained in the form of coordinates for convenience of description, but is not limited thereto as described above.

In an embodiment, the diagnosis apparatus may diagnose a neuropsychiatric disorder based on a difference value between an eye movement obtained in a coordinate format and an indicator movement (reference) coordinate on the display. For example, the diagnosis apparatus may be diagnosed through a pre-trained artificial intelligence neural network.

Specifically, when the eye movement obtained as described above has a latency higher than a certain level when compared with the movement predicted by the indicator, Alzheimer's dementia can be diagnosed.

Specifically, when tremor of a certain level or more occurs when the eye movement obtained as described above is compared with the movement predicted by the indicator, Parkinson's disease can be diagnosed.

According to an embodiment, one or more neuropsychiatric disorders may be simultaneously diagnosed using the eye movements obtained as described above.

According to one embodiment, one or more neuropsychiatric disorders can be diagnosed by using only eye movements as input values through the learned artificial intelligence neural network as well as neuropsychiatric diseases based on the difference between the eye movements and the movement coordinates of the indicator. can

According to an embodiment, a neuropsychiatric disorder may be diagnosed through an artificial intelligence neural network that is learned by using only the eye movement image as an input value.

The diagnosis method through eye movement requires movement data for at least a certain period of time for diagnosis, and may specifically be 30 seconds.

According to an embodiment, the diagnosis method through eye movement may be performed by dividing the trial using the indicator into a plurality of times in order to secure at least a predetermined time for diagnosis. Through this, it is possible to prevent the user from leaving.

2.1.2. Diagnosis using pupil size change

7 is a view illustrating a change in a pupil size according to an exemplary embodiment.

A user may concentrate according to a specific stimulus of the diagnostic device or a change in pupil size may occur when light is irradiated.

In the case of neuropsychiatric patients, there may be no change in pupil size or there may be less change in pupil size compared to normal individuals.

According to an exemplary embodiment, when the change in the pupil size does not change by more than a certain threshold in response to a specific stimulus provided through the diagnostic apparatus, the neuropsychiatric disorder may be diagnosed.

According to an embodiment, the presence or absence of a neuropsychiatric disorder may be diagnosed by an artificial intelligence neural network that has been trained using a change value of the pupil size as an input value.

2.1.3. Diagnosis Using Facial Tone Recognition

According to an embodiment, a neuropsychiatric disorder may be determined based on an image of a part of the body including a face or a finger by using a photographing unit included in the diagnosis apparatus.

Specifically, when a change of more than a threshold value occurs by comparing the PPG in the basic state obtained through a photograph or video of a photographed face or fingers, and the PPG after a specific task is progressed, it can be determined as a neuropsychiatric disorder. It may be photographed through the front or rear camera, and may be photographed simultaneously.

According to an embodiment, the specific task may include whether to determine a fragrance using a fragrance patch.

According to an embodiment, the specific task may include whether to judge the played sound.

According to an embodiment, the specific task may include moving the eyeball according to an indicator.

2.1.4. Diagnosis using figure overlay

8 is a diagram illustrating a test method that can be used for diagnosing a neuropsychiatric disorder according to an exemplary embodiment.

According to an embodiment, the inspection method may be a figure overlapping drawing, and the figure may be a figure having various shapes, such as a polygonal shape such as a pentagon or a star shape.

In the case where the diagnosis method is applied to the same user, the figure overlapping may present the shape of the figure in consideration of past learning results. This can prevent the result of lowering the diagnosis accuracy through learning in the iterative process.

According to an embodiment, in the test method, the user may draw a line on the touch panel by using a means for drawing a line on the touch panel with respect to the diagnostic apparatus including the touch panel.

According to an exemplary embodiment, in the test method, the user may overwrite a given figure on paper and then conduct the user's test by capturing the image by a photographing unit of the diagnostic apparatus.

Specifically, a neuropsychiatric disorder may be diagnosed in consideration of the area of the overlapping area between the figures and the accuracy of the figures based on the user's figure overlapping result. A normal person can overwrite the figure with the given figure without difficulty. However, in the case of a neuropsychiatric patient, it may not be possible to clearly draw the suggested figure (eg, a pentagon, etc.), and the overlapping area of the two figures may be small or absent, unlike a normal person. In this case, it is judged that there is a cognitive abnormality, and dementia can be diagnosed. Additionally, when a figure is drawn, the line segments that make up the figure may not be drawn close to a straight line, and the pen pressure (if data is collected by the touch panel can be measured) or the line segment while the line segment is drawn. The thickness may be different, and in this case, it can be diagnosed as Parkinson's disease.

2.1.5. Diagnosis using body part movement

When the diagnostic apparatus requests execution of a specific task, the user's neuropsychiatric disorder may be diagnosed through input data to an input unit of the user's diagnostic apparatus. This may be by a touch input, may be photographing information collected by a photographing unit of the diagnosis apparatus, or may be input of eye movement information.

According to an embodiment, a specific task may use a pelli-robson chart, and a neuropsychiatric disorder such as Parkinson's or dementia may be diagnosed by analyzing the performance speed and accuracy. This is a method that can perform a cognitive test by varying the transparency of a certain alphabet.

According to an embodiment, a specific task may use a dot kinematogram. In this case, the user matches the directions indicated by the plurality of indicators by the indicators indicating the plurality of directions.

According to an embodiment, the specific task may be to allow the user to walk a certain distance.

2.1.6. Diagnosis using birefringence

By calculating the birefringence of the eyeball using the camera and flash of the diagnostic device, vitreous turbidity and retinal abnormalities can be calculated, thereby diagnosing neuropsychiatric disorders. In this case, when the light emitted from the flash of the diagnostic device is reflected back inside the eyeball, the light can be collected from a plurality of cameras, and the reflected light can be combined to calculate the birefringence of the eyeball.

Specifically, the neuropsychiatric disease is dementia.

Additionally, when a certain amount of vitreous hemorrhage is detected, Tust syndrome can be diagnosed.

2.2. Neuropsychiatric disease diagnosis environment

2.2.1. Operating environment within the diagnostic unit

The method for diagnosing neuropsychiatric disorders described above may be driven in a foreground mode of the diagnostic apparatus, and may also be driven in a background mode.

Here, the foreground mode may refer to an operation mode for diagnosing a neuropsychiatric disorder by explicitly displaying an indicator instructing the user to perform eye movement and using the user's eye movement corresponding to the movement of the indicator.

In addition, the background mode may refer to an operation mode that does not explicitly display the indicator, but displays one or more objects and diagnoses a neuropsychiatric disorder using a user's eye movement corresponding to the movement of the object. The object may not be explicitly recognized as an indicator by the user. However, the user may unconsciously perform eye movements according to the movement of the object, and accordingly, in the background mode, the diagnostic apparatus can diagnose a neuropsychiatric disorder using the user's unconscious eye movement according to the movement of the object. have.

Here, in order to operate in the background mode, the controller of the diagnostic apparatus may perform an additional operation.

9 is a block diagram for driving neuropsychiatric disease diagnosis in a background mode according to an exemplary embodiment.

According to an embodiment, it is possible to determine whether a neuropsychiatric disorder exists by comparing the user's eye movement with a path predicted in advance by analyzing the content in the background mode to predict the movement of the subject in the content. The content may be content pre-stored in the diagnosis apparatus, and the diagnosis apparatus includes a unit for pre-analyzing the content of the content that is reproduced in real time as well as the previously-stored content, so that the content of the content is analyzed in advance. It is possible to obtain the configuration of objects included in the , and the movement direction/speed of each object, and use these objects as indicators. An eye movement corresponding to a time when the object is displayed may be acquired, and a neurological disease may be determined using the eye movement at the corresponding time.

According to an embodiment, a target object satisfying a criterion such as a predetermined movement speed among the objects may be separately selected and used as an indicator for diagnosis.

According to one embodiment, when an event in which a new object appears in the content occurs, the new object may be set as an indicator, and the user's response to the stimulus such as the appearance of the new object may be obtained to determine whether a neuropsychiatric disorder is present. . Specifically, in the case of a user with a cognitive impairment, in the stimulus for the appearance of a new object, the new object may show a movement with less attention compared to a normal user.

According to an embodiment, the analysis of neuropsychiatric disorders in the background mode of the content may be performed in consideration of the caption of the content. Specifically, it is possible to diagnose a neuropsychiatric disorder by acquiring gaze movements that occur as a result of reading subtitles. In addition, it is possible to diagnose a neuropsychiatric disorder by acquiring the time that the gaze on the subtitles stays.

2.2.2. Diagnostic device usable distance

In order to diagnose a neuropsychiatric disorder, a minimum separation distance and a maximum separation distance of the distance between the diagnostic device and the eyeball to be measured may be provided.

10 is a diagram illustrating a minimum separation distance between a diagnostic apparatus and an eyeball according to an exemplary embodiment.

The minimum separation distance between the diagnostic apparatus and the eye may be set in consideration of a distance at which both eyes of the user may be photographed by the photographing unit in consideration of the FOV of the photographing unit of the diagnostic apparatus.

Referring to FIG. 10 , this distance may be calculated by Equation 1 below.

[Equation 1]

Here, FOV _h is the field of view by horizontal of the smartphone front camera, h is the distance from the center of both eyes to the smartphone, is the length of the smartphone, is the lateral length of the eyes, and is the distance between the eyes.

11 is a diagram illustrating a maximum separation distance between a diagnostic apparatus and an eyeball according to an exemplary embodiment.

In order to diagnose a neuropsychiatric disorder, the distance between the diagnostic device and the eyeball must be limited enough to measure eye tremor. The ocular tremor can be distinguished when the ocular tremor is at least 1 pixel in the acquired image (FIG. 11 a), and the maximum separation distance between the imaging unit of the diagnostic device and the eyeball can be set in consideration of the FOV of the imaging unit of the diagnostic device. There is (Fig. 11b).

Referring to FIG. 11 , this distance may be calculated by Equation 2 below.

[Equation 2]

Here, FOV _v is the field of view by vertical of the smartphone front camera, h is the distance from the eye to the smartphone, amp _e is the eyeball amplitude, and Pix _v is the number of vertical pixels acquired by the camera.

2.2.3. Diagnostics-related telemedicine optimization

For diagnosis by a specialist such as a doctor, the device may be configured to facilitate online diagnosis.

12 is a diagram illustrating an indicator inverted in a diagnostic apparatus, according to an exemplary embodiment.

Referring to FIG. 12 , a neuropsychiatric disorder may be determined based on the user's eye movement following the movement of the indicator.

According to an exemplary embodiment, since the diagnostic apparatus including the photographing unit has a feature of being left and right inverted when capturing an image, it may be possible to reverse the image to the original direction.

The configuration for the convenience of online diagnosis by an expert is not limited to the above configuration.

2.3. Neuropsychiatric disease diagnosis neural network examples

13 is a schematic diagram illustrating a serial structure of a neural network for outputting a neuropsychiatric disorder diagnosis result, according to an embodiment.

A first input may be input to the first neural network, and a result output from the first neural network may be input to the second neural network as an input value. A diagnosis result may be output based on the result output from the second neural network.

According to an embodiment, in diagnosing a neuropsychiatric disorder, the first input may be an eye movement. The result output from the first neural network may be eye movement coordinates. The result output from the second neural network may be a neuropsychiatric disease diagnosis result.

According to an embodiment, in diagnosing a skin disease, the first input may be a user's skin input data. The result output from the first neural network may be the user's skin input data preprocessed by the neural network. This may be a skin segmented clot. The result output from the second neural network may be a skin disease diagnosis result.

14 is a schematic diagram illustrating a parallel structure of a neural network for outputting a neuropsychiatric disease diagnosis result, according to an embodiment.

The first input may be respectively input to the first neural network and the second neural network, and a diagnosis result may be output by synthesizing the results output from the first neural network and the second neural network.

According to an embodiment, in diagnosing a neuropsychiatric disorder, the first input may be an eyeball image or a moving picture captured by a diagnosis device. The first neural network may be a neural network for diagnosing a neuropsychiatric disease by tracking eye movements, and the second neural network may be a neural network for diagnosing a neuropsychiatric disease through whether the iris is contracted/expanded. The synthesis of the output results may be synthesized and outputted when there is a diagnosis result of one or more neuropsychiatric diseases among the first neural network and the second neural network.

According to an embodiment, in diagnosing a skin disease, the first input may be a skin image of a user captured by a diagnosis apparatus. The first neural network may be a neural network for diagnosing a first skin disease, and the second neural network may be a neural network for diagnosing a second skin disease. The synthesis of the output results may be synthesized and outputted when there is a diagnosis result of one or more skin diseases among the first neural network and the second neural network. A plurality of neural networks may be included to diagnose a plurality of skin diseases.

15 is a schematic diagram illustrating the use of a plurality of neural networks to output a neuropsychiatric disease diagnosis result, according to an embodiment.

The first input may be input to the second neural network, and the second input may be input to the first neural network. By synthesizing the output values from each neural network, a disease diagnosis result can be output.

According to an embodiment, in diagnosing a neuropsychiatric disorder, the first input may be eye tracking data of a user photographed by the diagnosis apparatus, and the second input may be user's figure overlapping input data, or a second input. The input may be input data of a user's body part movement. The synthesis of the output results may be synthesized and outputted when there is a diagnosis result of one or more neuropsychiatric diseases among the first neural network and the second neural network.

According to an exemplary embodiment, in diagnosing a skin disease, the first input may be skin data of a user photographed by the diagnosis apparatus, and the second input may be user and family medical history or numeric data of the user. The synthesis of the output results may be synthesized and outputted when there is a diagnosis result of one or more skin diseases among the first neural network and the second neural network. A plurality of neural networks may be included to diagnose a plurality of skin diseases.

2.4. Diagnosis of neuropsychiatric disorders

16 is a diagram illustrating a diagnosis result of a neuropsychiatric disorder displayed on an output unit of a diagnosis apparatus according to an exemplary embodiment.

According to an embodiment, the diagnosis result may indicate whether it is normal/abnormal, and the name of the neuropsychiatric disease and whether the disease is applicable may be displayed as a probability.

According to an embodiment, the user may consult with an expert according to the neuropsychiatric disease diagnosis result diagnosed by the diagnosis device, and the diagnosis result diagnosed by the diagnosis device may be provided to the expert.

According to an exemplary embodiment, after the first diagnosis, the diagnosis apparatus may perform a diagnosis again, and may diagnose a neuropsychiatric disorder by summing up the diagnosis result that has been previously performed and the diagnosis result that has been performed again. In this case, the accuracy can be increased.

According to an exemplary embodiment, the diagnosis apparatus may prescribe a drug according to a diagnosis result, and may receive a record input to manage whether the drug is taken and whether the disease is in remission.

According to an embodiment, the diagnosis apparatus may provide whether the user should visit a hospital based on the severity of the disease according to the diagnosis result.

3. Diagnosis of skin diseases

Hereinafter, a method for diagnosing a skin disease using the aforementioned diagnostic apparatus or method will be described.

The skin diseases described below include eczema, urticaria, warts, seborrheic dermatitis, keloids, hyperhidrosis, ringworm, Pityriasis versicolor, Urticaria, Angioedema, Dermatographism, Eczema, diaper rash, viral skin disease, bacterial skin disease, benign tumor, hair loss, Skin vascular disease, dermal and subcutaneous fat tumors, papule scaly disease, animal parasitic skin disease, melanoma, athlete's foot, psoriasis, herpes zoster, acne or atopic dermatitis may include all kinds of diseases occurring on the skin.

3.1. How to diagnose skin diseases

According to an embodiment, a region predicted to have a skin disease is photographed using a photographing unit of the diagnostic apparatus, and the photographed result is used as an input to the above-mentioned neural network to determine whether the photographed region has a skin disease or a disease. can be diagnosed

3.2. How to depersonalize input data for skin disease diagnosis

According to an exemplary embodiment, an image captured by using the photographing unit of the diagnosis apparatus may be stored in the apparatus or server by covering or removing parts such as eyes, nose, and mouth that can identify an individual.

According to an exemplary embodiment, an image photographed using the photographing unit of the diagnosis apparatus may be divided into fragments so as not to distinguish parts such as eyes, nose, and mouth that can identify individuals, and may be diagnosed. Specifically, the fragmentation may provide a margin of a predetermined numerical range at the boundary line of each fragment so that an overlapping image range may be included in each fragment. In this case, individual privacy can be protected. In addition, since only the disease-predicted region is used for skin disease diagnosis through the artificial intelligence neural network according to the disease, the accuracy of the diagnosis device can be increased. In addition, since there is specificity for each expression site for each skin disease, learning of the artificial intelligence neural network can be easily performed, and the accuracy of diagnosis can be improved.

According to an embodiment, an image captured by using the photographing unit of the diagnosis apparatus may be stored only in the apparatus, and only a value for providing a diagnosis result may be transmitted to the server. In this case, individual privacy can be protected, and the security problem is alleviated.

3.3. Provides skin disease diagnosis results

The results of disease types and prescriptions according to the diagnosis of skin diseases through the following device may be provided to the user by synthesizing the additional diagnosis results of the specialist.

3.3.1. Provide types of skin diseases

17 is a diagram illustrating a diagnosis result of a skin disease displayed on an output unit of a diagnosis apparatus according to an exemplary embodiment.

Specifically, the diagnosis result may indicate whether it is normal/abnormal, and the name of a skin disease and whether the disease is applicable may be displayed as a probability.

Specifically, consultation with an expert may be performed according to the skin disease diagnosis result, and the diagnosis result may be provided to the expert.

According to an exemplary embodiment, the diagnosis may be performed again, and a skin disease may be diagnosed by summing up the previously performed diagnosis result and the re-processed diagnosis result. In this case, the accuracy can be increased.

According to an embodiment, the prognosis of a disease may be determined through time-series photographed images. Specifically, the diagnosis apparatus may receive a plurality of skin images of a user, and the corresponding images may be acquired in different time zones. It is possible to determine the prognosis of how the skin disease will change in the future by determining the change in the input data according to time.

According to an embodiment, the prognostic determination of the disease may be a learned artificial intelligence neural network. The artificial intelligence neural network can train the artificial intelligence neural network using a plurality of skin images with different time zones of the same individual, meaning that the time zones are spaced apart by a predetermined time interval. Specifically, the predetermined time interval may be a day, a week, a week, a month, etc., and the accuracy of the neural network increases only when data with a uniform time interval is learned. As the artificial intelligence neural network, RNN or CRNN may be used.

According to an embodiment, when there are a plurality of artificial intelligence neural networks learned for the various time zones, the artificial intelligence neural network may be configured as a parallel model in the diagnosis method of the diagnostic apparatus. However, the diagnostic method of the diagnostic apparatus is not limited thereto, and may consist of one neural network. A plurality of images may be used in inputting a user's skin image in the diagnosis apparatus, and the images may be configured so that the photographing time is separated from each other. When the photographing time of a plurality of images is far apart, each photographing time is obtained, the photographing time interval of the plurality of images can be obtained, and the skin disease prognosis is determined by inputting the plurality of images to the neural network model corresponding to the photographing time interval can do. If there is no neural network model corresponding to the photographing time interval, the diagnosis may be made through a neural network model having the most similar time interval. For example, when the time interval between the user's plurality of image input values is 13 days, the skin disease diagnosis may be made through the artificial intelligence neural network model trained for 2 weeks.

According to an embodiment, the artificial intelligence neural network can learn data of various time intervals to the neural network in advance, derive accuracy for each time interval data in advance, and use the accuracy to include neural network information applicable to each time interval By pre-stored look-up table, a skin disease diagnosis can be performed by selecting a neural network model according to the look-up table.

3.3.2. Provision of disease-related prescriptions

According to an embodiment, the diagnosis apparatus may diagnose a type of a photographed skin disease and prescribe a drug that can be used for the skin disease.

Here, the drug may be both an internal drug and an external drug.

Specifically, the prescription of the drug may be a prescription for a drug according to the location of the lesion and the size of the lesion, may include a specific application amount, and may be a step-by-step drug prescription according to the stage of disease progression.

According to an embodiment, in the case of diagnosing athlete's foot, depending on the location of the lesion, a liquid medicine may be prescribed when athlete's foot is diagnosed on toenails, and a medicine applied when athlete's foot is diagnosed on normal skin.

3.3.3. Provides a range of external drug application for skin diseases

18 is a view showing a range in which an external drug for a skin disease can be applied in a photographed skin disease image, according to an embodiment.

Referring to FIG. 18 , when the diagnostic apparatus outputs a diagnostic result, a range to which a drug should be applied may be provided.

Here, the range in which the drug should be applied may be the same as or wider than the range in which the skin disease occurs.

According to an embodiment, the application range of the drug may have a boundary spaced apart from the boundary of the skin disease determination site by a predetermined interval. The predetermined interval may be determined in consideration of the type of skin disease, the degree of the disease, and the site of the skin disease. In addition, the predetermined interval may be an arbitrarily set value.

According to one embodiment, the range to be applied of the drug may be provided in real time through a method such as AR. The corresponding result may be output through the display unit of the diagnosis apparatus, or may be output through a device providing separate AR content such as wearable glasses.

According to an embodiment, when using a learned artificial intelligence neural network, the range to which the drug should be applied may refer to a heat map or Grad-CAM result. For example, a heat map or Grad-CAM indicating relevance to a skin disease region in the artificial intelligence neural network may be output. In the heat map or Grad-CAM, the relevance to the skin disease area can be displayed at various levels through color classification. In an embodiment, the diagnostic apparatus may set an area indicated by a first level among a plurality of levels as a skin disease area, and may set an area indicated by a second level lower than the first level as a range in which the drug is to be applied.

According to an embodiment, in diagnosing a skin disease through the learned artificial intelligence neural network, the learning data provided for the learning of the artificial intelligence neural network may be provided by labeling a separate application range for the disease picture by an expert.

19 is a view showing that the amount of ointment applied to a skin disease site is different for each area, according to an embodiment.

Referring to FIG. 19 , the diagnostic apparatus may provide a result that the amount of the drug applied is appropriate for the first area, and provide a result that the amount of the drug applied to the second area is small.

Furthermore, the diagnostic apparatus may provide a result of instructing to additionally apply the drug to the second region in which the amount of the applied drug is small.

4. Diagnostic method Lightweight driving method

An artificial intelligence model for diagnosing neuropsychiatric and skin diseases can use the ensemble technique.

Specifically, networks (EfficientNet-B0, DensNet-121, ResNext-101, SEResNext-101) that have been previously reported to have good performance in other datasets can be used without training, and problems caused by data shortage can be solved. In addition, networks can be ensembled to learn by composing a teacher network that sees the result of extracting features with a pre-learned model and a student network extracting features based on it. Referring to FIG. 20, the upper blue block is the teacher network, and the lower green block is the student network. Based on the features from the above two, we can configure the student network to learn to imitate the teacher network.

Referring to FIG. 20, (a) and (b) are the previously used ensemble methods, (a) is a method for learning the student network by averaging the results of teacher networks, (b) is a random combination of existing networks way to learn it. (c) is a network configuration using the Curriculum-Training technique so that each network is sorted by performance and learned step-by-step from the network with the lowest performance to the network with the highest performance. In the case of using curriculum-training, the learning accuracy can be increased compared to the case of using the existing ensemble technique.

However, the present invention is not limited thereto, and a predetermined algorithm may be used to diagnose a neuropsychiatric disease or a skin disease. Even in this case, weight reduction can be achieved by optimizing the diagnostic device, and it can be driven by a mobile device. Here, the algorithm may be configured in the form of a command that can be processed by the control unit of the diagnosis apparatus.

The method according to the embodiment may be implemented in the form of program instructions that can be executed through various computer means and recorded in a computer-readable medium. The computer-readable medium may include program instructions, data files, data structures, etc. alone or in combination. The program instructions recorded on the medium may be specially designed and configured for the embodiment, or may be known and available to those skilled in the art of computer software. Examples of the computer-readable recording medium include magnetic media such as hard disks, floppy disks and magnetic tapes, optical media such as CD-ROMs and DVDs, and magnetic such as floppy disks. - includes magneto-optical media, and hardware devices specially configured to store and execute program instructions, such as ROM, RAM, flash memory, and the like. Examples of program instructions include not only machine language codes such as those generated by a compiler, but also high-level language codes that can be executed by a computer using an interpreter or the like. The hardware devices described above may be configured to operate as one or more software modules to perform the operations of the embodiments, and vice versa.

As described above, although the embodiments have been described with reference to the limited embodiments and drawings, various modifications and variations are possible from the above description by those skilled in the art. For example, the described techniques are performed in an order different from the described method, and/or the described components of the system, structure, apparatus, circuit, etc. are combined or combined in a different form than the described method, or other components Or substituted or substituted by equivalents may achieve an appropriate result.

Therefore, other implementations, other embodiments, and equivalents to the claims are also within the scope of the following claims.

Claims (8)

In the diagnosis assistance method for diagnosing the user's neurological disease using the user's eye movement,
displaying an indicator for acquiring the user's eye information and moving the indicator;
obtaining the user's eye information; and
inputting the user's eye information into a first neural network model and obtaining diagnostic assistance information on the neurological disease based on an output result
including,
The diagnostic auxiliary information is
At least one of information on whether the user has at least one neurological disease or information on a probability that the user has the at least one neurological disease,
The at least one neurological disease is
comprising at least one of dementia, Alzheimer's disease or Parkinson's disease;
Diagnostic aid methods.
According to claim 1,
The user's eye information includes a plurality of eyeball images of the user,
Diagnostic aid methods.
According to claim 1,
The user's left eye and right eye are included in the plurality of eyeball images of the user,
Diagnostic aid methods.
According to claim 1,
The user's eye information includes a coordinate value corresponding to the user's eye movement,
Diagnostic aid methods.
5. The method of claim 4,
The step of obtaining the user's eye information includes,
acquiring a plurality of eyeball images of the user;
obtaining a coordinate value of an eyeball center included in each of the plurality of eyeball images of the user;
time-sequentially classifying the coordinate values of the eyeball center included in each of the plurality of eyeball images; and
acquiring the coordinate values of the eyeball center divided in time series as coordinate values corresponding to the user's eye movement;
containing,
Diagnostic aid methods.
According to claim 1,
The indicator is displayed on the mobile device,
The mobile device includes an imaging device,
The plurality of eyeball images of the user are obtained through the imaging device,
The distance between the imaging device and the user's face is equal to or greater than a first reference distance,
The first reference distance is set based on a viewing angle of the imaging device,
Diagnostic aid methods.
According to claim 1,
The indicator is displayed on the mobile device,
The mobile device includes an imaging device,
A plurality of eyeball images of the user are obtained through the imaging device,
The distance between the imaging device and the user's face is within a second reference distance,
The second reference distance is set based on a viewing angle of the imaging device or a reference amplitude of eye movement,
Diagnostic aid methods.
According to claim 1,
obtaining prescription information based on the diagnosis auxiliary information;
further comprising,
Diagnostic aid methods.

Images