KR20210152249A - Apparatus and method for artificial intelligence based automatic analysis of video fluoroscopic swallowing study - Google Patents

Abstract

Disclosed are an apparatus and method for automatic analysis of video fluoroscopic swallowing study (VFSS) based on artificial intelligence, which can automate the analysis of VFSS based on an artificial intelligence technology, and a recording medium. According to an embodiment of the present invention, the apparatus for automatic analysis of VFSS based on artificial intelligence includes: a data input unit which is configured to receive image frames of a VFSS video acquired for an object to be analyzed, by a VFSS inspection device for VFSS; and a swallowing stage classifying unit which classifies the image frames of the VFSS video into swallowing stages including an oral stage, a pharyngeal stage, and an esophageal stage by an artificial intelligence model.

Description

Apparatus and method for artificial intelligence based automatic analysis of video fluoroscopic swallowing study

The present invention relates to an artificial intelligence-based video fluoroscopic swallowing test automation analysis apparatus and method for automating the classification of the swallowing phase and the determination of the level of swallowing disorder in a video fluoroscopic swallowing study (VFSS) based on artificial intelligence technology.

Dysphagia refers to symptoms of difficulty swallowing caused by abnormalities in the muscular nervous system or structural abnormalities in the upper esophagus. Dysphagia is a common symptom not only in elderly patients, but also in stroke patients and head and neck tumor patients. Typical complications include aspiration pneumonia, airway blockage, nutritional disorders, or dehydration. However, in most cases, it is difficult to treat the causative disease of dysphagia, and even the same causative disease causes various types of dysphagia. more helpful

A video fluoroscopic swallowing study (VFSS) that is currently recognized as a gold standard as a test method for diagnosing swallowing disorders is a video fluoroscopic swallowing study. According to the VFSS method, for example, structural abnormalities and movements of the oral cavity, pharynx, and esophagus can be evaluated using a barium contrast agent, and airway aspiration can be directly confirmed.

However, in the case of the conventional VFSS test method, the analysis after acquiring the VFSS digital image is performed by the analyst manually analyzing the movement path of the food mass through frame-by-frame analysis of the video or measuring the swallowing time for each swallowing step. Classification of disabilities. As such, since the image data analysis work after the VFSS test is performed manually depending on the experience and skill level of the analyst, the bias of the analysis result is high, and the reproducibility and objectivity are low. In addition, the results of the analysis may be different depending on the analyst, so the consistency of the results is inevitably low. In addition, since labor-intensive analysis that requires a lot of labor and time for the analyst is required, the analysis results are inevitably presented after a considerable amount of time is also required.

An object of the present invention is to provide an artificial intelligence-based video fluoroscopic swallowing test automation analysis apparatus and method, and a recording medium that automates the classification of the swallowing phase and the determination of the level of dysphagia in the video fluoroscopic swallowing test (VFSS) based on artificial intelligence technology.

An artificial intelligence-based video fluoroscopic swallowing test automated analysis apparatus according to an embodiment of the present invention includes a data input unit configured to receive image frames of a VFSS moving picture acquired for an analysis object by a VFSS test device for a video fluoroscopic swallowing test (VFSS) ; and a swallowing stage classification unit that classifies the image frames of the VFSS video into swallowing stages including the oral stage, pharyngeal stage, and esophageal stage by an artificial intelligence model.

Artificial intelligence-based video fluoroscopy automatic swallowing test analysis apparatus according to an embodiment of the present invention is data for generating the artificial intelligence model by learning the learning data including the VFSS learning image and examination data for each swallowing step for a plurality of patients with swallowing disorders It may further include a learning unit.

The artificial intelligence-based video fluoroscopic swallowing test automation analysis apparatus according to an embodiment of the present invention extracts an image section corresponding to each swallowing step classified by the artificial intelligence model from the FSS video, and corresponds to each swallowing step It may further include a swallowing time calculator configured to calculate a swallowing time for each swallowing step from the video section.

The swallowing stage classification unit may be configured to classify a swallowing disorder according to a swallowing disorder classification criterion by the artificial intelligence model. The swallowing disorder classification criteria may include a Penetration-Aspiration Scale, which is defined to classify a swallowing disorder as one of a plurality of penetration-aspiration scale levels.

The artificial intelligence model, based on the output values of a plurality of sub-AI models learned for each level of the plurality of penetration-aspiration scales defined in the penetration-aspiration scale, determines the swallowing disorder of the analysis target according to the penetration-aspiration scale It can be configured to classify.

The sub-AI model trained with respect to a first penetration-aspiration scale level among the plurality of penetration-aspiration scale levels may include: an image feature extraction unit configured to extract a plurality of image feature information from each image frame of the VFSS video; and a sub-artificial neural network configured to output a probability related to the first penetration-aspiration scale level based on the plurality of image feature information.

The image feature extraction unit may include: at least one convolution processing unit configured to generate a convolutional image by convolution processing each image frame of the VFSS video; and one or more sub-sampling processing units for sub-sampling the convolutional image in order to extract the plurality of image feature information.

The sub artificial neural network may include: an input layer including input nodes configured to receive the plurality of image feature information; an output layer comprising an output node configured to output a probability relating to the first penetration-aspiration scale level; and a hidden layer including hidden nodes connected between the input nodes and the output nodes in a fully connected layer structure.

The artificial intelligence-based video fluoroscopic swallowing test automation analysis apparatus according to an embodiment of the present invention detects a moving object in which motion has occurred except for a still object in the VFSS video, tracks the movement path, and provides the analysis target in the VFSS video. It may further include a moving object tracking unit configured to track the movement path of the food mass ingested by the.

In the artificial intelligence-based video fluoroscopic swallowing test automated analysis method according to an embodiment of the present invention, image frames of a VFSS video acquired for an analysis target by a VFSS test device for video fluoroscopic swallowing test (VFSS) are input by a data input unit receiving; and classifying, by the swallowing stage classification unit, the image frames of the VFSS video into swallowing stages including the oral stage, pharyngeal stage, and esophageal stage by an artificial intelligence model.

The artificial intelligence-based video fluoroscopic swallowing test automation analysis method according to an embodiment of the present invention is performed by learning, by a data learning unit, learning data including VFSS learning images and test data for a number of swallowing disorder patients for a number of swallowing disorders. The method may further include generating an intelligence model.

The artificial intelligence-based video fluoroscopic swallowing test automated analysis method according to an embodiment of the present invention extracts, by a swallowing time calculator, an image section corresponding to each swallowing stage classified by the artificial intelligence model from the VFSS video, The method may further include calculating a swallowing time for each swallowing step from the image section corresponding to each swallowing step.

The artificial intelligence-based video fluoroscopic swallowing test automation analysis method according to an embodiment of the present invention is defined to classify a swallowing disorder into any one of a number of penetration-aspiration scale levels by the artificial intelligence model Penetration- The method may further include classifying the swallowing disorder of the analysis target according to the Aspiration Scale).

In the step of classifying the swallowing disorder of the analysis target, the swallowing disorder of the analysis target is classified according to the penetration-aspiration scale based on the output values of a plurality of sub-AI models learned for each of the plurality of penetration-aspiration scale levels. may include steps.

In the step of classifying the swallowing disorder of the analysis target according to the penetration-aspiration scale, the first sub-AI model learned with respect to a first penetration-aspiration scale level among the plurality of penetration-aspiration scale levels includes the first predicting the probability of the penetration-aspiration scale level.

The step of predicting the probability of the first penetration-suction scale level includes, by the image feature extraction unit of the first sub-AI model, convolution processing each image frame of the VFSS video to generate a convolutional image, extracting a plurality of image feature information from each image frame by subsampling the convolutional image; and outputting, by a sub-artificial neural network having a fully connected layer structure of the first sub-AI model, a probability regarding the first penetration-suction scale level based on the plurality of image feature information. .

In the artificial intelligence-based video fluoroscopic swallowing test automation analysis method according to an embodiment of the present invention, a moving object in which a movement has occurred except for a still object in the VFSS video is detected by a moving object tracking unit to track the movement path, and the The method may further include tracking the movement path of the food mass ingested by the analysis target in the VFSS video.

According to an embodiment of the present invention, there is provided a computer-readable recording medium in which a program for executing the artificial intelligence-based video fluoroscopy automated swallowing test analysis method is recorded.

According to an embodiment of the present invention, there is provided an artificial intelligence-based video fluoroscopic swallowing test automation analysis apparatus and method, and a recording medium that automates the classification of the swallowing stage and the determination of the level of swallowing disorder of the video fluoroscopic swallowing test (VFSS) based on artificial intelligence technology do.

1 is a block diagram of an artificial intelligence-based video fluoroscopy automatic swallowing test analysis apparatus according to an embodiment of the present invention.
2 is a flowchart of an artificial intelligence-based video fluoroscopy automatic swallowing test analysis method according to an embodiment of the present invention.
3 is a diagram illustrating an oral stage, a pharyngeal stage, and an esophageal stage.
4 is a block diagram of an artificial intelligence model constituting an artificial intelligence-based video fluoroscopy automatic swallowing test analysis apparatus according to an embodiment of the present invention.
5 is an exemplary view of the penetration-aspiration scale for the evaluation of dysphagia.
6 is a block diagram of a sub-AI model constituting an artificial intelligence-based video fluoroscopy automatic swallowing test analysis apparatus according to an embodiment of the present invention.
7 is a flowchart of step S150 of FIG. 2 .
8 and 9 are conceptual views illustrating an artificial intelligence model constituting an artificial intelligence-based video fluoroscopy automatic swallowing test analysis apparatus according to an embodiment of the present invention.
10 is an exemplary diagram of a VFSS video.

Advantages and features of the present invention and methods of achieving them will become apparent with reference to the embodiments described below in detail in conjunction with the accompanying drawings. However, the present invention is not limited to the embodiments disclosed below, but may be implemented in various different forms, and only these embodiments allow the disclosure of the present invention to be complete, and common knowledge in the art to which the present invention pertains It is provided to fully inform those who have the scope of the invention, and the present invention is only defined by the scope of the claims. Like reference numerals refer to like elements throughout.

In the present specification, when a part "includes" a certain component, it means that other components may be further included, rather than excluding other components, unless otherwise stated. As used herein, '~ unit, ~ module' is a unit that processes at least one function or operation, and may refer to, for example, software, FPGA, or hardware component. The functions provided by '~ unit, ~ module' may be performed separately by a plurality of components, or may be integrated with other additional components. 'Part, ~ module' in this specification is not necessarily limited to software or hardware, and may be configured to be in an addressable storage medium, or may be configured to reproduce one or more processors. Hereinafter, an embodiment of the present invention will be described in detail with reference to the drawings.

1 is a block diagram of an artificial intelligence-based video fluoroscopy automatic swallowing test analysis apparatus according to an embodiment of the present invention. Referring to FIG. 1 , an artificial intelligence-based video fluoroscopic swallowing test automation analysis apparatus 100 according to an embodiment of the present invention automatically performs a video fluoroscopic swallowing test (VFSS) for an analysis target, such as a patient with dysphagia, based on artificial intelligence. It may include a swallowing test unit 100a configured to perform.

In addition, the artificial intelligence-based video fluoroscopy automatic swallowing test analysis apparatus 100 according to an embodiment of the present invention receives a VFSS video captured for an analysis target by a VFSS test device and outputs a data input unit ( 120), a controller 180 that controls the swallowing test unit 100a to execute a swallowing test function, a memory 190 for storing programs and various information for the swallowing test function of the swallowing tester 100a, and a swallowing tester 100a may include an output unit 170 for outputting various swallowing test results provided by

2 is a flowchart of an artificial intelligence-based video fluoroscopy automatic swallowing test analysis method according to an embodiment of the present invention. 1 and 2 , the swallowing test unit 100a may include a data learning unit 110 , a swallowing stage classification unit 130 , a swallowing time calculation unit 150 , and a moving object tracking unit 160 . .

The data learning unit 110 provides VFSS learning images and test result data for each swallowing disorder patient (or a person without a swallowing disorder) for various swallowing disorders (for example, the time taken for each swallowing step while the patient is performing the VFSS test, swallowing An artificial intelligence model (artificial intelligence probabilistic model) 140 may be generated by learning the learning data including a result of evaluation by a doctor or the like of a quantitative evaluation value according to the disability classification criterion (S110).

3 is a diagram illustrating an oral stage (a), a pharyngeal stage (b) and an esophageal stage (c). The swallowing phase may include an oral phase, pharyngeal phase and esophageal phase. The oral stage is a stage in which the mass eaten by the analysis target stays in the oral cavity (1), the pharyngeal stage is a stage in which the mass passes through the pharynx (2), and the esophageal stage is the stage in which the mass passes through the esophagus (3). may be a step.

The data learning unit 110 learns the artificial intelligence network by learning data divided by each swallowing step and swallowing disorder classification criteria, for example, Penetration-Aspiration Scale (PAS) for swallowing disorder classification, and VFSS It is possible to create an artificial intelligence model for classifying and extracting image sections for each swallowing step in the video, and for classifying swallowing disorders.

The data input unit 120 may receive the image frames of the VFSS moving picture acquired for the analysis target by the VFSS inspection device for video fluoroscopic swallowing test (VFSS) and transmit the received image frames to the swallowing test unit 100a ( S120 ). VFSS moving picture information may be digital data, and various data preprocessing such as a normalization algorithm may be performed on the collected data.

The swallowing phase classification unit 130 divides the image frames of the VFSS video into an oral phase and a pharyngeal phase using the artificial intelligence model 140 generated by the learning of the data learning unit 110 . And it can be classified into three swallowing phases (oral phase, pharyngeal phase, and esophageal phase) or four or more swallowing phases including an esophagus phase (S130).

The swallowing time calculator 150 extracts the video section corresponding to each swallowing stage classified by the artificial intelligence model 140 from the VFSS video, and calculates the swallowing time for each swallowing step from the video section corresponding to each swallowing step. can be (S140). The swallowing time calculator 150 may calculate the swallowing time for each swallowing step from a value obtained by multiplying the number of consecutive video frames corresponding to each swallowing step among the video frames of the VFSS video by the interval between two adjacent video frames, for example. have.

Also, the swallowing stage classification unit 130 may classify a swallowing disorder to be analyzed according to the swallowing disorder classification criteria by the artificial intelligence model 140 ( S150 ). In an embodiment, the dysphagia classification criterion may include a Penetration-Aspiration Scale (PAS). The penetration-aspiration scale may be defined to classify a dysphagia into one of a number of levels of the penetration-aspiration scale.

The moving object tracking unit 160 detects a moving object (eg, hyoid bone, food mass, etc.) in which motion has occurred in the VFSS video except for a static object, and tracks the movement path, and food mass consumed by the analysis target in the VFSS video. It is possible to track the movement path of (S160). Through this, it is possible to determine the cause of the dysphagia or determine airway aspiration.

The moving object tracking unit 160, for example, extracts objects from sequentially input image frames, generates a motion vector from the amount of coordinate change of the objects between adjacent image frames, and classifies the objects into a still object and a moving object. However, it is also possible to track the moving object in a different way.

4 is a block diagram of an artificial intelligence model constituting an artificial intelligence-based video fluoroscopy automatic swallowing test analysis apparatus according to an embodiment of the present invention. 1 to 4 , the artificial intelligence model 140 may include a plurality of sub artificial intelligence models 142 , 144 , 146 , and a penetration-suction scale determining unit 148 .

5 is an exemplary view of the penetration-aspiration scale for the evaluation of dysphagia. 6 is a block diagram of a sub-AI model constituting an artificial intelligence-based video fluoroscopy automatic swallowing test analysis apparatus according to an embodiment of the present invention. 7 is a flowchart of step S150 of FIG. 2 .

4 to 7, the artificial intelligence model 140 is a plurality of penetration-aspiration scale levels defined in the penetration-aspiration scale (eg, PAS1 to PAS8 shown in FIG. 5) of a plurality of learned sub Based on the output values (probability values) of the artificial intelligence models 142, 144, and 146, the swallowing disorder of the analysis target may be classified according to the penetration-aspiration scale.

The penetration-aspiration scale (PAS) can be divided into three categories: normal, penetration, and aspiration, and again, as shown in FIG. PAS8).

In FIG. 5 , PAS1 is a normal level at which food does not penetrate into the airways, PAS2 is a first penetration level where food enters the airways and remains on the vocal cords and there is no residue, and PAS3 is food flows into the airways and remains on the vocal cords and remains on the vocal cords. The second level of penetration in which water remains, PAS4, is the third penetration level, in which food enters the airway and contacts the vocal cords and no residue remains, and PAS5, where food enters the airways and contacts the vocal cords and residues remain. 4 indicates the level of penetration.

In addition, PAS6 indicates that food enters the airway and passes through the glottis, the first level of aspiration with no residue remaining under the glottis, PAS7 indicates that food enters the airway and passes through the glottis, and the patient responds through the glottis. The second aspiration level with residual residue underneath, PAS8, represents the third aspiration level at which food enters the airway and passes through the glottis, residual residue is present under the glottis, and the patient is unresponsive.

For probability prediction for each PAS level, each sub-AI model 142 , 144 , and 146 of the AI model 140 may include an image feature extractor 142a and a sub-artificial neural network 142b. The image feature extraction unit 142a may extract a plurality of pieces of image feature information (a plurality of feature maps) from each image frame of the VFSS video.

8 and 9 are conceptual diagrams illustrating an artificial intelligence model constituting an artificial intelligence-based video fluoroscopy automatic swallowing test analysis apparatus according to an embodiment of the present invention. 8 is a diagram illustrating a process of learning an artificial intelligence model.

Referring to FIGS. 1, 4, and 6 to 8 , the process of learning the artificial intelligence model will be described. The data learning unit 110 learns the artificial intelligence model using the learning data (training data) 10 . do.

The training data 10 includes positive learning data 20 corresponding to VFSS video and test data of a person without a swallowing disorder, and negative learning data 30 corresponding to VFSS video and test data of a person with a swallowing disorder. may include

Negative learning data 30 may be divided by swallowing stage and by swallowing disorder classification criteria, for example, a plurality of penetration-aspiration scale (PAS) levels may include sub-learning data 32, 34, 36 classified by level. have.

Each sub-learning data 32 , 34 , and 36 may be utilized for learning of each corresponding sub-AI model 142 , 144 , 146 . The sub-AI models 142, 144, and 146 may be provided as many as the number of penetration-suction scale (PAS) levels.

In this way, various VFSS videos are classified by swallowing disorder criteria (eg, PAS), and the video frames of each VFSS video are divided into consecutive frames for each swallowing step, and then, through artificial intelligence-based training using this, An artificial intelligence model 140 for classifying a level and a swallowing stage may be generated.

9 is a diagram illustrating a process of automatically performing a swallowing test by a learned artificial intelligence model. 10 is an exemplary diagram of a VFSS video. 1, 4, 6, 7, 9 and 10 , the image feature extracting unit 142a performs a plurality of image feature information (for example, in the image frame 40 of the VFSS video 50 ). , feature map) may include one or more convolution processing units 1422 and one or more sub-sampling processing units 1424 .

The convolution processing unit 1422 may generate a plurality of convolutional images (first feature maps) by convolutionally processing each image frame 40 of the VFSS video ( S152 ).

The convolution processing unit 1422 converts a reference image (eg, a 3×3 reference image, a 2×2 reference image, a 7×7 reference image, etc.) of a set pixel size into a VFSS image (or a previously generated convolution image). can be mapped to create a feature map.

The sub-sampling processing unit 1424 pools the convolutional images generated by the convolution processing unit 1422 in order to extract a plurality of image feature information (second feature maps) from the plurality of convolutional images (first feature maps). A subsampling process such as (pooling) may be performed (S154).

The convolution processing of the convolution processing unit 1422 and the sub-sampling processing of the sub-sampling processing unit 1424 may be sequentially and repeatedly performed a plurality of times, and the sub artificial neural network 142b through the final convolution processing or sub-sampling processing. Image characteristic information to be input may be generated.

The sub artificial neural network 142b is based on a plurality of image feature information (eg, feature maps) extracted by the image feature extraction unit 142a, for example, the first penetration-suction by the fully connected layer artificial neural network. A probability related to the scale level may be output (S156).

The sub artificial neural network 142b includes an input layer (IL) including input nodes receiving a plurality of image feature information, and an output layer (OL) including one or more output nodes for outputting a probability related to the first penetration-aspiration scale level and a hidden layer HL including hidden nodes connected between input nodes and output nodes in a fully connected layer structure.

The penetration-aspiration scale determining unit 148 penetrates the swallowing disorder of the analysis target based on the output values (probability values) of a plurality of sub-AI models 142, 144, and 146 learned for each of the plurality of penetration-aspiration scale levels. -Can be classified according to the suction scale (S158).

Each output value (probability value) of the sub-AI models 142, 144, and 146 may be output as a binary value of 0 or 1, or may be output as a probability value or numerical value within a preset range between 0 and 1.

Penetration-aspiration scale determining unit 148, for example, the probability value output from the first sub-AI model corresponding to the first PAS level among the plurality of sub-AI models 142, 144, 146 is 1, and the remaining When the probability value output from the sub-AI model is 0, the penetration-aspiration scale (PAS) of the analysis target may be determined as the first PAS level.

In addition, the penetration-aspiration scale determining unit 148 is a statistical value of the probability values output from a plurality of sub-AI models 142, 144, 146 (eg, average or each PAS level set or learned weight is reflected) It is also possible to determine the PAS level of an analysis target based on a weighted average, etc.).

In this way, when the patient's VFSS video 50 information is input, the user can classify the video for each swallowing step by successive frames, estimate the swallowing time for the corresponding swallowing step, and finally predict the level of swallowing disorder (PAS level). can

According to an embodiment of the present invention, it is possible to generate an artificial intelligence probabilistic model for classifying the swallowing stage and the level of swallowing disorder through artificial intelligence learning based on the results of data mining of vast VFSS test video data and test result information. Based on an intelligent probabilistic model, it is possible to automatically recognize and extract video frames for each swallowing step from a random VFSS test video, automatically predict the time required for each swallowing step and classification of swallowing disorders, and provide the swallowing test results as shown in Table 1. have.

Therefore, according to the embodiment of the present invention, the labor-intensive analysis work of the VFSS analyst who has to analyze each frame image of the VFSS video can be reduced, and the time measurement for each swallowing phase and the prediction of the degree of swallowing disorder can be accurately and consistent with results. It can be presented, so it is possible to improve the efficiency and accuracy of diagnosis of dysphagia.

In addition, according to an embodiment of the present invention, it is possible to detect a moving object (eg, hyoid bone, food mass, etc.) in which a motion has occurred except for a still object in the VFSS video to track the movement path.

That is, the movement state (movement range, movement trajectory, etc.) of organs related to swallowing, such as the hyoid bone, is calculated for each section corresponding to each swallowing stage, and compared with a preset normal movement state or abnormal movement state. It can be used to determine the cause of

In addition, according to an embodiment of the present invention, by tracking the movement path of the food mass ingested by the analysis target in the VFSS video, whether the food mass has penetrated into the airway, whether aspiration, etc. is a moving object It can be confirmed through the movement trajectory of

The embodiments described above may be implemented by a hardware component, a software component, and/or a combination of a hardware component and a software component. For example, the apparatus, method, and component described in the embodiments may include, for example, a processor, a controller, an arithmetic logic unit (ALU), a digital signal processor, a microcomputer, a field programmable gate (FPGA). Array), a programmable logic unit (PLU), a microprocessor, or any other device capable of executing and responding to instructions, may be implemented using one or more general purpose or special purpose computers.

The processing device may run an operating system and one or more software applications running on the operating system. A processing device may also access, store, manipulate, process, and generate data in response to execution of the software. For convenience of understanding, the processing device is sometimes described as being used, but one of ordinary skill in the art will recognize that the processing device includes a plurality of processing elements and/or a plurality of types of processing elements. It will be understood that this may include

For example, the processing device may include a plurality of processors or one processor and one controller. Other processing configurations are also possible, such as a Parallel Processor. The software may include a computer program, code, instructions, or a combination of one or more thereof, which configures a processing device to operate as desired or is independently or collectively processed You can command the device.

The software and/or data may be any kind of machine, component, physical device, virtual equipment, computer storage medium or device, to be interpreted by or provide instructions or data to the processing device. , or may be permanently or temporarily embody in a transmitted signal wave. The software may be distributed over networked computer systems and stored or executed in a distributed manner. Software and data may be stored in one or more computer-readable recording media.

The method according to the embodiment may be implemented in the form of program instructions that can be executed through various computer means and recorded in a computer-readable medium. The computer-readable medium may include program instructions, data files, data structures, etc. alone or in combination. The program instructions recorded on the medium may be specially designed and configured for the embodiment, or may be known and available to those skilled in the art of computer software.

Examples of the computer-readable recording medium include magnetic media such as hard disks, floppy disks and magnetic tapes, optical media such as CDROMs and DVDs, and ROM, RAM, and flash memory. Hardware devices specially configured to store and execute program instructions, such as, etc. are included. Examples of program instructions include not only machine language codes such as those generated by a compiler, but also high-level language codes that can be executed by a computer using an interpreter or the like. The hardware devices described above may be configured to operate as one or more software modules to perform the operations of the embodiments, and vice versa.

As described above, although the embodiments have been described with reference to the limited embodiments and drawings, various modifications and variations are possible from the above description by those skilled in the art. For example, the described techniques are performed in an order different from the described method, and/or the described components of the system, structure, apparatus, circuit, etc. are combined or combined in a different form than the described method, or other components Or substituted or substituted by equivalents may achieve an appropriate result. Therefore, other implementations, other embodiments, and equivalents to the claims are also within the scope of the following claims.

100: Artificial intelligence-based video fluoroscopy swallowing test automation analysis device
100a: swallowing test unit
110: data learning unit
120: data input unit
130: swallowing stage classification unit
140: artificial intelligence model
142, 144, 146: sub-AI model
142a: image feature extraction unit
142b: sub artificial neural network
148: penetration-aspiration scale determining part
150: swallowing time calculator
160: moving object tracking unit
170: output unit
180: control unit
190: memory

Claims (15)

a data input unit configured to receive image frames of a VFSS moving picture acquired for an analysis object by a VFSS inspection device for video fluoroscopic swallowing test (VFSS); and
Artificial intelligence-based video fluoroscopic swallowing test automation analysis device comprising a swallowing stage classification unit that classifies the image frames of the VFSS video into swallowing stages including the oral stage, pharyngeal stage, and esophageal stage by an artificial intelligence model. According to claim 1,
An artificial intelligence-based video fluoroscopic swallowing test automation analysis device further comprising a data learning unit for generating the artificial intelligence model by learning learning data including VFSS learning images and examination data for each swallowing disorder patient for a plurality of swallowing disorders. According to claim 1,
A swallowing time calculator configured to extract an image section corresponding to each swallowing step classified by the artificial intelligence model from the VFSS video and calculate the swallowing time for each swallowing step from the video section corresponding to each swallowing step. AI-based video fluoroscopy automatic swallowing test analysis device. According to claim 1,
The swallowing stage classification unit is configured to classify a swallowing disorder according to a swallowing disorder classification criterion by the artificial intelligence model,
The swallowing disorder classification criterion includes a penetration-aspiration scale that is defined to classify a swallowing disorder into any one of a plurality of penetration-aspiration scale levels. 5. The method of claim 4,
The artificial intelligence model is
Configured to classify the swallowing disorder of the analysis target according to the penetration-aspiration scale based on the output values of a plurality of sub-AI models learned for each penetration-aspiration scale level defined in the penetration-aspiration scale, artificial Intelligence-based video fluoroscopic swallowing test automation analysis device. 6. The method of claim 5,
The sub-AI model learned with respect to the first penetration-aspiration scale level among the plurality of penetration-aspiration scale levels is,
an image feature extraction unit configured to extract a plurality of image feature information from each image frame of the VFSS video; and
And a sub-artificial neural network configured to output a probability related to the first penetration-aspiration scale level based on the plurality of image feature information, AI-based video fluoroscopy automatic swallowing test analysis device. 7. The method of claim 6,
The image feature extraction unit,
one or more convolution processing units for generating a convolutional image by convolutionally processing each image frame of the VFSS video; and
One or more sub-sampling processing units for sub-sampling the convolutional image in order to extract the plurality of image feature information,
The sub artificial neural network is
an input layer including input nodes configured to receive the plurality of image characteristic information;
an output layer comprising an output node configured to output a probability relating to the first penetration-aspiration scale level; and
Artificial intelligence-based video fluoroscopic swallowing test automation analysis apparatus comprising a hidden layer comprising hidden nodes connected in a fully connected layer structure between the input nodes and the output nodes. 8. The method according to any one of claims 1 to 7,
Further comprising a moving object tracking unit configured to detect a moving object in which motion has occurred except for a still object in the VFSS moving image and track the moving path, and to track the moving path of the food mass eaten by the analysis target in the VFSS moving image. , AI-based video fluoroscopic swallowing test automation analysis device. receiving, by a data input unit, image frames of a VFSS moving picture acquired for an analysis object by a VFSS test device for video fluoroscopic swallowing test (VFSS); and
Artificial intelligence-based video fluoroscopic swallowing, comprising, by a swallowing stage classification unit, classifying the image frames of the VFSS video into swallowing stages including the oral stage, pharyngeal stage and esophageal stage by an artificial intelligence model Inspection automation analysis method. 10. The method of claim 9,
Artificial intelligence-based video fluoroscopic swallowing test automation, further comprising the step of generating the artificial intelligence model by learning, by the data learning unit, learning data including VFSS learning images and examination data for each swallowing stage for a plurality of patients with swallowing disorders analysis method. 10. The method of claim 9,
extracting, by the swallowing time calculator, an image section corresponding to each swallowing step classified by the artificial intelligence model from the VFSS video, and calculating the swallowing time for each swallowing step from the video section corresponding to each swallowing step Further comprising, artificial intelligence-based video fluoroscopy swallowing test automation analysis method. 10. The method of claim 9,
Classifying the dysphagia of the analysis subject according to the Penetration-Aspiration Scale, which is defined by the artificial intelligence model to classify the dysphagia into any one of a plurality of penetration-aspiration scale levels, further comprising: ,
The step of classifying the dysphagia of the analysis target is,
Based on the output values of a plurality of sub-AI models learned for each of the plurality of penetration-aspiration scale levels, classifying the swallowing disorder of the analysis target according to the penetration-aspiration scale, AI-based video fluoroscopic swallowing test Automated analysis methods. 13. The method of claim 12,
Classifying the dysphagia of the analysis target according to the penetration-aspiration scale,
Predicting the probability of the first penetration-aspiration scale level by a first sub-AI model learned with respect to the first penetration-aspiration scale level among the plurality of penetration-aspiration scale levels,
Predicting the probability of the first penetration-aspiration scale level comprises:
By the image feature extraction unit of the first sub-AI model, each image frame of the VFSS video is convolutionally processed to generate a convolutional image, and the convolutional image is subjected to subsampling processing to obtain a plurality of images from each image frame. extracting image feature information; and
Comprising the step of outputting a probability regarding the first penetration-suction scale level based on the plurality of image feature information by a sub-artificial neural network having a fully connected layer structure of the first sub-AI model, artificial intelligence A method for automated analysis of swallowing tests based on video fluoroscopy. 10. The method of claim 9,
By the moving object tracking unit, detecting a moving object in which motion has occurred except for a still object in the VFSS moving image, tracking the moving path, and tracing the moving path of the food mass consumed by the analysis target in the VFSS moving image. Further comprising, an artificial intelligence-based video fluoroscopic swallowing test automation analysis method. A computer-readable recording medium in which a program for executing the artificial intelligence-based video fluoroscopic swallowing test automation analysis method of any one of claims 9 to 14 is recorded.

Images