KR101900200B1 - Method of tuberculosis diagnosis based on a deep-learning - Google Patents

Abstract

A tuberculosis diagnosis method based on deep learning according to the present invention includes: a step of acquiring an image from a sputum slide manufactured for training using an image capturing apparatus; a learning step of learning a deep learning model to be used for tuberculosis determination with the acquired image; a verification step of verifying the accuracy of the deep learning model learned through the learning step; a step of determining whether a tuberculosis diagnosis is negative or positive by using a weight value of the deep learning model learned through the learning step and the verification step; and a display step of providing a diagnosis result to a user by displaying the diagnosis result from the tuberculosis determining step on a monitor. Accordingly, the present invention can quickly and conveniently diagnose tuberculosis germs.

Description

[0001] METHOD OF TUBERCULOSIS DIAGNOSIS BASED ON A DEEP-LEARNING [0002]

The present invention relates to a tuberculosis testing method, and more particularly, to a deep running-based tuberculosis testing method capable of reducing an inspection error through automation of a tuberculosis test.

The number of tuberculosis patients is growing steadily, despite the improvement in living and medical status. According to the WHO data, the number of tuberculosis infected during the year of 2013 is 9 million, of which 1.5 million died The number of patients with tuberculosis is increasing again in Korea, and it is the number one in OECD countries.

Methods for diagnosis of tuberculosis include sputum smear test, sputum culture test, drug susceptibility test, chest X-ray and tuberculin skin test (TST).

Among them, sputum smear test and cultivation are most widely used. Especially, a sputum smear test is essential in a public health center and it is evaluated as a very important test.

Sputum smear test is a test method to confirm the presence of Mycobacterium tuberculosis by microscopic examination of the patient's sputum (thin sputum) by thinly spreading it on the slide to confirm whether or not the tuberculosis is released.

Although this sputum smear test is widely used because it can determine the results with relatively simple experimental manipulation and quick time, accuracy of the test results may vary according to the skill of the tester, so that it is difficult to predict the presence of tuberculosis and the progress of treatment have.

In addition, in the case of a microscope used for sputum smear test in general, the inspector puts one slide on the floor at each inspection, changes the position and focus of the slide, There is a need to replace the slide with another slide.

In order to improve this, studies on automation method using machine learning which learns feature extraction and extracted features using image processing have been carried out. However, since it does not show enough sensitivity and specificity to replace actual inspection, There is a difficulty.

In addition, since the focus is not constant depending on the posture of the slides or the type of sputum smear, there is a problem that it is difficult to distinguish the type of mycobacteria because the focus is not adjusted even if the automatic photographing is performed.

In addition, since it is a method of confirming tuberculosis with naked eyes, there is a problem that the tester causes visual fatigue.

It is an object of the present invention to solve the above-mentioned problems, and it is an object of the present invention to provide a deep learning method capable of quickly and conveniently detecting a tubercle bacillus by a simple operation by automating a sputum smear test process from shooting to inspection of a slide by applying an image processing technique and a deep learning technique And to provide a method for testing tuberculosis.

Acquiring an image from a sputum slide formed for training using a video imaging device; A learning step for learning a deep learning model to be used for tuberculosis determination with the acquired image; A verification step of verifying the accuracy of the deep learning model learned through the learning step; Determining whether the tuberculosis voice is positive or negative using the weight value of the learned deep learning model through the learning step and the verification step, and displaying the test result from the tuberculosis determining step on a monitor to provide the result to the user Step.

The learning step for learning the deep learning model to be used for the tuberculosis determination using the acquired image may include a training image capturing step including a training image capturing device, a training image transmitting step for transmitting the captured training image, A training data collection step of collecting data necessary for learning, a training data generation step of generating data necessary for learning the deep learning model, and a deep learning model learning step of learning with the training data.

In the verification step, a part of the data collected in the training data collection step of the learning step is used for verification. The verification software uses the determination result obtained through the deep learning model and the negative or positive result of the tuberculosis determined through the culture result The sensitivity, specificity and accuracy of the deep learning model can be calculated by comparison.

Wherein the determining step includes a sample image photographing step of photographing an image from a sample slide according to a photographing range calculated through the photographing planning step in a single layer or a multi-layer photographing, a step of taking a single layer or a multi- A sample determining step of determining whether a tuberculosis is negative or positive using a weight value of an object classification model or an object detection model learned through the verification step; And a test result display step of displaying a result determined in the sample determination step.

According to this aspect, the present invention has an effect that the operation is simplified by automating sputum smear inspection and the detection of mycobacteria is convenient.

In addition, it is not necessary for the examiner to observe the microscope manually, and the inspection error according to the proficiency of the examiner can be reduced.

1 is a photographing apparatus installed in the TB research institute according to an embodiment of the present invention.
FIG. 2 is a method for calculating focus by object according to an embodiment of the present invention.
3 is a block diagram of a training data collection program of an object detection model according to an embodiment of the present invention.
4 is a view illustrating a Z-axis focus movement according to an embodiment of the present invention.
5 is a screen for enlarging or reducing an image according to an embodiment of the present invention.
6 is an image change in Z-axis movement according to an embodiment of the present invention.
FIG. 7 is a training data collection program user interface of an object classification model according to an embodiment of the present invention.
8 is a photographing planning step according to an embodiment of the present invention.

Hereinafter, embodiments of the present invention will be described in detail with reference to the accompanying drawings so that those skilled in the art can easily carry out the present invention. The present invention may, however, be embodied in many different forms and should not be construed as limited to the embodiments set forth herein. In order to clearly illustrate the present invention, parts not related to the description are omitted, and similar parts are denoted by like reference characters throughout the specification.

Hereinafter, a deep-run-based TB inspection method according to an embodiment of the present invention will be described with reference to the accompanying drawings.

FIG. 1 is an image capturing apparatus installed in a TB research institute according to an embodiment of the present invention. FIG. 2 is a method of calculating focus according to an object according to an embodiment of the present invention. FIG. 4 is a screen for moving a Z-axis focus according to an embodiment of the present invention, and FIG. 5 is a view for explaining a screen for zooming in or zooming out an image according to an embodiment of the present invention. 6 is an image change in Z-axis movement according to an embodiment of the present invention, FIG. 7 is a training data collection program user interface of an object classification model according to an embodiment of the present invention, and FIG. 8 is an embodiment .

As shown in Table 1, the structure of the present invention is largely composed of three steps using data, a deep learning model, and data and a deep learning model.

The data is composed of a single-layer type data and a multi-layer type.

The monolayer data means an image taken on one Z-axis in one X-axis.

The multi-layered type means a plurality of sets of images taken while moving the Z axis on one XY-axis line.

The deep learning model is composed of a classification model and a detection model.

The classification model refers to a model that extracts objects from an image and classifies the extracted objects into classes of previously learned classes.

The detection model refers to a model in which objects to be searched are learned in advance and then the presence and position of the objects are detected on an image (Detection).

Therefore, the classification and detection models differ in the training data and the applied fields for learning each model.

In the present invention, both models are applied in order to utilize the characteristics of each model and to compare the accuracy.

The three steps are the process of preparing a judgment model to be used for the examination of actual tuberculosis using the data and the deep learning model.

The three steps consist of a learning step, a verification step and a judgment step.

The learning step is a step for learning a deep learning model to be used in the tuberculosis determination. The learning step includes a training image capturing step including a training image capturing device, a training image transmitting step for transmitting the captured training image, A training data generation step of generating data necessary for learning the deep learning model, and a deep learning model learning step of learning with the training data.

The training image capturing step will be described in detail with reference to FIG. 1 and FIG.

Referring to FIGS. 1 and 2, a deep-running-based tuberculosis testing method according to an embodiment of the present invention acquires an image from a sputum slide prepared for training using an imaging apparatus.

The image capturing device includes a control PC, a three-axis (XYZ) platform, an optical system, a fluorescent illumination system, and a fluorescent camera.

By using the control PC, the operation of the three-axis station, the illumination system, and the fluorescence camera is controlled, and the sputum slide is photographed with the smudged sputum slide. At this time, it provides a multi-layer photographing in which a plurality of photographs are taken while moving the Z-axis when photographing.

In the case of the multi-layer imaging, the operation is repeated while moving the XY axis by capturing an image while moving the Z-axis at a predetermined distance from the XY point several times. In this case, the total number of images to be photographed is equal to the sum of the number of movements of the X axis and the number of movements of the Y axis multiplied by the number of movements of the Z axis. In this way, when one slide is photographed in a multi-layered manner, it is possible to acquire images of various focuses rather than one fixed focus, so that it is possible to solve the focus problem that may occur when the slide is not fixed flat on the slide fixture . Also, due to the difference in position of the Z axis between objects in the smear plane, it is possible to solve the problem of having different focuses for each object in one FOV (Field Of View).

If there are different focuses among the objects in the image taken at the same position, the individual focus for each object is obtained by the method of FIG.

Referring to FIG. 2, Step 1 shows the photographed images pic 1 to pic 3 moving in the Z-axis at the same XY position. A to D represent objects photographed in the image. As shown in Step 2 of FIG. 2, if objects have different focuses in one image, objects are detected in all images through object extraction. At this time, algorithms such as Connected component, Watershed, Determinant of hessian, Difference of Gaussian and Laplacian of Gaussian are used for object extraction.

If any object is detected in any one of several images of different Z-axes, the coordinates of the detected object are obtained, and the object located at the same coordinates of another Z-axis image is determined as the same object, To calculate the sharpness (Sharpness). Sharpness is a numerical value indicating the boundary of the shape of an object photographed by the camera, and has a higher value as the boundary is clearer and more accurate. For the calculation of sharpness, the Slanted edge algorithm, ISO 12233 standard, was used.

In this way, by separately calculating the focus of each photographed object, not focusing on the entire image, it is possible to solve the problem of having different focuses for each object, and it is possible to solve the problem that the shape of the object is sensitively deformed as the focus changes have.

In the training image transmission step, the image captured through the image capturing device is automatically transmitted to the collection server using a file transfer protocol.

In order to smoothly transmit a large number of slide images during the training image transmission, the image capture and acquisition steps are separated and a pipelined technique is applied.

When the image is directly transmitted immediately after the image is shot, the waiting time for the next slide is longer by the time of transferring the file, which is inefficient for shooting the slide. In order to solve this problem, it is necessary to separate the image shooting and the image transmission step so that the slide shooting and the slide transmission can be independently performed, that is, during the shooting of the next slide, Shooting and transmission are possible.

In the training data collection step, the training data is collected using the image stored in the server and the training data collection program through the training image transmission step.

The training data collection program provides the images stored on the collection server through the imaging equipment to a number of clinical pathologists through a PC or a touch screen device in the form of a web or mobile application.

For the training data collection program, the detection model and the classification model have separate collection programs for each model to collect data suitable for the two types of deep learning models.

In the case of the training data collection program for the object detection model, a plurality of clinical pathologists display the object identified as the tuberculosis in the image separately by using the touch screen device, Collect training data.

At this time, when collecting data using the touch screen device, it is possible to collect data not only by the finger touch but also by the capacitive touch pen.

3, the training data collection program of the object detection model includes a slide selection unit 11 for selecting a slide, a tool selection unit 12 for attaching various functions to a slide selected by the slide selection unit 11, A focus moving unit 13 for moving the focus of the selected slide, and a size adjusting unit 14 for adjusting the magnification of the selected slide screen.

When the slide selection unit 11 is clicked, the slide list photographed through the training data photographing equipment can be confirmed. If the slide ID to start the collection is selected, the training data collection for the slide can be started.

The tool selection unit 12 includes a back button for canceling the most recently selected mycobacterial area selected by the user, a pen size setting used for selecting the tubercle bacillus region, a pen color setting, an image original size view, A comparison button with the selected object area, an eraser capable of erasing a part of the selected area, an image movement mode and an image movement or drawing button for selecting a drawing mode, and a save information storage button.

The focus shifting unit 13 can confirm the Z-axis focus of the multi-layered image by moving the focus.

The size adjuster 14 provides a function of enlarging or reducing an image.

6 (a) is a conceptual view of an image change due to Z-axis focus movement, and FIG. 6 (b) is an illustration of another image change in Z-axis focus movement.

Referring to FIG. 7, in the case of a training data collection program for an object classification model, a plurality of clinical pathologists construct an object in which an object identified as a tubercle bacterium among the objects in the image is Yes, and an object identified as a non- .

Since the object classification model distinguishes the class of the patch image unlike the object detection model that collects the area data, it is not necessary to collect the area by using the touch screen device, and the deep learning model training Collect the data.

Next, in the training data generation step, the data collected through the training data collection step is processed into a learning form in a deep learning model to generate a training data set (Dataset).

Deep learning models for tuberculosis use two methods, Object Classification Model and Object Detection Model, and generate datasets in a form suitable for each model.

First, in the case of the object classification model, a patch image and label data for an object other than the Mycobacterium tuberculosis and the Mycobacterium tuberculosis are respectively generated. The patch image is extracted by using object extraction algorithm such as Connected Component, Watershed, Determinant of hessian, Difference of Gaussian, and Laplacian of Gaussian. Then, a new image is created for each object. The patch image has a size of N * N (e.g., 64 * 64 or 128 * 128).

At this time, the data set to be generated is generated in a single layer form or a multi layer form. In the case of the single-layered data set, the image has a TB image of one Z-axis corresponding to the focus calculated on an object-by-object basis, and an object label that is not a M. tuberculosis or M. tuberculosis. In the case of the multi-layered data set, a plurality of Z-axis images for an object other than the same Mycobacterium tuberculosis or Mycobacterium tuberculosis are combined into a single three-dimensional sample to form the label data of the corresponding sample. The label data for each patch image is generated using the labeled values collected through the training data collection step.

In the case of the object detection model, the original image containing the Mycobacterium on the image and the bounding box information on the Mycobacterial area are labeled. In this case, it is necessary to include the bounding box information for all regions corresponding to the Mycobacterium tuberculosis, and if there are several Mycobacterium tuberculosis bacteria in one image, they are all labeled. Like the object classification model, it is possible to check both images taken in single layer and multiple layer form, but the form of dataset only creates a single layer form. For the dataset, it has an image of mycobacterial zone for one Z-axis and a bounding box label for mycobacterial zone.

In the deep learning model learning step, the deep learning model is learned by using the data set generated according to the object classification model and the object detection model through the training data generation step.

In the case of the object classification model, a classification model for classifying objects other than the Mycobacterium tuberculosis and the Mycobacterium tuberculosis is learned by using the label data corresponding to 1: 1 with the patch image generated through the training data generation step. The patch image is learned by the input of the model and the label data is processed by the end-to-end method that gives the output result of the patch image. The model is constructed using the Convolutional Neural Network (CNN).

The learned results are stored as the model's architecture and weight values. The object classification model learns not only about the Mycobacterium tuberculosis but also about the non-TB microbes, which can cause the accuracy of the model to deteriorate because the shape and size are not regular.

In the case of the object detection model, a model for detecting the Mycobacterium tuberculosis is learned by using the image of the tuberculosis produced through the training data generation step and the bounding box label data.

As with the object classification model, it is an end-to-end learning method that learns the model by learning the label data from the input of the model. However, since it is not necessary to detect objects that are not Mycobacteria, . Therefore, it is possible to reduce judgment errors that may occur when learning about objects other than Mycobacterium tuberculosis. In the case of the object classification model, it is necessary to create a patch image separately, but in the case of the object detection model, it is possible to learn without generating the patch image by using only the bounding box information indicating the original image and the region of the M. tuberculosis.

The verification step is a step for verifying the accuracy of the deep learning model learned through the learning step, and is composed of verification software.

In the verification step, a part of the data collected in the training data collection step of the learning step is used for verification. At this time, the training data used in the learning step and the verification data used in the verification step are used so as not to overlap each other.

The verification software calculates the sensitivity, specificity, and accuracy of the deep learning model by comparing the negative results or the positive results of the determined tuberculosis with the determined results from the deep learning model and the culture results.

The determining step is a step of performing an actual tuberculosis examination using the learned and verified deep learning model through the learning step and the verification step and includes steps of photographing planning step, sample image photographing step, sample determination step, Including,

And an exposing step of displaying the result of the examination through the tuberculosis determining step on a monitor to provide the result to a user.

The photographing planning step is a step for correcting the posture of the slide mounted on the flooring before the photographing of the inspection image from the sample slide, and the slide is fixed to the flooring by predicting the three- It is possible to solve the focus problem that may occur when the slide is not fixed flat. Also, due to the difference in position of the Z axis between objects in the smear plane, it is possible to solve the problem of having different focuses for each object in one FOV. The photographing planning step is further composed of a photographing step for photographing plan, a sharpness calculating step and a photographing range calculating step.

In the imaging plan imaging step, at least four or more points are previously specified along the edge of the smear area excluding the bar code attachment area of the slide, and the image is taken at the corresponding point. At this time, the Z-axis is moved n times at each point.

In the sharpness calculation step, the peak point having the highest sharpness at each point is determined as the sharpest, that is, the Z-axis position with the best focus, by using the images photographed in the multi-layer through the image taking step for the shooting plan.

In the shooting range calculation step, a three-dimensional profile of the slide posture is derived using the Z-axis peak position information calculated at each point through the imaging plan imaging step and the sharpness calculation step. Using this derived slide posture information, the shooting range is calculated during the actual inspection.

8 is a schematic diagram of a photographing plan image capturing step, a sharpness calculating step, and a photographing range calculating step constituting a photographing planning step. In FIG. 8, the number of photographing points is represented by six for the purpose of explaining the invention, but the number of photographing points is not limited, and it can be reduced or expanded according to the application example.

Referring to FIG. 8 (a), P1 to P6 are six points designated in advance for attitude correction of the mounted slide. Moves 6 points while moving the XY axis, and obtains images in multiple layers while moving the Z axis at each point.

Referring to FIG. 8 (b), the sharpness calculation step is illustrated. In FIG. 8 (b), the sharpness is calculated from the images photographed in the Z-axis direction at the respective points P1 to P6.

The peak point having the highest value among the calculated sharpnesses in the total of n images taken from Z1 to Zn is determined as the sharpest Z-axis focus. The Z-axis points indicated in bold in Figure 8 (b) indicate the sharpest Z-axis focus calculated through sharpness.

Referring to FIG. 8 (c), an example of predicting the three-dimensional posture of a mounted slide using the sharpest Z-axis focus calculated at six points through the sharpness calculation step.

In the sample image photographing step, an image is taken as a single-layer or multi-layer image from a specimen slide according to the photographing range calculated through the photographing planning step. Since only the calculated area is photographed through the photographing planning step, the photographing of unnecessary areas in the slide can be minimized and the inspection time can be reduced accordingly. The captured image is temporarily stored in the control PC.

In the sample determination step, a single layer or a multi-layer image photographed through a sample imaging step is inputted, and weight values of the object classification model or the object detection model learned through the learning step and the verification step are used to determine the tuberculosis voice or benignity .

The sample determination step includes two methods of determining through a deep learning model trained with a deep learning model trained with a single layer data set, or a deep learning model trained with a multi-layer data set.

When the sample is judged, the object classification model and the object detection model learned through the learning step are used.

First, in the case of the object classification model, all objects are detected from a single-layer or multi-layered image through a sample imaging step. In the single layer mode, all objects in the image are detected and a patch image is generated. In the case of the multi-layer mode, all objects are detected at all layers, and the same object is grouped using the object position on the image to generate a three-dimensional patch image. Objects detected with a single layer or multi-layer patch image are classified as objects other than Mycobacterium tuberculosis and M. tuberculosis, and the number of objects classified as Mycobacterium tuberculosis is counted to determine whether the sample slides are negative or positive.

Next, in the case of the object detection model, images are taken in single layer or multiple layers taken through a sample imaging step and all of the mycobacteria are detected from the images. In the case of the single layer mode, the region and location of mycobacteria in the tomographic image are detected. In the case of the multi-layer mode, the Mycobacterium tuberculosis is detected in all the layers and the mycobacteria that are overlapped in position are judged to be the same. The number of tubercle bacilli extracted from the monolayer or multilayer images is counted to judge whether the sample slide is negative or positive.

At this time, the object classification model and the detection model are determined according to WHO's sputum score examination guidelines at the time of determination of a voice or a positive of a sample.

In the test result presentation stage, the test result from the tuberculosis judgment module is presented to the user by displaying the result on the monitor.

While the present invention has been particularly shown and described with reference to exemplary embodiments thereof, it is to be understood that the invention is not limited to the disclosed exemplary embodiments, It belongs to the scope of right.

11: Slide selector 12: Tool selector
13: focus shifter 14: size adjuster

Claims (4)

A step of acquiring an image by tomography and a multi-layer photographing from a sputum slide prepared for training by using a radiographic apparatus including a control PC, a three-axis (XYZ) scale, an optical system, a fluorescent illumination system and a fluorescent camera, A training image transmission step of continuously capturing and transmitting a large number of slides by separating the image transmission steps so that one image is automatically transmitted to the collection server using a file transfer protocol, Collecting training data by using a video and a training data collection program stored in a server through the training video transmission step and processing the data collected through the training data collection step into a learning- A data set is generated, which is an object classification model or object detection model. Training data generating step that separates the data set in the form of a suitable single-layer or multi-layer form in;
Learning the data set generated through the training data generation step to the object classification model and the object detection model;
A verification step of verifying accuracy using verification software based on the deep learning model learned through the learning step;
Before performing the actual blood test using the learned and verified deep learning model through the learning step and the verification step, the slides mounted on the table are corrected before the test images are taken from the sample slides, A step of photographing, a step of taking a single-layer or a multi-layer image of an image from a sample slide according to a photographing range calculated through the photographing planning step, a step of receiving a single-layer or multi- And a sample judgment step of judging whether a tuberculosis voice is positive or not using the weight value of the object classification model or the object detection model learned through the verification step; And
And displaying the result of the examination through the tuberculosis judging step to a user by displaying it on a monitor. The method according to claim 1,
The photographing planning step is a step of designating at least four or more points along the edge of the smearing area excluding the bar code attaching area of the slide, taking an image at the corresponding point, and photographing the photographing plan for moving the Z axis at least ten times A photographing step;
A sharpness calculation step of determining a peak point having the highest sharpness at each point as a Z-axis position with the best focus by using an image photographed in a plurality of layers at a predetermined point through the imaging plan imaging step;
Deriving a three-dimensional profile of the slide using the peak point position information of the Z-axis calculated through the sharpness calculation step, and calculating a shooting range at the actual inspection using the derived slide orientation information Wherein the deep-run-based tuberculosis test method comprises the steps of: delete delete

Images