KR20230171799A - Cervical image diagnosis preprocessing system - Google Patents

Abstract

The present invention relates to a cervical image reading pre-processing system that replaces cervical images that are not suitable for reading with readable images, and a cervical image classification model that classifies input cervical images as suitable or unsuitable for reading. class; an image restoration model that replaces and restores unreadable images input from the cervix image classification model with readable cervix images; and a cervix reading model that reads cervical lesions for the cervix images input from the cervix image classification model and the image restoration model and outputs the results.

Description

Cervical image diagnosis preprocessing system

The present invention relates to an automatic interpretation system for cervical images, and in particular, to a cervical image interpretation pre-processing system that replaces cervical images that are not suitable for interpretation with readable images.

One of the methods of diagnosing the cervix is cervix imaging, which involves taking pictures of the cervix with a camera, observing them, and diagnosing the risk. As an advanced form, systems that automatically diagnose the risk of cervical images captured with a camera using a deep learning-based neural network (artificial neural network) are also being introduced.

In the above-described methods, various types of reading interference elements exist in the cervix image obtained through the camera. Interference with reading may occur depending on camera performance, differences in shooting method, or photographer's intention. Examples of images that may interfere with reading include images that are too bright, blurred images, images using a green filter, and images of the cervix printed on a monitor or paper.

Even in actual clinical trials, it is common for specialists to treat images with elements that interfere with reading (hereinafter referred to as unreadable images) as unreadable images and not proceed with reading them. Even in systems that automatically read (diagnose) the risk of the cervix using an artificial neural network model, normal reading results cannot be obtained for video images that have elements that interfere with reading.

Therefore, in order to improve the reliability of interpretation of cervical images, it is necessary to remove images that interfere with interpretation in advance or replace them with readable images.

Republic of Korea Patent Publication No. 10-2316557

Accordingly, the present invention is an invention created in response to the above-mentioned need, and the main purpose of the present invention is to classify in advance cervix images with reading interference elements and replace them with readable images to improve the reliability of the reading results. The goal is to provide a cervical image interpretation preprocessing system that can be improved.

The cervical image reading pre-processing system according to an embodiment of the present invention to achieve the above-described object is a system installed and executed in the memory of a computer system,

A cervix image classification model that classifies the input cervix image into an image suitable for reading or an image unsuitable for reading;

an image restoration model that replaces and restores unreadable images input from the cervix image classification model with readable cervix images;

and a cervix reading model that reads cervical lesions for the cervix images input from the cervix image classification model and the image restoration model and outputs the results.

Furthermore, in the cervical image reading pre-processing system including the above-described configuration, the cervical image classification model is,

Learning is carried out by sharing the weights of the pre-trained cervix interpretation model, including the cervix image with a green filter applied, the cervix image with blur reflected, the cervix image displayed and captured on a monitor or paper, and the threshold value. It is characterized by classifying at least one of the cervix images with the brightness above or above as an unreadable image.

In addition, the image restoration model is,

An encoder pre-trained to extract features of a cervical image suitable for reading from an unreadable image input from the cervical image classification model during inference;

and a decoder unit that receives a content map containing the features of the cervical image extracted from the encoder, replaces it with a cervical image suitable for reading, and outputs the reconstruction.

In addition, the encoder of the cervical image reading preprocessing system described above is,

It is characterized by pre-learning by sharing the weights of the pre-trained cervical image classification model,

In the learning mode, the encoder monitors the features of the cervical image suitable for reading, the characteristics of the cervical image to which a green filter is applied, the characteristics of the cervical image to which shaking is reflected (blur), for the unreadable image input from the cervical image classification model in the learning mode. It is characterized by outputting a content map containing any one of the features of the cervical image displayed on paper and captured, and the characteristics of the cervical image with brightness above a threshold.

According to the means for solving the technical problem described above, the cervix image reading pre-processing system according to an embodiment of the present invention classifies the cervix image images in which reading interference elements exist, replaces them with clear readable images, and reads the cervix, resulting in This provides the effect of improving the reliability of interpretation of cervical images.

1 is an exemplary configuration diagram of a cervical image reading pre-processing system according to an embodiment of the present invention.
Figure 2 is an example configuration of the image restoration model in Figure 1.
FIG. 3 is a diagram for further explaining the operation during learning of the image restoration model shown in FIG. 1.
FIG. 4 is a diagram for further explaining the operation during inference of the image restoration model shown in FIG. 1.

The detailed description of the present invention described below refers to the accompanying drawings, which show by way of example specific embodiments in which the present invention may be practiced to make clear the objectives, technical solutions and advantages of the present invention. These embodiments are described in sufficient detail to enable those skilled in the art to practice the invention.

In addition, the term 'learning' throughout the detailed description and claims of the present invention refers to performing machine learning (deep learning) according to procedures, and is not intended to refer to mental operations such as human educational activities. will be understandable. Also, throughout the detailed description and claims of the present invention, the word 'comprise' and its variations are not intended to exclude other technical features, attachments, components or steps. Other objects, advantages and features of the invention will appear to those skilled in the art, partly from this description and partly from practice of the invention. The examples and drawings below are provided by way of example and are not intended to limit the invention. Moreover, the present invention encompasses all possible combinations of the embodiments shown herein. It should be understood that the various embodiments of the invention are different from one another but are not necessarily mutually exclusive. For example, features and features described herein with respect to one embodiment may be implemented in other embodiments without departing from the spirit and scope of the invention. Additionally, it should be understood that the location or arrangement of individual components within each disclosed embodiment may be changed without departing from the spirit and scope of the invention. Accordingly, the detailed description that follows is not intended to be taken in a limiting sense, and the scope of the invention is limited only by the appended claims, together with all equivalents to what those claims assert, if properly described. Similar reference numbers in the drawings refer to identical or similar functions across various aspects.

In this specification, unless otherwise indicated or clearly contradictory to the context, items referred to in the singular include plural unless the context otherwise requires. Additionally, in describing the present invention, if it is determined that a detailed description of a related known configuration or function may obscure the gist of the present invention, the detailed description will be omitted.

For reference, the CNN (convolutional neural network) model, one of the machine learning models used in the present invention, can be briefly described as a multi-layered artificial neural network. In other words, it is expressed as a deep neural network (deep neural network) in the sense that it is a network with a deep structure. It automatically learns the characteristics of each image by learning a large amount of data in a structure consisting of a multi-layer network, Through this, the network is trained in a way that minimizes the error of the objective function. At first glance, this can be expressed as a connection between nerve cells in the human brain, and thus is established as a representative example of artificial intelligence. In particular, CNN is a model suitable for classifying two-dimensional images such as images. It uses a convolution layer that creates a feature map by using multiple filters for each area of the image and reduces the size of the feature map. By repeating the pooling layer (sub-sampling layer), which allows extracting features that are invariant to changes in position or rotation, from low-level features such as points, lines, and surfaces to complex and meaningful high-level features. Features at various levels can be extracted, and ultimately, there is an advantage in building a classification model with higher accuracy by using the features extracted through a fully-connected layer as input to an existing model. .

Hereinafter, in order to enable those skilled in the art to easily practice the present invention, preferred embodiments of the present invention will be described in detail with reference to the attached drawings.

FIG. 1 illustrates a configuration diagram of a cervical image reading pre-processing system according to an embodiment of the present invention, and FIG. 2 illustrates a configuration diagram of the image restoration model 150 in FIG. 1 .

The cervical image reading pre-processing system according to an embodiment of the present invention can be implemented as an embedded system in a system previously applied and registered by the applicant, for example, a cervical image acquisition device, as well as in the computer system of a specialist (diagnostician). It can be implemented as a collection of program data (application program) that is installed and executable by the main processor of a computer system.

Figure 1 shows a cervical image reading pre-processing system installed in a specialist's computer system. The cervical image reading pre-processing system includes,

It includes a cervix image classification model 100 that classifies the input cervix image into a suitable or unreadable image.

An image suitable for reading is an image suitable for reading a cervical image and is assumed to be an image in a clear and fine state. In contrast, images that are unsuitable for interpretation are assumed to be a cervical image with a green filter applied, a cervical image with blur, a cervical image captured while displayed on a monitor or paper, and a cervical image with brightness above the threshold. These unreadable images are only examples, and all images designated as unreadable by a specialist may be considered unreadable images.

The cervical image interpretation pre-processing system includes an image restoration model (150) that replaces and restores an image that is unsuitable for interpretation input from the cervical image classification model (100) with a cervical image that is suitable for interpretation;

It includes a cervix reading model 200 that reads cervical lesions for the cervix images input from the cervix image classification model 100 and the image restoration model 150 and outputs the results.

Each of the above-described cervical image classification model (100), image restoration model (150), and cervical reading model (200) is a deep neural network with a learning mode (transfer learning) and an inference mode, especially a convolution neural network: CNN), and the encoder and decoder units that make up the image restoration model 150, which will be described later, can also be implemented using a convolutional neural network.

For reference, the cervix image classification model 100 is trained by sharing the weights of the pre-trained cervix interpretation model 200, but the cervix image with a green filter applied and the cervix with blurred cervix are used for learning. Training is conducted to classify at least one of the images, cervical images displayed on a monitor or paper and captured, and cervical images with brightness above the threshold as unreadable images. Accordingly, the cervical image classification model 100 includes a cervical image with a green filter applied in the inference stage, a cervical image with blur reflected, a cervical image captured and displayed on a monitor or paper, and a cervical image with brightness above the threshold. One or more of the images may be classified as unreadable images.

Meanwhile, the image restoration model 150 uses an encoder ( 152) Wow,

It includes a decoder unit 153 that receives a content map containing the characteristics of the cervical image extracted from the encoder 152, replaces it with a cervical image suitable for reading, and outputs the reconstructed image. Restoration here means restoring the content map with the characteristics of the cervical image back to the image level (an image with the original image size and clear and fine features). Considering the cervical image input to the system for interpretation, The image restored in the decoder 154 may be regarded as an image that ultimately replaces the input image.

As shown in FIG. 1, the encoder 152 also performs pre-training by sharing the weights of the pre-trained cervical image classification model 100, so that during inference, the green filter-applied cervical image is used as shown in FIG. 2. One or more features can be extracted among features, features of a cervical image reflected by blur (blur), features of a cervical image captured and displayed on a monitor or paper, and features of a cervical image with brightness above the threshold (Feature Maps). there is.

Meanwhile, as shown in FIG. 3, the decoder unit 153 includes a decoder 153-1 that restores a cervical image that has brightness characteristics above the threshold but also has clear and fine characteristics, and a green filter applied feature. A decoder (153-2) that restores a cervical image that has the characteristics of a cervical image that is displayed on a monitor or paper and has clear and fine features, but also restores a cervical image that has clear and fine features (153-3) and a decoder (153-4) that restores a cervical image with blurred features and clear and fine features, and restores a cervical image with clear and fine features, that is, features suitable for reading. It includes a decoder (153-5) that does.

The cervix interpretation model 200 includes a machine learning model that learns the characteristics of learning data set by a specialist, etc. in a learning mode, and in an inference mode, the machine learning model is preprocessed at the front end and restored or classified into an appropriate image. For cervical images, readout information regarding the occurrence of cervical cancer is generated and output through the user interface screen.

Hereinafter, the operation of the cervix image reading pre-processing system shown in FIG. 1 will be further explained with reference to the accompanying drawings.

FIG. 3 is a diagram for further explaining the operation of the image restoration model 150 shown in FIG. 1 during learning, and FIG. 4 illustrates the operation of the image restoration model 150 shown in FIG. 1 during inference. A drawing is shown for further explanation.

First, in an embodiment of the present invention, a method commonly used when learning a deep learning model is used in the initial stage of learning of another lip learning model by sharing the pre-trained weight of the backbone network that plays the role of feature extraction. That is, the cervix learning data of the cervix reading model 200 (cervix image data with green filter applied, cervix image data with blur reflected, cervix image data captured and displayed on a monitor or paper, brightness above the threshold) When learning the cervical image classification model (100), pre-learned weights for cervical image data (cervical image data with clear and fine features) are shared and learning is performed. In addition, when learning the encoder 152 of the image restoration model 150, the pre-trained weights from the cervical image classification model 100 are shared and learned.

In this way, in the learning mode, if the convolutional neural network models corresponding to the preprocessing part of the cervix image interpretation preprocessing system share pre-trained weights from other models, for example, the cervix image classification model 100 may generate an unsuitable cervix. As images, cervical images with a green filter applied, cervical images with blur reflected, cervical images captured while displayed on a monitor or paper, and cervical images with brightness above the threshold can be better classified as inappropriate images. In addition, the encoder 152 in the image restoration model 150 is used to detect inappropriate features of the cervical image, for example, characteristics of the cervical image to which a green filter is applied, characteristics of the cervical image to which shaking is reflected (blur), and It is possible to better extract the features of the displayed and captured cervical image, the features of the cervical image with brightness above the threshold, and the clear and fine cervical image features suitable for reading. Cervical images of clear and fine that are suitable for reading are also included in the data set for training the encoder 152.

However, the encoder 152 must be pre-trained so that features of the cervical image suitable for reading (clear and fine) are extracted for any unreadable image input from the cervical image classification model 100 during inference.

For this purpose, the decoder unit 154, which receives the content map containing the features of the cervical image extracted from the encoder 152 and restores and outputs it as a readable cervical image, provides brightness characteristics above a threshold and clear A decoder (153-1) that inputs a content map with and fine features and restores a cervical image (a) with brightness features above a threshold and clear and fine features, and a decoder (153-1) that has green filter applied features but also has clear and fine features. It has a decoder 153-2 that restores the cervix image (b), and a decoder that restores a cervix image (c) that has the characteristics of a cervix image displayed and captured on a monitor or paper, but also has clear and fine features. (153-3), a decoder (153-4) that restores a cervical image (d) with blurred features and clear and fine features, and a cervical image (e) with only clear and fine features. It includes a decoder 153-5 that restores.

As described above, the decoder unit 153, which includes five decoders, generates images containing the characteristics of the four inappropriate cervical images based on clear and fine images in the learning mode, and performs the learning process on the generated images. When set in parallel in a batch, each of the generated images is input to the combined model of the encoder 152 and the decoder 153-1, 2, 3, 4, and each decoder 153-1, 2 For the output images from ,3,4), learning is conducted in a direction that minimizes the loss with the input images.

Once this learning is completed, at the time of inference, the encoder 152 generates clear and fine features no matter which cervix image including clear and fine, brightness, green filter, and blur features is input, as shown in FIG. Outputs a content map with . Accordingly, the decoder 153-5 receives a content map containing the features (clear and fine) of the cervical image extracted from the encoder 152, restores a clear cervical image, and creates a cervical reading model (200 ) is output.

That is, the image restoration model 150 of the cervical image reading pre-processing system according to an embodiment of the present invention has characteristics of a cervical image suitable for reading for an image unsuitable for reading input from the cervical image classification model 100 at the time of inference. An encoder 152 pre-trained to be extracted, and a decoder unit 153 that receives a content map containing the characteristics of the cervical image extracted from the encoder 152 and restores and outputs a cervical image suitable for reading. By including it, it provides the effect of improving the reliability of the interpretation of the cervical image by replacing the cervical imaging image in which there are elements interfering with the interpretation with a clear image that can be simply read.

The above has been described with reference to the embodiments shown in the drawings, but these are merely examples, and those skilled in the art will understand that various modifications and other equivalent embodiments are possible. Therefore, the true technical protection scope of the present invention should be determined only by the appended claims.

Claims (5)

A cervix image classification model that classifies the input cervix image into a suitable or unreadable image;
an image restoration model that replaces and restores unreadable images input from the cervix image classification model with readable cervix images;
A cervical image reading pre-processing system comprising: a cervical reading model that reads cervical lesions for the cervical images input from the cervical image classification model and the image restoration model and outputs the results. The method of claim 1, wherein the cervical image classification model is:
Learning is carried out by sharing the weights of the pre-trained cervix interpretation model, including the cervix image with a green filter applied, the cervix image with blur reflected, the cervix image displayed and captured on a monitor or paper, and the threshold value. A pre-processing system for interpreting cervical images, characterized in that classifying at least one of the cervical images with a brightness above or equal to an unreadable image. The method of claim 1 or claim 2, wherein the image restoration model is:
An encoder pre-trained to extract features of a cervical image suitable for reading from an unreadable image input from the cervical image classification model during inference;
A pre-processing system for interpreting cervical images comprising a decoder unit that receives a content map containing features of the cervical image extracted from the encoder and restores the content by replacing it with a cervical image suitable for reading. The method of claim 3, wherein the encoder,
A pre-processing system for interpreting cervical images, characterized in that pre-learning is performed by sharing the weights of the pre-trained cervical image classification model. The method of claim 3, wherein the encoder,
In the learning mode, the characteristics of the cervical image suitable for reading for the unreadable image input from the cervical image classification model, the characteristics of the cervical image with a green filter applied, and the characteristics of the cervical image with blur reflected, are displayed on a monitor or paper. A pre-processing system for reading cervical images, characterized in that it outputs a content map containing any one of the characteristics of the displayed and captured cervical image and the characteristics of the cervical image having a brightness above a threshold.

Images