RU2717911C1 - Method of training a convolution neural network to carry out teleroentgenogram markings in a direct and lateral projections - Google Patents

Abstract

FIELD: computer equipment.

SUBSTANCE: invention relates to computer engineering and medicine. Method comprises steps of: creating a database (DB) with pre-analyzed and marked cephalometric points and their coordinates on teleroentgenograms in direct and lateral projections (TRGD and TRGL); based on the prepared DB, training at least one convolutional neural network (CNN) is marked with cephalometric points of the teleroentgenograms in the direct and lateral projections (TRGD and TRGL), wherein: the CNN input is fed with images from the DB, with points N marked thereon, and multilayer mask M for said points; CNN analyzes multilayer masks M and places them on N masks M_1 …M_N by number of marked points on input images; finding connectivity components for each layer of masks, wherein each component is predicted coordinates of points on images position; comparing predicted coordinates with coordinates of marked points on input images from DB and calculating root-mean-square deviation; trained at least one CNN is used for further marking with cephalometric points of teleroentgenogram in direct and lateral projections.

EFFECT: provision for training of a convolution neural network to carry out marking of teleroentgenograms in direct and lateral projections.

5 cl, 2 dwg

Description

FIELD OF TECHNOLOGY

The present technical solution relates to the field of computer engineering and medicine, in particular, to a method for training a convolutional neural network to carry out marking of tele-roentgenograms in front and side projections.

BACKGROUND

Anthropometric analysis of the facial section of the skull and soft tissues of the face is widely used in orthodontics and maxillofacial surgery, being one of the main diagnostic tools for diagnosing and choosing a treatment plan [1] [2] [3]. In addition, anthropometric approaches to the study of soft and hard tissues are used in traumatology, anthropology, archeology and forensic medicine [4].

At the same time, tele-roentgenography (TRH) is an important research method in orthodontics and maxillofacial surgery, which allows obtaining the necessary diagnostic information about the structure of the brain and facial parts of the skull for planning treatment. There are many methods of analysis that require the placement of cephalometric points on a teleradiogram with subsequent processing, which takes a considerable amount of time from the doctor.

The prior art solutions that describe various methods for placing cephalometric points for anthropometric analysis, US20180189421A1, US10117727B2.

However, prior art solutions have limited functionality. In particular, they do not use artificial neural networks for marking tele-roentgenograms in direct and lateral projections.

Currently, artificial neural networks are an important tool for solving many applied problems. They have already allowed to cope with a number of difficult problems and promise the creation of new inventions that can solve problems that so far only human can do. Artificial neural networks, as well as biological ones, are systems consisting of a huge number of functioning neuron processors, each of which performs some small amount of work assigned to it, while having a large number of connections with the rest, which characterizes power of network computing.

The use of artificial intelligence can improve the research method and greatly simplify the work of the doctor. In the claimed solution, a configuration of an artificial neural network (ANN) was developed, which makes it possible to place cephalometric points on tele-roentgenograms in direct and lateral projections with high accuracy. The ANN error in the claimed solution was only 0.5%.

The proposed approach requires 2-5 times faster than the traditional “manual” method of arranging cephalometric points, depending on the number of points and the complexity of the cephalometric analysis.

SUMMARY OF THE INVENTION

This technical solution is aimed at eliminating the disadvantages inherent in existing solutions from the prior art.

The technical problem to which the claimed technical solution is directed is the creation of a computer-implemented method for training the convolutional neural network to carry out the marking of X-ray patterns in front and side projections. Additional embodiments of the present invention are presented in the dependent claims.

The technical result of this decision is to provide training for the convolutional neural network to carry out the marking of tele-roentgenograms in the front and side projections

An additional technical result achieved in solving the above problem is to improve the quality and accuracy of the marking of teleretgenograms in the direct and lateral projections of the patient by a trained convolutional neural network.

In a preferred embodiment, a computer-implemented method for training a convolutional neural network (SNA) for marking tele-roentgenograms in front and side projections, comprising the steps in which:

- using a computing device, create a database (DB) with pre-analyzed and marked cephalometric points and their coordinates on tele-roentgenograms in the front and side projections (TRGP and TRGB);

- on the basis of the prepared database, they teach at least one SNA to mark the tele-roentgenograms in frontal and lateral projections (TRHP and TRGB) with cephalometric points, while during training:

• images from the database with points N marked on them and multilayer probability maps (masks) M for these points are fed to the SNA input;

• SNA analyzes multilayer masks M and sticks them onto N masks M_1 ... M_N according to the number of marked points on the input images;

• find connected components for each layer of masks, with each component being the predicted coordinates of the points in the images;

• compare the predicted coordinates with the coordinates of the marked points on the input images from the database and calculate the standard deviation;

- apply the trained at least one SNA for subsequent marking by cephalometric points of the teleradiogram in direct and lateral projections.

In a particular embodiment, the tele-roentgenograms in the database are saved as an xml file.

In another particular embodiment, the pre-processing of images of teleradiographs is carried out before they are submitted to the SNA by augmentation.

In another particular embodiment, connectivity components are found using the skimage.morphology.label function.

In another particular embodiment, cephalometric points are placed on such localizations as soft tissue, bone and dental tissues.

DESCRIPTION OF DRAWINGS

The implementation of the invention will be described hereinafter in accordance with the accompanying drawings, which are presented to illustrate the essence of the invention and in no way limit the scope of the invention. The following drawings are attached to the application:

FIG. 1 illustrates a diagram of a claimed solution;

FIG. 2 illustrates an example of a general circuit of a computer device.

DETAILED DESCRIPTION OF THE INVENTION

In the following detailed description of the implementation of the invention, numerous implementation details are provided to provide a clear understanding of the present invention. However, to a person skilled in the art, it will be obvious how the present invention can be used, both with and without implementation details. In other cases, well-known methods, procedures, and components have not been described in detail so as not to obscure the features of the present invention.

In addition, from the foregoing it will be clear that the invention is not limited to the above implementation. Numerous possible modifications, changes, variations and replacements preserving the essence and form of the present invention will be apparent to those skilled in the subject field.

The present invention is directed to providing a computer-implemented method for training a convolutional neural network, with the subsequent use of this trained SNA for marking tele-roentgenograms in front and side projections.

An artificial neural network (hereinafter - ANN) is a computational or logical circuit constructed from homogeneous processor elements, which are simplified functional models of neurons.

The convolutional neural network (CNN) is a special architecture of artificial neural networks aimed at efficient pattern recognition, and is part of deep learning technologies.

The number of layers of an artificial neural network is not limited by implementation options. As a trained neural network, a fully connected neural network, or a convolutional neural network, or a recurrent neural network, or a combination thereof, can be used, not limited to.

Layer of a neural network (English layer) - a set of neurons of a network, united by the features of their functioning.

As shown in FIG. 1, the claimed computer-implemented training of the convolutional neural network to carry out the marking of tele-roentgenograms in the front and side projections (100) is implemented as follows:

At step (101), using a computing device, a database (DB) is created with previously analyzed and marked cephalometric points and their coordinates on the tele-roentgenograms in the front and side projections (TRGP and TRGB).

Next, at step (102), on the basis of the prepared database, at least one SNA is trained to be marked with cephalometric points of the teleradiograms in the direct and lateral projections (TRGP and TRGB), while during training:

At step (103), images from the database with the points N marked on them and multilayer masks M for these points are fed to the SNA input.

At step (104), the SNA analyzes the multi-layer masks M and sticks them onto N masks M_1 ... M_N according to the number of marked points on the input images. Next, at step (105), connectivity components are found for each layer of masks, with each component being the predicted coordinates of the points in the images.

Then, at step (106), the predicted coordinates are compared with the coordinates of the marked points on the input images from the database and the standard deviation is calculated. A trained at least one SNA is used for subsequent marking by cephalometric points of a teleradiogram in a direct and lateral projection (107).

In the claimed solution, the tele-roentgenograms in the database are saved as an xml file.

SNA can be trained on a set of objects, which, for example, are images.

For training the SNA on TRGB, 64 primary (primary) and 98 secondary (additional) points were used. To teach the SNA on TRHP, 49 primary (primary) and 12 secondary (additional) points were used. The proposed method for teaching the SNA does not have a limit on the number of cephalometric points. For the initial training of the ANN, 130 images of each type were taken (TRGP and TRGB), and the remaining 25 were left for validation. Pictures were obtained from various sources, thereby ensuring the adaptability of the trained algorithm to new data.

On the basis of the initially trained SNA, a database is created with marked cephalometric points and their coordinates on the tele-roentgenograms in the front and side projections of at least 1000 tele-roentgenograms of each type.

Based on the prepared databases, SNAs are trained for dotting using a method based on multiclass image segmentation.

At the input of the SNA, images from the database are supplied, with points N marked on them, and multilayer masks M for these points. Masks are constructed using the OpenPose method [5], in part, the density of the two-dimensional normal distribution is taken with the expectation - the point itself. The covariance matrix is selected so that the points do not overlap much when drawn, but are quite noticeable.

This density is normalized so that there is one at the maximum, so we get an analog of probability. Then the probabilities are selected, above which the masks are assigned 1, and below - 0. When generating the 512x512 masks, diag was taken as the covariance matrix (4, 4). This is a 2x2 diagonal matrix with fours on the diagonal, i.e. mtaritsa:

(4 0)

(0 4).

After which the SNA learns to predict these masks. Then, all predictions that are greater than or equal to 0.5 are made 1, and the rest are zeroed out. Using the skimage.morphology.label function, connectivity components are found for each layer of the mask. Next, select the largest, which does not consist of 0, and its center is located. This center is the position of the desired point.

The following is the architecture of the SNA.

The implementation of the SNA is entirely based on Pytorch. This is the most flexible and convenient framework for working with neural networks.

To arrange the primary and secondary points, neural networks were used with the same architecture, accurate to the number of classes. In each case, two neural networks were used at once. The first network was SE_ResNeXt-50 [6], but FPN modified for multiclass segmentation was used to increase the accuracy of predictions [7]. This decision is due to the fact that there are a lot of classes, and networks whose decoders are based on the Unet architecture are intended initially for binary segmentation. Unet also works with a small number of classes. The SNS encoder was pre-trained on the ImageNet dataset.

Initially, images and masks had a size of 384x384. In training, a random 256x256 window was cut out of them, since such a network architecture implies strong augmentations. The following random augmentations were also used:

1. Random change in brightness;

2. Distortion over the grid;

3. Random change in contrast;

4. Random change in gamma;

5. Random zoom (from 0.8 to 1.2);

6. Random rotation of 8 degrees.

The borders were filled in black. Next, the data was collected in a series of 64 pictures and sent for training.

For the SNA, regression was performed using 240x240-sized images, which were trimmed to 224x224 by a random window during training, which is the standard input image size for networks of the ResNet type. The augmentations were as follows:

1. Random blur;

2. Random change in brightness;

3. Random zoom (from 0.95 to 1.11);

4. Random rotation of 10 degrees.

Next, the images were fed to the network input in series of 64.

Workout ANN.

As a SNA metric for segmentation, the Intersection Over Union (IOU) metric was used for each channel and averaged, and BCE * 0.3 + Jaccard * 0.7 was taken as a loss function. This ratio is most often used in segmentation tasks.

The training took place in 3 stages. In the first two, Adam was taken as an optimizer. On the first, the encoder was frozen, and only the decoder studied. LR was initially 0.003 and decreased by 2 times, if 5 eras there was no improvement in the metric. If 11 epochs did not improve, they proceeded to the second stage. On it, all weights were thawed and the entire network studied, according to the same principle, but already with LR = 0.001. Both the decline and the transition occurred during the 6th and 16th eras, respectively. At stage 3, the optimizer was RMSProp, and LR changed in cosine form from 0.001 to 0.000001. At the same time, its period was 24 epochs. SNA went through three such cycles.

The accuracy indicators of the positioning of cephalometric points by trained SNAs were as follows; mean square errors (MSE) or mean squared deviation (MSD) are given: 0.0000752 - 0.000273, Avg misses: 1.4-11.75, old MSE: 0.00148-0.00155.

Training SE_ResNeXt-50 took place in two stages.

Images 224x224 in size are fed into the trained neural network SE-ResNeXt-50.

The network extracts from each image a vector of 2048 attributes. Extracted features pass through a design of four Fully-Connected layer using Batch Normalization, ReLU and Dropout activation functions. At the output, we get a vector of 124 predicted coordinates belonging to 62 cephalometric points.

Retraining of the SNA is a phenomenon when the constructed model approximates the examples from the training set well, but relatively poorly works on the examples from the test sample that did not participate in the training. This is due to the fact that when constructing the SNA in the learning process in the sample, some random patterns are found that are absent in the general population.

To overcome this problem, Dropout, proposed by Srivastava N [8], was used in the last layers of the SNA. The idea of this method is to randomly zero out the output of a fraction of neurons in front of a fully connected layer when training a neural network. When obtaining predictions for a test set of images, instead of zeroing out a random fraction of the outputs of neurons, all outputs are halved. This fairly simple algorithm has worked well for fighting retraining.

The resulting model is trained by the method of back propagation of error. More information about the training mode will be written below.

Preparation of source images. To prevent possible retraining processes and to improve the generalizing ability of the model, strong augmentations were applied to the original images. Augmentations are intentional distortions and transformations of the original images in order to artificially increase the size of the training sample, which is a strong regularization for the model. That is, the SNA, viewing more different images during training, is better generalized and less retrained.

At first, all images were reduced to a size of 256x256 pixels with preservation of the proportion of the image so as not to distort the shape of the skull of patients. To keep the proportions, the images were supplemented to a square gray fill. Then, all the images went through preliminary random transformations:

1. Random Image Blur

2. Random brightness correction

3. Reflection of the image relative to the vertical axis

4. Changing the image scale by the number of times selected equally likely from a segment from 0.6 to 1.4

5. Rotate by a random angle from -30 ° to + 30 °.

After that, a random window measuring 224x224 pixels was cut out of the images and the coordinates of the cephalometric points were calculated accordingly so that they corresponded to the new images.

After the transformations, the images were collected in large series and submitted to the ANN for its training.

SNA training. SNA was trained to optimize the MSE (Mean Squared Error) loss function, which is standard for key point detection tasks, which is defined as follows:

, где

это предсказанные координаты цефалометрических точек, а

это их настоящие значения.

where

these are the predicted coordinates of the cephalometric points, and

these are their true meanings.

Adam was used as an optimizer [9].

Adam (from the English adaptive moment estimation) is an optimization algorithm that is an extension of the stochastic gradient descent method.

SNA training took place in two stages. At the first stage, the trained SE-ResNeXt-50 “froze”, and only the last four fully connected layers were trained. At this stage, images of 800 pieces were sent to the neural network. Each epoch (under the epoch is meant 100 optimizer steps, on each of which the SNA looks at a series of 800 images), MSE was measured on a validation sample of 10 images to control the learning process of the SNA. In the stated solution, the following training regimen was used: the network trained with an initial parameter of the learning rate equal to 0.0001; every time the MSE error in a validation sample of 10 images did not decrease over 10 epochs, the parameter of the learning rate was halved and the training was continued. The procedure was repeated until the error in the validation sample ceased to improve.

At the second stage, the SNA trained as a whole (along with the “frozen” at the first stage SE-ResNeXt-50) with a similar training regimen. However, at this stage, the images were sent to the SNA in a series of 185 images. This is due to the need to propagate the error through the deep network of SE-ResNeXt-50, which imposes restrictions on the size of a single series associated with the RAM of the video cards. Accordingly, at this stage, the era refers to the 100 steps of the optimizer, at each of which the SNA views 185 images.

To place cephalometric points on a test data set, the central part of 224x224 pixels in size is cut out of the image and sent to the SNA. The coordinates predicted by the neural network are recalculated back to match the original image.

To clarify the predictions, the claimed solution used Test Time Augmentation: five windows with a size of 224x224 pixels in the center and at the corners are cut from the original image, and also these windows are reflected relative to the vertical axis. Received 10 images are sent to the SNA. The predicted coordinates are appropriately recalculated and averaged by the arithmetic mean.

3. Results

As a metric in the claimed solution, we took RMSE = sqrt (MSE) = .... For the primary and secondary points, different SNAs were used. Next, we give an assessment of the results:

Primary: MSE: 0.0000752

Without misses: 0.0000705

Avg misses: 1.4

old MSE: 0.00148

Secondary: MSE: 0.00150

Without misses: 0.000273

Avg misses: 11.75

old MSE: 0.00155.

All Together: MSE: 0,0009371160494

Without misses: 0.0001878370843

Avg misses: 13.15

old MSE: 0.001522345679.

In the claimed solution, the user can move and adjust the position of cephalometric points, program cephalometric calculations and save them as a separate file in .pdf format.

In FIG. 2, a general diagram of a computing device (200) that provides the data processing necessary for the implementation of the claimed solution will be presented.

In the General case, the device (200) contains such components as: one or more processors (201), at least one memory (202), data storage means (203), input / output interfaces (204), I / O means ( 205), networking tools (206).

The processor (201) of the device performs the basic computational operations necessary for the operation of the device (200) or the functionality of one or more of its components. The processor (201) executes the necessary computer-readable instructions contained in the random access memory (202).

Memory (202), as a rule, is made in the form of RAM and contains the necessary program logic that provides the required functionality.

The data storage medium (203) can be implemented as HDD, SSD disks, RAID raid, network storage, flash memory, optical information storage devices (CD, DVD, MD, Blue-Ray disks), etc. The tool (203) allows for long-term storage of various types of information, for example, the aforementioned files with user data sets, a database containing records of time intervals measured for each user, user identifiers, etc.

Interfaces (204) are standard tools for connecting and working with a computer device, for example, USB, RS232, RJ45, LPT, COM, HDMI, PS / 2, Lightning, FireWire, etc.

The choice of interfaces (204) depends on the specific design of the device (200), which can be a personal computer, mainframe, server cluster, thin client, smartphone, laptop, be a part of a bank terminal, ATM, etc.

As I / O data means (205), a mouse, joystick, display (touch screen), projector, touchpad, keyboard, trackball, light pen, speakers, microphone, etc. can be used.

Network communication tools (206) are selected from a device that provides network reception and data transfer, for example, an Ethernet card, WLAN / Wi-Fi module, Bluetooth module, BLE module, NFC module, IrDa, RFID module, GSM modem, etc. Using means (205), the organization of data exchange via a wired or wireless data channel is provided, for example, WAN, PAN, LAN (LAN), Intranet, Internet, WLAN, WMAN or GSM.

The components of the device (200) are interfaced via a common data bus (210).

In the present application materials, the preferred disclosure was presented the implementation of the claimed technical solution, which should not be used as limiting other, private embodiments of its implementation, which do not go beyond the requested scope of legal protection and are obvious to specialists in the relevant field of technology.

Literature

[1] Lin H. H., Chuang Y. F., Weng J. L., et al. Comparative validity and reproducibility study of various landmark-oriented reference planes in 3-dimensional computed tomographic analysis for patients receiving orthognathic surgery. Plos one. 2015; 10: e0117604.

[2] van Vlijmen O.J., Maal T., Berge S.J., et al. A comparison between 2D and 3D cephalometry on CBCT scans of human skulls. Int. J. Oral. Maxillofac. Surg. 2010; 39: 156-160.

[3] Farronato G., Garagiola U., Dominici A., et al. “Ten-point” 3D cephalometric analysis using low-dosage cone beam computed tomography. Prog Orthod. 2010; 11: 2-12.

[4] Kuehne H. et al. Hmdb51: A large video database for human motion recognition // High Performance Computing in Science and Engineering ‘12. - Springer, Berlin, Heidelberg, 2013 .-- S. 571-582.

[5] Zhe Cao, et al. OpenPose: Realtime Multi-Person 2D Pose Estimation using Part Affinity Fields. (Submitted on 18 Dec 2018 (v1), last revised 30 May 2019 (this version, v2)).

[6] Hu J., Shen L., Sun G. Squeeze-and-excitation networks // arXiv preprint arXiv: 1709.01507. - 2017 .-- T. 7.

[7] Selim S. Seferbekov, et al. Feature Pyramid Network for Multi-Class Land Segmentation. (Submitted on 9 Jun 2018 (v1), last revised 19 Jun 2018 (this version, v2)).

[8] Srivastava N. et al. Dropout: a simple way to prevent neural networks from overfitting // The Journal of Machine Learning Research. - 2014. - T. 15. - No. 1. - S. 1929-1958.

[9] Kingma D. P., Ba J. Adam: A method for stochastic optimization // arXiv preprint arXiv: 1412.6980. - 2014.

Claims (12)

1. A computer-implemented method for training a convolutional neural network to carry out the marking of tele-roentgenograms in front and side projections, containing the stages in which: - using a computing device, create a database (DB) with pre-analyzed and marked cephalometric points and their coordinates on tele-roentgenograms in the front and side projections (TRGP and TRGB); - on the basis of the prepared database, at least one convolutional neural network (SNA) is trained to be marked with cephalometric points of the teleradiograms in the front and side projections (TRGP and TRGB), while during training: • images from the database with points N marked on them and multilayer masks M for these points are fed to the SNA input; • SNA analyzes multilayer masks M and sticks them onto N masks M_1 ... M_N according to the number of marked points on the input images; • find connected components for each layer of masks, with each component being the predicted coordinates of the points in the images; • compare the predicted coordinates with the coordinates of the marked points on the input images from the database and calculate the standard deviation; - apply trained at least one SNA for subsequent marking by cephalometric points of the teleradiogram in front and side projections. 2. The method according to p. 1, characterized in that the roentgenograms in the database are saved as an xml file. 3. The method according to p. 1, characterized in that they carry out the preprocessing of images of teleradiographs before they are submitted to the SNA by augmentation method. 4. The method according to p. 1, characterized in that the connected components are found using the function skimage.morphology.label. 5. The method according to p. 1, characterized in that the cephalometric points are placed on such localizations as: soft tissue, bone and dental tissues.

Images