CN116012269A - Image processing device, model training device, and model training method - Google Patents

Abstract

The invention provides an image processing device, a model training device and a model training method, which adaptively perform structural sensing so as to improve the precision of segmentation processing of medical images. The model training device according to the embodiment is used for training a segmentation model of a medical image, and comprises: an acquisition unit that acquires a sample medical image including a tubular object segmentation result as training data; a center line extraction unit that extracts a center line of a tubular object in the sample medical image; an expansion unit configured to expand the center line extracted by the center line extraction unit to obtain an expanded center line; a loss function setting unit that sets a loss function using a pixel matrix including the expanded center line as a weight matrix; and a learning unit configured to learn the training data using the loss function set by the loss function setting unit, and to output a segmentation model for segmenting the tubular object in the medical image.

Description

Image processing device, model training device, and model training method

Technical Field

The embodiment relates to an image processing device, a model training device, and a model training method.

Background

Conventionally, an image acquisition device such as a CT device, an X-ray imaging device, or a magnetic resonance device is used to acquire a medical image of a subject, and data of a specific region such as a blood vessel or an organ is extracted by dividing the medical image, thereby facilitating a reading.

The existing segmentation algorithm for segmenting medical images is generally a conventional segmentation algorithm for segmenting medical images based on pixel threshold values, a deep learning algorithm or an algorithm combining the two algorithms.

Most algorithms based on deep learning use different labels (label) for segmenting only substantially the same geometry or different structures, such as segmenting only the intrapulmonary vessels when segmenting pulmonary vessels, without considering vessel thickness, extrapulmonary vessel trunks, etc.

However, when the existing segmentation methods are aimed at some complex structures, the segmentation accuracy cannot be ensured, for example, when the pulmonary blood vessels are segmented, the pulmonary blood vessels show different structural characteristics at different parts of the lung, for example, the blood vessels at the root part are thicker and the winding relationship among different blood vessels is complex, on the other hand, the blood vessels at the tip are thinner and the distance among different blood vessels is relatively far, and meanwhile, other structures in the lung image, such as a lung door, an atrium and the like, are not tubular structures and are large in size, and the morphological difference of the segmented objects can reduce the efficiency and the accuracy of the blood vessel segmentation algorithm.

For the segmentation of a complex tubular structure such as a pulmonary blood vessel, a method for extracting the central line of the pulmonary blood vessel and performing neural network learning and segmentation based on the central line is also proposed. Specifically, in this method, a medical image is first roughly segmented, a region of interest including a specified blood vessel is then segmented, a center line of the blood vessel in the region of interest is obtained, and then a two-dimensional slice data perpendicular to the center line at a plurality of sampling points on the center line is obtained along the center line, and for the two-dimensional slice data, lung blood vessel segmentation is performed using a prediction model obtained by neural network learning and a segmentation result is output (see patent document 1: cn107563983 b).

However, in such a technique of dividing a blood vessel by using a center line, the geometric structure of the divided object is not distinguished, and the efficiency and accuracy of the blood vessel dividing algorithm are not high by using only the center line for determining the sampling point.

Disclosure of Invention

The present invention has been made in view of the above-described problems, and an object of the present invention is to provide an image processing apparatus, a model training apparatus, and a model training method that can adaptively perform structural sensing and improve accuracy of segmentation processing of medical images.

One aspect of the present invention is a model training device for training a segmentation model of a medical image, comprising: an acquisition unit that acquires a sample medical image including a tubular object segmentation result as training data; a center line extraction unit that extracts a center line of a tubular object in the sample medical image; an expansion unit configured to expand the center line extracted by the center line extraction unit to obtain an expanded center line; a loss function setting unit that sets a loss function using a pixel matrix including the expanded center line as a weight matrix; and a learning unit configured to learn the training data using the loss function set by the loss function setting unit, and to output a segmentation model for segmenting the tubular object in the medical image.

Another aspect of the present invention is a model training method for training a segmentation model of a medical image, comprising: an acquisition step of acquiring a sample medical image including a tubular object segmentation result as training data; a center line extraction step of extracting a center line of a tubular object in the sample medical image; an expansion step of expanding the center line extracted in the center line extraction step to obtain an expanded center line; a loss function setting step of setting a loss function using a pixel matrix including the expanded center line as a weight matrix; and a learning step of learning the training data by using the loss function set in the loss function setting step, thereby outputting a segmentation model for segmenting the tubular object in the medical image.

According to the model training device and the model training method, the weight matrix including the expanded central line is applied to the loss function, so that the structure sensing can be performed adaptively, the root of the thicker blood vessel is arranged tightly, the topological structure of the blood vessel is correct, the connectivity is improved, and the segmentation precision of the segmentation model is improved.

Another aspect of the present invention is a model training apparatus for training a segmentation model of a medical image, comprising: an acquisition unit that acquires a sample medical image including a tubular object segmentation result as training data; a sub-region dividing section that identifies differences in geometric features in the sample medical image, and divides the sample medical image into a plurality of sub-regions based on the differences in geometric features; a loss function setting unit configured to set a loss function in which a weight matrix is formed by setting, for a certain subregion, a pixel matrix including the tubular object division result in the subregion as the weight matrix of the subregion, and assigning different values to pixels of the tubular object division result in different subregions, so that the loss function is calculated by using the weight matrix for each subregion; and a learning unit configured to learn the training data using the loss function set by the loss function setting unit, and to output a segmentation model for segmenting the tubular object in the medical image.

Another aspect of the present invention is a model training method for training a segmentation model of a medical image, comprising: an acquisition step of acquiring a sample medical image including a tubular object segmentation result as training data; a sub-region dividing step of identifying differences in geometric features in the sample medical image, dividing the sample medical image into a plurality of sub-regions based on the differences in geometric features; a loss function setting step of setting a loss function in which a weight matrix is formed by setting a pixel matrix including the tubular object division result in a certain subregion as a weight matrix of the subregion and assigning different values to pixels of the tubular object division result in different subregions, so that the weight matrix is calculated by using the weight matrix for each subregion; and a learning step of learning the training data by using the loss function set in the loss function setting step, thereby outputting a segmentation model for segmenting the tubular object in the medical image.

According to the technical scheme, the loss function can be set according to the characteristics of different areas, the structure sensing can be performed in a self-adaptive mode, and the segmentation accuracy of the segmentation model can be improved.

Another aspect of the present invention is an image processing apparatus, comprising: a medical image acquisition unit that acquires a medical image of a subject; and a segmentation unit configured to segment the medical image acquired by the medical image acquisition unit, using a segmentation model generated by a model training device.

According to the above-described aspects, a segmentation model capable of adaptively performing structural sensing can be applied, and thus the segmentation accuracy of medical image segmentation processing can be improved.

Drawings

Fig. 1 is a block diagram showing an example of the functional configuration of a model training device and an image processing device according to the first embodiment.

Fig. 2 is a schematic diagram showing a medical image and a segmentation truth value according to the first embodiment.

Fig. 3 is a schematic view showing the extracted center line and the center line after expansion in the first embodiment.

Fig. 4 is a schematic diagram for explaining a comparison between the center line and blood vessels at different sites in the first embodiment.

Fig. 5 is a schematic diagram illustrating the configuration of a loss function in the first embodiment using a pixel matrix.

Fig. 6 is a flowchart for explaining the split model creation in the model training apparatus according to the first embodiment.

Fig. 7 is a flowchart illustrating an image dividing process in the image processing apparatus of the first embodiment.

Fig. 8 is a block diagram showing an example of the functional configuration of the model training device according to the modification of the first embodiment.

Fig. 9 is a block diagram showing an example of the functional configuration of the model training apparatus and the image processing apparatus according to the second embodiment.

Fig. 10 is a schematic diagram showing the division of the lung field area and the lung field area in the second embodiment.

Fig. 11 is a schematic diagram illustrating the configuration of a loss function in the second embodiment using a pixel matrix.

Fig. 12 is a flowchart for explaining the split model creation in the model training apparatus according to the second embodiment.

Description of the reference numerals

100. 100a, 100b model training means; a 10 acquisition unit; 20 a center line extraction unit; 30 expansion part; 40. 40b a loss function setting unit; 50. a 50b learning unit; 60 a display unit; a 70 receiving unit; 80 a sub-region dividing section; 200 image processing means; 210 dividing part; 220 a medical image acquisition unit.

Detailed Description

Hereinafter, preferred embodiments of the image processing apparatus, the model training apparatus, and the model training method according to the present invention will be described in detail with reference to the accompanying drawings.

The model training device and the image processing device according to the present invention are each constituted by a plurality of functional blocks as separate devices, and can be implemented by being installed as software in a separate device such as a computer having a CPU (central process unit: central processing unit) and a memory, or being installed in a plurality of devices in a distributed manner, and executing the respective functional blocks of the model training device or the image processing device stored in the memory by a certain processor. The present invention may be realized in hardware as a circuit capable of executing the respective functions of the model training apparatus or the image processing apparatus. The circuit implementing the model training device or the image processing device can transmit and receive data or collect data via a network such as the internet.

The model training device of the present invention is used for training a segmentation model of a medical image. Therefore, the model training apparatus of the present invention can be installed on the manufacturer side for generating the divided model, and the manufacturer can sell the generated divided model, so that the divided model can be purchased at the image acquisition site such as a hospital and placed in the image processing apparatus at the site. The image processing apparatus of the present invention can be installed at a site of medical image acquisition, and can perform segmentation processing on an image at the site. The image processing apparatus may be directly mounted in a medical image acquisition apparatus such as a CT apparatus or a magnetic resonance imaging apparatus as a part of the medical image acquisition apparatus. The model training device and the image processing device may be provided on site to train the model on site, or the model training device and the image processing device may be realized by the same processor.

In the following description, a case is described in which the model training device in the embodiment trains and generates a segmentation model for segmenting blood vessels in a medical image of this type, and the image processing device performs segmentation processing for the medical image of this type, taking a medical image captured with a lung blood vessel as a target as an example. However, the present invention is not limited to this, and the present invention may be applied to a process of dividing another tubular object existing in a medical image.

(first embodiment)

First, a first embodiment of the present invention will be described with reference to fig. 1 to 7.

Fig. 1 is a block diagram showing an example of the functional configuration of a model training device and an image processing device according to the first embodiment.

As shown in fig. 1, the model training apparatus 100 according to the first embodiment includes an acquisition unit 10, a center line extraction unit 20, an expansion unit 30, a loss function setting unit 40, and a learning unit 50.

The acquisition unit 10 acquires a sample medical image including a tubular object segmentation result as training data. When the model training apparatus 100 creates a segmentation model for segmenting the pulmonary blood vessels, the acquisition unit 10 inputs a medical image captured by targeting the pulmonary blood vessels and a training data set composed of segmentation truth values GT (ground truth) of the medical image. The sample medical image is, for example, a CT image of the chest in three dimensions. Of course, the sample medical image may be a medical image of other dimensions.

Fig. 2 is a schematic diagram showing a medical image and a segmentation truth value according to the first embodiment. Fig. 2 (a) shows a schematic diagram of a two-dimensional cross section of a CT image of a thoracic cavity in a certain three-dimensional volume in the training dataset. It can be seen that the morphology of the blood vessels in the lung region is diverse, with both fine vessel endings shown in the a region enclosed by the dashed box, and coarse vessel endings shown in the B region enclosed by the dashed box. The division of the vessel root and vessel tip may be based on anatomical regulations or may be based on predetermined criteria.

In addition, (b) in fig. 2 shows a segmentation truth GT, which is a result of segmentation of a pulmonary vessel in a CT image of a chest in a three-dimensional manner in a training dataset, and can be extracted from existing segmented data. The pixel values in fig. 2 (b) except for the pulmonary vessel portion are zero, and are shown in black in fig. 2 for convenience of display. That is, the true value GT is represented as a matrix of pixels in three dimensions, in which pixels of a portion of the pulmonary blood vessels are given values greater than 0, and pixels of portions other than the pulmonary blood vessels are given values of 0, thereby representing the division result.

The acquisition unit 10 may perform preprocessing and data enhancement related to pixel value adjustment on the acquired sample medical image by using a conventional method. This step may also be omitted in the case where the acquired sample medical image is already an image after data processing.

The center line extraction unit 20 extracts the center line of the tubular object in the sample medical image acquired by the acquisition unit 10. Specifically, the centerline extraction unit 20 extracts the centerline of the pulmonary vessel truth GT shown in fig. 2 (b) based on a centerline extraction method such as a skeletonizing, distance conversion extraction centerline method, or the like. The center line extraction method can be referred to in the prior art, and thus a detailed description thereof will be omitted here. The centerline thus extracted is typically a single-pixel centerline running along the vessel. Fig. 3 (a) is a schematic diagram showing the center line extracted by the center line extracting unit 20.

The expansion unit 30 expands the center line extracted by the center line extraction unit 20 to obtain an expanded center line. Specifically, the expansion unit 30 expands the center line to the surrounding area by a predetermined expansion range as if the center line is expanded to the surrounding area, thereby increasing the number of pixels occupied by the center line.

Fig. 3 is a schematic view showing the extracted center line and the center line after expansion in the first embodiment. Fig. 3 (a) shows an example of the center line extracted by the extracting unit 20, and fig. 3 (b) shows an example of the expanded center line obtained by expanding the center line shown in fig. 3 (a) by the expanding unit 30. As can be seen from the figure, the center line of fig. 3 (b) is significantly thicker than that of fig. 3 (a). As shown in fig. 3 (b), the expansion performed by the expansion unit 30 causes the center line after expansion to be represented as a three-dimensionally configured matrix of pixels, and pixels at the portion of the center line after expansion and pixels at the portion other than the center line after expansion are respectively assigned different pixel values, thereby representing the center line after expansion.

The expansion performed by the expansion unit 30 is performed in all dimensions on the image, and in the example shown in fig. 3, the expansion range is preferably the same in all directions. By performing a uniform inflation, different structures of the blood vessel can be better perceived. However, the expansion may be performed to different degrees, for example, the expansion ranges of the centerlines corresponding to the root and the tip of the blood vessel may be different.

The extent of expansion may be preset according to the characteristics of the tubular structure or the location. In the present embodiment, the tubular object is a pulmonary blood vessel, and therefore, the range of the expanded center line in a section perpendicular to the center line at the thickest diameter of the pulmonary blood vessel is preferably not beyond the outer edge of the corresponding blood vessel, that is, the expanded center line is thinner than the pulmonary blood vessel; at the narrowest diameter of the pulmonary vessel, the extent of the inflated centerline in a cross-section perpendicular to the centerline exceeds the outer edges of the corresponding vessel, i.e. the inflated centerline is thicker than the pulmonary vessel.

Here, since there is a large difference between the tip and the root of the pulmonary blood vessel anatomically, it is more preferable to focus on the difference between the tip and the root of the pulmonary blood vessel, and the expansion range of the expansion unit 30 is set so that the center line after expansion is thinner than the pulmonary blood vessel at the root of the pulmonary blood vessel and the center line after expansion is thicker than the pulmonary blood vessel at the tip of the pulmonary blood vessel.

Fig. 4 is a schematic diagram for explaining a comparison between the center line and blood vessels at different sites in the first embodiment. Fig. 4 (a) shows a case where the inflated centerline and the pulmonary blood vessel truth value GT are superimposed and displayed on the medical image in the region a in fig. 2 (a). Wherein the catheter tip and its corresponding inflated centerline are shown with different shading patterns. As shown, at the distal end of the vessel, the cross-sectional area of the inflated centerline is larger than the vessel, so that the inflated centerline covers a partial area outside the vessel. On the other hand, fig. 4 (B) shows a case where the inflated centerline and the pulmonary blood vessel true value GT are superimposed and displayed on the medical image in the region B in fig. 2 (a). Wherein the arterial root, the corresponding inflated centerline of the arterial root, the venous root and the corresponding inflated centerline of the venous root are shown with different shading patterns. As shown in the figure, the cross-sectional area of the expanded center line is smaller at the root of the arterial blood vessel and the vein blood vessel than at the root of the venous blood vessel, and therefore, although the shadows of the artery and the vein are connected together, the expanded center line corresponding to the root of the arterial blood vessel and the expanded center line corresponding to the root of the vein are separated from each other by a certain distance, and the root of the different blood vessels can be better distinguished according to the center lines.

The loss function setting unit 40 sets a loss function used in the deep learning, in which a pixel matrix including the expanded center line is used as a weight matrix, so that an influence factor related to the expanded center line is added to the learning of the segmentation model.

Specifically, the loss function setting unit 40 obtains a matrix having these pixel values as elements as a weight matrix in the loss function by assigning a high value, for example, 10, to the pixels of the center line after expansion in the pixel matrix and a low value, for example, 1, to the background pixels other than the center line after expansion in the pixel matrix, as shown in fig. 3 (b).

Assuming that the pulmonary vessel true value GT is y, the prediction segmentation result of the segmentation model on the sample medical image is set as

Setting the pixel matrix including the expanded center line after the pixel values are given as described above as ω _cl In the case of (2), the loss function can also be expressed in the form of a function as +.>

The specific loss function L may be set in a conventional manner, and may be, for example, in the form of a mean square error (mean square error), cross entropy (cross entropy), or Dice.

Taking the mean square error as an example, a weight matrix may be given to the square of the absolute value of the difference between the true value and the predicted result, thereby forming a loss function shown in the following equation (1).

/>

Wherein ω from the viewpoint of the pixel matrix constituting the image _cl 、y、

Are pixel matrixes which are equal to the original image of the sample medical image in size and omega _cl The weight matrix is a matrix of pixels in which pixels of the expanded center line are given a large value and pixels of other regions other than the expanded center line are given a small value. y represents a segmentation truth value GT,/as a segmentation result of the sample medical image>

The segmentation prediction result obtained by inputting the original image of the sample medical image into the segmentation model is shown. Wherein y, ->

A pixel matrix which is the result of conventional segmentation, i.e., y is the positive value given to the pixels of the segmented pulmonary blood vessels and the pixels of the other regions than the pulmonary blood vesselsA pixel matrix of 0. />

The pixel matrix is obtained by assigning a large value to pixels of the pulmonary blood vessel predicted by the segmentation model and assigning 0 to pixels of the region other than the predicted pulmonary blood vessel.

For omega _cl The rule of the pixel assignment of (c) may be set in advance, and the value of the pixel assigned to the center line after expansion may be set as desired as long as there is a difference between the value assigned to the pixel other than the center line after expansion.

The learning unit 50 learns the training data using the loss function set by the loss function setting unit 40, and outputs a segmentation model for segmenting a tubular object in the medical image.

The specific learning method may be one of various conventional deep learning methods, for example, a method in which a sample medical image is set as an input layer and a segmentation truth value GT is set as an output layer by using a neuron network to construct an initial segmentation model for segmentation processing, and training is performed by using a loss function represented by the formula (1) to generate a trained segmentation model.

In the deep learning process of the learning unit 50, the segmentation model is evaluated using the pixel matrix including the center line after expansion as a loss function of the weight matrix as described above. Fig. 5 shows a partial schematic view of such a pixel matrix. For example, fig. 5 (a) shows a weight matrix ω formed by a pixel matrix including the center line after expansion on a two-dimensional cross section of the pulmonary vessel tip of the region a in fig. 2 (a) _cl Pixel matrix y of segmentation truth GT and pixel matrix of segmentation prediction result

Wherein, the pixel value of the central line or the peripheral of the pulmonary blood vessel is different from the background and is represented as light color. Moreover, it can be seen that the pixel area of the central line after the expansion of the vascular tip is larger than the pulmonary vascular area and the pixel matrix +. >

Is the pulmonary vessel area in the middle. In this way, when calculating the loss function, the boundary of the blood vessel and the portion other than the blood vessel have a large influence on the result, and thus more attention can be given to the minute blood vessel.

In addition, pixel values may be assigned to the pixel matrix so as to distinguish between arteries and veins, and fig. 5 (B) shows a weight matrix ω composed of a pixel matrix including the center line after expansion on a two-dimensional cross section of the pulmonary vessel root of the region B in fig. 2 (a) _cl Pixel matrix y of segmentation truth GT and pixel matrix of segmentation prediction result

Wherein pixels of arteries and veins are distinguished to give different pixel values, such as a pixel matrix ω including the expanded center line shown in fig. 5 (b) _cl The center line of the pulmonary vein root is given a first value and appears gray, the center line of the pulmonary artery root is given a second value and appears white, and the pixels of the background are given a third value and appear black, the first value, the second value, and the third value being different from each other and all being non-zero values, whereby the pixel values of the pixel matrix after such assignment form a weight matrix. The first value and the second value set above may be the same.

In addition, the pixel matrix y of the division true value GT and the pixel matrix of the division prediction result shown in fig. 5 (b)

The pulmonary veins and pulmonary arteries also appear as different pixel values, with the background pixels having values of 0, but for ease of viewing, the background pixels are shown black. As can be seen from FIG. 5 (b), the pixel area of the expanded center line of the root of the blood vessel is smaller than the lung vessel area and the pixel matrix in the pixel matrix y>

In the lung vessel area, in this way, when calculating the loss functionSince the central part of the vessel is weighted more than the vessel edges, the central part of the vessel has a greater influence on the results. Thereby helping to distinguish between closely adjacent different vessels.

The learning unit 50 forms a weight matrix ω by using a pixel matrix including the expanded center line _cl Pixel matrix y composed of true value GT and pixel matrix composed of prediction segmentation result

Substituting the loss function into the formula (1) to obtain the corresponding loss function for training.

After completion of learning, the learning unit 50 outputs the learned segmentation model.

On the other hand, returning to fig. 1, the image processing apparatus 200 receives the segmentation model generated by the model training apparatus 100 for the segmentation process of the medical image.

The image processing apparatus 200 may be installed in a device such as a hospital that can acquire medical images of a subject, and the device may perform processing by executing a corresponding processing function by a processor in the device. As shown in fig. 1, the image processing apparatus 200 includes a medical image acquisition unit 220 and a segmentation unit 210.

The medical image acquisition unit 220 acquires a medical image of the subject to be processed, which may be a CT image, an ultrasound image, or the like acquired in situ. The segmentation unit 210 segments the medical image acquired by the medical image acquisition unit 220 using the segmentation model generated by the model training apparatus 100. Specifically, the segmentation unit 210 may perform preprocessing such as converting the medical image into a format more suitable for the segmentation model, and then substituting the medical image into the segmentation model, thereby outputting the segmentation result from the segmentation model.

The specific flow of creating the segmentation model is described below.

Fig. 6 is a flowchart for explaining the split model creation in the model training apparatus according to the first embodiment. As shown in fig. 6, when starting to create a model, first, the acquisition unit 10 acquires a sample medical image and a blood vessel segmentation truth GT as training data (step S601). After the acquired sample medical images satisfy the predetermined number of training sets, the acquisition unit 10 performs processing such as preprocessing of the acquired sample medical images and data enhancement to make the medical images more suitable for the generation of a model (step S602).

Next, the process advances to step S603, where the center line extraction unit 20 extracts the center line of the tubular object in the processed sample medical image from the sample medical image. Then, the expansion unit 30 expands the center line extracted by the center line extraction unit 20 to obtain an expanded center line (step S604).

Next, the flow proceeds to step S605, where the loss function setting unit 40 sets a loss function used in the deep learning, and sets the loss function to a pixel matrix including the expanded center line as a weight matrix. The learning unit 50 learns using the processed medical image, the blood vessel segmentation truth value GT, and the loss function set by the loss function setting unit 40, and generates a blood vessel segmentation model of the learned medical image (step S606).

Fig. 7 is a flowchart illustrating an image segmentation process in the image processing apparatus according to the first embodiment. When the medical image to be processed acquired in situ is subjected to the segmentation process, first, the medical image acquisition unit 220 acquires a medical image of the subject to be processed (step S701).

Next, the segmentation unit 210 performs image preprocessing on the acquired medical image (step S702), and then substitutes the preprocessed image into the segmentation model generated by the model training device 100 to segment the medical image (step S703), thereby outputting a segmentation result from the segmentation model (step S704).

According to the first embodiment, the pixel matrix including the expanded center line is used as the weight matrix in the learning of the segmentation model, so that different parts of the blood vessel can be weighted differently by using the expanded center line, so that the loss function is more sensitive to the structure, for example, at the root of the blood vessel, the expanded center line is thinner than the blood vessel, and thus, by giving higher weight to the region of the root of the blood vessel corresponding to the expanded center line, the distinction between the different blood vessels in close proximity can be assisted. On the other hand, at the distal end of the blood vessel, the center line after inflation is thicker than the blood vessel, and more attention can be given to the minute blood vessel. Therefore, the structure sensing can be performed in a self-adaptive manner, so that the blood vessel topological structure is correct and the connectivity is improved at the root part with thicker blood vessel and compact arrangement, and the segmentation precision of the manufactured model can be improved.

(modification of the first embodiment)

In the first embodiment, the range of the center line expansion is set in advance, but the range of the center line expansion may be modified.

Fig. 8 is a block diagram showing an example of the functional configuration of the model training device according to the modification of the first embodiment. As shown in fig. 8, the model training apparatus 100a includes a display unit 60 and a receiving unit 70 in addition to the acquisition unit 10, the center line extraction unit 20, the expansion unit 30, the loss function setting unit 40, and the learning unit 50, which are similar to those of the first embodiment.

The display unit 60 displays a certain sample medical image superimposed on the automatically inflated center line to the user, and the user can change the inflation range of the inflated center line by an operation such as input or drag while observing the effect of the superimposed display. When the user changes the expansion range of the expanded center line, the reception unit 70 can receive the change of the expansion range of the expanded center line. The user can apply the changed expansion range to all the training data by changing the expansion range of the center line of a certain sample medical image. The loss function setting unit 40 sets a loss function using the changed expansion range, and the learning unit 70 learns the loss function using the changed expansion range.

In the first embodiment, the same value is given to both the expanded center line corresponding to the root of the blood vessel and the expanded center line corresponding to the tip of the blood vessel in the pixel matrix including the expanded center line, but different values may be given to the expanded center line corresponding to the root of the blood vessel and the expanded center line corresponding to the tip of the blood vessel, for example, a larger value may be given to a portion to be emphasized. The distribution of pixel values in the pixel matrix including the expanded center line can be adjusted as needed.

In the first embodiment, the loss function L is described by taking the mean square error as an example. However, the form of the loss function L is not limited to this, and for example, a loss function of the following expression (2) may be set in the form of cross entropy (cross entropy). The loss function of the following expression (3) may be set in the form of a Dice coefficient. Other forms of Focal Loss, etc. or combinations of the above may also be used.

In the above formulas (2) and (3), ω _cl Is a weight matrix containing the expanded center line, y _c Is the segmentation truth value GT,

is the segmentation prediction result.

(second embodiment)

A second embodiment of the present invention will be described with reference to fig. 9 to 12. The configuration of the model training device according to the second embodiment is different from that of the first embodiment, and mainly different points will be described below, and the same or similar configurations are denoted by the same reference numerals, and overlapping description will be omitted as appropriate.

Fig. 9 is a block diagram showing an example of the functional configuration of the model training apparatus and the image processing apparatus according to the second embodiment.

As shown in fig. 9, the model training apparatus 100b of the second embodiment includes an acquisition unit 10, a sub-region dividing unit 80, a loss function setting unit 40b, and a learning unit 50b.

The acquisition unit 10 acquires a sample medical image including a tubular object segmentation result as training data. When the model training device 100b creates a segmentation model for segmenting the pulmonary blood vessels, the acquisition unit 10 inputs a medical image captured by targeting the pulmonary blood vessels as shown in fig. 2 and a training data set composed of segmentation truth values GT (ground truth) of the medical image.

The acquisition unit 10 may perform preprocessing and data enhancement related to pixel value adjustment on the acquired sample medical image by using a conventional method. This step may also be omitted in the case where the acquired sample medical image is already an image after data processing.

The sub-region dividing section 80 identifies differences in geometric features in the sample medical image, and divides the sample medical image into a plurality of sub-regions based on the differences in geometric features.

Taking the process of dividing the pulmonary blood vessels as an example, the difference in structure between the lung field region and the lung field region is large, so that the morphology of the blood vessels present therein is also different. For example, in the lung field, the pulmonary portal, atrium, etc. are larger in volume, and the blood vessel portions present therein are also larger in volume, with little difference between foreground and background. In the field areas of the lung, the vessels are finer tubes, with large differences in volumes of foreground and background. Thus, the lung field region may be divided into different sub-regions than the lung field region.

Fig. 10 is a schematic diagram showing the division of the lung field area and the lung field area in the second embodiment. As shown in fig. 10, the portion of the lung parenchyma covered with the mesh-like hatching is divided into lung field areas, and the areas of the other parts of the subject except the lung field areas are divided into lung field areas. The lung field region and the lung field region may be divided according to an anatomical specification, or may be divided by setting specific dividing conditions in advance.

The loss function setting unit 40b sets a loss function used in the deep learning, in which a loss function is calculated by using a weight matrix for each sub-region, wherein a pixel matrix including the tube division result in a sub-region is used as a weight matrix for the sub-region, and pixels of the tube division result in different sub-regions are assigned with different values to form a weight matrix, and a loss function in which different weight matrices are designed for each sub-region is set.

Specifically, since there is a large difference in the morphology of the segmented object in the lung field region and the lung field region, by setting different weights for the lung field region and the lung field region in the loss function, the perception of the geometry can be formed in model training.

For example, when the pulmonary vessel true value GT is set to y, the prediction segmentation result of the segmentation model on the sample medical image is set to

In the case of (2), the loss function setting unit 40b sets the loss function as:

in the above-described loss function, m is a pixel matrix as large as the original image, and is a mask (mask) for the lung, and by setting 1 to the pixel of the target sub-region and 0 to the pixels other than the target sub-region, only the target sub-region can be extracted. For example, if m is a mask for extracting the lung field region

Is the loss function of the lung field region, +.>

Is the loss function of the lung field area.

In addition, ω is a constant that is used to measure the specific gravity of the lung field area loss function over the total loss function.

ω ₁ Is a pixel matrix which is equal to the original image as the weight matrix of the blood vessels in the lung field, wherein, a larger value x is given to the pixels of the blood vessel true value ₁ While pixels other than the true value of the blood vessel are assigned 1.

ω ₂ Is a pixel matrix which is equal to the original image as the weight matrix of the blood vessels in the lung field, wherein, the pixels of the blood vessel true value are given another larger value x ₂ While pixels other than the true value of the blood vessel are assigned 1.

In general, preference is given to taking x ₁ ＞x ₂ So that the vascular portion of the lung field region is weighted more heavily.

L ₁ 、L ₂ The loss function is expressed in the same meaning as that of L in the first embodiment, and may be in the form of, for example, a mean square error (mean square error), cross entropy (cross entropy), or Dice.

The learning unit 50b learns the training data using the loss function set by the loss function setting unit 40b, and outputs a segmentation model for segmenting the tubular object in the medical image.

The specific learning method may be one of various conventional deep learning methods, for example, in which a neural network is used, a sample medical image is used as an input layer, a segmentation truth value GT is used as an output layer, an initial segmentation model for performing segmentation processing is constructed, and training is performed by using a loss function set by the loss function setting unit 40b, thereby generating a trained segmentation model.

In the deep learning process of the learning unit 50, the segmentation model is evaluated using the loss function equation set to calculate the loss function re-summation for each of the different sub-regions as described above. Fig. 11 shows a partial schematic of such a pixel matrix. For example, FIG. 11 shows a mask m, a weight matrix ω, on a two-dimensional cross-section of the pulmonary vessel tip at the boundary of the lung field region and the lung field region in FIG. 10 ₁ 、ω ₂ . The mask m shown here is a mask for a lung field region, and is represented by a gray color by giving a non-zero value to pixels in the lung field region and by giving 0 to pixels in other regions (here, lung field regions). Furthermore, the weight matrix ω ₁ Wherein the white part is the pixel of a certain blood vessel in the lung field area and is given a value x ₁ Pixels of the other regions are given 1. Weight matrix omega ₂ In which the white part is the lungThe pixels of a certain blood vessel in the field area are given a value x ₂ Pixels of the other regions are given 1. Here, due to the weight matrix ω ₁ The vessels in the lung field area are masked by the mask m and are therefore not in the weight matrix omega ₁ Showing blood vessels in the lung field, again due to the weight matrix omega ₂ The vessels in the lung fields are masked by the mask (1-m) and are therefore not in the weight matrix omega ₂ Showing blood vessels in the lung field region.

The learning unit 50 uses the mask m and the weight matrix ω of the sample medical image ₁ 、ω ₂ Pixel matrix y composed of true value GT and pixel matrix composed of prediction segmentation result

Substituting the training data into a loss function formula for training.

After completion of learning, the learning unit 50 outputs the learned segmentation model.

On the other hand, returning to fig. 9, the image processing apparatus 200 receives the segmentation model generated by the model training apparatus 100 for the segmentation process of the medical image.

The image processing apparatus 200 includes a medical image acquisition unit 220 and a dividing unit 210, as in the first embodiment. The medical image acquisition unit 220 acquires a medical image of the subject to be processed, which may be a CT image, an ultrasound image, or the like acquired in situ. The segmentation unit 210 segments the medical image acquired by the medical image acquisition unit 220 using the segmentation model generated by the model training apparatus 100.

The specific flow of creating the segmentation model is described below.

Fig. 12 is a flowchart for explaining the split model creation in the model training apparatus according to the second embodiment. As shown in fig. 12, when starting to create a model, first, the acquisition unit 10 acquires a sample medical image and a blood vessel segmentation truth GT as training data (step S1201). After the acquired sample medical images satisfy the predetermined number of training sets, the acquisition unit 10 performs processing such as preprocessing of the acquired sample medical images and data enhancement to make the medical images more suitable for the generation of a model (step S1202).

Next, the step S1203 is advanced, and the sub-region dividing section 80 recognizes a difference in geometric configuration in the image to thereby divide the lung field region and the lung field region from the sample medical image.

Next, the flow proceeds to step S1204, where the loss function setting unit 40b sets a loss function in which a pixel matrix including the intrapulmonary vessels in the truth value GT of the intrapulmonary region is used as a weight matrix of the intrapulmonary region and a pixel matrix including the extrapulmonary vessels in the truth value GT of the intrapulmonary region is used as a weight matrix of the intrapulmonary region.

Thus, in step S1205, the learning unit 50 learns using the processed medical image, the blood vessel segmentation truth value GT, and the loss function set by the loss function setting unit 40b, and generates a blood vessel segmentation model of the learned medical image.

The flow of the image dividing process in the image processing apparatus according to the second embodiment is the same as that of the first embodiment, and therefore, the description thereof is omitted here.

According to the second embodiment, by assigning different weighting matrices to different subregions in the loss function according to the difference in structure between the different subregions, the loss function can be more sensitive to the difference in structure, and the segmentation accuracy of the created model can be improved.

The components of the devices according to the above embodiments are functionally conceptual, and do not necessarily have to be physically configured as shown in the drawings. That is, the specific form of dispersion and integration of the devices is not limited to the illustrated form, and all or a part of the devices may be functionally or physically dispersed and integrated in arbitrary units according to various loads, use conditions, and the like. Further, all or any part of the processing functions performed by the respective devices may be realized by a CPU and a program that is executed by the CPU, or may be realized as hardware based on wired logic.

The image processing apparatus, the model training apparatus, and the model training method described in the above embodiments can be realized by executing a program prepared in advance by a computer such as a personal computer or a workstation. The program can be distributed via a network such as the internet. The program can also be recorded on a computer-readable nonvolatile recording medium such as a hard disk, a Flexible Disk (FD), or a CD-ROM, MO, DVD, and read from the recording medium by a computer to be executed.

While the present invention has been described with reference to several embodiments, these embodiments are presented by way of example and are not intended to limit the scope of the invention. These embodiments can be implemented in various other forms, and various omissions, substitutions, and changes can be made without departing from the spirit of the invention. These embodiments and modifications are included in the scope and gist of the invention, and are also included in the invention described in the claims and their equivalents.

Claims (14)

1. A model training device for training a segmentation model of a medical image is characterized by comprising:

an acquisition unit that acquires a sample medical image including a tubular object segmentation result as training data;

a center line extraction unit that extracts a center line of a tubular object in the sample medical image;

an expansion unit configured to expand the center line extracted by the center line extraction unit to obtain an expanded center line;

a loss function setting unit that sets a loss function using a pixel matrix including the expanded center line as a weight matrix; and

and a learning unit configured to learn the training data using the loss function set by the loss function setting unit, and to output a segmentation model for segmenting the tubular object in the medical image.

2. The model training apparatus of claim 1 wherein,

the expansion unit expands the center line so that the expanded center line is thinner than the tubular at the thickest diameter of the tubular and thicker than the tubular at the thinnest diameter of the tubular.

3. The model training apparatus of claim 1 wherein,

The tube is a pulmonary vessel.

4. The model training apparatus of claim 3 wherein,

the expansion unit expands the center line so that the expanded center line is thinner than the pulmonary blood vessel at the root of the pulmonary blood vessel and thicker than the pulmonary blood vessel at the tip of the pulmonary blood vessel.

5. The model training apparatus of claim 1 wherein,

the device also comprises:

a display unit for displaying the sample medical image and the inflated center line in a superimposed manner; and

and a receiving unit that receives a change in the expansion range of the expanded center line.

6. The model training apparatus of claim 1 wherein,

the loss function setting unit assigns a first value to a pixel of the expanded center line of the tubular object in the pixel matrix, and assigns a second value different from the first value to other pixels other than the expanded center line, thereby forming a weight matrix of the loss function.

7. The model training apparatus of claim 3 wherein,

the loss function setting unit assigns a first value to the center line of the vein after inflation in the pixel matrix, assigns a second value to the center line of the artery after inflation in the pixel matrix, and assigns a third value to the pixels other than the center line after inflation, thereby forming a weight matrix of the loss function, wherein the first value, the second value, and the third value are different from each other.

8. The model training apparatus of claim 3 wherein,

the loss function setting unit assigns a first value to the center line of the root of the pulmonary blood vessel after inflation in the pixel matrix, and assigns a second value to the center line of the tip of the pulmonary blood vessel after inflation in the pixel matrix, wherein the first value is larger than the second value.

9. A model training device for training a segmentation model of a medical image is characterized by comprising:

an acquisition unit that acquires a sample medical image including a tubular object segmentation result as training data;

a sub-region dividing section that identifies differences in geometric features in the sample medical image, and divides the sample medical image into a plurality of sub-regions based on the differences in geometric features;

a loss function setting unit configured to set a loss function in which a weight matrix is formed by setting, for a certain subregion, a pixel matrix including the tubular object division result in the subregion as the weight matrix of the subregion, and assigning different values to pixels of the tubular object division result in different subregions, so that the loss function is calculated by using the weight matrix for each subregion; and

And a learning unit configured to learn the training data using the loss function set by the loss function setting unit, and to output a segmentation model for segmenting the tubular object in the medical image.

10. The model training apparatus of claim 9 wherein,

the tube is a pulmonary vessel.

11. The model training apparatus of claim 10 wherein,

the sub-region dividing section divides a lung field region and a lung field region in the sample medical image into different sub-regions,

the loss function setting unit assigns a first value to a pixel of the tube segmentation result of the lung field region, which is a weight matrix of the lung field region, and assigns a second value to a pixel of the tube segmentation result of the lung field region, which is a weight matrix of the lung field region, the first value being greater than the second value.

12. An image processing apparatus, comprising:

a medical image acquisition unit that acquires a medical image of a subject; and

a segmentation unit configured to segment the medical image acquired by the medical image acquisition unit using a segmentation model generated by the model training apparatus according to claim 1 or 9.

13. A model training method for training a segmentation model of a medical image, comprising:

an acquisition step of acquiring a sample medical image including a tubular object segmentation result as training data;

a center line extraction step of extracting a center line of a tubular object in the sample medical image;

an expansion step of expanding the center line extracted in the center line extraction step to obtain an expanded center line;

a loss function setting step of setting a loss function using a pixel matrix including the expanded center line as a weight matrix; and

and a learning step of learning the training data by using the loss function set in the loss function setting step, thereby outputting a segmentation model for segmenting the tubular object in the medical image.

14. A model training method for training a segmentation model of a medical image is characterized by comprising:

an acquisition step of acquiring a sample medical image including a tubular object segmentation result as training data;

a sub-region dividing step of identifying differences in geometric features in the sample medical image, dividing the sample medical image into a plurality of sub-regions based on the differences in geometric features;

A loss function setting step of setting a loss function in which a weight matrix is formed by setting a pixel matrix including the tubular object division result in a certain subregion as a weight matrix of the subregion and assigning different values to pixels of the tubular object division result in different subregions, so that the weight matrix is calculated by using the weight matrix for each subregion; and

and a learning step of learning the training data by using the loss function set in the loss function setting step, thereby outputting a segmentation model for segmenting the tubular object in the medical image.

Images