JP2023143628A - Image generation device, method and program, learning device, method and program, segmentation model, and image processing device, method and program - Google Patents

Abstract

To make it possible to provide a machine learning model allowing segmentation to be accurately performed, in an image generation device, method and program, a learning device, method and program, a segmentation model, and an image processing device, method and program.SOLUTION: A processor obtains an original image and a mask image in which a mask is applied to one or more regions representing each of one or more objects including a target object in the original image, derives a pseudo mask image by processing the mask in the mask image, and derives a pseudo image having a region based on the mask included in the pseudo mask image and having the same representation form as the original image, on the basis of the original image and the pseudo mask image.SELECTED DRAWING: Figure 3

Description

The present disclosure relates to an image generation device, a method, and a program, a learning device, a method, and a program, a segmentation model, and an image processing device, a method, and a program.

As a machine learning model for handling images, a convolutional neural network (hereinafter abbreviated as CNN) that performs semantic segmentation to identify objects included in an image on a pixel-by-pixel basis is known. For example, Non-Patent Document 1 proposes segmentation using a U-shaped convolutional neural network (U-Net).

Furthermore, in the medical field, medical images are segmented using machine learning models, and the degree of disease progression is determined for the segmented regions.

On the other hand, training a machine learning model requires a large amount of training data. However, in the medical field, it is difficult to collect data on rare diseases, making it difficult to provide machine learning models that can accurately perform segmentation.

Therefore, in order to train a machine learning model for detecting skin cancer, we use existing medical images and ground truth data that masks skin cancer areas in medical images to detect skin of various sizes. A method has been proposed to generate a pseudo image that includes . Furthermore, a method has also been proposed in which the spatial distribution of a three-dimensional object with an existing shape, such as a chair, is learned and an image of a chair with an unknown shape is generated (see Non-Patent Document 3).

U-Net: Convolutional Networks for Biomedical Image Segmentation, Olaf Ronneberger et al., 2015 Mask2Lesion: Mask-Constrained Adversarial Skin Lesion Image Synthesis, Kumar Abhishek et al., 2019 The shape variational autoencoder: A deep generative model of part-segmented 3D objects, C. Nash et al., 2017

However, just by using correct data for existing images as in the method described in Non-Patent Document 2, it is not possible to generate data of features that are rarely included in existing learning data. For this reason, even if generated images are added to learning data, it is difficult to construct a machine learning model that can accurately segment objects that are rarely included in existing images. Furthermore, as in the method described in Non-Patent Document 3, it is difficult to construct a machine learning model that can accurately segment objects that are different from existing objects by simply changing the shape of existing objects. It is. In particular, in the case of rare diseases such as advanced cancer, cancer tissue often invades surrounding areas, so it is desired to segment such advanced cancers with high accuracy.

The present disclosure has been made in view of the above circumstances, and aims to provide a machine learning model that can perform segmentation with high accuracy.

An image generation device according to a first aspect of the present disclosure includes at least one processor,
the processor obtains an original image and a mask image in which a mask is applied to one or more regions representing each of one or more objects including the target object in the original image;
Derive a pseudo mask image by processing the mask in the mask image,
Based on the original image and the pseudo mask image, a pseudo image is derived that has a region based on the mask included in the pseudo mask image and has the same expression format as the original image.

An image generation device according to a second aspect of the present disclosure is the image generation device according to the first aspect of the present disclosure, in which the pseudo mask image and the pseudo image are used to learn a segmentation model for segmenting objects included in the image. It may also be used as training data.

An image generation device according to a third aspect of the present disclosure may be the image generation device according to the second aspect of the present disclosure, in which the processor accumulates the pseudo mask image and the pseudo image as teacher data.

An image generation device according to a fourth aspect of the present disclosure is an image generation device according to any one of the first to third aspects of the present disclosure, in which the processor is configured to generate a target object in a class different from a class indicated by the target object. It may be possible to derive a pseudo mask image that can generate a pseudo image including the following.

"Different classes" means that the shape of the target object is different, or if the target object is a lesion included in a medical image, the degree of progression of the lesion is different. Furthermore, a "different class" refers to a class that appears less frequently than other classes, or does not appear at all, among the teacher data held. For this reason, by ``deriving a pseudo mask image that can generate a pseudo image that includes the target object in a class different from the class indicated by the target object,'' we prepare training data for classes that are small or absent in existing training data. can do. Therefore, by training a segmentation model using such training data together with existing training data, it is possible to construct a segmentation model that can segment target objects for images that include target objects that are rarely applied. can.

An image generation device according to a fifth aspect of the present disclosure is an image generation device according to any one of the first to fourth aspects of the present disclosure, in which the processor is an evaluation index in clinical practice regarding medical images. A pseudo mask image may be derived by processing the mask so that at least one of the shape and progression of the lesion is different from the lesion included in the original image based on the lesion shape evaluation index. .

An image generation device according to a sixth aspect of the present disclosure is the image generation device according to any one of the first to fifth aspects of the present disclosure. A pseudo mask image may be derived by processing the mask until it has a shape that is evaluated as a lesion based on the following.

An image generation device according to a seventh aspect of the present disclosure is an image generation device according to any one of the first to sixth aspects of the present disclosure, in which the processor includes at least one image having a predetermined density, color, or texture. A pseudo image having density, color, or texture according to the style image may be generated by referring to one style image.

A "style image" is an image that represents an object of the same type as the target object, having the density, color, and texture that the target object may have.

An image generation device according to an eighth aspect of the present disclosure is the image generation device according to any one of the first to seventh aspects of the present disclosure, in which the processor receives an instruction regarding the degree of processing of the mask and responds to the instruction. A pseudo mask image may be derived by processing a mask based on the mask.

In the image generation device according to a ninth aspect of the present disclosure, in the image generation device according to the eighth aspect of the present disclosure, the processor specifies the position of the end point of the mask after processing and the amount of processing. It may also be accepted as an instruction.

An image generation device according to a tenth aspect of the present disclosure is the image generation device according to the eighth or ninth aspect of the present disclosure, wherein the processor receives an instruction for the degree of processing of the mask according to preset constraint conditions. It may be something.

An image generation device according to an eleventh aspect of the present disclosure is an image generation device according to any one of the first to tenth aspects of the present disclosure, in which the original image includes a plurality of objects, a target object and a target object other than the target object. When some regions of other objects have an inclusive relationship with each other, in the mask image, the areas that have an inclusive relationship may be given a different mask from those that do not have an inclusive relationship.

An image generation device according to a twelfth aspect of the present disclosure is the image generation device according to the eleventh aspect of the present disclosure, in which when the other object in the inclusion relationship is a fixed object in the original image, the processor Alternatively, a pseudo mask image may be derived by processing a mask applied to a target object to match the shape of a mask applied to a fixed object.

An image generation device according to a thirteenth aspect of the present disclosure is the image generation device according to any one of the first to twelfth aspects of the present disclosure, in which when the original image is a three-dimensional image, the processor The pseudo mask image may be derived by processing the mask while maintaining the three-dimensional continuity of the mask applied to the area.

An image generation device according to a fourteenth aspect of the present disclosure is an image generation device according to any one of the first to thirteenth aspects of the present disclosure, wherein the original image is a three-dimensional medical image;
The target object may be a lesion included in a medical image.

An image generation device according to a fifteenth aspect of the present disclosure is an image generation device according to a fourteenth aspect of the present disclosure, wherein the medical image includes a rectum of a human body;
The target object is rectal cancer, and the other object other than the target object is at least one of the mucosal layer of the rectum, the submucosa of the rectum, the muscularis propria of the rectum, the subserosa of the rectum, and a background other than these. It's okay.

An image generation device according to a sixteenth aspect of the present disclosure is an image generation device according to a fourteenth aspect of the present disclosure, wherein the medical image includes a joint of a human body;
The target object is a bone forming a joint, and the object other than the target object may be a background other than the bones forming a joint.

A learning device according to a seventeenth aspect of the present disclosure includes at least one processor,
The processor performs machine learning using, as training data, a set of a plurality of pseudo images and pseudo mask images generated by the image generation device according to any one of the first to sixteenth aspects of the present disclosure. A segmentation model is constructed to segment one or more object regions including a target object included in an image.

A learning device according to an eighteenth aspect of the present disclosure is a learning device according to a seventeenth aspect of the present disclosure, wherein the processor further performs machine learning using a plurality of sets of original images and mask images as training data. Alternatively, a segmentation model may be constructed by

The segmentation model according to the nineteenth aspect of the present disclosure is constructed by the learning device according to the seventeenth or eighteenth aspect of the present disclosure.

An image processing device according to a twentieth aspect of the present disclosure includes at least one processor,
The processor uses the segmentation model according to the nineteenth aspect of the present disclosure to segment the region of one or more objects including the target object included in the target image to be processed. Derive a mask image in which the object is masked.

An image processing apparatus according to a twenty-first aspect of the present disclosure is the image processing apparatus according to the twentieth aspect of the present disclosure, in which the processor uses a discriminant model that discriminates the class of the target object included in the mask image to The class of the target object masked in the image may be determined.

An image generation method according to a twenty-second aspect of the present disclosure obtains an original image and a mask image in which a mask is applied to one or more regions representing each of one or more objects including a target object in the original image,
Derive a pseudo mask image by processing the mask in the mask image,
Based on the original image and the pseudo mask image, a pseudo image is derived that has a region based on the mask included in the pseudo mask image and has the same expression format as the original image.

A learning method according to a twenty-third aspect of the present disclosure includes performing machine learning using a set of a plurality of pseudo images and a pseudo mask image generated by the image generation method according to the twenty-second aspect of the present disclosure as training data. A segmentation model is constructed to segment one or more object regions including the target object included in the input image.

An image processing method according to a twenty-fourth aspect of the present disclosure uses the segmentation model according to the nineteenth aspect of the present disclosure to segment one or more object regions including a target object included in a target image to be processed. By doing so, a mask image in which one or more objects included in the target image is masked is derived.

An image generation program according to a twenty-fifth aspect of the present disclosure includes a step of acquiring an original image and a mask image in which a mask is applied to one or more regions representing each of one or more objects including a target object in the original image;
A procedure for deriving a pseudo mask image by processing a mask in the mask image;
Based on the original image and the pseudo mask image, the computer executes a procedure for deriving a pseudo image that has a region based on a mask included in the pseudo mask image and has the same expression format as the original image.

A learning program according to a twenty-sixth aspect of the present disclosure performs machine learning using a set of a plurality of pseudo images and a pseudo mask image generated by the image generation method according to the twenty-second aspect of the present disclosure as training data. This causes the computer to execute a procedure for constructing a segmentation model that segments one or more object regions including the target object included in the input image.

An image processing program according to a twenty-seventh aspect of the present disclosure uses the segmentation model according to the nineteenth aspect of the present disclosure to segment a region of one or more objects including a target object included in a target image to be processed. This causes the computer to execute a procedure for deriving a mask image in which one or more objects included in the target image are masked.

According to the present disclosure, a machine learning model that can perform segmentation with high accuracy can be provided.

A diagram showing a schematic configuration of a diagnosis support system to which an image generation device, a learning device, and an image processing device according to an embodiment of the present disclosure are applied. A diagram showing the hardware configuration of an image generation device and a learning device according to the present embodiment Functional configuration diagram of an image generation device and a learning device according to this embodiment Diagram schematically showing a cross section of the rectum to explain the progression of rectal cancer Diagram showing an example of original image and mask image Diagram showing the mask processing screen for rectal cancer Diagram showing the mask processing screen for rectal cancer Diagram showing the mask processing screen for rectal cancer Diagram showing another example of mask processing Diagram showing another example of mask processing Diagram schematically showing the generator and its learning Diagram showing pseudo-mask and pseudo-image A diagram showing the hardware configuration of an image processing device according to this embodiment Functional configuration diagram of an image processing device according to this embodiment Diagram showing the display screen Flowchart of image generation processing in this embodiment Flowchart of learning processing in this embodiment Flowchart of image processing in this embodiment Diagram showing the mask processing screen for bone spurs Diagram showing the mask processing screen for bone spurs Diagram showing another example of the mask processing screen Diagram showing another example of the mask processing screen Diagram showing another example of the mask processing screen Diagram showing another example of the mask processing screen Diagram showing another example of the mask processing screen Diagram showing another example of the mask processing screen Diagram showing another example of the mask processing screen Diagram showing another example of the mask processing screen Diagram showing another example of the mask processing screen Diagram showing another example of the mask processing screen

Embodiments of the present disclosure will be described below with reference to the drawings. First, the configuration of a medical information system to which an image generation device, a learning device, and an image processing device according to this embodiment are applied will be described. FIG. 1 is a diagram showing a schematic configuration of a medical information system. In the medical information system shown in FIG. 1, a computer 1 including an image generation device and a learning device according to the present embodiment, a computer 2 containing an image processing device according to the present embodiment, an imaging device 3, and an image storage server 4 are connected to a network. 5 and is connected in a communicable state.

The computer 1 includes an image generation device and a learning device according to the present embodiment, and has the image generation program and learning program according to the present embodiment installed therein. The computer 1 may be a workstation or a personal computer, or a server computer connected thereto via a network. The image generation program and the learning program are stored in a storage device of a server computer connected to a network or in a network storage in a state that can be accessed from the outside, and are downloaded and installed in the computer 1 according to a request. Alternatively, it is recorded and distributed on a recording medium such as a DVD (Digital Versatile Disc) or a CD-ROM (Compact Disc Read Only Memory), and installed on the computer 1 from the recording medium.

The computer 2 includes an image processing apparatus according to the present embodiment, and has an image processing program according to the present embodiment installed therein. The computer 2 may be a workstation or a personal computer, or a server computer connected thereto via a network. The image processing program is stored in a storage device of a server computer connected to a network or in a network storage in a state that can be accessed from the outside, and is downloaded and installed in the computer 2 according to a request. Alternatively, it is recorded and distributed on a recording medium such as a DVD or CD-ROM, and installed on the computer 2 from the recording medium.

The imaging device 3 is a device that generates a three-dimensional image representing the region of the subject by photographing the region to be diagnosed. Specifically, the imaging device 3 is a device that generates a three-dimensional image representing the region of the subject. ) equipment, etc. A three-dimensional image composed of a plurality of tomographic images generated by the imaging device 3 is transmitted to the image storage server 4 and stored. Note that in this embodiment, the imaging device 3 is an MRI device, and generates an MRI image of a human body, which is a subject, as a three-dimensional image. Furthermore, in this embodiment, the three-dimensional image is a three-dimensional image including the vicinity of the rectum of the human body. Therefore, when a patient with rectal cancer is photographed, the three-dimensional image includes the rectal cancer.

The image storage server 4 is a computer that stores and manages various data, and includes a large-capacity external storage device and database management software. The image storage server 4 communicates with other devices via a wired or wireless network 5 and sends and receives image data and the like. Specifically, various data including image data of a three-dimensional image generated by the photographing device 3 is acquired via a network, and is stored and managed in a recording medium such as a large-capacity external storage device. The image storage server 4 also stores training data for constructing machine learning models for deriving pseudo images, detecting abnormal areas, and determining classes of abnormal areas, as described later. has been done. Note that the storage format of image data and the communication between each device via the network 5 are based on a protocol such as DICOM (Digital Imaging and Communication in Medicine).

Next, an image generation device and a learning device according to this embodiment will be explained. FIG. 2 is a diagram showing the hardware configuration of the image generation device and the learning device according to this embodiment. As shown in FIG. 2, an image generation device and a learning device (hereinafter referred to as image generation device) 20 includes a CPU (Central Processing Unit) 11, a nonvolatile storage 13, and a memory 16 as a temporary storage area. . The image generation device 20 also includes a display 14 such as a liquid crystal display, an input device 15 such as a keyboard and a mouse, and a network I/F (InterFace) 17 connected to the network 5. The CPU 11, storage 13, display 14, input device 15, memory 16, and network I/F 17 are connected to the bus 18. Note that the CPU 11 is an example of a processor in the present disclosure.

The storage 13 is realized by an HDD (Hard Disk Drive), an SSD (Solid State Drive), a flash memory, or the like. The storage 13 as a storage medium stores an image generation program 12A and a learning program 12B. The CPU 11 reads the image generation program 12A and the learning program 12B from the storage 13, expands them into the memory 16, and executes the expanded image generation program 12A and learning program 12B.

Next, the functional configurations of the image generation device and learning device according to this embodiment will be explained. FIG. 3 is a diagram showing the functional configuration of the image generation device and the learning device according to this embodiment. As shown in FIG. 3, the image generation device 20 includes an information acquisition section 21, a pseudo mask derivation section 22, a pseudo image derivation section 23, and a learning section 24. When the CPU 11 executes the image generation program 12A, the CPU 11 functions as an information acquisition section 21, a pseudo mask derivation section 22, and a pseudo image derivation section 23. Furthermore, the CPU 11 functions as the learning section 24 by executing the learning program 12B.

The information acquisition unit 21 acquires an original image G0 used for deriving a pseudo image, which will be described later, from the image storage server 4. The information acquisition unit 21 also acquires teacher data for constructing a learned model, which will be described later, from the image storage server 4.

Here, in this embodiment, the original image G0 is stored together with a mask image M0 in which a mask is applied to the object area included in the original image G0. Furthermore, when the original image G0 includes rectal cancer, information representing the stage of the rectal cancer is added to the original image G0.

The mask may be applied to the original image G0 by manual operation using the input device 15, or by segmenting the original image G0. Semantic segmentation is used for segmentation, which performs class classification by labeling all pixels of an image pixel by pixel. Semantic segmentation is performed using a semantic segmentation model, which is a machine learning model constructed by machine learning to extract the region of an object included in an image. The semantic segmentation model will be described later. Note that the original image G0 and mask image M0 stored in the image storage server 4 may be those derived by the image processing device of this embodiment described later.

In the present embodiment, rectal cancer is detected from a target image to be processed by an image processing apparatus to be described later. Therefore, the original image G0 identifies the respective areas of rectal cancer, the mucosal layer of the rectum, the submucosa of the rectum, the muscularis propria of the rectum, the subserosa of the rectum, and other background areas included in the original image G0. A mask for this purpose is added to the original image G0 as a mask image M0. The rectal cancer included in the original image G0 is an example of the target object, and the mucosal layer of the rectum, the submucosa of the rectum, the muscularis propria of the rectum, the subserosa of the rectum, and the background other than these are other than the target object. This is an example of an object.

In the early stages, rectal cancer exists only in the mucosal layer, but as rectal cancer progresses, it spreads outward from the mucosal layer, invades the submucosa and muscularis propria, and forms an inclusive relationship with the submucosa and muscularis propria. Become what you have. FIG. 4 is a diagram schematically showing a cross section of the rectum to explain the progression of rectal cancer. As shown in FIG. 4, the rectum 30 consists of a mucosal layer 31, a submucosal layer 32, a muscularis propria 33, and a subserosal layer 34. In FIG. 4, regions of rectal cancer 35 are shown hatched according to the degree of progression of the cancer, that is, the stage of the cancer. As shown in FIG. 4, rectal cancer 35 at early stages T1 and T2 is located in the mucosal layer 31 of the rectum 30, but as rectal cancer 35 progresses, it invades the submucosa 32 (stage T3ab) and further It invades the muscular layer 33 (stage T3cd) and gradually approaches the subserosal layer 34 (stage T3MRF+). As it progresses further, it breaks through the muscularis propria 33 and the subserosal layer 34 (stage T4a) and reaches other organs (stage T4b). Note that each stage of rectal cancer corresponds to a different class in the present disclosure.

FIG. 5 is a diagram showing an example of the original image G0 and the mask image M0. The original image G0 shown in FIG. 5 shows a tomographic plane obtained by cutting a portion of the rectum where rectal cancer exists in a direction intersecting the central axis of the rectum. In the original image G0 shown in FIG. It has an inclusive relationship with the muscularis propria and the muscularis propria 33. In addition, in FIG. 5, the rectal cancer 35 has not invaded the subserosal layer 34.

In this embodiment, areas in an inclusive relationship in the original image G0 are given a different mask from areas that are not in an inclusive relationship. For example, in the mask image M0 shown in FIG. 5, masks M1, M2, M3, and M4 are applied to the mucosal layer 31, the submucosa layer 32, the muscularis propria 33, and the subserosa layer 34, respectively. In the rectal cancer 35, a mask M5 is given to a region that is not included in any tissue of the rectum. In the rectal cancer 35, a mask M6 is applied to an area in an inclusive relationship with the mucosal layer 31, a mask M7 is applied to an area in an inclusive relationship with the submucosal layer 32, and a mask M7 is applied to an area in an inclusive relationship with the muscularis propria 33. is given mask M8. Note that in FIG. 5, the background mask is omitted.

Note that when deriving the mask image M0 using a segmentation model, in order to segment regions that have an inclusion relationship using a semantic segmentation model, correct data that masks regions that have an inclusion relationship and areas that do not have an inclusion relationship differently must be used. A segmentation model can be constructed by preparing training data including the following and performing machine learning.

The pseudo mask deriving unit 22 derives a pseudo mask image by processing the mask included in the mask image M0 of the original image G0. For this purpose, the display control unit 74 displays a mask processing screen on the display 14. FIG. 6 is a diagram showing a mask processing screen. As shown in FIG. 6, the mask processing screen 40 includes a mask image M0, a pseudo mask image Mf0 including a mask Mf0 obtained by processing the mask M0 included in the mask image M0, and a rectum included in the original image G0. A pull-down menu 26 for setting the three-dimensional model 41 and the degree of deformation of the three-dimensional model 41, a processing button 27 for performing mask processing, and a conversion button 28 for deriving a pseudo image are displayed. has been done.

Note that the mask image M0 and pseudo-mask image Mf0 shown in FIG. 6 show tomographic planes obtained by cutting a portion of the rectum where rectal cancer exists in a direction intersecting the central axis of the rectum. For the purpose of explanation, the mask image M0 shown in FIG. 6 is an image in which a mask Ms0 is applied only to the rectal cancer region to be processed in the original image G0. Further, in the pseudo mask image Mf0, only the rectal cancer mask to be processed is given the reference code Msf0. Since the mask image M0 and the pseudo mask image Mf0 shown in FIG. 6 are before mask processing, the masks Ms0 and Msf0 given to each have the same shape. Here, the rectal cancer shown in FIG. 6 is at stage T3ab as the stage of the rectal cancer shown in FIG. 4.

The three-dimensional model 41 is a three-dimensional image derived by extracting only the region of rectal cancer in the original image G0 and performing volume rendering. By operating the input device 15, the operator can change the orientation of the three-dimensional model 41 in all directions.

The pull-down menu 26 allows selection of the stage of rectal cancer. That is, the pull-down menu 26 allows selection of rectal cancer stages T1, T2, T3ab, T3cd, T3MRF+, T4a, and T4b shown in FIG. 4.

In the present embodiment, the pseudo mask deriving unit 22 derives a pseudo mask image that can generate a pseudo image including a target object of a class different from the class indicated by the target object. That is, when deriving the pseudo mask image Mf0, the pseudo mask deriving unit 22 uses the three-dimensional model 41 to generate a pseudo image containing rectal cancer at a stage different from the stage of rectal cancer included in the original image G0. Process the mask by transforming it. For example, if the stage of rectal cancer included in the original image G0 is stage T1, which exists only in the mucosal layer, the three-dimensional model 41 of the rectal cancer is transformed so as to extend to the muscularis propria. 41 corresponds to, for example, rectal cancer at stage T3ab, which has progressed from stage T1. Furthermore, by deforming the three-dimensional model 41 of rectal cancer so as to break through the muscularis propria, the three-dimensional model 41 corresponds to stage T4a rectal cancer.

Therefore, the operator looks at the displayed mask image M0 and selects the stage of rectal cancer to be generated as a pseudo image from the pull-down menu 26. FIG. 6 shows a state in which stage T3MRF+ is selected. When the operator selects the processing button 27 after selecting the stage of rectal cancer, the pseudo mask deriving unit 22 creates a three-dimensional model so that the stage of rectal cancer represented by the three-dimensional model 41 is T3MRF+. Transform 41. Furthermore, the pseudo-mask deriving unit 22 processes the mask Msf0 given to the pseudo-mask image Mf0 so that it conforms to the shape of the deformed three-dimensional model 41.

Note that when deriving the pseudo mask image Mf0, the pseudo mask deriving unit 22 determines whether the shape and/or degree of progression of the lesion is included in the original image G0 based on a lesion shape evaluation index that is a clinical evaluation index for medical images. The three-dimensional model 41 is deformed so that it becomes a lesion different from that of the lesion. That is, the shape of the rectal cancer included in the original image G0 is changed so that the shape and/or the degree of progression of the rectal cancer changes from stage T3ab to stage T3MRF+, for example. Alternatively, the three-dimensional model 41 is deformed until a normal organ in a medical image has a shape that can be evaluated as a lesion based on clinical measurement indicators.

Further, the pseudo mask deriving unit 22 processes the mask while maintaining the three-dimensional continuity of the mask applied to the rectal cancer. For example, when deforming the three-dimensional model 41 so as to extend it toward the outside of the rectum, the degree of deformation is made smaller as the distance from the center of the rectum in the three-dimensional model 41 increases. As a result, the three-dimensional model 41 can be transformed while maintaining the three-dimensional continuity of the original three-dimensional model 41, and as a result, the three-dimensional continuity of the mask applied to rectal cancer can be changed. You can process the mask while holding it.

Furthermore, the three-dimensional model 41 corresponds to rectal cancer, and deforming the three-dimensional model 41 to advance the stage of rectal cancer causes rectal cancer to invade the submucosa of the rectum and even the muscularis propria. That will happen. Rectal cancer expands and deforms as it progresses, but the way it deforms depends on the shape of the rectum. That is, rectal cancer expands and deforms to fit the shape of the rectum. Here, the rectum is fixed within the original image G0 without moving or deforming. For this reason, the pseudo mask deriving unit 22 deforms the three-dimensional model 41 in accordance with the shape of the mask applied to the fixed submucosal layer and muscularis propria around the rectal cancer. Thereby, it is possible to derive a pseudo mask image Mf0 that represents rectal cancer in a natural shape that matches the shape of the rectum.

FIG. 7 shows a state in which the three-dimensional model 41 has been transformed from stage T3ab rectal cancer to stage T3MRF+ rectal cancer and a region 41A has been added on the mask processing screen 40. As the three-dimensional model 41 is deformed, a mask Msf0 obtained by processing the mask Ms0 of the mask image M0 is added to the pseudo mask image Mf0.

Note that the operator may be able to instruct the degree of deformation of the three-dimensional model 41. FIG. 8 is a diagram showing a mask processing screen on which the operator can instruct the degree of deformation of the three-dimensional model. Note that in FIG. 8, the same reference numerals are given to the same components as in FIG. 6, and detailed explanations are omitted here. As shown in FIG. 8, the mask processing screen 40 displays a scale 42 for adjusting the degree of deformation of the three-dimensional model 41, in place of the pull-down menu 26 and processing button 27 shown in FIG.

On the mask processing screen 40A shown in FIG. 8, the operator specifies the degree of deformation of the three-dimensional model 41 by moving the knob 42A of the scale 42 using the input device 15. As a result, the pseudo mask deriving unit 22 deforms the three-dimensional model 41 so as to extend it from the specified position, and further transforms the mask Msf0 given to the pseudo mask image Mf0 into the shape of the extended three-dimensional model 41. Process it to fit. In this case as well, the pseudo mask deriving unit 22 determines whether the shape and/or degree of progression of the lesion is included in the original image G0 based on the lesion shape evaluation index, which is a clinical evaluation index for the medical image specified by the operator. The three-dimensional model 41 is deformed so that it becomes a lesion different from that of the lesion. Further, the pseudo mask deriving unit 22 processes the mask while maintaining the three-dimensional continuity of the mask applied to the rectal cancer.

Note that the derivation of the pseudo mask image Mf0 is not limited to that according to the stage of rectal cancer specified as described above. For example, in order to generate a mask having multiple spinous processes such as infiltrated lymph nodes, in the mask processing screen 40 shown in FIG. Alternatively, a pull-down menu for selecting whether to add spinous processes may be displayed. Further, in addition to the scale 42 shown in FIG. 8, a pull-down menu for selecting whether or not to add spinous processes may be displayed. In this case, the three-dimensional model 41 on the mask processing screen 40 has the infiltrated lymph nodes 43 added thereto as shown in FIG. It will be added.

In order to generate a mask having small protrusions such as vascular invasion, in addition to or in place of the pull-down menu 26 for selecting the stage of rectal cancer, in the mask processing screen 40 shown in FIG. A pull-down menu may be displayed for selecting whether to grant or not. Furthermore, in addition to the scale 42 shown in FIG. 8, a pull-down menu for selecting whether or not to provide protrusions may be displayed. Note that the position where the protrusion is to be provided and the position of the tip of the protrusion in rectal cancer are specified by the operator using the input device 15. In this case, the three-dimensional model 41 on the mask processing screen 40 has a vascular invasion 44 added thereto by interpolating the protrusion placement position and the tip position of the protrusion by, for example, spline interpolation, as shown in FIG. . Note that in the three-dimensional model 41 shown in FIG. 10, two blood vessel invasions are added. Further, the pseudo mask image Mf0 is obtained by adding two areas of blood vessel invasion 44 to the rectal cancer mask.

Further, the pseudo mask deriving unit 22 derives information representing the stage of rectal cancer for the derived pseudo mask image Mf0. Here, in the mask processing screen 40 shown in FIG. 6, since the stage of rectal cancer is selected by the operator from the pull-down menu 26, the selected stage of rectal cancer may be used as is. On the other hand, on the mask processing screen 40A shown in FIG. 8, the degree of processing is instructed by the operator. Therefore, the pseudo-mask deriving unit 22 determines the stage of the rectal cancer included in the pseudo-mask image Mf0 according to the depth of the mask Msf0 from the mucous membrane layer 31, and adds information representing the stage to the pseudo-mask image Mf0. Give. Note that the information representing the stage of rectal cancer in the pseudo mask image Mf0 may be based on input from the input device 15 by the operator.

When the conversion button 28 is selected on the mask processing screen 40, the pseudo image deriving unit 23 derives a pseudo image having a region based on the mask included in the pseudo mask image Mf0, based on the original image G0 and the pseudo mask image Mf0. do. For this purpose, the pseudo image deriving unit 23 includes a generator 50 trained using Generative Adversarial Networks (GAN). The generator 50 is constructed by learning to output a pseudo image having a region based on the mask included in the mask image when the original image G0 containing rectal cancer and a mask image are input.

In this embodiment, a pseudo image means an image having the same expression format as the image acquired by the modality that acquired the original image G0. That is, when the original image G0 is an MRI image acquired by an MRI imaging device, the pseudo image means an image having the same expression format as the MRI image. Here, the expression format is the same means that structures having the same composition are expressed with the same density or brightness.

Here, the generator 50 uses, for example, the SPADE method of generating a pseudo image from a mask image described in "Semantic Image Synthesis with Spatially-Adaptive Normalization, Park et al., arXiv:1903.07291v2 [cs.CV] 5 Nov 2019". A constructed generator can be used.

FIG. 11 is a diagram schematically showing a generator and its learning. As shown in FIG. 11, generator 50 includes an encoder 51 and a decoder 52. In this embodiment, the generator 50 constitutes a generative adversarial network (GAN) together with a discriminator 53, which will be described later. In the example shown in FIG. 11, the learning image S1 is an MRI image including the rectum, and the learning image in which masks are applied to the rectal cancer, mucosal layer, submucosal layer, muscularis propria, and subserosal layer in the learning image S1. Teacher data S0 consisting of a mask image S2 is prepared.

The encoder 51 constituting the generator 50 is a convolutional neural network (CNN), which is one of the multilayer neural networks in which a plurality of processing layers are hierarchically connected. When the image S1 is input, a latent expression z0 representing the feature amount of the MRI image including the rectum is output.

The decoder 52 encodes the latent expression z0 output by the encoder 51, applies the masks of the individual regions included in the learning mask image S2 to generate regions represented by each mask, and A pseudo image S3 having regions based on the respective regions and having the same expression format as the learning image S1 is output.

The discriminator 53 determines whether the input image is a real image or a pseudo image generated by the generator 50, and outputs a determination result TF0. Here, the actual image is not an image generated by the generator 50, but an original image obtained by photographing a subject with the photographing device 3. On the other hand, the pseudo image is an image that has the same expression format as the original image generated from the mask image by the generator 50.

In this embodiment, the discriminator 53 is trained so as to correctly determine whether the input image is a real image or a pseudo image generated by the generator 50 as the determination result TF0. Further, the generator 50 is trained so that a pseudo image resembling the real image is derived from the input mask image, and the discriminator 53 determines that the determination result TF0 is incorrect. This allows the generator 50 to generate a pseudo image that is not identified by the discriminator 53 and has the same representation format as the real MRI image.

The pseudo image deriving unit 23 uses the generator 50 constructed in this manner to derive a pseudo image from the original image G0 and the pseudo mask image Mf0 derived by the pseudo mask deriving unit 22. For example, when the original image G0 and the pseudo mask image Mf0 as shown in FIG. 12 are input, the generator 50 outputs the pseudo image Gf0 having the same expression format as the original image G0. The derived pseudo image Gf0 is stored in the storage 13 together with the pseudo mask image Mf0 and information representing the stage of rectal cancer. Here, in the present embodiment, existing images that are not generated by the image generation device according to the present embodiment, that is, the original image S0, the mask image M0 in which the rectal cancer in the original image S0 is masked, and the rectal Information representing the stage of cancer is stored in the storage 13 as training data for learning a segmentation model to be described later. In the present embodiment, the pseudo image Gf0, the pseudo mask image Mf0, and the information representing the stage of rectal cancer derived by the image generation device 1 according to the present embodiment are stored in the storage 13 in addition to the existing training data. Accumulated as teacher data. Further, the pseudo image Gf0 may transmit the pseudo mask image Mf0 and information representing the stage of rectal cancer to the image storage server 4, where they may be stored together with existing teacher data.

The learning unit 24 learns a segmentation model that segments an MRI image including the rectum into a plurality of regions. In this embodiment, we learn a semantic segmentation model that segments an MRI image into rectal cancer, the mucosal layer of the rectum, the submucosa of the rectum, the muscularis propria of the rectum, the subserosa of the rectum, and other background regions. and build a trained semantic segmentation model. As is well known, a semantic segmentation model (hereinafter referred to as SS (Semantic Segmentation) model) is a machine learning model that outputs an output image in which each pixel of an input image is given a mask representing the extraction target (class). It is. In this embodiment, the input image to the SS model is an MRI image including the rectal region, and the output image is the rectal cancer in the MRI image, the mucosal layer of the rectum, the submucosa of the rectum, the muscularis propria of the rectum, and the rectal region. This is a mask image in which the subserosa layer and other background areas are masked. The SS model is constructed using a convolutional neural network (CNN) such as ResNet (Residual Networks) or U-Net (U-shaped Networks).

When learning the SS model, in addition to existing training data, that is, training data consisting of a combination of the original image G0 and the mask image M0 of the original image G0, the pseudo mask image Mf0 and the pseudo image derived by the pseudo mask derivation unit 22 are used. Teacher data consisting of a combination of pseudo images Gf0 derived by the unit 23 is used. In the existing teacher data, the original image G0 and pseudo image Gf0 are learning data, and the mask image M0 and pseudo mask image Mf0 are correct data. In the teacher data including the pseudo image Gf0 and the pseudo mask image Mf0, the pseudo image Gf0 is the learning data, and the pseudo mask image Mf0 is the correct data.

During learning, the original image G0 and pseudo image Gf0 are input to the SS model, and a mask image in which objects included in these images are segmented is output. Next, the difference between the mask image output by the SS model and the mask image M0 and pseudo mask image Mf0, which are correct data, is derived as a loss. Then, learning of the SS model is repeated using a plurality of pieces of training data so that the loss is small, and the SS model is constructed.

The learning unit 24 also learns a discrimination model for determining the stage of rectal cancer for MRI images including the rectum, and constructs a learned discrimination model. In this embodiment, the input images to the discriminant model are an MRI image including the rectal region and a mask image obtained by segmenting the MRI image, and the output is the stage of rectal cancer included in the MRI image. The discriminant model is also constructed using a convolutional neural network such as ResNet or U-Net.

When learning the discriminant model, in addition to the existing training data, which is a combination of the original image G0, the mask image M0 of the original image G0, and the information representing the stage of rectal cancer included in the original image G0, a pseudo mask is used. The pseudo mask image Mf0 derived by the deriving unit 22, the pseudo image Gf0 derived by the pseudo image deriving unit 23, and the training data consisting of a combination of information representing the stage of rectal cancer included in the pseudo image Gf0 are used. In the existing teacher data, the original image G0 and the mask image M0 are learning data, and the information representing the stage of rectal cancer in the original image G0 is correct data. In the training data including pseudo image Gf0 and pseudo mask image Mf0, pseudo image Gf0 and pseudo mask image Mf0 are learning data, and information representing the stage of rectal cancer is correct data.

During learning, the original image G0 and mask image M0, as well as the pseudo image Gf0 and pseudo mask image Mf0, are input to the discriminant model, and information representing the stage of rectal cancer included in these images is output. Information representing the stage of rectal cancer is the probability of being at each stage of rectal cancer. Probability takes a value between 0 and 1. Next, the difference between the probability of each stage of rectal cancer and the stage of rectal cancer in the correct data is derived as a loss. Here, the stage of rectal cancer output by the discriminant model (T1, T2, T3, T4) = (0.1, 0.1, 0.7, 0.1), and the correct data is (0, 0 , 1, 0), the difference between the probability of each stage of rectal cancer output by the discriminant model and the probability of each stage of rectal cancer in the correct data is derived as a loss. Then, learning of the discriminant model is repeated using a plurality of pieces of training data so that the loss is small, and the discriminant model is constructed.

Note that the discrimination model may be constructed so that when only the mask image for the input image is input, information representing the stage of rectal cancer included in the input image is output. In this case, for learning the discriminant model, training data includes a mask image M0 for the original image G0, teacher data including information representing the stage of rectal cancer included in the original image G0 as correct data, and training data for the pseudo image Gf0. The pseudo mask image Mf0 of is used as learning data, and the teacher data including information representing the stage of rectal cancer included in the pseudo image Gf0 as correct data is used.

Next, an image processing apparatus according to this embodiment will be explained. FIG. 13 is a diagram showing the hardware configuration of the image processing device according to this embodiment. As shown in FIG. 13, the image processing device 60 includes a CPU 61, a nonvolatile storage 63, and a memory 66 as a temporary storage area. The image processing device 60 also includes a display 64 such as a liquid crystal display, an input device 65 such as a keyboard and a mouse, and a network I/F (InterFace) 67 connected to the network 5. The CPU 61, storage 63, display 64, input device 65, memory 66, and network I/F 67 are connected to a bus 68. Note that the CPU 61 is an example of a processor in the present disclosure.

An image processing program 62 is stored in the storage 63. The CPU 61 reads the image processing program 62 from the storage 63, loads it into the memory 66, and executes the loaded image processing program 62.

Next, the functional configuration of the image processing apparatus according to this embodiment will be explained. FIG. 14 is a diagram showing the functional configuration of the image processing device according to this embodiment. As shown in FIG. 14, the image processing device 60 includes an image acquisition section 71, a segmentation section 72, a discrimination section 73, and a display control section 74. When the CPU 61 executes the image processing program 62, the CPU 61 functions as an image acquisition section 71, a segmentation section 72, a discrimination section 73, and a display control section 74.

The image acquisition unit 71 acquires a target image T0 to be processed from the image storage server 4. The target image T0 is an MRI image including the patient's rectum.

The segmentation unit 72 segments the object region included in the target image T0 and derives a mask image TM0 in which the object region included in the target image T0 is masked. In this embodiment, each region of the rectal cancer, the mucosal layer of the rectum, the submucosa of the rectum, the muscularis propria of the rectum, the subserosa of the rectum, and the background other than these, which are included in the target image T0, is segmented. A mask image TM0 in which a mask is applied to a region is derived. For this purpose, an SS model 72A constructed by the learning device according to the present embodiment is applied to the segmentation unit 72.

The determining unit 73 determines the stage of rectal cancer included in the target image T0, and outputs the determination result. For this purpose, the discrimination model 73A constructed by the learning device according to the present embodiment is applied to the discrimination unit 73. The target image T0 and the mask image TM0 of the target image T0 derived by the segmentation unit 72 are input to the discrimination model 73A, and outputs the determination result of the stage of rectal cancer included in the target image T0.

The display control unit 74 displays on the display 64 the mask image TM0 derived by the segmentation unit 72 and the determination result of the stage of rectal cancer derived by the determination unit 73. FIG. 15 is a diagram showing a display screen of the mask image TM0 and the discrimination results. As shown in FIG. 15, the target image T0, mask image TM0, and determination result 81 are displayed on the display screen 80. In addition, in FIG. 15, the determination result is "stage T3".

Next, the processing performed in this embodiment will be explained. FIG. 16 is a flowchart of image generation processing in this embodiment. First, the information acquisition unit 21 acquires the original image G0 and the mask image M0 from the image storage server 4 (step ST1). Next, the pseudo mask deriving unit 22 derives a pseudo mask image Mf0 by processing the mask (step ST2). Then, the pseudo image deriving unit 23 derives a pseudo image Gf0 having a region based on the pseudo mask (step ST3), and extracts information representing the stage of rectal cancer included in the pseudo image Gf0, the pseudo mask image Mf0, and the pseudo image Gf0. , is stored as teacher data in the storage 13 or the image storage server 4 together with existing teacher data (step ST4), and the process ends.

FIG. 17 is a flowchart of learning processing in this embodiment. First, the learning unit 24 acquires teacher data consisting of a combination of the pseudo image Gf0 and the pseudo mask image Mf0 (step ST11). The learning unit 24 then learns the SS model using the teacher data (step ST12). As a result, a trained SS model is constructed.

FIG. 18 is a flowchart of image processing in this embodiment. First, the image acquisition unit 71 acquires the target image T0 to be processed from the image storage server 4 (step ST21). Then, the segmentation unit 72 segments the target image T0 using the SS model 72A to derive a mask image TM0 (step ST22). Next, the discrimination unit 73 derives the discrimination result of the stage of rectal cancer included in the target image T0 using the discrimination model 73A (step ST23). Then, the display control unit 74 displays the mask image TM0 and the discrimination result on the display screen (step ST24), and the process ends.

Here, because there are few cases of rare diseases such as advanced cancer that occur less frequently, it is not possible to prepare a sufficient amount of training data to build a machine learning model for segmentation and stage discrimination. . For this reason, it is difficult to provide a machine learning model that can accurately segment rare diseases or accurately distinguish classes of rare diseases, such as stages of advanced cancer.

In this embodiment, the pseudo mask image Mf0 is derived by processing the mask in the mask image M0 for the original image G0, and the pseudo image Gf0 having a region based on the pseudo mask image Mf0 is derived. As a result, it is possible to prepare training data for constructing a segmentation model that includes a target object of a class that is not present in existing training data or is not present at all. For example, a pseudo image Gf0 including advanced rectal cancer can be prepared as training data. Therefore, by deriving a pseudo image Gf0 for rare diseases, accumulating it together with existing training data, and using it to train a learning model for segmentation and stage discrimination, it is possible to accurately segment rare diseases. A sufficient amount of training data can be prepared to enable the Therefore, it is possible to provide a machine learning model that can accurately segment rare diseases and accurately discriminate classes of rare diseases, such as stages of advanced cancer, for target images to be processed. It becomes possible.

Note that in the above embodiment, the target object is rectal cancer, but the present invention is not limited to this. The target object can be a lesion such as cancer or tumor in an organ or structure other than the rectum. For example, a pseudo mask and a pseudo image can be derived, and an SS model can be constructed using a bone spur in a joint as a target object. This will be described below as another embodiment.

Here, osteophytes are cartilage on articular surfaces that enlarge and proliferate, gradually becoming hard and ossified into "thorns", and are a characteristic finding of osteoarthritis that occurs around articular surfaces. It is one of the In such a case, a pseudo mask may be derived so as to form osteophytes using the bones forming the joint as target objects, and a pseudo image in which osteophytes are formed on the bones forming the joint may be derived.

FIG. 19 is a diagram showing a mask processing screen for bone spurs. As shown in FIG. 19, the mask processing screen 90 includes a mask image M0, a pseudo mask image Mf0 obtained by processing the mask in the mask image M0, a three-dimensional model 91 of the knee joint included in the original image G0, and a three-dimensional model 91. A scale 92 for adjusting the degree of deformation is displayed. Here, the original image G0 is an MRI image of the patient's knee joint. Furthermore, the three-dimensional knee joint model 91 represents the vicinity of the tibia joint. Note that no osteophyte is formed in the joint in the original image G0.

Note that for the purpose of explanation, a mask Ms0 is provided only to the region of the tibia to be processed in the mask image M0 shown in FIG. 19. Further, in the pseudo mask image Mf0, only the mask Msf0 of the tibia to be processed is given a reference numeral. Since the mask image M0 and the pseudo mask image Mf0 shown in FIG. 19 are before mask processing, the masks applied to each are the same.

The three-dimensional model 91 is a three-dimensional image derived by extracting only the region near the tibia joint in the original image G0 and performing volume rendering. By operating the input device 15, the operator can change the orientation of the three-dimensional model 91 in all directions. Furthermore, the mask processing location is designated by designating a desired position in the three-dimensional model 91 using the input device 15. Then, by moving the knob 92A of the scale 92 using the input device 15, the degree of mask processing is specified. As a result, the pseudo mask deriving unit 22 processes the three-dimensional model 91 to extend it from the specified position as the starting point, as shown in FIG. A mask Msf1 is provided to match the shape of the dimensional model 91.

At this time, the pseudo mask deriving unit 22 processes the mask while maintaining the three-dimensional continuity of the mask applied to the tibia. For example, when deforming the three-dimensional model 91 to extend a designated position, the degree of deformation is made smaller as the distance from the designated position in the three-dimensional model 91 increases. As a result, the three-dimensional model 91 can be transformed while maintaining the three-dimensional continuity of the original three-dimensional model 91, and as a result, the three-dimensional continuity of the mask applied to the tibia can be maintained. However, the mask can be processed to have bone spurs formed therein.

The pseudo-image deriving unit 23 derives a pseudo-image Gf0 including the tibia on which osteophytes are formed from the pseudo-mask image Mf0 on which osteophytes are formed. The learning unit 24 then learns the SS model using the pseudo mask image Mf0 and the pseudo image Gf0 in which bone spurs are formed as teacher data. Thereby, it is possible to construct a trained SS model that can accurately segment osteophytes in an MRI image including a knee joint. Therefore, by applying the SS model constructed in this manner to the segmentation unit 72 of the image processing apparatus according to this embodiment, it is possible to precisely segment the osteophyte region in an MRI image including the knee joint.

Furthermore, by using the pseudo image Gf0 that includes the tibia with osteophytes formed in the joint as training data, it is also possible to construct a discriminant model that determines the presence or absence of osteophytes in MRI images that include the tibia joint. . When constructing a discriminant model that determines the presence or absence of osteophytes, the original image that includes the tibia without osteophytes is also used as training data.

Note that in the other embodiments described above, the pseudo mask image Mf0 and further pseudo image Gf0 are derived by processing the original image G0 that does not include osteophytes, but the added lesions are limited to osteophytes. It's not something you can do. A pseudo mask image Mf0 and a pseudo image Gf0 are derived by processing the original image G0 so as to add a lesion to the original image G0, which includes an organ to which an arbitrary lesion is to be added but does not contain a lesion. You may also do so.
I will write an embodiment for converting the code later.

In each of the embodiments described above, when the pseudo mask deriving unit 22 processes a mask to derive the pseudo mask image Mf0, a constraint condition is set for the deformation of the mask, and the degree of processing of the mask is determined according to the constraint condition. It may also be possible to accept specifications. FIG. 21 is a diagram showing another example of the mask processing screen. The mask processing screen 100 shown in FIG. 21 includes a pseudo mask image Mf0, a condition list 101 for setting constraint conditions, a pull-down menu 102 for setting the degree of mask deformation, and a conversion for deriving a pseudo image. A button 103 is displayed.

In the pseudo mask image Mf0 displayed on the mask processing screen 100 shown in FIG. 21, a mask Msf0 is applied to the rectal cancer area, and a mask Msf1 is applied to the rectal area. It is possible to set whether or not to process the masks Msf0 and Msf1 included in the pseudo mask image Mf0 displayed on the mask processing screen 100. For example, by a predetermined operation such as moving the mouse cursor over a desired mask and right-clicking, it is possible to set whether or not to process the selected mask. In this embodiment, it is assumed that the rectal mask Msf1 is not processed and the rectal cancer mask Msf0 is processed. Note that in the following description, a mask set not to be processed will be referred to as a fixed mask.

In the condition list 101, constraint conditions for deformation of the mask to be processed are displayed in a selectable manner using check boxes. As shown in Figure 21, the constraint conditions include, for example, "rotation around the center of the fixed mask", "inscribed in the fixed mask", "distance from the fixed mask is □mm", and "rotation according to the center of gravity of the fixed mask". ” and “Rotation along edge of fixed mask” are displayed. The operator can select one or more desired constraint conditions from the condition list 101. Note that for the constraint condition "distance from the fixed mask is □mm", it is possible to input a numerical value of the distance. In this embodiment, it is assumed that "rotation around the center of gravity of the fixed mask" is selected as the constraint condition. Further, in the pull-down menu 102, similarly to the pull-down menu 26 shown in FIG. 6, the stages of rectal cancer T1, T2, T3ab, T3cd, T3MRF+, T4a, and T4b can be selected.

When "rotate around the center of the fixed mask" is selected as the constraint condition, the center of gravity C0 of the rectal mask Msf1, which is the fixed mask, is displayed on the pseudo mask image Mf0. Thereby, when the operator deforms and processes the mask Mfs0 using the input device 15, the deformation of the mask Msf0 is restricted so that it can only rotate about the center of gravity C0 of the fixed mask Msf1. FIG. 22 shows a state in which the mask Msf0 is rotated. Note that in FIG. 22, the mask Msf0 before deformation is shown by a virtual line. Further, the operator can select the stage of rectal cancer from the pull-down menu 102. In FIG. 22, T3ab, which is the same stage as the currently displayed rectal cancer, is displayed.

After the mask Msf0 is processed and the pseudo mask image Mf0 is derived, when the conversion button 103 is selected, the pseudo image derivation unit 23 converts the processed mask Msf0 and the selected stage of rectal cancer. A pseudo image Gf0 is derived.

FIG. 23 is a diagram showing another example of the mask processing screen. In the pseudo mask image Mf0 displayed on the mask processing screen 105 shown in FIG. 23, the region of the mask Mfs0 shown in FIG. 21 that overlaps with the rectum is set as a fixed mask Mfs2. As a result, only the mask Mfs0 existing in the region outside the rectum can be deformed. Further, here, it is assumed that "the distance from the fixed mask is □ mm" is selected as the constraint condition, and 3 mm is input as the distance. Thereby, when the operator deforms and processes the mask Mfs0 using the input device 15, the deformation of the mask Msf0 is restrained so that the farthest position from the fixed mask is 3 mm or less. FIG. 24 shows a state in which the mask Msf0 is deformed. As shown in FIG. 24, the protruding portion of the mask Msf0 is deformed so that the distance from the fixed masks Msf1 and Msf2 is a specified distance (ie, 3 mm). Note that the outline of the deformation mask Msf0 is shown by a virtual line.

After the mask Msf0 is processed and the pseudo mask image Mf0 is derived, when the conversion button 103 is selected, the pseudo image derivation unit 23 converts the processed mask Msf0 and the selected stage of rectal cancer. A pseudo image Gf0 is derived.

FIG. 25 is a diagram showing another example of the mask processing screen. In the mask image Mf0 displayed on the mask processing screen 106 shown in FIG. 25, masks Mfs0 and Msf1 similar to those in FIG. 21 are set. It is assumed that "rotation along the edge of fixed mask" is selected in the condition list 101. When "rotation along the edge of the fixed mask" is selected in the condition list 101, the intersection points C2 and C3 between the outline of the rectal mask Msf1, which is a fixed mask, and the outline of the mask Msf0 are displayed in the pseudo mask image Mf0. be done. Thereby, when the operator deforms and processes the mask Mfs0 using the input device 15, the deformation of the mask Msf0 is restricted so that it can only rotate along the edge of the fixed mask Msf1. FIG. 26 shows a state in which the mask Msf0 is rotated.

After the mask Msf0 is processed and the pseudo mask image Mf0 is derived, when the conversion button 103 is selected, the pseudo image derivation unit 23 converts the processed mask Msf0 and the selected stage of rectal cancer. A pseudo image Gf0 is derived.

FIG. 27 is a diagram showing another example of the mask processing screen. In the processing screen shown in FIG. 8 above, the mask Msf0 is transformed using the scale 42, but in the mask processing screen 107 shown in FIG. 27, the operator directly transforms the mask Msf0 by operating the input device 15. It was designed to do so. In this case, the operator can deform and process the mask Msf0 as shown in FIG. 28 by dragging the point C4 at the end of the mask Msf0 toward the point C5 using the input device 15. In this case, instead of dragging the point C4, only the point C5 may be specified using the input device 15. For example, the position of point C5 may be specified by double-clicking the position of point C5. In this case, the mask Msf0 is deformed and processed as shown in FIG. 28 along with the designation of the point C5.

Note that in the mask processing screen 107 shown in FIG. 28, the shape of the tip of the deformed mask Msf0 may be set. For example, a pull-down menu for selecting whether to add spinous processes may be displayed so that a mask having spinous processes shown in FIG. 9 can be generated. FIG. 29 is a diagram showing a mask processing screen on which a pull-down menu for selecting the presence or absence of spinous processes is displayed. As shown in FIG. 29, when "Present" is selected in the spinous process pull-down menu 109, when the operator drags the point C4 at the end of the mask Msf0 toward the point C5, the destination points C4, C5 The mask Msf0 is modified and processed so that a mask having a spinous process is added to the tip of the mask.

Further, in each of the embodiments described above, the pseudo mask image Mf0 is derived by deforming the rectal cancer to enlarge it or adding bone spurs, but the present invention is not limited to this. The technology of the present disclosure can also be applied when deriving a pseudo image Gf0 for a disease that causes contraction.

For example, when deriving a pseudo image Gf0 that includes a disease of vascular stenosis, if the pseudo mask image Mf0 is derived by processing the original image G0 that does not include vascular stenosis so that the blood vessel mask becomes thinner, then It is possible to derive a pseudo image Gf0.

Further, in the above embodiment, the pseudo image deriving unit 23 derives the pseudo image Gf0 having the same expression format as the original image G0, but the present invention is not limited to this. When the pseudo image derivation unit 23 derives the pseudo image, at least one of the density, color, and texture of the pseudo image Gf0 may be changed. In this case, a plurality of style images having predetermined density, color, and texture are stored in the storage 13, and a pseudo image Gf0 is derived to have density, color, or texture selected from the plurality of style images. Bye. FIG. 30 is a diagram showing a mask processing screen on which a plurality of style images are displayed. Note that the mask processing screen 120 shown in FIG. 30 displays a style image list 122 including a plurality of style images on the mask processing screen 40 shown in FIGS. 6 and 7, and also displays a pseudo image Gf0. .

The style image list 122 displayed on the mask processing screen 120 shown in FIG. 30 includes style images of six different styles. The upper row of the list 122 is style images 123A, 123B, 123C of three different colors, and the lower row is style images 123A, 124B, 124C of three different textures. Note that a style image of density may be included in addition to color and texture, or style images of two or all of density, color, and texture may be included. After the mask Msf0 is processed and the pseudo mask image Mf0 is derived, the operator selects a style image with a desired color and texture from the style image list 122 before selecting the execution button 29. Here, it is assumed that the color style image 123B is selected. After selecting the style image, when the operator selects the execution button 29, the pseudo image derivation unit 23 derives a pseudo image Gf0 containing rectal cancer in a color corresponding to the selected style image.

Further, in the above embodiment, for example, the information acquisition unit 21, pseudo mask derivation unit 22, pseudo image derivation unit 23, and learning unit 24 of the image generation device 20, the image acquisition unit 71, the segmentation unit 72, As a hardware structure of a processing unit (Processing Unit) such as the determination unit 73 and the display control unit 74 that executes various processes, the following various processors can be used. As mentioned above, the various processors mentioned above include the CPU, which is a general-purpose processor that executes software (programs) and functions as various processing units, as well as circuits such as FPGA (Field Programmable Gate Array) after manufacturing. A programmable logic device (PLD), which is a processor whose configuration can be changed, and a dedicated electrical device, which is a processor with a circuit configuration specifically designed to execute a specific process, such as an ASIC (Application Specific Integrated Circuit) Includes circuits, etc.

One processing unit may be composed of one of these various types of processors, or a combination of two or more processors of the same type or different types (for example, a combination of multiple FPGAs or a combination of a CPU and an FPGA). ). Further, the plurality of processing units may be configured with one processor.

As an example of configuring a plurality of processing units with one processor, firstly, as typified by computers such as a client and a server, one processor is configured with a combination of one or more CPUs and software, There is a form in which this processor functions as a plurality of processing units. Second, there are processors that use a single IC (Integrated Circuit) chip to implement the functions of an entire system including multiple processing units, as typified by System On Chip (SoC). be. In this way, various processing units are configured using one or more of the various processors described above as a hardware structure.

Furthermore, as the hardware structure of these various processors, more specifically, an electric circuit (Circuitry) that is a combination of circuit elements such as semiconductor elements can be used.

1, 2 Computer 3 Photographing device 4 Image storage server 5 Network 11, 61 CPU
12A Image generation program 12B Learning program 13, 63 Storage 14, 64 Display 15, 65 Input device 16, 66 Memory 17, 67 Network I/F
18, 68 Bus 20 Image processing device 21 Information acquisition section 22 Pseudo-mask derivation section 23 Pseudo-image derivation section 24 Learning section 26, 102 Pull-down menu 27 Processing button 28, 103 Conversion button 30 Rectum 31 Mucosal layer 32 Submucosal layer 33 Muscular layer propria 34 Subserosa layer 35 Rectal cancer 40, 90, 100, 105, 106, 107, 120 Mask processing screen 41, 91 3D model 41A Added area 42, 92 Scale 42A, 92A Pump 43 Infiltrated lymph node 44 Vascular invasion 50 Generator 51 Encoder 52 Decoder 53 Discriminator 62 Image processing program 71 Image acquisition section 72 Segmentation section 72A Segmentation model 73 Discrimination section 73A Discrimination model 74 Display control section 80 Display screen 81 Discrimination result 101 Condition list 122 Style image list 123A to 123C , 124A to 124C Style image C0 Center of gravity C2 to C5 points G0 Original image Gf0 Pseudo image M0 Mask image M1 to M8, Ms0, Msf0, Msf1 Mask Mf0 Pseudo mask image S0 Teacher data S1 Learning image S2 Learning mask image S3 Pseudo image T0 Target image TF0 Discrimination result TM0 Mask image z0 Latent expression

Claims (27)

comprising at least one processor;
The processor includes:
Obtaining an original image and a mask image in which a mask is applied to one or more areas representing each of one or more objects including the target object in the original image,
Deriving a pseudo mask image by processing the mask in the mask image,
An image generation device that derives, based on the original image and the pseudo mask image, a pseudo image that has a region based on a mask included in the pseudo mask image and has the same expression format as the original image. The image generation device according to claim 1, wherein the pseudo mask image and the pseudo image are used as training data for learning a segmentation model that segments the object included in an image. The image generation device according to claim 2, wherein the processor accumulates the pseudo mask image and the pseudo image as the teacher data. The image generation device according to claim 1, wherein the processor derives the pseudo mask image capable of generating the pseudo image including a target object of a class different from a class indicated by the target object. With respect to the medical image, the processor is configured to process the medical image so that at least one of the shape and degree of progression of the lesion is different from that of the lesion included in the original image, based on a lesion shape evaluation index that is an evaluation index in clinical practice. The image generation device according to claim 1, wherein the pseudo mask image is derived by processing a mask. Image generation according to claim 1, wherein the processor derives the pseudo mask image by processing the mask until a normal organ has a shape that is evaluated as a lesion based on a clinical measurement index with respect to a medical image. Device. 2. The pseudo image according to claim 1, wherein the processor refers to at least one style image having a predetermined density, color, or texture, and generates the pseudo image having a density, color, or texture according to the style image. Image generation device. The image generation device according to claim 1, wherein the processor receives an instruction regarding the degree of processing of the mask, and derives the pseudo mask image by processing the mask based on the instruction. The image generation device according to claim 8, wherein the processor receives a designation of a position of an end point of the mask after processing and a designation of a processing amount as an instruction of the degree of processing. 9. The image generation device according to claim 8, wherein the processor receives an instruction regarding the degree of processing of the mask according to preset constraint conditions. If the original image includes a plurality of the objects, and some regions of the target object and other objects other than the target object are in an inclusive relationship with each other, the mask image includes the regions that have the inclusive relationship with each other. The image generation device according to claim 1, wherein a mask different from that for areas that are not in an inclusive relationship is provided. When the other object in the inclusion relationship is a fixed object in the original image, the processor adjusts the mask given to the target object to the shape of the mask given to the fixed object. The image generation device according to claim 11, wherein the pseudo mask image is derived by processing the pseudo mask image. When the original image is a three-dimensional image, the processor derives the pseudo mask image by processing the mask while maintaining the three-dimensional continuity of the mask applied to the region of the target object. The image generation device according to claim 1. The original image is a three-dimensional medical image,
The image generation device according to claim 1, wherein the target object is a lesion included in the medical image. the medical image includes a rectum of a human body;
The target object is rectal cancer, and the other object other than the target object is at least one of the mucosal layer of the rectum, the submucosa of the rectum, the muscularis propria of the rectum, the subserosa of the rectum, and a background other than these. The image generation device according to claim 14. the medical image includes a joint of a human body;
15. The image generation device according to claim 14, wherein the target object is a bone forming a joint, and the object other than the target object is a background other than the bone forming the joint. comprising at least one processor;
The processor includes:
By performing machine learning using a set of a plurality of pseudo images and pseudo mask images generated by the image generation device according to claim 1 as training data, one or more objects including a target object included in an input image can be A learning device that builds a segmentation model that segments regions. The processor includes:
The learning device according to claim 17, further constructing the segmentation model by performing machine learning using a plurality of sets of original images and mask images as training data. A segmentation model constructed by the learning device according to claim 17. comprising at least one processor;
The processor segments one or more object regions including the target object included in the target image to be processed, using the segmentation model according to claim 19, thereby segmenting one or more object regions included in the target image to be processed. An image processing device that derives a mask image in which an object is masked. The image processing apparatus according to claim 20, wherein the processor determines the class of the target object masked in the mask image using a discriminant model that determines the class of the target object included in the mask image. Obtaining an original image and a mask image in which a mask is applied to one or more areas representing each of one or more objects including the target object in the original image,
Deriving a pseudo mask image by processing the mask in the mask image,
An image generation method that derives, based on the original image and the pseudo mask image, a pseudo image that has a region based on a mask included in the pseudo mask image and has the same expression format as the original image. By performing machine learning using a set of a plurality of pseudo images and pseudo mask images generated by the image generation method according to claim 22 as training data, one or more objects including the target object included in the input image are A learning method for building a segmentation model that segments regions. The one or more objects included in the target image to be processed are masked by segmenting the region of one or more objects including the target object included in the target image to be processed using the segmentation model according to claim 19. An image processing method for deriving a mask image. a step of acquiring an original image and a mask image in which a mask is applied to one or more regions representing each of one or more objects including the target object in the original image;
a step of deriving a pseudo mask image by processing a mask in the mask image;
and causing a computer to execute a procedure for deriving a pseudo image having a region based on a mask included in the pseudo mask image and having the same expression format as the original image, based on the original image and the pseudo mask image. Image generation program. By performing machine learning using a set of a plurality of pseudo images and pseudo mask images generated by the image generation method according to claim 22 as training data, one or more objects including the target object included in the input image are A learning program that causes a computer to execute the steps to build a segmentation model that segments an area. The one or more objects included in the target image to be processed are masked by segmenting the region of one or more objects including the target object included in the target image to be processed using the segmentation model according to claim 19. An image processing program that causes a computer to execute the steps to derive a mask image.