JP2019208903A - Medical image processor, medical image processing method, medical image processing program - Google Patents

Abstract

To provide a medical image processor capable of suppressing affection of noise in imaging of an analyte, and improving image quality of a plurality of phases of medical images.SOLUTION: A medical image processor comprises: an acquisition part for acquiring a plurality of medical images obtained by imaging multi times, a first analyte in time series; and a processor for generating a plurality of prediction images using a predictor which generates a prediction image corresponding to residual medical images out of the plurality of medical images, on the basis of some medical images out of the plurality of medical images. The processor sets at least two medical images forming a part of the plurality of medical images to training data, sets the residual medical image to teacher data, and sequentially changes a combination of some medical images and residual medical image, for causing the predictor to perform machine learning, and inputs some medical images out of the plurality of medical images to the predictor, then sequentially changes a combination of some medical images, for generating a plurality of prediction images.SELECTED DRAWING: Figure 6

Description

  The present disclosure relates to a medical image processing apparatus, a medical image processing method, and a medical image processing program.

  Conventionally, an image processing method is known in which a maximum value for each voxel is synthesized in volume data of a plurality of phases arranged in time series to generate synthesized volume data. In this image processing method, for example, a common virtual ray is projected on volume data of a plurality of phases, the value of one point that is the maximum value on the common virtual ray is obtained, and the maximum on the common virtual ray is obtained. The pixel value of each pixel of one image is determined using the value of one point which is a value (see, for example, Patent Document 1).

JP 2008-035895 A

  Since the pixel value of each pixel of one image obtained in Patent Document 1 is the maximum value of each virtual ray with respect to volume data, this image is also referred to as a 4DMIP (4D Maximum Intensity Projection) image. In the 4DMIP image, since the maximum value of each virtual ray with respect to the volume data is the pixel value, for example, when high luminance noise is included in a part of the voxel, this high luminance noise can be reflected as the maximum value in the pixel value.

  In addition, when a subject is imaged by a medical imaging device such as a CT (Computed Tomography) device, the subject is imaged in a real space including noise such as random noise, so that high luminance noise is completely removed. It is difficult to image the subject in a state where Therefore, in the 4DMIP image based on the volume data obtained by the medical imaging apparatus, high luminance noise is included, so that the pixel value (luminance value) is higher than the pixel value based on the actual signal component (real signal component). It becomes high, that is, the image tends to be white.

  The present disclosure has been made in view of the above circumstances, a medical image processing apparatus, a medical image processing method, and a medical image processing method capable of improving the image quality of a plurality of phases of medical images by suppressing the influence of noise during imaging of a subject. A medical image processing program is provided.

  According to one aspect of the present disclosure, an acquisition unit that acquires a plurality of medical images obtained by imaging a first subject a plurality of times in a time series, and some medical images of the plurality of medical images A processing unit that generates a plurality of predicted images using a predictor that generates a predicted image corresponding to the remaining medical images of the plurality of medical images, and the processing unit includes the plurality of predicted images. At least two medical images constituting a part of the medical images of the plurality of medical images are used as training data, and the remaining medical images of the plurality of medical images are used as teacher data. The medical image is sequentially changed, the predictor is machine-learned, a part of the plurality of medical images is input to the predictor, and the combination of the part of the medical images is sequentially changed. A plurality of the predicted images are generated.

  One aspect of the present disclosure is a medical image processing method in a medical image processing apparatus, the step of acquiring a plurality of medical images obtained by imaging a first subject a plurality of times in a time series, and the plurality of the plurality of medical images Generating a plurality of predicted images using a predictor that generates predicted images corresponding to the remaining medical images of the plurality of medical images based on some of the medical images. And the step of generating the predicted image uses at least two medical images forming a part of the plurality of medical images as training data, and uses the remaining medical images of the plurality of medical images as training data. Changing the combination of the part of medical images and the remaining part of the medical image in order as teacher data, causing the predictor to perform machine learning, and predicting part of the medical images of the plurality of medical images Input , Including the sequentially changing the combination of some of the medical image, and generating a plurality of the prediction image.

  One aspect of the present disclosure is a medical image processing program for causing a computer to execute each step of the medical image processing method.

  According to the present disclosure, it is possible to improve the image quality of a plurality of phases of medical images by suppressing the influence of noise during imaging of a subject.

1 is a block diagram illustrating a hardware configuration example of a medical image processing apparatus according to a first embodiment. 1 is a block diagram illustrating a functional configuration example of a medical image processing apparatus according to a first embodiment. The figure for demonstrating the production example of each prediction image, and the production example of an output MIP image The figure for demonstrating an example of the learning phase and prediction image generation phase of a predictor The flowchart which shows the outline | summary of the operation example of a medical image processing apparatus. The flowchart which shows the operation example which concerns on the machine learning and prediction image generation of the predictor by the medical image processing apparatus in 1st Embodiment. The figure which shows an example of the output MPR image produced | generated by 4DMIP composition, and the output MPR image produced | generated by Deep 4DMIP composition The figure which shows an example of the output ray-cast image produced | generated by 4DMIP composition, and the output ray-cast image produced | generated by Deep 4DMIP composition The figure which shows an example of the appropriateness information according to the synthetic | combination method of volume data of multiple phases The figure which shows the example of an image according to the synthetic | combination method of the volume data of multiple phases, and the site | part in a test object The figure which shows the example of an image according to the synthesis method and rendering method of volume data of multiple phases A graph showing an example of a change in pixel value in each phase for each part of the head of the subject The figure which shows an example of the output MIP image by which the volume data of the single phase of the to-be-observed subject were rendered The figure which shows an example of the output ray cast image by which the volume data of the single phase of the to-be-observed subject were rendered The figure which shows an example of the output MIP image by which the synthetic | combination volume data by which 4DMIP synthesis | combination of the volume data of the several phase of the to-be-observed subject was rendered The figure which shows an example of the output ray cast image by which the synthetic | combination volume data by which 4DMIP synthesis | combination of the volume data of the several phase of the to-be-observed subject was rendered An example of an output MIP image in which composite volume data in which volume data of a plurality of phases of a subject to be observed is combined with Deep 4DMIP is rendered when the predictor performs machine learning using volume data of a subject other than the subject to be observed Illustration An example of an output raycast image in which composite volume data in which volume data of a plurality of phases of a subject to be observed is combined with Deep 4DMIP is rendered when the predictor performs machine learning using the volume data of the subject other than the subject to be observed Figure showing FIG. 3 is a block diagram illustrating an example of a functional configuration of a medical image processing apparatus according to a second embodiment. The figure for demonstrating an example of the machine learning of a predictor and a discriminator The figure which shows an example of the real volume data of the same subject as a prediction object, prediction volume data, and difference volume data The figure which shows an example of the real volume data, the prediction volume data, and difference volume data of the subject different from the subject to be predicted The figure which shows an example of the output ray cast image at the time of phase average 4DMIP composition The figure which shows an example of the output MPR image at the time of phase average 4DMIP composition, and the pixel value of the specific range in an output MPR image The figure which shows an example of the output raycast image at the time of Deep GAN 4DMIP composition The figure which shows an example of the output MPR image at the time of Deep GAN 4DMIP composition, and the pixel value of the specific range in an output MPR image The figure which shows an example of the output MPR image of the brain at the time of phase average 4DMIP composition The figure which shows an example of the output MPR image of the brain at the time of Deep GAN 4DMIP composition The figure which shows an example of the image by which real volume data, prediction volume data, and difference volume data at the time of Deep 4DMIP composition were rendered The figure which shows an example of the image by which real volume data at the time of Deep GAN 4DMIP composition, prediction volume data, and difference volume data were rendered The flowchart which shows the operation example which concerns on the machine learning and prediction image generation of the predictor and discriminator by the medical image processing apparatus in 2nd Embodiment.

  Hereinafter, embodiments of the present disclosure will be described with reference to the drawings.

(First embodiment)
FIG. 1 is a block diagram illustrating a configuration example of a medical image processing apparatus 100 according to the first embodiment. The medical image processing apparatus 100 includes a port 110, a user interface (UI) 120, a display 130, a processor 140, and a memory 150.

  A CT apparatus 200 is connected to the medical image processing apparatus 100. The medical image processing apparatus 100 acquires volume data from the CT apparatus 200 and performs processing on the acquired volume data. The medical image processing apparatus 100 may be configured by a PC (Personal Computer) and software installed in the PC.

  The CT apparatus 200 irradiates a living body with X-rays and captures an image (CT image) using the difference in X-ray absorption by tissues in the body. Examples of the living body include a human body. A living body is an example of a subject.

  A plurality of CT images may be taken in time series. The CT apparatus 200 generates volume data including information on an arbitrary location inside the living body. Arbitrary locations inside the living body may include various organs (for example, brain, heart, kidney, large intestine, small intestine, lung, breast, breast gland, prostate, lung). By capturing a CT image, pixel values (CT value, voxel value) of each pixel (voxel) in the CT image are obtained. The CT apparatus 200 transmits volume data as a CT image to the medical image processing apparatus 100 via a wired line or a wireless line.

  Specifically, the CT apparatus 200 includes a gantry (not shown) and a console (not shown). The gantry includes an X-ray generator (not shown) and an X-ray detector (not shown), detects X-rays transmitted through the human body by imaging at a predetermined timing designated by the console, Obtain line detection data. The X-ray generator includes an X-ray tube (not shown). The console is connected to the medical image processing apparatus 100. The console acquires a plurality of X-ray detection data from the gantry and generates volume data based on the X-ray detection data. The console transmits the generated volume data to the medical image processing apparatus 100. The console may include an operation unit (not shown) for inputting patient information, imaging conditions related to CT imaging, imaging conditions related to administration of a contrast agent, and other information. The operation unit may include an input device such as a keyboard and a mouse.

  The CT apparatus 200 can also acquire a plurality of three-dimensional volume data by continuously capturing images and generate a moving image. The moving image data using a plurality of three-dimensional volume data is also referred to as 4D (four-dimensional) data.

  The CT apparatus 200 may capture a CT image at each of a plurality of timings. The CT apparatus 200 may capture a CT image in a state where the subject is contrasted. The CT apparatus 200 may capture a CT image in a state where the subject is not contrasted.

  A port 110 in the medical image processing apparatus 100 includes a communication port and an external apparatus connection port, and acquires volume data obtained from a CT image. The acquired volume data may be sent immediately to the processor 140 for various processing, or may be stored in the memory 150 and then sent to the processor 140 for various processing when necessary. Further, the volume data may be acquired via a recording medium or a recording medium.

  Volume data imaged by the CT apparatus 200 may be sent from the CT apparatus 200 to a picture data server (PACS: Picture Archiving and Communication Systems) (not shown) and stored. The port 110 may acquire volume data from the image data server instead of acquiring from the CT apparatus 200. Thus, the port 110 functions as an acquisition unit that acquires various data such as volume data.

  The UI 120 may include a touch panel, a pointing device, a keyboard, or a microphone. The UI 120 receives an arbitrary input operation from the user of the medical image processing apparatus 100. Users may include doctors, radiologist, or other paramedic staff.

  The UI 120 accepts operations such as designation of a region of interest (ROI) in volume data and setting of luminance conditions. Regions of interest may include regions of various tissues (eg, blood vessels, bronchi, organs, bones, brain, heart, feet, neck, blood flow). The tissue may widely include living tissues such as a diseased tissue, a normal tissue, an organ, and an organ. Further, the UI 120 may accept operations such as specifying a region of interest and setting a luminance condition in volume data or an image based on the volume data (for example, a three-dimensional image or a two-dimensional image described later).

  The display 130 may include an LCD (Liquid Crystal Display) and displays various types of information. Various types of information may include a three-dimensional image or a two-dimensional image obtained from volume data. The three-dimensional image may include a volume rendering image, a surface rendering image, a virtual endoscopic image (VE image), a CPR (Curved Planar Reconstruction) image, and the like. The volume rendering image is a RaySum image (also simply referred to as “SUM image”), a MIP (Maximum Intensity Projection) image, a MinIP (Minimum Intensity Projection) image, an Average image, or a Raycast image. May be included. The two-dimensional image may include an axial image, a sagittal image, a coronal image, an MPR (Multi Planer Reconstruction) image, and the like.

  The memory 150 includes a primary storage device of various ROM (Read Only Memory) and RAM (Random Access Memory). The memory 150 may include a secondary storage device such as a hard disk drive (HDD) or a solid state drive (SSD). The memory 150 may include a tertiary storage device such as a USB memory or an SD card. The memory 150 stores various information and programs. The various information may include volume data acquired by the port 110, an image generated by the processor 140, setting information set by the processor 140, and various programs. The memory 150 is an example of a non-transitory recording medium on which a program is recorded.

  The processor 140 may include a CPU (Central Processing Unit), a DSP (Digital Signal Processor), or a GPU (Graphics Processing Unit). The processor 140 functions as a processing unit 160 that performs various processes and controls by executing a medical image processing program stored in the memory 150.

  FIG. 2 is a block diagram illustrating a functional configuration example of the processing unit 160.

  The processing unit 160 includes an area extraction unit 161, an image generation unit 162, a registration processing unit 163, a prediction processing unit 164, and a display control unit 166. The prediction processing unit 164 includes a predictor (Predictor) 171.

  The processing unit 160 controls each unit of the medical image processing apparatus 100. Each unit included in the processing unit 160 may be realized as a different function by one piece of hardware, or may be realized as a different function by a plurality of pieces of hardware. In addition, each unit included in the processing unit 160 may be realized by a dedicated hardware component.

  The region extraction unit 161 may perform a segmentation process on the volume data. In this case, the UI 120 receives an instruction from the user, and the instruction information is sent to the area extraction unit 161. The region extraction unit 161 may perform segmentation processing from volume data and extract a region of interest based on the instruction information by a known method. Further, the region of interest may be manually set according to a detailed instruction from the user. When the observation target is determined in advance, the region extraction unit 161 may extract a region of interest including the observation target by performing segmentation processing from the volume data without a user instruction. The extracted region may include regions of various tissues (for example, blood vessels, bronchi, organs, bones, brains, hearts, feet, necks, blood flows, breasts, breasts, tumors).

  Based on the plurality of volume data (for example, volume data arranged in time series) acquired by the port 110, the image generation unit 162 combines the plurality of volume data to generate one volume data (for example, combined volume data). It's okay. The image generation unit 162 may generate a 3D image or a 2D image based on the volume data acquired or generated via the port 110 or the like. Based on the one or more volume data acquired by the port 110, the image generation unit 162 uses the specified region or the region extracted by the region extraction unit 161 to generate volume data (for example, synthetic volume data) or 3 A two-dimensional image or a two-dimensional image may be generated.

  The image generation unit 162 may generate volume data (for example, combined volume data) as an output image based on the registered multiple-phase volume data. The image generation unit 162 may generate a three-dimensional image or a two-dimensional image as an output image based on the registered multiple-phase volume data. The image generation unit 162 may generate a three-dimensional image as an output image based on the registered three-phase three-dimensional image. The image generation unit 162 may generate a two-dimensional image as an output image based on the registered two-phase two-dimensional image. The plurality of phases of data and images may be a plurality of data and images arranged in time series.

  The image generation unit 162 may generate one composite volume data with the maximum value of each voxel as the pixel value in the multi-phase volume data. This synthesis method is also referred to as 4DMIP synthesis. As a specific method of 4DMIP synthesis, the method described in Patent Document 1 may be used. In 4DMIP composition, the pixel value tends to be large.

  The image generation unit 162 divides a plurality of phases (for example, eight phases) into a plurality of phase groups (for example, three phase groups), calculates an average value of pixel values of each voxel in each volume data in the plurality of phase groups, One synthetic volume data may be generated with the maximum value among the average values of the pixel values of each voxel in each phase group as the pixel value. This synthesis method is also referred to as phase average 4DMIP synthesis.

  The image generation unit 162 may apply a Gaussian filter to the volume data obtained by the phase average 4DMIP synthesis, and generate one synthesized volume data as the pixel value of each voxel. This synthesis method is also referred to as a filtered phase average 4DMIP synthesis.

  The image generation unit 162 may generate one composite volume data by using a value obtained by adding the pixel values of the voxels in the volume data of a plurality of phases as a pixel value. This synthesis method is also referred to as TSUM synthesis. In TSUM synthesis, both noise components and actual signal components are easily deleted.

  The image generation unit 162 may generate one composite volume data with the maximum value of each voxel in the multi-phase predicted volume data generated by the machine-learned predictor 171 as a pixel value. This synthesis method may be referred to as Deep 4DMIP synthesis. In Deep 4DMIP synthesis, a specific method implemented by 4DMIP synthesis may be applied to derivation of the maximum value of each voxel in the predicted volume data of a plurality of phases.

The registration processing unit 163 detects movement of each unit included in the volume data based on a plurality of volume data (CT images) obtained in time series, and generates movement information. In this case, the registration processing unit 163 performs motion analysis on the deformation of the CT image between a plurality of phases based on the CT images of a plurality of phases, and acquires motion information in the CT images. Specific methods of motion analysis are described in, for example, Reference Patent Document 1 and Reference Patent Document 2. These are examples of non-rigid registration, but may be rigid registration.
(Reference Patent Document 1: US Pat. No. 8311300)
(Reference Patent Document 2: Japanese Patent No. 5408493)

  The registration processing unit 163 recognizes to which position an arbitrary position in the subject has moved by analyzing the movement of each point or observation site using the movement information. The registration processing unit 163 registers (positions) the volume data of a plurality of phases arranged in time series based on the result of the motion analysis.

  The prediction processing unit 164 includes a predictor 171 that generates predicted volume data based on the volume data imaged by the CT apparatus 200. The volume data imaged by the CT apparatus 200 is volume data imaged in real space. The predictor 171 may be configured by hardware or software. The predictor 171 generates predicted volume data corresponding to the remaining volume data of the plurality of phase volume data based on a part of the volume data of the plurality of phases of the same subject. Good. Volume data imaged in real space is data on which noise (for example, random noise) is superimposed, but predicted volume data is data in which this noise is suppressed.

  The prediction processing unit 164 causes the predictor 171 to perform machine learning (Machine Learning) using the training data and the teacher data. Machine learning may include a deep neural network (DNN), that is, so-called deep learning. In the machine learning, the prediction processing unit 164 may use a part of the volume data of the plurality of phases as training data. The prediction processing unit 164 may use the remaining volume data among the plurality of phases of volume data as teacher data. Then, the prediction processing unit 164 sequentially changes the combination of the part of the volume data serving as training data, and sequentially changes the remaining volume data serving as teacher data in accordance with this, and causes the predictor 171 to perform machine learning. You may let me. The machine learning method is arbitrary, and may be CNN (Convolutional Neural Network), Random Forest, Structured SVM (Support Vector Machine), NN (Nearest Neighbor), or the like. In this embodiment, CNN will be mainly described.

  The image generation unit 162 generates a three-dimensional output image (display target image) based on the volume data captured in at least one real space and at least one prediction volume data generated by the predictor 171. An image or a two-dimensional image may be generated. The image generation unit 162 includes a three-dimensional image or two-dimensional image based on volume data captured in at least one real space, and a three-dimensional image or two-dimensional image based on at least one predicted volume data generated by the predictor 171. Based on the image, a three-dimensional image or a two-dimensional image as an output image may be generated.

  The display control unit 166 displays various data, information, and images on the display 130. The display control unit 166 may visualize volume data (output volume data, for example, synthesized volume data) indicated in a three-dimensional space. In this case, the image generation unit 162 may generate a three-dimensional image by rendering the output volume data by an arbitrary rendering method. In addition, the image generation unit 162 may extract an arbitrary cross section of the output volume data and generate a two-dimensional image. The display control unit 166 may visualize the output volume data by displaying the generated three-dimensional image or two-dimensional image.

  The volume data, the three-dimensional image, and the two-dimensional image are examples of medical images. In the following, volume data is mainly exemplified, but the present invention can also be applied to other medical images.

Next, the detailed operation of the predictor 171 will be described.
FIG. 3 is a diagram for explaining a generation example of each predicted image and a generation example of an output image. Note that “0” to “7” shown in FIG. 3 indicate phase numbers.

  The predicted image is an image generated by the predictor 171, and includes volume data (predicted volume data) generated by the predictor 171, a three-dimensional image or a two-dimensional image (predicted three-dimensional image) generated by the predictor 171. Image, predicted two-dimensional image), or the like. The output image is an image to be output. Volume data (output volume data) output (for example, displayed) by the display 130 or the like, 3D image or 2D image output by the display 130 or the like (output 3D image or Output two-dimensional image). Here, the use of predicted volume data as the predicted image and output volume data as the output image is mainly exemplified.

  The prediction processing unit 164 acquires an input image group of n (n: positive integer) (n = 8 in FIG. 3) phase. Each input image included in the input image group is input to the predictor 171. Each input image included in the input image group may be volume data from the CT apparatus 200 acquired via the port 110. Each input image included in the input image group is input to the predictor 171. The input image may include volume data of a non-contrast phase captured in a non-contrast-enhanced state, or volume data of a contrast phase captured in a contrast-enhanced state of contrast.

  The predictor 171 operates according to the control of the prediction processing unit 164. The predictor 171 predicts the i-th volume data of the n phases based on the volume data of the phases other than the i-phase of the n phases, and generates a predicted image YGi. The predicted image YGi may be predicted volume data.

  For example, in FIG. 3, the predictor 171 uses the first phase (i = 1) of the 8 phases (the 0th phase to the 7th phase) as the volume data of the phases other than the 1 phase (0). , 2, 3, 4, 5, 6 and 7), the prediction image YG1 is generated. Further, the predictor 171 converts the volume data of the second phase (i = 2) of the eight phases into phases other than the two phases of the eight phases, that is, 0, 1, 3, 4, 5, 6, 7 Prediction is performed based on the volume data of the seven phases, and a predicted image YG2 is generated.

  The predictor 171 sequentially changes the value of i to generate predicted images YGi corresponding to the i-th volume data. In FIG. 3, the predictor 171 generates eight predicted images YG0 to YG7 corresponding to the eight phases. As a result, a predicted image group including n predicted images YGi is obtained.

  Note that the predictor 171 may predict the i-th volume data of the n phases based on the volume data of some phases other than the i phase of the n phases, and generate the predicted image YGi. Good. For example, the predictor 171 converts the volume data of the first phase (i = 1) out of the 8 phases (the 0th phase to the 7th phase) into some phases (0, The prediction image YG1 may be generated by prediction based on volume data of the second, fourth, and sixth phases).

  The predictor 171 may perform machine learning using training data and teacher data. In machine learning, the predictor 171 may use the i-th volume data of n phases as teacher data. The predictor 171 may use volume data of phases other than the i phase of the n phases as training data. Further, the predictor 171 may use volume data of a part of the phases other than the i phase of the n phases as training data. The predictor 171 may perform machine learning to input training data and output teacher data. In machine learning, the relationship between training data and teacher data may be non-linear.

  Therefore, the predictor 171 performs the machine learning to obtain the desired volume data of the i phase of the n phase without depending on the volume data of the phase other than the i phase of the n phase as the training data. Can be derived. In this case, it can be expected that the structure of the subject's tissue can be satisfactorily left in the predicted image.

  Further, the training data may be volume data of a phase other than the i phase that is captured in real space. The teacher data may be i-phase volume data captured in real space. Therefore, machine learning is performed based on the volume data of each phase to which various noises are added so that one volume data of each phase is obtained. Therefore, the medical image processing apparatus 100 can perform machine learning so that various noises cancel each other as a result of machine learning to generate a predicted image. Therefore, the medical image processing apparatus 100 can obtain, as predicted volume data, volume data from which noise has been removed from volume data captured in an i-phase real space.

  Further, the predictor 171 may set the amount of training data or input data input at a time for machine learning or prediction image generation for each pixel (voxel) of volume data or to a certain size ( Each pixel group included in the patch) or all the pixels in the volume data may be used. When the training data and the input data are set for each pixel, the predictor 171 refers to only one voxel to be processed and does not refer to surrounding voxels, so that the propagation of information in the spatial direction is reduced. When training data and input data are set for each pixel group included in the patch or for all pixels, the predictor 171 takes into account the propagation of information in the spatial direction because it refers to the surrounding voxels in addition to the voxels to be processed. It is possible to improve machine learning accuracy and prediction accuracy.

  Further, as the predictor 171, one common predictor 171 may be provided to generate a predicted image of each phase, or different predictors 171 may be provided for each phase. This is because the predictor 171 performs machine learning and generates a predicted image in each phase.

  In addition, when machine learning is performed at the end of a plurality of phases (for example, the 0th phase or the 7th phase in 8 phases) or when a predicted image is generated, the volume data of the phases before and after that may be absent. In this case, as the absent volume data, assume that the volume data is a black image in which all the pixel values of the volume data are 0, or the opposite end (for example, in the case of the seventh phase, the zeroth phase). In the case of the 0th phase, it may be assumed that there is volume data of the 7th phase).

  In addition, learning data (for example, training data and teacher data) used for machine learning in the predictor 171 may be stored and prepared in advance in the memory 150 or the like. Therefore, the learning data may include a plurality of phases of volume data arranged in time series in the subject.

  The predictor 171 may perform machine learning each time it acquires volume data of a plurality of phases arranged in time series from the CT apparatus 200, and may update the machine learning result (machine learning result) and store it in the memory 150. . The predictor 171 may acquire the machine learning result stored in the memory 150 at a timing when learning data is necessary, such as when a predicted image is generated.

  The image generation unit 162 combines the n predicted images YGi included in the predicted image group by any combining method, and generates output volume data. For example, when performing Deep 4DMIP composition, the image generation unit 162 derives the maximum pixel value of the same pixel along the time axis.

  When the predicted image is predicted volume data, the image generation unit 162 may generate one output volume data as an output image based on the n predicted volume data. In this case, the image generation unit 162 may perform volume rendering on the output volume data to generate an output 3D image or an output 2D image. In addition, when the predicted image is a predicted three-dimensional image, the image generation unit 162 may generate one output three-dimensional image as an output image based on the n predicted three-dimensional images. Further, when the predicted image is a predicted two-dimensional image, the image generation unit 162 may generate one output two-dimensional image as an output image based on the n predicted two-dimensional images.

  The image generation unit 162 may visualize the output volume data by various volume rendering methods. For example, when the output volume data is visualized by the MIP method, this output image is also referred to as an output MIP image. Further, when the output image is visualized by the ray cast method, this output image is also referred to as an output ray cast image. Further, when the output image is visualized by MPR, this output image is also referred to as an output MPR image.

  The registration processing unit 163 may register (align) the n predicted images YGi. That is, the registration processing unit 163 may align the predicted images so that each pixel of the predicted image YGi corresponds to each pixel of another predicted image. Thereby, the medical image processing apparatus 100 can suppress the positional shift of each predicted image, and can improve the accuracy of the pixel value of each pixel of the output image. Further, the registration processing unit 163 may perform registration (position alignment) for n input image groups. That is, the registration processing unit 163 may align the input images so that each pixel of the input image corresponds to each pixel of the other input image. Thereby, the medical image processing apparatus 100 can suppress the positional shift of each input image, and can improve the accuracy of the pixel value of each pixel of each predicted image and the output image. The registration processing unit 163 may register the input image and the predicted image.

  FIG. 4 is a diagram for explaining an example of a learning phase and a prediction image generation phase of the predictor 171. The learning phase is a phase in which the predictor 171 performs machine learning. The predicted image generation phase is a phase in which a predicted image is generated by a machine-learned predictor 171.

  The memory 150 may include a learning DB 151 that stores learning data (for example, a study) for machine learning by the predictor 171. The learning data may be learning data for a subject to be observed (a prediction image generation target), or may be learning data for a subject that is not an observation target (not a prediction image generation target). That is, the subject related to the learning data and the subject related to the actual prediction image may be the same subject or different subjects. The learning DB 151 may hold a large amount of learning data for machine learning by the predictor 171 and may include at least one of learning data for a subject to be observed and learning data for a subject not to be observed. . Further, the learning DB 151 may hold a learning result.

  In FIG. 4, V indicates volume data. i and k indicate phase numbers. A plurality of volume data having different phases may be registered. V′i is predicted volume data generated corresponding to the phase i-th volume data.

  In the learning phase, Vk (k ≠ i) is used as training data among the volume data of a plurality of phases acquired through the port 110 for machine learning. That is, the predictor 171 inputs volume data of phases other than the i-th as training data. The training data may be training data for machine learning held in the learning DB 151. The predictor 171 uses Vi as teacher data among the volume data of a plurality of phases acquired via the port 110. The teacher data may be machine learning teacher data stored in the learning DB 151. The predictor 171 performs machine learning so as to output Vi as teacher data when Vk (k ≠ i) as training data is input. In this case, the predictor 171 actually outputs V′i, but performs machine learning so that V′i approaches Vi. The predictor 171 stores the result of machine learning in the learning DB 151. The prediction processing unit 164 may cause the predictor 171 to learn using observation target data as training data and teacher data. The prediction processing unit 164 may cause the predictor 171 to learn using data other than the observation target as training data and teacher data. Further, the prediction processing unit 164 may cause the predictor 171 to learn step by step by exchanging a large number of data other than the observation target as training data and teacher data.

  In addition, the predictor 171 performs known overlearning measures such as regularization processing, dropout processing (learning by randomly deactivating nodes), k-division cross-validation, and the like in order to suppress overlearning. Good. Regularization processing provides a penalty for increasing complexity when learning a model. Regularization may include, as a penalty, L1 regularization using the sum of absolute values of learning model parameters, L2 regularization using the sum of squares of learning model parameters, and the like.

  In the predicted image generation phase, the predictor 171 uses Vk (k ≠ i) among the volume data (target for generating predicted images) of a plurality of phases to be observed acquired via the port 110 for generating predicted images. Input data. This input data may be the same as the training data acquired for machine learning. The predictor 171 refers to the machine learning result stored in the learning DB 151 and calculates V′i that is output data based on Vk (k ≠ i) that is input data. In this case, the machine-learned predictor 171 can derive V′i having a value close to Vi that is i-phase volume data acquired via the port 110. For example, in the case of n phases (for example, n = 8), the predictor 171 assumes that i = 0, 1,... 7 and outputs V′0, V′1,. 7 is generated. This output data becomes the predicted volume data.

  The image generation unit 162 acquires V′0, V′1,... V′7 as predicted volume data of each phase of i = 0, 1,. Combined to generate combined volume data. Note that the registration processing unit 163 may perform registration (registration) of each predicted volume data before the synthesized volume data is generated based on each predicted volume data. The processing unit 160 may repeat the generation and registration of the composite volume data by the image generation unit 162 in order. Thereby, the medical image processing apparatus 100 can perform image processing with further noise removal.

  FIG. 5 is a flowchart showing an outline of an operation example of the medical image processing apparatus 100.

  First, the port 110 acquires volume data of a plurality of phases arranged in time series from the CT apparatus 200 (S11). The registration processing unit 163 aligns the acquired volume data of a plurality of phases (S12).

  The prediction processing unit 164 generates a machine-learned predictor 171 based on a plurality of phases of volume data (machine learning data) (S13). Details of the specific machine learning of the predictor 171 will be described later.

  The prediction processing unit 164 uses a plurality of phases of volume data (prediction image generation data) and the predictor 171 to generate a plurality of prediction volume data corresponding to the volume data of each phase. The prediction processing unit 164 generates composite volume data based on the plurality of predicted volume data (S14).

  The image generation unit 162 performs volume rendering on the combined volume data, and generates a volume rendering image (for example, a MIP image) (S15).

  According to the processing of FIG. 5, the medical image processing apparatus 100 can improve the generation accuracy of the predicted volume data by generating the predicted volume data using the machine-learned predictor 171. Also, the medical image processing apparatus 100 can suppress the positional deviation of each voxel in each volume data by aligning the volume data of each phase, and can improve the statistical accuracy of the pixel value for each voxel. Therefore, the medical image processing apparatus 100 can obtain more accurate pixel values, and can more accurately draw the state of the subject based on the synthesized volume data.

  FIG. 6 is a flowchart illustrating an operation example related to machine learning and prediction image generation of the predictor 171 by the medical image processing apparatus 100. In FIG. 6, it is assumed that there are 0 to n phases.

  First, the port 110 acquires volume data V0 to Vn of a plurality of phases arranged in time series from the CT apparatus 200 (S21). The registration processing unit 163 aligns the acquired multi-phase volume data V0 to Vn (S22).

  The prediction processing unit 164 causes the predictor 171 to perform machine learning for the phases i = 0 to n using the volume data V0 to Vn (excluding Vi) (S23). In this case, the prediction processing unit 164 uses the volume data V0 to Vn (excluding Vi) acquired in S21 as training data for machine learning, and uses the volume data Vi acquired in S21 as teacher data for machine learning. .

  The prediction processing unit 164 inputs volume data V0 to Vn (excluding Vi) to the predictor 171 for the phases i = 0 to n, and generates predicted volume data V′i (S24). Therefore, predicted volume data V′0 to V′n are obtained.

  The image generation unit 162 calculates the maximum value of the pixel value (voxel value) in each corresponding voxel for the predicted volume data V′0 to V′n, and generates maximum value volume data Vmax (S25). That is, the image generation unit 162 generates the maximum value volume data Vmax by Deep 4DMIP composition.

  The image generation unit 162 visualizes the maximum value volume data Vmax by, for example, the MIP method (S26). That is, the image generation unit 162 performs volume rendering on the maximum volume data Vmax using, for example, the MIP method, and generates an output image (output MIP image).

  According to the process of FIG. 6, the medical image processing apparatus 100 can visualize the volume data of each phase. For example, the medical image processing apparatus 100 can easily be visually recognized by the user even if the observation target is a time-sequential observation target that moves and appears in each pixel of the volume data depending on the phase. For example, when observing a non-specimen while contrasting, the region of the subject to be imaged by the contrast agent varies depending on the phase as the contrast agent flows. Depending on the phase, for example, only a part of the blood vessel may be imaged, but by synthesizing the volume data of each phase, the pixel value related to the observation target of each phase can be expressed in one volume data. Therefore, when observing a phase in which only the artery is contrasted or a phase in which only the vein is contrasted, it is possible to visualize the change in time series by image processing with one volume data without continuing to flow a contrast medium into the subject. . Therefore, the medical image processing apparatus 100 can reduce the time for injecting the contrast agent, and can reduce the burden on the kidney of the subject. Further, the medical image processing apparatus 100 can generate predictive volume data in which noise is suppressed by performing machine learning of the predictor 171 and can visualize a moderately bright image.

  FIG. 7A is a diagram showing an output MPR image gzx1 generated by 4DMIP composition and an output MPR image GZ1 generated by Deep 4DMIP composition. FIG. 7B is a diagram illustrating an output ray-cast image gzx2 generated by 4DMIP combining and an output ray-cast image GZ2 generated by Deep 4DMIP combining. In the output MPR images gzx1, GZ1 and the output ray cast images gzx2, GZ2, the same part of the same subject is shown.

  In FIG. 7A, in the output MPR image gzx1 generated by the 4DMIP composition, by applying the maximum value, the high luminance noise is easily selected as the pixel value, and is generally white, that is, the luminance value ( Pixel value) is high. On the other hand, the output MPR image GZ1 generated by the Deep 4DMIP synthesis is less in noise and reduced in overall whiteness compared to the output MPR image gzx1, and has a high diagnostic significance. It can be determined that it is a value. Therefore, the diagnostic accuracy by the user can be improved.

  In FIG. 7B, in the output MPR image gzx2 generated by the 4DMIP synthesis, noise other than the actual blood vessel is also reflected, and it is difficult to see the actual blood vessel. On the other hand, the output MPR image GZ2 generated by the Deep 4DMIP composition is suppressed in noise and the drawing of noise components is reduced as compared with the output MPR image gzx2, so that a blood vessel or the like that is meaningful for diagnosis is discriminated. It is easy. Therefore, the diagnostic accuracy by the user can be improved.

  That is, in the output MPR image gzx1 generated by the 4DMIP composition, the composite volume data is created by simply using the maximum value for each voxel of the volume data of a plurality of phases, so that high luminance noise is easily visualized. In volume data of a plurality of phases, it is difficult to predict noise in volume data of an arbitrary phase based only on volume data of a phase in the vicinity of the phase. In the case of volume data obtained by imaging with the CT apparatus 200, it is possible to predict the state of noise in the target pixel by using a neighboring pixel located in the vicinity of the target pixel in the same phase. Further, a signal other than noise in the volume data of the target phase (for example, a steady portion or a blood vessel whose CT value changes) can be predicted based on a signal other than noise in the volume data of the previous and subsequent phases. This is because signals other than noise are related to some extent in successive phases.

  Therefore, it can be assumed that the medical image processing apparatus 100 can distinguish between a noise component in volume data and a component other than noise in volume data (for example, a contrast agent).

  FIG. 8 is a diagram illustrating an example of suitability information for each method of synthesizing volume data of a plurality of phases. Synthesis methods may include 4DMIP synthesis, phase average 4DMIP synthesis, phase average 4DMIP synthesis with filter, Deep 4DMIP synthesis, and the like. Note that FIG. 8 illustrates a case where a plurality of phases are divided into three phase groups in the phase average 4DMIP synthesis (the same applies to the description of the following drawings).

  When the synthetic volume data is generated by 4DMIP synthesis, there is a lot of noise in the synthetic volume data, and the thin blood vessels (thin blood vessels) in the synthetic volume data become brighter (whiter). ) Brighten up. That is, the pixel value becomes the largest. The fine blood vessel may be a blood vessel having a thickness close to the lower limit that can be expressed by a CT image, for example, a blood vessel having a thickness of about 0.5 mm.

  When the synthetic volume data is generated by the phase average 4DMIP synthesis, the noise in the synthetic volume data is moderate, and the thin blood vessels in the synthetic volume data become slightly dark.

  When synthetic volume data is generated by phase average 4DMIP synthesis with a filter, there is little noise in the synthetic volume data and dark blood vessels in the synthetic volume data are darkened.

  When synthetic volume data is generated by Deep 4DMIP synthesis, it is preferable because there is little noise in the synthetic volume data and the thin blood vessels in the synthetic volume data are bright. Therefore, in Deep 4DMIP synthesis, noise can be reduced compared to other synthesis methods.

  FIG. 9 is a diagram showing an example of an image for each method of synthesizing volume data of a plurality of phases and for each region in a subject. In FIG. 9, volume data synthesis methods may include Deep 4DMIP synthesis, 4DMIP synthesis, phase average 4DMIP synthesis, TSUM synthesis, and the like.

  In FIG. 9, three parts in the subject are shown. In FIG. 9, each image in the uppermost row shows a part b1, each image in the middle stage shows a part b2, and each image in the lowermost part shows a part b3. In FIG. 9, MPR images of combined volume data generated by the respective combining methods are shown as output images.

  In the output image synthesized with Deep 4DMIP, the fine blood vessels do not disappear, the thin blood vessels do not become thin (the luminance does not become small), and the noise disappears. For example, in the uppermost part b1, the fine blood vessels are easier to see than other synthesis methods (see the vicinity of the tip of the leftmost column and the uppermost two arrows in FIG. 9). Further, for example, in the middle part b2, noise is harder to see than in other synthesis methods (see the vicinity of the tip of the leftmost column and the middle arrow in FIG. 9). That is, as a result of the machine learning of the predictor 171, it can be understood that each predicted volume data is generated so that a real signal component remains and no noise component remains.

  In the output image synthesized by the phase average 4DMIP, the thin blood vessels are slightly disappeared and thinned, and the noise is slightly disappeared. In the output image synthesized by 4DMIP, the fine blood vessels do not disappear, the fine blood vessels are buried in noise, and high luminance noise increases. That is, compared with simple 4DMIP, the phase averaged 4DMIP smoothes the whole pixel value of the output image somewhat, and the drawing of fine blood vessels and noise is reduced.

  In the TSUM synthesized output image, the fine blood vessels disappear, become thinner, and the noise disappears. In other words, the luminance value for displaying the output image is adjusted under the influence of the high luminance value in the entire output image, so that necessary signal components (also referred to as real signal components) and noise components are easily removed as well. . The actual signal component may be a component that physically exists over a plurality of phases, such as a blood vessel and a bone.

  FIG. 10 is a diagram illustrating an example of an image for each synthesis method and rendering method for volume data of a plurality of phases. In FIG. 10, the volume data synthesis method may include 4DMIP synthesis, phase average 4DMIP synthesis with filter, Deep 4DMIP synthesis, and the like. Rendering methods may include MPR, raycast, MIP, etc. In FIG. 10, each output image subjected to phase average 4DMIP synthesis with a filter has a luminance value doubled.

  The 4DMIP synthesized output image contains a lot of noise and the entire image is whitish. Further, in the output ray cast image and the output MIP image, there are many noise components in addition to the blood vessels, and the blood vessels are unclear.

  In the output image synthesized by the phase average 4DMIP with filter, noise is reduced, but the actual signal component is also smoothed and reduced. For this reason, in the output ray cast image and the output MIP image, there are fewer blood vessels to be drawn than actual blood vessels, and necessary information may be missing.

  In Deep 4DMIP synthesis, noise is reduced and the actual signal component is maintained without being deleted. Therefore, in the output ray cast image and the output MIP image, the travel of the blood vessel is appropriately drawn and is easy for the user to observe.

  FIG. 11 is a graph showing an example of changes in pixel values in each phase of each part (organization) in the head of the subject. That is, FIG. 11 shows the displacement in the time direction for each region (for each tissue) in the head of the subject. In FIG. 11, the brain (Brain, Brain 2 to 5), the relatively bright blood vessel (Bright Vessel), the relatively dark brain (Dark Brain 1 to 3), and the relatively dark blood vessel (Dark). Vessel), sagittal sinus, relatively thin blood vessels (Thin Vessel, Thin Vessel 2), etc. are shown. In FIG. 11, it is assumed that a contrast medium is started to be administered to a subject as necessary in phase 0 (t = 0). The pixel values shown in FIG. 11 may be pixel values of volume data (actual volume data) obtained by imaging with the CT apparatus 200.

  In the brain and the relatively dark brain, there is almost no change in pixel values in each phase. This means that no contrast agent is administered to the subject, or a contrast agent is administered to the subject, but the contrast agent does not flow into these brain regions.

  In a relatively thin blood vessel, a sagittal sinus, a relatively dark blood vessel, and a relatively light blood vessel, the pixel value changes to some extent in each phase. Specifically, in the early stage of a plurality of phases, the contrast agent sequentially reaches and flows into each part, so that the pixel value increases sequentially. At the end of a plurality of phases, the contrast value flows out from each part, so that the pixel values are sequentially reduced. In addition, since the time for the contrast agent to flow is different for each part, the phase where the pixel value reaches a peak may be different for each part. For example, in a blood vessel, since the contrast medium flows into the artery first, the pixel value of the artery increases first, and since the contrast medium flows into the vein later, the pixel value of the vein increases later.

  Next, an output image of the observation target when learning is performed using the volume data of the subject other than the observation target will be described.

  FIG. 12A is a diagram illustrating an example of an output MIP image in which single-phase volume data of a subject to be observed is rendered. FIG. 12B is a diagram illustrating an example of an output ray cast image in which single-phase volume data of a subject to be observed is rendered.

  FIG. 12C is a diagram illustrating an example of an output MIP image in which composite volume data in which volume data of a plurality of phases of a subject to be observed is combined with 4DMIP is rendered. FIG. 12D is a diagram illustrating an example of an output raycast image obtained by rendering composite volume data obtained by 4DMIP synthesis of volume data of a plurality of phases of a subject to be observed.

  FIG. 12E shows an output in which synthesized volume data in which volume data of a plurality of phases of a subject to be observed is subjected to Deep 4DMIP synthesis is rendered when the predictor 171 performs machine learning using the volume data of the subject other than the subject to be observed. It is a figure which shows an example of a MIP image. FIG. 12F shows an output in which synthesized volume data obtained by deep 4DMIP synthesis of volume data of a plurality of phases of a subject to be observed when the predictor 171 performs machine learning using the volume data of the subject other than the observation subject is rendered. It is a figure which shows an example of a ray cast image.

  Referring to FIGS. 12A to 12F, the medical image processing apparatus 100 outputs an observation target output image (for example, an output MIP image or an output raycast image) even when machine learning is performed using volume data of a subject other than the observation target. It can be understood that a clear image is obtained in which the noise component is suppressed and the actual signal component is not suppressed. Therefore, the user can easily and clearly grasp the state of the observation target by checking the rendered output image, and can perform a highly accurate diagnosis. In addition, the predictor 171 can perform learning using not only data specific to the patient to be observed but also data regarding various patients by performing machine learning using volume data of a subject different from the subject to be observed. , Increase machine learning variations.

  As described above, the medical image processing apparatus 100 can acquire a plurality of volume data in time series and generate volume data with good image quality. In the conventional deep learning, the image quality of the volume data itself used for learning is not improved, but the medical image processing apparatus 100 can improve the image quality of the time series data by using the time series volume data. The same effect can be obtained even when using volume data other than the subject to be observed. Further, the medical image processing apparatus 100 uses the time-series data itself for which image quality is desired to be improved as teacher data for machine learning by the predictor 171, so that there is no noise-free data in real space. Can be obtained, and output data (for example, synthetic volume data and output image) using the time series data can be obtained.

  In the first embodiment, the predictor 171 uses a part of the volume data of the plurality of phases of volume data (actual volume data) for the same subject. An example of generating predicted volume data corresponding to the remaining volume data is shown. Instead, the predictor 171 may use real volume data as a plurality of phases of volume data that is the basis for generation of predicted volume data, or a mixture of real volume data and predicted volume data. Alternatively, all real volume data may be used. That is, based on the volume data of a plurality of phases including the predicted volume data of an arbitrary phase generated by the predictor 171, predicted volume data of other phases may be generated.

  Therefore, the predictor 171 may generate predicted volume data by the predictor 171, generate next predicted volume data based on the generated predicted volume data, and repeat this process. In this case, volume data of a plurality of phases (including predicted volume data and actual volume data) may be aligned each time. As a result, the medical image processing apparatus 100 can gradually reduce noise by generating predicted volume data each time and improve the alignment accuracy, thereby improving the image quality of the synthesized volume data.

(Second Embodiment)
In the first embodiment, the predictor 171 is machine-learned to obtain synthetic volume data based on a plurality of phases of volume data. In the second embodiment, the classifier 172, which will be described later, is machine-learned together with the predictor 171 to obtain combined volume data based on volume data of a plurality of phases.

  In the second embodiment, description of the same configuration, operation, processing, and the like as in the first embodiment may be omitted or simplified.

  Similar to the first embodiment, the medical image processing apparatus 100A according to the second embodiment includes a port 110, a UI 120, a display 130, a processor 140, and a memory 150. In this embodiment, the processor 140 functions as a processing unit 160A that performs various processes and controls by executing a medical image processing program stored in the memory 150.

  FIG. 13 is a block diagram illustrating a functional configuration example of the medical image processing apparatus 100A according to the second embodiment. The processing unit 160A includes an area extraction unit 161, an image generation unit 162, a registration processing unit 163, a prediction processing unit 164, an identification processing unit 165, and a display control unit 166. The identification processing unit 165 includes a discriminator 172. Note that the predictor 171 can also be said to be a generator that generates a predicted image. In the processing unit 160A of FIG. 13, the same functional components as those of the processing unit 160 illustrated in FIG. 2 are denoted by the same reference numerals, and the description thereof is omitted or simplified.

  The region extraction unit 161 cuts out and extracts a part of volume data (for example, actual volume data or predicted volume data). The region extraction unit 161 may extract an arbitrary part or tissue of the subject in the volume data.

  The identification processing unit 165, based on the volume data of a plurality of phases, the arbitrary part p in the real volume data of any phase i imaged in real space, and the arbitrary part p in the predicted volume data of any phase j Difference volume data indicating the difference between the two may be generated. It is preferable that the actual volume data and the predicted volume data in the corresponding phase (phase i) coincide with each other. However, in the case where the actual signal component is included, a difference occurs in the actual signal component between the two volume data. Since the differential volume data ideally does not contain a real signal component, it can also be said to be noise volume data. Here, phase i and phase j may be the same or different. The identification processing unit 165 may generate differential volume data indicating differences between various parts p in the actual volume data and various parts p in the predicted volume data. Therefore, the identification processing unit 165 may generate difference volume data relating to various parts p.

  The identification processing unit 165 generates a plurality of combinations (data pairs) of the predicted volume data of the extracted part and the generated differential volume data. In a data pair, the phases of predicted volume data and differential volume data may be different. In the data pair, the predicted volume data and the difference volume data have the same part in the subject. For example, the identification processing unit 165 determines whether the data pair is a combination of the predicted volume data and the difference volume data of the same part or a combination of the predicted volume data and the difference volume data of a different part. Good. For example, the probability of combining the same parts and the probability of combining different parts may be 50% each.

  The identification processing unit 165 identifies whether or not the difference volume data included in the data pair corresponds to the part p in the predicted volume data. It is desirable that the part p in the predicted volume data generated by the predictor 171 completely coincides with the part p in the actual volume data and no difference appears in the difference volume data. This is an indication for identifying which part of the predicted volume data the volume data corresponds to. Therefore, when the difference volume data includes many difference components, the identification processing unit 165 can easily identify whether or not the difference volume data corresponds to the portion p in the predicted volume data.

  The identification processing unit 165 notifies the predictor 171 of the prediction processing unit 164 of an identification result (for example, information indicating whether identification is successful or unsuccessful) as to whether or not the difference volume data and the predicted volume data correspond. .

  The identification processing unit 165 causes the classifier 172 to perform machine learning using the training data and the teacher data. This machine learning may include GAN (Generative Adversarial Network) (for example, Conditional GAN). In machine learning, the identification processing unit 165 may use a data pair of difference volume data of an arbitrary part corresponding to the same phase and predicted volume data of an arbitrary part as training data. The identification processing unit 165 may use, as teacher data, information indicating whether or not the difference volume data of any part corresponds to the predicted volume data of any part.

  The identification processing unit 165 performs machine learning using the training data and the teacher data so that the identification of whether or not the difference volume data corresponds to the predicted volume data succeeds (the accuracy rate increases). That is, the identification processing unit 165 may perform machine learning so that the identification of whether or not the difference volume data of an arbitrary part and the predicted volume data of an arbitrary part are the same part data is successful.

  Upon receiving the notification of the identification result, the prediction processing unit 164 can recognize whether or not the discriminator 172 has identified the correspondence between the differential volume data and the predicted volume data. The predictor 171 performs machine learning so that the correspondence between the differential volume data of an arbitrary part and the predicted volume data of an arbitrary part is not identified by the identifier 172. This machine learning may include GAN (for example, Conditional GAN, GAN using CNN).

  The image generation unit 162 may generate one composite volume data using the maximum value of each voxel as the pixel value in the predicted volume data of a plurality of phases generated by the predictor 171 machine-learned together with the discriminator 172. This synthesis method may be referred to as Deep GAN 4DMIP synthesis. In Deep GAN 4DMIP synthesis, a specific method implemented by 4DMIP synthesis may be applied to derivation of the maximum value of each voxel in a plurality of phases of predicted volume data.

  Next, the roles of the predictor 171 and the identifier 172 will be described.

  Here, i may be a phase number. Vi may be volume data of a phase for which prediction is desired. p and q indicate part identification information for identifying the part in the volume data.

  As described above, the predictor 171 generates the predicted volume data V′i. Since noise cannot be predicted, the predicted volume data V′i does not include a noise component. On the other hand, the predictor 171 easily deletes a minute structure (for example, a fine structure of a bone or a blood vessel) from the actual signal component due to the L1 loss property of machine learning together with the noise component. Therefore, when the predictor 171 deletes the micro structure, a penalty is applied.

  The predictor 171 generates differential volume data (V′i−Vi). In the differential volume data, it is desirable that each pixel value is a uniform noise component (A). Actually, due to the performance of the predictor 171, the minute volume component (B) may remain as a minute edge component together with the noise component (A) in the differential volume data. The microstructure component (B) is an example of an actual signal component. Therefore, the (A + B) component may remain in the differential volume data.

  When the difference volume data (V′p−Vp) and the predicted volume data V′q are compared, it is desirable that it is not possible to identify whether p = q or p ≠ q. Therefore, the difference volume data (V It is desirable that the microstructure component (B) is not contained in 'p-Vp).

  The difference volume data (V′p−Vp) and the predicted volume data V′q are data of different parts in the subject. The difference volume data (V′p−Vp) and the predicted volume data V′q may be the same phase data or different phase data.

  A classifier 172 can be defined to perform the above identification. The discriminator 172 acquires a data pair of any combination of the differential volume data (V′p−Vp) and the predicted volume data V′q. For example, the discriminator 172 outputs a value of 0 when identifying that p = q, and outputs a value of 1 when identifying that p ≠ q. The success or failure of this output result (identification result) is evaluated and reflected in subsequent predictions by the predictor 171.

  The loss (Generator Loss) related to the predictor 171 includes a reconstruction error, that is, a difference (prediction error) of predicted volume data with respect to actual volume data, and successful identification by the classifier 172. The discriminator loss associated with the discriminator 172 includes misidentification, that is, discrimination failure by the discriminator 172.

  Next, machine learning of the predictor 171 and the discriminator 172 will be described.

  FIG. 14 is a diagram for explaining an example of machine learning of the predictor 171 and the discriminator 172. In FIG. 14, V indicates volume data. i and k indicate phase numbers. A plurality of volume data having different phases are preferably registered, but it is not essential.

  Further, in FIG. 14, p and q indicate part identification information in the volume data. The part identification information indicates information for identifying various parts (arbitrary organs, blood vessels, bones, tissues, lesions, etc.) in the subject. Vp is a part of the volume data, and indicates volume data of a part identified by the part number p in the subject.

  As in the first embodiment, the predictor 171 performs machine learning so as to output Vi that is teacher data when Vk (k ≠ i) that is training data is input. In this case, the predictor 171 actually outputs V′i, but performs machine learning so that V′i approaches Vi.

  The identification processing unit 165 generates differential volume data (V′p−Vp) of an arbitrary part p of the subject for the phase i corresponding to the predicted volume data generated by the predictor 171 among the plurality of phases. . The identification processing unit 165 generates predicted volume data V′q of an arbitrary part q of the subject for phase i corresponding to the predicted volume data generated by the predictor 171 among the plurality of phases. The parts p and q here may be a range corresponding to a patch.

  The identification processing unit 165 inputs a data pair of the difference volume data (V′p−Vp) and the predicted volume data V′q to the classifier 172. In this case, the identification processing unit 165 generates a data pair to be input to the classifier 172 by arbitrarily combining the differential volume data (V′p−Vp) and the predicted volume data V′q. The arbitrary combination may include a combination of data of the same part (q = p) and a combination of data of different parts (q ≠ p).

  Since the data pair of the predicted volume data and difference volume data of the same part (for example, blood vessel and blood vessel) is data of the same part, that is, a corresponding part, it is also called a corresponding pair (corresponding pattern). A pair of predicted volume data and difference volume data of different parts (for example, blood vessels and bones) is data of different parts, that is, non-corresponding parts, and is also referred to as a non-corresponding pair (non-corresponding pattern). Therefore, the identification processing unit 165 determines the corresponding pair (V′p−Vp, V′q) (q = p) or the non-corresponding pair (V′p−Vp, V′q) (q ≠ p) as training data. To the discriminator 172.

  When the identification processing unit 165 generates a pair of the differential volume data (V′p−Vp) and the predicted volume data V′q, the differential volume data (V′p−Vp) and the predicted volume data V′q The success or failure (true / false, that is, Real / Fake) of whether or not corresponds to is recognized in advance. This success / failure information may be held in the memory 150, for example. The identification processing unit 165 inputs data indicating success or failure corresponding to the training data (for example, Real / Fake, value 0 or value 1) to the classifier 172 as teacher data.

  The discriminator 172 performs machine learning so that training data is input and teacher data is output. Specifically, the discriminator 172 identifies whether or not the differential volume data (V′p−Vp) and the predicted volume data V′q correspond, that is, whether the pair is a corresponding pair or a non-corresponding pair. Machine learning may be performed so that is correct.

  The identification processing unit 165 notifies the prediction unit 171 of the prediction processing unit 164 of the success / failure information of the corresponding / non-corresponding identification result in the machine learning by the classifier 172.

  The prediction processing unit 164 adds information on the success or failure of the discrimination result by the discriminator 172 as one parameter for machine learning by the predictor 171. The predictor 171 performs machine learning so that the discrimination result by the discriminator 172 is incorrect. That is, the predictor 171 generates the predicted volume data V′i corresponding to the predicted volume data V′q in which the discriminator 172 cannot distinguish the difference volume data (V′p−Vp) and the predicted volume data V′q. May be machine learning. Thus, the predictor 171 receives a penalty when the discriminator 172 succeeds in discrimination.

  In this way, the medical image processing apparatus 100A takes into consideration the success or failure of the discrimination result by the discriminator 172 for the prediction by the predictor 171, so that the predictor 171 can perform the machine learning rather than the case where the predictor 171 performs machine learning alone. By implementing both the machine learning and the machine learning of the discriminator 172, the prediction accuracy (prediction volume data generation accuracy) by the predictor 171 can be improved.

  For example, if the predictor 171 is ideal, the actual volume data Vi of the same phase i and the predicted volume data V′i have the same actual signal component other than the noise component, so that the differential volume data includes only the noise component. Including noise volume data. In this case, the pair input to the discriminator 172 is a pair of differential volume data, that is, noise volume data and predicted volume data. Since the noise volume data is only a noise component regardless of the part of the subject, the discriminator 172 cannot identify whether it is a corresponding pair or a non-corresponding pair. On the other hand, when the predictor 171 is not ideal, a structure (real signal component) other than noise remains in the differential volume data. In this case, the discriminator 172 can identify whether the pair is a corresponding pair or a non-corresponding pair based on the actual signal component of the differential volume data and the actual signal component included in the predicted volume data. Therefore, the predictor 171 can approach the ideal predictor 171 by performing machine learning so that the discriminator 172 cannot discriminate whether it is a corresponding pair or a non-corresponding pair.

  FIG. 15 is a diagram illustrating an example of real volume data, predicted volume data, and differential volume data of the same subject as the prediction target.

  In FIG. 15, three images in the left column show actual images obtained by volume rendering of actual volume data. Three images in the middle row indicate predicted images in which the predicted volume data is volume-rendered. Three images in the right column show differential images obtained by volume rendering of the differential volume data. Further, the upper three images, the middle three images, and the lower three images indicate images of different parts of the same subject. Three images at each stage show images of the same part. Specifically, each uppermost image in FIG. 15 shows a part b11, each middle image shows a part b12, and each lowermost image shows a part b13. Note that the subject in FIG. 15 is the same as the subject to be predicted by the predictor 171.

  FIG. 16 is a diagram illustrating an example of actual volume data, predicted volume data, and differential volume data of a subject different from the subject to be predicted.

  In FIG. 16, the three images in the left column show the MPR actual image of the actual volume data. Three images in the middle row indicate predicted images in which the predicted volume data is volume-rendered. Three images in the right column show differential images obtained by volume rendering of the differential volume data. Further, the upper three images, the middle three images, and the lower three images indicate images of different parts of the same subject. Three images at each stage show images of the same part. Specifically, each uppermost image in FIG. 16 shows a part b21, each middle image shows a part b22, and each lowermost image shows a part b23. Note that the subject in FIG. 16 is a subject different from the subject to be predicted by the predictor 171.

  For example, in the middle part of FIG. 16, it is estimated that the part p1 of the predicted volume data v1 appears as the part p3 of the differential volume data v3. That is, it is estimated that the difference between the pixel value of the actual volume data v2 and the pixel value of the predicted volume data v1 appears. This corresponds to a prediction error by the predictor 171. Therefore, when there is a prediction error, the discriminator 172 can easily identify whether or not the predicted volume data v1 and the difference volume data v3 correspond.

  FIG. 17A is a diagram illustrating an example of an output raycast image when phase average 4DMIP is combined. FIG. 17B is a diagram illustrating an example of an output MPR image in the case of phase average 4DMIP synthesis and pixel values in a specific range in the output MPR image. The output ray cast image and the output MPR image in FIGS. 17A and 17B are images of the same part in the same subject.

  In FIG. 17B, a graph g11 indicating pixel values on the straight line L11 (here, unit: HU) is superimposed on the output MPR image. In the graph g11, the peak value (about 150) of the pixel value on the straight line L11 at the point p11 existing on the straight line L11.

  FIG. 17C is a diagram illustrating an example of an output ray-cast image when Deep GAN 4DMIP is combined. FIG. 17D is a diagram illustrating an example of an output MPR image when Deep GAN 4DMIP synthesis is performed, and pixel values in a specific range in the output MPR image.

  In FIG. 17D, a graph g12 indicating pixel values (here, unit: HU) on the straight line L12 is superimposed on the output MPR image. In the graph g12, the peak value (about 330) of the pixel value on the straight line L12 is obtained at a point p12 existing on the straight line L12.

  Comparing FIG. 17A and FIG. 17B with FIG. 17C and FIG. 17D, in FIG. 17A and FIG. 17B, there is a lot of noise, and the drawing of thin blood vessels is somewhat clear in the output raycast image, and the drawing of thin blood vessels in the output MPR image. Is unclear (the peak value of the pixel value is about 150).

  On the other hand, in FIGS. 17C and 17D, there is little noise, drawing of the fine blood vessel is quite clear in the output raycast image (see the vicinity of the tip of the two arrows in FIG. 17C), and drawing of the thin blood vessel is made in the output MPR image. It can be understood that the pixel value is clear (the peak value of the pixel value is about 330). Therefore, the user can easily and clearly grasp the state of the observation target by checking the output image in which the volume data synthesized by Deep GAN 4DMIP is rendered, and can perform a highly accurate diagnosis.

  FIG. 18A is a diagram showing an example of an output MPR image of the brain when the phase average 4DMIP synthesis is performed. FIG. 18B is a diagram illustrating an example of an output MPR image of the brain when Deep GAN 4DMIP is synthesized.

  Comparing FIG. 18A and FIG. 18B, it can be understood that the boundary between the lung white matter and the white matter in the brain is unclear in FIG. 18A, but the boundary between the lung white matter and the white matter in the brain is clear in FIG. 18B. (See the vicinity of the tip of the arrow in FIG. 18B).

  FIG. 19A is a diagram illustrating an example of an image in which real volume data, predicted volume data, and differential volume data of an arbitrary part when Deep 4DMIP is combined are rendered. FIG. 19B is a diagram illustrating an example of actual volume data, predicted volume data, and differential volume data of an arbitrary part when Deep GAN 4DMIP is combined.

  Comparing FIG. 19A and FIG. 19B, it can be understood that in FIG. 19A, the difference volume data includes a little other than the noise component, and a shadow indicating the presence of the tissue is generated in the vicinity of the tip of the arrow. In other words, in Deep 4DMIP synthesis, a blood vessel portion is slightly deleted in the predicted volume data, and the pixel value of the blood vessel portion becomes smaller, and the shadow value is generated with the pixel value of the predicted volume data <the pixel value of the actual volume data. . On the other hand, in FIG. 19B, it can be understood that other than the noise component of the differential volume data is further reduced, and it is difficult to confirm the shadow near the tip of the arrow. That is, it can be understood that the generation accuracy of predicted volume data can be improved by performing Deep GAN 4DMIP synthesis.

  FIG. 20 is a flowchart illustrating an operation example related to machine learning and prediction image generation of the predictor 171 and the identifier 172 by the medical image processing apparatus 100A. In FIG. 20, it is assumed that there are 0 to n phases.

  First, the port 110 acquires volume data V0 to Vn of a plurality of phases arranged in time series from the CT apparatus 200 (S31). The registration processing unit 163 aligns the acquired multiple-phase volume data V0 to Vn (S32).

  The prediction processing unit 164 causes the predictor 171 to perform machine learning for the phases i = 0 to n using the volume data V0 to Vn (excluding Vi) (S33). In this case, the prediction processing unit 164 uses the volume data V0 to Vn (excluding Vi) acquired in S31 as training data for machine learning, and uses the volume data Vi acquired in S31 as teacher data for machine learning. .

  The prediction processing unit 164 inputs the volume data V0 to Vn (excluding Vi) to the predictor 171 for the phases i = 0 to n, and generates the predicted volume data V′i (S34). Therefore, predicted volume data V′0 to V′n are obtained.

  The identification processing unit 165 generates differential volume data NVi = | V′i−Vi | for the phases i = 0 to n (S35). That is, the differential volume data may be indicated by the absolute value of the difference between the pixel value of each pixel of the predicted volume data and the pixel value of each pixel of the actual volume data.

  The identification processing unit 165 uses the data pair [NVi, V′j] of the differential volume data NVi and the predicted volume data V′j for the phase i = 0 to n and the phase j = 0 to n to identify the identifier 172. Machine learning (S36). In this case, the discriminator 172 performs machine learning using the data pair [NVi, V′j] as training data and information indicating whether the data pair is a corresponding pair or a non-corresponding pair as teacher data. For example, if the discriminator 172 identifies that the training data is a corresponding pair (ie, i = j), it outputs true (eg, value 0) and identifies that the training data is a non-corresponding pair (ie, i ≠ j). Then, false (for example, value 1) is output. This output result is evaluated.

  The identification processing unit 165 notifies (feeds back) the prediction processing unit 164 of the success or failure of the identification result by the classifier 172. Based on the notification from the identification processing unit 165, the prediction processing unit 164 causes the predictor 171 to perform machine learning so that it is not correctly identified by the classifier 172 (S37).

  The image generation unit 162 calculates the maximum value of the pixel value (voxel value) in each corresponding voxel for the predicted volume data V′0 to V′n, and generates maximum value volume data Vmax (S38). That is, the image generation unit 162 generates the maximum value volume data Vmax by Deep GAN 4DMIP composition.

  The image generation unit 162 visualizes the maximum value volume data Vmax by, for example, the MIP method (S39). That is, the image generation unit 162 performs volume rendering on the maximum volume data Vmax using, for example, the MIP method, and generates an output image (output MIP image).

  According to the processing in FIG. 20, the medical image processing apparatus 100 </ b> A can perform machine learning by the predictor 171 so that the generation accuracy of predicted volume data is increased. Further, the medical image processing apparatus 100A can perform machine learning so that the identification accuracy for identifying the correspondence between the predicted volume data and the difference volume data is increased by the classifier 172. Therefore, the predictor 171 can generate predicted volume data so that the discriminator 172 does not identify it, and can generate predicted volume data with higher image quality than when the discriminator 172 is not used. Therefore, the medical image processing apparatus 100A can further suppress the influence of noise at the time of imaging of the subject and further improve the accuracy of deriving a synthesized image obtained by synthesizing a plurality of phases of medical images.

  Note that the processing unit 160 may provide a loop that proceeds to S33 after S37. That is, the machine learning of the predictor 171 and the discriminator 172 may be repeatedly performed. Accordingly, the medical image processing apparatus 100A can sequentially improve the predictor 171 and the discriminator 172, and can improve the prediction performance by the predictor 171 and the discrimination performance by the discriminator 172. The processing unit 160 may proceed to the processing of S38 when the number of loops is equal to or greater than the threshold th.

  While various embodiments have been described above with reference to the drawings, it goes without saying that the present disclosure is not limited to such examples. It will be apparent to those skilled in the art that various changes and modifications can be made within the scope of the claims, and these are naturally within the technical scope of the present disclosure. Understood.

  The above embodiment has illustrated that the predictor 171 performs machine learning and prediction based on time-series real volume data. In addition, the classifier 172 illustrated performing machine learning and identification based on time-series actual volume data, predicted volume data, and differential volume data. Instead, the predictor 171 may perform machine learning and prediction based on time-series real volume data and information (for example, perfusion data) based on the real volume data. The discriminator 172 may perform machine learning and identification based on time-series actual volume data, predicted volume data, differential volume data, and information (for example, perfusion data) regarding these volume data. Thereby, the medical image processing apparatuses 100 and 100A can further improve machine learning accuracy and prediction accuracy.

  In the above embodiment, three-dimensional volume data is input to the predictor 171 when predictive images are generated (at the time of actual prediction) or machine learning, and three-dimensional volume data (predicted volume data) is output from the predictor 171 ( Generated). A medical image other than the three-dimensional volume data may be input to the predictor 171 and output from the predictor 171. For example, at the time of predictive image generation or machine learning, for example, two-dimensional slice data obtained by the CT apparatus 200 may be input to the predictor 171 and two-dimensional slice data may be output (generated) from the predictor 171. . In addition, a three-dimensional image or a two-dimensional image generated based on the volume data is input to the predictor 171 when a predicted image is generated or machine learning is performed, and a three-dimensional image or a two-dimensional image is output (generated) from the predictor 171. May be.

  In the above embodiment, the data pair of the three-dimensional volume data is input to the discriminator 172, and the discrimination result as to whether it is a corresponding pair or a non-corresponding pair is output from the discriminator 172. Note that a pair of medical images other than three-dimensional volume data may be input to the discriminator 172, and an identification result indicating whether the pair is a corresponding pair or an incompatible pair may be output from the discriminator 172. For example, two-dimensional slice data may be input to the discriminator 172, and an identification result indicating whether the pair is a corresponding pair or a non-corresponding pair may be output from the discriminator 172. In addition, a three-dimensional image or a two-dimensional image generated based on the volume data may be input to the discriminator 172, and an identification result indicating whether it is a corresponding pair or a non-corresponding pair may be output from the discriminator 172.

  In the above-described embodiment, the time-series volume data is an example of a medical image representing, for example, the transition of a contrast agent, but other data (for example, an MRI relaxation image) may be used. The volume data may be visualized by a known method (for example, MIP, MinIP, RaySUM, MPR). Further, when the data generated by the predictor 171 is slice data or a two-dimensional image, it may be visualized by a two-dimensional image.

  For example, the image generation unit 162 may generate one composite volume data by using the minimum value of each voxel as the pixel value in the multi-phase volume data. This synthesis method is also referred to as TminIP synthesis. In TMinIP synthesis, the pixel value tends to be small. The image generation unit 162 may generate the synthesized volume data using a statistical value (for example, the minimum value (Min)) other than the maximum value among the pixel values of the corresponding pixels in the n predicted images YGi as the pixel value. Good. The medical image processing apparatuses 100 and 100A perform TMinIP composition so that, when the CT apparatus 200 images a subject in a contrast state, depending on the type of contrast agent, composite volume data (an image in which the composite volume data is rendered) Can be easily observed.

  Further, the image generation unit 162 may exclude the average value (Average) from the above statistical values and exclude the generation of the composite volume data using the average value. As a result, the medical image processing apparatuses 100 and 100A average the pixel values of the respective pixels of each predicted image YGi, thereby suppressing noise and smoothing the signal values actually required in each predicted image YGi. Can be prevented.

  The image generation unit 162 may generate an output image without using a Gaussian filter. This makes it difficult for the pixel values in the output image to be smoothed, and for example, the peripheral part of the blood vessel is difficult to disappear. That is, the medical image processing apparatuses 100 and 100A can maintain the drawing performance of the peripheral part of the blood vessel.

  In the above embodiment, the image generation unit 162 exemplifies generating Vmax by 4DMIP as a method for synthesizing volume data of a plurality of phases. However, other synthesis methods (for example, average value synthesis, median value synthesis, minimum value) A plurality of volume data may be synthesized using (combining).

  In the above-described embodiment, the image generation unit 162 is configured to perform known noise removal processing or image filtering processing (for example, an image using a Gaussian filter) before, during or after combining multi-phase volume data using 4DMIP or the like. Filter processing) may be performed.

  In the above embodiment, the volume data as a captured CT image is transmitted from the CT apparatus 200 to the medical image processing apparatuses 100 and 100A. Instead, the volume data may be transmitted to a server on the network and stored in the server or the like so that the volume data is temporarily accumulated. In this case, when necessary, the port 110 of the medical image processing apparatus 100 or 100A may acquire volume data from a server or the like via a wired line or a wireless line, or via an arbitrary storage medium (not shown). You may get it.

  In the above embodiment, the volume data as a captured CT image is transmitted from the CT apparatus 200 to the medical image processing apparatuses 100 and 100A via the port 110. This includes the case where the CT apparatus 200 and the medical image processing apparatuses 100 and 100A are substantially combined into one product. Further, the case where the medical image processing apparatuses 100 and 100A are handled as a console of the CT apparatus 200 is also included.

  In the above-described embodiment, it is exemplified that an image is captured by the CT apparatus 200 and volume data including information inside the living body is generated. However, an image may be captured by another apparatus to generate volume data. Other devices include MRI devices, PET (Positron Emission Tomography) devices, angiographic devices (Angiography devices), or other modality devices. In addition, the PET apparatus may be used in combination with other modality apparatuses.

  In the above embodiment, the human body is exemplified as the subject, but an animal body may be used.

  In the present disclosure, a program that realizes the functions of the medical image processing apparatus of the above-described embodiment is supplied to the medical image processing apparatus via a network or various storage media, and a program that is read and executed by a computer in the medical image processing apparatus is also provided. Scope of application.

  As described above, in the medical image processing apparatus 100 of the above-described embodiment, the acquisition unit (for example, the port 110) has a plurality of medical images (for example, volume) obtained by imaging the first subject a plurality of times in time series. Data). The processing unit 160 uses the predictor 171 to generate a plurality of predicted images (for example, predicted volume data). The predictor 171 predicts images (for example, V ′) corresponding to the remaining medical images (for example, V0) of the plurality of medical images based on some of the plurality of medical images (for example, V1 to V7). 0) is generated. The processing unit 160 uses at least two medical images constituting a part of the plurality of medical images as training data, uses the remaining medical images among the plurality of medical images as teacher data, and combines some medical images. Then, the remaining medical images are sequentially changed to cause the predictor 171 to perform machine learning. The processing unit 160 inputs a part of the plurality of medical images to the predictor 171 and sequentially changes a combination of the part of the medical images to thereby generate a plurality of predicted images (for example, V′0 to V ′). 7) is generated.

  As a result, the medical image processing apparatus 100 performs machine learning using a part of the medical images in the plurality of phases arranged in time series, and thereby the remaining medical images in the plurality of phases and the predicted images corresponding to the phases. Machine learning is possible so that the characteristics of That is, the medical image processing apparatus 100 causes the predictor 171 to perform machine learning using a medical image in a phase other than the predicted phase among a plurality of phases as training data and a medical image in a predicted phase as teacher data to perform prediction. The image quality of the medical image in the phase to be improved can be improved (for example, noise included in the medical image is reduced). Therefore, when the imaging accuracy of the remaining medical image is low (for example, when blurring occurs due to the movement of the subject at the imaging timing of the remaining medical image, the medical image processing apparatus 100 (eg, the CT apparatus 200). Even when the above problem occurs), the influence of low imaging accuracy can be reduced.

  Moreover, although the medical image processing apparatus 100 includes noise in the actual medical image, the medical image of each phase can be improved by generating a predicted image. In addition, although no noise-free data exists in the real space, the medical image processing apparatus 100 can generate an ideal image without noise by generating a predicted image corresponding to the medical image of each phase. Phase medical images can be brought close to ideal 4D data.

  In this manner, the medical image processing apparatus 100 can improve the image quality of a plurality of phases of medical images by suppressing the influence of noise during imaging of the subject.

  In addition, the processing unit 160 uses at least the first medical image (for example, V0) and the second medical image (for example, V1) as training data of the predictor 171 and uses the third medical image (for example, V2) of the predictor 171. As the teacher data, the predictor 171 may be machine-learned. The processing unit 160 may cause the predictor 171 to perform machine learning using at least the second medical image and the third medical image as training data for the predictor 171 and using the first medical image as teacher data for the predictor 171. The processing unit 160 may cause the predictor 171 to perform machine learning using at least the third medical image and the first medical image as training data for the predictor 171 and the second medical image as teacher data for the predictor 171. The processing unit 160 may input at least the first medical image and the second medical image to the predictor 171 to generate a first predicted image (for example, V′2). The processing unit 160 may input at least the second medical image and the third medical image to the predictor 171 to generate a second predicted image (for example, V′0). The processing unit 160 may input at least the third medical image and the first medical image to the predictor 171 to generate a third predicted image (for example, V′1). The processing unit 160 may combine at least the first predicted image, the second predicted image, and the third predicted image to obtain a combined image.

  As a result, the medical image processing apparatus 100 performs machine learning using the first medical image, the second medical image, and the third medical image arranged in time series, thereby the third medical image and the first medical image. Learning can be performed so that the features of the predicted image approach each other. Therefore, the medical image processing apparatus 100 has a problem in the medical image imaging device when the imaging accuracy of the third medical image is low (for example, when blurring occurs due to the movement of the subject at the imaging timing of the third medical image). However, the influence of low imaging accuracy can be reduced.

  The medical image processing apparatus 100 generates 4D data (for example, a moving image) as a synthesized image with reduced noise by generating a synthesized image using a predicted image, although noise is included in an actual medical image. Can be obtained. In addition, data without noise does not exist in real space, but the medical image processing apparatus 100 can approximate ideal 4D data by generating an ideal image without noise.

  In this manner, the medical image processing apparatus 100 can improve the derivation accuracy of a composite image in which a plurality of phases of medical images are combined by suppressing the influence of noise during imaging of the subject.

  The processing unit 160 may cause the predictor 171 to perform machine learning after aligning the first medical image, the second medical image, and the third medical image.

  As a result, the medical image processing apparatus 100 deteriorates the prediction accuracy when the position of the subject is shifted in time series. However, by aligning a plurality of medical images on the time series, deterioration in prediction accuracy can be suppressed. .

  The processing unit 160 may cause the predictor 171 to perform machine learning using a plurality of medical images on a time series of a second subject that is different from the first subject. In this case, the acquisition unit obtains the fourth medical image, the fifth medical image, and the sixth medical image obtained by imaging the second subject different from the first subject at least three times in time series. Medical images may be acquired. The processing unit 160 may cause the predictor 171 to perform machine learning using at least the fourth medical image and the fifth medical image as training data for the predictor 171 and the sixth medical image as teacher data for the predictor 171. The processing unit 160 may cause the predictor 171 to perform machine learning using at least the fifth medical image and the sixth medical image as training data for the predictor 171 and the fourth medical image as teacher data for the predictor 171. The processing unit 160 may cause the predictor 171 to perform machine learning using at least the sixth medical image and the fourth medical image as training data for the predictor 171 and the fifth medical image as teacher data for the predictor 171.

  Thereby, the medical image processing apparatus 100 can share the predictors 171 for various subjects (patients). That is, the medical image processing apparatus 100 can cause the predictor 171 to perform machine learning using both the volume data of a subject other than the subject to be predicted and the volume data of the subject to be predicted. In this case, the medical image processing apparatus 100 can take into account various subject states and tissue configurations compared to the case where the predictor 171 performs machine learning using only the subject to be predicted. It is possible to suppress local convergence on the result. Therefore, for example, even when predictive images of a large number of subjects are generated using the predictor 171, various subject states and tissue configurations can be taken into account, and prediction accuracy can be improved.

  In addition, the processing unit 160 determines, from the third medical image, a medical image (for example, Vp) of the first region (for example, region p) in the first subject and a second region (for example, different from the first region). You may acquire the medical image (for example, Vq) of the site | part q). The processing unit 160 may acquire a predicted image (for example, V′p) of the first part and a predicted image (for example, V′q) of the second part in the first subject from the first predicted image. . The processing unit 160 may generate a first difference image (for example, V′p−Vp) indicating a difference between the medical image of the first part and the predicted image of the first part. The processing unit 160 may generate a second difference image indicating a difference between the medical image of the second part and the predicted image of the second part. The processing unit 160 includes a difference image (for example, V′p−Vp) of the first difference image and the second difference image, a predicted image of the first part, and a predicted image of the second part. You may generate | occur | produce the pair (for example, data pair) with the prediction image of any site | part. The processing unit 160 may cause the predictor 171 to perform machine learning so that the discrimination by the discriminator 172 that identifies whether any difference image in the pair corresponds to the predicted image of any part does not succeed. .

  When discrimination by the discriminator 172 fails, it means that the discriminator 172 cannot associate the difference image with the predicted image, that is, the actual image component of the predicted image is not included in the difference image, and only the noise component is included. Means that On the other hand, when the discrimination by the discriminator 172 is successful, it means that the discriminator 172 can associate the difference image with the predicted image, that is, the actual signal component (for example, a minute structure) of the predicted image remains in the difference image. Means that Therefore, the difference image includes a real signal component (such as a blood vessel as a minute structure), which means that there is room for improvement in prediction accuracy. The medical image processing apparatus 100 </ b> A can suppress the deletion of a minute structure or the like exceeding the noise component by the predictor 171 by causing the predictor 171 to perform machine learning so that the identification by the identifier 172 is not successful, thereby improving the prediction accuracy. it can.

  That is, the predictor 171 can prevent the original structure from remaining in the predicted image. Therefore, the identification accuracy by the classifier 172 can be reduced. This means that the difference image and the predicted image cannot be distinguished. Therefore, the remaining minute area of the subject in the difference image is reduced and the noise component is increased, so that the determination accuracy of the part in the subject is also reduced. By feeding back this identification result to the predictor 171, the predictor 171 can generate a predicted image so that the discrimination accuracy by the discriminator 172 is lowered. Therefore, the remaining of the minute region in the difference image is reduced, so that the difference between the medical image and the predicted image is reduced, and the medical image and the predicted image are brought closer. Therefore, the classifier 172 can assist the predictor 171 in improving the prediction accuracy of the predicted image.

  In addition, the processing unit 160 uses the pair of the first difference image and the predicted image of the first part as training data of the discriminator 172, and the first difference image and the predicted image of the first part correspond to each other. The discriminator 172 may be machine-learned using the correspondence information indicating as the teacher data. The processing unit 160 uses the pair of the first difference image and the predicted image of the second part as training data of the discriminator 172, and indicates that the first difference image and the predicted image of the second part do not correspond. The discriminator 172 may be machine-learned using non-corresponding information as teacher data.

  Thereby, the medical image processing apparatus 100A can sequentially learn whether or not the difference image input to the classifier 172 corresponds to the predicted image of an arbitrary part. Therefore, the medical image processing apparatus 100A can improve the identification accuracy for identifying whether or not the difference image newly input by the classifier 172 corresponds to the predicted image of an arbitrary part. Thereby, since the predictor 171 performs machine learning so that it is not discriminated by the discriminator 172, the generation accuracy of the predicted image by the predictor 171 can be improved.

  The processing unit 160 is an image arranged in time series, and may generate a composite image by rendering using the Raycast method, the MIP method, the RaySUM method, or the MinIP method.

  Accordingly, the medical image processing apparatuses 100 and 100A can visualize the composite image by various rendering methods. For example, when a composite image is visualized according to the MIP method, the luminance (pixel value) tends to be excessively large by taking the maximum value, and the composite image is difficult to observe, but according to the medical image processing apparatuses 100 and 100A. The luminance value can be suppressed from becoming excessively large, and the composite image can be easily observed.

  The present disclosure is useful for a medical image processing apparatus, a medical image processing method, a medical image processing program, and the like that can suppress the influence of noise during imaging of a subject and improve the image quality of a plurality of phases of medical images.

100 Medical Image Processing Device 110 Port 120 User Interface (UI)
130 Display 140 Processor 150 Memory 151 Learning DB
160 processing unit 161 region extraction unit 162 image generation unit 163 registration processing unit 164 prediction processing unit 165 identification processing unit 166 display control unit 171 predictor 172 classifier 200 CT device

Claims (9)

An acquisition unit for acquiring a plurality of medical images obtained by imaging the first subject a plurality of times in time series;
Generating a plurality of predicted images using a predictor that generates a predicted image corresponding to the remaining medical images of the plurality of medical images based on some of the medical images of the plurality of medical images. A processing unit;
With
The processor is
At least two medical images constituting a part of the plurality of medical images are used as training data, and the remaining medical images of the plurality of medical images are used as teacher data. The remaining medical images are sequentially changed, and the predictor is machine-learned,
A part of the plurality of medical images is input to the predictor, a combination of the part of the medical images is sequentially changed to generate a plurality of the prediction images;
Medical image processing apparatus. The plurality of medical images include a first medical image, a second medical image, and a third medical image,
The plurality of predicted images include a first predicted image, a second predicted image, and a third predicted image,
The processor is
At least the first medical image and the second medical image as training data of the predictor, the third medical image as teacher data of the predictor, and machine learning of the predictor,
At least the second medical image and the third medical image as training data of the predictor, the first medical image as teacher data of the predictor, and machine learning of the predictor,
At least the third medical image and the first medical image as training data of the predictor, the second medical image as teacher data of the predictor, and machine learning of the predictor,
Inputting at least the first medical image and the second medical image to the predictor to generate the first predicted image;
Inputting at least the second medical image and the third medical image to the predictor to generate the second predicted image;
Inputting at least the third medical image and the first medical image to the predictor to generate the third predicted image;
Synthesizing at least the first predicted image, the second predicted image, and the third predicted image to obtain a combined image;
The medical image processing apparatus according to claim 1. The processing unit aligns the first medical image, the second medical image, and the third medical image, and then causes the predictor to perform the machine learning.
The medical image processing apparatus according to claim 2. The acquisition unit includes a fourth medical image, a fifth medical image, and a sixth medical image obtained by imaging a second subject different from the first subject at least three times in time series. Get an image,
The processor is
At least the fourth medical image and the fifth medical image as training data of the predictor, the sixth medical image as teacher data of the predictor, and machine learning of the predictor,
At least the fifth medical image and the sixth medical image as training data of the predictor, the fourth medical image as teacher data of the predictor, and machine learning of the predictor,
Machine learning of the predictor using at least the sixth medical image and the fourth medical image as training data of the predictor, and the fifth medical image as teacher data of the predictor,
The medical image processing apparatus according to claim 2 or 3. The processor is
Obtaining from the third medical image a medical image of a first part of the first subject and a medical image of a second part different from the first part;
From the first predicted image, obtain a predicted image of the first part and a predicted image of the second part in the first subject,
Generating a first difference image indicating a difference between the medical image of the first part and the predicted image of the first part;
Generating a second difference image indicating a difference between the medical image of the second part and the predicted image of the second part;
A pair of any one of the first difference image and the second difference image and a predicted image of any part of the predicted image of the first part and the predicted image of the second part. Produces
Causing the predictor to machine-learn so that identification by a discriminator for identifying whether or not any one of the difference images in the pair corresponds to a predicted image of any one of the parts,
The medical image processing apparatus according to any one of claims 2 to 4. The processor is
Correspondence indicating that the pair of the first difference image and the predicted image of the first part is training data of the classifier, and the first difference image and the predicted image of the first part correspond to each other. Using the information as teacher data, let the classifier machine-learn,
A pair of the first difference image and the predicted image of the second part is used as training data for the discriminator, and the first difference image and the predicted image of the second part do not correspond to each other. The medical image processing apparatus according to claim 5, wherein the classifier is machine-learned using correspondence information as teacher data. The processing unit visualizes the composite image according to the Raycast method, the MIP method, the RaySUM method, or the MinIP method.
The medical image processing apparatus according to any one of claims 2 to 6. A medical image processing method in a medical image processing apparatus,
Obtaining a plurality of medical images obtained by imaging the first subject a plurality of times in time series;
Generating a plurality of predicted images using a predictor that generates a predicted image corresponding to the remaining medical images of the plurality of medical images based on some of the medical images of the plurality of medical images. Steps,
Have
The step of generating the predicted image includes
At least two medical images constituting a part of the plurality of medical images are used as training data, and the remaining medical images of the plurality of medical images are used as teacher data. Sequentially changing the remaining medical images and machine learning the predictor;
Including inputting a part of the plurality of medical images to the predictor and sequentially changing a combination of the part of the medical images to generate a plurality of the predicted images.
Medical image processing method.   A medical image processing program for causing a computer to execute the medical image processing method according to claim 8.