WO2023060735A1 - 图像生成模型训练及图像生成方法、装置、设备和介质 - Google Patents

图像生成模型训练及图像生成方法、装置、设备和介质 Download PDF

Info

Publication number
WO2023060735A1
WO2023060735A1 PCT/CN2021/136942 CN2021136942W WO2023060735A1 WO 2023060735 A1 WO2023060735 A1 WO 2023060735A1 CN 2021136942 W CN2021136942 W CN 2021136942W WO 2023060735 A1 WO2023060735 A1 WO 2023060735A1
Authority
WO
WIPO (PCT)
Prior art keywords
image
modality
image generation
generation model
images
Prior art date
Application number
PCT/CN2021/136942
Other languages
English (en)
French (fr)
Inventor
邹莉娴
刘新
梁栋
郑海荣
Original Assignee
中国科学院深圳先进技术研究院
Priority date (The priority date is an assumption and is not a legal conclusion. Google has not performed a legal analysis and makes no representation as to the accuracy of the date listed.)
Filing date
Publication date
Application filed by 中国科学院深圳先进技术研究院 filed Critical 中国科学院深圳先进技术研究院
Publication of WO2023060735A1 publication Critical patent/WO2023060735A1/zh

Links

Images

Classifications

    • G—PHYSICS
    • G06—COMPUTING; CALCULATING OR COUNTING
    • G06N—COMPUTING ARRANGEMENTS BASED ON SPECIFIC COMPUTATIONAL MODELS
    • G06N3/00—Computing arrangements based on biological models
    • G06N3/02—Neural networks
    • G06N3/04—Architecture, e.g. interconnection topology
    • G—PHYSICS
    • G06—COMPUTING; CALCULATING OR COUNTING
    • G06N—COMPUTING ARRANGEMENTS BASED ON SPECIFIC COMPUTATIONAL MODELS
    • G06N3/00—Computing arrangements based on biological models
    • G06N3/02—Neural networks
    • G06N3/08—Learning methods
    • G—PHYSICS
    • G06—COMPUTING; CALCULATING OR COUNTING
    • G06T—IMAGE DATA PROCESSING OR GENERATION, IN GENERAL
    • G06T7/00—Image analysis
    • G—PHYSICS
    • G06—COMPUTING; CALCULATING OR COUNTING
    • G06T—IMAGE DATA PROCESSING OR GENERATION, IN GENERAL
    • G06T7/00—Image analysis
    • G06T7/30—Determination of transform parameters for the alignment of images, i.e. image registration
    • G—PHYSICS
    • G06—COMPUTING; CALCULATING OR COUNTING
    • G06V—IMAGE OR VIDEO RECOGNITION OR UNDERSTANDING
    • G06V10/00—Arrangements for image or video recognition or understanding
    • G06V10/20—Image preprocessing
    • G06V10/25—Determination of region of interest [ROI] or a volume of interest [VOI]
    • G—PHYSICS
    • G06—COMPUTING; CALCULATING OR COUNTING
    • G06V—IMAGE OR VIDEO RECOGNITION OR UNDERSTANDING
    • G06V10/00—Arrangements for image or video recognition or understanding
    • G06V10/70—Arrangements for image or video recognition or understanding using pattern recognition or machine learning
    • G06V10/82—Arrangements for image or video recognition or understanding using pattern recognition or machine learning using neural networks

Definitions

  • the embodiments of the present application relate to the technical field of image processing, for example, to an image generation model training and image generation method, device, device, and medium.
  • medical imaging provides a variety of medical images for clinical diagnosis, including ultrasound images, magnetic resonance images, computer tomography images, positron emission tomography (Positron emission tomography, PET) images and other modal images.
  • Different modalities of medical images have their own advantages and disadvantages in terms of spatial and temporal resolution, and can provide different information about relevant organs and tissues of the human body.
  • the embodiment of the present application discloses an image generation model training and image generation method, device, device and medium.
  • the embodiment of the present application provides a method for training an image generation model, the method comprising:
  • the embodiment of the present application provides an image generation method, the method comprising:
  • the original modality image is input to the target image generation model trained by the image generation model training method described in any embodiment to obtain the target conversion modality image.
  • an image generation model training device which includes:
  • the sample acquisition module is configured to acquire different modal images of the site of interest, and form at least one set of image pairs based on the different modal images;
  • the sample data input module is configured to, for each image pair in the at least one group of image pairs, input the first modality image in the image pair into the preset image generation model, and use the image pair in the
  • the second modal image is used as a learning standard, and supervised model training is performed on the preset image generation model;
  • the model training module is configured to obtain a target image generation model in response to the loss function of the preset image generation model meeting a preset convergence condition, and complete the model training process.
  • the embodiment of the present application also provides an image generating device, which includes:
  • An image data acquisition module configured to acquire an original modality image, and determine a target conversion modality of the original modality image
  • the image generation module is configured to input the original modality image into the target image generation model trained by the image generation model training method described in any embodiment, to obtain an image of the target converted modality.
  • the embodiment of the present application also provides a computer device, the computer device comprising:
  • processors one or more processors
  • memory for storing one or more programs
  • the one or more processors are made to implement an image generation model training method or an image generation method as provided in any embodiment of the present application.
  • the embodiment of the present application also provides a computer-readable storage medium, on which a computer program is stored, and when the program is executed by a processor, an image generation model training method as provided in any embodiment of the present application is implemented. or image generation methods.
  • FIG. 1 is a flow chart of an image generation model training method provided in Embodiment 1 of the present application;
  • FIG. 2 is a schematic structural diagram of an image generation model provided in Embodiment 1 of the present application.
  • FIG. 3 is a flowchart of an image generation method provided in Embodiment 2 of the present application.
  • Fig. 4 is a schematic diagram of an example of image generation provided by Embodiment 2 of the present application.
  • Fig. 5 is a schematic structural diagram of an image generation model training device provided in Embodiment 3 of the present application.
  • FIG. 6 is a schematic structural diagram of an image generation device provided in Embodiment 4 of the present application.
  • FIG. 7 is a schematic structural diagram of a computer device provided in Embodiment 5 of the present application.
  • the embodiment of the present application discloses an image generation model training and image generation method, device, device and medium.
  • FIG. 1 is a flow chart of an image generation model training method provided in Embodiment 1 of the present application.
  • This embodiment can train models for generating medical images of different modalities.
  • the method can be executed by an image generation model training device, which can be realized by software and/or hardware, and integrated in a computer device with application development function.
  • the image generation model training method includes the following steps:
  • images of different modalities refer to images formed by different imaging principles and methods.
  • medical imaging including imaging modalities such as ultrasound imaging, magnetic resonance imaging, electronic computed tomography imaging, and positron emission tomography imaging
  • medical images of different modalities can provide different information about relevant organs and tissues of the human body.
  • ultrasound imaging can coherently and dynamically observe the movement and function of organs, and can track lesions without being limited by their imaging layers. It is a patient-friendly and widely used medical imaging modality.
  • Magnetic resonance imaging has the advantages of high soft tissue resolution, multiple imaging parameters, large amount of image information and no ionization damage, and has been widely used in clinic.
  • MRI is sensitive to patient motion and prone to artifacts, and is insensitive to calcification.
  • Positron emission tomography is a nuclear imaging technique (also known as molecular imaging), which can display metabolic processes in vivo and is mostly used for functional imaging, but it also has ionizing radiation, and the spatial structure information on the image is difficult to distinguish.
  • the site of interest is usually the site where the lesion is located in the clinical image, which is the site that the clinician needs to observe, that is, the target site during the imaging process.
  • at least two modal imaging can be performed on the image of each interested part, and the medical images in the corresponding modalities can be obtained as training samples for the image generation model. If you want to obtain more comprehensive structural organization information of the region of interest, you can obtain images in all imaging modalities of each sensitive region.
  • multiple modal images can be combined in pairs to obtain multiple sets of image pairs.
  • the medical images of the region of interest include ultrasound images, magnetic resonance images, computerized tomography images and positron emission tomography images.
  • Ultrasound images and magnetic resonance images can be combined into a set of image pairs
  • ultrasound images and computerized tomography images can be combined into a set of image pairs
  • ultrasound images and positron emission tomography images can be combined into a set of image pairs
  • magnetic resonance images can be combined into a set of image pairs.
  • the image and the electron computed tomography image form a set of image pairs
  • the magnetic resonance image and the positron emission tomography image form a set of image pairs
  • the electron computed tomography images and the positron emission tomography images form a set of image pairs.
  • the first modality image may be any image in the image pair, and the second modality image is another image in the image pair.
  • the first modality image may be confirmed according to requirements for subsequent image generation. Exemplarily, in clinical practice, it usually takes less time to acquire ultrasound images and the cost is lower.
  • the ultrasound image can be used as the first modality image in the image pair
  • the other image in the image pair is used as the second modality image.
  • two image generation models are trained simultaneously.
  • the first modality image and the second modality image are respectively one of ultrasound images, magnetic resonance images, computerized tomography images or positron emission tomography images, and the first modality
  • the first modality image and the second modality image are images of different modalities.
  • the first modality image is registered to the second modality image, so that the space coordinate system where the first modality image is located is consistent with the second modality image.
  • the spatial coordinate systems where the two modal images are located remain consistent.
  • the registered first modality image is used as input data of the preset image generation model, and input into the preset image generation model, and then model training is performed.
  • the spatial registration of images can use methods such as Fourier transform method, affine transform method, maximum mutual information method or scale invariant feature transformation method for spatial transformation. Or you can also use the original image and its spatial registration image to form a sample pair to train the image registration model. After the training is completed, you can directly input the first modal image into the trained image registration model, and the image registration model The quasi-model is output to obtain a registration map of the first modality image after the coordinate transformation.
  • the preset image generation model can be a convolutional neural network or a non-convolutional neural network. Any deep learning neural network structure that can extract and learn image features and further output cross-modal images is fine.
  • the structure of the preset image generation model may refer to the structure shown in FIG. 2 .
  • the preset image generation model includes an input layer, a convolutional layer, a residual density network (Residual Dense Net, RDN), an output connection layer, and an output layer.
  • the first modal image passes through the input layer to realize image input; then, it passes through two convolutional layers for shallow feature extraction.
  • the extracted features are used as the input of the residual density block (Residual Dense Block, RDB) in the subsequent residual density network.
  • RDB residual Dense Block
  • each convolutional layer is not only input to the next convolutional layer, but also input to all subsequent convolutional layers of the RDB to form a density connection.
  • the purpose of dense connections is to make both deep and shallow convolutional layers work.
  • the output of these convolutional layers is connected (concatenate) together for fusion registration, and then a global residual learning is performed through a convolutional layer and a residual connection.
  • the result of the learning is applied to a convolutional layer for further deep feature extraction, and finally the image is output by the output layer, which is the second modality image converted from the first modality image.
  • the supervised learning process is to train the preset image generation model through known input and output training samples, so as to obtain an optimal image generation model, that is, the target image generation model.
  • the loss function of the preset image generation model is calculated by comparing the result of each output with the corresponding known output image, and when the loss function does not converge, adjust the parameters in the preset image generation model Continue learning until the loss function converges, satisfy the preset condition (preset convergence condition), determine the parameters of the preset image generation model, obtain the target image generation model, and complete the training process.
  • the target image generation model can be used to generate the second modality image from the first modality image. Therefore, after the image of the first modality is collected, the medical image of the second modality can be obtained through the target image generation model without further clinical image scanning.
  • different modality images of the part of interest are acquired in advance, and at least one group of image pairs is formed based on the different modality images; for each group of image pairs, the first modality image in the image pair is input into the preset image generation model, and use the second modality image in the image pair as the learning standard to perform supervised model training on the preset image generation model; in response to the preset image generation model, the loss function satisfies the preset convergence conditions, the target image generation model is obtained, and the model training process is completed.
  • the embodiment of the present application solves the time-consuming and costly problem of scanning and acquiring images of different modalities in order to obtain comprehensive information of the region of interest, realizes the generation of images of other modalities based on images of one modality, and improves the acquisition of multiple modalities.
  • the efficiency of a single modality image saves the time and cost of acquiring multi-modal images.
  • Fig. 3 is a flow chart of an image generation method provided by Embodiment 2 of the present application.
  • This embodiment belongs to the same inventive concept as the image generation model training method in the above embodiment, and describes how to use the trained image generation model to generate The process of targeting the modal image.
  • the method can be executed by an image generating device, and the device can be realized by software and/or hardware, and integrated in a computer device with application development function.
  • the image generation method includes the following steps:
  • the original modality image refers to an image of a modality that has been acquired clinically, and may be a modality in images such as an ultrasound image, a magnetic resonance image, an electronic computer tomography image, or a positron emission tomography image. state image.
  • the target conversion modality is the modality of the target image that is expected to be generated according to the original modality image, and may be one or more modalities different from the original modality image.
  • the target image generation model is the target image generation model trained according to the image generation model training method in the above embodiment, and the training samples for training the target image generation model are determined according to the original modality and the target conversion modality.
  • the image of the original modality is input into the trained target image generation model, and the image of the target converted modality can be generated by the target image generation model.
  • multiple target image generation models can be pre-trained, and the original modality images are simultaneously input into multiple trained target image generation models to obtain Multiple target transform modal images.
  • the target conversion modality includes a first modality, a second modality, and a third modality
  • the first image generation model that generates the first modality image from the original modality image can be pre-trained, and the training is performed by the first modality
  • the first image generation model, the second image generation model and the third image generation model are cascaded to obtain a new target image generation model.
  • the first modality image, the second modality image and the third modality image can be sequentially obtained.
  • multiple image generation models between multiple target modalities include a first image generation model that generates a first modal image from an original modal image, and a second image generation model that generates a second modal image from a first modal image.
  • the image generation model is a second image generation model for generating a third modality image from the second modality image.
  • multiple image generation models between multiple target modalities may also be multiple image generation models between two target modalities.
  • the modality-one image is an original modality image, which is an ultrasound image.
  • the modality 2 image is a target conversion modality image, which is a magnetic resonance image.
  • the modality 1 image is input into the deep learning neural network, that is, the target image generation model, and the modality 2 image can be obtained from the model output.
  • the original modality image is input into the target image generation model, so as to obtain the image in the target converted modality.
  • the embodiment of the present application solves the time-consuming and costly problem of scanning and acquiring images of different modalities in order to obtain comprehensive information of the region of interest, realizes the generation of images of other modalities based on images of one modality, and improves the acquisition of multiple modalities.
  • the efficiency of a single modality image saves the time and cost of acquiring multi-modal images.
  • FIG. 5 is a schematic structural diagram of an image generation model training device provided in Embodiment 3 of the present application. This embodiment can train models for generating medical images of different modalities.
  • the device can be implemented by software and/or hardware, integrated In computer equipment with application development capabilities.
  • the image generation model training device includes: a sample acquisition module 310 , a sample data input module 320 and a model training module 330 .
  • the sample acquisition module 310 is configured to acquire different modality images of the site of interest, and forms at least one group of image pairs based on the different modality images;
  • the sample data input module 320 is configured to, for each group of image pairs, The first modal image in the image pair is input into the preset image generation model, and the second modal image in the image pair is used as a learning standard to carry out supervised model training on the preset image generation model;
  • model training Module 330 configured to obtain a target image generation model in response to the loss function of the preset image generation model meeting a preset convergence condition, and complete the model training process.
  • different modality images of the part of interest are acquired in advance, and at least one group of image pairs is formed based on the different modality images; for each group of image pairs, the first modality image in the image pair is input into the preset image generation model, and use the second modality image in the image pair as the learning standard to perform supervised model training on the preset image generation model; when the loss function of the preset image generation model satisfies the preset convergence condition When , the target image generation model is obtained, and the model training process is completed.
  • the embodiment of the present application solves the time-consuming and costly problem of scanning and acquiring images of different modalities in order to obtain comprehensive information of the region of interest, realizes the generation of images of other modalities based on images of one modality, and improves the acquisition of multiple modalities.
  • the efficiency of a single modality image saves the time and cost of acquiring multi-modal images.
  • the image generation model training device further includes an image registration module, configured to, before performing supervised model training on the preset image generation model, align the first modality image to the second Two-modal image registration.
  • sample data input module 320 is further configured to: input the registered first modality image into the preset image generation model as input data of the preset image generation model.
  • the image registration module is also set to:
  • the first modality image and the second modality image are respectively one of ultrasound images, magnetic resonance images, computerized tomography images or positron emission tomography images, and the The first modality image and the second modality image are images of different modality.
  • the image generation model training device provided in the embodiment of the present application can execute the image generation model training method provided in any embodiment of the present application, and has corresponding functional modules and beneficial effects for executing the method.
  • Fig. 6 is a schematic structural diagram of an image generating device provided in Embodiment 4 of the present application.
  • This embodiment can generate medical images of different modalities based on one modality image.
  • the device can be implemented by software and/or hardware, and integrated in In computer equipment with application development capabilities.
  • the image generation device includes: an image data acquisition module 410 and an image generation module 420 .
  • the image data acquisition module 410 is configured to acquire the original modality image, and determine the target conversion modality of the original modality image; the image generation module 420 is configured to input the original modality image to the
  • the target image generation model trained by the above image generation model training method is used to obtain the image of the target conversion mode.
  • the original modality image is input into the target image generation model, so as to obtain the image in the target converted modality.
  • the embodiment of the present application solves the time-consuming and costly problem of scanning and acquiring images of different modalities in order to obtain comprehensive information of the region of interest, realizes the generation of images of other modalities based on images of one modality, and improves the acquisition of multiple modalities.
  • the efficiency of a single modality image saves the time and cost of acquiring multi-modal images.
  • the image generation module 420 is further configured to: when the target conversion modality includes multiple target modalities, obtain an image generation model between the multiple target modalities through pre-training, and combine the multiple target modalities The image generation model cascades between target modalities to obtain an updated target image generation model;
  • the image generating device provided in the embodiment of the present application can execute the image generating method provided in any embodiment of the present application, and has corresponding functional modules and beneficial effects for executing the method.
  • FIG. 7 is a schematic structural diagram of a computer device provided in Embodiment 5 of the present application.
  • FIG. 7 shows a block diagram of an exemplary computer device 12 suitable for implementing embodiments of the present application.
  • the computer device 12 shown in FIG. 7 is only an example, and should not limit the functions and scope of use of this embodiment of the present application.
  • the computer device 12 can be any terminal device with computing capability, such as an intelligent controller, a server, a mobile phone and other terminal devices.
  • computer device 12 takes the form of a general-purpose computing device.
  • Components of computer device 12 may include, but are not limited to, one or more processors or processing units 16 , system memory (memory) 28 , bus 18 connecting various system components including system memory 28 and processing unit 16 .
  • Bus 18 represents one or more of several types of bus structures, including a memory bus or memory controller, a peripheral bus, an accelerated graphics port, a processor, or a local bus using any of a variety of bus structures.
  • these architectures include but are not limited to Industry Standard Architecture (ISA, Industry Standard Architecture) bus, Micro Channel Architecture (MCA, Micro Channel Architecture) bus, Enhanced ISA bus, Video Electronics Standards Association (VESA, Video Electronics Standards Association) local bus and peripheral component interconnect (PCI, Peripheral Component Interconnect) bus.
  • Computer device 12 typically includes a variety of computer system readable media. These media can be any available media that can be accessed by computer device 12 and include both volatile and nonvolatile media, removable and non-removable media.
  • System memory 28 may include computer system readable media in the form of volatile memory, such as random access memory (RAM, Random Access Memory) 30 and/or cache memory 32 .
  • Computer device 12 may further include other removable/non-removable, volatile/nonvolatile computer system storage media.
  • storage system 34 may be used to read and write to non-removable, non-volatile magnetic media (not shown in FIG. 7, commonly referred to as a "hard drive”).
  • a disk drive for reading and writing to removable nonvolatile disks e.g., "floppy disks”
  • removable nonvolatile optical disks e.g., CD-ROM, DVD-ROM or other optical media
  • each drive may be connected to bus 18 via one or more data media interfaces.
  • System memory 28 may include at least one program product having a set (eg, at least one) of program modules configured to perform the functions of various embodiments of the present application.
  • Program/utility 40 may be stored, for example, in system memory 28 as a set (at least one) of program modules 42 including, but not limited to, an operating system, one or more application programs, other program modules, and program data, each or some combination of these examples may include the implementation of the network environment.
  • the program modules 42 generally perform the functions and/or methods of the embodiments described herein.
  • the computer device 12 may also communicate with one or more external devices 14 (e.g., a keyboard, pointing device, display 24, etc.), and with one or more devices that enable a user to interact with the computer device 12, and/or with Any device (eg, network card, modem, etc.) that enables the computing device 12 to communicate with one or more other computing devices. Such communication may occur through input/output (I/O) interface 22 .
  • the computer device 12 can also communicate with one or more networks (such as a local area network (LAN, Local Area Network), a wide area network (WAN, Wide Area Network) and/or a public network, such as the Internet) through the network adapter 20. As shown in FIG.
  • network adapter 20 communicates with other modules of computer device 12 via bus 18 .
  • bus 18 It should be appreciated that although not shown in FIG. 7 , other hardware and/or software modules may be used in conjunction with computer device 12, including but not limited to: microcode, device drivers, redundant processing units, external disk drive arrays, RAID (Redundant Arrays of Independent Disks, disk array) systems, tape drives, and data backup storage systems.
  • the processing unit 16 executes a variety of functional applications and data processing by running the program stored in the system memory 28, such as implementing the image generation model training method provided in the embodiment of the present application, the method includes:
  • the processing unit 16 executes various functional applications and data processing by running the program stored in the system memory 28.
  • the image generation method provided in the embodiment of the present application can also be implemented, and the method includes:
  • the original modality image is input to the target image generation model trained by the image generation model training method described in any embodiment to obtain the target conversion modality image.
  • the sixth embodiment provides a computer-readable storage medium on which a computer program is stored.
  • the image generation model training method provided in any embodiment of the present application is implemented, including:
  • a computer-readable storage medium provided in this embodiment has a computer program stored thereon, and when the program is executed by a processor, the image generation method provided in any embodiment of the present application can also be implemented, including:
  • the original modality image is input to the target image generation model trained by the image generation model training method described in any embodiment to obtain the target conversion modality image.
  • the computer storage medium in the embodiments of the present application may use any combination of one or more computer-readable media.
  • the computer readable medium may be a computer readable signal medium or a computer readable storage medium.
  • a computer-readable storage medium may be, for example but not limited to, an electrical, magnetic, optical, electromagnetic, infrared, or semiconductor system, device, or device, or any combination thereof.
  • a computer-readable storage medium may be any tangible medium that contains or stores a program that can be used by or in conjunction with an instruction execution system, apparatus, or device.
  • a computer readable signal medium may include a data signal carrying computer readable program code in baseband or as part of a carrier wave traveling as a data signal. Such propagated data signals may take many forms, including but not limited to electromagnetic signals, optical signals, or any suitable combination of the foregoing.
  • a computer-readable signal medium may also be any computer-readable medium other than a computer-readable storage medium, which can send, propagate, or transmit a program for use by or in conjunction with an instruction execution system, apparatus, or device. .
  • the storage medium may be a non-transitory storage medium.
  • the program code contained on the computer-readable medium can be transmitted by any appropriate medium, including but not limited to: wireless, electric wire, optical cable, RF (Radio Frequency, radio frequency), etc., or any suitable combination of the above.
  • Computer program codes for performing the operations of the present application may be written in one or more programming languages or combinations thereof, including object-oriented programming languages such as Java, Smalltalk, C++, and conventional A procedural programming language, such as the "C" language or similar programming language.
  • the program code may execute entirely on the user's computer, partly on the user's computer, as a stand-alone software package, partly on the user's computer and partly on a remote computer or entirely on the remote computer or server.
  • the remote computer can be connected to the user computer through any kind of network, including a local area network (LAN) or a wide area network (WAN), or it can be connected to an external computer (such as through the Internet using an Internet service provider). connect).
  • LAN local area network
  • WAN wide area network
  • connect such as AT&T, MCI, Sprint, EarthLink, MSN, GTE, etc.
  • each module or each step of the above-mentioned application can be realized by a general-purpose computing device, and they can be concentrated on a single computing device, or distributed on a network formed by multiple computing devices.
  • they can be implemented with executable program codes of computer devices, so that they can be stored in storage devices and executed by computing devices, or they can be made into individual integrated circuit modules, or a plurality of modules in them Or the steps are fabricated into a single integrated circuit module to realize.
  • the application is not limited to any specific combination of hardware and software.

Landscapes

  • Engineering & Computer Science (AREA)
  • Theoretical Computer Science (AREA)
  • Physics & Mathematics (AREA)
  • General Physics & Mathematics (AREA)
  • Evolutionary Computation (AREA)
  • General Health & Medical Sciences (AREA)
  • Computer Vision & Pattern Recognition (AREA)
  • Health & Medical Sciences (AREA)
  • Artificial Intelligence (AREA)
  • Computing Systems (AREA)
  • Software Systems (AREA)
  • Data Mining & Analysis (AREA)
  • Life Sciences & Earth Sciences (AREA)
  • Biomedical Technology (AREA)
  • Biophysics (AREA)
  • Computational Linguistics (AREA)
  • Multimedia (AREA)
  • Molecular Biology (AREA)
  • General Engineering & Computer Science (AREA)
  • Mathematical Physics (AREA)
  • Medical Informatics (AREA)
  • Databases & Information Systems (AREA)
  • Image Analysis (AREA)
  • Measuring And Recording Apparatus For Diagnosis (AREA)

Abstract

本申请实施例公开了一种图像生成模型训练及图像生成方法、装置、设备和介质，其中，方法包括：获取感兴趣部位的不同模态图像，并基于所述不同模态图像组成至少一组图像对；针对每一组图像对，将图像对中的第一模态图像输入到预设图像生成模型中，并以图像对中的第二模态图像作为学习标准，对所述预设图像生成模型进行有监督的模型训练；响应于所述预设图像生成模型的损失函数满足预设收敛条件，得到目标图像生成模型，完成模型训练过程。

Description

图像生成模型训练及图像生成方法、装置、设备和介质
本申请要求在2021年10月11日提交中国专利局、申请号为202111180703.3的中国专利申请的优先权，以上申请的全部内容通过引用结合在本申请中。
技术领域
本申请实施例涉及图像处理技术领域，例如涉及一种图像生成模型训练及图像生成方法、装置、设备和介质。
背景技术
随着医学工程和计算机技术的发展,医学影像学为临床诊断提供了多种模态的医学图像，包括超声图像、磁共振图像、电子计算机断层扫描图像、正电子发射断层扫描(Positron emission tomography，PET)图像等多种模态图像。不同模态的医学影像在时空分辨率方面有自身的优缺点，可以提供人体相关脏器和组织的不同信息。
发明内容
本申请实施例公开一种图像生成模型训练及图像生成方法、装置、设备和介质。
第一方面，本申请实施例提供了一种图像生成模型训练方法，该方法包括：
获取感兴趣部位的不同模态图像，并基于所述不同模态图像组成至少一组图像对；
针对所述至少一组图像对中的每一组图像对，将图像对中的第一模态图像输入到预设图像生成模型中，并以图像对中的第二模态图像作为学习标准，对所述预设图像生成模型进行有监督的模型训练；
响应于所述预设图像生成模型的损失函数满足预设收敛条件，得到目标图像生成模型，完成模型训练过程。
第二方面，本申请实施例提供了一种图像生成方法，该方法包括：
获取原始模态图像，并确定所述原始模态图像的目标转换模态；
将所述原始模态图像输入至由任一实施例所述的图像生成模型训练方法训练得到的目标图像生成模型，得到目标转换模态的图像。
第三方面，本申请实施例还提供了一种图像生成模型训练装置，该装置包括：
样本获取模块，设置为获取感兴趣部位的不同模态图像，并基于所述不同模态图像组成至少一组图像对；
样本数据输入模块，设置为针对所述至少一组图像对中的每一组图像对，将图像对中的第一模态图像输入到预设图像生成模型中，并以图像对中的第二模态图像作为学习标准，对所述预设图像生成模型进行有监督的模型训练；
模型训练模块，设置为响应于所述预设图像生成模型的损失函数满足预设收敛条件，得到目标图像生成模型，完成模型训练过程。
第四方面，本申请实施例还提供了一种图像生成装置，该装置包括：
图像数据获取模块，设置为获取原始模态图像，并确定所述原始模态图像的目标转换模态；
图像生成模块，设置为将所述原始模态图像输入至由任一实施例所述的图像生成模型训练方法训练得到的目标图像生成模型，得到目标转换模态的图像。
第五方面，本申请实施例还提供了一种计算机设备，所述计算机设备包括：
一个或多个处理器；
存储器，用于存储一个或多个程序；
当所述一个或多个程序被所述一个或多个处理器执行，使得所述一个或多个处理器实现如本申请任意实施例所提供的一种图像生成模型训练方法或图像生成方法。
第六方面，本申请实施例还提供了一种计算机可读存储介质，其上存储有计算机程序，该程序被处理器执行时实现如本申请任意实施例所提供的一种图像生成模型训练方法或图像生成方法。
附图说明
图1是本申请实施例一提供的一种图像生成模型训练方法的流程图；
图2是本申请实施例一提供的一种图像生成模型结构示意图；
图3是本申请实施例二提供的一种图像生成方法流程图；
图4是本申请实施例二提供的一种图像生成实例的示意图；
图5是本申请实施例三提供的一种图像生成模型训练装置结构示意图；
图6是本申请实施例四提供的一种图像生成装置结构示意图；
图7是本申请实施例五提供的一种计算机设备的结构示意图。
具体实施方式
在临床中若是能够同时获取到感兴趣部位的多个模态图像，并对不同模态图像的信息进行集成，十分有利于医生对患者病情的诊断。
然而，不同模态的医学图像成像原理不同，分辨率不同，成像参数也不同。若分别扫描不同模态的图像，非常耗时，特别在时间有限的情况下，不利于实时观察诊断，而且不同模态的信息存在冗余性。
为应对上述状况，本申请实施例公开一种图像生成模型训练及图像生成方法、装置、设备和介质。
下面结合附图和实施例对本申请作进一步说明。可以理解的是，此处所描述的具体实施例仅仅用于解释本申请，而非对本申请的限定。另外还需要说明的是，为了便于描述，附图中仅示出了与本申请相关的部分而非全部结构。
实施例一
图1为本申请实施例一提供的一种图像生成模型训练方法的流程图，本实施例可对生成不同模态医学图像的模型进行训练。该方法可以由图像生成模型训练装置执行，该装置可以由软件和/或硬件的方式来实现，集成于具有应用开发功能的计算机设备中。
如图1所示，图像生成模型训练方法包括以下步骤：
S110、获取感兴趣部位的不同模态图像，并基于所述不同模态图像组成至少一组图像对。
其中，不同模态的图像指通过不同的成像原理和方法进行成像的图像。在医学成像中，包括超声成像、磁共振成像、电子计算机断层扫描成像和正电子发射断层扫描成像等成像模态，不同模态的医学影像可以提供人体相关脏器和组织的不同信息。例如，超声成像可以连贯地、动态地观察脏器的运动和功能，可以追踪病变而不受其成像分层的限制，是一种对患者价格友好且普及面广的医学成像模式。磁共振成像则具有软组织分辨率高、成像参数多、图像信息量大和无电离损害等优点,已在临床广泛应用。但是，磁共振成像对病人体动敏感，易产生伪影，同时对钙化不敏感。电子计算机断层扫描成像是根据人体不同组织对X线的吸收与透过率的不同进行成像，具有高空间分辨率的特征，但对软组织的成像不如磁共振成像敏感，且具有电离辐射。正电子发射断层扫描成像是一种核成像技术(也称为分子成像)，可以显示体内代谢过程，多用于功能成像，但其同样存在电离辐射，且影像上的空间结构信息难以分辨。
而感兴趣部位通常是在临床影像中，病灶点所在的部位，是临床医生需要观察的部位，也即成像过程中的目标部位。根据不同模态图像的需求，针对每 一个感兴趣部位的图像可以进行至少两种模态成像，并获取相应模态下的医学影像，以作为图像生成模型的训练样本。若是希望能够获取感兴趣区域的较为全面的结构组织信息，可以获取各感性区域全部成像模态下的图像。
示例性地，多个模态图像之间可以两两组合得到多组图像对。示例性的，感兴趣部位的医学图像包括超声图像、磁共振图像、电子计算机断层扫描图像和正电子发射断层扫描图像。可以将超声图像和磁共振图像组成一组图像对，将超声图像和电子计算机断层扫描图像组成一组图像对，将超声图像和正电子发射断层扫描图像组成一组图像对，将磁共振图像和电子计算机断层扫描图像组成一组图像对，将磁共振图像和正电子计算机断层扫描图像组成一组图像对，将电子计算机断层扫描图像和正电子计算机断层扫描图像组成一组图像对。
S120、针对每一组图像对，将图像对中的第一模态图像输入到预设图像生成模型中，并以图像对中的第二模态图像作为学习标准，对所述预设图像生成模型进行有监督的模型训练。
在每一个图像对中，第一模态图像可以是图像对中的任意一个图像，那么第二模态图像则是此图像对中的另外一个图像。可以根据后续图像生成的需求确认第一模态图像。示例性的，在临床中，通常获取超声图像所耗的时间较少，且成本较低，可以在包含超声成像模态图像的图像对中，将超声图像作为图像对中的第一模态图像，图像对中的另一图像作为第二模态图像。或者，针对第一模态图像和第二模态图像的不同情况，同时训练两个图像生成模型。
那么，可以理解的是，第一模态图像和第二模态图像分别是超声图像、磁共振图像、电子计算机断层扫描图像或正电子发射断层扫描图像等图像中的一种，且第一模态图像和第二模态图像是不同模态的图像。
在一种实施方式中，在对预设图像生成模型进行有监督的模型训练之前，将第一模态图像向第二模态图像配准，使第一模态图像所在的空间坐标系与第二模态图像所在的空间坐标系保持一致。然后，将配准后的第一模态图像作为预设图像生成模型的输入数据，输入到所述预设图像生成模型中，再进行模型训练。图像的空间配准可以使用傅里叶变换法、仿射变换法、最大互信息法或者尺度不变特征变换法等方法进行空间变换。或者还可以通过原始图像和其空间配准图像组成样本对，对图像配准模型进行训练，训练完成后，可以直接将第一模态图像输入到训练好的图像配准模型中，由图像配准模型输出得到坐标转换后的第一模态图像的配准图。
预设图像生成模型可以是卷积神经网络，也可以是非卷积神经网络，任意 可以进行图像特性提取和学习，并进一步输出跨模态图像的深度学习神经网络结构皆可以。示例性的，预设图像生成模型的结构可参考图2所示的结构。在图2中，预设图像生成模型包括输入层、卷积层、残差密度网络(Residual Dense Net，RDN)、输出连接层和输出层。第一模态图像经过输入层实现图像输入；然后，经过两个卷积层进行浅层特征提取。提取的特征作为后面残差密度网络中的残差密度块(Residual Dense Block，RDB)的输入，每个残差密度模块(例如，RDB 1，RDB 2，……，RDB N)中有若干个卷积层。每个卷积层的输出不仅输入到下一个卷积层，同时输入到RDB后续所有卷积层，构成密度连接。密度连接的目的是，使得深层卷积层和浅层卷积层均发挥作用。再将这些卷积层的输出连接(concatenate)在一起进行融合配准，后再经过一个卷积层和残差连接进行全局残差学习。最终将学习的结果再进行一个卷积层进一步进行深层的特征提取，最后由输出层输出图像，即为由第一模态图像转换后的第二模态图像。
图2中，符号
Figure PCTCN2021136942-appb-000001
表示两者的输出共同作为下一级的输入。
S130、响应于所述预设图像生成模型的损失函数满足预设收敛条件，得到目标图像生成模型，完成模型训练过程。
有监督的学习过程是通过已知输入和输出的训练样本去训练预设图像生成模型，从而得到一个最优的图像生成模型，即目标图像生成模型。在模型训练的过程中，通过比较每一次输出的结果和对应的已知输出图像进行比较，计算预设图像生成模型的损失函数，当损失函数未收敛时，调整预设图像生成模型中的参数继续学习，直到损失函数收敛，满足预设条件(预设收敛条件)，确定预设图像生成模型的参数，得到目标图像生成模型，完成训练过程。
那么，该目标图像生成模型便可以用于由第一模态图像生成第二模态图像。从而，在采集到第一模态的图像后，无需再进行临床上的图像扫描，通过该目标图像生成模型即可获得第二模态的医学图像。
本实施例，预先获取感兴趣部位的不同模态图像，并基于所述不同模态图像组成至少一组图像对；针对每一组图像对，将图像对中的第一模态图像输入到预设图像生成模型中，并以图像对中的第二模态图像作为学习标准，对预设图像生成模型进行有监督的模型训练；响应于预设图像生成模型的损失函数满足预设收敛条件，得到目标图像生成模型，完成模型训练过程。本申请实施例，解决了为获取感兴趣区域全面的信息，扫描获取不同模态的图像，耗时且成本高的问题，实现了基于一种模态的图像生成其他模态图像，提高获取多个模态 图像的效率，节约获取多模态图像的时间成本。
实施例二
图3为本申请实施例二提供的一种图像生成方法的流程图，本实施例与上述实施例中的图像生成模型训练方法属于同一个发明构思，描述了利用已经训练得到的图像生成模型生成目标模态图像的过程。该方法可以由图像生成装置执行，该装置可以由软件和/或硬件的方式来实现，集成于具有应用开发功能的计算机设备中。
如图3所示，图像生成方法包括以下步骤：
S210、获取原始模态图像，并确定所述原始模态图像的目标转换模态。
其中，原始模态图像是指在临床上已经获取到的一种模态的图像，可以是超声图像、磁共振图像、电子计算机断层扫描图像或正电子发射断层扫描图像等图像中的一种模态的图像。目标转换模态则是希望根据原始模态图像生成的目标图像的模态，可以是一种或多种与原始模态图像不同的模态。
S220、将所述原始模态图像输入至由任一实施例所述的图像生成模型训练方法训练得到的目标图像生成模型，得到目标转换模态的图像。
其中，目标图像生成模型是根据上述实施例中的图像生成模型训练方法训练得到的目标图像生成模型，训练目标图像生成模型时的训练样本是根据原始模态和目标转换模态确定的。将原始模态的图像输入到训练好的目标图像生成模型中，便可有由目标图像生成模型生成目标转换模态的图像。
在一实施例中，当目标转换模态包括多个目标模态时，可以预先训练多个目标图像生成模型，将原始模态图像同时输入到多个训练好的目标图像生成模型中，以得到多个目标转换模态图像。
或者，还可以预先训练得到多个目标模态间的图像生成模型，并将多个目标模态间的图像生成模型级联得到更新后的目标图像生成模型；将所述原始模态图像输入至所述更新后的目标图像生成模型，得到所述多个目标模态的图像。示例性的，目标转换模态包括第一模态、第二模态和第三模态，可以预先训练由原始模态图像生成第一模态图像的第一图像生成模型，训练由第一模态图像生成第二模态图像的第二图像生成模型，训练由第二模态图像生成第三模态图像的第二图像生成模型。再例如，将第一图像生成模型、第二图像生成模型和第三图像生成模型级联，得到新的目标图像生成模型。将原始模态图像输入到这一新的目标图像生成模型，便可以依次得到第一模态图像、第二模态图像和第三模态图像。
例如，多个目标模态间的多个图像生成模型，包括，由原始模态图像生成第一模态图像的第一图像生成模型，由第一模态图像生成第二模态图像的第二图像生成模型，由第二模态图像生成第三模态图像的第二图像生成模型。
例如，多个目标模态间的多个图像生成模型，还可为，两个目标模态间的多个图像生成模型。
在一个实例中，如图4所示，模态一图像为原始模态图像，是超声图像。模态二图像是目标转换模态图像，是磁共振图像。将模态一图像输入到深度学习神经网络，即目标图像生成模型中，便可以由模型输出得到模态二图像。
本实施例，通过预先训练由原始模态图像生成目标转换模态图像的目标图像生成模型，将原始模态图像输入到目标图像生成模型中，从而得到目标转换模态下的图像。本申请实施例，解决了为获取感兴趣区域全面的信息，扫描获取不同模态的图像，耗时且成本高的问题，实现了基于一种模态的图像生成其他模态图像，提高获取多个模态图像的效率，节约获取多模态图像的时间成本。
实施例三
图5为本申请实施例三提供的图像生成模型训练装置的结构示意图，本实施例可对生成不同模态医学图像的模型进行训练，该装置可以由软件和/或硬件的方式来实现，集成于具有应用开发功能的计算机设备中。
如图5所示，图像生成模型训练装置包括：样本获取模块310、样本数据输入模块320和模型训练模块330。
样本获取模块310，设置为获取感兴趣部位的不同模态图像，并基于所述不同模态图像组成至少一组图像对；样本数据输入模块320，设置为针对每一组图像对，将图像对中的第一模态图像输入到预设图像生成模型中，并以图像对中的第二模态图像作为学习标准，对所述预设图像生成模型进行有监督的模型训练；模型训练模块330，设置为响应于所述预设图像生成模型的损失函数满足预设收敛条件，得到目标图像生成模型，完成模型训练过程。
本实施例，预先获取感兴趣部位的不同模态图像，并基于所述不同模态图像组成至少一组图像对；针对每一组图像对，将图像对中的第一模态图像输入到预设图像生成模型中，并以图像对中的第二模态图像作为学习标准，对预设图像生成模型进行有监督的模型训练；当预设图像生成模型的损失函数满足预设收敛条件时，得到目标图像生成模型，完成模型训练过程。本申请实施例，解决了为获取感兴趣区域全面的信息，扫描获取不同模态的图像，耗时且成本高的问题，实现了基于一种模态的图像生成其他模态图像，提高获取多个模态 图像的效率，节约获取多模态图像的时间成本。
在一种实施方式中，图像生成模型训练装置还包括图像配准模块，设置为在对所述预设图像生成模型进行有监督的模型训练之前，将所述第一模态图像向所述第二模态图像配准。
相应的，样本数据输入模块320还设置为：将配准后的第一模态图像作为所述预设图像生成模型的输入数据，输入到所述预设图像生成模型中。
在一种实施方式中，所述图像配准模块还设置为：
采用预设配准算法或预训练的图像配准神经网络，将所述第一模态图像向所述第二模态图像配准。
在一种实施方式中，所述第一模态图像和所述第二模态图像分别是超声图像、磁共振图像、电子计算机断层扫描图像或正电子发射断层扫描图像中的一种，且所述第一模态图像和所述第二模态图像是不同模态的图像。
本申请实施例所提供的图像生成模型训练装置可执行本申请任意实施例所提供的图像生成模型训练方法，具备执行方法相应的功能模块和有益效果。
实施例四
图6为本申请实施例四提供的图像生成装置的结构示意图，本实施例可基于一种模态图像生成不同模态医学图像，该装置可以由软件和/或硬件的方式来实现，集成于具有应用开发功能的计算机设备中。
如图6所示，图像生成装置包括：图像数据获取模块410和图像生成模块420。
图像数据获取模块410，设置为获取原始模态图像，并确定所述原始模态图像的目标转换模态；图像生成模块420，设置为将所述原始模态图像输入至由任一实施例所述的图像生成模型训练方法训练得到的目标图像生成模型，得到目标转换模态的图像。
本实施例，通过预先训练由原始模态图像生成目标转换模态图像的目标图像生成模型，将原始模态图像输入到目标图像生成模型中，从而得到目标转换模态下的图像。本申请实施例，解决了为获取感兴趣区域全面的信息，扫描获取不同模态的图像，耗时且成本高的问题，实现了基于一种模态的图像生成其他模态图像，提高获取多个模态图像的效率，节约获取多模态图像的时间成本。
在一实施例中，图像生成模块420还设置为：当所述目标转换模态包括多个目标模态时，预先训练得到所述多个目标模态间的图像生成模型，并将所述多个目标模态间的图像生成模型级联得到更新后的目标图像生成模型；
将所述原始模态图像输入至所述更新后的目标图像生成模型，得到所述多个目标模态的图像。
本申请实施例所提供的图像生成装置可执行本申请任意实施例所提供的图像生成方法，具备执行方法相应的功能模块和有益效果。
实施例五
图7为本申请实施例五提供的一种计算机设备的结构示意图。图7示出了适于用来实现本申请实施方式的示例性计算机设备12的框图。图7显示的计算机设备12仅仅是一个示例，不应对本申请实施例的功能和使用范围带来任何限制。计算机设备12可以任意具有计算能力的终端设备，如智能控制器及服务器、手机等终端设备。
如图7所示，计算机设备12以通用计算设备的形式表现。计算机设备12的组件可以包括但不限于：一个或者多个处理器或者处理单元16，系统存储器(内存)28，连接不同系统组件(包括系统存储器28和处理单元16)的总线18。
总线18表示几类总线结构中的一种或多种，包括存储器总线或者存储器控制器，外围总线，图形加速端口，处理器或者使用多种总线结构中的任意总线结构的局域总线。举例来说，这些体系结构包括但不限于工业标准体系结构(ISA，Industry Standard Architecture)总线，微通道体系结构(MCA，Micro Channel Architecture)总线，增强型ISA总线、视频电子标准协会(VESA，Video Electronics Standards Association)局域总线以及外围组件互连(PCI，Peripheral Component Interconnect)总线。
计算机设备12典型地包括多种计算机系统可读介质。这些介质可以是任何能够被计算机设备12访问的可用介质，包括易失性和非易失性介质，可移动的和不可移动的介质。
系统存储器28可以包括易失性存储器形式的计算机系统可读介质，例如随机存取存储器(RAM，Random Access Memory)30和/或高速缓存存储器32。计算机设备12可以进一步包括其它可移动/不可移动的、易失性/非易失性计算机系统存储介质。仅作为举例，存储系统34可以用于读写不可移动的、非易失性磁介质(图7未显示，通常称为“硬盘驱动器”)。尽管图7中未示出，可以提供用于对可移动非易失性磁盘(例如“软盘”)读写的磁盘驱动器，以及对可移动非易失性光盘(例如CD-ROM,DVD-ROM或者其它光介质)读写的光盘驱动器。在这些情况下，每个驱动器可以通过一个或者多个数据介质接口与 总线18相连。系统存储器28可以包括至少一个程序产品，该程序产品具有一组(例如至少一个)程序模块，这些程序模块被配置以执行本申请各实施例的功能。
具有一组(至少一个)程序模块42的程序/实用工具40，可以存储在例如系统存储器28中，这样的程序模块42包括但不限于操作系统、一个或者多个应用程序、其它程序模块以及程序数据，这些示例中的每一个或某种组合中可能包括网络环境的实现。程序模块42通常执行本申请所描述的实施例中的功能和/或方法。
计算机设备12也可以与一个或多个外部设备14(例如键盘、指向设备、显示器24等)通信，还可与一个或者多个使得用户能与该计算机设备12交互的设备通信，和/或与使得该计算机设备12能与一个或多个其它计算设备进行通信的任何设备(例如网卡，调制解调器等等)通信。这种通信可以通过输入/输出(I/O)接口22进行。并且，计算机设备12还可以通过网络适配器20与一个或者多个网络(例如局域网(LAN，Local Area Network)，广域网(WAN，Wide Area Network)和/或公共网络，例如因特网)通信。如图7所示，网络适配器20通过总线18与计算机设备12的其它模块通信。应当明白，尽管图7中未示出，可以结合计算机设备12使用其它硬件和/或软件模块，包括但不限于：微代码、设备驱动器、冗余处理单元、外部磁盘驱动阵列、RAID(Redundant Arrays of Independent Disks，磁盘阵列)系统、磁带驱动器以及数据备份存储系统等。
处理单元16通过运行存储在系统存储器28中的程序，从而执行多种功能应用以及数据处理，例如实现本申请实施例所提供的图像生成模型训练方法，该方法包括：
获取感兴趣部位的不同模态图像，并基于所述不同模态图像组成至少一组图像对；
针对每一组图像对，将图像对中的第一模态图像输入到预设图像生成模型中，并以图像对中的第二模态图像作为学习标准，对所述预设图像生成模型进行有监督的模型训练；
响应于所述预设图像生成模型的损失函数满足预设收敛条件，得到目标图像生成模型，完成模型训练过程。
处理单元16通过运行存储在系统存储器28中的程序，从而执行多种功能应用以及数据处理，例如还可以实现本申请实施例所提供的图像生成方法，该方法包括：
获取原始模态图像，并确定所述原始模态图像的目标转换模态；
将所述原始模态图像输入至由任一实施例所述的图像生成模型训练方法训练得到的目标图像生成模型，得到目标转换模态图像。
实施例六
本实施例六提供了一种计算机可读存储介质，其上存储有计算机程序，该程序被处理器执行时实现如本申请任意实施例所提供的图像生成模型训练方法，包括：
获取感兴趣部位的不同模态图像，并基于所述不同模态图像组成至少一组图像对；
针对每一组图像对，将图像对中的第一模态图像输入到预设图像生成模型中，并以图像对中的第二模态图像作为学习标准，对所述预设图像生成模型进行有监督的模型训练；
响应于所述预设图像生成模型的损失函数满足预设收敛条件，得到目标图像生成模型，完成模型训练过程。
此外，本实施例提供的一种计算机可读存储介质，其上存储有计算机程序，该程序被处理器执行时还可以实现如本申请任意实施例所提供的图像生成方法，包括：
获取原始模态图像，并确定所述原始模态图像的目标转换模态；
将所述原始模态图像输入至由任一实施例所述的图像生成模型训练方法训练得到的目标图像生成模型，得到目标转换模态图像。
本申请实施例的计算机存储介质，可以采用一个或多个计算机可读的介质的任意组合。计算机可读介质可以是计算机可读信号介质或者计算机可读存储介质。计算机可读存储介质例如可以是但不限于：电、磁、光、电磁、红外线、或半导体的系统、装置或器件，或者任意以上的组合。计算机可读存储介质的更具体的例子(非穷举的列表)包括：具有一个或多个导线的电连接、便携式计算机磁盘、硬盘、随机存取存储器(RAM，Random Access Memory)、只读存储器(ROM，Read-Only Memory)、可擦式可编程只读存储器(EPROM或闪存)、光纤、便携式紧凑磁盘只读存储器(CD-ROM)、光存储器件、磁存储器件、或者上述的任意合适的组合。在本文件中，计算机可读存储介质可以是任何包含或存储程序的有形介质，该程序可以被指令执行系统、装置或者器件使用或者与其结合使用。
计算机可读的信号介质可以包括在基带中或者作为载波一部分传播的数据 信号，其中承载了计算机可读的程序代码。这种传播的数据信号可以采用多种形式，包括但不限于电磁信号、光信号或上述的任意合适的组合。计算机可读的信号介质还可以是计算机可读存储介质以外的任何计算机可读介质，该计算机可读介质可以发送、传播或者传输用于由指令执行系统、装置或者器件使用或者与其结合使用的程序。
存储介质可以是非暂态(non-transitory)存储介质。
计算机可读介质上包含的程序代码可以用任何适当的介质传输，包括但不限于：无线、电线、光缆、RF(Radio Frequency，射频)等等，或者上述的任意合适的组合。
可以以一种或多种程序设计语言或其组合来编写用于执行本申请操作的计算机程序代码，所述程序设计语言包括面向对象的程序设计语言，诸如Java、Smalltalk、C++，还包括常规的过程式程序设计语言，诸如“C”语言或类似的程序设计语言。程序代码可以完全地在用户计算机上执行、部分地在用户计算机上执行、作为一个独立的软件包执行、部分在用户计算机上部分在远程计算机上执行、或者完全在远程计算机或服务器上执行。在涉及远程计算机的情形中，远程计算机可以通过任意种类的网络，包括局域网(LAN)或广域网(WAN)，连接到用户计算机，或者，可以连接到外部计算机(例如利用因特网服务提供商来通过因特网连接)。
本领域普通技术人员应该明白，上述的本申请的各模块或各步骤可以用通用的计算装置来实现，它们可以集中在单个计算装置上，或者分布在多个计算装置所组成的网络上，可选地，他们可以用计算机装置可执行的程序代码来实现，从而可以将它们存储在存储装置中由计算装置来执行，或者将它们分别制作成各个集成电路模块，或者将它们中的多个模块或步骤制作成单个集成电路模块来实现。这样，本申请不限制于任何特定的硬件和软件的结合。
本领域技术人员会理解，本申请不限于这里所述的特定实施例，对本领域技术人员来说能够进行多种变化、重新调整和替代而不会脱离本申请的保护范围。因此，虽然通过以上实施例对本申请进行了说明，但是本申请不仅仅限于以上实施例，在不脱离本发明构思的情况下，还可以包括更多其他等效实施例，而本申请的范围由所附的权利要求范围决定。

Claims (10)

  1. 一种图像生成模型训练方法，所述方法包括：
    获取感兴趣部位的不同模态图像，并基于所述不同模态图像组成至少一组图像对；
    针对所述至少一组图像对中的每一组图像对，将图像对中的第一模态图像输入到预设图像生成模型中，并以图像对中的第二模态图像作为学习标准，对所述预设图像生成模型进行有监督的模型训练；
    响应于所述预设图像生成模型的损失函数满足预设收敛条件，得到目标图像生成模型，完成模型训练过程。
  2. 根据权利要求1所述的方法，在对所述预设图像生成模型进行有监督的模型训练之前，所述方法还包括：
    将所述第一模态图像向所述第二模态图像配准；
    将配准后的第一模态图像作为所述预设图像生成模型的输入数据，输入到所述预设图像生成模型中。
  3. 根据权利要求2所述的方法，其中，所述将所述第一模态图像向所述第二模态图像配准，包括：
    采用预设配准算法或预训练的图像配准神经网络，将所述第一模态图像向所述第二模态图像配准。
  4. 根据权利要求1-3中任一所述的方法，其中，所述第一模态图像和所述第二模态图像分别是超声图像、磁共振图像、电子计算机断层扫描图像或正电子发射断层扫描图像中的一种，且所述第一模态图像和所述第二模态图像是不同模态的图像。
  5. 一种图像生成方法，所述方法包括：
    获取原始模态图像，并确定所述原始模态图像的目标转换模态；
    将所述原始模态图像输入至由权利要求1-4中任一所述的图像生成模型训练方法训练得到的目标图像生成模型，得到目标转换模态的图像。
  6. 根据权利要求5所述的方法，当所述目标转换模态包括多个目标模态时，所述方法还包括：
    预先训练得到所述多个目标模态间的多个图像生成模型，并将所述多个目标模态间的多个图像生成模型级联得到更新后的目标图像生成模型；
    将所述原始模态图像输入至所述更新后的目标图像生成模型，得到所述多个目标模态的图像。
  7. 一种图像生成模型训练装置，所述装置包括：
    样本获取模块，设置为获取感兴趣部位的不同模态图像，并基于所述不同模态图像组成至少一组图像对；
    样本数据输入模块，设置为针对所述至少一组图像对中的每一组图像对，将图像对中的第一模态图像输入到预设图像生成模型中，并以图像对中的第二模态图像作为学习标准，对所述预设图像生成模型进行有监督的模型训练；
    模型训练模块，设置为响应于所述预设图像生成模型的损失函数满足预设收敛条件，得到目标图像生成模型，完成模型训练过程。
  8. 一种图像生成装置，所述装置包括：
    图像数据获取模块，设置为获取原始模态图像，并确定所述原始模态图像的目标转换模态；
    图像生成模块，设置为将所述原始模态图像输入至由权利要求1-4中任一所述的图像生成模型训练方法训练得到的目标图像生成模型，得到目标转换模态的图像。
  9. 一种计算机设备，所述计算机设备包括：
    一个或多个处理器；
    存储器，设置为存储一个或多个程序；
    当所述一个或多个程序被所述一个或多个处理器执行，使得所述一个或多个处理器实现如权利要求1-4中任一所述的图像生成模型训练方法，或，如权利要求5或6所述的图像生成方法。
  10. 一种计算机可读存储介质，所述计算机可读存储介质上存储有计算机程序，所述计算机程序被处理器执行时实现如权利要求1-4中任一所述的图像生成模型训练方法，或，如权利要求5或6所述的图像生成方法。
PCT/CN2021/136942 2021-10-11 2021-12-10 图像生成模型训练及图像生成方法、装置、设备和介质 WO2023060735A1 (zh)

Applications Claiming Priority (2)

Application Number Priority Date Filing Date Title
CN202111180703.3A CN115965567A (zh) 2021-10-11 2021-10-11 图像生成模型训练及图像生成方法、装置、设备和介质
CN202111180703.3 2021-10-11

Publications (1)

Publication Number Publication Date
WO2023060735A1 true WO2023060735A1 (zh) 2023-04-20

Family

ID=85894827

Family Applications (1)

Application Number Title Priority Date Filing Date
PCT/CN2021/136942 WO2023060735A1 (zh) 2021-10-11 2021-12-10 图像生成模型训练及图像生成方法、装置、设备和介质

Country Status (2)

Country Link
CN (1) CN115965567A (zh)
WO (1) WO2023060735A1 (zh)

Citations (5)

* Cited by examiner, † Cited by third party
Publication number Priority date Publication date Assignee Title
CN109166087A (zh) * 2018-09-29 2019-01-08 上海联影医疗科技有限公司 医学图像的风格转换方法、装置、医学设备、影像系统及存储介质
CN109285200A (zh) * 2018-08-23 2019-01-29 上海连叶智能科技有限公司 一种基于人工智能的多模态医学影像的转换方法
US20200202502A1 (en) * 2018-12-19 2020-06-25 General Electric Company Methods and system for transforming medical images into different styled images with deep neural networks
CN111724450A (zh) * 2019-03-20 2020-09-29 上海科技大学 基于深度学习的医学图像重构系统、方法、终端、及介质
CN112819687A (zh) * 2021-01-21 2021-05-18 浙江大学 基于无监督神经网络的跨域图像转换方法、装置、计算机设备和存储介质

Patent Citations (5)

* Cited by examiner, † Cited by third party
Publication number Priority date Publication date Assignee Title
CN109285200A (zh) * 2018-08-23 2019-01-29 上海连叶智能科技有限公司 一种基于人工智能的多模态医学影像的转换方法
CN109166087A (zh) * 2018-09-29 2019-01-08 上海联影医疗科技有限公司 医学图像的风格转换方法、装置、医学设备、影像系统及存储介质
US20200202502A1 (en) * 2018-12-19 2020-06-25 General Electric Company Methods and system for transforming medical images into different styled images with deep neural networks
CN111724450A (zh) * 2019-03-20 2020-09-29 上海科技大学 基于深度学习的医学图像重构系统、方法、终端、及介质
CN112819687A (zh) * 2021-01-21 2021-05-18 浙江大学 基于无监督神经网络的跨域图像转换方法、装置、计算机设备和存储介质

Non-Patent Citations (1)

* Cited by examiner, † Cited by third party
Title
LEON A GATYS, ECKER ALEXANDER S, BETHGE MATTHIAS: "Image Style Transfer Using Convolutional Neural Networks", 1 January 2016 (2016-01-01), pages 1 - 10, XP055298103, Retrieved from the Internet <URL:http://www.cv-foundation.org/openaccess/content_cvpr_2016/papers/Gatys_Image_Style_Transfer_CVPR_2016_paper.pdf> [retrieved on 20160829], DOI: 10.1109/CVPR.2016.265 *

Also Published As

Publication number Publication date
CN115965567A (zh) 2023-04-14

Similar Documents

Publication Publication Date Title
CN109493328B (zh) 医学图像显示方法、查看设备以及计算机设备
CN109697741B (zh) 一种pet图像重建方法、装置、设备及介质
CN109961491B (zh) 多模态图像截断补偿方法、装置、计算机设备和介质
CN109035234B (zh) 一种结节检测方法、装置和存储介质
CN110414631B (zh) 基于医学图像的病灶检测方法、模型训练的方法及装置
CN110473186B (zh) 一种基于医学图像的检测方法、模型训练的方法及装置
CN111368849B (zh) 图像处理方法、装置、电子设备及存储介质
CN111369562B (zh) 图像处理方法、装置、电子设备及存储介质
Li et al. Application of image fusion in diagnosis and treatment of liver cancer
CN111080583B (zh) 医学图像检测方法、计算机设备和可读存储介质
CN106485691A (zh) 信息处理装置、信息处理系统和信息处理方法
CN115994902A (zh) 医学图像分析方法、电子设备及存储介质
CN114092475B (zh) 病灶长径确定方法、图像标注方法、装置及计算机设备
CN115830017B (zh) 基于图文多模态融合的肿瘤检测系统、方法、设备及介质
US11449210B2 (en) Method for providing an image base on a reconstructed image group and an apparatus using the same
US20230062672A1 (en) Ultrasonic diagnostic apparatus and method for operating same
CN113888566B (zh) 目标轮廓曲线确定方法、装置、电子设备以及存储介质
WO2023060735A1 (zh) 图像生成模型训练及图像生成方法、装置、设备和介质
Wang et al. Diagnosis of fetal total anomalous pulmonary venous connection based on the post‐left atrium space ratio using artificial intelligence
WO2019208130A1 (ja) 医療文書作成支援装置、方法およびプログラム、学習済みモデル、並びに学習装置、方法およびプログラム
WO2022227193A1 (zh) 肝脏区域分割方法、装置、电子设备及存储介质
CN114037830A (zh) 增强图像生成模型的训练方法、图像处理方法及装置
CN113538395A (zh) 图像处理方法、装置、设备、可读存储介质
CN113052930A (zh) 一种胸部dr双能量数字减影图像生成方法
Zhang et al. Enhancing the depth perception of DSA images with 2D–3D registration

Legal Events

Date Code Title Description
121 Ep: the epo has been informed by wipo that ep was designated in this application

Ref document number: 21960462

Country of ref document: EP

Kind code of ref document: A1