KR20220012406A - Model training method, image processing and registration method, apparatus, apparatus, medium - Google Patents

Abstract

An embodiment of the present invention provides a model training method, an image processing and registration method, and related apparatus, apparatus, storage medium and program product, the model may be applied to medical image processing, and the model training method includes at least one obtaining a sample image of - the sample image is provided with labeling information; performing content extraction on each of the at least one sample image by using the de-entanglement model to obtain content feature data of the at least one sample image; and training a preset network model using the content feature data and labeling information of at least one sample image to obtain a network model for processing the target image, wherein the de-entanglement model includes two models belonging to different domains. It is obtained by training using a type of training image, among which one type of training image and a sample image belong to a first domain, and another type of training image and a target image belong to a second domain.

Description

Model training method, image processing and registration method, apparatus, apparatus, medium

Cross-referencing of related applications

The present invention is filed based on a Chinese patent application with an application number of 202011193221.7 and an application date of October 30, 2020, and claims priority to the Chinese patent application, all contents of the Chinese patent application are incorporated herein by reference. are cited

The present invention relates to the field of artificial intelligence technology, and more particularly, to a model training method, an image processing and registration method, and related apparatus, apparatus, storage medium and program product.

As artificial intelligence technologies such as neural networks and deep learning develop, the method of training a network model and using the trained network model to complete image processing tasks such as image segmentation, image classification, target detection, and image registration is gradually becoming more and more popular with people. It is gaining popularity.

In the actual application process, taking medical images as an example, computed tomography (CT) images and magnetic resonance (MR) images belong to different domains, so a network model obtained by training through CT images When performing related processing on MR images using

The present invention provides a model training method, an image processing and registration method, and a related apparatus, apparatus, storage medium and program product.

A first aspect of the present invention comprises the steps of: obtaining at least one sample image, the sample image being provided with labeling information; performing content extraction on at least one sample image by using a disentanglement model, respectively, to obtain content feature data of the at least one sample image; and training a preset network model using the content feature data and labeling information of at least one sample image to obtain a network model for processing the target image, wherein the de-entanglement model includes two models belonging to different domains. It is obtained by training using a type of training image, of which one type of training image and a sample image belong to the first domain, and the other type of training image and the target image belong to the second domain. do.

Therefore, the detangling model obtained by training through one type of training image belonging to the first domain together with the sample image and another type of training image belonging to the second domain together with the target image is a sample image with labeling information. And by mapping the target image to obtain content feature data belonging to the same domain, the network model obtained by training using the content feature data and labeling information of the training image can be similarly applied to the target image, and further labeling is performed. It is not necessary, so it is possible to reduce the labeling cost of images in different domains and improve the precision of image processing.

Here, the sample image and the target image are medical images, and the labeling information is labeling information for a living organ; and/or, the sample image and the target image are three-dimensional images, and the labeling information is labeling information about the three-dimensional target.

Therefore, the sample image and the target image are set as medical images, and the labeling information is labeling information of a living organ, so it can be applied to the medical image field; Since the sample image and the target image are set as a three-dimensional image, and the labeling information is set as the labeling information for the three-dimensional target, it can be applied to the processing of a three-dimensional image.

Here, the preset network model includes any one of an image segmentation network model, an image classification network model, and a target detection network model.

Therefore, by setting the preset network model to any one of an image segmentation network model, an image classification network model, and a target detection network model, based on different types of preset network models, network models applied to different image processing tasks are trained to can be obtained, thereby completing different processing tasks for images in different domains.

Here, the training of the detangling model may include: acquiring a first type training image belonging to a first domain and a second type training image belonging to a second domain; Content and style extraction of the first type training image and the second type training image are respectively performed on the first type training image and the second type training image by using the original detangling model, and the content characteristic data and style characteristic data of the first type training image, and the second type training image obtaining content characteristic data and style characteristic data of reconstructing using the content feature data and style feature data of the first type training image and the content feature data and style feature data of the second type training image to obtain a reconstructed image; obtaining a loss value of the de-entanglement model based on the reconstructed image; and adjusting a parameter of the detangling model based on the loss value.

Accordingly, feature extraction is performed on the first type training image belonging to the first domain and the second type training image belonging to the second domain by using the original de-entanglement model, and content characteristic data and style characteristic data obtained by extraction are performed. By reconstructing based on , based on the reconstructed image obtained by reconstructing, it is possible to obtain a loss value of the de-entanglement model, and further, by adjusting the parameters of the de-entanglement model based on the loss value, the de-entanglement model is acquired by training can do.

Here, the step of reconstructing using the content feature data and style feature data of the first type training image and the content feature data and style feature data of the second type training image to obtain a reconstructed image includes: Reconstructed using content feature data and style feature data to obtain a first intra-domain reconstructed image, and reconstructed using content feature data and style feature data of a second type training image to obtain a second intra-domain reconstructed image to do; and reconstructing using the style characteristic data of the first type training image and the content characteristic data of the second type training image to obtain a first cross-domain reconstructed image, the style characteristic data of the second type training image and the first type training Reconstructing using the content feature data of the image to obtain a second cross-domain reconstructed image, and based on the reconstructed image, obtaining a loss value of the detangling model includes: the first type training image and the second obtaining a first intra-domain loss value based on the difference between the first intra-domain reconstructed images, and obtaining a second intra-domain loss value based on the difference between the second type training image and the second intra-domain reconstructed image step; Based on the difference between the content feature data of both the first type training image and the second cross-domain reconstructed image, a first content loss value is obtained, and between the style feature data of both the first type training image and the first cross-domain reconstructed image is obtained. Based on the difference, a first style loss value is obtained, and based on the difference between the content feature data of both the second type training image and the first cross-domain reconstructed image, a second content loss value is obtained, and a second type training image is obtained. obtaining a second style loss value based on a difference between the style feature data of both the image and the second cross-domain reconstructed image; Based on the difference between the first type training image and the first cross-domain reconstructed image, a first cross-domain loss value is obtained, and based on the difference between the second type training image and the second cross-domain reconstructed image, a second obtaining a cross domain loss value; and performing weighting processing on the obtained loss value to obtain a loss value of the de-entanglement model.

Therefore, based on the intra-domain loss value and the cross-domain loss value, and the loss value corresponding to the content feature data and the style feature data after image reconstruction, weight processing is performed to obtain the loss value of the detangling model, , furthermore, parameter adjustment can be performed based on the overall loss value, and images of different domains can be extracted by the de-entanglement model so that the obtained content feature data are located in the same domain.

A second aspect of the present invention provides the steps of: performing content extraction on a target image using a de-entanglement model to obtain content characteristic data of the target image; and processing the content characteristic data of the target image by using the processing network model to obtain a processing result of the target image, wherein the entanglement model is obtained by training using two types of training images belonging to different domains. It provides an image processing method in which one type of training image and a target image belong to the same domain, and the processing network model is obtained by training using the model training method of the first aspect.

Therefore, since the processing network model obtained by training through the model training method of the first aspect can be applied not only to the sample image for training the processing network model, but also to the target image, it is necessary to perform further labeling Therefore, it is possible to reduce the labeling cost of images in different domains and improve the precision of image processing.

A third aspect of the present invention includes the steps of: obtaining a first image to be registered and a second image to be registered; performing content extraction on the first image to be registered and the second image to be registered using the de-entanglement model, respectively, to obtain content feature data of the first image to be registered and content feature data of the second image to be registered; determining a registration parameter based on the content characteristic data of the first image to be registered and the content characteristic data of the second image to be registered; and registering the first image to be registered and the second image to be registered using the registration parameter, wherein the detangling model is obtained by training using two types of training images belonging to different domains, and the One type of training image and the first image to be matched belong to a first domain, and the other type of training image and the second image to be matched belong to a second domain.

Accordingly, the detangling model obtained by training through one type of training image belonging to the first domain together with the first image to be registered and another type of training image belonging to the second domain together with the second image to be registered is extracted. The content feature data of the first image to be matched and the content feature data of the second image to be matched belong to the same domain, so that the matching parameters of both can be obtained and registered directly based on the content feature data of both. Therefore, it is possible to implement unsupervised registration between images of different domains, further reducing the cost of image registration of different domains, and improving the accuracy.

wherein the first image to be registered and the second image to be registered are medical images; and/or, the first image to register and the second image to register are three-dimensional images; and/or, the matching parameter includes at least one of a stiffness variation parameter and a strain variation parameter.

Therefore, by setting the first image to be registered and the second image to be registered as medical images, image registration can be applied to registration of medical images; By setting the first image to be registered and the second image to be registered as three-dimensional images, image registration can be applied to registration of three-dimensional images; The registration parameter may include at least one of a stiffness change parameter and a deformation change parameter, so that the first image to be registered and the second image to be registered are changed correspondingly based on the actual situation of both, and then registration can be implemented; It is advantageous for improving the accuracy of registration.

Here, the determining of the registration parameter based on the content characteristic data of the first image to be registered and the content characteristic data of the second image to be registered includes: the content characteristic data of the first image to be registered and the content of the second image to be registered determining, based on the feature data, a registration parameter corresponding to the first image to be registered; The step of registering the first to-be-registered image and the second to-be-registered image using the registration parameter includes performing change processing on the first image to be registered using the registration parameter to change a change corresponding to the first image to be registered. acquiring an image; and superimposing the pixel points of the change image to the corresponding pixel points of the second to-be-registered image.

Therefore, by performing change processing on the first image to be registered using the registration parameter corresponding to the first image to be registered, a change image corresponding to the first image to be registered can be obtained, and the pixel point of the change image can be obtained. Unsupervised registration between images of different domains can be implemented by overlapping the corresponding pixel points of the second image to be registered.

Here, the training of the detangling model may include: acquiring a first type training image belonging to a first domain and a second type training image belonging to a second domain; Content and style extraction of the first type training image and the second type training image are respectively performed on the first type training image and the second type training image by using the original detangling model, and the content characteristic data and style characteristic data of the first type training image, and the second type training image obtaining content characteristic data and style characteristic data of reconstructing using the content feature data and style feature data of the first type training image and the content feature data and style feature data of the second type training image to obtain a reconstructed image; obtaining a loss value of the de-entanglement model based on the reconstructed image; and adjusting a parameter of the detangling model based on the loss value.

Accordingly, feature extraction is performed on the first type training image belonging to the first domain and the second type training image belonging to the second domain by using the original de-entanglement model, and content characteristic data and style characteristic data obtained by extraction are performed. By reconstructing based on , based on the reconstructed image obtained by reconstructing, it is possible to obtain a loss value of the de-entanglement model, and further, by adjusting the parameters of the de-entanglement model based on the loss value, the de-entanglement model is acquired by training can do.

A fourth aspect of the present invention provides a model training apparatus comprising an image acquisition module, a content extraction module and a model training module, wherein the image acquisition module is configured to acquire at least one sample image, wherein the sample image includes labeling information provided; the content extraction module is configured to perform content extraction on each of the at least one sample image by using the de-entanglement model, to obtain content characteristic data of the at least one sample image; The model training module is configured to train a preset network model by using the content feature data and labeling information of at least one sample image to obtain a network model for processing the target image, wherein the de-entanglement model is configured to perform two tasks belonging to different domains. It is obtained by training using dog-type training images, among which one type of training image and sample image belong to the first domain, and the other type of training image and target image belong to the second domain.

A fifth aspect of the present invention provides an image processing apparatus comprising a content extraction module and a network processing module, wherein the content extraction module performs content extraction on a target image using a de-entanglement model, so that the content features of the target image configured to acquire data; The network processing module is configured to process the content characteristic data of the target image by using the processing network model to obtain a processing result of the target image, and the detangling model is trained using two types of training images belonging to different domains. is obtained, one type of training image and the target image belong to the same domain, and the processing network model is obtained by training using the model training apparatus of the fourth aspect.

A sixth aspect of the present invention provides an image registration device comprising an image acquisition module, a content extraction module, a parameter determination module and an image registration module, wherein the image acquisition module is configured to acquire a first image to be registered and a second image to be registered consists; The content extraction module performs content extraction on the first to-be-matched image and the second to-be-matched image using the entanglement de-entanglement model, respectively, to obtain the content characteristic data of the first image to be registered and the content characteristic data of the second image to be registered configured to obtain; The parameter determining module is configured to determine a registration parameter according to the content characteristic data of the first image to be registered and the content characteristic data of the second image to be registered; The image registration module registers the first image to be registered and the second image to be registered by using the registration parameter, and the detangling model is obtained by training using two types of training images belonging to different domains, among which One type of training image and the first image to be registered belong to the first domain, and the other type of training image and the second image to be registered belong to the second domain.

A seventh aspect of the present invention provides an electronic device comprising a memory and a processor coupled to each other, wherein the processor executes program instructions stored in the memory to implement the model training method of the first aspect, or The image processing method of the second aspect is implemented, or the image registration method of the third aspect is implemented.

An eighth aspect of the present invention provides a computer readable storage medium having program instructions stored thereon, wherein when the program instructions are executed by a processor, the model training method of the first aspect is implemented, or the image processing of the second aspect A method is implemented, or an image registration method of the third aspect is implemented.

A ninth aspect of the present invention provides a computer program product, the computer program product comprising one or more instructions, wherein the one or more instructions are executed by a processor, the method comprising: implemented, or the image processing method of the second aspect is implemented, or the image registration method of the third aspect is suitable to be implemented.

In the above means, the detangling model obtained by training through one type of training image belonging to the first domain together with the sample image, and another type of training image belonging to the second domain together with the target image is provided with labeling information. By mapping the sample image and the target image to obtain content feature data belonging to the same domain, a network model obtained by training using the content feature data and labeling information of the training image can be similarly applied to the target image, and further labeling It is not necessary to carry out, so it is possible to reduce the labeling cost of images in different domains and improve the precision of image processing.

1 is a flow schematic diagram of an embodiment of a model training method provided in the present invention.
2 is a flow schematic diagram of an embodiment of a method for training a de-entanglement model provided in the present invention.
3 is a schematic diagram of another embodiment of the method for training a de-entanglement model provided in the present invention.
4 is a flow diagram of an embodiment of the image processing method provided in the present invention.
5 is a flow diagram of an embodiment of the image registration method provided in the present invention.
6 is a process schematic diagram of an embodiment of image registration provided in the present invention.
7 is a schematic diagram of a framework of an embodiment of the model training apparatus provided in the present invention.
8 is a schematic diagram of a framework of an embodiment of the image processing apparatus provided in the present invention.
9 is a schematic diagram of a framework of an embodiment of the image matching apparatus provided in the present invention.
10 is a schematic diagram of a framework of an embodiment of an electronic device provided in the present invention.
11 is a schematic diagram of a framework of an embodiment of a storage device provided in the present invention.

Hereinafter, the means of the embodiment of the present invention will be described in detail in conjunction with the drawings in the specification.

In the following description, details such as specific system structures, interfaces, techniques, and the like are set forth for purposes of explanation and not limitation, so that embodiments of the present invention may be more clearly understood.

As used herein, the terms “system” and “network” are often used interchangeably herein. As used herein, the term “and/or” is an association relationship that describes an associated object, and indicates that three relationships may exist. For example, A and/or B means that only A exists, and A and B exist simultaneously. , we can represent three cases in which only B exists. In addition, the symbol “/” in the present specification generally indicates that an object related before and after is in one “or” relationship. In addition, "a plurality" in this specification refers to two or two or more.

Referring to FIG. 1, FIG. 1 is a flow schematic diagram of an embodiment of a model training method provided in the present invention. It may include the following steps.

Step S11: Acquire at least one sample image.

In this embodiment, the sample image is provided with labeling information. For example, in the case of a training image segmentation network model, the labeling information may be contour information of a target object in the training image; Alternatively, in the case of a training image classification network model, the labeling information may be category information corresponding to a target object in the training image; Alternatively, in the case of a training target detection network model, the labeling information may be location information of a target object in a training image, and may be set according to an application scenario, but is not limited thereto.

In one implementation scenario, in order to improve the accuracy of the network model obtained by training, the number of sample images may be plural, for example, 500, 700, 900, etc., without limitation herein. .

In one implementation scenario, the sample image may be a medical image such as a CT image, an MR image, etc., so that the network model obtained by training can be applied to the field of medical imaging. , for example, may be labeling information for organs such as pancreas and kidney, and may be labeling information for organ tissue, for example, labeling information for ligament and cartilage tissue, or labeling information for organ lesion tissue; For example, it may be labeling information for a hematoma region and an ulcer region, but is not limited thereto.

In one implementation scenario, the sample image may be a three-dimensional image, such as a three-dimensional nuclear magnetic resonance image, or a three-dimensional image, so that the network model obtained by training can perform corresponding processing on the three-dimensional image. A CT image and a type B ultrasound image may be a 3D image obtained through 3D reconstruction, but the present disclosure is not limited thereto.

Step S12: Perform content extraction on each of the at least one sample image by using the de-entanglement model to obtain content feature data of the at least one sample image.

In this embodiment, the de-entanglement model is obtained by training using two types of training images belonging to different domains, one type of training image and a sample image belonging to the first domain, and the other type of training image belongs to the first domain. The image and the target image belong to the second domain. Image data of images of different domains follow different data distributions. For example, CT images and MR images collected by different kinds of medical image detection instruments belong to different domains; or the daytime image and the nighttime image belong to different domains; or the image taken by the camera and the hand-drawn image belong to different domains; Alternatively, the landscape photographed in summer and the landscape photographed in winter belong to different domains, and will not be described here individually.

및 이미지

에 대해 콘텐츠 특징 및 스타일 특징을 추출하여 획득한 후, 재구성하여 이미지

및 이미지

를 획득한다. 구체적으로, 제1 도메인에 속하는 제1 타입 트레이닝 이미지를 이미지

로, 제2 도메인에 속하는 제2 타입 트레이닝 이미지를 이미지

로 획득하고, 원래의 얽힘 해제 모델을 이용하여 제1 타입 트레이닝 이미지

및 제2 타입 트레이닝 이미지

에 대해 각각 콘텐츠 및 스타일 수출을 수행하여, 제1 타입 트레이닝 이미지

의 콘텐츠 특징 데이터

및 스타일 특징 데이터

, 및 제2 타입 트레이닝 이미지

의 콘텐츠 특징 데이터

및 스타일 특징 데이터

를 획득한다. 다음, 제1 타입 트레이닝 이미지

의 콘텐츠 특징 데이터

및 스타일 특징 데이터

, 및 제2 타입 트레이닝 이미지

의 콘텐츠 특징 데이터

및 스타일 특징 데이터

를 이용하여 재구성하여, 재구성 이미지를 획득한다. 재구성하는 과정은 도 2 및 도 3을 결합하여 참조하며, 도 3은 본 발명에서 제공되는 얽힘 해제 모델 트레이닝 방법의 다른 실시예의 모식도이다. 도 2에 도시된 바와 같이, 제1 타입 트레이닝 이미지

의 콘텐츠 특징 데이터

및 스타일 특징 데이터

를 이용하여 재구성하여, 제1 인트라 도메인 재구성 이미지

를 획득하고, 제2 타입 트레이닝 이미지

의 콘텐츠 특징 데이터

및 스타일 특징 데이터

를 이용하여 재구성하여, 제2 인트라 도메인 재구성 이미지

를 획득하며, 도 3에 도시된 바와 같이, 제1 타입 트레이닝 이미지

의 스타일 특징 데이터

및 제2 타입 트레이닝 이미지

의 콘텐츠 특징 데이터

를 이용하여 재구성하여, 제1 크로스 도메인 재구성 이미지

를 획득하고, 제2 타입 트레이닝 이미지

의 스타일 특징 데이터

및 제1 타입 트레이닝 이미지

의 콘텐츠 특징 데이터

를 이용하여 재구성하여, 제2 크로스 도메인 재구성 이미지

를 획득한다. 상기 인트라 도메인 재구성 및 크로스 도메인 재구성을 통해, 제1 인트라 도메인 재구성 이미지

, 제2 인트라 도메인 재구성 이미지

, 제1 크로스 도메인 재구성 이미지

및 제2 크로스 도메인 재구성 이미지

를 각각 획득할 수 있다. 이를 토대로, 재구성 이미지에 기반하여, 얽힘 해제 모델의 손실값을 획득함으로써, 손실값에 기반하여 얽힘 해제 모델의 파라미터를 조정할 수 있다.Referring to FIG. 2 in combination, FIG. 2 is a schematic diagram of an embodiment of a method for training a de-entanglement model provided in the present invention. As shown in Figure 2, images of different domains using the same distribution reconstruction process

and images

After extracting and acquiring content features and style features for

and images

to acquire Specifically, the image of the first type training image belonging to the first domain

As an image, the second type training image belonging to the second domain

, and using the original unentanglement model, the first type training image

and a second type training image

By performing content and style export for each, the first type training image

content feature data of

and style feature data

, and the second type training image

content feature data of

and style feature data

to acquire Next, the first type training image

content feature data of

and style feature data

, and the second type training image

content feature data of

and style feature data

Reconstructed using , to obtain a reconstructed image. The reconstructing process is referred to in combination with FIGS. 2 and 3, and FIG. 3 is a schematic diagram of another embodiment of a method for training a detangling model provided in the present invention. As shown in Fig. 2, the first type training image

content feature data of

and style feature data

Reconstructed using the first intra-domain reconstructed image

to obtain a second type training image

content feature data of

and style feature data

Reconstructed using , the second intra domain reconstructed image

, and as shown in FIG. 3 , the first type training image

of style feature data

and a second type training image

content feature data of

Reconstructed using the first cross-domain reconstructed image

to obtain a second type training image

of style feature data

and the first type training image

content feature data of

Reconstructed using a second cross-domain reconstructed image

to acquire Through the intra-domain reconstruction and cross-domain reconstruction, the first intra-domain reconstruction image

, the second intra-domain reconstruction image

, first cross-domain reconstructed image

and a second cross-domain reconstruction image.

can be obtained individually. Based on this, by obtaining a loss value of the de-entanglement model based on the reconstructed image, it is possible to adjust the parameters of the de-entanglement model based on the loss value.

와 제1 인트라 도메인 재구성 이미지

사이의 차이에 기반하여, 제1 인트라 도메인 손실값을 획득할 수 있고, 제2 타입 트레이닝 이미지

와 제2 인트라 도메인 재구성 이미지

사이의 차이에 기반하여, 제2 인트라 도메인 손실값을 획득할 수 있으며, 제1 타입 트레이닝 이미지

의 콘텐츠 특징 데이터

및 제2 크로스 도메인 재구성 이미지

의 콘텐츠 특징 데이터

간의 차이에 기반하여, 제1 콘텐츠 손실값을 획득할 수 있고, 여기서, 콘텐츠 특징 데이터

는 제2 크로스 도메인 재구성 이미지

를 디코딩하여 획득된 것이며, 제1 타입 트레이닝 이미지

의 스타일 특징 데이터

및 제1 크로스 도메인 재구성 이미지

의 스타일 특징 데이터

간의 차이에 기반하여, 제1 스타일 손실값을 획득할 수 있고, 여기서, 스타일 특징 데이터

는 제1 크로스 도메인 재구성 이미지

를 디코딩하여 획득된 것이며, 제2 타입 트레이닝 이미지

의 콘텐츠 특징 데이터

및 제1 크로스 도메인 재구성 이미지

의 콘텐츠 특징 데이터

간의 차이에 기반하여, 제2 콘텐츠 손실값을 획득할 수 있고, 여기서, 콘텐츠 특징 데이터

는 제1 크로스 도메인 재구성 이미지

를 디코딩하여 획득된 것이며, 제2 타입 트레이닝 이미지

의 스타일 특징 데이터

및 제2 크로스 도메인 재구성 이미지

의 스타일 특징 데이터

간의 차이에 기반하여, 제2 스타일 손실값을 획득할 수 있고, 여기서, 스타일 특징 데이터

는 제2 크로스 도메인 재구성 이미지

를 디코딩하여 획득된 것이며, 제1 타입 트레이닝 이미지

와 제1 크로스 도메인 재구성 이미지

사이의 차이에 기반하여, 제1 크로스 도메인 손실값을 획득할 수 있고, 제2 타입 트레이닝 이미지

와 제2 크로스 도메인 재구성 이미지

사이의 차이에 기반하여, 제2 크로스 도메인 손실값을 획득할 수 있으며, 상기 획득된 손실값에 대해 가중 처리를 수행하여, 얽힘 해제 모델의 손실값을 획득한다. 구체적으로, 제1 인트라 도메인 손실값과 제2 인트라 도메인 손실값의 가중치는 동일할 수 있고(예를 들어, 모두

임), 제1 스타일 손실값과 제2 스타일 손실값의 가중치도 동일할 수 있으며(예를 들어, 모두

임), 제1 콘텐츠 손실값과 제2 콘텐츠 손실값의 가중치도 동일할 수 있으며(예를 들어, 모두

임), 제1 크로스 도메인 손실값과 제2 크로스 도메인 손실값의 가중치도 동일할 수 있으며(예를 들어, 모두 1임), 일 구현 시나리오에서, 상기

,

,

은 실제 상황에 따라 설정될 수 있고, 여기에서 이에 대해 제한하지 않는다.Specifically, referring to FIGS. 3 and 3 in conjunction with each other, the first type training image

and the first intra-domain reconstruction image

Based on the difference between the first intra-domain loss value may be obtained, the second type training image

and the second intra-domain reconstruction image

Based on the difference between them, a second intra-domain loss value may be obtained, and a first type training image

content feature data of

and a second cross-domain reconstruction image.

content feature data of

A first content loss value may be obtained based on the difference between the content characteristic data

is the second cross-domain reconstructed image

It is obtained by decoding the first type training image

of style feature data

and a first cross-domain reconstructed image

of style feature data

Based on the difference between, a first style loss value may be obtained, wherein the style feature data

is the first cross-domain reconstructed image

It is obtained by decoding the second type training image

content feature data of

and a first cross-domain reconstructed image

content feature data of

A second content loss value may be obtained based on the difference between the content characteristic data

is the first cross-domain reconstruction image

It is obtained by decoding the second type training image

of style feature data

and a second cross-domain reconstruction image.

of style feature data

A second style loss value may be obtained based on the difference between the style feature data

is the second cross-domain reconstructed image

It is obtained by decoding the first type training image

and the first cross-domain reconstruction image

Based on the difference between the first cross-domain loss value can be obtained, the second type training image

and the second cross-domain reconstruction image

Based on the difference between them, a second cross-domain loss value may be obtained, and a weighting process is performed on the obtained loss value to obtain a loss value of the detangling model. Specifically, the weights of the first intra-domain loss value and the second intra-domain loss value may be the same (eg, both

), the weights of the first style loss value and the second style loss value may also be the same (for example, both

), the weight of the first content loss value and the second content loss value may also be the same (eg, both

), the weights of the first cross-domain loss value and the second cross-domain loss value may also be the same (eg, both are 1), and in one implementation scenario, the

,

,

may be set according to an actual situation, and is not limited thereto.

및 이미지

에 대해 콘텐츠 특징 및 스타일 특징을 추출하여 획득한 후, 재구성하여 이미지

및 이미지

를 획득한 것을 도시한다. 도 3에 도시된 것은 교차 모달리티 생성 과정이며,

이미지 및

이미지에 대해 추출하여 획득된 콘텐츠 특징이 교환된 후, 스타일 특징은 유지되고, 재구성하여

및

이 획득되며, 이러한 방법으로 얽힘 해제 모델을 지속적으로 트레이닝시킬 수 있다. 상기 얽힘 해제 모델을 이용하여, 상이한 도메인의 적어도 두 개의 이미지에서 대응되는 적어도 두 개 그룹의 콘텐츠 특징 데이터를 추출하여 획득함으로써, 두 개 그룹의 콘텐츠 특징 데이터가 동일한 분포를 갖도록 한 후, 적어도 두 개 그룹의 콘텐츠 특징 데이터 또는 트레이닝 이미지 분할 네트워크 모델, 이미지 분류 네트워크 모델, 타깃 검출 네트워크 모델 등 모델에 기반하여, 상기 기설정된 네트워크 모델를 트레이닝시켜, 이미지에 대해 분할을 수행하기 위한 네트워크 모델, 이미지 분류를 위한 네트워크 모델, 및 타깃 검출을 위한 네트워크 모델을 각각 획득할 수 있어, 트레이닝하여 획득된 네트워크 모델이 제1 도메인 및 제2 도메인에 속하는 타깃 이미지에 동시에 적용될 수 있도록 한다.Taken together, Figure 2 shows images of different domains using the same distribution reconstruction process.

and images

After extracting and acquiring content features and style features for

and images

is shown to have been obtained. 3 is a cross-modality generation process,

image and

After the content features extracted and obtained for the image are exchanged, the style features are maintained and reconstructed

and

is obtained, and in this way the de-entanglement model can be continuously trained. Using the de-entanglement model, by extracting and obtaining corresponding at least two groups of content feature data from at least two images of different domains, so that the content feature data of the two groups have the same distribution, at least two Based on a group's content feature data or a training image segmentation network model, an image classification network model, a target detection network model, etc., by training the preset network model to perform segmentation on an image, a network model for image classification It is possible to obtain a network model and a network model for target detection, respectively, so that the network model obtained by training can be simultaneously applied to target images belonging to the first domain and the second domain.

의 콘텐츠 특징 데이터

및 제2 타입 트레이닝 이미지

의 콘텐츠 특징 데이터

가 최종적으로 동일한 데이터 분포를 따르도록 할 수 있으며, 이때 얽힘 해제 모델 트레이닝이 완료되었다고 간주할 수 있다.In the actual training process, the parameters of the detangling model can be adjusted based on the obtained loss values, and the steps of feature extraction, image reconstruction, and loss calculation can be re-performed by repeating the above process, so that the parameters of the detangling model can be continuously maintained. The first type training image obtained through the de-entanglement model through multiple cycles and parameter adjustment.

content feature data of

and a second type training image

content feature data of

Finally, it can be made to follow the same data distribution, and at this time, it can be considered that the detangling model training is complete.

In one implementation scenario, the de-entanglement model may include a content encoder and a style encoder, each configured to encode an image to obtain content feature data and style feature data, where the content encoder may contain multiple strided convolutional layers configured to downsample the input image and multiple residual blocks for further processing, all convolutional layers being all instance normalized processing ( Instance Normalization). The style encoder may include multiple strided convolutional layers configured to downsample the input image, one global average pooling layer, and one fully connected layer, instance Since the normalization process removes the average value and variance of the original feature, and a lot of style feature information is included therein, the style encoder does not use the instance normalization process. In addition, the de-entanglement model further includes a decoder configured to reconstruct an image based on the content feature data and the style feature data, specifically, the decoder processes the content feature data through a group of residual blocks, Finally, the image is reconstructed using multiple upsampling, where a self-adaptive instance normalization may be further introduced into the residual block, and the self-adaptive instance normalization may be dynamically generated from the style feature data using a multilayer perceptron. .

In the present embodiment, the content characteristic data may include data describing image content, for example, data describing outlines of people, buildings, and landscapes among images. The style feature data may include data describing image color and style, for example, data describing color series (eg, warm color series, cool color series, etc.) such as people, buildings, and landscapes in the image. It may include, or may include data describing a style (eg, a realistic style, an impressive style) such as a person, a building, a landscape, etc. in the image, and will not be described in detail here by example.

Step S13: Train a preset network model using the content feature data and labeling information of at least one sample image to obtain a network model for processing the target image.

In one implementation scenario, the sample image and the target image may be medical images such as CT images, MR images, and the like, so that the network model obtained by training can be applied to the field of medical imaging. In addition, the labeling information may be labeling information for a living organ.

In one implementation scenario, the sample image and the target image are three-dimensional images, and the labeling information is labeling information for the three-dimensional target, so that the network model obtained by training can perform corresponding processing on the three-dimensional image . The three-dimensional image may be a three-dimensional nuclear magnetic resonance image, or the three-dimensional image may be a three-dimensional image obtained by performing a three-dimensional reconstruction of a CT image or a type B ultrasound image, but the present invention is not limited thereto.

In one implementation scenario, in order to adapt to different application scenarios, the preset network module may be set to any one of an image segmentation network model, an image classification network model, and a target detection network model according to an actual situation. For example, in the case of image segmentation, the image segmentation network model may include, but is not limited to, a Fully Convolutional Network (FCN), Segnet, Enet, and the like; Alternatively, in the case of image classification, the image classification network model may include, but is not limited to, a VGG network, an Inception network, a Resnet network, and the like; Alternatively, in the case of target detection, the target detection network model may include, but is not limited to, a You Only Look Once (YOLO) network, a Single Shot MultiBox Detector (SSD) network, and the like, but is not limited thereto. By training the preset network model, it is possible to obtain a network model for performing segmentation on an image, a network model for image classification, and a network model for target detection, respectively, so that the network model obtained by training is the first It allows it to be simultaneously applied to the target image belonging to the domain and the second domain.

In one implementation scenario, the sample image is a CT image of the skull and brain, the target image is an MR image of the skull and brain, a first type training image belonging to a first domain together with the sample image, and a second type with the target image By training and acquiring the de-entanglement network through the second type training image belonging to By training a preset network model using data and labeling information, it is possible to obtain a network model for processing MR images of the skull and brain. Other application scenarios can be inferred in this way, and examples are not further described here.

In the above means, the detangling model obtained by training through one type of training image belonging to the first domain together with the sample image, and another type of training image belonging to the second domain together with the target image is provided with labeling information. By mapping the sample image and the target image to obtain content feature data belonging to the same domain, a network model obtained by training using the content feature data and labeling information of the training image can be similarly applied to the target image, and further labeling It is not necessary to carry out, so it is possible to reduce the labeling cost of images in different domains and improve the precision of image processing.

Referring to FIG. 4, FIG. 4 is a flowchart of an embodiment of the image processing method provided in the present invention. It may include the following steps.

Step S41: Content extraction is performed on the target image by using the de-entanglement model to obtain content characteristic data of the target image.

In the present embodiment, the de-entanglement model is obtained by training using two types of training images belonging to different domains, one type of training image and the target image belonging to the same field. The training process of the de-entanglement model may refer to the relevant steps of the foregoing embodiments, which are not further described herein.

In one implementation scenario, the target image may be an MR image of the skull and brain, and content extraction is performed on the MR image of the skull and brain using a de-entanglement model to obtain content feature data of the MR image of the skull and brain can do. Other application scenarios can be inferred in this way, and examples are not further described here.

Step S42: Process the content characteristic data of the target image by using the processing network model to obtain a processing result of the target image.

In this embodiment, the processing network model is obtained by using the steps of any one of the above model training method embodiments, and may refer to the relevant steps of the foregoing embodiments, which are not further described herein.

In one implementation scenario, still taking as an example that the target image is an MR image of the skull and brain, the processing network model may be an image segmentation network model, and the image segmentation network model is the content feature data of the CT image of the skull and brain and its labeling It is obtained by training using information (eg, lacunar infarction contour information), the content characteristic data of the CT image of the skull and brain are obtained by extracting using the de-entanglement model, and the de-entanglement model is also obtained by using the de-entanglement model. It is obtained by training using a training image belonging to the same domain as the MR image and a training image belonging to the same domain as the CT image of the skull and brain. Using the image segmentation network model obtained by processing the content feature data of the MR images of the skull and brain, it is obtained by segmenting the hiatus cerebral infarction in the MR images of the skull and brain.

In another implementation scenario, still taking as an example that the target image is an MR image of the skull and brain, the processing network model may be an image classification network model, and the image classification network model is the content characteristic data of the CT image of the skull and brain and its classification It is obtained by training using the labeling information corresponding to the lacunar infarction, hemorrhage, cerebral white matter thinning, brain atrophy, etc. The entanglement model is also obtained by training using a training image belonging to the same domain as the MR image of the skull and brain, and a training image belonging to the same domain as the CT image of the skull and brain, and the training process is Reference may be made to the relevant steps of the above-described embodiments, whereby the image classification network model obtained by training is used to process the content characteristic data of the MR images of the skull and brain, to which categories (eg, the MR images of the skull and brain) belong. : Determining any one of lacunar cerebral infarction, hemorrhage, cerebral white matter thinning, and brain atrophy).

In another implementation scenario, still taking as an example that the target image is an MR image of the skull and brain, the processing network model may be a target detection network model, and the target detection network model includes content characteristic data of the CT image of the skull and brain and its It is obtained by training using the labeling information (eg, location information) of the target area of the target object (eg, lacunar infarction, hemorrhage, etc.) included in the It is obtained by extracting using , and the entanglement model is also obtained by training using a training image belonging to the same domain as the MR image of the skull and brain, and a training image belonging to the same domain as the CT image of the skull and brain. , the training process may refer to the relevant steps of the above-described embodiments, whereby the target detection network model obtained by training is used to process the content characteristic data of the MR images of the skull and brain, so that among the MR images of the skull and brain, The target area of the target object (lacquer cerebral infarction, hemorrhage, etc.) is acquired.

Other application scenarios can be inferred in this way, and examples are not further described here.

In the means, the processing network model obtained by training through the model training method of the first aspect can be applied not only to the sample image for training the processing network model, but also to the target image, so that the labeling is further performed. It is not necessary to do this, so it is possible to reduce the labeling cost of images in different domains and improve the precision of image processing.

Referring to FIG. 5, FIG. 5 is a flowchart of an embodiment of the image registration method provided in the present invention. It may include the following steps.

Step S51: Acquire a first image to be registered and a second image to be registered.

In this embodiment, the first image to be registered and the second image to be registered may belong to different domains. For example, the first image to be registered may be a CT image, the second image to be registered may be an MR image, or the first image to be registered may be a depth image obtained by scanning with a laser radar, 2 The image to be registered may be a visible light image obtained by photographing with a camera, but the present invention is not limited thereto.

Step S52: Perform content extraction on the first image to be registered and the second image to be registered by using the de-entanglement model, respectively, to obtain content feature data of the first image to be registered and content feature data of the second image to be registered do.

In this embodiment, the de-entanglement model is obtained by training using two types of training images belonging to different domains, and one type of training image and the first image to be matched belongs to the first domain, and the other type of training image belongs to the first domain. The training image of the type and the second image to be matched belong to the second domain. In the training process of the de-entanglement model, a first type training image belonging to the first domain and a second type training image belonging to the second domain are acquired, and the first type training image and the second type training image are obtained using the original de-entanglement model. The first type training image by performing content and style extraction on the training image, respectively, to obtain content feature data and style feature data of the first type training image, and content feature data and style feature data of the second type training image by reconstructing using the content feature data and style feature data of the second type training image, and the content feature data and style feature data of the second type training image to obtain a reconstructed image, and based on the reconstructed image, obtain a loss value of the detangling model, , and further adjusting the parameters of the de-entanglement model based on the loss value. Specifically, reference may be made to the relevant steps of the above-described embodiments, which are not further described herein.

Through the detangling model obtained by training, by performing content extraction on the first image to be registered and the second image to be registered, respectively, the content feature data of the first image to be registered and the second image to be registered Since the content characteristic data can be made to follow the same data distribution, it is convenient for subsequent processing.

Referring to FIG. 6 in combination, FIG. 6 is a schematic diagram of a process of an embodiment of image registration provided in the present invention, and as shown in FIG. 6 , the first image to be matched 601 is a CT image of the skull and brain. , the second image to be registered 602 is an MR image of the skull and brain, a training image belonging to the first domain together with a CT image of the skull and brain, and a training image belonging to the second domain together with the MR image of the skull and brain can be obtained by training the de-entanglement model using to obtain the content feature data 605 and style feature data 606 of the first image to be matched 601, and the content feature data 603 and style feature data 604 of the second image to be matched 602 can be obtained

Step S53: Determine a registration parameter according to the content characteristic data of the first image to be registered and the content characteristic data of the second image to be registered.

In one implementation scenario, the registration parameter may include at least one of a stiffness change parameter and a deformation change parameter so as to be able to determine the registration parameter based on the actual situation of the first image to register and the second image to register. When the registration parameter is a stiffness change parameter, the image only undergoes rotational change and/or translational change, but its shape is not changed, and when the registration parameter is a deformation change parameter, the shape of the image is changed.

Continuing to refer to FIG. 6 in combination, based on the content characteristic data 605 of the first image to be registered 601 and the content characteristic data 603 of the second image to be registered 602, the first image to be registered (601) and a registration parameter 607 of the second image to be registered 602 are obtained.

In one implementation scenario, during the image registration process, one of the first image to be registered and the second image to be registered may be set as a moving image, and the other may be set as a fixed image. For example, if the first image to be registered is set as a motion image, a registration parameter corresponding to the first image to be registered is determined based on the content characteristic data of the first image to be registered and the content characteristic data of the second image to be registered. can decide

Step S54: Using the registration parameter, the first image to be registered and the second image to be registered are registered.

In one implementation scenario, during the image registration process, one of the first image to be registered and the second image to be registered may be set as a moving image, and the other may be set as a fixed image. For example, a first image to be registered is set as a motion image, a registration parameter of the first image to be registered is determined based on the first image to be registered and a second image to be registered, and the first image to be registered is determined using the registration parameter. Perform change processing (eg, stiffness change, deformation change) on the image to be registered to obtain a change image corresponding to the first image to be registered, and set pixel points of the change image to corresponding pixels of the second image to be registered By superimposing the points, image registration can be implemented.

From Fig. 6, the content characteristic data 603 and the content characteristic data 605 respectively corresponding to the second image to be registered 602 and the first image to be registered 601 are used with the same modality (same distribution), so that unsupervised It can be seen that image registration is performed directly. Continuing to refer to FIG. 6 in combination, the content of the second image to be registered 602 is by using the first image to be registered 601 as a moving image and the second image to be registered 602 as a fixed image. Using the registration parameter 607 by determining a registration parameter 607 of the second image to register 602 based on the feature data 603 and the content characteristic data 605 of the first image to register 601 Thus, the first image to be registered 601 is subjected to change processing such as the stiffness change processing to obtain a change image 608, and pixel points of the obtained change image 608 are then matched to the second image to register 602 ) to the corresponding pixel point to complete image registration.

In one implementation scenario, in order to further improve the accuracy of image registration, registration parameters of a first image to register and a second image to be registered are determined, and change processing is performed on the first image to be registered by using the registration parameters. , after acquiring the corresponding change image, aggregate the difference between the change image and the second image to be registered, and fine-tune the registration parameter based on the difference obtained by counting, and then use the fine-tuned registration parameter to , continuously performing change processing on the first to-be-registered image to obtain a corresponding change image, and aggregate the difference between the change image and the second to-be-registered image, and based on the difference obtained by counting, the fine fine-tuning the adjusted registration parameter again, and performing change processing on the first image to be registered using the fine-tuned registration parameter until the difference between the obtained change image and the second image to be registered is within an acceptable range The above steps may be repeated.

In the above means, the detangling model obtained by training through one type of training image belonging to the first domain together with the first image to be registered, and another type of training image belonging to the second domain together with the second image to be registered is , so that the extracted content feature data of the first image to be matched and the content feature data of the second image to be matched belong to the same domain, based on the content feature data of both, directly obtain and register both matching parameters Therefore, it is possible to implement unsupervised registration between images of different domains, further reducing the cost of image registration of different domains, and improving the accuracy.

An embodiment of the present invention provides an unsupervised domain self-adaptation method based on image de-suppression, thereby greatly reducing data labeling cost, and also solving unsupervised matching between different modalities. However, in the related art, it is necessary to consume a lot of labeling labor cost to perform labeling even in one data distribution, or to generate another data distribution by using a generative adversarial network method, so that the overall precision of the entanglement provided in the embodiment of the present invention is Not as effective as the unlock method.

An embodiment of the present invention can change the latent variable to a content characteristic in the same encoding space and a style characteristic in a different encoding space, so that images of different data distributions follow the same distribution in the content characteristic. However, the related techniques generally use shared latent variables, so the de-entanglement effect is not good.

Embodiments of the present invention can be used for deployment or training of models of different data distributions, reduce labeling costs, accelerate product precision, and apply to products such as intelligent medical-assisted diagnosis, cloud platform-assisted smart diagnosis, remote diagnosis system, etc. do.

Referring to FIG. 7, FIG. 7 is a schematic diagram of a framework of an embodiment of the model training apparatus 70 provided in the present invention. The model training apparatus 70 includes an image acquisition module 71, a content extraction module 72 and a model training module 73, wherein the image acquisition module 71 is configured to acquire at least one sample image, The image is provided with labeling information; the content extraction module 72 is configured to perform content extraction on each of the at least one sample image by using the de-entanglement model, to obtain content characteristic data of the at least one sample image; The model training module 73 is configured to train a preset network model using the content characteristic data and labeling information of the at least one sample image to obtain a network model configured to process the target image, wherein the de-entanglement model is different It is obtained by training using two types of training images belonging to the domain, of which one type of training image and sample image belong to the first domain, and the other type of training image and the target image belong to the second domain. .

In the above means, the detangling model obtained by training through one type of training image belonging to the first domain together with the sample image, and another type of training image belonging to the second domain together with the target image is provided with labeling information. By mapping the sample image and the target image to obtain content feature data belonging to the same domain, a network model obtained by training using the content feature data and labeling information of the training image can be similarly applied to the target image, and further labeling It is not necessary to carry out, so it is possible to reduce the labeling cost of images in different domains and improve the precision of image processing.

In some embodiments, the sample image and the target image are medical images, and the labeling information is labeling information for a living organ; and/or, the sample image and the target image are three-dimensional images, and the labeling information is labeling information about the three-dimensional target.

Unlike the above-described embodiment, the sample image and the target image are set as medical images, and the labeling information is labeling information of a living organ, so it can be applied to the medical image field; Since the sample image and the target image are set as a three-dimensional image, and the labeling information is set as the labeling information for the three-dimensional target, it can be applied to the processing of a three-dimensional image.

In some embodiments, the preset network model includes any one of an image segmentation network model, an image classification network model, and a target detection network model.

Unlike the above-described embodiment, by setting the preset network model to any one of an image segmentation network model, an image classification network model, and a target detection network model, based on different types of preset network models, different image processing tasks are applied. It can be obtained by training a network model to complete different processing tasks for images in different domains.

In some embodiments, the model training apparatus 70 further includes a training image acquisition module, configured to acquire a first type training image belonging to the first domain and a second type training image belonging to the second domain, The device 70 performs content and style extraction on the first type training image and the second type training image, respectively, using the original de-entanglement model, so that the content characteristic data and the style characteristic data of the first type training image; and an image extraction module, configured to acquire content characteristic data and style characteristic data of the second type training image, wherein the model training apparatus 70 includes: content characteristic data and style characteristic data of the first type training image; and and an image reconstruction module, configured to reconstruct using the content feature data and style feature data of the second type training image to obtain a reconstructed image, wherein the model training device 70 is configured to de-entangle based on the reconstructed image and a loss calculation module, configured to obtain a loss value of the model, wherein the model training apparatus 70 further includes a parameter adjustment module, configured to adjust a parameter of the detangling model based on the loss value.

Unlike the above-described embodiment, feature extraction is performed on the first type training image belonging to the first domain and the second type training image belonging to the second domain using the original de-entanglement model, and content features obtained by extraction By reconstructing based on data and style feature data, based on the reconstructed image obtained by reconstructing, it is possible to obtain a loss value of the de-entanglement model, and further, by adjusting the parameters of the de-entanglement model based on the loss value, de-entanglement It can be obtained by training the model.

In some embodiments, the image reconstruction module is configured to reconstruct using the content feature data and style feature data of the first type training image to obtain a first intra domain reconstruction image, and the content feature data and style of the second type training image an intra-domain reconstruction sub-module, configured to reconstruct using the feature data to obtain a second intra-domain reconstruction image, wherein the image reconstruction module includes: the style feature data of the first type training image and the content of the second type training image; Reconstructed using feature data to obtain a first cross-domain reconstructed image, and reconstructed using the style feature data of the second type training image and content feature data of the first type training image to obtain a second cross-domain reconstructed image and a cross-domain reconfiguration sub-module, configured to obtain. The loss calculation module is configured to obtain a first intra-domain loss value based on the difference between the first type training image and the first intra-domain reconstructed image, and to obtain a difference between the second type training image and the second intra domain reconstructed image. an intra-domain loss calculating sub-module, configured to obtain a second intra-domain loss value based on the difference, wherein the loss calculating module is configured to: to obtain a first content loss value, and based on the difference between the content characteristic data of both the second type training image and the first cross-domain reconstructed image, a content loss calculation submodule, configured to obtain a second content loss value; The method further comprises: a loss calculation module, configured to obtain a first style loss value based on a difference between the style feature data of both the first type training image and the first cross-domain reconstructed image, the second type training image and the second and a style loss calculation submodule, configured to obtain a second style loss value based on a difference between the style feature data of both the cross-domain reconstructed images, wherein the loss calculation module includes: the first type training image and the first cross-domain a cross, configured to obtain a first cross-domain loss value based on a difference between the reconstructed images, and obtain a second cross-domain loss value based on a difference between the second type training image and the second cross-domain reconstructed image It further includes a domain loss calculation submodule, wherein the loss calculation module further includes a loss weighting submodule, configured to perform weighting processing on the obtained loss value to obtain a loss value of the de-entanglement model.

Unlike the above-described embodiment, the loss value of the detangling model by performing weighting processing based on the intra-domain loss value and the cross-domain loss value, and the loss value corresponding to the content feature data and the style feature data after image reconstruction can be obtained, and further, parameter adjustment can be performed based on the overall loss value, and images of different domains can be extracted by the de-entanglement model so that the obtained content feature data are located in the same domain.

Referring to FIG. 8, FIG. 8 is a schematic diagram of a framework of an embodiment of the image processing apparatus 80 provided in the present invention. The image processing apparatus 80 includes a content extraction module 81 and a network processing module 82, and the content extraction module 81 performs content extraction on the target image using the entangling model, and configured to obtain content characteristic data; The network processing module 82 is configured to process the content characteristic data of the target image by using the processing network model to obtain a processing result of the target image, wherein the detangling model uses two types of training images belonging to different domains is obtained by training, one type of training image and target image belong to the same domain, and the processing network model is obtained by using the model training apparatus of any one of the above model training apparatus embodiments.

In the above means, the processing network model obtained by training through the model training apparatus of any one model training apparatus embodiment can be applied not only to a sample image for training the processing network model, but also to a target image. , it is not necessary to perform further labeling, thereby reducing the labeling cost of images of different domains and improving the precision of image processing.

Referring to FIG. 9, FIG. 9 is a schematic diagram of a framework of an embodiment of the image matching apparatus 90 provided in the present invention. The image registration apparatus 90 includes an image acquisition module 91 , a content extraction module 92 , a parameter determination module 93 and an image registration module 94 , and the image acquisition module 91 includes a first image to be registered and acquiring a second image to be registered; The content extraction module 92 performs content extraction on the first to-be-matched image and the second to-be-registered image by using the de-entanglement model, respectively, so that the content characteristic data of the first to-be-matched image and the content of the second to-be-registered image are configured to acquire feature data; the parameter determining module 93 is configured to determine the registration parameter according to the content characteristic data of the first image to be registered and the content characteristic data of the second image to be registered; The image registration module 94 is configured to register the first image to register and the second image to register using the registration parameter, and the detangling model is obtained by training using two types of training images belonging to different domains. one type of training image and the first image to be matched belong to the first domain, and the other type of training image and the second image to be matched belong to the second domain.

In the above means, the detangling model obtained by training through one type of training image belonging to the first domain together with the first image to be registered, and another type of training image belonging to the second domain together with the second image to be registered is , so that the extracted content feature data of the first image to be matched and the content feature data of the second image to be matched belong to the same domain, based on the content feature data of both, directly obtain and register both matching parameters Therefore, it is possible to implement unsupervised registration between images of different domains, further reducing the cost of image registration of different domains, and improving the accuracy.

In some embodiments, the first image to register and the second image to register are medical images; and/or, the first image to register and the second image to register are three-dimensional images; and/or, the matching parameter includes at least one of a stiffness variation parameter and a strain variation parameter.

Unlike the above-described embodiment, by setting the first image to be registered and the second image to be registered as medical images, image registration can be applied to registration of medical images; by setting the first image to be registered and the second image to be registered as three-dimensional images, image registration can be applied to registration of three-dimensional images; The registration parameter may include at least one of a stiffness change parameter and a deformation change parameter, so that the first image to be registered and the second image to be registered are changed correspondingly based on the actual situation of both, and then registration can be implemented, so that the image It is advantageous for improving the accuracy of registration.

In some embodiments, the parameter determining module 93 is configured to determine, based on the content characteristic data of the first image to register and the content characteristic data of the second image to register, a registration parameter corresponding to the first image to register, and , the image registration module 94 includes a change processing submodule, configured to perform change processing on the first image to be registered by using the registration parameter to obtain a change image corresponding to the first image to be registered, the image The registration module 94 further includes a pixel overlap module, configured to superimpose the pixel points of the change image with the corresponding pixel points of the second to-be-registered image.

Unlike the above-described embodiment, by performing change processing on the first image to be registered using the registration parameter corresponding to the first image to be registered, a change image corresponding to the first image to be registered can be obtained, and the change Unsupervised registration between images of different domains can be implemented by superimposing the pixel points of the image to the corresponding pixel points of the second image to be registered.

In some embodiments, the image matching device 90 further includes a training image acquisition module, configured to acquire a first type training image belonging to the first domain and a second type training image belonging to the second domain, The device 90 performs content and style extraction on the first type training image and the second type training image, respectively, using the original de-entanglement model, so that the content characteristic data and the style characteristic data of the first type training image; and an image extraction module, configured to acquire content characteristic data and style characteristic data of the second type training image, wherein the model training apparatus 90 includes: content characteristic data and style characteristic data of the first type training image; and and an image reconstruction module, configured to reconstruct using the content characteristic data and style characteristic data of the second type training image to obtain a reconstructed image, wherein the model training apparatus 90 is configured to de-entangle based on the reconstructed image and a loss calculation module, configured to obtain a loss value of the model, wherein the model training apparatus 90 further includes a parameter adjustment module, configured to adjust a parameter of the detangling model based on the loss value.

Unlike the above-described embodiment, feature extraction is performed on the first type training image belonging to the first domain and the second type training image belonging to the second domain using the original de-entanglement model, and content features obtained by extraction By reconstructing based on data and style feature data, based on the reconstructed image obtained by reconstructing, it is possible to obtain a loss value of the de-entanglement model, and further, by adjusting the parameters of the de-entanglement model based on the loss value, de-entanglement It can be obtained by training the model.

Referring to FIG. 10 , FIG. 10 is a schematic diagram of a framework of an embodiment of an electronic device 100 provided in the present invention. The electronic device 100 includes a memory 101 and a processor 102 coupled to each other, the processor 102 executing program instructions stored in the memory 101, implement the steps, or implement the steps of any of the above image processing method embodiments, or are configured to implement the steps of any of the above image registration method embodiments. In the present embodiment, the electronic device 100 may include a mobile device such as a laptop computer, a tablet PC, and a smart phone, and may include a terminal device such as a microcomputer and a server, but is not limited thereto.

The processor 102 controls itself and the memory 101 to implement the steps of any one model training method embodiment above, or implement the steps of any one image processing method embodiment above, or any of the above. and implement the steps of one image registration method embodiment. The processor 102 may be referred to as a central processing unit (CPU). The processor 102 may be one integrated circuit chip with signal processing capability. The processor 102 may be a general purpose processor, a digital signal processor (DSP), an application specific integrated circuit (ASIC), a field programmable gate array (FPGA), or other programmable logic. devices, discrete gate or transistor logic devices, discrete hardware components, and the like. A general purpose processor may be a microprocessor, or the processor may be any conventional processor or the like. Also, the processor 102 may be jointly implemented by an integrated circuit chip.

The means can reduce the labeling cost of images of different domains and improve the precision of image processing.

Referring to FIG. 11, FIG. 11 is a schematic diagram of a framework of an embodiment of the computer-readable storage medium 110 provided in the present invention. The computer-readable storage medium 110 stores program instructions 111 that can be executed by a processor, and the program instructions 111 implement the steps of any one of the model training method embodiments, or any one of the above. A step of an image processing method embodiment of is implemented, or a step of any one of the above image registration method embodiments is configured to be implemented.

An embodiment of the present invention further provides a computer program product, wherein the computer program product includes one or more instructions, wherein the one or more instructions are loaded by a processor to implement the any one model training method. It is suitable for the example step to be implemented, or the step of any one of the above image processing method embodiments to be implemented, or the step of any one of the above image registration method embodiments to be implemented.

The means can reduce the labeling cost of images of different domains and improve the precision of image processing.

In each of the embodiments provided in the embodiments of the present invention, it should be understood that the disclosed method and apparatus may be implemented in different ways. For example, the device embodiments described above are merely exemplary, for example, a partition of a module or a unit is only a logical function partition, and there may be other partitioning methods in actual implementation, for example, a unit Or a component may be combined or integrated into one other system, or some features may be omitted or not implemented. Further, the couplings or direct couplings or communication connections between each other as indicated or discussed may be implemented via some communication interface, and the indirect couplings or communication connections of the device or unit may be electrical, mechanical, or other form.

A unit described as a separate member may or may not be physically separate, and a member represented as a unit may or may not be a physical unit, that is, may be located in one place or may be distributed in a network unit. Some or all of the units may be selected according to actual needs to realize the purpose of the means of the present embodiment.

In addition, each functional unit in the embodiment of the present invention may be integrated into one processing unit, each unit may be physically present separately, or two or two or more units may be integrated into one unit. . The integrated unit may be implemented using a hardware format or may be implemented using a software functional unit.

When the integrated unit is implemented in the form of a software functional unit and sold or used as an independent product, it may be stored in one computer-readable storage medium. Based on this understanding, the technical solutions of the embodiments of the present invention essentially or the part contributing to the prior art, or all or part of the technical solutions may be implemented in the form of a software product, and the computer software product is one One computer device (which may be a personal computer, server, or network device, etc.) or processor of is stored on the storage medium. The aforementioned storage medium includes a variety of media capable of storing a program code, such as a USB memory, a removable storage device, a read-only memory (ROM), a random access memory (RAM), a magnetic disk or an optical disk. include

In this embodiment, the detangling model obtained by training through one type of training image belonging to the first domain together with the sample image and another type of training image belonging to the second domain together with the target image includes labeling information. By mapping the sample image and the target image to obtain content feature data belonging to the same domain, the network model obtained by training using the content feature data and labeling information of the training image can be applied to the target image as well, Since there is no need to perform more, it is possible to reduce the labeling cost of images in different domains and improve the precision of image processing.

Claims (20)

A model training method comprising:
obtaining at least one sample image, the sample image being provided with labeling information;
obtaining content feature data of the at least one sample image by performing content extraction on each of the at least one sample image using a de-entanglement model; and
training a preset network model using the content feature data and the labeling information of the at least one sample image to obtain a network model for processing a target image, wherein the de-entanglement model belongs to a different domain A model obtained by training using two types of training images, wherein one type of the training image and the sample image belong to a first domain, and the other type of the training image and the target image belong to a second domain. training method. According to claim 1,
the sample image and the target image are medical images, and the labeling information is labeling information for a living organ; and
the sample image and the target image are three-dimensional images, and the labeling information is labeling information for a three-dimensional target; At least one of the model training methods. According to claim 1,
The preset network model includes any one of an image segmentation network model, an image classification network model, and a target detection network model. According to claim 1,
To obtain by training the de-entanglement model, the method comprises:
acquiring a first type training image belonging to the first domain and a second type training image belonging to the second domain;
Content and style extraction of the first type training image and the second type training image are respectively performed on the first type training image and the second type training image by using the original detangling model, so that the content characteristic data and style characteristic data of the first type training image, and the second type acquiring content feature data and style feature data of a type training image;
obtaining a reconstructed image by reconstructing using the content feature data and style feature data of the first type training image, and the content feature data and style feature data of the second type training image;
obtaining a loss value of the de-entanglement model based on the reconstructed image; and
Based on the loss value, the model training method further comprising the step of adjusting a parameter of the de-entanglement model. 5. The method of claim 4,
Reconstructing using the content feature data and style feature data of the first type training image, and the content feature data and style feature data of the second type training image to obtain a reconstructed image,
Reconstructed using the content feature data and style feature data of the first type training image to obtain a first intra domain reconstructed image, and reconstructed using the content feature data and style feature data of the second type training image, acquiring a second intra-domain reconstruction image; and
Reconstructing using the style characteristic data of the first type training image and the content characteristic data of the second type training image to obtain a first cross-domain reconstructed image, the style characteristic data of the second type training image and the second type of training image Reconstructing using the content feature data of the type 1 training image, comprising the step of obtaining a second cross-domain reconstructed image,
Based on the reconstructed image, obtaining the loss value of the de-entanglement model comprises:
based on a difference between the first type training image and the first intra domain reconstructed image, obtain a first intra domain loss value, and based on the difference between the second type training image and the second intra domain reconstructed image to obtain a second intra-domain loss value;
Based on the difference between the content characteristic data of the first type training image and the second cross-domain reconstructed image, a first content loss value is obtained, and a style characteristic of the first type training image and the first cross-domain reconstructed image is obtained. a first style loss value is obtained based on a difference between the data, and a second content loss value is obtained based on a difference between the content feature data of the second type training image and the first cross-domain reconstructed image; obtaining a second style loss value based on a difference between the second type training image and the style feature data of the second cross-domain reconstructed image;
obtain a first cross-domain loss value based on a difference between the first type training image and the first cross-domain reconstructed image, and based on the difference between the second type training image and the second cross-domain reconstructed image to obtain a second cross-domain loss value; and
and performing weighting processing on the obtained loss value to obtain a loss value of the de-entanglement model. An image processing method comprising:
performing content extraction on a target image using a de-entanglement model to obtain content characteristic data of the target image; and
processing the content characteristic data of the target image using a processing network model to obtain a processing result of the target image, wherein the de-entanglement model is trained using two types of training images belonging to different domains. The training image and the target image of one type belong to the same domain, and the processing network model is obtained by training using the method according to any one of claims 1 to 5. image processing method. An image registration method comprising:
acquiring a first image to be registered and a second image to be registered;
Content extraction is performed on the first image to be matched and the second image to be matched using the de-entanglement model, respectively, to obtain the content feature data of the first image to be registered and the content feature data of the second image to be registered obtaining;
determining a registration parameter based on the content characteristic data of the first image to be matched and the content characteristic data of the second image to be matched; and
using the registration parameter to register the first image to be registered and the second image to be registered, wherein the detangling model is obtained by training using two types of training images belonging to different domains. wherein one type of the training image and the first to-be-registered image belong to a first domain, and the other type of the training image and the second to-be-registered image belong to a second domain. 8. The method of claim 7,
the first image to be registered and the second image to be registered are medical images;
the first image to be registered and the second image to be registered are three-dimensional images; and
wherein the matching parameter includes at least one of a stiffness variation parameter and a strain variation parameter; At least one of the image registration methods. 8. The method of claim 7,
Determining a registration parameter based on the content characteristic data of the first image to be matched and the content characteristic data of the second image to be matched includes:
determining the registration parameter corresponding to the first image to be registered based on the content characteristic data of the first image to be registered and the content characteristic data of the second image to be registered;
The step of registering the first image to be matched and the second image to be matched using the matching parameter may include:
performing change processing on the first image to be registered using the registration parameter to obtain a change image corresponding to the first image to be registered; and
and superimposing a pixel point of the change image to a corresponding pixel point of the second image to be registered. 8. The method of claim 7,
To obtain by training the de-entanglement model, the method comprises:
acquiring a first type training image belonging to the first domain and a second type training image belonging to the second domain;
Content and style extraction of the first type training image and the second type training image are respectively performed on the first type training image and the second type training image by using the original detangling model, so that the content characteristic data and style characteristic data of the first type training image, and the second type acquiring content feature data and style feature data of a type training image;
obtaining a reconstructed image by reconstructing using the content feature data and style feature data of the first type training image, and the content feature data and style feature data of the second type training image;
obtaining a loss value of the de-entanglement model based on the reconstructed image; and
Based on the loss value, the image registration method further comprising the step of adjusting a parameter of the de-entanglement model. A model training device comprising:
an image acquisition module, configured to acquire at least one sample image, the sample image being provided with labeling information;
a content extraction module, configured to perform content extraction on each of the at least one sample image by using a de-entanglement model to obtain content feature data of the at least one sample image;
a model training module, configured to train a preset network model using the content feature data and the labeling information of the at least one sample image to obtain a network model for processing a target image, wherein the de-entanglement model comprises: It is obtained by training using two types of training images belonging to different domains, one type of the training image and the sample image belong to a first domain, and the other type of the training image and the target image belong to a second domain. A model training device belonging to a domain. 12. The method of claim 11,
the sample image and the target image are medical images, and the labeling information is labeling information for a living organ; and
the sample image and the target image are three-dimensional images, and the labeling information is labeling information for a three-dimensional target; At least one of a model training device. 12. The method of claim 11,
The preset network model is a model training apparatus including any one of an image segmentation network model, an image classification network model, and a target detection network model. 12. The method of claim 11,
The device is
a training image acquiring module, configured to acquire a first type training image belonging to the first domain and a second type training image belonging to the second domain;
Content and style extraction of the first type training image and the second type training image are respectively performed on the first type training image and the second type training image by using the original detangling model, so that the content characteristic data and style characteristic data of the first type training image, and the second type an image extraction module, configured to acquire content feature data and style feature data of the type training image;
an image reconstruction module, configured to reconstruct using the content feature data and style feature data of the first type training image, and the content feature data and style feature data of the second type training image, to obtain a reconstructed image;
a loss calculation module, configured to obtain a loss value of the de-entanglement model based on the reconstructed image; and
and a parameter adjustment module, configured to adjust a parameter of the de-entanglement model based on the loss value. 15. The method of claim 14,
The image reconstruction module,
Reconstructed using the content feature data and style feature data of the first type training image to obtain a first intra domain reconstructed image, and reconstructed using the content feature data and style feature data of the second type training image, an intra-domain reconstruction sub-module, configured to acquire a second intra-domain reconstruction image; and
Reconstructing using the style characteristic data of the first type training image and the content characteristic data of the second type training image to obtain a first cross-domain reconstructed image, the style characteristic data of the second type training image and the second type of training image a cross-domain reconstruction sub-module, configured to reconstruct using the content characteristic data of the type 1 training image to obtain a second cross-domain reconstruction image;
The loss calculation module,
based on a difference between the first type training image and the first intra domain reconstructed image, obtain a first intra domain loss value, and based on the difference between the second type training image and the second intra domain reconstructed image an intra-domain loss calculating sub-module, configured to obtain a second intra-domain loss value;
Based on the difference between the content characteristic data of the first type training image and the second cross-domain reconstructed image, a first content loss value is obtained, and the content characteristic of the second type training image and the first cross-domain reconstructed image a content loss calculation submodule, configured to obtain a second content loss value based on the difference between the data;
Based on the difference between the style feature data of the first type training image and the first cross-domain reconstructed image, a first style loss value is obtained, and the style characteristics of the second type training image and the second cross-domain reconstructed image are obtained. a style loss calculation submodule, configured to obtain a second style loss value based on the difference between the data;
obtain a first cross-domain loss value based on a difference between the first type training image and the first cross-domain reconstructed image, and based on the difference between the second type training image and the second cross-domain reconstructed image a cross-domain loss calculating sub-module, configured to obtain a second cross-domain loss value; and
and a loss weighting submodule, configured to perform weighting processing on the obtained loss value to obtain a loss value of the de-entanglement model. An image processing device comprising:
a content extraction module, configured to perform content extraction on a target image by using the entanglement model to obtain content characteristic data of the target image; and
a network processing module, configured to process the content characteristic data of the target image by using a processing network model to obtain a processing result of the target image;
The de-entanglement model is obtained by training using two types of training images belonging to different domains, the training image of one type and the target image belong to the same domain, and the processing network model according to claim 11 . An image processing device that is obtained using a model training device according to the An image matching device comprising:
an image acquiring module, configured to acquire a first image to be registered and a second image to be registered;
Content extraction is performed on the first image to be matched and the second image to be matched using the de-entanglement model, respectively, to obtain the content feature data of the first image to be registered and the content feature data of the second image to be registered a content extraction module, configured to acquire;
a parameter determining module, configured to determine a registration parameter based on the content characteristic data of the first image to be registered and the content characteristic data of the second image to be registered; and
an image registration module, configured to register the first image to be registered and the second image to be registered by using the registration parameter;
The de-entanglement model is obtained by training using two types of training images belonging to different domains, one type of the training image and the first image to be matched belong to a first domain, and the other type of the training image belongs to a first domain. The image and the second image to be matched belong to a second domain. As an electronic device,
A memory and a processor coupled to each other, wherein the processor executes program instructions stored in the memory to implement the model training method according to any one of claims 1 to 5, or An electronic device that implements an image processing method or an image registration method according to any one of claims 7 to 10. A computer readable storage medium comprising:
A program instruction is stored, and when the program instruction is executed by the processor, a model training method according to any one of claims 1 to 5 is implemented, or an image processing method according to claim 6 is implemented, or A computer readable storage medium on which the image registration method according to any one of claims 7 to 10 is implemented. A computer program product comprising:
The computer program product comprises one or more instructions, the one or more instructions being loaded and executed by a processor to implement a model training method according to any one of claims 1 to 5, or A computer program product implementing the image processing method according to claim 6 , or implementing the image registration method according to any one of claims 7 to 10 .

Images