WO2022011984A1

WO2022011984A1 - Image processing method and apparatus, electronic device, storage medium, and program product

Info

Publication number: WO2022011984A1
Application number: PCT/CN2020/140330
Authority: WO
Inventors: 张宏; 张靖阳; 夏清
Original assignee: 上海商汤智能科技有限公司
Priority date: 2020-07-16
Filing date: 2020-12-28
Publication date: 2022-01-20
Also published as: TW202203854A; CN111798498A; TWI767614B; KR20220016212A; JP2022543531A

Abstract

Embodiments of the present invention relate to an image processing method and apparatus, an electronic device, a storage medium, and a program product. The method comprises: acquiring a first segmentation result of a target object in a first image; acquiring a second segmentation result of a target object in a second image; and obtaining a deformation field between the first image and the second image according to the first segmentation result and the second segmentation result, wherein the deformation field comprises a position transformation relationship of each pixel point of the target object between the first image and the second image.

Description

Image processing method and apparatus, electronic device, storage medium and program product

CROSS-REFERENCE TO RELATED APPLICATIONS

The present disclosure is based on the Chinese patent application with the application number of 202010686919.6 and the filing date of July 16, 2020, and claims the priority of the Chinese patent application. The entire content of the Chinese patent application is hereby incorporated into the present disclosure in its entirety. .

technical field

The present disclosure relates to the technical field of image processing, and in particular, to an image processing method and apparatus, an electronic device, a storage medium, and a program product.

Background technique

Coronary heart disease has become one of the diseases with the highest mortality in the world, and the common treatment option is percutaneous coronary intervention. Percutaneous coronary intervention is the use of a catheter to dilate the narrowed part of the blood vessel under the guidance of intraoperative X-rays to achieve the purpose of treatment. However, during the operation, the blood vessels displayed in the X-ray image of the coronary artery will become invisible as the contrast agent dissipates, which brings great challenges to the doctor, and the success rate of the operation also depends on the actual experience of the doctor .

Computed tomography angiography (CTA) images taken before surgery can well demonstrate the vascular structure, but because they cannot be captured in real-time during surgery, it is impossible to give doctors guidance during surgery.

SUMMARY OF THE INVENTION

The embodiments of the present disclosure provide an image processing method and apparatus, an electronic device, a storage medium, and a program product.

According to an aspect of the embodiments of the present disclosure, an image processing method is provided, including:

Obtain the first segmentation result of the target object in the first image; obtain the second segmentation result of the target object in the second image; obtain the first image and the second segmentation result according to the first segmentation result and the second segmentation result A deformation field between second images, wherein the deformation field includes a positional transformation relationship of each pixel of the target object between the first image and the second image.

Through the above process, the positional transformation relationship of each pixel of the target object between the first image and the second image can be determined. Using this position transformation relationship, the image information of the target object between the first image and the second image can be fused into the same coordinate system, so that the image information of the target object contained in the first image and the second image can be used at the same time. The subsequent operations that need to be performed provide comprehensive guidance; moreover, since the position transformation relationship is the transformation relationship corresponding to each pixel point of the target object, the information fusion of the target object between the first image and the second image can have a higher level. accuracy.

In a possible implementation manner, the obtaining the deformation field between the first image and the second image according to the first segmentation result and the second segmentation result includes: A segmentation result and the second segmentation result are input to the first neural network to obtain the deformation fields of the first image and the second image.

Through the above process, on the one hand, the neural network can be used to realize the end-to-end deformation field prediction. Compared with determining the position transformation relationship by pixel, the acquisition time of the deformation field can be greatly shortened, the efficiency of the deformation field acquisition can be improved, and the whole process can be effectively improved. The efficiency of the image processing process and the subsequent image registration process; on the other hand, the deformation field obtained by the neural network can include the positional transformation relationship of each pixel between the first image and the second image, which can maximize the deformation field. The degree of freedom improves the precision and accuracy of the deformation field, thereby improving the accuracy of the entire image processing process and the subsequent image registration process.

In a possible implementation manner, the first image includes a three-dimensional image, and the second image includes a two-dimensional image; and the first segmentation result and the second segmentation result are used to obtain the first segmentation result. The deformation field between the image and the second image includes: converting the first segmentation result into a two-dimensional third segmentation result according to the collection information of the second image; converting the third segmentation result with the The second segmentation result is input to the first neural network to obtain the deformation field between the first image and the second image.

Through the above process, the collection information of the two-dimensional second image can be used to project the first segmentation result of the first image to a two-dimensional plane, so that the first image and the second image can be obtained according to the two two-dimensional segmentation results. Therefore, the obtained deformation field can more accurately reflect the transformation relationship between the first image and the second image of the target object, and improve the accuracy and effect of image processing.

In a possible implementation manner, the method further includes: registering the first image and the second image according to the deformation field to obtain a registration result.

Through the above process, the obtained deformation field can be used to flexibly integrate the target object information contained in the first image and the target object information contained in the second image into one coordinate system, so that the target object-based Operation provides comprehensive and effective guidance.

In a possible implementation manner, the method further includes: obtaining an error loss of the first neural network according to the deformation field; and training the first neural network according to the error loss.

In the embodiment of the present disclosure, by obtaining the error loss of the first neural network according to the deformation field, and then training the first neural network according to the error loss, the transformation relationship between the two input images of the first neural network can be directly used to The first neural network is trained without additional training images or labeled data, which reduces the difficulty and cost of training while ensuring the training accuracy of the first neural network.

In a possible implementation manner, the obtaining the error loss of the first neural network according to the deformation field includes: registering the first segmentation result according to the deformation field to obtain a registration After the first segmentation result, the error between the registered first segmentation result and the second segmentation result is used as the error loss of the first neural network; or, according to the deformation field, for all The second segmentation result is registered to obtain the registered second segmentation result, and the error between the registered second segmentation result and the first segmentation result is used as the error of the first neural network. loss; according to the deformation field, the first segmentation result is registered to obtain the registered first segmentation result, and the error between the registered first segmentation result and the second image is calculated As the error loss of the first neural network; or, according to the deformation field, the second segmentation result is registered to obtain the registered second segmentation result, and the registered second segmentation result is obtained. The error between the result and the first image serves as the error loss for the first neural network.

Through the above process of obtaining the error loss of the first neural network, an appropriate method can be flexibly selected to determine the error loss of the first neural network according to the actual situation, thereby improving the flexibility and convenience of training the first neural network.

In a possible implementation manner, the obtaining the first segmentation result of the target object in the first image includes: inputting the first image into a second neural network to obtain the target object in the first image The first segmentation result of the first segmentation result of the target object, wherein the first neural network is further configured to obtain the difference between the first image and the second image according to the first segmentation result and the second segmentation result deformation field.

The target object in the first image is segmented through the second neural network or the first neural network to obtain the first segmentation result, which can effectively improve the obtaining efficiency of the first segmentation result. At the same time, since the second neural network or the first neural network can be obtained by training the first training image containing the target object annotation, the first segmentation result obtained based on the second neural network or the first neural network can have a higher Accurate segmentation effect. The target object in the first image is segmented through the first neural network to obtain a first segmentation result, and the deformation field between the first image and the second image is further obtained through the first neural network. Through the above process, the accuracy of the obtained deformation field can be further improved on the basis of improving the segmentation effect of the first segmentation result, and the acquisition process from the first image end to the deformation field end can be directly realized through the first neural network.

In a possible implementation manner, the acquiring the second segmentation result of the target object in the second image includes: inputting the second image into a third neural network to obtain the target object in the second image The second segmentation result of the second segmentation result of the target object, wherein the first neural network is further configured to obtain the difference between the first image and the second image according to the first segmentation result and the second segmentation result deformation field.

The target object in the second image is segmented through the third neural network or the first neural network to obtain the second segmentation result, which can effectively improve the obtaining efficiency of the second segmentation result. At the same time, since the third neural network or the first neural network can be obtained by training the second training image containing the target object annotation, the second segmentation result obtained based on the third neural network or the first neural network can have higher Accurate segmentation effect. The target object in the second image is segmented through the first neural network to obtain a second segmentation result, and the deformation field between the first image and the second image is obtained through the first neural network. Through the above process, on the basis of improving the segmentation effect of the second segmentation result, the accuracy of the obtained deformation field can be further improved, and the acquisition process from the second image end to the deformation field end can be directly realized through the first neural network, and can also be obtained through the first neural network. The first neural network directly realizes the acquisition process from the two image ends of the first image and the second image to the deformation field end.

In a possible implementation manner, the first image includes an electronic computed tomography angiography CTA image, the second image includes an X-ray image, and the target object includes a coronary artery object.

When the first image includes a CTA image, the second image includes an X-ray image, and the target object includes a coronary artery, the image processing method proposed in the embodiment of the present disclosure can effectively predict the deformation field between the CTA image and the X-ray image , thereby unifying the two modal data of coronary surgery into the same coordinate system, compensating for coronary blood vessels that cannot be seen on X-ray images during coronary surgery, providing better guidance for coronary surgery, and reducing the need for doctors. The complexity of the operation increases the success rate of the operation.

According to an aspect of the embodiments of the present disclosure, there is provided an image processing apparatus, including:

The first segmentation module is configured to obtain the first segmentation result of the target object in the first image; the second segmentation module is configured to obtain the second segmentation result of the target object in the second image; the deformation field acquisition module is configured to The first segmentation result and the second segmentation result obtain a deformation field between the first image and the second image, wherein the deformation field includes the target object between the first image and the second image. The position transformation relationship of each pixel point between the second images.

According to an aspect of the embodiments of the present disclosure, there is provided an electronic device, comprising: a processor; a memory configured to store instructions executable by the processor; wherein the processor is configured to invoke the instructions stored in the memory , to perform the above image processing method.

According to an aspect of the embodiments of the present disclosure, there is provided a computer-readable storage medium having computer program instructions stored thereon, the computer program instructions implementing the above-mentioned image processing method when executed by a processor.

According to an aspect of the embodiments of the present disclosure, a computer program product is provided, the program product stores one or more program instructions, and the program instructions are loaded and executed by a processor to implement the above-mentioned image processing method.

In the embodiment of the present disclosure, the first segmentation result and the second segmentation result of the target object in the first image and the second image are obtained respectively, so as to obtain the first segmentation result and the second segmentation result according to the first segmentation result and the second segmentation result. and the deformation field between the second image. Through the above process, the positional transformation relationship of each pixel of the target object between the first image and the second image can be determined, and the image information of the target object between the first image and the second image can be fused by using the positional transformation relationship to the same coordinate system, so that the image information of the target object contained in the first image and the second image can be used at the same time to provide comprehensive guidance for the subsequent operations of the target object; Therefore, the information fusion of the target object between the first image and the second image can have higher precision.

It is to be understood that the foregoing general description and the following detailed description are exemplary and explanatory only and are not limiting of the disclosed embodiments.

Other features and aspects of embodiments of the present disclosure will become apparent from the following detailed description of exemplary embodiments with reference to the accompanying drawings.

Description of drawings

The accompanying drawings, which are incorporated into and constitute a part of this specification, illustrate embodiments consistent with the present disclosure, and together with the description, serve to explain the technical solutions of the present disclosure.

FIG. 1 shows a flowchart of an image processing method according to an embodiment of the present disclosure.

FIG. 2 shows a schematic diagram of a training process of a registration neural network in an application example according to the present disclosure.

FIG. 3 shows a block diagram of an image processing apparatus according to an embodiment of the present disclosure.

FIG. 4 shows a block diagram of an electronic device according to an embodiment of the present disclosure.

FIG. 5 shows a block diagram of an electronic device according to an embodiment of the present disclosure.

detailed description

Various exemplary embodiments, features and aspects of embodiments of the present disclosure will be described in detail below with reference to the accompanying drawings. The same reference numbers in the figures denote elements that have the same or similar functions. While various aspects of the embodiments are shown in the drawings, the drawings are not necessarily drawn to scale unless otherwise indicated.

The word "exemplary" is used exclusively herein to mean "serving as an example, embodiment, or illustration." Any embodiment described herein as "exemplary" is not necessarily to be construed as preferred or advantageous over other embodiments.

The term "and/or" in this article is only an association relationship to describe the associated objects, indicating that there can be three kinds of relationships, for example, A and/or B, it can mean that A exists alone, A and B exist at the same time, and A and B exist independently B these three cases. In addition, the term "at least one" herein refers to any combination of any one of the plurality or at least two of the plurality, for example, including at least one of A, B, and C, and may mean including from A, B, and C. Any one or more elements selected from the set of B and C.

In addition, in order to better illustrate the embodiments of the present disclosure, numerous implementation details are given in the following detailed description. It should be understood by those skilled in the art that embodiments of the present disclosure may be practiced without certain details. In some instances, methods, means, components and circuits well known to those skilled in the art have not been described in detail so as not to obscure the subject matter of the embodiments of the present disclosure.

FIG. 1 shows a flowchart of an image processing method according to an embodiment of the present disclosure. The method may be applied to an image processing apparatus, and the image processing apparatus may be a terminal device, a server, or other processing devices. Wherein, the terminal device may be user equipment (User Equipment, UE), mobile device, user terminal, terminal, cellular phone, cordless phone, Personal Digital Assistant (PDA), handheld device, computing device, in-vehicle device, available wearable devices, etc.

In some possible implementations, the image processing method may be implemented by the processor calling computer-readable instructions stored in the memory.

As shown in Figure 1, the image processing method may include:

Step S11, obtaining a first segmentation result of the target object in the first image.

Step S12, acquiring a second segmentation result of the target object in the second image.

Step S13, obtaining a deformation field between the first image and the second image according to the first segmentation result and the second segmentation result, wherein the deformation field includes each pixel of the target object between the first image and the second image position transformation relationship.

The target object can be any object that needs to be registered between the two images. Its implementation form can be flexibly determined according to the actual application scenario of the image processing method proposed in the embodiments of the present disclosure.

The image processing method proposed by the embodiments of the present disclosure can be flexibly applied to various scenarios according to actual requirements. For example, in a possible implementation manner, the method proposed by the embodiments of the present disclosure may be applied in a surgical procedure, for example, may be used to register an image captured before surgery and an image captured during surgery, Or register the images taken before the operation and the images taken after the operation, etc. In this case, the realization form of the target object can be flexibly changed according to the different objects of the operation. In one example, the method proposed by the embodiments of the present disclosure may be applied to coronary artery surgery, such as percutaneous coronary intervention, etc. In this case, the target object may be a coronary artery object or the like.

In a possible implementation manner, the method proposed by the embodiments of the present disclosure can also be applied to other scenarios, for example, can be applied to the process of diagnosing a patient's disease, for example, can be used to diagnose a patient for a certain period of time In this case, the realization form of the target object can be flexibly changed according to the different positions of the monitored lesions. In one example, the method proposed by the embodiments of the present disclosure may be applied to monitor the condition of the patient's heart, in this case, the target object may be a heart object or the like.

For the convenience of description, the subsequent disclosed embodiments are described by taking the image processing method used for the operation of the coronary artery as an example, and the target object is a coronary artery object. Flexible expansion is performed according to the subsequent disclosed embodiments, and will not be expanded one by one.

The realization forms of the first image and the second image can also be flexibly determined according to the application scenario of the image processing method. The second image may be an image captured at different time periods before, during, or after coronary artery surgery, and the actual selection is not limited to the following disclosed embodiments. For example, the first image may be an image captured before surgery, and the second image may be an image captured during surgery. In a possible implementation manner, the first image and the second image may also be images with different attributes or types, for example, the first image may be a three-dimensional image, the second image may be a two-dimensional image, and the like.

Various possible implementation forms of the above-mentioned first image and second image can also be flexibly combined with each other. For example, in a possible implementation form, the first image can include a three-dimensional CTA image captured before surgery , the second image may include an X-ray image taken during the operation, and the target object may include a coronary artery object. When the first image includes a CTA image, the second image includes an X-ray image, and the target object includes a coronary artery, the image processing method proposed in the embodiment of the present disclosure can effectively predict the deformation field between the CTA image and the X-ray image , thereby unifying the two modal data of coronary surgery into the same coordinate system, compensating for coronary blood vessels that cannot be seen on X-ray images during coronary surgery, providing better guidance for coronary surgery, and reducing the need for doctors. The complexity of the operation increases the success rate of the operation.

Since the implementation forms of the first image and the second image are not limited, correspondingly, the number of the first image and the second image is not limited in the embodiments of the present disclosure, and can be flexibly selected according to the actual situation, and is not limited to the following disclosure Example. In a possible implementation manner, the second image may include multiple X-ray images, that is, the registration between the CTA image and the multiple X-ray images may be implemented. In one example, the multiple X-ray images may be multiple X-ray images captured in real time of a coronary artery object during coronary surgery, by matching the CTA images with the multiple X-ray images captured during the surgery. It can realize real-time image registration in coronary surgery, so as to better display the position of blood vessels in real time during the operation, and provide real-time and accurate guidance and assistance to the doctor during the operation.

After the first image, the second image and the target object are determined, the first segmentation result of the target object can be obtained from the first image and the first segmentation result of the target object can be obtained from the second image through step S11 and step S12 respectively. Divide the result. Among them, the numbers such as "first" and "second" in the first segmentation result and the second segmentation result are only used to distinguish the segmentation results obtained from different images, and do not limit the realization form of the segmentation results. In fact, the realization forms of the first segmentation result and the second segmentation result are flexibly determined by the realization forms of the corresponding segmented images and the target object. The implementation form of step S11 and step S12 is not limited. For details, please refer to the following disclosed embodiments, which will not be expanded here. It should be noted that, in the embodiment of the present disclosure, the implementation order of step S11 and step S12 is not limited, and step S11 and step S12 may be performed sequentially in a certain order according to requirements, or may be performed simultaneously.

After the first segmentation result and the second segmentation result are obtained, the deformation field between the first image and the second image can be determined based on the first segmentation result and the second segmentation result through step S13, wherein the deformation field can reflect the The position transformation relationship of each pixel point between the first image and the second image. The implementation form of step S13 can be flexibly selected according to the actual situation. For details, please refer to the subsequent disclosed embodiments, which will not be expanded here.

As described in the above disclosed embodiments, the manner of obtaining the first segmentation result of the target object from the first image is not limited. In a possible implementation manner, the first segmentation result may be obtained from the first image by applying any blood vessel segmentation algorithm in the image. In a possible implementation manner, step S11 may include:

The first image is input to the second neural network to obtain a first segmentation result of the target object in the first image, wherein the second neural network is trained by the first training image containing the target object annotation. or,

The first image is input into the first neural network to obtain the first segmentation result of the target object in the first image, wherein the first neural network is also used to obtain the first image and the second segmentation result according to the first segmentation result and the second segmentation result. The deformation field between the two images.

It can be seen from the above disclosed embodiments that, in a possible implementation manner, the target object in the first image can be segmented through the second neural network having the segmentation function, thereby obtaining the first segmentation result. The implementation form of the second neural network can be flexibly determined according to the actual situation, and is not limited to the following disclosed embodiments. In a possible implementation manner, a convolutional neural network (U-Net) can be used as the second neural network. The first training image for training the second neural network can also be flexibly selected according to the actual situation of the first image. In a possible implementation manner, in the case that the first image includes a CTA image, the first training image may include Pixel-by-pixel vessel annotated CTA images.

It can also be seen from the above disclosed embodiments that, in a possible implementation manner, the target object in the first image can be segmented through the first neural network with the segmentation function, thereby obtaining the first segmentation result. Among them, based on the above disclosed embodiments, it can be seen that the first neural network can not only be used to segment the target object in the first image, but also has a deformation field acquisition function, that is, it can be used to obtain a deformation field according to the first segmentation result and the first Divide the result into two to obtain the deformation field between the first image and the second image. In this case, the first neural network may sequentially obtain the first segmentation result of the first image by inputting the first image and the second segmentation result, and obtain the first image according to the first segmentation result and the second segmentation result and the deformation field between the second image.

In the case where the first neural network can be used to segment the target object in the first image and obtain the deformation field between the first image and the second image, the first neural network can be the same as the above-mentioned second neural network , the training is performed by the first training image containing the target object annotation, and the training can also be performed according to the first segmentation result and the second segmentation result, wherein the first segmentation result can be the target object annotation in the first training image. Therefore, in In a possible implementation manner, the first neural network may be trained by using the first training image and the second segmentation result marked with the target object.

In some other embodiments, the implementation form and training process of the first neural network can also be flexibly selected according to the actual situation. For details, please refer to the subsequent disclosed embodiments, which will not be expanded here.

The target object in the first image is segmented by the second neural network or the first neural network to obtain the first segmentation result, which can effectively improve the obtaining efficiency of the first segmentation result. At the same time, due to the second neural network or the first neural network It can be obtained by training on the first training image containing the target object label. Therefore, the first segmentation result obtained based on the second neural network or the first neural network can have a higher-precision segmentation effect.

In some other embodiments, the target object in the first image is segmented through the first neural network to obtain a first segmentation result, and the deformation field between the first image and the second image is further obtained through the first neural network Through the above process, the accuracy of the obtained deformation field can be further improved on the basis of improving the segmentation effect of the first segmentation result, and the acquisition process from the first image end to the deformation field end can be directly realized through the first neural network.

Similarly, the manner of obtaining the first segmentation result of the target object from the second image is not limited. In a possible implementation manner, any blood vessel segmentation algorithm applied to the image can also be used to obtain the second segmentation result from the second image. In a possible implementation manner, step S12 may include:

The second image is input to the third neural network to obtain a second segmentation result of the target object in the second image, wherein the third neural network is trained by the second training image containing the target object annotation. or,

Inputting the second image into the first neural network to obtain a second segmentation result of the target object in the second image, wherein the first neural network is also used to obtain the first image according to the first segmentation result and the second segmentation result and the deformation field between the second image.

It can be seen from the above disclosed embodiments that, in a possible implementation manner, the target object in the second image may be segmented through a third neural network with a segmentation function, thereby obtaining a second segmentation result. The implementation form of the third neural network can be flexibly determined according to the actual situation, and is not limited to the following disclosed embodiments. In a possible implementation manner, the U-Net network can also be used as the third neural network. The second training image for training the third neural network can also be flexibly selected according to the actual situation of the second image. In a possible implementation manner, when the second image includes an X-ray image, the first training image can be X-ray images containing pixel-by-pixel vessel annotations.

It can also be seen from the above disclosed embodiments that, in a possible implementation manner, the target object in the second image can be segmented by using the first neural network with segmentation function to obtain the second segmentation result. Among them, based on the above disclosed embodiments, it can be seen that the first neural network can not only be used to segment the target object in the second image, but also has a deformation field acquisition function, that is, it can be used to obtain a deformation field according to the first segmentation result and the third Divide the result into two to obtain the deformation field between the first image and the second image. In this case, the first neural network can sequentially obtain the second segmentation result of the second image by inputting the second image and the first segmentation result, and obtain the first image according to the second segmentation result and the first segmentation result and the deformation field between the second image.

As described in the above disclosed embodiments, the first neural network may also be used to segment the target object in the first image, so in a possible implementation manner, the first neural network may also include segmenting the first image , segmenting the second image and obtaining the deformation field. In this case, the first neural network can obtain the first segmentation result and the first segmentation result of the first image by inputting the first image and the second image, respectively. The second segmentation result of the two images, and the deformation field between the first image and the second image is obtained according to the first segmentation result and the second segmentation result.

In the case where the first neural network can be used to segment the target object in the second image and obtain the deformation field between the first image and the second image, the first neural network can be similar to the third neural network described above. In the same way, training can be performed on the second training image containing the target object annotation, and training can also be performed according to the first segmentation result and the second segmentation result, wherein the second segmentation result can be the target object annotation in the second training image. Therefore, In a possible implementation manner, the first neural network can be trained by using the second training image marked with the target object and the first segmentation result. In some other embodiments, the first neural network can both perform training on the first image In the case of segmenting, segmenting the second image, and obtaining the deformation field, the first neural network can be simultaneously trained by the first training image containing the target object annotation and the second training image containing the target object annotation.

It should be noted that the labels such as "first", "second" and "third" in the first neural network, the second neural network, and the third neural network in the embodiments of the present disclosure are only used to distinguish different The functional neural network does not limit the implementation form of the neural network. In the embodiments of the present disclosure, the implementation forms of the first neural network, the second neural network, and the third neural network may be the same or different.

The target object in the second image is segmented by the third neural network or the first neural network to obtain the second segmentation result, which can effectively improve the obtaining efficiency of the second segmentation result. At the same time, due to the third neural network or the first neural network It can be obtained by training on the second training image containing the target object label. Therefore, the second segmentation result obtained based on the third neural network or the first neural network can have a higher-precision segmentation effect.

In some other embodiments, the target object in the second image is segmented through the first neural network to obtain a second segmentation result, and the deformation field between the first image and the second image is further obtained through the first neural network , through the above process, on the basis of improving the segmentation effect of the second segmentation result, the accuracy of the obtained deformation field can be further improved, and the acquisition process from the second image end to the deformation field end can be directly realized through the first neural network, and also The acquisition process from the two image ends, the first image and the second image, to the deformation field end is directly realized through the first neural network.

After the first segmentation result and the second segmentation result are obtained, the deformation field between the first image and the second image may be acquired through step S13. In a possible implementation manner, step S13 may include:

The first segmentation result and the second segmentation result are input into the first neural network to obtain the deformation fields of the first image and the second image.

It can be seen from the above disclosed embodiments that, in a possible implementation manner, the pixel position transformation relationship between the first segmentation result and the second segmentation result can be performed by the first neural network with the function of obtaining the deformation field. Extraction, thereby obtaining the deformation field between the first segmentation result and the second segmentation result. In a possible implementation, the deformation field between the first segmentation result and the second segmentation result can be directly used as the deformation field between the first image and the second image; in a possible implementation , or according to the relationship between the first image and the first segmentation result, and the relationship between the second image and the second segmentation result, this deformation field can be correspondingly converted into the transformation relationship between the two images, so as to obtain The deformation field between the first image and the second image.

The implementation form of the first neural network can be flexibly determined according to the actual situation, and is not limited to the following disclosed embodiments. In a possible implementation manner, a U-Net network can be used as the first neural network. How to train the first neural network so that it can determine the deformation field according to the input first segmentation result and the second segmentation result, the training process can refer to the following disclosed embodiments, which will not be expanded here.

The first segmentation result and the second segmentation result are processed through the first neural network to obtain the deformation field between the first image and the second image. Through the above process, on the one hand, the neural network can be used to realize the end-to-end deformation Compared with determining the position transformation relationship pixel by pixel, field prediction can greatly shorten the acquisition time of the deformation field, improve the acquisition efficiency of the deformation field, and then effectively improve the efficiency of the entire image processing process and subsequent image registration process; on the other hand, The deformation field obtained by the neural network can include the positional transformation relationship of each pixel between the first image and the second image, which can maximize the degree of freedom of the deformation field, improve the precision and accuracy of the deformation field, and thus improve the overall Accuracy of image processing and subsequent image registration.

As described in the above disclosed embodiments, the first image and the second image may have different properties. In a possible implementation manner, the first image may include a three-dimensional image, and the second image may include a two-dimensional image. In this case, the process of acquiring the deformation field between the first image and the second image according to the first segmentation result and the second segmentation result can be flexibly changed. Therefore, in a possible implementation manner, step S13 may include:

Step S131, converting the first segmentation result into a two-dimensional third segmentation result according to the collection information of the second image;

Step S132, the third segmentation result and the second segmentation result are input to the first neural network to obtain the deformation field between the first image and the second image.

As described in the above disclosed embodiments, since the first image includes a three-dimensional image and the second image includes a two-dimensional image, correspondingly, the first segmentation result obtained from the first image may be a three-dimensional segmentation result, and the second The second segmentation result obtained in the image may be a two-dimensional segmentation result, and converting the first segmentation result into a two-dimensional segmentation result can facilitate subsequent acquisition of the deformation between the three-dimensional first image and the two-dimensional second image. Therefore, in a possible implementation manner, step S131 may be used to convert the first segmentation result into a two-dimensional third segmentation result according to the acquisition information of the second image.

Wherein, the collection information of the second image may be any information related to the collection angle or collection method of the second image during the second image collection process, and its implementation form can be flexibly determined according to the actual situation, and is not limited to the following disclosure Example. In a possible implementation manner, in the case where the second image is an X-ray image, the acquisition information may include header file information of Digital Imaging and Communications in Medicine (DICOM) of the second image. Reading the DICOM header file information can determine the angle at which the X-ray image was taken.

The manner of converting the first segmentation result into the two-dimensional third segmentation result according to the collected information is also not limited, and can be flexibly determined according to the actual situation of the collected information. In a possible implementation manner, in the case where the collection information may include DICOM header file information, the shooting angle of the second image may be determined according to the DICOM header file information, and the first segmentation result may be projected according to the shooting angle , to get the third segmentation result. The manner of projecting the first segmentation result is not limited. In an example, the projected third segmentation result may be obtained by using a ray projection algorithm.

After the two-dimensional third segmentation result is obtained, the third segmentation result and the second segmentation result may be input into the first neural network through step S132 to obtain the deformation field between the first image and the second image. For the processing methods of the first neural network and the first neural network on the third segmentation result and the second segmentation result, reference may be made to the processing methods of the first neural network on the first segmentation result and the second segmentation result in the above disclosed embodiments, It is not repeated here.

It should be noted that, in the embodiment of the present disclosure, after the first segmentation result is processed as the third segmentation result, the deformation field obtained by the first neural network based on the third segmentation result and the second segmentation result is the third segmentation result. The deformation field between the segmentation result and the second segmentation result, in a possible implementation, this deformation field can be directly used as the deformation field between the first image and the second image; in a possible implementation , it is also possible to further process this deformation field according to the corresponding relationship between the transformation of the first image to the third segmentation result and the transformation of the second image to the second segmentation result to obtain the difference between the first image and the second image. direct deformation field. With the different realization forms of the deformation field, the subsequent processing operations performed on the first image and the second image by using the deformation field may also change accordingly.

In a possible implementation manner, the process of converting the first segmentation result into the third segmentation result can also be implemented by the first neural network. In this case, the first neural network can directly convert the first segmentation result and the second segmentation result as input, the conversion from the first segmentation result to the third segmentation result is sequentially performed inside the neural network, and the deformation between the first image and the second image is obtained according to the third segmentation result and the second segmentation result. field.

In the case where the first image includes a three-dimensional image and the second image includes a two-dimensional image, the first segmentation result is converted into a two-dimensional third segmentation result according to the acquisition information of the second image, so that the third segmentation result and the The second segmentation result is input to the first neural network to obtain the deformation field between the first image and the second image. Through the above process, the first segmentation result of the first image can be divided into Projection to a two-dimensional plane, so as to obtain the deformation field between the first image and the second image according to the two two-dimensional segmentation results, so that the obtained deformation field can more accurately reflect the difference between the first image and the second image of the target object. The transformation relationship between the two images improves the accuracy and effect of image processing.

After the deformation field between the first image and the second image is acquired, the first image and the second image can be processed correspondingly by using the deformation field, such as the registration mentioned in the above disclosed embodiments. Therefore, in a possible implementation manner, the method proposed by the embodiment of the present disclosure may further include:

According to the deformation field, the first image and the second image are registered to obtain a registration result.

As described in the above disclosed embodiments, the deformation field can reflect the positional transformation relationship of each pixel of the target object between the first image and the second image. Therefore, the target object in the first image and the second image can be transformed by the deformation field. The target object in the image is transformed into the same coordinate system, so as to realize the registration between the first image and the second image, and obtain the registration result.

The registration process can be flexibly determined according to the actual situation of the deformation field. As described in the above disclosed embodiments, in a possible implementation manner, the deformation field may be the deformation field between the segmentation results, such as the deformation field between the first segmentation result and the second segmentation result, or the third segmentation result The result or the deformation field between the second segmentation results, etc. In this case, the process of registering the first image and the second image can be the process of deforming the corresponding segmentation results according to the deformation field, that is, Transform the first segmentation result into the coordinate system of the second segmentation result using the deformation field, transform the third segmentation result into the coordinate system of the second segmentation result using the deformation field, and transform the second segmentation result into the first segmentation result using the deformation field or the coordinate system used to transform the second segmentation result to the third segmentation result by using the deformation field, etc.

In a possible implementation manner, the deformation field can also be obtained by further processing the deformation field between the images on the basis of the deformation field of the segmentation result, that is, the direct deformation field between the first image and the second image, In this case, the process of registering the first image and the second image may be a deformation process by directly processing the first image or the second image, that is, using the deformation field to transform the first image to the second image. The coordinate system of the second image, or the coordinate system of the second image is transformed into the first image by using the deformation field, etc.

In a possible implementation, the registration process may not be limited to the image or the coordinate system where the segmentation result is located. For example, a deformation field may be used to register both the first image and the second image to a preset coordinate system, or both the first segmentation result and the second segmentation result are registered to a preset coordinate system, and so on.

The actual registration method is not limited in the embodiments of the present disclosure, and is not limited to the following disclosed embodiments. In one example, the registration result can be obtained by comparing the images to be registered by using Spatial Transformer Networks (STN).

By registering the first image and the second image according to the deformation field to obtain the registration result, the obtained deformation field can be used to flexibly combine the target object information contained in the first image with the target object contained in the second image Information is unified and fused into a single coordinate system to provide comprehensive and effective guidance on the object-based operations to be performed.

As described in the above disclosed embodiments, in a possible implementation manner, the first neural network may be used to obtain the deformation field. In order to make the acquired deformation field more accurate, the first neural network can be trained to have higher accuracy. That is, the image processing method proposed by the embodiment of the present disclosure can also be used in the training process of the first neural network. In this case, in a possible implementation manner, the image processing method proposed by the embodiment of the present disclosure may include:

Step S13, obtaining a deformation field between the first image and the second image according to the first segmentation result and the second segmentation result.

Step S14, according to the deformation field, obtain the error loss of the first neural network.

Step S15, train the first neural network according to the error loss.

For the implementation process of steps S11 to S13, reference may be made to the above disclosed embodiments, and details are not described herein again. The first neural network may be an untrained neural network, or may be a trained but incompletely trained neural network.

After the deformation field is obtained, the error loss of the first neural network can be obtained according to the deformation field. The acquisition method of the error loss can be flexibly selected according to the actual situation. In a possible implementation manner, step S14 may include:

Step S141, according to the deformation field, register the first segmentation result, obtain the registered first segmentation result, and use the error between the registered first segmentation result and the second segmentation result as the error of the first neural network. error loss. or,

Step S142: According to the deformation field, the second segmentation result is registered to obtain the registered second segmentation result, and the error between the registered second segmentation result and the first segmentation result is used as the error of the first neural network. error loss. or,

Step S143: According to the deformation field, the first segmentation result is registered to obtain the registered first segmentation result, and the error between the registered first segmentation result and the second image is used as the error of the first neural network. loss. or,

Step S144, register the second segmentation result according to the deformation field, obtain the registered second segmentation result, and use the error between the registered second segmentation result and the first image as the error of the first neural network loss.

Since the deformation field can reflect the transformation relationship between the first segmentation result and the second segmentation, the first segmentation result after registration can be obtained by using the deformation field output by the first neural network to register the first segmentation result. , if the deformation field is completely accurate, the registered first segmentation result and the second segmentation result will be consistent. Therefore, through the error between the registered first segmentation result and the second segmentation result, the first The error of the deformation field output by the neural network is used as the error loss of the first neural network to train the first neural network, which can improve the accuracy of the first neural network obtained after training.

Similarly, the deformation field can also be used to register the second segmentation result, so that the error between the registered second segmentation result and the first segmentation result can be used to determine the error of the deformation field output by the first neural network. , and then determine the error loss of the first neural network.

In a possible implementation manner, the acquisition of the error loss of the first neural network may be configured during the training process of the first neural network, and during the training process, the first segmentation result input to the first neural network may be marked with is located on the first image in the form of , and the second segmentation result may also be located on the second image in the form of annotations. Therefore, in this case, the error between the registered first segmentation result and the second image where the second segmentation result is located, or the error between the registered second segmentation result and the first segmentation result The error between the first images is used as the error of the first neural network.

As mentioned in the above disclosed embodiments, the deformation field may be the deformation field between the first segmentation result and the second segmentation result, the deformation field between the third segmentation result and the second segmentation result, or the deformation field between the third segmentation result and the second segmentation result. The deformation field between one image and the second image, etc., therefore, with the different objects pointed by the deformation field, the determined error can change flexibly. For example, when the deformation field is the difference between the third segmentation result and the second segmentation result In the case of the deformation field between the two, the deformation field can be used to register the third segmentation result to obtain the registered third segmentation result, and then determine the third segmentation result according to the error between the registered third segmentation result and the second segmentation result. The error loss of a neural network, etc., other implementation forms can be flexibly expanded according to the above disclosed embodiments, and will not be listed one by one here. For the registration method, reference may be made to the above disclosed embodiments, and details are not described herein again.

The manner of calculating the error between different objects can be flexibly selected according to the actual situation, and is not limited to the following disclosed embodiments. In one example, the calculation method of a loss function such as Mean Squared Error (MSE) or Normalized Cross Correlation (NCC) can be used to determine the error between different objects.

After the error loss of the first neural network is obtained, the first neural network can be trained through step S15, and the training method can be flexibly determined according to the actual situation, and is not limited to the following disclosed embodiments. In a possible implementation manner, various network parameters and the like in the first neural network may be updated by using the method of back propagation according to the error loss of the first neural network.

Preoperative CTA images can show the three-dimensional vascular structure well, but since CTA images cannot be captured in real time during the operation, it is necessary to register the preoperative CTA and intraoperative X-ray images to fuse them into the same coordinate system to provide doctors with information. Better guidance reduces the complexity of the surgery for doctors and improves the success rate of surgery.

The coronary registration method in the related art regards the registration problem as an optimization problem, defines a similarity to measure the distance between two blood vessels, and iteratively optimizes the distance to find an optimal transformation matrix. Another scheme extends the nearest iterative point method from point sets to curves, and proposes an iterative nearest curve algorithm for the registration of curve structures. There is also a probabilistic and statistical precise registration scheme for coherent point drift, which defines the registration of two point sets as a probability density estimation problem. The above schemes require iterative optimization using the most recent iterative point or the coherent point drift, which is often difficult to meet the requirements of intraoperative real-time performance. At the same time, deformations such as B-splines or thin-plate splines are used in the scheme, which cannot well meet the complex vessel deformation, resulting in low registration accuracy.

Deep learning techniques have made great achievements in the field of computer vision and also provide new solutions for medical image registration. One scheme trains a fully convolutional neural network to perform non-rigid registration of 3D brain MR images using "self-supervision"; the other uses normalized cross-correlation to train a fully convolutional neural network to predict deformation fields, to register 4D cardiac MR images; another scheme uses convolutional neural networks and spatial transformation networks to register T1-weighted brain MR images; another scheme uses convolutional neural networks and spatial transformation networks to register T1-weighted brain MR images registration; another approach utilizes a transfer learning-based approach to separately register X-ray and cardiac sequence images. Although the above registration methods have achieved great success, they are difficult to meet the multi-modal and multi-dimensional registration problem.

The embodiments of the present disclosure propose an end-to-end coronary registration method. In the embodiment of the present disclosure, the blood vessel bundle of the preoperative CTA image and the blood vessel bundle of the intraoperative X-ray image are firstly segmented, and the data of two different modalities and dimensions are unified into one coordinate system by using the ray projection method, and then input into a single coordinate system. The deformation field is directly predicted in the U-Net network. The method of the embodiment of the present disclosure can predict the deformation field end-to-end, and meet the requirements of intraoperative real-time performance while ensuring the registration accuracy.

The embodiment of the present disclosure proposes an image processing method, which can perform real-time registration of a preoperative CTA image and an intraoperative X-ray image of a coronary artery. The image processing process can be as follows:

The 3D U-Net network (ie the second neural network in the above disclosed embodiment) is used to segment the preoperative CTA image (ie the first image in the above disclosed embodiment), and the blood vessels in the CTA image are extracted bundle (that is, the first segmentation result in the above disclosed embodiment);

The U-Net network (ie the third neural network in the above disclosed embodiment) is used to segment the intraoperative X-ray image (ie the second image in the above disclosed embodiment), and the blood vessel bundles in the X-ray image (ie the above-mentioned the second segmentation result in the disclosed embodiment);

Read the header file information of the DICOM in the X-ray image (that is, the acquisition information in the above-mentioned disclosed embodiments), and use the light projection algorithm to generate a digitally reconstructed radiological image for the blood vessel bundle in the CTA image, and obtain a two-dimensional blood vessel projection map (ie the third segmentation result in the above disclosed embodiment);

Input the two-dimensional blood vessel projection map and the blood vessel bundle in the X-ray image into the registration neural network (ie, the first neural network in the above disclosed embodiment), and output the corresponding deformation field; the registration neural network used in this process For the deep learning network, the deformation field can be directly predicted end-to-end, which greatly improves the real-time performance; at the same time, the process predicts the displacement of each pixel point, maximizes the degree of freedom of the deformation field, and improves the registration accuracy.

Using the output deformation field, the two-dimensional blood vessel projection map or the blood vessel bundle in the X-ray image can be transformed to complete the registration process.

In some embodiments, since the above-mentioned registration process needs to utilize 3D U-Net network, U-Net network and registration neural network, the application example of the present disclosure also proposes an image processing method, which can be used for each of the above-mentioned neural networks. To train:

Perform pixel-by-pixel blood vessel labeling on the preoperative CTA image to obtain the corresponding label, and use the original image of the preoperative CTA image and the corresponding label (that is, the first training image containing the target object label in the above disclosed embodiment) to train for 3D U-Net network for CTA vessel segmentation;

Perform pixel-by-pixel blood vessel labeling on the intraoperative X-ray image to obtain the corresponding label, and use the original image of the intraoperative X-ray image and the corresponding label (that is, the second training image containing the target object label in the above disclosed embodiment) to train U-Net network for X-ray vessel segmentation.

FIG. 2 shows a schematic diagram of a training process of a registration neural network in an application example of the present disclosure. As shown in FIG. 2 , the training process may be:

Use the trained 3D U-Net network and U-Net network to segment the preoperative CTA image 201 and the intraoperative X-ray image 202, respectively, to obtain the blood vessel bundle 203 in the CTA image and the blood vessel bundle 204 in the X-ray image;

Read the DICOM header file information in the X-ray image, determine the shooting angle of the intraoperative X-ray image, and use the light projection algorithm to project the blood vessel bundle 203 in the CTA image to obtain the projected blood vessel bundle 205;

Input the projected blood vessel bundle 205 and the blood vessel bundle 204 in the X-ray image into an untrained initial registration neural network 206 (which can be a U-Net network), and output a vascular bundle) of the same size as the predicted deformation field 207, the predicted deformation field containing the displacement of each pixel;

Using the spatial transformation network 208, the projected vascular bundle is deformed according to the predicted deformation field to obtain the deformed vascular bundle 209;

Calculate the loss function 210 between the deformed blood vessel bundle 209 and the blood vessel bundle 205 in the X-ray image. The calculation method can use mean square error or normalized cross-correlation, etc., and then use the back propagation algorithm to update the registration neural network. parameters to complete the training process of the registration neural network.

Through the above process, the embodiment of the present disclosure can obtain an end-to-end coronary registration network. By inputting preoperative CTA images and intraoperative X-ray images, the network can directly predict the deformation field and complete the registration task: Using the trained U-Net to segment the X-ray image and the intraoperative X-ray image to obtain the blood vessel bundle of the X-ray image; read the DICOM header file information, and use the preoperative CTA blood vessel bundle to generate the projected blood vessel bundle; The beams are fed into the registration network and the deformation fields are obtained.

In the embodiment of the present disclosure, there is no need to iteratively optimize the similarity measure function to find the optimal transformation relationship between the preoperative CTA image and the intraoperative X-ray image, but directly realize the end-to-end transformation through a deep learning network, that is, a registration neural network. Deformation field prediction. After experimental verification, the above-mentioned registration process is performed on the Central Processing Unit (CPU), and the deformation field can be obtained within 1 second, which greatly improves the real-time performance of the registration process. At the same time, since the predicted deformation field includes the displacement of each pixel point in the vascular bundle, the degree of freedom of deformation is maximized. Compared with the transformation methods based on B-splines or thin-plate splines, the configuration proposed in the application example of the present disclosure is improved. The registration method can greatly improve the registration accuracy.

In the actual application process, after obtaining the preoperative CTA image and the intraoperative X-ray image, the radiologist can use the method proposed in the application example of the present disclosure to perform fast and accurate registration, and unify the data of the two modalities into the same one In the coordinate system, it compensates for the problem of coronary vessels that cannot be seen on intraoperative X-ray images. At the same time, since the application example of the present disclosure can perform real-time registration of the coronary arteries included in the preoperative CTA image and the intraoperative X-ray image, the intraoperative X-ray image can better display the position of the catheter, so that the doctor can perform the operation during the operation. Have a better judgement of the direction in which the catheter is traveling.

It should be noted that, the image processing method in the embodiment of the present disclosure is not limited to be applied to the above-mentioned processing of coronary images of the heart, and may be applied to any image processing, which is not limited in the embodiment of the present disclosure.

It can be understood that the above method embodiments mentioned in the embodiments of the present disclosure can be combined with each other to form a combined embodiment without violating the principle and logic. Those skilled in the art can understand that, in the above-mentioned method of the specific embodiment, the execution order of each step should be determined by its function and possible internal logic.

In addition, the embodiments of the present disclosure also provide image processing apparatuses, electronic devices, computer-readable storage media, and programs, all of which can be used to implement any image processing method provided by the embodiments of the present disclosure, and the corresponding technical solutions and descriptions and refer to the methods Some of the corresponding records will not be repeated.

FIG. 3 shows a block diagram of an image processing apparatus according to an embodiment of the present disclosure. The image processing apparatus may be a terminal device, a server, or other processing devices. Wherein, the terminal device may be user equipment (User Equipment, UE), mobile device, user terminal, terminal, cellular phone, cordless phone, Personal Digital Assistant (PDA), handheld device, computing device, in-vehicle device, available wearable devices, etc.

In some possible implementations, the image processing apparatus may be implemented by a processor invoking computer-readable instructions stored in a memory.

As shown in FIG. 3 , the image processing apparatus 30 may include:

The first segmentation module 31 is configured to obtain a first segmentation result of the target object in the first image.

The second segmentation module 32 is configured to obtain a second segmentation result of the target object in the second image.

The deformation field acquiring module 33 is configured to obtain a deformation field between the first image and the second image according to the first segmentation result and the second segmentation result, wherein the deformation field includes the target object between the first image and the second image The position transformation relationship of each pixel of .

In a possible implementation manner, the deformation field acquisition module is configured to input the first segmentation result and the second segmentation result into the first neural network to obtain the deformation fields of the first image and the second image.

In a possible implementation manner, the first image includes a three-dimensional image, and the second image includes a two-dimensional image; the deformation field acquisition module is configured to convert the first segmentation result into a two-dimensional third image according to the acquisition information of the second image Segmentation result; input the third segmentation result and the second segmentation result to the first neural network to obtain the deformation field between the first image and the second image.

In a possible implementation manner, the image processing apparatus 30 further includes: a registration module, configured to register the first image and the second image according to the deformation field to obtain a registration result.

In a possible implementation manner, the image processing apparatus 30 further includes: an error acquisition module, configured to acquire the error loss of the first neural network according to the deformation field; train.

In a possible implementation manner, the error acquisition module is configured to register the first segmentation result according to the deformation field, obtain the registered first segmentation result, and compare the registered first segmentation result with the second segmentation result. The error between the results is used as the error loss of the first neural network; or, according to the deformation field, the second segmentation result is registered to obtain the registered second segmentation result, and the registered second segmentation result is compared with the first segmentation result. The error between the first segmentation results is used as the error loss of the first neural network; according to the deformation field, the first segmentation results are registered to obtain the registered first segmentation results, and the registered first segmentation results and the first segmentation results are obtained. The error between the two images is used as the error loss of the first neural network; or, according to the deformation field, the second segmentation result is registered to obtain the registered second segmentation result, and the registered second segmentation result is compared with The error between the first images is used as the error loss of the first neural network.

In a possible implementation manner, the first segmentation module is configured to input the first image into the second neural network to obtain a first segmentation result of the target object in the first image, wherein the second neural network is marked by including the target object The first training image is trained; or, the first image is input into the first neural network, and the first segmentation result of the target object in the first image is obtained, wherein the first neural network is also used for according to the first segmentation result and the first segmentation result As a result of the binary segmentation, the deformation field between the first image and the second image is obtained.

In a possible implementation manner, the second segmentation module is configured to input the second image into a third neural network to obtain a second segmentation result of the target object in the second image, wherein the third neural network is marked by including the target object The second training image is trained; or, the second image is input to the first neural network to obtain the second segmentation result of the target object in the second image, wherein the first neural network is also used to As a result of the binary segmentation, the deformation field between the first image and the second image is obtained.

In a possible implementation, the first image includes an electronic computed tomography angiography CTA image, the second image includes an X-ray image, and the target object includes a coronary artery object.

An embodiment of the present disclosure further provides a computer-readable storage medium, on which computer program instructions are stored, and when the computer program instructions are executed by a processor, the above-mentioned image processing method is implemented. The computer-readable storage medium may be a non-volatile computer-readable storage medium.

An embodiment of the present disclosure further provides an electronic device, comprising: a processor; a memory configured to store instructions executable by the processor; wherein the processor is configured to invoke the instructions stored in the memory to execute the above method.

Embodiments of the present disclosure also provide a computer program product, including computer-readable codes. When the computer-readable codes are run on a device, a processor in the device executes the image processing method for implementing the image processing method provided by any of the above embodiments. instruction.

Embodiments of the present disclosure further provide another computer program product for storing computer-readable instructions, which, when executed, cause the computer to perform the operations of the image processing method provided by any of the foregoing embodiments.

The electronic device may be provided as a terminal, server or other form of device.

FIG. 4 shows a block diagram of an electronic device 800 according to an embodiment of the present disclosure. For example, electronic device 800 may be a mobile phone, computer, digital broadcast terminal, messaging device, game console, tablet device, medical device, fitness device, and personal digital assistant, among other terminals.

4, an electronic device 800 may include one or more of the following components: a processing component 802, a memory 804, a power supply component 806, a multimedia component 808, an audio component 810, an input/output (I/O) interface 812 , sensor component 814 , and communication component 816 .

The processing component 802 generally controls the overall operations of the electronic device 800, such as operations associated with display, phone calls, data communications, camera operations, and recording operations. The processing component 802 can include one or more processors 820 to execute instructions to perform all or some of the steps of the methods described above. Additionally, processing component 802 may include one or more modules that facilitate interaction between processing component 802 and other components. For example, processing component 802 may include a multimedia module to facilitate interaction between multimedia component 808 and processing component 802.

Memory 804 is configured to store various types of data to support operation at electronic device 800 . Examples of such data include instructions for any application or method operating on electronic device 800, contact data, phonebook data, messages, pictures, videos, and the like. The memory 804 may be implemented by any type of volatile or non-volatile storage device or a combination thereof, such as Static Random-Access Memory (SRAM), Electrically Erasable Programmable Read-Only Memory (Electrically Erasable Programmable Read-Only Memory) Erasable Programmable Read-Only Memory, EEPROM), Erasable Programmable Read-Only Memory (EPROM), Programmable Read-Only Memory (PROM), Read Only Memory (Read Only Memory) Memory, ROM), magnetic memory, flash memory, magnetic disk or optical disk.

Power supply assembly 806 provides power to various components of electronic device 800 . Power supply components 806 may include a power management system, one or more power supplies, and other components associated with generating, managing, and distributing power to electronic device 800 .

Multimedia component 808 includes a screen that provides an output interface between the electronic device 800 and the user. In some embodiments, the screen may include a Liquid Crystal Display (LCD) and a touch panel (TouchPanel, TP). If the screen includes a touch panel, the screen may be implemented as a touch screen to receive input signals from a user. The touch panel includes one or more touch sensors to sense touch, swipe, and gestures on the touch panel. The touch sensor may not only sense the boundaries of a touch or swipe action, but also detect the duration and pressure associated with the touch or swipe action. In some embodiments, the multimedia component 808 includes a front-facing camera and/or a rear-facing camera. When the electronic device 800 is in an operation mode, such as a shooting mode or a video mode, the front camera and/or the rear camera may receive external multimedia data. Each of the front and rear cameras can be a fixed optical lens system or have focal length and optical zoom capability.

Audio component 810 is configured to output and/or input audio signals. For example, the audio component 810 includes a microphone (Microphone, MIC) configured to receive external audio signals when the electronic device 800 is in an operating mode, such as a calling mode, a recording mode, and a voice recognition mode. The received audio signal may be further stored in memory 804 or transmitted via communication component 816 . In some embodiments, audio component 810 also includes a speaker for outputting audio signals.

The I/O interface 812 provides an interface between the processing component 802 and a peripheral interface module, which may be a keyboard, a click wheel, a button, or the like. These buttons may include, but are not limited to: home button, volume buttons, start button, and lock button.

Sensor assembly 814 includes one or more sensors for providing status assessment of various aspects of electronic device 800 . For example, sensor assembly 814 can detect the open/closed state of electronic device 800 and the relative positioning of the assembly. For example, the components are the display and keypad of the electronic device 800, the sensor component 814 can also detect a change in the position of the electronic device 800 or a component of the electronic device 800, the presence or absence of user contact with the electronic device 800, the orientation of the electronic device 800, or the presence or absence of contact with the electronic device 800. Acceleration/deceleration and temperature change of electronic device 800. Sensor assembly 814 may include a proximity sensor configured to detect the presence of nearby objects in the absence of any physical contact. Sensor assembly 814 may also include a light sensor, such as a Complementary Metal-Oxide-Semiconductor (CMOS) or Charge Coupled Device (CCD) image sensor, for use in imaging applications. In some embodiments, the sensor assembly 814 may also include an acceleration sensor, a gyroscope sensor, a magnetic sensor, a pressure sensor, or a temperature sensor.

Communication component 816 is configured to facilitate wired or wireless communication between electronic device 800 and other devices. The electronic device 800 can access a wireless network based on a communication standard, such as Wireless Fidelity (Wi-Fi), the 2nd Generation (The 2nd Generation, 2G) or the 3rd Generation (The 3nd Generation) , 3G) or their combination. In one exemplary embodiment, the communication component 816 receives broadcast signals or broadcast related information from an external broadcast management system via a broadcast channel. In an exemplary embodiment, the communication component 816 also includes a Near Field Communication (NFC) module to facilitate short-range communication. For example, the NFC module may be based on Radio Frequency Identification (RFID) technology, Infrared Data Association (IrDA) technology, Ultra Wide Band (UWB) technology, Bluetooth (Blue Tooth, BT) technology and other technologies to achieve.

In an exemplary embodiment, the electronic device 800 may be implemented by one or more Application Specific Integrated Circuits (ASICs), Digital Signal Processors (DSPs), Digital Signal Processing Devices (Digital Signal Processing Devices) , DSPD), Programmable Logic Device (PLD), Field Programmable Gate Array (FPGA), controller, microcontroller, microprocessor, or other electronic component implementation, used to perform the above method.

In an exemplary embodiment, a non-volatile computer-readable storage medium, such as a memory 804 comprising computer program instructions executable by the processor 820 of the electronic device 800 to perform the above method is also provided.

FIG. 5 shows a block diagram of an electronic device 1900 according to an embodiment of the present disclosure. For example, the electronic device 1900 may be provided as a server. 5, electronic device 1900 includes processing component 1922, which may include one or more processors, and memory resources represented by memory 1932 for storing instructions executable by processing component 1922, such as applications. An application program stored in memory 1932 may include one or more modules, each corresponding to a set of instructions. Additionally, the processing component 1922 is configured to execute instructions to perform the above-described methods.

The electronic device 1900 may also include a power supply assembly 1926 configured to perform power management of the electronic device 1900, a wired or wireless network interface 1950 configured to connect the electronic device 1900 to a network, and an I/O interface 1958. Electronic device 1900 may operate based on an operating system stored in memory 1932, such as Windows Server™, Mac OS X™, Unix™, Linux™, FreeBSD™, or the like.

In an exemplary embodiment, a non-volatile computer-readable storage medium is also provided, such as memory 1932 comprising computer program instructions executable by processing component 1922 of electronic device 1900 to perform the above-described method.

Embodiments of the present disclosure may be systems, methods and/or computer program products. The computer program product may include a computer-readable storage medium having computer-readable program instructions loaded thereon for causing a processor to implement various aspects of the embodiments of the present disclosure.

A computer-readable storage medium may be a tangible device that can hold and store instructions for use by the instruction execution device. The computer-readable storage medium may be, for example, but not limited to, an electrical storage device, a magnetic storage device, an optical storage device, an electromagnetic storage device, a semiconductor storage device, or any suitable combination of the foregoing. Computer-readable storage media may include: portable computer disks, hard disks, random access memory (RAM), read-only memory, erasable programmable read-only memory (EPROM or flash memory), static random access memory, Portable Compact Disc Read-Only Memory (CD-ROM), Digital Video Disc (DVD), memory sticks, floppy disks, mechanical coding devices, such as punch cards on which instructions are stored Or the protruding structure in the groove, and any suitable combination of the above. Computer-readable storage media, as used herein, are not to be construed as transient signals per se, such as radio waves or other freely propagating electromagnetic waves, electromagnetic waves propagating through waveguides or other transmission media (eg, light pulses through fiber optic cables), or through electrical wires transmitted electrical signals.

The computer readable program instructions described herein may be downloaded to various computing/processing devices from a computer readable storage medium, or to an external computer or external storage device over a network such as the Internet, a local area network, a wide area network, and/or a wireless network. The network may include copper transmission cables, fiber optic transmission, wireless transmission, routers, firewalls, switches, gateway computers, and/or edge servers. A network adapter card or network interface in each computing/processing device receives computer-readable program instructions from a network and forwards the computer-readable program instructions for storage in a computer-readable storage medium in each computing/processing device .

Computer program instructions for performing operations of embodiments of the present disclosure may be assembly instructions, Industry Standard Architecture (ISA) instructions, machine instructions, machine-related instructions, pseudocode, firmware instructions, state setting data, or in a form of Source or object code written in any combination of programming languages, including object-oriented programming languages such as Smalltalk, C++, etc., as well as conventional procedural programming languages such as C or similar programming languages. The computer readable program instructions may execute entirely on the user's computer, partly on the user's computer, as a stand-alone software package, partly on the user's computer and partly on a remote computer, or entirely on the remote computer or server implement. In the case of a remote computer, the remote computer may be connected to the user's computer through any kind of network, including a Local Area Network (LAN) or Wide Area Network (WAN), or may be connected to an external computer (eg, using the Internet service provider to connect via the Internet). In some embodiments, electronic circuits, such as programmable logic circuits, field programmable gate arrays, or programmable logic arrays, that can execute computer readable program instructions are personalized by utilizing state information of computer readable program instructions , thereby implementing various aspects of the embodiments of the present disclosure.

Aspects of embodiments of the present disclosure are described herein with reference to flowchart illustrations and/or block diagrams of methods, apparatus (systems) and computer program products according to embodiments of the disclosure. It will be understood that each block of the flowchart illustrations and/or block diagrams, and combinations of blocks in the flowchart illustrations and/or block diagrams, can be implemented by computer readable program instructions.

These computer readable program instructions may be provided to a processor of a general purpose computer, special purpose computer or other programmable data processing apparatus to produce a machine that causes the instructions when executed by the processor of the computer or other programmable data processing apparatus , resulting in means for implementing the functions/acts specified in one or more blocks of the flowchart and/or block diagrams. These computer readable program instructions can also be stored in a computer readable storage medium, these instructions cause a computer, programmable data processing apparatus and/or other equipment to operate in a specific manner, so that the computer readable medium on which the instructions are stored includes An article of manufacture comprising instructions for implementing various aspects of the functions/acts specified in one or more blocks of the flowchart and/or block diagrams.

Computer readable program instructions can also be loaded onto a computer, other programmable data processing apparatus, or other equipment to cause a series of operational steps to be performed on the computer, other programmable data processing apparatus, or other equipment to produce a computer-implemented process , thereby causing instructions executing on a computer, other programmable data processing apparatus, or other device to implement the functions/acts specified in one or more blocks of the flowcharts and/or block diagrams.

The flowchart and block diagrams in the Figures illustrate the architecture, functionality, and operation of possible implementations of systems, methods and computer program products according to various embodiments of the present disclosure. In this regard, each block in the flowchart or block diagrams may represent a module, segment, or portion of instructions, which comprises one or more functions for implementing the specified logical function(s) executable instructions. In some alternative implementations, the functions noted in the blocks may occur out of the order noted in the figures. For example, two blocks in succession may, in fact, be executed substantially concurrently, or the blocks may sometimes be executed in the reverse order, depending upon the functionality involved. It is also noted that each block of the block diagrams and/or flowchart illustrations, and combinations of blocks in the block diagrams and/or flowchart illustrations, can be implemented in dedicated hardware-based systems that perform the specified functions or actions , or can be implemented in a combination of dedicated hardware and computer instructions.

The computer program product can be implemented in hardware, software or a combination thereof. In an optional embodiment, the computer program product may be embodied as a computer storage medium, and in another optional embodiment, the computer program product may be embodied as a software product, such as a software development kit (Software Development Kit, SDK), etc. Wait.

Various embodiments of the present disclosure have been described above, and the foregoing descriptions are exemplary, not exhaustive, and not limiting of the disclosed embodiments. Numerous modifications and variations will be apparent to those of ordinary skill in the art without departing from the scope and spirit of the described embodiments. The terminology used herein was chosen to best explain the principles of the various embodiments, the practical application or improvement over the technology in the marketplace, or to enable others of ordinary skill in the art to understand the various embodiments disclosed herein.

Industrial Applicability

Embodiments of the present disclosure relate to an image processing method and apparatus, an electronic device, a storage medium, and a program product. The method includes: acquiring a first segmentation result of a target object in a first image; acquiring a second segmentation result of the target object in a second image; and obtaining the first segmentation result according to the first segmentation result and the second segmentation result. A deformation field between an image and the second image, wherein the deformation field includes a positional transformation relationship of each pixel of the target object between the first image and the second image. Through the above process, the information fusion of the target object between the first image and the second image can have higher precision.

Claims

An image processing method, comprising:

obtaining the first segmentation result of the target object in the first image;

obtaining the second segmentation result of the target object in the second image;

According to the first segmentation result and the second segmentation result, a deformation field between the first image and the second image is obtained, wherein the deformation field includes the target object in the first image and the position transformation relationship of each pixel point between the second image.
The method according to claim 1, wherein the obtaining a deformation field between the first image and the second image according to the first segmentation result and the second segmentation result comprises:

The first segmentation result and the second segmentation result are input into a first neural network to obtain a deformation field between the first image and the second image.
The method of claim 1, wherein the first image comprises a three-dimensional image and the second image comprises a two-dimensional image;

The obtaining of the deformation field between the first image and the second image according to the first segmentation result and the second segmentation result includes:

converting the first segmentation result into a two-dimensional third segmentation result according to the collection information of the second image;

The third segmentation result and the second segmentation result are input into the first neural network to obtain the deformation field between the first image and the second image.
The method according to claim 2 or 3, wherein the method further comprises:

According to the deformation field, the first image and the second image are registered to obtain a registration result.
The method according to claim 2 or 3, wherein the method further comprises:

obtaining the error loss of the first neural network according to the deformation field;

The first neural network is trained according to the error loss.
The method according to claim 5, wherein the obtaining the error loss of the first neural network according to the deformation field comprises:

According to the deformation field, the first segmentation result is registered to obtain the registered first segmentation result, and the error between the registered first segmentation result and the second segmentation result is taken as the error loss of the first neural network; or,

According to the deformation field, the second segmentation result is registered to obtain the registered second segmentation result, and the error between the registered second segmentation result and the first segmentation result is taken as the error loss of the first neural network; or,

According to the deformation field, the first segmentation result is registered to obtain the registered first segmentation result, and the error between the registered first segmentation result and the second image is used as the the error loss of the first neural network; or,

According to the deformation field, the second segmentation result is registered to obtain the registered second segmentation result, and the error between the registered second segmentation result and the first image is used as the The error loss of the first neural network is described.
The method according to any one of claims 2 to 6, wherein the acquiring the first segmentation result of the target object in the first image comprises:

Inputting the first image to a second neural network to obtain a first segmentation result of the target object in the first image, wherein the second neural network is trained by using the first training image marked with the target object ;or,

Inputting the first image to the first neural network to obtain a first segmentation result of the target object in the first image, wherein the first neural network is further configured to obtain a first segmentation result according to the first segmentation result With the second segmentation result, a deformation field between the first image and the second image is obtained.
The method according to any one of claims 2 to 7, wherein the acquiring the second segmentation result of the target object in the second image comprises:

Inputting the second image into a third neural network to obtain a second segmentation result of the target object in the second image, wherein the third neural network is trained by using the second training image marked with the target object ;or,

Inputting the second image to the first neural network to obtain a second segmentation result of the target object in the second image, wherein the first neural network is further configured to obtain a second segmentation result according to the first segmentation result With the second segmentation result, a deformation field between the first image and the second image is obtained.
8. The method of any one of claims 1 to 8, wherein the first image comprises a CTA image, the second image comprises an X-ray image, and the target object comprises a coronary artery object.
An image processing device, comprising:

a first segmentation module, configured to obtain a first segmentation result of the target object in the first image;

a second segmentation module, configured to obtain a second segmentation result of the target object in the second image;

A deformation field acquisition module, configured to obtain a deformation field between the first image and the second image according to the first segmentation result and the second segmentation result, wherein the deformation field includes the target The position transformation relationship of each pixel of the object between the first image and the second image.
The apparatus according to claim 10, wherein the deformation field acquisition module is further configured to:

The first segmentation result and the second segmentation result are input into the first neural network to obtain the deformation fields of the first image and the second image.
The apparatus according to claim 10, wherein the first image includes a three-dimensional image, and the second image includes a two-dimensional image; the deformation field acquisition module is further configured to:

According to the collection information of the second image, the first segmentation result is converted into a two-dimensional third segmentation result; the third segmentation result and the second segmentation result are input into the first neural network, and the difference between the first image and the second image is obtained. deformation field.
The apparatus of claim 11 or 12, wherein the apparatus further comprises:

The registration module is configured to perform registration on the first image and the second image according to the deformation field to obtain a registration result.
The apparatus of claim 11 or 12, wherein the apparatus further comprises:

an error obtaining module, configured to obtain the error loss of the first neural network according to the deformation field;

A training module configured to train the first neural network according to the error loss.
The apparatus of claim 11, wherein,

The error acquisition module is further configured to perform registration on the first segmentation result according to the deformation field, obtain a registered first segmentation result, and compare the registered first segmentation result with the The error between the second segmentation results is used as the error loss of the first neural network; or,

According to the deformation field, the second segmentation result is registered to obtain the registered second segmentation result, and the error between the registered second segmentation result and the first segmentation result is taken as the error loss of the first neural network; or,

According to the deformation field, the first segmentation result is registered to obtain the registered first segmentation result, and the error between the registered first segmentation result and the second image is used as the the error loss of the first neural network; or,

According to the deformation field, the second segmentation result is registered to obtain the registered second segmentation result, and the error between the registered second segmentation result and the first image is used as the The error loss of the first neural network is described.
The device according to any one of claims 11 to 15, wherein,

The first segmentation module is further configured to input the first image into a second neural network to obtain a first segmentation result of the target object in the first image, wherein the second neural network includes the target object-labeled first training image for training; or,

Inputting the first image to the first neural network to obtain a first segmentation result of the target object in the first image, wherein the first neural network is further configured to obtain a first segmentation result according to the first segmentation result With the second segmentation result, a deformation field between the first image and the second image is obtained.
The apparatus of any one of claims 11 to 16, wherein,

The second segmentation module is configured to input the second image into a third neural network to obtain a second segmentation result of the target object in the second image, wherein the third neural network includes the target object-labeled second training images for training; or,

Inputting the second image into the first neural network to obtain a second segmentation result of the target object in the second image, wherein the first neural network is further configured to obtain a second segmentation result according to the first segmentation result With the second segmentation result, a deformation field between the first image and the second image is obtained.
18. The apparatus of any one of claims 10 to 17, wherein the first image includes a CTA image, the second image includes an X-ray image, and the target object includes a coronary artery object.
An electronic device comprising:

processor;

a memory configured to store instructions executable by the processor;

wherein the processor is configured to invoke the memory-stored instructions to perform the method of any one of claims 1-9.
A computer-readable storage medium, storing computer program instructions in the storage medium, the computer program instructions implementing the method of any one of claims 1 to 9 when executed by a processor.
A computer program product having computer-readable instructions stored in the program product, the computer-readable instructions implementing the method according to any one of claims 1 to 9 when executed.