CN115115772A

CN115115772A - Key structure reconstruction method and device based on three-dimensional image and computer equipment

Info

Publication number: CN115115772A
Application number: CN202210384845.XA
Authority: CN
Inventors: 秦陈陈; 常健博; 姚建华; 冯铭; 刘翌勋; 王任直; 陈亦豪
Original assignee: Tencent Technology Shenzhen Co Ltd; Peking Union Medical College Hospital Chinese Academy of Medical Sciences
Current assignee: Tencent Technology Shenzhen Co Ltd; Peking Union Medical College Hospital Chinese Academy of Medical Sciences
Priority date: 2022-04-13
Filing date: 2022-04-13
Publication date: 2022-09-27

Abstract

The application relates to a three-dimensional image-based key structure reconstruction method, a three-dimensional image-based key structure reconstruction device, a computer device, a storage medium and a computer program product. The method comprises the following steps: carrying out pre-sampling processing on an original three-dimensional medical image of a target part to obtain a three-dimensional medical image with a specified size; performing semantic segmentation processing on the three-dimensional medical image with the specified size to obtain a corresponding three-dimensional marked image; post-sampling processing is carried out on the basis of the three-dimensional marked image to obtain a target marked image meeting a high-resolution condition; segmenting the target marked image based on the probability value corresponding to each voxel in the three-dimensional marked image to obtain at least one three-dimensional structural image; and respectively reconstructing each three-dimensional structure image to obtain a target three-dimensional structure model respectively corresponding to each key structure. The method remarkably improves the accuracy of reconstructing the key structure by fusing the computer vision technology and the machine learning technology.

Description

Key structure reconstruction method and device based on three-dimensional image and computer equipment

Technical Field

The present application relates to the field of image technologies, and in particular, to a method, an apparatus, a computer device, a storage medium, and a computer program product for reconstructing a key structure based on a three-dimensional image, and a method, an apparatus, a computer device, a storage medium, and a computer program product for training an image processing model.

Background

With the development of computer technology, various medical scanning apparatuses, such as a CT (Computed Tomography) apparatus, a magnetic resonance apparatus, an ultrasound apparatus, etc., have appeared, and different parts of different objects can be scanned by the medical scanning apparatuses to obtain corresponding medical images. With the development of the technology, some scenes for relevant application of medical images have appeared, such as segmentation of the medical images to assist medical teaching or medical recognition, etc.

The traditional method for segmenting the medical image usually adopts an image value segmentation method, but because the scanning resolution of medical equipment is too low, the axial resolution of the medical image is low and an obvious fault is often presented, when the medical image is directly segmented by the traditional image value segmentation method, the obtained image also has the phenomenon of the fault, the quality of the segmented image is not good enough, and the requirement of visualization is difficult to meet.

Disclosure of Invention

In view of the foregoing, it is necessary to provide a method, an apparatus, a computer device, a computer readable storage medium and a computer program product for reconstructing a critical structure based on three-dimensional images, and a method, an apparatus, a computer device, a storage medium and a computer program product for training an image processing model.

In one aspect, the application provides a key structure reconstruction method based on a three-dimensional image. The method comprises the following steps:

carrying out pre-sampling treatment on an original three-dimensional medical image of a target part to obtain a three-dimensional medical image with a specified size, wherein the target part comprises at least one key structure;

performing semantic segmentation processing on the three-dimensional medical image with the specified size to obtain a corresponding three-dimensional labeled image, wherein each voxel in the three-dimensional labeled image corresponds to a probability value belonging to each key structure;

carrying out post-sampling processing based on the three-dimensional marked image to obtain a target marked image meeting a high-resolution condition;

segmenting the target marked image based on the probability value corresponding to each voxel in the three-dimensional marked image to obtain at least one three-dimensional structural image, wherein each three-dimensional structural image corresponds to one key structure;

and respectively reconstructing each three-dimensional structure image to obtain a target three-dimensional structure model respectively corresponding to each key structure, wherein the target three-dimensional structure model is used for carrying out visual display.

On the other hand, the application also provides a key structure reconstruction device based on the three-dimensional image. The device comprises:

the system comprises a pre-sampling module, a pre-sampling module and a pre-sampling module, wherein the pre-sampling module is used for performing pre-sampling processing on an original three-dimensional medical image of a target part to obtain a three-dimensional medical image with a specified size, and the target part comprises at least one key structure;

the semantic segmentation module is used for performing semantic segmentation processing on the three-dimensional medical image with the specified size to obtain a corresponding three-dimensional labeled image, and each voxel in the three-dimensional labeled image corresponds to a probability value belonging to each key structure;

the post-sampling module is used for performing post-sampling processing on the basis of the three-dimensional marked image to obtain a target marked image meeting a high-resolution condition;

the structure segmentation module is used for segmenting the target marked image based on the probability value corresponding to each voxel in the three-dimensional marked image to obtain at least one three-dimensional structure image, and each three-dimensional structure image corresponds to one key structure;

and the three-dimensional reconstruction module is used for respectively reconstructing each three-dimensional structure image to obtain a target three-dimensional structure model respectively corresponding to each key structure, and the target three-dimensional structure model is used for carrying out visual display.

On the other hand, the application also provides computer equipment. The computer equipment comprises a memory and a processor, wherein the memory stores a computer program, and the processor realizes the steps of the key structure reconstruction method based on the three-dimensional image when executing the computer program.

In another aspect, the present application also provides a computer-readable storage medium. The computer readable storage medium has a computer program stored thereon, and the computer program, when executed by a processor, implements the steps of the above-mentioned three-dimensional image-based key structure reconstruction method.

In another aspect, the present application also provides a computer program product. The computer program product comprises a computer program, and the computer program realizes the steps of the above-mentioned key structure reconstruction method based on three-dimensional images when being executed by a processor.

According to the three-dimensional image-based key structure reconstruction method, the device, the computer equipment, the storage medium and the computer program product, the original three-dimensional medical image of the target part is subjected to pre-sampling processing, so that the multi-modal original three-dimensional medical images with different thicknesses and sizes are adjusted to the three-dimensional medical image with the specified size, a corresponding three-dimensional mark image is obtained through semantic segmentation processing, each voxel in the three-dimensional mark image corresponds to a probability value belonging to each key structure, and each probability value can be used as a basis for subsequently segmenting at least one key structure. The post-sampling processing is carried out on the basis of the three-dimensional labeled image to obtain the target labeled image meeting the high-resolution condition, so that the precision of subsequent segmentation is improved, the target labeled image is segmented on the basis of the probability value corresponding to each voxel in the three-dimensional labeled image, and each key structure can be accurately segmented from the three-dimensional medical image. And finally, reconstructing each three-dimensional structure image, wherein the obtained target three-dimensional structure model can accurately represent a corresponding key structure, has high resolution, eliminates the phenomenon of fault and meets the requirement of visualization.

On the other hand, the application also provides a training method of the image processing model. The method comprises the following steps:

acquiring original three-dimensional medical image samples of a target part and structural labels respectively corresponding to the original three-dimensional medical image samples;

carrying out pre-sampling processing on the original three-dimensional medical image sample to obtain a three-dimensional medical image sample with a specified size;

semantic segmentation processing is carried out on the three-dimensional medical image sample with the specified size through an image processing model to be trained to obtain a predicted three-dimensional labeled image, and each voxel in the predicted three-dimensional labeled image corresponds to a predicted probability value belonging to each key structure;

constructing a target loss function based on the difference between the predicted three-dimensional marked image and the structural label corresponding to the corresponding original three-dimensional medical image sample;

and adjusting model parameters of the image processing model based on the target loss function, continuing model training based on the image processing model after the model parameters are adjusted until a training stopping condition is reached, and obtaining the trained image processing model.

On the other hand, the application also provides a training device of the image processing model. The device comprises:

the acquisition module is used for acquiring original three-dimensional medical image samples of the target part and structural labels respectively corresponding to the original three-dimensional medical image samples;

the pre-sampling module is used for performing pre-sampling processing on the original three-dimensional medical image sample to obtain a three-dimensional medical image sample with a specified size;

the prediction module is used for carrying out semantic segmentation processing on the three-dimensional medical image sample with the specified size through an image processing model to be trained to obtain a predicted three-dimensional labeled image, and each voxel in the predicted three-dimensional labeled image corresponds to a prediction probability value belonging to each key structure;

the construction module is used for constructing a target loss function based on the difference between the predicted three-dimensional marked image and the structural label corresponding to the corresponding original three-dimensional medical image sample;

and the parameter adjusting module is used for adjusting the model parameters of the image processing model based on the target loss function, continuing model training based on the image processing model after the model parameters are adjusted until a training stopping condition is reached, and obtaining the trained image processing model.

On the other hand, the application also provides computer equipment. The computer device comprises a memory and a processor, wherein the memory stores a computer program, and the processor realizes the steps of the training method of the image processing model when executing the computer program.

In another aspect, the present application also provides a computer-readable storage medium. The computer-readable storage medium, on which a computer program is stored, which, when being executed by a processor, carries out the steps of the above-mentioned training method of the image processing model.

In another aspect, the present application also provides a computer program product. The computer program product comprises a computer program which, when being executed by a processor, carries out the steps of the method for training an image processing model as described above.

According to the training method, the training device, the computer equipment, the storage medium and the computer program product of the image processing model, the original three-dimensional medical image samples with the structural labels are obtained, pre-sampling processing is firstly carried out, and the original three-dimensional medical image samples with different sizes and thicknesses are uniformly converted into the three-dimensional medical image samples with the specified sizes so as to meet the input requirements of a subsequent image processing model; performing semantic segmentation processing on the three-dimensional medical image sample with the specified size through an image processing model to be trained to obtain a predicted three-dimensional labeled image, and constructing a target loss function according to the difference between the structure labels corresponding to the predicted three-dimensional labeled image and the corresponding original three-dimensional medical image sample, so that the training process of the image processing model is performed towards the direction of minimizing the target loss function, and the accuracy of the trained image processing model meets the requirement; and in the training process, continuously adjusting model parameters of the image processing model based on the target loss function, and stopping training until a training stopping condition is reached, thereby obtaining the trained image processing model. The image processing model trained in the mode is high in accuracy, high-resolution image segmentation can be achieved, and then the accuracy of a follow-up reconstructed three-dimensional structure model can be guaranteed.

Drawings

FIG. 1 is a diagram illustrating an exemplary application environment of a method for reconstructing a critical structure based on three-dimensional images;

FIG. 2 is a schematic flowchart illustrating a method for reconstructing a critical structure based on three-dimensional images according to an embodiment;

FIG. 3 is a diagram showing residual blocks in 3D U-Net according to an embodiment;

FIG. 4A is a flowchart illustrating the steps of segmenting a target marker image in one embodiment;

FIG. 4B is a schematic representation of multi-channel probability values in one embodiment;

FIG. 5 is a flowchart illustrating steps of a reconstruction process for a three-dimensional structure image according to an embodiment;

FIG. 6 is a flow chart illustrating the steps of registration in one embodiment;

FIG. 7 is a flowchart illustrating the steps of visualization in one embodiment;

FIG. 8 is a flowchart illustrating the steps of training an image processing model in one embodiment;

FIG. 9 is a flowchart illustrating the labeling of a sample of an original three-dimensional medical image according to one embodiment;

FIG. 10 is a schematic diagram of the steps of filling of a sample of an original three-dimensional medical image in one embodiment;

FIG. 11 is a schematic view of an application scenario of a three-dimensional image-based key structure reconstruction method in an embodiment;

FIG. 12A is a schematic view of the overall flow in one embodiment;

FIG. 12B is a schematic view showing the whole flow in another embodiment;

FIG. 12C is a diagram illustrating the segmentation and reconstruction effects in one embodiment;

FIG. 12D is a diagram illustrating the segmentation and reconstruction effects in another embodiment;

FIG. 13 is a flowchart illustrating a method for training an image processing model according to one embodiment;

FIG. 14 is a flowchart illustrating the labeling of a sample of an original three-dimensional medical image according to one embodiment;

FIG. 15 is a block diagram of an embodiment of a three-dimensional image based critical structure reconstruction apparatus;

FIG. 16 is a block diagram showing the construction of an image processing model training apparatus according to an embodiment;

FIG. 17 is a diagram illustrating an internal structure of a computer device according to an embodiment.

Detailed Description

In order to make the objects, technical solutions and advantages of the present application more apparent, the present application is described in further detail below with reference to the accompanying drawings and embodiments. It should be understood that the specific embodiments described herein are merely illustrative of the present application and are not intended to limit the present application.

In a conventional image reconstruction method, an image of a corresponding tissue or organ portion is often obtained by using a threshold segmentation method. In an actual clinical scene, due to the fact that the scanning resolution of medical equipment is too low, the axial resolution of a medical image is low, obvious faults are displayed, and the accuracy of a reconstructed image is low.

In an actual clinical scene, the scanning resolution of the medical scanning equipment is low, and a high-resolution three-dimensional medical image is difficult to obtain. The three-dimensional medical image is formed by overlapping multiple slices (Slice), and can be divided into a thick-layer image and a thin-layer image according to different hardware specifications of the scanning probe. The thick-layer image has a relatively large thickness and relatively small total slices, so that the thick-layer image presents an obvious fault situation, an accurate key structure cannot be reconstructed according to the fault situation, registration accuracy in a medical treatment process is seriously affected, and the segmentation and visualization effects of the key structure are extremely poor. The thin layer image has a thinner layer thickness and a larger number of total slices than the thick layer image, but the thin layer image is limited by the scanning principle of the medical scanning equipment, and the situation that the thin layer image is a fault cannot be avoided.

Therefore, the embodiment of the application provides a method, a device, a computer device, a storage medium and a computer program product for reconstructing a key structure based on a three-dimensional image based on an artificial intelligence technology, combines a computer vision technology and a machine learning technology, encodes a discretely encoded three-dimensional medical image into a continuous Hidden Space (Hidden Space) through an end-to-end three-dimensional convolutional neural network based on image continuity analysis, outputs a high-resolution three-dimensional structure image in a decoding stage, and reconstructs the three-dimensional structure image to obtain an accurate three-dimensional structure model.

Among them, Artificial Intelligence (AI) is a theory, method, technique and application system that simulates, extends and expands human Intelligence using a digital computer or a machine controlled by a digital computer, senses the environment, acquires knowledge and uses the knowledge to obtain the best result. In other words, artificial intelligence is a comprehensive technique of computer science that attempts to understand the essence of intelligence and produce a new intelligent machine that can react in a manner similar to human intelligence. Artificial intelligence is the research of the design principle and the realization method of various intelligent machines, so that the machines have the functions of perception, reasoning and decision making.

The artificial intelligence technology is a comprehensive subject and relates to the field of extensive technology, namely the technology of a hardware level and the technology of a software level. The artificial intelligence base technologies generally include technologies such as sensors, dedicated artificial intelligence chips, cloud computing, distributed storage, big data processing technologies, operation/interaction systems, mechatronics, and the like. The artificial intelligence software technology mainly comprises a computer vision technology, a voice processing technology, a natural language processing technology, machine learning/deep learning and the like.

Computer Vision technology (CV) is a science for researching how to make a machine "see", and more specifically, it refers to that a camera and a Computer are used to replace human eyes to perform machine Vision such as identification, follow-up and measurement on a target, and further perform graphic processing, so that the Computer processing becomes an image more suitable for human eyes to observe or transmit to an instrument to detect. As a scientific discipline, computer vision research-related theories and techniques attempt to build artificial intelligence systems that can capture information from images or multidimensional data. The computer vision technology generally includes technologies such as image processing, image Recognition, image semantic understanding, image retrieval, OCR (Optical Character Recognition), video processing, video semantic understanding, video content/behavior Recognition, three-dimensional object reconstruction, 3D technology, virtual reality, augmented reality, synchronous positioning, map construction, and the like, and also includes common biometric technologies such as face Recognition, fingerprint Recognition, and the like.

Machine Learning (ML) is a multi-domain cross subject, and relates to multiple subjects such as probability theory, statistics, approximation theory, convex analysis and algorithm complexity theory. The special research on how a computer simulates or realizes the learning behavior of human beings so as to acquire new knowledge or skills and reorganize the existing knowledge structure to continuously improve the performance of the computer. Machine learning is the core of artificial intelligence, is the fundamental approach for computers to have intelligence, and is applied to all fields of artificial intelligence. Machine learning and deep learning generally include techniques such as artificial neural networks, belief networks, reinforcement learning, transfer learning, inductive learning, and formal learning.

With the research and progress of artificial intelligence technology, the artificial intelligence technology is developed and applied in a plurality of fields, such as common smart homes, smart wearable devices, virtual assistants, smart speakers, smart marketing, unmanned driving, automatic driving, unmanned aerial vehicles, robots, smart medical care, smart customer service, and the like.

The following describes technical schemes related to an artificial intelligence three-dimensional image-based key structure reconstruction method, an image processing model training method and the like provided by the embodiment of the application.

The method for reconstructing a key structure based on a three-dimensional image, provided by the embodiment of the application, can be applied to an application environment as shown in fig. 1. Wherein the terminal 102 communicates with the server 104 via a network. The data storage system may store data that the server 104 needs to process. The data storage system may be integrated on the server 104, or may be placed on the cloud or other server. In some embodiments, the terminal 102 or the server 104 sequentially performs pre-sampling processing, semantic segmentation processing, and post-sampling processing on an original three-dimensional medical image of a target portion, which is locally stored or acquired through a network, and performs segmentation to obtain at least one three-dimensional structural image, and performs reconstruction processing on each three-dimensional structural image to obtain a target three-dimensional structural model corresponding to each key structure.

In other embodiments, the server 104 obtains an original three-dimensional medical image transmitted by the terminal 102 or acquired through a network, sequentially performs pre-sampling processing, semantic segmentation processing, post-sampling processing, and segmentation to obtain at least one three-dimensional structural image, and performs reconstruction processing on each three-dimensional structural image to obtain a target three-dimensional structural model corresponding to each key structure. The server 104 may transmit the target three-dimensional structure model to the terminal 102 for visual display by the terminal 102.

The three-dimensional structure model of the target obtained by the terminal 102 or the server 104 may be visually displayed on the terminal 102 or by other devices, such as a medical scanning device externally connected with a display device, a separate display device, and the like. The medical scanning device is, for example, one or more of a CT (Computed Tomography) machine, a nuclear magnetic resonance apparatus, an ultrasound scanner, and the like. The display device includes, but is not limited to, a display screen, a projection device, or a virtual reality imaging device, and the display screen may be one or more of a liquid crystal display screen, an electronic ink display screen, or the like.

The terminal 102 may be, but not limited to, various personal computers, notebook computers, smart phones, tablet computers, internet of things devices, and portable wearable devices, where the internet of things devices may be one or more of smart speakers, smart televisions, smart air conditioners, and smart car-mounted devices. The portable wearable device may be one or more of a smart watch, a smart bracelet, and a head-mounted device, among others.

The server 104 may be an independent physical server, a server cluster or a distributed system formed by a plurality of physical servers, or a cloud server providing basic cloud computing services such as a cloud service, a cloud database, cloud computing, a cloud function, cloud storage, a Network service, cloud communication, a middleware service, a domain name service, a security service, a CDN (Content Delivery Network), a big data and artificial intelligence platform, and the like.

In some embodiments, as shown in fig. 2, a method for reconstructing a critical structure based on a three-dimensional image is provided, and the method may be applied to a terminal or a server, or may be executed by the terminal and the server in cooperation. The method is applied to computer equipment as an example, and comprises the following steps:

step S202, the original three-dimensional medical image of the target part is subjected to pre-sampling processing to obtain a three-dimensional medical image with a specified size, and the target part comprises at least one key structure.

The original three-dimensional image is obtained by acquiring a medical image of a target portion of a target object based on a medical scanning device, and includes multiple modalities, such as but not limited to one or more of a CT image, an MRI (Magnetic resonance Imaging) image, and an ultrasound image. Wherein the target object may be a human body, an animal body, or a non-biological object; the target site is a site area on the target object. Taking a human body or an animal body as an example, the target part can be one or more of a head part, a neck part, a trunk part, four limbs and the like; for non-living objects, the target site may be a portion or all of the area of the non-living object.

Wherein, the critical structure refers to a critical anatomical structure that can be applied for registration and visualization during medical treatment, including but not limited to one or more of skin, brain, skull, ventricle, femur, and lesion, etc. The target site may include one or more critical structures. Taking the target part as the head as an example, the target part comprises skin, brain, skull, ventricles of brain, and the like.

Due to the hardware limitation of the medical scanning equipment, the image size and resolution of the original three-dimensional medical image obtained by the computer equipment are usually different, and if the original three-dimensional medical image is not preprocessed, the calculation amount is greatly increased, and the accuracy of the subsequent three-dimensional structural image and the three-dimensional structural model is influenced. Therefore, the original three-dimensional medical images with different specifications can be converted to the specified size through the pre-sampling processing so as to adapt to the original three-dimensional medical images with different thicknesses and sizes and to adapt to the input requirements of the subsequent processing.

Specifically, the computer device performs pre-sampling processing on the original three-dimensional medical image of the target part to obtain a three-dimensional medical image with a specified size. The pre-sampling process may be implemented by means of interpolation, and the interpolation process includes, but is not limited to, one or more of linear interpolation, bilinear interpolation, cubic curve interpolation, and the like.

Illustratively, the computer device adjusts the size of the acquired original three-dimensional medical image to a specified size of 176 × 176 × 176 by linear interpolation, the specified size of the three-dimensional medical image being an input of the subsequent neural network.

Step S204, performing semantic segmentation processing on the three-dimensional medical image with the specified size to obtain a corresponding three-dimensional labeled image, wherein each voxel in the three-dimensional labeled image corresponds to a probability value belonging to each key structure.

The semantic segmentation processing is to classify each voxel in the three-dimensional medical image so as to obtain different semantic interpretable categories corresponding to each voxel, wherein the semantic interpretable categories are used for representing at least one key structure to which each voxel belongs respectively.

Specifically, the computer device inputs a three-dimensional medical image with a specified size into a pre-trained three-dimensional convolutional neural network, the three-dimensional convolutional neural network sequentially performs encoding processing (Encode) and decoding processing (Decode), low-order image features in the three-dimensional medical image are obtained through the encoding processing, the low-order image features are converted into high-order image features through the decoding processing, and finally, a three-dimensional labeled image corresponding to the three-dimensional medical image is output. Wherein each voxel in the three-dimensional labeled image corresponds to a probability value belonging to each key structure, for example, for a voxel in the three-dimensional labeled image, assuming that three key structures A, B and C are included in the three-dimensional medical image, the voxel corresponds to three probability values belonging to the key structures, for example, (0.1,0.8,0.3), where 0.1 represents the probability value that the voxel belongs to the key structure a, 0.8 represents the probability value that the voxel belongs to the key structure B, and 0.3 represents the probability value that the voxel belongs to the key structure C.

The three-dimensional Convolutional Neural network is a three-dimensional Convolutional Neural Network (CNN) and comprises an input layer, at least one hidden layer and at least one output layer. The input layer is used for receiving a three-dimensional medical image with a specified size; the hidden layer comprises a convolution layer and an activation function layer, and even can comprise at least one of a normalization layer, a pooling layer and a fusion layer; the output layer is used for outputting a feature map containing image feature information. The connection mode of each layer is determined according to the connection relation of each layer in the neural network structure. For example, a connection relationship between the front layer and the back layer set based on data transfer, a connection relationship between the front layer and the back layer and full connection are set based on the convolutional kernel size in each hidden layer, and the like.

Illustratively, in the embodiment of the present application, the encoding process and the decoding process are performed by using 3D U-Net (a three-dimensional convolutional neural network having a U-shaped structure). The U-shaped structure of 3D U-Net includes a downsampling path and an upsampling path, each path including 4 identical modules. The downsampling Block consists of a Residual Block (Residual Block) as shown in fig. 3 and a convolutional layer with step size of 2. The residual block shown in fig. 3 includes, for example, two identical blocks, which include an active layer (RELU), a Convolution layer (Conv) having a size of 3 × 3 × 3, and a Normalization layer (BN). The residual module adopts a residual connection mode between the convolution layers. The up-sampling module consists of up-sampling layers in jump connection, and the feature maps with the same scale can be fused into feature maps with stronger semantic information through the convolutional layers and the up-sampling layers.

In the process of semantic segmentation processing, a bed board of the target part in the image during scanning, attachments for fixing the target part and the like can be removed through semantic segmentation, so that the influence of irrelevant objects on the accuracy of the three-dimensional structure image and the three-dimensional structure model can be avoided.

And step S206, carrying out post-sampling processing based on the three-dimensional mark image to obtain a target mark image meeting a high-resolution condition.

In order to further improve the resolution of the image and enable a subsequently reconstructed three-dimensional structure model to be more accurate, on the basis that the three-dimensional convolutional neural network encoding processing and decoding processing output a three-dimensional marked image, the computer equipment further performs post-sampling processing on the three-dimensional marked image so as to improve the resolution of the image.

Specifically, the computer device performs post-sampling processing on the three-dimensional marker image based on the three-dimensional marker image obtained through the three-dimensional convolutional neural network to obtain a three-dimensional marker image satisfying a high resolution condition, namely, a target marker image. The post-sampling process may be implemented by means of interpolation, and the interpolation process includes, but is not limited to, one or more of linear interpolation, bilinear interpolation, cubic curve interpolation, and the like.

Wherein the resolution of the target mark image is higher than the resolution of the three-dimensional medical image with the specified size. Illustratively, the computer device performs an up-sampling process through a post-sampling layer, and interpolates the three-dimensional marker image to a physical resolution of 1mm by means of linear interpolation, thereby satisfying a high resolution condition. The high resolution condition may be determined according to actual requirements, for example, a value specifically defining the resolution, and the like.

And S208, segmenting the target marked image based on the probability value corresponding to each voxel in the three-dimensional marked image to obtain at least one three-dimensional structural image, wherein each three-dimensional structural image corresponds to one key structure.

Specifically, after the post-sampling processing, the computer device segments the high-resolution target marker image, segments voxels belonging to the same key structure in the target marker image, and thereby obtains a three-dimensional structure image including the key structure. When the three-dimensional medical image contains a plurality of key structures, the computer equipment obtains a plurality of three-dimensional structure images containing a single key structure. Illustratively, the computer device may segment voxels belonging to the same critical structure in the target marker image by means of thresholding.

In some embodiments, the post-sampling processing step may also be performed after the step of segmenting the target marker image, that is, performing interpolation processing on each three-dimensional structure image separately to improve the resolution of the three-dimensional structure image.

And step S210, respectively reconstructing each three-dimensional structure image to obtain target three-dimensional structure models respectively corresponding to the key structures, wherein the target three-dimensional structure models are used for visual display.

The reconstruction processing refers to a processing procedure of reconstructing a key structure in the three-dimensional structure image into a visualized three-dimensional structure model.

Specifically, the computer equipment determines a key structure to be reconstructed from the three-dimensional structure image, namely a target key structure, and extracts all corresponding voxels of the target key structure in the three-dimensional structure image; and the computer equipment constructs the surface of the target key structure according to the voxels to obtain a target three-dimensional structure model corresponding to the target key structure. The target three-dimensional structure model obtained through reconstruction can be displayed visually in preoperative registration, intraoperative navigation and other scenes. The visual display includes, but is not limited to, one or more of a graphic display, a VR (Virtual Reality) display, an AR (Augmented Reality) display, and an MR (Mixed Reality) display.

According to the key structure reconstruction method based on the three-dimensional image, the original three-dimensional medical image of the target part is subjected to pre-sampling processing, so that the multi-modal original three-dimensional medical images with different thicknesses and sizes are adjusted to the three-dimensional medical image with the specified size, a corresponding three-dimensional marking image is obtained through semantic segmentation processing, each voxel in the three-dimensional marking image corresponds to a probability value belonging to each key structure, and each probability value can be used as a basis for subsequently segmenting at least one key structure. The post-sampling processing is carried out on the basis of the three-dimensional labeled image to obtain the target labeled image meeting the high-resolution condition, so that the precision of subsequent segmentation is improved, the target labeled image is segmented on the basis of the probability value corresponding to each voxel in the three-dimensional labeled image, and each key structure can be accurately segmented from the three-dimensional medical image. And finally, reconstructing each three-dimensional structural image, wherein the obtained target three-dimensional structural model can accurately represent a corresponding key structure, has high resolution, eliminates the fault phenomenon and meets the requirement of visualization.

The scheme of the application can be adapted to multi-modal original three-dimensional medical images, the application range is wide, the discretely coded three-dimensional medical images can be coded into a continuous hidden space based on the continuity analysis of the images, and then the three-dimensional structural images with high resolution are output, so that the three-dimensional structural model obtained by reconstruction is more accurate, and the three-dimensional effect can be well presented. The method can obtain the three-dimensional structural image with high resolution, can also carry out fine and accurate key structure reconstruction, and reduces the error of subsequent registration. Moreover, since the whole architecture is end-to-end, the whole architecture can be executed on a GPU (Graphics Processing Unit), and the computing resources of a Central Processing Unit (CPU) are saved.

Due to different specifications of medical scanning equipment, different hardware conditions of the probe and different configured scanning intervals, the thickness and the size of original three-dimensional medical images obtained by scanning are different. To this end, in some embodiments, the pre-sampling the original three-dimensional medical image of the target region to obtain a three-dimensional medical image with a specified size includes: acquiring an original three-dimensional medical image obtained by acquiring a medical image of a target part of a target object; and carrying out interpolation processing on the original three-dimensional medical image through a front sampling layer to obtain a three-dimensional medical image with a specified size.

Specifically, the medical scanning device performs medical image acquisition on a target part of a target object to obtain an original three-dimensional medical image corresponding to the target part. The computer equipment acquires the original three-dimensional medical image transmitted by the medical scanning equipment, and performs pre-sampling processing on the original three-dimensional medical image of the target part through a pre-sampling layer which is arranged in advance according to output size information configured by the pre-sampling layer to obtain a three-dimensional medical image with a specified size specified by the output size information.

The pre-sampling process may be implemented by means of interpolation, and the interpolation process includes, but is not limited to, one or more of linear interpolation, bilinear interpolation, cubic curve interpolation, and the like. Illustratively, the computer device adjusts the size of the acquired original three-dimensional medical image to a specified size of 176 × 176 × 176 by linear interpolation, the specified size of the three-dimensional medical image being an input of the subsequent neural network.

In the above embodiment, interpolation processing is performed on the original three-dimensional medical image through the front sampling layer, the three-dimensional medical image interpolation processing method can be adapted to original three-dimensional medical images of different thicknesses and different sizes acquired by various medical scanning devices, the application range is wider, and through the front sampling processing, the subsequent probability value is more accurate, and then the reconstructed three-dimensional structure model is more accurate.

As mentioned above, the computer device performs encoding and decoding processing by using a pre-trained three-dimensional convolutional neural network, in some embodiments, the pre-trained three-dimensional convolutional neural network includes two parts, namely an encoding network and a decoding network, and accordingly performs semantic segmentation processing on the three-dimensional medical image with the specified size to obtain a corresponding three-dimensional labeled image, including: extracting a plurality of continuous characteristic vectors in the three-dimensional medical image with the specified size through a coding network; fusing the continuous feature vectors through a decoding network to obtain a feature map of the three-dimensional medical image corresponding to the specified size; and normalizing the feature map, converting the normalized feature map through an activation layer, and outputting a three-dimensional marking image.

Specifically, the computer device inputs the three-dimensional medical image with the specified size obtained after pre-sampling into a coding network, performs down-sampling through the coding network, codes each voxel in the three-dimensional medical image into a continuous feature vector space, and obtains a plurality of continuous feature vectors, wherein the plurality of continuous feature vectors form a feature map. And then, inputting the obtained feature map into a decoding network by the computer equipment, carrying out up-sampling by the decoding network, restoring the feature map to the specified size when the three-dimensional medical image is input into the coding network, and fusing a plurality of feature vectors to obtain the feature map with stronger semantic feature information. And finally, the computer equipment normalizes the feature map, converts the normalized feature map through an activation layer, acquires the probability value of each key structure corresponding to each voxel in the three-dimensional medical image with the specified size, and outputs the three-dimensional label image carrying the information of the probability value.

Illustratively, the computer device normalizes the feature map at the specified resolution by calculating a mean, a standard deviation, and the like of each voxel in the feature map. For example, the formula for the normalization process can be as follows:

wherein, I _in Is the value of the voxel in the input feature map, min (I) _in ) Minimum of voxels, max (I) _in ) Is the maximum value of the voxel, I _in The values of the voxels in the feature map output after the normalization processing are obtained.

After the normalization processing, the computer device converts the normalized feature map through the activation layer to output a three-dimensional label image including information having probability values.

In the embodiment, continuous feature vectors are extracted through the coding network and the decoding network, feature fusion is carried out, based on image continuity analysis, the discretely coded three-dimensional medical image is coded into a continuous hidden space through an end-to-end three-dimensional convolutional neural network, a high-resolution three-dimensional marked image is output in a decoding stage, accurate semantic segmentation is provided, and the accuracy of a subsequent three-dimensional structure model reconstruction is further ensured.

In some embodiments, as shown in fig. 4A, segmenting the target labeled image based on the probability value corresponding to each voxel in the three-dimensional labeled image to obtain at least one three-dimensional structure image, includes:

step S402, determining probability threshold values respectively corresponding to the key structures;

step S404, comparing the probability value of each voxel in the three-dimensional label image belonging to the corresponding key structure with the probability threshold corresponding to the corresponding key structure for each key structure, and determining a target voxel matched with the corresponding key structure from the target label image based on the comparison result;

and step S406, segmenting the target voxels respectively corresponding to each key structure to obtain at least one three-dimensional structure image.

Specifically, the computer device obtains a plurality of probability thresholds that are preset corresponding to respective critical structures. For each voxel in the three-dimensional labeled image, according to the probability value corresponding to each key structure corresponding to the voxel, the computer equipment compares the probability value with a probability threshold, and according to the numerical comparison result of the probability value and the probability threshold, the voxel meeting the threshold condition is determined as a target voxel belonging to the key structure corresponding to the threshold condition. For example, if the probability value is greater than (or less than) a preset threshold, the voxel is determined as a target voxel belonging to the corresponding key structure. For each voxel, the computer device performs the above processing, thereby determining all target voxels corresponding to each key structure.

For example, assuming that there are three key structures, each voxel in the three-dimensional labeled image has three probability values, e.g., (0.1,0.8,0.3), where 0.1 represents the probability value that the voxel belongs to key structure a, 0.8 represents the probability value that the voxel belongs to key structure B, and 0.3 represents the probability value that the voxel belongs to key structure C. According to the preset probability threshold value of each key structure being 0.7, the computer equipment can determine that the voxel belongs to the key structure B because the probability value corresponding to the key structure B in the probability values corresponding to the voxel is greater than the probability threshold value. If a plurality of probability values corresponding to the voxel are larger than the probability threshold, the computer device determines that the voxel belongs to a plurality of corresponding key structures.

Thus, for each key structure, the computer device segments all target voxels corresponding to the key structure to obtain a three-dimensional structure image corresponding to the key structure; when the three-dimensional medical image comprises a plurality of key structures, the computer equipment can be used for segmenting to obtain a plurality of three-dimensional structure images.

In some embodiments, the three-dimensional marker image includes a plurality of channels therein, each channel representing a critical structure. Accordingly, each voxel in the three-dimensional label image carries a probability value under each channel. Illustratively, as shown in fig. 4B, each channel in the three-dimensional marker image corresponds to a key structure, for example, channel a corresponds to skin, channel B corresponds to brain, and channel C corresponds to skull. For one of the voxels, respectively, the probability value of the respective key structure belonging to each voxel under each channel, as is also shown in fig. 4B, for example, the probability value in the probability map corresponding to the skin is 0.1, the probability value in the probability map corresponding to the brain is 0.8, and the probability value in the probability map corresponding to the skull is 0.06.

In the above embodiment, by setting the probability threshold and segmenting the three-dimensional labeled image obtained by semantic segmentation, all voxels belonging to the same key structure can be rapidly and accurately segmented from the original image to obtain the three-dimensional structure image only including the key structure, so that a visual graph for a single key structure can be provided, and accordingly, the key structure can be reconstructed in a targeted manner, and the influence of other key structures on the accuracy of the reconstructed three-dimensional structure model is avoided.

In some embodiments, as shown in fig. 5, the reconstructing process is performed on each three-dimensional structure image to obtain a three-dimensional structure model corresponding to each critical structure, and the reconstructing process includes:

step S502, for each three-dimensional structure image, establishing an initial three-dimensional structure model matched with a key structure contained in the three-dimensional structure image, wherein the surface of the key structure in the initial three-dimensional structure model is composed of a plurality of patches in preset shapes.

Step S504, the surfaces of the key structures in the initial three-dimensional structure models are smoothed, and target three-dimensional structure models corresponding to the key structures are obtained.

Specifically, for the three-dimensional structure image obtained by segmentation, the computer device performs surface modeling according to the three-dimensional structure image, and forms the surface of the key structure in the three-dimensional structure image through a plurality of patches with preset shapes to obtain an initial three-dimensional structure model of the key structure. The preset shape includes, but is not limited to, one or more of a triangle, a rectangle, a hexagon, an octagon, and the like. Illustratively, the computer device converts the three-dimensional segmented image into an initial three-dimensional structure model by a Discrete Marching Cube function in a VTK (Visualization Toolkit).

Because the data volume is usually large, in order to improve the smoothness, the computer equipment performs smoothing treatment on the surfaces of the key structures in the structure model to obtain the target three-dimensional structure models respectively corresponding to the key structures. Illustratively, the computer device may filter through a Windowed Sinc (window filtering) function in the VTK to smooth the surface of the initial three-dimensional structure model.

In order to increase the calculation speed, the number of patches needs to be controlled to a certain number at maximum, so as to keep a fixed calculation time length. Different number thresholds may be set for different critical structures. For example, since the skin is available for subsequent registration, requiring higher accuracy, the number of patches of the three-dimensional structural model corresponding to the skin can be set to 2000, and other critical structures such as bones and the like can be set to a smaller number according to the visualization requirements.

In the above embodiment, the three-dimensional structure image is modeled, the image is converted into a further visualized three-dimensional structure model, and the three-dimensional structure model is finer through a surface patch and smooth filtering, so that each key structure in the target part can be accurately represented, and the accuracy in the following scenes such as preoperative registration and preoperative navigation is higher.

After the target three-dimensional structure model is obtained, the target three-dimensional structure model may be applied in the registered scene. For example, to provide accurate navigation or achieve accurate virtual reality rendering during a medical procedure, images of a real scene acquired during the medical procedure need to be registered. Registration refers to registration of the surface layers (or surfaces). In some embodiments, the critical structures of the target site include surface structures and internal structures. Wherein, for the organism, the surface structure is skin, and the internal structure is brain, femur, etc. covered by skin; for non-living organisms, the surface structure is, for example, the outer surface (or outer skin) and the internal structure is, for example, a specific structure of the interior. Accordingly, in some embodiments, as shown in fig. 6, the method further comprises:

step S602, acquiring a real scene image obtained by image-capturing a target portion in a real scene, and determining real surface point cloud data corresponding to the real scene image.

Step S604, converting the target three-dimensional structure model corresponding to the surface structure into reconstructed surface point cloud data, and registering the reconstructed surface point cloud data and the real surface point cloud data.

The real scene image is a three-dimensional image obtained by scanning a target region in a real scene by a medical scanning apparatus or other imaging devices, such as a point cloud scanner and a depth camera.

Specifically, the computer device acquires a real scene image and converts the real scene image into real surface point cloud data. For the target three-dimensional structure model, the computer equipment obtains reconstructed surface point cloud data corresponding to the surface structure by determining the connection point of each patch in the target three-dimensional structure model and removing the patch part. The computer equipment registers the reconstructed surface point cloud data and the real surface point cloud data through rigid body transformation (including translation, rotation and mirror inversion), determines the coordinate value of each voxel in the reconstructed surface point cloud data and the mapping relation between the coordinate values of each voxel in the real surface point cloud data, and accordingly aligns the surface layer position in the real scene with the surface layer structure in the target three-dimensional structure model.

Illustratively, for the target three-dimensional structure model, an ICP (Iterative Closest Point) algorithm may be used to perform rigid transformation and non-rigid transformation to establish a mapping relationship of a coordinate system between reconstructed surface Point cloud data obtained by converting the target three-dimensional structure model and real surface Point cloud data, so as to determine a relative position relationship of the key structure.

In the above embodiment, the reconstructed surface point cloud data obtained by converting the target three-dimensional structure model is used, and the real surface point cloud data obtained by converting the real scene image is registered and aligned, so that the method has higher precision and better registration effect.

After the registration is completed, in some embodiments, as shown in fig. 7, the above method further includes the following steps to achieve visualization of the target three-dimensional structural model:

step S702, after the registration of the reconstructed surface point cloud data and the real surface point cloud data is completed, a target three-dimensional structure model corresponding to each internal structure of the target part and the relative position of the target three-dimensional structure model in the real scene image are determined based on the registration result.

Step S704, displaying the target three-dimensional structure model corresponding to each internal structure in the real scene image in an overlapping manner according to the relative position of the target three-dimensional structure model corresponding to each internal structure in the real scene image.

Specifically, after the registration of the reconstructed surface point cloud data and the real surface point cloud data is completed, the computer device determines a mapping relation between a coordinate value of each voxel in the reconstructed surface point cloud data and a coordinate value of each voxel in the real surface point cloud data based on the registration result, so as to determine the relative position of the reconstructed surface point cloud data corresponding to the target three-dimensional structure model corresponding to each internal structure in the real scene image. Therefore, according to the relative positions of the target three-dimensional structure models corresponding to the internal structures in the real scene image, the computer equipment can display the target three-dimensional structure models corresponding to the internal structures in the real scene image in an overlapping mode, and therefore three-dimensional visual display based on VR/AR/MR is achieved.

Illustratively, after the reconstructed surface point cloud data obtained by converting the target three-dimensional model corresponding to the skin is registered with the real surface point cloud data corresponding to the real scene image, each internal structure of the human brain is displayed in the real 3D image in a mixed reality manner, for example, the lesion may be projected on the real human brain in an overlapping manner to provide accurate positioning.

In the above embodiment, the target three-dimensional structure models corresponding to the internal structures are displayed in the corresponding positions in the real-scene image in an overlapping manner according to the registration result, so that the virtual internal structure can be viewed in the real-scene image in real time, the visualization of the internal structure is realized, and the precision is high.

As mentioned above, the computer device performs encoding and decoding processing by using a pre-trained three-dimensional convolutional neural network, which may be an image processing model obtained through machine learning, and the encoding and decoding processing of the three-dimensional medical image with the specified size is realized by processing the image processing model. Correspondingly, in some embodiments, as shown in fig. 8, the step of training the image processing model includes:

step S802, obtaining original three-dimensional medical image samples of the target part and structural labels respectively corresponding to the original three-dimensional medical image samples.

The original three-dimensional medical image sample includes, but is not limited to, one or more of a thick layer image, a thin layer image, and a pseudo-thick layer image. The pseudo-thick-layer image is obtained by extracting slices of the thin-layer image at equal intervals and performing interpolation processing. The spacing is typically between 1mm and 7 mm. In order to improve the accuracy of the image processing model during training, sample enhancement can be performed in other manners, such as random inversion and random transposition of the three-dimensional medical image.

Specifically, the computer device obtains a plurality of original three-dimensional medical image samples including a target portion, wherein each original three-dimensional medical image sample corresponds to a corresponding structure label, and the structure labels are used for representing key structures contained in the original three-dimensional medical image samples. Illustratively, the structure label may indicate whether a voxel belongs to a certain critical structure, and to which critical structure it belongs. For example, a structure label of "0" indicates that the voxel does not belong to any critical structure, a structure label of "1" is used to indicate that the voxel belongs to critical structure a, a structure label of "2" is used to indicate that the voxel belongs to critical structure B … …, and so on.

Step S804, the original three-dimensional medical image sample is subjected to pre-sampling processing, and a three-dimensional medical image sample with a specified size is obtained.

Specifically, the computer device performs pre-sampling processing on the original three-dimensional medical image of the target part through a pre-sampling layer which is arranged in advance according to output size information configured by the pre-sampling layer, so as to obtain a three-dimensional medical image with a specified size specified by the output size information, and the three-dimensional medical image sample with the specified size is used as the input of a subsequent image processing model.

Step S806, performing semantic segmentation on the three-dimensional medical image sample with the specified size through an image processing model to be trained, to obtain a predicted three-dimensional labeled image, where each voxel in the predicted three-dimensional labeled image corresponds to a prediction probability value belonging to each key structure.

Specifically, the computer device inputs a three-dimensional medical image sample with a specified size into an image processing model to be trained, the image processing model to be trained sequentially performs encoding processing and decoding processing, low-order image features in the three-dimensional medical image are obtained through the encoding processing, the low-order image features are converted into high-order image features through the decoding processing, and finally a predicted three-dimensional label image corresponding to the three-dimensional medical image sample is output.

Step S808, constructing a target loss function based on the difference between the predicted three-dimensional marked image and the structural label corresponding to the corresponding original three-dimensional medical image sample.

Specifically, the computer device compares the prediction probability value corresponding to each voxel in the prediction three-dimensional labeled image obtained by semantic segmentation with the structure label corresponding to each voxel in the corresponding original three-dimensional medical image sample, and constructs a loss function for the corresponding key structure according to the difference between the prediction probability value and the actual structure label. Since the image processing model can be used in the segmentation of a plurality of key structures, when the image processing model processes a plurality of key structures, the computer device constructs a total loss function, i.e. a target loss function, corresponding to the image processing model according to the loss function corresponding to each key structure.

In some embodiments, the constructing an object loss function based on a difference between the predicted three-dimensional labeled image and a structural label corresponding to the corresponding original three-dimensional medical image sample comprises: determining initial losses respectively corresponding to each key structure based on the coincidence degree between the predicted three-dimensional marked image and the structure label corresponding to the corresponding original three-dimensional medical image sample; and constructing a target loss function according to the initial loss and the preset weight corresponding to each key structure.

The preset weight is used for representing the importance degree of the key structure. For example, the skin accuracy is very critical, and directly affects the accuracy of the subsequent registration, so the value of the preset weight corresponding to the key structure of the skin may be larger than the values of the preset weights corresponding to other key structures.

Specifically, the computer device determines initial losses corresponding to the critical structures based on the degree of coincidence between the predicted three-dimensional marker image and the structure labels corresponding to the corresponding original three-dimensional medical image samples. And each key structure is preset with a corresponding preset weight, and the computer equipment can construct a target loss function according to the preset weight and the initial loss.

Illustratively, the initial loss L _dice The formula of (c) can be as follows:

wherein, X is the output three-dimensional marking image, and Y is the original three-dimensional medical image sample marked with the structure label for reference. L is _dice For characterizing the degree of overlap, L, between the output probability map and a standard probability map _dice The larger the size, the higher the degree of overlap between the probability map and the standard probability map; when the probability map completely overlaps the standard probability map, L _dice 1. Accordingly, according to presets corresponding to each key structureWeight, target loss function L _total This can be shown as follows:

wherein n is the number of key structures, λ _i Is a predetermined weight, L, of the critical structure i _i Initial loss L for critical structure i _dice 。

In the above embodiment, different preset weights are respectively set for different key structures, and a final target loss function is determined according to the different preset weights and the initial loss, so that the training effect on the image processing model is better, and the training accuracy can be improved.

And step S810, adjusting model parameters of the image processing model based on the target loss function, continuing model training based on the image processing model after the model parameters are adjusted, and stopping training until a training stopping condition is reached to obtain the trained image processing model.

Specifically, in the training process, the computer device adjusts model parameters of the image processing model according to the target loss function, continues model training on the image processing model towards the direction of the minimum target loss function, and stops training until reaching a training stop condition to obtain the trained image processing model. The training stopping condition includes, but is not limited to, at least one of the training time reaching a preset duration, the training times reaching a preset number, the target loss function reaching a preset threshold, and the like.

In the embodiment, the image processing model is trained, and the parameter update of the target loss function constraint image processing model in the training process is constructed, so that the trained image processing model can be quickly and accurately coded and decoded in subsequent use, and the precision of subsequent segmentation and reconstruction is improved.

In some embodiments, the structural labels corresponding to the critical structures may be manually labeled. In other embodiments, the structural labels corresponding to the key structures may be obtained by a labeling algorithm. To ensure that the reconstruction accuracy is sufficiently high, accordingly, in some embodiments, as shown in fig. 9, the step of acquiring the structure label of the original three-dimensional medical image sample corresponding to any critical structure includes:

step S902, performing slicing processing on the unmarked original three-dimensional medical image sample to obtain a plurality of sliced samples.

Specifically, the computer device performs equidistant slicing processing on unlabelled original three-dimensional medical image samples according to a certain interval, so as to obtain a plurality of slice samples corresponding to one original three-dimensional medical image. Illustratively, the pitch is, for example, 1mm to 7 mm.

And step S904, acquiring a preset threshold value matched with any key structure, and respectively performing threshold segmentation on each section sample based on the preset threshold value to obtain a binary section sample.

Specifically, the computer device obtains preset threshold values matched with the key structures, and performs threshold segmentation on the slice samples based on the preset threshold values to obtain a plurality of binary slice samples. For example, a preset threshold corresponding to the skin is set to-390, each slice sample is subjected to threshold segmentation based on the preset threshold, and a voxel value greater than the threshold is marked as 1 and filled with white (or black), whereas the voxel value is marked as 0 and filled with black (or white), and the like, thereby obtaining a plurality of binarized slice samples.

And step S906, performing cavity filling processing on the binaryzation slice sample to obtain a slice sample to be labeled.

Since the threshold segmentation is only roughly performed by the preliminary segmentation, the segmentation result is not accurate enough. Therefore, specifically, the computer device performs cavity filling processing on the binarized sliced sample, and fills the cavity or hole generated by the threshold segmentation, thereby obtaining the sliced sample to be labeled.

In some embodiments, the performing cavity filling processing on the binarized sliced sample to obtain a sliced sample to be labeled includes: determining at least one voxel in a binarization slice sample as a seed voxel, and determining a connected domain corresponding to the seed voxel; filling voxels with the same attributes as the seed voxels based on preset voxel values in a connected domain corresponding to the seed voxels to obtain filled slice samples; and summing the filling slice samples after negation and the corresponding binaryzation segmentation samples to obtain the to-be-labeled slice samples.

Specifically, the computer device determines at least one voxel as a seed voxel in the binarized slice sample and determines its connected domain from the seed voxel. In the connected domain of the seed voxels, the computer device fills the voxels with the same attribute as the seed voxels based on preset voxel values to obtain filled slice samples. The attribute identity is, for example, the same gray scale value. The preset voxel values are, for example, voxel values representing black and white, but not limited to this, and may also be, for example, voxel values representing red and blue, and the like. After filling, the computer device performs negation on the filled slice samples obtained through filling, and sums the filled slice samples obtained through negation and the corresponding binaryzation segmentation samples, so that the slice samples to be labeled are obtained.

In some cases, such as a target portion of the head, the nose tip and the back of the brain are sometimes beyond the image range, so setting two or more seed voxels can improve robustness. Illustratively, as shown in fig. 10, the computer device determines voxels of the upper left corner and the lower right corner as seed voxels in the binarized slice sample (a), and performs filling by using a FloodFill algorithm function of OpenCV (computer vision and machine learning software library) to obtain a filled slice sample (b). And (c) the computer equipment performs negation on the filling section sample (b) to obtain the filling section sample obtained after negation, such as a mask image (c) of an internal hole, and sums the filling section sample obtained after negation and the corresponding binarization segmentation sample to obtain a final section sample to be labeled (d). In some cases, manual correction may be performed manually to improve the accuracy of the labeling due to the possibility of incorrect filling at the ear and nostril for automatic filling.

In the embodiment, the cavity filling processing is performed on the binarization slice sample, the threshold segmentation boundary is very accurate, the internal holes are effectively removed, the labeling can be quickly and accurately realized, and the accuracy of image segmentation and three-dimensional structure model reconstruction in subsequent training and application is further improved.

Step S908, labeling voxels representing the critical structures in the slice sample to be labeled based on the structural label corresponding to any critical structure, to obtain a labeled slice sample carrying the structural label.

Specifically, the computer device labels all voxels representing a certain key structure in the to-be-labeled slice sample with a structure label corresponding to the key structure, so as to obtain a labeled slice sample with the structure label. Illustratively, for all voxels characterizing the skin, the computer device labels the structural label corresponding to the skin, resulting in a labeled slice sample carrying the structural label of the skin.

Step S910, determining a structural label corresponding to the original three-dimensional medical image sample to which each slice sample belongs according to the structural label corresponding to each slice sample.

Specifically, the computer device determines the structural label corresponding to the original three-dimensional medical image sample by combining the structural labels corresponding to the slice samples according to the original three-dimensional medical image sample to which each slice sample belongs. Therefore, the original three-dimensional medical image sample carrying the structural label can be used as the input of the image processing model for the training process of the image processing model.

In the above embodiment, the accuracy of the trained image processing model is higher by accurately labeling the key structure, so that the accuracy of the three-dimensional structure model reconstructed by the output of the image processing model is higher.

The three-dimensional image-based key structure reconstruction method provided by the embodiment of the application can be applied to scenes such as rigid body transformation, non-rigid body transformation and the like of model reconstruction. Or, the method may also be applied to medical treatment application scenarios in neurosurgery and orthopedics, as shown in fig. 11, taking medical treatment of a brain as an example, before treatment, a CT image and an MRI image are scanned by a medical scanning device, key structures are segmented by the steps in the above embodiments, and a three-dimensional structure model corresponding to each key structure is obtained by reconstruction. The three-dimensional structure model corresponding to the skin can be subjected to intraoperative registration with real surface point cloud data obtained through scanning by a point cloud scanner/depth camera and the like, and a registration result after registration can be subjected to mixed reality rendering for medical treatment, assisting an operator in positioning a key structure and the like.

In one specific example, the target site is exemplified as a head, including a variety of critical structures such as skin, skull, brain, ventricle, and lesion. The common image reconstruction is an image-to-image reconstruction mode, and the image reconstructed by the mode is not directed at a key structure, so that the key structure obtained by subsequent segmentation still has a fault feeling, and the accuracy is poor, so that the requirement of medical treatment cannot be met. The segmentation and reconstruction method provided by the embodiment of the application can meet the requirement of visualization, can acquire an accurate and high-resolution three-dimensional structure model, and reduces the registration error in the navigation process. The overall flow of the key structure reconstruction method based on the three-dimensional image in the embodiment of the application is shown in fig. 12A, and the computer device performs interpolation processing on a thick-layer original image through a pre-sampling layer, performs semantic segmentation through a three-dimensional convolution neural network, performs post-sampling processing, and performs image segmentation to obtain a thin-layer segmented image.

Taking the original image of the thick layer as a thick layer CT image as an example, as shown in fig. 12B, the computer device performs pre-sampling by linear interpolation, adjusts the size of the acquired original three-dimensional medical image to a specified size of 176 × 176 × 176, and the specified size three-dimensional medical image is used as an input of the subsequent 3D U-Net, and outputs a three-dimensional marker image also in a specified size by 3 DU-Net. In order to further improve the resolution, the computer equipment carries out post-sampling processing of up-sampling on the three-dimensional mark image and interpolates the three-dimensional mark image to the physical resolution of 1mm interval, so that the actual scene requirement is met. And then obtaining a three-dimensional structure image corresponding to the key structure through image segmentation. Illustratively, as shown in fig. 12C and 12D, the results of the first row in fig. 12C and 12D, which are obtained by directly performing threshold segmentation on the original 5mm thick layer CT image, respectively, can be seen to have very low resolution and significant jaggies and faults. The result of the second line is a segmented image and a three-dimensional structure model obtained by the three-dimensional image-based key structure reconstruction method provided by the embodiment of the application, the resolution is higher, the details are more accurate, the surface is smoother, and the accuracy is ensured particularly for regions such as canthus, nose tip, and vertex in the skin.

Based on the same inventive concept, as shown in fig. 13, an embodiment of the present application further provides a training method for implementing an image processing model of the above-mentioned three-dimensional image-based key structure reconstruction method, where the method may be applied to a terminal or a server, or may be executed by the terminal and the server in a cooperation manner. The method is applied to computer equipment as an example, and comprises the following steps:

step S1302, acquiring original three-dimensional medical image samples of the target portion and structural labels corresponding to the original three-dimensional medical image samples respectively.

Step S1304, performing pre-sampling processing on the original three-dimensional medical image sample to obtain a three-dimensional medical image sample of a specified size.

Step 1306, performing semantic segmentation processing on the three-dimensional medical image sample with the specified size through an image processing model to be trained to obtain a predicted three-dimensional labeled image, wherein each voxel in the predicted three-dimensional labeled image corresponds to a prediction probability value belonging to each key structure.

Step S1310, constructing a target loss function based on a difference between the predicted three-dimensional labeled image and the structural label corresponding to the corresponding original three-dimensional medical image sample.

Specifically, the computer device determines initial losses corresponding to the critical structures based on the degree of coincidence between the predicted three-dimensional marker image and the structure labels corresponding to the corresponding original three-dimensional medical image samples. And each key structure is preset with corresponding preset weight, and the computer equipment can construct a target loss function according to the preset weight and the initial loss.

Illustratively, the initial loss L _dice The formula of (c) can be as follows:

wherein X is the output three-dimensional label imageAnd Y is the original three-dimensional medical image sample marked with the structural label for reference. L is a radical of an alcohol _dice For characterizing the degree of overlap, L, between the output probability map and a standard probability map _dice The larger the size, the higher the degree of overlap between the probability map and the standard probability map; when the probability map completely overlaps the standard probability map, L _dice 1. Accordingly, the objective loss function L is based on the preset weights corresponding to each key structure _total This can be shown as follows:

And step S1312, adjusting model parameters of the image processing model based on the target loss function, continuing model training based on the image processing model after the model parameters are adjusted, and stopping training until a training stopping condition is reached to obtain the trained image processing model.

In the training method of the image processing model, the original three-dimensional medical image samples with the structural labels are obtained, pre-sampling processing is firstly carried out, and the original three-dimensional medical image samples with different sizes and thicknesses are uniformly converted into the three-dimensional medical image samples with the specified size so as to meet the input requirement of a subsequent image processing model; semantic segmentation processing is carried out on the three-dimensional medical image sample with the specified size through an image processing model to be trained to obtain a predicted three-dimensional marked image, and a target loss function is constructed according to the difference between the structure labels corresponding to the predicted three-dimensional marked image and the corresponding original three-dimensional medical image sample, so that the training process of the image processing model is carried out towards the direction of minimizing the target loss function, and the accuracy of the trained image processing model meets the requirement; and in the training process, continuously adjusting model parameters of the image processing model based on the target loss function, and stopping training until a training stopping condition is reached, thereby obtaining the trained image processing model. The image processing model trained in the mode is high in accuracy, high-resolution image segmentation can be achieved, and then the accuracy of a follow-up reconstructed three-dimensional structure model can be guaranteed.

In some embodiments, the structural labels corresponding to the critical structures may be manually labeled. In other embodiments, the structure label corresponding to the key structure may be obtained through a labeling algorithm. In order to ensure that the reconstruction accuracy is sufficiently high, in some embodiments, as shown in fig. 14, the step of acquiring the structure label of the original three-dimensional medical image sample corresponding to any critical structure includes:

step S1402, a plurality of slice samples are obtained by slicing the unmarked original three-dimensional medical image sample.

Step S1404, obtaining a preset threshold value matched with any key structure, and performing threshold segmentation on each slice sample based on the preset threshold value, to obtain a binarized slice sample.

And step S1406, performing cavity filling processing on the binarized sliced sample to obtain a sliced sample to be labeled.

In some embodiments, the performing cavity filling processing on the binarized sliced sample to obtain a sliced sample to be labeled includes: determining at least one voxel in a binarization slice sample as a seed voxel, and determining a connected domain corresponding to the seed voxel; filling voxels with the same attribute as the seed voxels on the basis of a preset voxel value in a connected domain corresponding to the seed voxels to obtain a filling slice sample; and summing the filling slice samples after negation and the corresponding binaryzation segmentation samples to obtain the to-be-labeled slice samples.

In some cases, such as a target portion of the head, the nose tip and the back of the brain are sometimes beyond the image range, so setting two or more seed voxels can improve robustness. Illustratively, as further shown in fig. 10, the computer device determines voxels of the top left corner and the bottom right corner as seed voxels in the binarized slice sample (a), and performs filling by using a FloodFill algorithm function of OpenCV (computer vision and machine learning software library) to obtain a filled slice sample (b). And (c) the computer equipment performs negation on the filling section sample (b) to obtain the filling section sample obtained after negation, such as a mask image (c) of an internal hole, and sums the filling section sample obtained after negation and the corresponding binarization segmentation sample to obtain a final section sample to be labeled (d). In some cases, manual correction may be performed manually to improve the accuracy of the labeling due to the possibility of incorrect filling at the ear and nostril for automatic filling.

In the embodiment, the cavity filling processing is carried out on the binaryzation section sample, the threshold segmentation boundary is very accurate, the internal hole is effectively removed, the labeling can be quickly and accurately realized, and the accuracy of image segmentation and three-dimensional structure model reconstruction in subsequent training and application is further improved.

And step S1408, labeling the voxels representing the critical structures in the to-be-labeled slice sample based on the structural label corresponding to any critical structure, and obtaining a labeled slice sample carrying the structural label.

Step S1410, determining a structural label corresponding to the original three-dimensional medical image sample to which each slice sample belongs, according to the structural label corresponding to each slice sample.

In the above embodiment, the accuracy of the trained image processing model is higher by accurately labeling the key structure, and thus the accuracy of the three-dimensional structure model reconstructed by the output of the image processing model is higher.

In a specific example, taking CT image as an example, in the sample preparation stage of training, 200 cases of thin layer CT and thick layer CT are collected respectively, and corresponding key structures, such as skin, brain, ventricles, lesions, etc., are labeled. Because subsequent registration is involved, the labeling of the skin is very important, and in order to ensure that the reconstruction accuracy is high enough, the labeling of the skin adopts a semi-automatic labeling algorithm. The computer device obtains a binarized sample of the skin by using threshold segmentation. Due to the large number of cavities in the skin, the computer device sets seed voxels in the top left and bottom right corners of the sliced specimen, which are filled using the OpenCV FloodFill function. And after the filling result of the Flood Fill is obtained, the computer equipment sums the mask of the internal hole obtained after the filled image is inverted and the binarization sample to obtain the final filling result. In order to allow the network to encode and decode both the thick layer image and the thin layer image, the training samples may include a thin layer CT image, a thick layer CT image, and a pseudo-thick layer CT image. In addition, the number of samples is also increased by conventional online data enhancement (including random flipping, random transposing, etc. operations).

It should be understood that, although the steps in the flowcharts related to the embodiments are shown in sequence as indicated by the arrows, the steps are not necessarily executed in sequence as indicated by the arrows. The steps are not performed in the exact order shown and described, and may be performed in other orders, unless explicitly stated otherwise. Moreover, at least a part of the steps in the flowcharts related to the above embodiments may include multiple steps or multiple stages, which are not necessarily performed at the same time, but may be performed at different times, and the order of performing the steps or stages is not necessarily sequential, but may be performed alternately or alternately with other steps or at least a part of the steps or stages in other steps.

Based on the same inventive concept, the embodiment of the present application further provides a three-dimensional image-based critical structure reconstruction device for implementing the above-mentioned three-dimensional image-based critical structure reconstruction method. The implementation scheme for solving the problem provided by the apparatus is similar to the implementation scheme described in the above method, so specific limitations in one or more embodiments of the apparatus for reconstructing a key structure based on a three-dimensional image provided below can be referred to the limitations in the above method for reconstructing a key structure based on a three-dimensional image, and are not described herein again.

In some embodiments, as shown in fig. 15, there is provided a three-dimensional image based critical structure reconstruction apparatus 1500, comprising: pre-sampling module 1501, semantic segmentation module 1502, post-sampling module 1503, structure segmentation module 1504, and three-dimensional reconstruction module 1505, wherein:

the pre-sampling module 1501 is configured to perform pre-sampling processing on an original three-dimensional medical image of a target portion, so as to obtain a three-dimensional medical image of a specified size, where the target portion includes at least one key structure.

The semantic segmentation module 1502 is configured to perform semantic segmentation on the three-dimensional medical image with the specified size to obtain a corresponding three-dimensional labeled image, where each voxel in the three-dimensional labeled image corresponds to a probability value belonging to each key structure.

And the post-sampling module 1503 is configured to perform post-sampling processing on the three-dimensional marker image to obtain a target marker image meeting a high resolution condition.

A structure segmentation module 1504, configured to segment the target labeled image based on a probability value corresponding to each voxel in the three-dimensional labeled image, to obtain at least one three-dimensional structure image, where each three-dimensional structure image corresponds to one key structure.

The three-dimensional reconstruction module 1505 is configured to perform reconstruction processing on each three-dimensional structure image to obtain a target three-dimensional structure model corresponding to each key structure, where the target three-dimensional structure model is used for performing visual display.

In some embodiments, the pre-sampling module 1501 is further configured to acquire an original three-dimensional medical image obtained by acquiring a medical image of a target portion of a target object; and carrying out interpolation processing on the original three-dimensional medical image through a front sampling layer to obtain a three-dimensional medical image with a specified size.

In some embodiments, semantic segmentation module 1502 is further configured to extract a plurality of continuous feature vectors in the three-dimensional medical image of the specified size through a coding network; fusing the continuous feature vectors through a decoding network to obtain a feature map of the three-dimensional medical image corresponding to the specified size; and normalizing the feature map, converting the normalized feature map through an activation layer, and outputting a three-dimensional marking image.

In some embodiments, the structure segmentation module 1504 is further configured to determine probability thresholds corresponding to the respective critical structures; for each key structure, comparing the probability value of each voxel in the three-dimensional marking image belonging to the corresponding key structure with the probability threshold corresponding to the corresponding key structure, and determining a target voxel matched with the corresponding key structure from the target marking image based on the comparison result; and segmenting the target voxel corresponding to each key structure respectively to obtain at least one three-dimensional structure image.

In some embodiments, the three-dimensional reconstruction module 1505 is further configured to, for each three-dimensional structure image, create an initial three-dimensional structure model matching a key structure contained in the three-dimensional structure image, where a surface of the key structure in the initial three-dimensional structure model is composed of a plurality of patches of a preset shape; and smoothing the surfaces of the key structures in the initial three-dimensional structure models to obtain target three-dimensional structure models respectively corresponding to the key structures.

In some embodiments, the key structure of the target region includes a surface structure and an internal structure, and the apparatus further includes a registration module configured to acquire a real scene image obtained by image-capturing the target region in a real scene, and determine real surface point cloud data corresponding to the real scene image; and converting the target three-dimensional structure model corresponding to the surface structure into reconstructed surface point cloud data, and registering the reconstructed surface point cloud data and the real surface point cloud data.

In some embodiments, the apparatus further includes a display module, configured to determine, based on a registration result, a relative position of a target three-dimensional structure model corresponding to each internal structure of the target portion in the real scene image after registration of the reconstructed surface point cloud data and the real surface point cloud data is completed; and displaying the target three-dimensional structure model corresponding to each internal structure in the real scene image in an overlapping manner according to the relative position of the target three-dimensional structure model corresponding to each internal structure in the real scene image.

In some embodiments, the encoding and decoding process of the three-dimensional medical image with the specified size is implemented by an image processing model, the apparatus may include or be externally connected with a training module, and the training module is used for the training step of the image processing model and includes: acquiring original three-dimensional medical image samples of a target part and structural labels respectively corresponding to the original three-dimensional medical image samples; carrying out pre-sampling processing on the original three-dimensional medical image sample to obtain a three-dimensional medical image sample with a specified size; semantic segmentation processing is carried out on the three-dimensional medical image sample with the specified size through an image processing model to be trained to obtain a predicted three-dimensional labeled image, and each voxel in the predicted three-dimensional labeled image corresponds to a prediction probability value belonging to each key structure; constructing a target loss function based on the difference between the predicted three-dimensional marked image and the structural label corresponding to the corresponding original three-dimensional medical image sample; and adjusting model parameters of the image processing model based on the target loss function, continuing model training based on the image processing model after the model parameters are adjusted until a training stopping condition is reached, and obtaining the trained image processing model.

In some embodiments, the training module is further configured to slice the unlabeled original three-dimensional medical image sample to obtain a plurality of slice samples; acquiring a preset threshold value matched with any key structure, and respectively carrying out threshold value segmentation on each section sample based on the preset threshold value to obtain a binary section sample; carrying out cavity filling treatment on the binarization slice sample to obtain a slice sample to be marked; marking voxels for representing the key structures in the section sample to be marked based on the structure label corresponding to any key structure to obtain a marked section sample carrying the structure label; and determining the structural label corresponding to the original three-dimensional medical image sample to which each slice sample belongs according to the structural label corresponding to each slice sample.

In some embodiments, the training module is further configured to determine at least one voxel in the binarized slice sample as a seed voxel, and determine a connected domain corresponding to the seed voxel; filling voxels with the same attributes as the seed voxels based on preset voxel values in a connected domain corresponding to the seed voxels to obtain filled slice samples; and summing the filling slice samples after negation and the corresponding binaryzation segmentation samples to obtain the to-be-labeled slice samples.

In some embodiments, the training module is further configured to determine initial losses corresponding to the respective critical structures based on a degree of coincidence between the predicted three-dimensional labeled image and the structural labels corresponding to the respective original three-dimensional medical image samples; and constructing a target loss function according to the initial loss and the preset weight corresponding to each key structure.

All or part of each module in the above-mentioned key structure reconstruction device based on three-dimensional images can be realized by software, hardware and their combination. The modules can be embedded in a hardware form or independent from a processor in the computer device, and can also be stored in a memory in the computer device in a software form, so that the processor can call and execute operations corresponding to the modules.

Based on the same inventive concept, the embodiment of the present application further provides a training device of an image processing model for implementing the above-mentioned training method of an image processing model. The implementation scheme for solving the problem provided by the apparatus is similar to the implementation scheme described in the above method, so the specific limitations in the following embodiments of the training apparatus for one or more image processing models may refer to the limitations on the training method for the image processing model in the above description, and are not described herein again.

In some embodiments, as shown in fig. 16, there is provided an image processing model training apparatus 1600, comprising: an obtaining module 1601, a sampling module 1602, a predicting module 1603, a constructing module 1604, and a parameter adjusting module 1605, wherein:

the obtaining module 1601 is configured to obtain an original three-dimensional medical image sample of a target portion and a structure label corresponding to each original three-dimensional medical image sample.

The sampling module 1602 is configured to perform pre-sampling processing on the original three-dimensional medical image sample to obtain a three-dimensional medical image sample with a specified size.

The prediction module 1603 is configured to perform semantic segmentation on the three-dimensional medical image sample with the specified size through an image processing model to be trained to obtain a predicted three-dimensional labeled image, wherein each voxel in the predicted three-dimensional labeled image corresponds to a prediction probability value belonging to each key structure.

A constructing module 1604, configured to construct a target loss function based on a difference between the predicted three-dimensional labeled image and a corresponding structural label corresponding to the original three-dimensional medical image sample.

And the parameter adjusting module 1605 is configured to adjust the model parameters of the image processing model based on the target loss function, and continue model training based on the image processing model after the model parameters are adjusted until a training stop condition is reached, so as to obtain a trained image processing model.

In some embodiments, the obtaining module 1601 is further configured to slice the unlabeled original three-dimensional medical image sample to obtain a plurality of slice samples; acquiring a preset threshold value matched with any key structure, and respectively carrying out threshold value segmentation on each section sample based on the preset threshold value to obtain a binary section sample; carrying out cavity filling treatment on the binarization slice sample to obtain a slice sample to be marked; labeling voxels representing the critical structures in the to-be-labeled slice sample based on the structural label corresponding to any critical structure to obtain a labeled slice sample carrying the structural label; and determining the structural label corresponding to the original three-dimensional medical image sample to which each slice sample belongs according to the structural label corresponding to each slice sample.

In some embodiments, the obtaining module 1601 is further configured to determine at least one voxel in the binarized slice sample as a seed voxel, and determine a connected component corresponding to the seed voxel; filling voxels with the same attribute as the seed voxels on the basis of a preset voxel value in a connected domain corresponding to the seed voxels to obtain a filling slice sample; and summing the filling slice samples after negation and the corresponding binaryzation segmentation samples to obtain the to-be-labeled slice samples.

In some embodiments, the constructing module 1604 is further configured to determine initial losses corresponding to the critical structures, respectively, based on a degree of coincidence between the predicted three-dimensional labeled image and the structure labels corresponding to the corresponding original three-dimensional medical image samples; and constructing a target loss function according to the initial loss and the preset weight corresponding to each key structure.

In some embodiments, a computer device is provided, which may be a terminal, and its internal structure diagram may be as shown in fig. 17. The computer apparatus includes a processor, a memory, an input/output interface, a communication interface, a display unit, and an input device. The processor, the memory and the input/output interface are connected by a system bus, and the communication interface, the display unit and the input device are connected by the input/output interface to the system bus. Wherein the processor of the computer device is configured to provide computing and control capabilities. The memory of the computer device comprises a nonvolatile storage medium and an internal memory. The non-volatile storage medium stores an operating system and a computer program. The internal memory provides an environment for the operating system and the computer program to run on the non-volatile storage medium. The input/output interface of the computer device is used for exchanging information between the processor and an external device. The communication interface of the computer device is used for carrying out wired or wireless communication with an external terminal, and the wireless communication can be realized through WIFI, a mobile cellular network, NFC (near field communication) or other technologies. The computer program is executed by a processor to implement a method for reconstructing a key structure based on three-dimensional images or a method for training an image processing model. The display unit of the computer equipment is used for forming a visual picture, and can be a display screen, a projection device or a virtual reality imaging device, the display screen can be a liquid crystal display screen or an electronic ink display screen, the input device of the computer equipment can be a touch layer covered on the display screen, a key, a track ball or a touch pad arranged on a shell of the computer equipment, and can also be an external keyboard, a touch pad or a mouse and the like.

Those skilled in the art will appreciate that the architecture shown in fig. 17 is merely a block diagram of some of the structures associated with the disclosed aspects and is not intended to limit the computing devices to which the disclosed aspects apply, as particular computing devices may include more or less components than those shown, or may combine certain components, or have a different arrangement of components.

In some embodiments, there is further provided a computer device comprising a memory and a processor, the memory having stored therein a computer program, the processor implementing the steps of the above method embodiments when executing the computer program.

In some embodiments, a computer-readable storage medium is provided, on which a computer program is stored, which computer program, when being executed by a processor, carries out the steps of the above-mentioned method embodiments.

In some embodiments, a computer program product is provided, comprising a computer program which, when executed by a processor, performs the steps in the above-described method embodiments.

It should be noted that the images (including three-dimensional medical images, three-dimensional medical image samples, real scene images, etc.) referred to in the present application are all authorized or fully authorized information and data by each party, and the collection, use and processing of the related images need to comply with the relevant laws and regulations and standards of the relevant countries and regions.

It will be understood by those skilled in the art that all or part of the processes of the methods of the embodiments described above can be implemented by hardware instructions of a computer program, which can be stored in a non-volatile computer-readable storage medium, and when executed, can include the processes of the embodiments of the methods described above. Any reference to memory, database, or other medium used in the embodiments provided herein may include at least one of non-volatile and volatile memory. The nonvolatile Memory may include Read-Only Memory (ROM), magnetic tape, floppy disk, flash Memory, optical Memory, high-density embedded nonvolatile Memory, resistive Random Access Memory (ReRAM), Magnetic Random Access Memory (MRAM), Ferroelectric Random Access Memory (FRAM), Phase Change Memory (PCM), graphene Memory, and the like. Volatile Memory can include Random Access Memory (RAM), external cache Memory, and the like. By way of illustration and not limitation, RAM can take many forms, such as Static Random Access Memory (SRAM) or Dynamic Random Access Memory (DRAM), among others. The databases involved in the embodiments provided herein may include at least one of relational and non-relational databases. The non-relational database may include, but is not limited to, a block chain based distributed database, and the like. The processors referred to in the embodiments provided herein may be general purpose processors, central processing units, graphics processors, digital signal processors, programmable logic devices, quantum computing based data processing logic devices, etc., without limitation.

The technical features of the above embodiments can be arbitrarily combined, and for the sake of brevity, all possible combinations of the technical features in the above embodiments are not described, but should be considered as the scope of the present specification as long as there is no contradiction between the combinations of the technical features.

The above-mentioned embodiments only express several embodiments of the present application, and the description thereof is more specific and detailed, but not construed as limiting the scope of the present application. It should be noted that, for a person skilled in the art, several variations and modifications can be made without departing from the concept of the present application, which falls within the scope of protection of the present application. Therefore, the protection scope of the present application should be subject to the appended claims.

Claims

1. A key structure reconstruction method based on three-dimensional images is characterized by comprising the following steps:

post-sampling processing is carried out on the basis of the three-dimensional marked image to obtain a target marked image meeting a high-resolution condition;

2. The method according to claim 1, wherein the pre-sampling processing of the original three-dimensional medical image of the target portion to obtain a three-dimensional medical image with a specified size comprises:

acquiring an original three-dimensional medical image obtained by acquiring a medical image of a target part of a target object;

and carrying out interpolation processing on the original three-dimensional medical image through a front sampling layer to obtain a three-dimensional medical image with a specified size.

3. The method according to claim 1, wherein the performing semantic segmentation processing on the three-dimensional medical image with the specified size to obtain a corresponding three-dimensional labeled image comprises:

extracting a plurality of continuous characteristic vectors in the three-dimensional medical image with the specified size through a coding network;

fusing the continuous feature vectors through a decoding network to obtain a feature map of the three-dimensional medical image corresponding to the specified size;

and normalizing the feature map, converting the normalized feature map through an activation layer, and outputting a three-dimensional marking image.

4. The method according to claim 1, wherein the segmenting the target labeled image based on the probability value corresponding to each voxel in the three-dimensional labeled image to obtain at least one three-dimensional structural image comprises:

determining probability threshold values respectively corresponding to the key structures;

for each key structure, comparing the probability value of each voxel in the three-dimensional marking image belonging to the corresponding key structure with the probability threshold corresponding to the corresponding key structure, and determining a target voxel matched with the corresponding key structure from the target marking image based on the comparison result;

and segmenting the target voxel corresponding to each key structure respectively to obtain at least one three-dimensional structure image.

5. The method according to claim 1, wherein the performing reconstruction processing on each three-dimensional structure image to obtain a target three-dimensional structure model corresponding to each key structure respectively comprises:

for each three-dimensional structure image, establishing an initial three-dimensional structure model matched with a key structure contained in the three-dimensional structure image, wherein the surface of the key structure in the initial three-dimensional structure model is formed by a plurality of patches in preset shapes;

and smoothing the surfaces of the key structures in the initial three-dimensional structure models to obtain target three-dimensional structure models respectively corresponding to the key structures.

6. The method of claim 1, wherein the critical structures of the target site include surface structures and internal structures, the method further comprising:

acquiring a real scene image obtained by carrying out image acquisition on the target part in a real scene, and determining real surface point cloud data corresponding to the real scene image;

and converting the target three-dimensional structure model corresponding to the surface structure into reconstructed surface point cloud data, and registering the reconstructed surface point cloud data and the real surface point cloud data.

7. The method of claim 6, further comprising:

after the registration of the reconstructed surface point cloud data and the real surface point cloud data is finished, determining a target three-dimensional structure model corresponding to each internal structure of the target part and the relative position of the target three-dimensional structure model in the real scene image based on a registration result;

and displaying the target three-dimensional structure models corresponding to the internal structures in the real scene image in an overlapping manner according to the relative positions of the target three-dimensional structure models corresponding to the internal structures in the real scene image.

8. The method according to any one of claims 1 to 7, wherein the semantic segmentation processing of the three-dimensional medical image of the specified size is implemented by an image processing model, and the training step of the image processing model comprises:

semantic segmentation processing is carried out on the three-dimensional medical image sample with the specified size through an image processing model to be trained to obtain a predicted three-dimensional labeled image, and each voxel in the predicted three-dimensional labeled image corresponds to a prediction probability value belonging to each key structure;

9. The method of claim 8, wherein the step of obtaining the structural label of the original three-dimensional medical image sample corresponding to any critical structure comprises:

slicing the unmarked original three-dimensional medical image sample to obtain a plurality of sliced samples;

acquiring a preset threshold value matched with any one key structure, and respectively carrying out threshold value segmentation on each section sample based on the preset threshold value to obtain a binaryzation section sample;

carrying out cavity filling treatment on the binaryzation section sample to obtain a section sample to be labeled;

marking voxels representing key structures in the slice sample to be marked based on the structure label corresponding to any key structure to obtain a marked slice sample carrying the structure label;

and determining the structural label corresponding to the original three-dimensional medical image sample to which each slice sample belongs according to the structural label corresponding to each slice sample.

10. The method according to claim 9, wherein the performing a cavity filling process on the binarized slice sample to obtain a slice sample to be labeled comprises:

determining at least one voxel in the binarized slice sample as a seed voxel, and determining a connected domain corresponding to the seed voxel;

filling voxels with the same attributes as the seed voxels based on preset voxel values in a connected domain corresponding to the seed voxels to obtain filled slice samples;

and summing the filling slice sample after negation and the corresponding binaryzation segmentation sample to obtain a to-be-labeled slice sample.

11. The method of claim 8, wherein constructing an object loss function based on differences between the predicted three-dimensional labeled image and the corresponding structural labels of the original three-dimensional medical image samples comprises:

determining initial losses respectively corresponding to each key structure based on the coincidence degree between the predicted three-dimensional marked image and the structure label corresponding to the corresponding original three-dimensional medical image sample;

and constructing a target loss function according to the initial loss and the preset weight corresponding to each key structure.

12. A method of training an image processing model, the method comprising:

13. The method of claim 12, wherein the step of obtaining the structural label of the original three-dimensional medical image sample corresponding to any critical structure comprises:

carrying out cavity filling treatment on the binarization slice sample to obtain a slice sample to be marked;

14. The method according to claim 13, wherein the performing a cavity filling process on the binarized sliced sample to obtain a sliced sample to be labeled comprises:

15. The method of claim 14, wherein constructing an object loss function based on differences between the predicted three-dimensional labeled image and the corresponding structural labels of the original three-dimensional medical image samples comprises:

16. A three-dimensional image-based key structure reconstruction device is characterized by comprising:

the semantic segmentation module is used for performing semantic segmentation processing on the three-dimensional medical image with the specified size to obtain a corresponding three-dimensional label image, and each voxel in the three-dimensional label image corresponds to a probability value belonging to each key structure;

17. An apparatus for training an image processing model, the apparatus comprising:

the sampling module is used for carrying out pre-sampling processing on the original three-dimensional medical image sample to obtain a three-dimensional medical image sample with a specified size;

18. A computer device comprising a memory and a processor, the memory storing a computer program, characterized in that the processor, when executing the computer program, implements the steps of the method of any of claims 1 to 15.

19. A computer-readable storage medium, on which a computer program is stored, which, when being executed by a processor, carries out the steps of the method of any one of claims 1 to 15.

20. A computer program product comprising a computer program, characterized in that the computer program realizes the steps of the method of any one of claims 1 to 15 when executed by a processor.