WO2022147783A1 - 脑结构三维重建方法、装置及终端设备 - Google Patents

脑结构三维重建方法、装置及终端设备 Download PDF

Info

Publication number
WO2022147783A1
WO2022147783A1 PCT/CN2021/070934 CN2021070934W WO2022147783A1 WO 2022147783 A1 WO2022147783 A1 WO 2022147783A1 CN 2021070934 W CN2021070934 W CN 2021070934W WO 2022147783 A1 WO2022147783 A1 WO 2022147783A1
Authority
WO
WIPO (PCT)
Prior art keywords
brain
point cloud
image
neural network
convolutional neural
Prior art date
Application number
PCT/CN2021/070934
Other languages
English (en)
French (fr)
Inventor
王书强
胡博闻
申妍燕
Original Assignee
中国科学院深圳先进技术研究院
Priority date (The priority date is an assumption and is not a legal conclusion. Google has not performed a legal analysis and makes no representation as to the accuracy of the date listed.)
Filing date
Publication date
Application filed by 中国科学院深圳先进技术研究院 filed Critical 中国科学院深圳先进技术研究院
Priority to US18/026,498 priority Critical patent/US20230343026A1/en
Priority to PCT/CN2021/070934 priority patent/WO2022147783A1/zh
Publication of WO2022147783A1 publication Critical patent/WO2022147783A1/zh

Links

Images

Classifications

    • G—PHYSICS
    • G06—COMPUTING; CALCULATING OR COUNTING
    • G06T—IMAGE DATA PROCESSING OR GENERATION, IN GENERAL
    • G06T17/00—Three dimensional [3D] modelling, e.g. data description of 3D objects
    • G—PHYSICS
    • G06—COMPUTING; CALCULATING OR COUNTING
    • G06T—IMAGE DATA PROCESSING OR GENERATION, IN GENERAL
    • G06T19/00—Manipulating 3D models or images for computer graphics
    • G06T19/20—Editing of 3D images, e.g. changing shapes or colours, aligning objects or positioning parts
    • G—PHYSICS
    • G06—COMPUTING; CALCULATING OR COUNTING
    • G06T—IMAGE DATA PROCESSING OR GENERATION, IN GENERAL
    • G06T9/00—Image coding
    • G06T9/002—Image coding using neural networks
    • G—PHYSICS
    • G06—COMPUTING; CALCULATING OR COUNTING
    • G06T—IMAGE DATA PROCESSING OR GENERATION, IN GENERAL
    • G06T2210/00—Indexing scheme for image generation or computer graphics
    • G06T2210/41—Medical
    • G—PHYSICS
    • G06—COMPUTING; CALCULATING OR COUNTING
    • G06T—IMAGE DATA PROCESSING OR GENERATION, IN GENERAL
    • G06T2210/00—Indexing scheme for image generation or computer graphics
    • G06T2210/56—Particle system, point based geometry or rendering
    • G—PHYSICS
    • G06—COMPUTING; CALCULATING OR COUNTING
    • G06T—IMAGE DATA PROCESSING OR GENERATION, IN GENERAL
    • G06T2219/00—Indexing scheme for manipulating 3D models or images for computer graphics
    • G06T2219/20—Indexing scheme for editing of 3D models
    • G06T2219/2004—Aligning objects, relative positioning of parts

Definitions

  • the present application belongs to the technical field of artificial intelligence, and in particular relates to a method, device and terminal equipment for three-dimensional reconstruction of brain structure.
  • the embodiments of the present application provide a three-dimensional reconstruction method, device and terminal device of a brain structure, which can convert a 2D image of the brain into a 3D point cloud to provide doctors with more visual information.
  • an embodiment of the present application provides a three-dimensional reconstruction method of a brain structure, the method includes: acquiring a 2D image of the brain, and inputting the 2D image of the brain into a trained 3D brain point cloud reconstruction model for processing , the output is the 3D point cloud of the brain;
  • the 3D brain point cloud reconstruction model includes: ResNet encoder and graph convolutional neural network, the ResNet encoder is used to extract the encoded feature vector of the 2D image of the brain, and the graph convolutional neural network uses It is used to construct a 3D point cloud of the brain from the encoded feature vectors.
  • the encoded feature information of the image can be effectively extracted by the ResNet encoder, and the encoded feature information can guide the graph convolutional neural network to accurately construct a 3D point cloud.
  • the 2D image of the information is reconstructed into a 3D point cloud with richer and more accurate information, which can provide doctors with more and more accurate visual information for the lesion during the diagnosis and treatment process, thereby assisting doctors to make better decisions.
  • the graph convolutional neural network includes: multiple groups of alternately arranged graph convolution modules and branch modules, the graph convolution module is used to adjust the position coordinates of the point cloud, and the branch module is used to expand the number of point clouds.
  • the branch module can expand the number of point clouds to the target number, and the graph convolution module can adjust the position coordinates of the point cloud and reduce the dimension of the coordinates to 3 dimensions, so that it can correctly describe the target features .
  • the 3D point cloud can be generated top-down, and the relative position of the point cloud can be fully utilized while retaining the position information of the ancestor point cloud, thereby improving the accuracy of the reconstructed 3D point cloud.
  • the 3D brain point cloud reconstruction model is obtained based on the obtained training sample set and the corresponding discriminator training, the training sample set includes multiple training samples, and each training sample includes 2D brain image samples and 2D brain image samples.
  • the training method of the 3D brain point cloud reconstruction model includes: for each training sample, inputting the 2D brain image sample in the training sample into the initial neural network model to obtain the predicted 3D point cloud;
  • the 3D point cloud and the 3D point cloud samples in the training samples are input into the discriminator, and the training sample discrimination results are obtained; according to the discrimination results of each training sample, the loss function of the 3D brain point cloud reconstruction model and the discriminator loss function.
  • a 3D brain point cloud reconstruction model is obtained.
  • the graph convolutional neural network and the discriminator in the neural network model constitute a generative adversarial network, which does not require supervised learning during the training process, which reduces the training complexity of the model and improves the generalization ability of the model.
  • the method for obtaining training samples includes: obtaining a 3D image of the brain; performing image preprocessing on the 3D image of the brain and then slicing to obtain a 2D brain image sample; obtaining a 3D point cloud sample of the brain according to the 3D image .
  • the preprocessed 3D point cloud image is cut in different directions, and the clearest 2D point cloud image is selected.
  • the image is used as the input of the ResNet encoder, which can improve the accuracy of 3D brain point cloud reconstruction.
  • L E and G represent the loss values corresponding to the 3D brain point cloud reconstruction model
  • ⁇ 1 and ⁇ 2 are constants
  • L KL represents the KL divergence
  • Z is the distribution of the encoded feature vector generated by the ResNet encoder
  • z is the encoded feature vector
  • G( ) represents the output of the graph convolutional neural network
  • D( ) represents the discriminator
  • E( ) represents the expectation
  • LCD is the 3D point cloud and 3D point predicted by the initial neural network model Chamfer distance between cloud samples.
  • the loss function corresponding to the discriminator is:
  • E( ⁇ ) is the expectation
  • G( ⁇ ) is the output of the graph convolutional neural network
  • D( ⁇ ) is the discriminator
  • Y is the 3D point cloud sample
  • R is the 3D point cloud sample distribution
  • ⁇ gp is a constant
  • the loss function of the 3D brain point cloud reconstruction model is constructed by combining the chamfering distance loss function and the bulldozer distance loss function, which is higher than the classification accuracy of the existing model trained only by the chamfering distance as the loss function. If it is higher, the accuracy of the network can be improved, and the edge distortion of the 3D point cloud can be avoided, and the generation quality of the point cloud image can be improved.
  • an embodiment of the present application provides a three-dimensional reconstruction device for brain structure, the device includes: an acquisition unit for acquiring a 2D image of the brain; a reconstruction unit for inputting the 2D image of the brain into a The trained 3D brain point cloud reconstruction model is processed, and the 3D point cloud of the brain is obtained by outputting; the 3D brain point cloud reconstruction model includes: a ResNet encoder and a graph convolutional neural network, and the ResNet encoder uses For extracting the encoded feature vector of the 2D image of the brain, the graph convolutional neural network is used to construct a 3D point cloud of the brain according to the encoded feature vector.
  • an embodiment of the present application provides a terminal device, including a memory, a processor, and a computer program stored in the memory and running on the processor, where the processor implements the first aspect when the processor executes the computer program. any method.
  • an embodiment of the present application provides a computer-readable storage medium, where the computer-readable storage medium stores a computer program, and when the computer program is executed by a processor, any one of the methods in the first aspect is implemented.
  • an embodiment of the present application provides a computer program product that, when the computer program product runs on a processor, causes the processor to execute the method described in any one of the foregoing first aspects.
  • FIG. 1 is a schematic structural diagram of a 3D brain point cloud reconstruction model provided by the present application.
  • FIG. 2 is a schematic flowchart of a method for three-dimensional reconstruction of a brain structure provided by the present application
  • FIG. 3 is a schematic structural diagram of a 3D brain point cloud reconstruction training model provided by the present application.
  • FIG. 4 is a schematic diagram of a training process of a 3D brain point cloud reconstruction model provided by the present application.
  • FIG. 5 is a schematic structural diagram of a three-dimensional reconstruction device for brain structure provided by the present application.
  • FIG. 6 is a schematic structural diagram of a terminal device provided by the present application.
  • references in this specification to "one embodiment” or “some embodiments” and the like mean that a particular feature, structure or characteristic described in connection with the embodiment is included in one or more embodiments of the present application.
  • appearances of the phrases “in one embodiment,” “in some embodiments,” “in other embodiments,” “in other embodiments,” etc. in various places in this specification are not necessarily All refer to the same embodiment, but mean “one or more but not all embodiments” unless specifically emphasized otherwise.
  • the terms “including”, “including”, “having” and their variants mean “including but not limited to” unless specifically emphasized otherwise.
  • Point cloud is a data structure that describes the shape and structure of a specific object in three-dimensional space. It has the advantages of small space complexity, simple storage form and high computing performance. Compared with the flat space of 2D images, 3D point cloud data contains more spatial structure information, which can provide doctors with more visual information, thereby assisting doctors in better diagnosis and treatment. Therefore, it is of great significance to reconstruct 2D images into accurate and clear 3D point clouds.
  • the present application provides a three-dimensional reconstruction method, apparatus and terminal device of a brain structure.
  • the 2D image of the brain can be converted into a 3D point cloud to provide doctors with better visual information, so that they can make better diagnosis and treatment.
  • FIG. 1 is a 3D brain point cloud reconstruction model provided by this application.
  • the model includes: ResNet (residual network) encoder and Graph Convolutional Network (GCN).
  • ResNet residual network
  • GCN Graph Convolutional Network
  • the graph convolutional neural network as the generator of the 3D brain point cloud reconstruction model, includes multiple groups of alternately set branch modules and graph convolution modules.
  • the 2D image of the brain is input into the ResNet encoder, and the ResNet encoder can extract the encoded feature vector of the 2D image.
  • the ResNet encoder first quantizes the 2D image into a feature vector with a certain mean and variance and obeys a Gaussian distribution, and then randomly extracts a high-dimensional encoded feature vector with a preset dimension from the feature vector (for example, a 96-dimensional encoded feature vector), And pass the encoded feature vector to the graph convolutional neural network.
  • the encoded feature vector is an initial point cloud of the input graph convolutional neural network, and the coordinate dimension is 96.
  • the branch module is used to expand the number of point clouds, and the graph convolution module is used to adjust the position coordinates of each point cloud. Using the branch module and the graph convolution module alternately can accurately reconstruct the brain's 3D point cloud.
  • the 2D image of the brain may be MRI (Magnetic Resonance Imaging), CT (Computed Tomography), PET (Positron Emission Computed Tomography), DTI (Diffusion Tensor Imaging) or FMRI (Functional Magnetic) taken at any angle. Resonance Imaging) and other images.
  • FIG. 2 is a flowchart of an embodiment of a method for three-dimensional reconstruction of a brain structure provided by the present application.
  • the execution body of the method may be an image data acquisition device, such as a positron emission tomography PET device, a CT device, or an MRI. equipment and other terminal equipment. It can also be a control device of an image data acquisition device, a computer, a robot, a mobile terminal and other terminal devices. As shown in Figure 2, the method includes:
  • 2D images can be brain images such as MRI, CT, PET, DTI or fMRI taken at any angle.
  • MRI magnetic resonance imaging
  • CT computed tomography
  • PET PET
  • DTI magnetic resonance imaging
  • fMRI magnetic resonance imaging
  • the 2D image of the brain is input into the ResNet encoder to obtain an encoded feature vector.
  • the ResNet encoder first quantizes the 2D image of the brain into a feature vector with a certain mean ⁇ and a variance ⁇ and obeys a Gaussian distribution, and then randomly extracts a 96-dimensional encoded feature vector z from the feature vector, and Pass the encoded feature vector z to the graph convolutional neural network.
  • the encoded feature vector is used as the initial point cloud of the input graph convolutional neural network, with a quantity of 1 and a coordinate dimension of 96.
  • the graph convolutional neural network constructs a 3D point cloud of the brain according to the encoded feature vector.
  • the graph convolutional neural network includes a plurality of alternately arranged branch modules and graph convolution modules, wherein the branch module can map a point cloud into multiple point clouds, then, through multiple branch modules, the An initial point cloud is gradually expanded to a target number of point clouds.
  • the graph convolution module is used to adjust the position coordinates of each point cloud.
  • the coordinate dimension of each input point cloud is increased or reduced through multiple graph convolution modules to gradually reduce the coordinate dimension of the point cloud from 96 dimensions. for 3 dimensions. Therefore, by alternately setting multiple graph convolution modules and branch modules, the graph convolutional neural network finally generates a 3D point cloud with a certain number of point clouds, and each point cloud has 3-dimensional position coordinates.
  • the branch module obeys formula (1):
  • the branch module may copy the coordinates of each point cloud in the upper layer into n pieces respectively. If there are a (i ⁇ a) point clouds in the upper layer, and the coordinates of each point cloud are copied into n, the branch module of this layer can expand the number of point clouds to a ⁇ n, and the a The ⁇ n point cloud coordinates are passed to the next layer.
  • the ResNet encoder will be an initial After the point cloud is input into the graph convolutional neural network, each branch module in the graph convolutional neural network copies the coordinates of each point cloud into n, then the predicted 3D point cloud finally generated by the graph convolutional neural network Contains n b point clouds.
  • each branch module can also be different. For example, if the expansion factor of the first layer branch module is 5, the ResNet encoder can expand an initial point cloud input into 5 point clouds. The expansion factor of the branch module of the second layer is 10, then after the second layer receives 5 point clouds, the 5 point clouds can be expanded into 50 point clouds.
  • the graph convolution module obeys formula (2):
  • Equation (2) represents the K perceptrons in the lth layer; is a fully connected layer, which represents the mapping relationship between the lth layer node and the l+1th layer node; Represents the i-th node in the l-th layer
  • the collection of all nodes (that is, ancestor nodes) of the corresponding layers 1 to l-1; is a sparse matrix; Represents the feature distribution from each ancestor node of the lth layer node to the l+1th layer node; b l is the bias parameter; ⁇ ( ⁇ ) represents the activation function.
  • the encoded feature information of the image can be effectively extracted through the ResNet encoder, and the encoded feature information can guide the graph convolutional neural network to accurately construct the 3D point cloud.
  • the 2D image of the information is reconstructed into a 3D point cloud with richer and more accurate information, which can provide doctors with more and more accurate visual information for the lesion during the diagnosis and treatment process, thereby assisting doctors to make better decisions.
  • the 3D brain point cloud model provided in this application can also reconstruct the 3D point cloud of other organs in the medical field, and can also be used in construction, manufacturing and other fields, such as rebuilding houses, 3D point clouds for crafts and more.
  • FIG. 3 is a 3D brain point cloud reconstruction training model provided by this application.
  • the model includes: ResNet encoder, graph convolutional neural network, and discriminator.
  • the graph convolutional neural network and the discriminator constitute a generative adversarial generative network.
  • the 3D point cloud and 3D point cloud samples predicted by the graph convolutional neural network are input into the discriminator to obtain the discriminant result. Iterative training is performed according to the discrimination results, the loss function of the 3D brain point cloud reconstruction model and the loss function of the discriminator, and the 3D brain point cloud reconstruction model is obtained.
  • the trained 3D brain point cloud reconstruction model can be used to construct the 3D point cloud corresponding to the 2D image of the brain.
  • the training flow chart of the 3D brain point cloud reconstruction model is shown in Figure 4.
  • the training process is as follows:
  • the training sample set includes a plurality of training samples, and each training sample includes a 2D brain image sample and a 3D point cloud sample of the brain corresponding to the 2D brain image sample.
  • First obtain a 3D image of the brain and then perform image preprocessing on the 3D image of the brain and then slice it to obtain the corresponding 2D brain image sample.
  • the corresponding 3D point cloud sample of the brain can also be obtained.
  • the 3D point cloud samples of the brain are real 3D brain point cloud images.
  • a 3D brain MRI image Take a 3D brain MRI image as an example. First obtain the real 3D brain MRI image, after preprocessing the real 3D brain MRI image, slice the 3D brain MRI image from different directions, and select the 2D slice image near the best plane as the training sample 2D brain image samples. And, a 3D point cloud sample is acquired based on the first 3D brain MRI image.
  • preprocessing the real 3D brain MRI image includes: cleaning and denoising, removing skull and removing neck.
  • the 2D slice images near the optimal plane can be selected by artificially selecting the clearest and largest 2D slice images, or selecting the 2D slice images in the middle layers as the 2D brain image samples.
  • a 2D sample image can be represented as I H ⁇ W , where H and W represent the length and width of the image, respectively.
  • I H ⁇ W into the ResNet encoder
  • ResNet can quantize the features of the input image I H ⁇ W into a Gaussian distribution vector with a specific mean ⁇ and variance ⁇ , and randomly extract a 96-dimensional encoded feature vector from the vector.
  • z ⁇ N( ⁇ , ⁇ 2 ) and pass the encoded feature vector z to the graph convolutional neural network.
  • ResNet can calculate the KL divergence by formula (3).
  • L KL is the KL divergence
  • X is the total number of Q values or P values
  • Q(x) is the xth probability distribution obtained by the encoder according to the encoding feature vector z
  • P(x) is The preset xth probability distribution.
  • the discriminator includes multiple fully connected layers.
  • the input of the discriminator is the predicted 3D point cloud and 3D point cloud samples.
  • the discriminator can judge the true and false probability of each point cloud in the predicted 3D brain point cloud. If it must be true, the probability is 1, and if it must be false, the probability is 1. is 0.
  • ResNet encoder and graph convolutional neural network use the same loss function and are trained together, while the discriminator is trained separately.
  • the ResNet encoder and graph convolutional neural network loss function is formula (4):
  • L E, G are the loss functions of the ResNet encoder and graph convolutional neural network; ⁇ 1 and ⁇ 2 are constants; L KL is the KL divergence in formula (1); Z is the ResNet The distribution of the encoded feature vector generated by the encoder; z represents the encoded feature vector, which is equivalent to Q(x); G(z) is the 3D point cloud predicted by the graph convolutional neural network; D(G(z)) represents the graph The value obtained after the 3D point cloud predicted by the convolutional neural network is input to the discriminator; E( ) represents the expectation; LCD is the chamfering distance between the 3D point cloud predicted by the graph convolutional neural network and the 3D point cloud sample ( Chamfer Distance, CD), the chamfer distance can be expressed as formula (5):
  • Y is the real coordinate matrix of all 3D point clouds
  • y is a point cloud coordinate vector in matrix Y
  • Y′ is the predicted coordinate matrix of all 3D point clouds obtained by the graph convolutional neural network
  • y ' is a point cloud coordinate vector in matrix Y'.
  • Y is an m ⁇ 3 matrix composed of m point cloud coordinates
  • y is a coordinate vector of size 1 ⁇ 3 corresponding to a point cloud in Y.
  • the loss function of the discriminator is derived from the Earth Mover Distance (EMD) loss function, which can be expressed as formula (6):
  • the loss function of the discriminant and the loss function of the 3D brain point cloud reconstruction model meet the requirements at the same time, it means that the model has converged, the initial 3D brain point cloud reconstruction model has been trained, and the trained 3D brain point cloud reconstruction model is obtained. .
  • the trained 3D brain point cloud reconstruction model can be used to construct 3D point clouds corresponding to 2D images.
  • the 3D brain point cloud reconstruction model provided by the embodiments of this application integrates the ResNet encoder and the graph convolutional neural network.
  • the discriminator is incorporated into the training model, so that the graph convolutional neural network and the discriminator form a generative adversarial generative network.
  • the ResNet encoder can effectively extract the encoded information feature vector of the input image, which provides a priori guidance for the training of the generative adversarial network, and makes the training process of the generative adversarial network easier.
  • the present application expands the number of point clouds and adjusts the position coordinates of the point clouds by alternately using the graph convolution module and the branch module, so that the 3D point cloud predicted by the graph convolutional neural network is more accurate.
  • the model is trained by integrating the chamfering distance loss function and the bulldozer distance loss function, which has a higher classification accuracy than the existing model trained only by the chamfering distance loss function.
  • Table 1 is some comparison results between the 3D brain point cloud reconstruction model provided by the application and the PointOutNet model (a 3D point cloud reconstruction model) on indicators such as chamfering distance, point-to-point error, and classification accuracy. As can be seen from Table 1, the 3D brain point cloud reconstruction model provided by this application is superior to the PointOutNet model in these three indicators.
  • FIG. 5 is a schematic structural diagram of a three-dimensional reconstruction device for brain structure provided by the present application.
  • the brain structure three-dimensional reconstruction device 500 includes: an acquisition unit 501 , a reconstruction unit 504 and a storage unit 505 .
  • the acquisition unit 501 is used to acquire a 2D image of the brain.
  • the storage unit 505 is used for storing the trained 3D brain point cloud reconstruction model.
  • the reconstruction unit 504 is used to input the 2D image of the brain into the trained 3D brain point cloud reconstruction model for processing, and output the 3D point cloud of the brain;
  • the trained 3D brain point cloud reconstruction model includes: ResNet coding
  • the ResNet encoder is used to extract the encoded feature vector of the 2D image of the brain, and the graph convolutional neural network is used to construct the 3D point cloud of the brain based on the encoded feature vector.
  • the obtaining unit 501 is further configured to obtain a 3D image of the brain, and the storage unit 505 is configured to store the training sample set.
  • the apparatus for three-dimensional reconstruction of brain structure further includes an image processing unit 502 and a training unit 503 .
  • the image processing unit 502 is used for preprocessing and slicing the 3D image of the brain obtained by the obtaining unit 501 to obtain a training sample set.
  • the training sample set includes a plurality of training samples, and each training sample includes a 2D brain image sample and 3D point cloud samples of the brain corresponding to the 2D brain image samples.
  • the preprocessing includes cleaning and denoising, removing skull and neck, slicing the preprocessed 3D image of the brain from different angles, and selecting the 2D slice image near the best plane as the 2D sample image in the training sample.
  • the training unit 503 is used for training the 3D brain point cloud reconstruction model.
  • the 2D brain image samples from the training sample are input into the initial neural network model, resulting in a predicted 3D point cloud of the brain.
  • Iterative training is performed according to the discrimination results, the loss function of the 3D brain point cloud reconstruction model and the loss function of the discriminator, and the 3D brain point cloud reconstruction model is obtained.
  • FIG. 6 is a schematic structural diagram of a 3D point cloud reconstruction device provided by the present application.
  • the device 600 may be a terminal device or a server or a chip.
  • the device 600 includes one or more processors 601, and the one or more processors 601 can support the device 600 to implement the methods described in the above method embodiments.
  • the processor 601 may be a general purpose processor or a special purpose processor.
  • the processor 601 may be a central processing unit (CPU).
  • the CPU may be used to control the device 600, execute software programs, and process data of the software programs.
  • the device 600 may include a communication unit 605 to enable input (reception) and output (transmission) of signals.
  • the device 600 can be a chip, and the communication unit 605 can be an input and/or output circuit of the chip, or the communication unit 605 can be a communication interface of the chip, and the chip can be used as a terminal device or a network device or other electronic device. component.
  • the device 600 may be a terminal device or a server, the communication unit 605 may be a transceiver of the terminal device or the server, or the communication unit 605 may be a transceiver circuit of the terminal device or the server.
  • the device 600 may include one or more memories 602 on which a program 604 is stored, and the program 604 may be executed by the processor 601 to generate instructions 603, so that the processor 601 executes the above method according to the instructions 603. method described in the example.
  • the memory 602 may also store data (eg, a 3D point cloud reconstruction model).
  • the processor 601 may also read data stored in the memory 602 , the data may be stored at the same storage address as the program 604 , or the data may be stored at a different storage address from the program 604 .
  • the processor 601 and the memory 602 may be provided separately, or may be integrated together, for example, integrated on a system on chip (system on chip, SOC) of the terminal device.
  • SOC system on chip
  • the steps in the above method embodiments may be implemented by logic circuits in the form of hardware or instructions in the form of software in the processor 601 .
  • the processor 601 may be a CPU, a digital signal processor (DSP), an application specific integrated circuit (ASIC), a field programmable gate array (FPGA) or other programmable logic devices , for example, discrete gates, transistor logic devices, or discrete hardware components.
  • DSP digital signal processor
  • ASIC application specific integrated circuit
  • FPGA field programmable gate array
  • An embodiment of the present application also provides a network device, the network device includes: at least one processor, a memory, and a computer program stored in the memory and executable on the at least one processor, the processor executing The computer program implements the steps in any of the foregoing method embodiments.
  • Embodiments of the present application further provide a computer-readable storage medium, where a computer program is stored in the computer-readable storage medium, and when the computer program is executed by a processor, the steps in the foregoing method embodiments can be implemented.
  • the embodiments of the present application provide a computer program product, when the computer program product runs on the cleaning robot, the steps in the foregoing method embodiments can be implemented when the cleaning robot executes.

Abstract

一种脑结构三维重建方法、装置及终端设备。该方法包括：获取脑部的2D图像，将所述脑部的2D图像输入到已训练的3D脑部点云重建模型中进行处理，输出得到所述脑部的3D点云；所述3D脑部点云重建模型包括：ResNet编码器和图卷积神经网络，所述ResNet编码器用于提取所述脑部的2D图像的编码特征向量，所述图卷积神经网络用于根据所述编码特征向量构建所述脑部的3D点云。基于本申请提供的脑结构三维重建方法可以将脑部的2D图像转化成脑部的3D点云，为医生提供更多的视觉信息，便于医生更好地进行诊疗。

Description

脑结构三维重建方法、装置及终端设备 技术领域
本申请属于人工智能技术领域，尤其涉及一种脑结构三维重建方法、装置及终端设备。
背景技术
近年来，随着医学手术方法的不断发展，微创手术和机器人导航手术已逐渐应用于脑外科手术，医生可以通过微型探头对手术部位进行观察，但是微型探头的视角有限，且微型探头采集的图像属于二维(Second Dimension，2D)图像，无法为医生提供更多的视觉信息，不利用医生对病变部位进行精确的诊断和分析。与2D图像的扁平空间相比，三维(Three Dimensional，3D)点云数据包含有更多的空间结构信息，可以为医生提供更多的视觉信息，从而辅助医生更好地进行诊疗。因此，将2D图像重建为准确且清晰的3D点云具有重要意义。
发明内容
本申请实施例提供了一种脑结构三维重建方法、装置及终端设备，可以将脑部的2D图像转化成3D点云，为医生提供更多的视觉信息。
第一方面，本申请实施例提供了一种脑结构三维重建方法，该方法包括：获取脑部的2D图像，将脑部的2D图像输入到已训练的3D脑部点云重建模型中进行处理，输出得到脑部的3D点云；3D脑部点云重建模型包括：ResNet编码器和图卷积神经网络，ResNet编码器用于提取脑部的2D图像的编码特征向量，图卷积神经网络用于根据编码特征向量构建脑部的3D点云。
基于本申请提供的一种脑结构三维重建方法，通过ResNet编码器可以有效地提取图像的编码特征信息，编码特征信息可以引导图卷积神经网络精准地构建3D点云，该方法可以将包含有限信息的2D图像重建成信息更加丰富、更加准确的3D点云，在诊疗的过程中可以针对病变部位为医生提供更多、更加准确地视觉信息，从而辅助医生更好地决策。
可选地，图卷积神经网络包括：多组交替设置的图卷积模块和分支模块，图卷积模块用于调整点云的位置坐标，分支模块用于扩充点云的个数。
基于上述可选方式，分支模块可以将点云的个数扩充至目标个数，图卷积模块可以调整点云的位置坐标并将坐标的维度降低至3维，使其可以正确地描述目标特征。通过交替使用图卷积模块和分支模块可以自顶向下生成3D点云，在保留祖先点云位置信息的情况下，充分利用点云相对位置，从而提高了重建的3D点云的准确性。
可选地，3D脑部点云重建模型是基于获取的训练样本集和对应的判别器训练得到的，训练样本集中包括多个训练样本，每个训练样本中包括2D脑部图像样本和2D脑部图像样本对应的脑部的3D点云样本。
可选地，3D脑部点云重建模型的训练方法包括：对于每个训练样本，将训练样本中的2D脑部图像样本输入到初始神经网络模型中，得到预测的3D点云；将预测的3D点云与训练样本中的3D点云样本输入到判别器中，得到训练样本判别结果；根据每个训练样本的判别结果、3D脑部点云重建模型的损失函数和判别器的损失函数进行迭代训练，得到3D脑部点云重建模型。
基于上述可选方式，神经网络模型中的图卷积神经网络和判别器构成生成对抗网络，在训练的过程中无需进行监督学习，降低了模型的训练复杂度，提高模型的泛化能力。
可选地，训练样本的获取方法包括：获取脑部的3D图像；对脑部的3D图像进行图像预处理后进行切片，得到2D脑部图像样本；根据3D图 像得到脑部的3D点云样本。
基于上述可选方式，对获取的3D点云图像进行预处理去除噪声后，便于后续的图像处理，在不同方向上对预处理后的3D点云图像进行切割，并选取最清晰的一张2D图像作为ResNet编码器的输入，可以提高3D脑部点云重建的准确度。
可选地，3D脑部点云重建模型对应的损失函数为L E,G＝λ 1L KL+λ 2L CD+E z～Z[D(G(z))]；
其中，L E,G表示3D脑部点云重建模型对应的损失值；λ 1和λ 2为常数；L KL表示KL散度；Z为所述ResNet编码器生成的编码特征向量的分布；z为编码特征向量；G(·)表示图卷积神经网络的输出，D(·)表示判别器，E(·)表示期望；L CD为所述初始神经网络模型预测的3D点云与3D点云样本之间的倒角距离。
可选地，判别器对应的损失函数为：
Figure PCTCN2021070934-appb-000001
其中，
Figure PCTCN2021070934-appb-000002
表示3D点云样本与所述初始神经网络模型预测的3D点云之间线性分割的采样，
Figure PCTCN2021070934-appb-000003
E(·)为期望；G(·)表示图卷积神经网络的输出，D(·)表示判别器；Y表示3D点云样本，R表示3D点云样本分布；λ gp为常数；
Figure PCTCN2021070934-appb-000004
为梯度算子。
基于上述可选方式，融合倒角距离损失函数和推土机距离损失函数构建3D脑部点云重建模型的损失函数，比现有的仅通过倒角距离作为损失函数进行训练得到的模型的分类准确率更高，可以提高网络的精度，同时可以避免3D点云的边缘失真，提高点云图像的生成质量。
第二方面，本申请实施例提供了一种脑结构三维重建装置，该装置包括：获取单元，用于获取脑部的2D图像；重建单元，用于将所述脑部的2D图像输入到已训练的3D脑部点云重建模型中进行处理，输出得到所述 脑部的3D点云；所述3D脑部点云重建模型包括：ResNet编码器和图卷积神经网络，所述ResNet编码器用于提取所述脑部的2D图像的编码特征向量，所述图卷积神经网络用于根据所述编码特征向量构建所述脑部的3D点云。
第三方面，本申请实施例提供了一种终端设备，包括存储器、处理器以及存储在存储器中并可在处理器上运行的计算机程序，处理器执行所述计算机程序时实现第一方面中的任意一种方法。
第四方面，本申请实施例提供了一种计算机可读存储介质，计算机可读存储介质存储有计算机程序，计算机程序被处理器执行时实现第一方面中的任意一种方法。
第五方面，本申请实施例提供一种计算机程序产品，当计算机程序产品在处理器上运行时，使得处理器执行上述第一方面中任一项所述的方法
可以理解的是，上述第二方面至第五方面的有益效果可以参见上述第一方面中的相关描述，在此不再赘述。
附图说明
为了更清楚地说明本申请实施例中的技术方案，下面将对实施例或现有技术描述中所需要使用的附图作简单地介绍，显而易见地，下面描述中的附图仅仅是本申请的一些实施例，对于本领域普通技术人员来讲，在不付出创造性劳动性的前提下，还可以根据这些附图获得其他的附图。
图1是本申请提供的一种3D脑部点云重建模型的结构示意图；
图2是本申请提供的一种脑结构三维重建方法的流程示意图；
图3是本申请提供的一种3D脑部点云重建训练模型的结构示意图；
图4是本申请提供的一种3D脑部点云重建模型的训练流程示意图；
图5是本申请提供的一种脑结构三维重建装置的结构示意图；
图6是本申请提供的一种终端设备的结构示意图。
具体实施方式
以下描述中，为了说明而不是为了限定，提出了诸如特定系统结构、技术之类的具体细节，以便透彻理解本申请实施例。然而，本领域的技术人员应当清楚，在没有这些具体细节的其它实施例中也可以实现本申请。在其它情况中，省略对众所周知的系统、装置、电路以及方法的详细说明，以免不必要的细节妨碍本申请的描述。
应当理解，在本申请说明书和所附权利要求书中使用的术语“和/或”是指相关联列出的项中的一个或多个的任何组合以及所有可能组合，并且包括这些组合。
另外，在本申请说明书和所附权利要求书的描述中，术语“第一”、“第二”、“第三”等仅用于区分描述，而不能理解为指示或暗示相对重要性。
在本申请说明书中描述的参考“一个实施例”或“一些实施例”等意味着在本申请的一个或多个实施例中包括结合该实施例描述的特定特征、结构或特点。由此，在本说明书中的不同之处出现的语句“在一个实施例中”、“在一些实施例中”、“在其他一些实施例中”、“在另外一些实施例中”等不是必然都参考相同的实施例，而是意味着“一个或多个但不是所有的实施例”，除非是以其他方式另外特别强调。术语“包括”、“包含”、“具有”及它们的变形都意味着“包括但不限于”，除非是以其他方式另外特别强调。
近年来，随着医学手术方法的不断发展，微创手术和机器人导航手术已逐渐应用于脑外科手术，医生可以通过微型探头对手术部位进行观察，但是微型探头的视角有限，微型探头采集的图像属于2D图像，无法为医生提供更多的视觉信息，不利用医生对病变部位进行精确的诊断和分析。
点云是一种描述三维空间中特定对象形状结构的一种数据结构，具有空间复杂度小、储存形式简单和计算性能高等优点。与2D图像的扁平空间相比， 3D点云数据包含有更多的空间结构信息，可以为医生提供更多的视觉信息，从而辅助医生更好地进行诊疗。因此，将2D图像重建为准确且清晰的3D点云具有重要意义。
为此，本申请提供了一种脑结构三维重建方法、装置及终端设备。可以将脑部的2D图像转化成3D点云，为医生提供更好的视觉信息，从而可以更好地进行诊疗。
下面结合附图对本申请提供的3D脑部点云重建模型及其训练方法，和脑结构三维重建方法进行详细介绍。
图1是本申请提供的一种3D脑部点云重建模型。该模型包括：ResNet(残差网络)编码器和图卷积神经网络(Graph Convolutional Network，GCN)。其中，图卷积神经网络作为3D脑部点云重建模型的生成器，包括多组交替设置的分支模块和图卷积模块。
在本实施例中，将脑部的2D图像输入到ResNet编码器中，ResNet编码器可以提取2D图像的编码特征向量。ResNet编码器先将2D图像量化成具有一定均值和方差的且服从高斯分布的特征向量，再从特征向量中随机抽取预设维度的高维编码特征向量(例如，96维的编码特征向量)，并将编码特征向量传递给图卷积神经网络。该编码特征向量即为输入图卷积神经网络的一个初始点云，坐标维度为96。
图卷积神经网络中，分支模块用于扩充点云的个数，图卷积模块用于调整每个点云的位置坐标，交替使用分支模块和图卷积模块可以精准地重建出脑部的3D点云。
在一个实施例中，脑部的2D图像可以是以任意角度拍摄的MRI(Magnetic Resonance Imaging)、CT(Computed Tomography)、PET(Positron Emission Computed Tomography)、DTI(Diffusion Tensor Imaging)或FMRI(Functional Magnetic Resonance Imaging)等图像。
图2为本申请提供的一种脑结构三维重建方法的一个实施例的流程图， 该方法的执行主体可以是影像数据采集设备，例如正电子发射型计算机断层显像PET设备、CT设备或者MRI设备等终端设备。还可以是影像数据采集设备的控制设备、计算机、机器人、移动终端等终端设备。如图2所示，该方法包括：
S201，获取脑部的2D图像。
该2D图像的大小满足ResNet编码器的输入要求。2D图像可以是以任意角度拍摄的MRI、CT、PET、DTI或者fMRI等脑部影像。当然，为了得到更精确的3D点云，可以选择具有更多脑部特征的角度拍摄得到2D图像。
S202，将脑部的2D图像输入到ResNet编码器中，得到编码特征向量。
在本实施例中，ResNet编码器先将脑部的2D图像量化成具有一定均值μ和方差σ的且服从高斯分布的特征向量，再从特征向量中随机抽取96维的编码特征向量z，并将编码特征向量z传递给图卷积神经网络。该编码特征向量作为输入图卷积神经网络的初始点云，数量为1，坐标维度为96。
S203，图卷积神经网络根据编码特征向量构建脑部的3D点云。
如图1所示，图卷积神经网络包括多个交替设置的分支模块和图卷积模块，其中，分支模块可以将一个点云映射成多个点云，那么，通过多个分支模块可以将1个初始点云逐渐扩充到目标个数的点云。图卷积模块用于调整各个点云的位置坐标，通过多个图卷积模块对输入的每个点云的坐标维度进行升维或者降维，以逐渐将点云的坐标维度从96维降为3维。因此，通过交替设置的多个图卷积模块和分支模块，使得图卷积神经网络最终生成一个具有特定点云个数的3D点云，每个点云具有3维的位置坐标。
其中，分支模块服从公式(1)：
Figure PCTCN2021070934-appb-000005
公式(1)中，
Figure PCTCN2021070934-appb-000006
表示图卷积神经网络第l层网络中的第i个点云；
Figure PCTCN2021070934-appb-000007
表示图卷积神经网络第l+1层网络中的第i个点云；
Figure PCTCN2021070934-appb-000008
表示图卷积神经网络第l+1层网络中的第i+1个点云；
Figure PCTCN2021070934-appb-000009
表示图卷积神经网络第l+1层网络中的第 i+n个点云。
也就是说，在本实施例中，分支模块可以将上层中的每个点云的坐标分别复制成n个。若上层有a(i∈a)个点云，将每个点云的坐标复制成n个，则本层的分支模块可以将点云的个数扩充为a×n个，并将所述a×n个点云坐标传递给下一层。若图卷积神经网络中包括b(l∈b，且b≥1，b为正整数)b个分支模块，且每个分支模块的扩展倍数相同，均为n，则ResNet编码器将一个初始点云输入到图卷积神经网络中后，图卷积神经网络中的每个分支模块将每个点云的坐标复制成n个，则图卷积神经网络最终生成的预测的3D点云中包含n b个点云。
当然，每个分支模块的扩展倍数也可以不同。比如说第一层分支模块的扩展倍数是5，可以将ResNet编码器将输入的一个初始点云扩充为5个点云。第二层分支模块的扩展倍数是10，则第二层接收到5个点云后即可将5个点云扩充为50个点云。
图卷积模块服从公式(2)：
Figure PCTCN2021070934-appb-000010
公式(2)中，
Figure PCTCN2021070934-appb-000011
表示第l层中的K个感知机；
Figure PCTCN2021070934-appb-000012
为全连接层，表示第l层节点与第l+1层节点之间的映射关系；
Figure PCTCN2021070934-appb-000013
表示与第l层中第i个节点
Figure PCTCN2021070934-appb-000014
对应的第1至l-1层的所有节点(即祖先节点)的合集；
Figure PCTCN2021070934-appb-000015
为稀疏矩阵；
Figure PCTCN2021070934-appb-000016
表示第l层节点的各祖先节点到第l+1层节点的特征分布；b l为偏置参数；σ(·)表示激活函数。
基于本申请提供的一种3D点云重建方法，通过ResNet编码器可以有效地提取图像的编码特征信息，编码特征信息可以引导图卷积神经网络精准地构建3D点云，该方法可以将包含有限信息的2D图像重建成信息更加丰富、更加准确的3D点云，在诊疗的过程中可以针对病变部位为医生提供更多、更加准确地视觉信息，从而辅助医生更好地决策。
当然，除了重建3D脑部点云以外，本申请提供的3D脑部点云模型也可以重建医学领域中其它各个器官的3D点云，还可以应用于建筑、制造业等领域，例如重建房屋、工艺品等的3D点云。
图3是本申请提供的一种3D脑部点云重建训练模型。该模型包括：ResNet编码器、图卷积神经网络和判别器。其中，图卷积神经网络和判别器构成生成对抗生成网络。将图卷积神经网络预测的3D点云和3D点云样本输入到判别器中，得到判别结果。根据判别结果、3D脑部点云重建模型的损失函数和判别器的损失函数进行迭代训练，得到3D脑部点云重建模型。训练好的3D脑部点云重建模型可用于构建脑部的2D图像对应的3D点云。
3D脑部点云重建模型的训练流程图如图4所示。训练过程如下：
S401，获取训练样本集。
训练样本集中包括多个训练样本，每个训练样本包括2D脑部图像样本和与2D脑部图像样本对应的脑部的3D点云样本。首先获取脑部的3D图像，然后对脑部的3D图像进行图像预处理后进行切片，得到对应2D脑部图像样本，根据脑部的3D图像还可以得到对应脑部的3D点云样本，脑部的3D点云样本为真实的3D脑部点云图像。
示例性的，以3D脑部MRI图像为例。首先获取真实的3D脑部MRI图像，在对真实的3D脑部MRI图像进行预处理后，从不同方向上对3D脑部MRI图像进行切片，选取最佳平面附近的2D切片图像作为训练样本中的2D脑部图像样本。并且，基于第一3D脑部MRI图像获取3D点云样本。
在一个实施例中，对真实的3D脑部MRI图像进行预处理包括：清洗去噪、去除颅骨和去除脖骨。
在一个实施例中，最佳平面附近的2D切片图像可以通过人为的选取最清晰且最大的2D切片图像，或者选取中间几层的2D切片图像作为2D脑部图像样本。
S402，通过ResNet编码器提取训练样本集的编码特征向量。
在一种可能的实现方式中，一个2D样本图像可以表示为I H×W，其中H和W分别表示图像的长度和宽度。将I H×W输入到ResNet编码器中，ResNet可以将输入图像I H×W的特征量化成一个具有特定均值μ和方差σ的高斯分布向量，并从向量中随机抽取96维的编码特征向量z～N(μ,σ 2)，并将编码特征向量z传递给图卷积神经网络。ResNet可以通过公式(3)计算KL散度。
Figure PCTCN2021070934-appb-000017
公式(3)中，L KL为KL散度；X为Q值或P值的总个数；Q(x)为编码器根据编码特征向量z得到的第x个概率分布；P(x)为预设的第x个概率分布。
S403，将编码特征向量输入到图卷积神经网络中，得到预测的3D点云。
该步骤具体地实现方式如上述S203所述，此处不再赘述。
S404，将预测的3D点云与3D点云样本输入到判别器中进行训练。
在本实施例中，如图3所示，判别器包括多个全连接层。判别器的输入为预测的3D点云与3D点云样本，判别器可以判断预测的3D脑部点云中每个点云的真假概率，一定为真则概率为1，一定为假则概率为0。并根据点云的实际真假情况计算预测的3D点云G(z)与3D点云样本Y之间差值，差值可以表示为公式G(z)-Y。
在训练的过程中，ResNet编码器和图卷积神经网络使用相同的损失函数并一同训练，判别器则单独训练。ResNet编码器和图卷积神经网络损失函数为公式(4)：
L E,G＝λ 1L KL+λ 2L CD+E z～Z[D(G(z))]                (4)
公式(4)中，L E,G为ResNet编码器和图卷积神经网络的损失函数；λ 1和λ 2为常数；L KL为公式(1)中的KL散度；Z为所述ResNet编码器生成的编码特征向量的分布；z表示所述编码特征向量，相当于Q(x)；G(z)为图卷积神经网络预测的3D点云；D(G(z))表示图卷积神经网络预测的3D点云输入 到判别器后得到的值；E(·)表示期望；L CD为图卷积神经网络预测的3D点云与3D点云样本之间的倒角距离(Chamfer Distance，CD)，该倒角距离可以表示为公式(5)：
Figure PCTCN2021070934-appb-000018
在公式(5)中，Y为真实的所有3D点云坐标矩阵，y为矩阵Y中的一个点云坐标向量；Y′为图卷积神经网络得到的预测的所有3D点云坐标矩阵，y′为矩阵Y′中的一个点云坐标向量。示例性的，若Y为由m个点云坐标组成的m×3的矩阵，则y为Y中一个点云对应的大小为1×3的坐标向量。
判别器的损失函数由推土机距离(Earth Mover Distance，EMD)损失函数推导而成，具体可以表示为公式(6)：
Figure PCTCN2021070934-appb-000019
在公式(6)中，
Figure PCTCN2021070934-appb-000020
表示3D点云样本与预测的3D点云之间线性分割的采样，即3D点云样本与预测的3D点云之间的差值，
Figure PCTCN2021070934-appb-000021
E(·)为期望；D(G(z))表示图卷积神经网络预测的3D点云G(z)输入到判别器后得到的值；D(Y)表示3D点云样本Y输入到判别器后得到的值；R为3D点云样本分布；λ gp为常数；
Figure PCTCN2021070934-appb-000022
为梯度算子。
当判别其的损失函数和3D脑部点云重建模型的损失函数同时满足要求时，表示模型已经收敛，初始3D脑部点云重建模型已经完成训练，得到训练好的3D脑部点云重建模型。
训练好的3D脑部点云重建模型可以用于构建2D图像对应的3D点云。本申请实施例提供的3D脑部点云重建模型融合了ResNet编码器和图卷积神经网络。在训练模型中融入判别器，使图卷积神经网络和判别器构成生成对抗生成网络。ResNet编码器可以有效提取输入图像的编码信息特征向量，为生成对抗网络的训练提供了先验指导，使生成对看网络的训练过程更加简便。且本申请通过交替使用图卷积模块和分支模块扩充点云的个数以及调整点云 的位置坐标，使得图卷积神经网络预测的3D点云更加精确。在训练的过程中，将倒角距离损失函数与推土机距离损失函数进行融合对模型进行训练，比现有的仅通过倒角距离损失函数进行训练得到的模型的分类准确率更高。
表1为本申请提供的3D脑部点云重建模型与PointOutNet模型(一种3D点云重建模型)在倒角距离、点到点误差和分类准确率等指标上的一些对比结果。从表1中可以看出，本申请提供的3D脑部点云重建模型在这三种指标上都优于PointOutNet模型。
表1
Figure PCTCN2021070934-appb-000023
图5是本申请提供的一种脑结构三维重建装置的结构示意图。该脑结构三维重建装置500包括：获取单元501和重建单元504和存储单元505。获取单元501用于获取脑部的2D图像。存储单元505用于存储已训练的3D脑部点云重建模型。重建单元504用于将脑部的2D图像输入到已训练的3D脑部点云重建模型中进行处理，输出得到脑部的3D点云；已训练的3D脑部点云重建模型包括：ResNet编码器和图卷积神经网络，ResNet编码器用于提取脑部的2D图像的编码特征向量，图卷积神经网络用于根据编码特征向量构建脑部的3D点云。
在一个实施例中，获取单元501还用于获取脑部的3D图像，存储单元505用于存储训练样本集。
在一种可能的实现方式中，脑结构三维重建装置还包括图像处理单元502 和训练单元503。
其中，图像处理单元502用于对获取单元501获取到的脑部的3D图像进行预处理和切片，得到训练样本集，训练样本集中包括多个训练样本，每个训练样本包括2D脑部图像样本和与2D脑部图像样本对应的脑部的3D点云样本。预处理包括清洗去噪、去除颅骨和去除脖骨，从不同角度对预处理后的脑部的3D图像进行切片，选取最佳平面附近的2D切片图像作为训练样本中的2D样本图像。
训练单元503用于训练3D脑部点云重建模型。对于每个训练样本，将训练样本中的2D脑部图像样本输入到初始神经网络模型中，得到预测的脑部的3D点云。将预测的脑部的3D点云与训练样本中的脑部的3D点云样本输入到判别器中，得到判别结果。根据判别结果、3D脑部点云重建模型的损失函数和判别器的损失函数进行迭代训练，得到3D脑部点云重建模型。
图6是本申请提供的一种3D点云重建设备的结构示意图。设备600可以是终端设备或服务器或芯片。设备600包括一个或多个处理器601，该一个或多个处理器601可支持设备600实现上述方法实施例中描述的方法。处理器601可以是通用处理器或者专用处理器。例如，处理器601可以是中央处理器(central processing unit，CPU)。CPU可以用于对设备600进行控制，执行软件程序，处理软件程序的数据。
在一个是实施例中，设备600可以包括通信单元605，用以实现信号的输入(接收)和输出(发送)。例如，设备600可以是芯片，通信单元605可以是该芯片的输入和/或输出电路，或者，通信单元605可以是该芯片的通信接口，该芯片可以作为终端设备或网络设备或其它电子设备的组成部分。又例如，设备600可以是终端设备或服务器，通信单元605可以是该终端设备或该服务器的收发器，或者，通信单元605可以是该终端设备或该服务器的收发电路。
在另一个实施例中，设备600中可以包括一个或多个存储器602，其上存 有程序604，程序604可被处理器601运行，生成指令603，使得处理器601根据指令603执行上述方法实施例中描述的方法。
在其它实施例中，存储器602中还可以存储有数据(例如3D点云重建模型)。可选地，处理器601还可以读取存储器602中存储的数据，该数据可以与程序604存储在相同的存储地址，该数据也可以与程序604存储在不同的存储地址。
处理器601和存储器602可以单独设置，也可以集成在一起，例如，集成在终端设备的系统级芯片(system on chip，SOC)上。
处理器601执行3D点云重建方法的具体方式可以参见上述实施例中的相关描述。
应理解，上述方法实施例的各步骤可以通过处理器601中的硬件形式的逻辑电路或者软件形式的指令完成。处理器601可以是CPU、数字信号处理器(digital signal processor，DSP)、专用集成电路(application specific integrated circuit，ASIC)、现场可编程门阵列(field programmable gate array，FPGA)或者其它可编程逻辑器件，例如，分立门、晶体管逻辑器件或分立硬件组件。
本申请实施例还提供了一种网络设备，该网络设备包括：至少一个处理器、存储器以及存储在所述存储器中并可在所述至少一个处理器上运行的计算机程序，所述处理器执行所述计算机程序时实现上述任意各个方法实施例中的步骤。
本申请实施例还提供了一种计算机可读存储介质，所述计算机可读存储介质存储有计算机程序，所述计算机程序被处理器执行时实现可实现上述各个方法实施例中的步骤。
本申请实施例提供了一种计算机程序产品，当计算机程序产品在清洁机器人上运行时，使得清洁机器人执行时实现可实现上述各个方法实施例中的步骤。
以上所述实施例仅用以说明本申请的技术方案，而非对其限制；尽管参 照前述实施例对本申请进行了详细的说明，本领域的普通技术人员应当理解：其依然可以对前述各实施例所记载的技术方案进行修改，或者对其中部分技术特征进行等同替换；而这些修改或者替换，并不使相应技术方案的本质脱离本申请各实施例技术方案的精神和范围，均应包含在本申请的保护范围之内。

Claims (10)

  1. 一种脑结构三维重建方法，其特征在于，所述方法包括：
    获取脑部的2D图像，将所述脑部的2D图像输入到已训练的3D脑部点云重建模型中进行处理，输出得到所述脑部的3D点云；
    所述3D脑部点云重建模型包括：ResNet编码器和图卷积神经网络，所述ResNet编码器用于提取所述脑部的2D图像的编码特征向量，所述图卷积神经网络用于根据所述编码特征向量构建所述脑部的3D点云。
  2. 根据权利要求1所述的方法，其特征在于，所述图卷积神经网络包括：多组交替设置的图卷积模块和分支模块，所述图卷积模块用于调整点云的位置坐标，所述分支模块用于扩充点云的个数。
  3. 根据权利要求1或2所述的方法，其特征在于，所述3D脑部点云重建模型是基于获取的训练样本集和对应的判别器训练得到的，所述训练样本集中包括多个训练样本，每个训练样本中包括的2D脑部图像样本和所述2D脑部图像样本对应的脑部的3D点云样本。
  4. 根据权利要求3所述的方法，其特征在于，所述3D脑部点云重建模型的训练方法包括：
    对于每个训练样本，将所述训练样本中的2D脑部图像样本输入到初始神经网络模型中，得到预测的3D点云；
    将所述预测的3D点云与所述训练样本中的3D点云样本输入到所述判别器中处理，得到所述训练样本的判别结果；
    根据每个训练样本的判别结果、所述3D脑部点云重建模型的损失函数和所述判别器的损失函数进行迭代训练，得到所述3D脑部点云重建模型。
  5. 根据权利要求4所述的方法，其特征在于，所述3D脑部点云重建模型的损失函数为：L E,G＝λ 1L KL+λ 2L CD+E z～Z[D(G(z))]；
    其中，L E,G表示3D脑部点云重建模型对应的损失值；λ 1和λ 2为常数； L KL表示KL散度；Z为所述ResNet编码器生成的编码特征向量的分布；z表示所述编码特征向量；G(·)表示图卷积神经网络的输出，D(·)表示判别器，E(·)表示期望；L CD为所述初始神经网络模型预测的3D点云与3D点云样本之间的倒角距离。
  6. 根据权利要求4所述的方法，其特征在于，所述判别器的损失函数为：
    Figure PCTCN2021070934-appb-100001
    其中，
    Figure PCTCN2021070934-appb-100002
    表示3D点云样本与所述初始神经网络模型预测的3D点云之间线性分割的采样，
    Figure PCTCN2021070934-appb-100003
    E(·)为期望；G(·)表示图卷积神经网络的输出，D(·)表示判别器；Y表示3D点云样本，R表示3D点云样本分布；λ gp为常数；
    Figure PCTCN2021070934-appb-100004
    为梯度算子。
  7. 根据权利要求3所述的方法，其特征在于，训练样本的获取方法包括：
    获取脑部的3D图像；
    对所述脑部的3D图像进行图像预处理后进行切片，得到2D脑部图像样本；
    根据所述3D图像得到所述脑部的3D点云样本。
  8. 一种脑结构三维重建装置，其特征在于，包括：
    获取单元，用于获取脑部的2D图像；
    重建单元，用于将所述脑部的2D图像输入到已训练的3D脑部点云重建模型中进行处理，输出得到所述脑部的3D点云；所述3D脑部点云重建模型包括：ResNet编码器和图卷积神经网络，所述ResNet编码器用于提取所述脑部的2D图像的编码特征向量，所述图卷积神经网络用于根据所述编码特征向量构建所述脑部的3D点云。
  9. 一种终端设备，包括存储器、处理器以及存储在所述存储器中并可在所述处理器上运行的计算机程序，其特征在于，所述处理器执行所述计 算机程序时实现如权利要求1至9任一项所述的方法。
  10. 一种计算机可读存储介质，所述计算机可读存储介质存储有计算机程序，其特征在于，所述计算机程序被处理器执行时实现如权利要求1至9任一项所述的方法。
PCT/CN2021/070934 2021-01-08 2021-01-08 脑结构三维重建方法、装置及终端设备 WO2022147783A1 (zh)

Priority Applications (2)

Application Number Priority Date Filing Date Title
US18/026,498 US20230343026A1 (en) 2021-01-08 2021-01-08 Method and device for three-dimensional reconstruction of brain structure, and terminal equipment
PCT/CN2021/070934 WO2022147783A1 (zh) 2021-01-08 2021-01-08 脑结构三维重建方法、装置及终端设备

Applications Claiming Priority (1)

Application Number Priority Date Filing Date Title
PCT/CN2021/070934 WO2022147783A1 (zh) 2021-01-08 2021-01-08 脑结构三维重建方法、装置及终端设备

Publications (1)

Publication Number Publication Date
WO2022147783A1 true WO2022147783A1 (zh) 2022-07-14

Family

ID=82357783

Family Applications (1)

Application Number Title Priority Date Filing Date
PCT/CN2021/070934 WO2022147783A1 (zh) 2021-01-08 2021-01-08 脑结构三维重建方法、装置及终端设备

Country Status (2)

Country Link
US (1) US20230343026A1 (zh)
WO (1) WO2022147783A1 (zh)

Citations (5)

* Cited by examiner, † Cited by third party
Publication number Priority date Publication date Assignee Title
CN109389671A (zh) * 2018-09-25 2019-02-26 南京大学 一种基于多阶段神经网络的单图像三维重建方法
US20190130562A1 (en) * 2017-11-02 2019-05-02 Siemens Healthcare Gmbh 3D Anisotropic Hybrid Network: Transferring Convolutional Features from 2D Images to 3D Anisotropic Volumes
US20200027269A1 (en) * 2018-07-23 2020-01-23 Fudan University Network, System and Method for 3D Shape Generation
CN111382300A (zh) * 2020-02-11 2020-07-07 山东师范大学 基于组对深度特征学习的多视图三维模型检索方法及系统
CN111598998A (zh) * 2020-05-13 2020-08-28 腾讯科技（深圳）有限公司 三维虚拟模型重建方法、装置、计算机设备和存储介质

Patent Citations (5)

* Cited by examiner, † Cited by third party
Publication number Priority date Publication date Assignee Title
US20190130562A1 (en) * 2017-11-02 2019-05-02 Siemens Healthcare Gmbh 3D Anisotropic Hybrid Network: Transferring Convolutional Features from 2D Images to 3D Anisotropic Volumes
US20200027269A1 (en) * 2018-07-23 2020-01-23 Fudan University Network, System and Method for 3D Shape Generation
CN109389671A (zh) * 2018-09-25 2019-02-26 南京大学 一种基于多阶段神经网络的单图像三维重建方法
CN111382300A (zh) * 2020-02-11 2020-07-07 山东师范大学 基于组对深度特征学习的多视图三维模型检索方法及系统
CN111598998A (zh) * 2020-05-13 2020-08-28 腾讯科技（深圳）有限公司 三维虚拟模型重建方法、装置、计算机设备和存储介质

Also Published As

Publication number Publication date
US20230343026A1 (en) 2023-10-26

Similar Documents

Publication Publication Date Title
US10964071B2 (en) Systems and methods for data and model-driven image reconstruction and enhancement
KR102210474B1 (ko) 양전자방출 단층촬영 시스템 및 그것을 이용한 영상 재구성 방법
US7844087B2 (en) Method for segmentation of lesions
CN109523584A (zh) 图像处理方法、装置、多模态成像系统、存储介质及设备
CN112598790A (zh) 脑结构三维重建方法、装置及终端设备
CN109598697A (zh) 二维乳腺造影数据组的确定
WO2023044605A1 (zh) 极端环境下脑结构的三维重建方法、装置及可读存储介质
CN110251231A (zh) 超声三维重建的方法及装置
Yasmin et al. Brain image reconstruction: A short survey
Qiu et al. A deep learning approach for segmentation, classification, and visualization of 3-D high-frequency ultrasound images of mouse embryos
WO2022147783A1 (zh) 脑结构三维重建方法、装置及终端设备
CN108596900B (zh) 甲状腺相关性眼病医学影像数据处理装置、方法、计算机可读存储介质及终端设备
CN114913262B (zh) 采样模式与重建算法联合优化的核磁共振成像方法及系统
WO2022163513A1 (ja) 学習済みモデルの生成方法、機械学習システム、プログラムおよび医療画像処理装置
Beetz et al. Reconstructing 3D cardiac anatomies from misaligned multi-view magnetic resonance images with mesh deformation U-nets
CN115908610A (zh) 一种基于单模态pet图像获取衰减校正系数图像的方法
CN115861464A (zh) 基于多模mri同步生成的伪ct合成方法
Oulbacha et al. MRI to C‐arm spine registration through Pseudo‐3D CycleGANs with differentiable histograms
CN112669450B (zh) 人体模型构建方法和个性化人体模型构建方法
CN113052840B (zh) 一种基于低信噪比pet图像的处理方法
CN114972026A (zh) 图像处理方法和存储介质
WO2024055493A1 (zh) 基于深度相位相关的异构三维观测配准方法、介质及设备
US20230206477A1 (en) Image processing method, image processing device, program, and trained model
Joseph et al. LSOA Optimized Curve Fitting-based 3D Reconstruction of Brain Tumor
Labate et al. Leveraging Model and Data-Driven Methods in Medical Imaging

Legal Events

Date Code Title Description
121 Ep: the epo has been informed by wipo that ep was designated in this application

Ref document number: 21916838

Country of ref document: EP

Kind code of ref document: A1

NENP Non-entry into the national phase

Ref country code: DE

122 Ep: pct application non-entry in european phase

Ref document number: 21916838

Country of ref document: EP

Kind code of ref document: A1