CN117392312A - New view image generation method of monocular endoscope based on deformable nerve radiation field - Google Patents
New view image generation method of monocular endoscope based on deformable nerve radiation field Download PDFInfo
- Publication number
- CN117392312A CN117392312A CN202311280252.XA CN202311280252A CN117392312A CN 117392312 A CN117392312 A CN 117392312A CN 202311280252 A CN202311280252 A CN 202311280252A CN 117392312 A CN117392312 A CN 117392312A
- Authority
- CN
- China
- Prior art keywords
- image
- radiation field
- monocular endoscope
- depth
- deformable
- Prior art date
- Legal status (The legal status is an assumption and is not a legal conclusion. Google has not performed a legal analysis and makes no representation as to the accuracy of the status listed.)
- Pending
Links
- 238000000034 method Methods 0.000 title claims abstract description 54
- 230000005855 radiation Effects 0.000 title claims abstract description 40
- 210000005036 nerve Anatomy 0.000 title claims abstract description 28
- 238000009877 rendering Methods 0.000 claims abstract description 9
- 238000004519 manufacturing process Methods 0.000 claims abstract description 7
- 230000000007 visual effect Effects 0.000 claims abstract description 6
- 238000012549 training Methods 0.000 claims description 15
- 230000001537 neural effect Effects 0.000 claims description 13
- 238000005070 sampling Methods 0.000 claims description 8
- 230000006870 function Effects 0.000 description 18
- 238000004590 computer program Methods 0.000 description 7
- 238000010586 diagram Methods 0.000 description 7
- 238000003860 storage Methods 0.000 description 6
- 238000012986 modification Methods 0.000 description 4
- 230000004048 modification Effects 0.000 description 4
- 238000012545 processing Methods 0.000 description 4
- 210000004872 soft tissue Anatomy 0.000 description 3
- 230000004075 alteration Effects 0.000 description 2
- 230000015572 biosynthetic process Effects 0.000 description 2
- 230000003287 optical effect Effects 0.000 description 2
- 238000005457 optimization Methods 0.000 description 2
- 238000003786 synthesis reaction Methods 0.000 description 2
- 238000012360 testing method Methods 0.000 description 2
- 230000009286 beneficial effect Effects 0.000 description 1
- 238000007796 conventional method Methods 0.000 description 1
- 238000003745 diagnosis Methods 0.000 description 1
- 210000002249 digestive system Anatomy 0.000 description 1
- 201000010099 disease Diseases 0.000 description 1
- 208000037265 diseases, disorders, signs and symptoms Diseases 0.000 description 1
- 238000000605 extraction Methods 0.000 description 1
- 238000001727 in vivo Methods 0.000 description 1
- 210000000936 intestine Anatomy 0.000 description 1
- 230000003902 lesion Effects 0.000 description 1
- 230000007721 medicinal effect Effects 0.000 description 1
- 238000003062 neural network model Methods 0.000 description 1
- 230000008447 perception Effects 0.000 description 1
- 238000010882 preoperative diagnosis Methods 0.000 description 1
- 238000011160 research Methods 0.000 description 1
- 210000002784 stomach Anatomy 0.000 description 1
- 238000013526 transfer learning Methods 0.000 description 1
Classifications
-
- G—PHYSICS
- G06—COMPUTING; CALCULATING OR COUNTING
- G06T—IMAGE DATA PROCESSING OR GENERATION, IN GENERAL
- G06T17/00—Three dimensional [3D] modelling, e.g. data description of 3D objects
-
- G—PHYSICS
- G06—COMPUTING; CALCULATING OR COUNTING
- G06N—COMPUTING ARRANGEMENTS BASED ON SPECIFIC COMPUTATIONAL MODELS
- G06N3/00—Computing arrangements based on biological models
- G06N3/02—Neural networks
- G06N3/08—Learning methods
- G06N3/0895—Weakly supervised learning, e.g. semi-supervised or self-supervised learning
-
- G—PHYSICS
- G06—COMPUTING; CALCULATING OR COUNTING
- G06T—IMAGE DATA PROCESSING OR GENERATION, IN GENERAL
- G06T7/00—Image analysis
- G06T7/50—Depth or shape recovery
- G06T7/55—Depth or shape recovery from multiple images
-
- G—PHYSICS
- G06—COMPUTING; CALCULATING OR COUNTING
- G06T—IMAGE DATA PROCESSING OR GENERATION, IN GENERAL
- G06T7/00—Image analysis
- G06T7/60—Analysis of geometric attributes
- G06T7/62—Analysis of geometric attributes of area, perimeter, diameter or volume
-
- G—PHYSICS
- G06—COMPUTING; CALCULATING OR COUNTING
- G06T—IMAGE DATA PROCESSING OR GENERATION, IN GENERAL
- G06T7/00—Image analysis
- G06T7/70—Determining position or orientation of objects or cameras
- G06T7/73—Determining position or orientation of objects or cameras using feature-based methods
-
- G—PHYSICS
- G06—COMPUTING; CALCULATING OR COUNTING
- G06T—IMAGE DATA PROCESSING OR GENERATION, IN GENERAL
- G06T7/00—Image analysis
- G06T7/80—Analysis of captured images to determine intrinsic or extrinsic camera parameters, i.e. camera calibration
-
- G—PHYSICS
- G06—COMPUTING; CALCULATING OR COUNTING
- G06T—IMAGE DATA PROCESSING OR GENERATION, IN GENERAL
- G06T2207/00—Indexing scheme for image analysis or image enhancement
- G06T2207/10—Image acquisition modality
- G06T2207/10024—Color image
-
- G—PHYSICS
- G06—COMPUTING; CALCULATING OR COUNTING
- G06T—IMAGE DATA PROCESSING OR GENERATION, IN GENERAL
- G06T2207/00—Indexing scheme for image analysis or image enhancement
- G06T2207/10—Image acquisition modality
- G06T2207/10028—Range image; Depth image; 3D point clouds
-
- G—PHYSICS
- G06—COMPUTING; CALCULATING OR COUNTING
- G06T—IMAGE DATA PROCESSING OR GENERATION, IN GENERAL
- G06T2207/00—Indexing scheme for image analysis or image enhancement
- G06T2207/10—Image acquisition modality
- G06T2207/10068—Endoscopic image
-
- G—PHYSICS
- G06—COMPUTING; CALCULATING OR COUNTING
- G06T—IMAGE DATA PROCESSING OR GENERATION, IN GENERAL
- G06T2207/00—Indexing scheme for image analysis or image enhancement
- G06T2207/20—Special algorithmic details
- G06T2207/20081—Training; Learning
-
- G—PHYSICS
- G06—COMPUTING; CALCULATING OR COUNTING
- G06T—IMAGE DATA PROCESSING OR GENERATION, IN GENERAL
- G06T2207/00—Indexing scheme for image analysis or image enhancement
- G06T2207/20—Special algorithmic details
- G06T2207/20084—Artificial neural networks [ANN]
-
- G—PHYSICS
- G06—COMPUTING; CALCULATING OR COUNTING
- G06T—IMAGE DATA PROCESSING OR GENERATION, IN GENERAL
- G06T2207/00—Indexing scheme for image analysis or image enhancement
- G06T2207/30—Subject of image; Context of image processing
- G06T2207/30244—Camera pose
Abstract
The invention relates to a method for generating a monocular endoscope new view image based on a deformable nerve radiation field, which comprises the following steps: acquiring a plurality of visual angle original images and camera parameters acquired by a monocular endoscope; based on the original image, mask image manufacturing is carried out to obtain Mask images of the original images with multiple view angles; inputting the original image into a pre-trained monocular endoscope depth estimation network model to obtain a dense depth map; obtaining an original image and an effective area of a dense depth map based on the Mask image; based on the camera parameters and the effective area, implicitly reconstructing by adopting a deformable nerve radiation field to obtain space volume density information and space color information; and rendering the space volume density information and the space color information by a volume to obtain an image generated by the new view angle. Compared with the prior art, the method has the advantages of improving the generation accuracy of the new view angle image and the like.
Description
Technical Field
The invention relates to the technical field of medical image reconstruction and new view angle generation, in particular to a method for generating a new view angle image of a monocular endoscope based on a deformable nerve radiation field.
Background
Monocular endoscopes have been widely used for examination, diagnosis and treatment of the esophageal digestive system such as the stomach, intestines, etc. However, monocular endoscopic image sequences typically provide only a limited field of view, which limits the physician's view of the lesion, particularly at stenosed or curved sites. The new view generation technique can synthesize a new view from image data of a plurality of views, thereby expanding the field of view of an original image. Therefore, the research of generating the new visual angle of the monocular endoscope image is very important for assisting doctors in accurately diagnosing and improving the medical effect and the treatment quality.
Some new view angle generation methods have been proposed by researchers. The conventional method cannot restore fine appearance details. In recent years, methods based on neural radiation fields have performed well in the field of natural image new view generation. However, since monocular endoscopic images possess special complex features such as limited field of view and depth of the input image, black dead area occlusion, and in vivo soft tissue deformation. The use of neural radiation field-based methods to accomplish three-dimensional reconstruction of soft tissue and new view angle generation in a dynamic minimally invasive surgical environment remains a challenging task.
Disclosure of Invention
The invention aims to provide a monocular endoscope new view image generation method based on a deformable nerve radiation field, which improves the accuracy of new view image generation.
The aim of the invention can be achieved by the following technical scheme:
a method for generating a new view image of a monocular endoscope based on a deformable neural radiation field, comprising the steps of:
acquiring a plurality of visual angle original images and camera parameters acquired by a monocular endoscope;
based on the original image, mask image manufacturing is carried out to obtain Mask images of the original images with multiple view angles;
inputting the original image into a pre-trained monocular endoscope depth estimation network model to obtain a dense depth map;
obtaining an original image and an effective area of a dense depth map based on the Mask image;
based on the camera parameters and the effective area, implicitly reconstructing by adopting a deformable nerve radiation field to obtain space volume density information and space color information;
and rendering the space volume density information and the space color information by a volume to obtain an image generated by the new view angle.
Further, the camera parameters are acquired using Colmap.
Further, the camera parameters include camera pose and internal parameters.
Further, the specific steps of obtaining the Mask image include:
selecting one input labelme from the original images with multiple view angles;
manually manufacturing a closed-loop binary Mask at the edges of the invalid area and the effective area to obtain Mask images of a single original image;
and expanding based on the Mask images of the single original image to obtain Mask images corresponding to the original images with multiple view angles.
Further, the training process of the monocular endoscope depth estimation network model specifically comprises the following steps:
acquiring data sets formed by different monocular endoscope videos to finish the pre-training of a monocular endoscope depth estimation network model, and initializing network parameters;
retraining the network model based on the multiple view original images to complete the training process of the monocular endoscope depth estimation network model.
Further, in the training process of the monocular endoscope depth estimation network model, a scale-invariant loss function is adopted to adjust the network weight, and the expression is as follows:
wherein L is 1 Represents a scale-invariant loss value,representing the i-th original image input into the dense depth map predicted by the monocular endoscope depth estimation network model,/for the monocular endoscope depth estimation network model>And (3) representing a sparse depth map corresponding to the ith original image, and q represents a pixel.
Further, the obtaining process of the effective area of the original image and the dense depth map specifically includes:
and judging whether the pixels of the original image and the dense depth map are positioned in a black shielding area or not based on the Mask image, if so, positioning the pixels in an invalid area, and if not, positioning the pixels in an effective area.
Further, the specific step of obtaining the spatial volume density information and the spatial color information includes:
obtaining 3D points corresponding to the pixels based on the effective areas;
and inputting the camera parameters and the 3D points corresponding to the effective areas into a deformation network and a standard network of the trained deformable nerve radiation field, and outputting space volume density information and space color information, wherein in the process of training the deformation network and the standard network of the deformable nerve radiation field, the training is performed by building color loss and depth loss to build total loss.
Further, the expression of the total loss function is:
L＝L RGB +L Depth
wherein L is the total loss, L RGB For colour loss, L Depth For depth loss, C i (p)、C i-gt (p) is the color of the pixel p of the ith image and the color of the pixel p of the ith original image predicted based on the canonical network, D i (p)、D i-gt (p) is the depth value of the pixel p of the ith image and the depth value of the pixel p of the effective depth map of the ith image predicted based on the canonical network, N S Delta is a parameter of the depth loss function for a collection of sampled rays in the input view.
Further, the expression of the volume rendering is:
wherein C is i (p) predicting the color value of the corresponding pixel point p in the 2D image, D is the direction of the light emitted from the center of the camera through the pixel p, h is the position parameter representing the sampling point on the ray, τ i (h) Representing the ray from the nearest point h n To the furthest point h f P i (h) Is a 3D point on the camera ray transformed into canonical space using a deformation network.
Compared with the prior art, the invention has the following beneficial effects:
(1) Aiming at the problem that the input view field and depth of the monocular endoscope sequence image are limited, the monocular endoscope depth estimation is utilized to predict a dense depth map, reconstruction optimization is carried out in a follow-up deformable nerve radiation field, and scene depth information in a sparse view angle is recovered, so that an accurate new view angle image is obtained.
(2) The deformable nerve radiation field can model the shape and the color in the scene through a deformation network and a standard network, and can better process the deformation and the movement in the scene, thereby avoiding the problems of blurring, artifact and the like.
(3) The invention adopts labelme to create the mask image which can obviously distinguish the invalid region and the effective region so as to standardize the ray sampling range, so that the deformable nerve radiation field only carries out clear modeling on the effective region in the image so as to improve the result.
(4) The invention realizes accurate new view synthesis, can be applied to the field of medical images, helps doctors to more accurately observe the details of disease parts, and makes more reliable preoperative diagnosis.
Drawings
FIG. 1 is a schematic flow chart of a method according to an embodiment of the invention;
FIG. 2 is a diagram of an overall architecture of an embodiment of the present invention;
FIG. 3 is a comparison chart of the new view result graph obtained by predicting the new view by the new view generation model trained in the embodiment of the present invention and the results of a plurality of existing methods.
Detailed Description
The invention will now be described in detail with reference to the drawings and specific examples. The present embodiment is implemented on the premise of the technical scheme of the present invention, and a detailed implementation manner and a specific operation process are given, but the protection scope of the present invention is not limited to the following examples.
The embodiment provides a method for generating a new view image of a monocular endoscope based on a deformable nerve radiation field, as shown in fig. 1, the method comprises the following steps:
step 1: and acquiring a plurality of visual angle original images and camera parameters acquired by the monocular endoscope.
The camera parameters of the original multiple view images are acquired using Colmap and a rough depth image estimation is performed. Camera angles and internal references generated by SfM (Structure from Motion, motion structure restoration) with camera parameters of Colmap are used to guide implicit modeling of a scene described later; the coarse depth image is a sparse depth map generated by SfM of Colmap to guide retraining of the monocular endoscopic depth estimation network described later.
The camera parameters and coarse depth map estimation operations include the steps of:
1-1: inputting original multiple view images into a Colmap;
1-2: feature extraction, feature matching and sparse reconstruction are performed successively on the pictures to obtain camera intrinsic parameters, camera pose and sparse 3D point clouds. The generated camera internal parameters and camera pose are used for guiding the implicit modeling of a scene described later, and the generated sparse 3D point cloud can be used for guiding the retraining of a monocular endoscope depth estimation network described later;
1-4: the camera internal parameters, the camera pose and the sparse 3D point cloud output are stored as bin formats;
1-5: the file is converted into LLFF format to facilitate reading of the variable neural radiation field model.
Step 2: and based on the original image, making a Mask image to obtain Mask images of the original images with multiple view angles.
The mask image making in the step 2 comprises the following operations:
2-1: selecting one input labelme from the original multiple view images;
2-2: and manually manufacturing a closed-loop binary Mask at the edges of the black invalid region and the effective region to obtain a Mask image. In the binary mask, a black pixel region corresponds to an effective region within the endoscopic image, and a white pixel region corresponds to a black ineffective region of the endoscopic image. This image can be used to distinguish between an invalid region and an valid region of the image. Outputting and storing the manufactured mask image into json format;
2-3: converting a json format Mask image obtained by 2-2 into a png format, and then expanding the png format Mask image into a plurality of Mask images I corresponding to the original plurality of view images mask 。
Step 3: inputting the original image into a pre-trained monocular endoscope depth estimation network model to obtain a dense depth map.
This step of acquiring a dense depth map includes the operations of:
3-1: monocular endoscopic depth estimation networks are an existing, self-supervising, dense depth estimation network that performs pre-training on data sets composed of different monocular endoscopic videos. Initializing network parameters by using a pre-trained dense depth estimation model, so that transfer learning can be realized, and a reliable dense depth map can be generated;
3-2: inputting the original multiple view images into a monocular endoscope depth estimation network, and using the sparse depth map generated by SfM from the step 1 as a self-supervision signal to finely tune the pre-trained monocular endoscope depth estimation network;
3-3: in order to avoid the acquired depth map having scale ambiguity, the weights of the monocular endoscope depth estimation network are fine-tuned using standard back propagation with a loss function 1;
the loss function 1 is a scale invariant loss function expression as follows:
wherein,representing the i-th original image input into the dense depth map predicted by the monocular endoscope depth estimation network model,/for the monocular endoscope depth estimation network model>Representing the rough depth image corresponding to the i-th original image.
3-4: the network model trains 150 epochs in total;
3-5: inputting the original multiple view images into a trained monocular endoscope depth estimation network model to obtain a dense depth map I predicted by the model ddepth 。
Step 4: and obtaining an original image and an effective area of the dense depth map based on the Mask image.
The realization method for acquiring the effective area of the original image comprises the following steps: mask image I obtained according to step 2 mask Determining whether the pixel of the original image is located at a position not blocked by the black invalid pixelAn effective area. If the pixel is located in the black shielding area, the pixel is not subjected to subsequent ray sampling. If the pixel is located in the effective area, step 5 is executed, and the randomly extracted pixel located in the effective area projects rays and samples to obtain 3D position coordinates.
The implementation method for acquiring the effective area of the dense depth map comprises the following steps: mask image I obtained according to step 2 mask It is determined whether the pixels of the dense depth map are located in an active area that is not occluded by black inactive pixels. If the pixel is located in the black shielding region, the depth value of the pixel is set to 0. If the pixel is located in the effective area, the original pixel depth information is reserved. Thereby, an effective depth map I can be obtained ldepth 。I ldepth The implicit deformable neural radiation field can be trained as a self-supervising signal in combination with the loss function 3.
Step 5: based on the camera parameters and the effective area, implicit reconstruction is performed by using a deformable neural radiation field to obtain spatial volume density information and spatial color information.
The implicit reconstruction of the scene at this step includes the following operations:
5-1: the D-NeRF (Deformable Neural Radiance Fields), deformable neural radiation field, proposes a deformation network and a canonical network, consisting of two cascaded multi-layer perceptrons (Multilayer Perceptron, MLP) capable of reconstructing and rendering rigid and non-rigid scenes acquired by monocular cameras. The invention reconstructs and renders the deformed scene in the monocular endoscopic image based on D-NeRF. The reconstruction process is shown in fig. 2, where randomly extracted pixels located within the active area of the original image project rays and sample to obtain 3D position coordinates (x, y, z). The position offset (Deltax, deltay, deltaz) from the 1 st original image to the i-th original image is predicted by a deformation network. The new position coordinates (x+Δx, y+Δy, z+Δz) and the viewing angle generated by SfM of step 1 are then fed into the canonical network to obtain the reconstructed color and bulk density. Through training of the network, a field state with continuous color and volume density in a three-dimensional space can be obtained as an implicit expression of the scene.
5-2: total loss function sampling loss function 2 and lossThe loss functions 3 are combined so as to self-supervise the training of the deformable neural radiation field of the endoscope scene. The loss function 2 is the color C of the pixel p of the ith image predicted based on canonical network i Color C of pixel p of (p) and ith original image i-gt (p) established color loss L RGB . The loss function 3 is the depth value D of the pixel p of the ith image predicted based on the canonical network i (p) and depth value D of pixel p of effective depth map of ith image obtained in step 4 i-gt (p) established depth loss L Depth . The total loss function can be expressed as:
L＝L RGB +L Depth (2)
wherein N is S Representing a set of sampled rays in the input view;
5-3: the MLP neural network model trains a total of 200k epochs, the batch size of the ray is set to 4096, and each pixel is sampled 32 times along the ray.
Step 6: and rendering the space volume density information and the space color information by the volume to obtain an image generated by the new view angle.
The new view angle generation of the scene comprises the following specific operation steps:
6-1: inputting the 3D position coordinates and the observation visual angles corresponding to the pixels in the images of the test set into a deformation network and a standard network of the trained deformable nerve radiation field to obtain the color and volume density of the sampling points;
6-2: according to the color C and the volume density sigma of the sampling point obtained by 6-1, predicting the color value C of the corresponding pixel point p in the 2D image through a volume rendering formula i (p) and then generating the whole new view angle image. The volume rendering formula is as follows:
where d is the direction of the light from the camera center through pixel p, h is the position parameter representing the sampling point on the ray, τ i (h) Representing the ray from the nearest point h n To the furthest point h f P i (h) Is a 3D point on the camera ray transformed into canonical space using a deformation network.
The method can utilize the constructed data set to train and test the network model, and accurately synthesize the high-fidelity view.
The method of the embodiment selects a plurality of sparse view images with different view angles from a public endoscope data set Nerthus dataset to verify network efficiency. The method is used for generating new angles of monocular endoscopic images, and the method is compared with 7 new angle generation methods based on nerve radiation fields (NeRF), nerve radiation fields aiming at dynamic scenes (D-NeRF), dietNeRF, guide optimization of indoor multi-view stereoscopic vision nerve radiation fields (NerfingMVS), depth supervision NeRF (DS-NeRF), fast dynamic radiation fields based on time perception nerve voxels (TiNeuVox) and nerve radiation characteristic fields (NRFF). FIG. 3 is a detailed graph comparison of the predicted outcome of the method of the present invention and the prior art method. As can be seen from the comparison result of FIG. 3, the method of the present invention can accurately infer the geometry and appearance of the inner wall of the real soft tissue by performing new view angle synthesis on the monocular endoscopic image, and the most clear detail is obtained. Furthermore, the proposed method does not produce unreasonable pixels in invalid occlusion areas. Therefore, the new view synthesized by the method of this embodiment is closer to the original color of the group trunk.
The above functions, if implemented in the form of software functional units and sold or used as a stand-alone product, may be stored in a computer readable storage medium. Based on this understanding, the technical solution of the present invention may be embodied essentially or in a part contributing to the prior art or in a part of the technical solution, in the form of a software product stored in a storage medium, comprising several instructions for causing a computer device (which may be a personal computer, a server, a network device, etc.) to perform all or part of the steps of the method according to the embodiments of the present invention. And the aforementioned storage medium includes: a U-disk, a removable hard disk, a Read-Only Memory (ROM), a random access Memory (RAM, random Access Memory), a magnetic disk, or an optical disk, or other various media capable of storing program codes.
It will be appreciated by those skilled in the art that embodiments of the present invention may be provided as a method, system, or computer program product. Accordingly, the present invention may take the form of an entirely hardware embodiment, an entirely software embodiment or an embodiment combining software and hardware aspects. Furthermore, the present invention may take the form of a computer program product embodied on one or more computer-usable storage media (including, but not limited to, disk storage, CD-ROM, optical storage, and the like) having computer-usable program code embodied therein. The scheme in the embodiment of the invention can be realized by adopting various computer languages, such as object-oriented programming language Java, an transliteration script language JavaScript and the like.
The present invention is described with reference to flowchart illustrations and/or block diagrams of methods, apparatus (systems) and computer program products according to embodiments of the invention. It will be understood that each flow and/or block of the flowchart illustrations and/or block diagrams, and combinations of flows and/or blocks in the flowchart illustrations and/or block diagrams, can be implemented by computer program instructions. These computer program instructions may be provided to a processor of a general purpose computer, special purpose computer, embedded processor, or other programmable data processing apparatus to produce a machine, such that the instructions, which execute via the processor of the computer or other programmable data processing apparatus, create means for implementing the functions specified in the flowchart flow or flows and/or block diagram block or blocks.
These computer program instructions may also be stored in a computer-readable memory that can direct a computer or other programmable data processing apparatus to function in a particular manner, such that the instructions stored in the computer-readable memory produce an article of manufacture including instruction means which implement the function specified in the flowchart flow or flows and/or block diagram block or blocks.
These computer program instructions may also be loaded onto a computer or other programmable data processing apparatus to cause a series of operational steps to be performed on the computer or other programmable apparatus to produce a computer implemented process such that the instructions which execute on the computer or other programmable apparatus provide steps for implementing the functions specified in the flowchart flow or flows and/or block diagram block or blocks.
While preferred embodiments of the present invention have been described, additional variations and modifications in those embodiments may occur to those skilled in the art once they learn of the basic inventive concepts. It is therefore intended that the following claims be interpreted as including the preferred embodiments and all such alterations and modifications as fall within the scope of the invention.
It will be apparent to those skilled in the art that various modifications and variations can be made to the present invention without departing from the spirit or scope of the invention. Thus, it is intended that the present invention also include such modifications and alterations insofar as they come within the scope of the appended claims or the equivalents thereof.
Claims (10)
1. A method for generating a new view image of a monocular endoscope based on a deformable neural radiation field, comprising the steps of:
acquiring a plurality of visual angle original images and camera parameters acquired by a monocular endoscope;
based on the original image, mask image manufacturing is carried out to obtain Mask images of the original images with multiple view angles;
inputting the original image into a pre-trained monocular endoscope depth estimation network model to obtain a dense depth map;
obtaining an original image and an effective area of a dense depth map based on the Mask image;
based on the camera parameters and the effective area, implicitly reconstructing by adopting a deformable nerve radiation field to obtain space volume density information and space color information;
and rendering the space volume density information and the space color information by a volume to obtain an image generated by the new view angle.
2. A method of generating a new view image of a monocular endoscope based on a deformable neural radiation field according to claim 1, wherein the camera parameters are acquired using Colmap.
3. The method for generating a new view image of a monocular endoscope based on a deformable neural radiation field of claim 1, wherein the camera parameters include camera pose and internal parameters.
4. The method for generating a new view image of a monocular endoscope based on a deformable nerve radiation field according to claim 1, wherein the specific step of obtaining the Mask image comprises:
selecting one input labelme from the original images with multiple view angles;
manually manufacturing a closed-loop binary Mask at the edges of the invalid area and the effective area to obtain Mask images of a single original image;
and expanding based on the Mask images of the single original image to obtain Mask images corresponding to the original images with multiple view angles.
5. The method for generating a new view image of a monocular endoscope based on a deformable nerve radiation field according to claim 1, wherein the training process of the monocular endoscope depth estimation network model specifically comprises the following steps:
acquiring data sets formed by different monocular endoscope videos to finish the pre-training of a monocular endoscope depth estimation network model, and initializing network parameters;
retraining the network model based on the multiple view original images to complete the training process of the monocular endoscope depth estimation network model.
6. The method for generating a new view image of a monocular endoscope based on a deformable nerve radiation field according to claim 5, wherein the training process of the monocular endoscope depth estimation network model adopts a scale-invariant loss function to adjust network weights, and the expression is as follows:
wherein L is 1 Represents a scale-invariant loss value,representing the i-th original image input into the dense depth map predicted by the monocular endoscope depth estimation network model,/for the monocular endoscope depth estimation network model>And (3) representing a sparse depth map corresponding to the ith original image, and q represents a pixel.
7. The method for generating a new view image of a monocular endoscope based on a deformable nerve radiation field according to claim 1, wherein the process of obtaining the original image and the effective area of the dense depth map specifically comprises:
and judging whether the pixels of the original image and the dense depth map are positioned in a black shielding area or not based on the Mask image, if so, positioning the pixels in an invalid area, and if not, positioning the pixels in an effective area.
8. The method for generating a new view image of a monocular endoscope based on a deformable nerve radiation field according to claim 1, wherein the specific step of acquiring spatial volume density information and spatial color information comprises:
obtaining 3D points corresponding to the pixels based on the effective areas;
and inputting the camera parameters and the 3D points corresponding to the effective areas into a deformation network and a standard network of the trained deformable nerve radiation field, and outputting space volume density information and space color information, wherein in the process of training the deformation network and the standard network of the deformable nerve radiation field, the training is performed by building color loss and depth loss to build total loss.
9. The method for generating a new view image of a monocular endoscope based on a deformable nerve radiation field of claim 8, wherein the expression of the total loss function is:
L＝L RGB +L Depth
wherein L is the total loss, L RGB For colour loss, L Depth For depth loss, C i (p)、C i-gt (p) is the color of the pixel p of the ith image and the color of the pixel p of the ith original image predicted based on the canonical network, D i (p)、D i-gt (p) is the depth value of the pixel p of the ith image and the depth value of the pixel p of the effective depth map of the ith image predicted based on the canonical network, N S Delta is a parameter of the depth loss function for a collection of sampled rays in the input view.
10. The method for generating a new view image of a monocular endoscope based on a deformable neural radiation field of claim 1, wherein the expression of the volume rendering is:
wherein C is i (p) predicting the color value of the corresponding pixel point p in the 2D image, D is the direction of the light emitted from the center of the camera through the pixel p, h is the position parameter representing the sampling point on the ray, τ i (h) Representing the ray from the nearest point h n To the furthest point h f P i (h) Is a 3D point on the camera ray transformed into canonical space using a deformation network.
Priority Applications (1)
| Application Number | Priority Date | Filing Date | Title |
|---|---|---|---|
| CN202311280252.XA CN117392312A (en) | 2023-09-28 | 2023-09-28 | New view image generation method of monocular endoscope based on deformable nerve radiation field |
Applications Claiming Priority (1)
| Application Number | Priority Date | Filing Date | Title |
|---|---|---|---|
| CN202311280252.XA CN117392312A (en) | 2023-09-28 | 2023-09-28 | New view image generation method of monocular endoscope based on deformable nerve radiation field |
Publications (1)
| Publication Number | Publication Date |
|---|---|
| CN117392312A true CN117392312A (en) | 2024-01-12 |
Family
ID=89440089
Family Applications (1)
| Application Number | Title | Priority Date | Filing Date |
|---|---|---|---|
| CN202311280252.XA Pending CN117392312A (en) | 2023-09-28 | 2023-09-28 | New view image generation method of monocular endoscope based on deformable nerve radiation field |
Country Status (1)
| Country | Link |
|---|---|
| CN (1) | CN117392312A (en) |
Cited By (2)
| Publication number | Priority date | Publication date | Assignee | Title |
|---|---|---|---|---|
| CN117745924A (en) * | 2024-02-19 | 2024-03-22 | 北京渲光科技有限公司 | Neural rendering method, system and equipment based on depth unbiased estimation |
| CN117745924B (en) * | 2024-02-19 | 2024-05-14 | 北京渲光科技有限公司 | Neural rendering method, system and equipment based on depth unbiased estimation |
-
2023
- 2023-09-28 CN CN202311280252.XA patent/CN117392312A/en active Pending
Cited By (2)
| Publication number | Priority date | Publication date | Assignee | Title |
|---|---|---|---|---|
| CN117745924A (en) * | 2024-02-19 | 2024-03-22 | 北京渲光科技有限公司 | Neural rendering method, system and equipment based on depth unbiased estimation |
| CN117745924B (en) * | 2024-02-19 | 2024-05-14 | 北京渲光科技有限公司 | Neural rendering method, system and equipment based on depth unbiased estimation |
Similar Documents
| Publication | Publication Date | Title |
|---|---|---|
| US11806189B2 (en) | Three-dimensional segmentation from two-dimensional intracardiac echocardiography imaging | |
| US20210406591A1 (en) | Medical image processing method and apparatus, and medical image recognition method and apparatus | |
| Yi et al. | Generative adversarial network in medical imaging: A review | |
| Dou et al. | Pnp-adanet: Plug-and-play adversarial domain adaptation network at unpaired cross-modality cardiac segmentation | |
| CN111932550B (en) | 3D ventricle nuclear magnetic resonance video segmentation system based on deep learning | |
| CN112598649B (en) | 2D/3D spine CT non-rigid registration method based on generation of countermeasure network | |
| Rezaei et al. | Whole heart and great vessel segmentation with context-aware of generative adversarial networks | |
| Lee et al. | Identifying women with mammographically-occult breast cancer leveraging GAN-simulated mammograms | |
| CN114359310A (en) | 3D ventricle nuclear magnetic resonance video segmentation optimization system based on deep learning | |
| CN116583880A (en) | Multimodal image processing technique for training image data generation and use thereof for developing a unimodal image inference model | |
| Seo et al. | Neural contrast enhancement of CT image | |
| Zhang et al. | MinimalGAN: diverse medical image synthesis for data augmentation using minimal training data | |
| Wang et al. | MSE-Fusion: Weakly supervised medical image fusion with modal synthesis and enhancement | |
| Chatterjee et al. | A survey on techniques used in medical imaging processing | |
| Poonkodi et al. | 3d-medtrancsgan: 3d medical image transformation using csgan | |
| Iddrisu et al. | 3D reconstructions of brain from MRI scans using neural radiance fields | |
| CN117392312A (en) | New view image generation method of monocular endoscope based on deformable nerve radiation field | |
| Zhou et al. | Synchronizing detection and removal of smoke in endoscopic images with cyclic consistency adversarial nets | |
| Hayat et al. | Combined Channel and Spatial Attention-based Stereo Endoscopic Image Super-Resolution | |
| CN113850710A (en) | Cross-modal medical image accurate conversion method | |
| Tawfeeq et al. | Predication of Most Significant Features in Medical Image by Utilized CNN and Heatmap. | |
| Mathew et al. | SoftEnNet: Symbiotic Monocular Depth Estimation and Lumen Segmentation for Colonoscopy Endorobots | |
| CN110570355A (en) | Multi-scale automatic focusing super-resolution processing system and method | |
| Jeong et al. | Depth estimation of endoscopy using sim-to-real transfer | |
| Zhao et al. | Perception-oriented generative adversarial network for retinal fundus image super-resolution |
Legal Events
| Date | Code | Title | Description |
|---|---|---|---|
| PB01 | Publication | ||
| PB01 | Publication | ||
| SE01 | Entry into force of request for substantive examination | ||
| SE01 | Entry into force of request for substantive examination |