CN117204796B - Multispectral imaging method and device of abdominal cavity endoscope - Google Patents
Multispectral imaging method and device of abdominal cavity endoscope Download PDFInfo
- Publication number
- CN117204796B CN117204796B CN202311486571.6A CN202311486571A CN117204796B CN 117204796 B CN117204796 B CN 117204796B CN 202311486571 A CN202311486571 A CN 202311486571A CN 117204796 B CN117204796 B CN 117204796B
- Authority
- CN
- China
- Prior art keywords
- image
- obtaining
- spectrum
- detection area
- detection
- Prior art date
- Legal status (The legal status is an assumption and is not a legal conclusion. Google has not performed a legal analysis and makes no representation as to the accuracy of the status listed.)
- Active
Links
- 238000000034 method Methods 0.000 title claims abstract description 43
- 210000000683 abdominal cavity Anatomy 0.000 title claims abstract description 36
- 238000000701 chemical imaging Methods 0.000 title claims abstract description 36
- 238000001514 detection method Methods 0.000 claims abstract description 175
- 238000001228 spectrum Methods 0.000 claims abstract description 159
- 230000003595 spectral effect Effects 0.000 claims abstract description 88
- 238000003384 imaging method Methods 0.000 claims abstract description 43
- 238000012545 processing Methods 0.000 claims abstract description 29
- 238000004422 calculation algorithm Methods 0.000 claims abstract description 24
- 238000013527 convolutional neural network Methods 0.000 claims abstract description 20
- 230000003287 optical effect Effects 0.000 claims description 29
- 230000008569 process Effects 0.000 claims description 16
- 238000004590 computer program Methods 0.000 claims description 12
- 230000006870 function Effects 0.000 claims description 9
- 230000000877 morphologic effect Effects 0.000 claims description 6
- 238000003708 edge detection Methods 0.000 claims description 5
- 230000009286 beneficial effect Effects 0.000 abstract 1
- 238000012549 training Methods 0.000 description 7
- 230000009466 transformation Effects 0.000 description 6
- 230000000694 effects Effects 0.000 description 4
- 238000005516 engineering process Methods 0.000 description 4
- 230000009471 action Effects 0.000 description 3
- 238000013528 artificial neural network Methods 0.000 description 3
- 238000004364 calculation method Methods 0.000 description 3
- 230000008859 change Effects 0.000 description 3
- 238000006243 chemical reaction Methods 0.000 description 3
- 238000010183 spectrum analysis Methods 0.000 description 3
- 238000000547 structure data Methods 0.000 description 3
- 238000004458 analytical method Methods 0.000 description 2
- 239000000284 extract Substances 0.000 description 2
- 238000000605 extraction Methods 0.000 description 2
- 238000010191 image analysis Methods 0.000 description 2
- 230000003902 lesion Effects 0.000 description 2
- 238000013507 mapping Methods 0.000 description 2
- 239000011159 matrix material Substances 0.000 description 2
- 238000005457 optimization Methods 0.000 description 2
- 230000001360 synchronised effect Effects 0.000 description 2
- 230000000007 visual effect Effects 0.000 description 2
- 230000003187 abdominal effect Effects 0.000 description 1
- 238000013375 chromatographic separation Methods 0.000 description 1
- 238000000354 decomposition reaction Methods 0.000 description 1
- 238000013461 design Methods 0.000 description 1
- 238000003745 diagnosis Methods 0.000 description 1
- 238000010586 diagram Methods 0.000 description 1
- 230000001747 exhibiting effect Effects 0.000 description 1
- 238000005286 illumination Methods 0.000 description 1
- 238000002357 laparoscopic surgery Methods 0.000 description 1
- 238000005259 measurement Methods 0.000 description 1
- 238000002324 minimally invasive surgery Methods 0.000 description 1
- 238000012986 modification Methods 0.000 description 1
- 230000004048 modification Effects 0.000 description 1
- 238000003672 processing method Methods 0.000 description 1
- 238000009877 rendering Methods 0.000 description 1
- 230000003068 static effect Effects 0.000 description 1
- 238000001356 surgical procedure Methods 0.000 description 1
- 210000001835 viscera Anatomy 0.000 description 1
Classifications
-
- Y—GENERAL TAGGING OF NEW TECHNOLOGICAL DEVELOPMENTS; GENERAL TAGGING OF CROSS-SECTIONAL TECHNOLOGIES SPANNING OVER SEVERAL SECTIONS OF THE IPC; TECHNICAL SUBJECTS COVERED BY FORMER USPC CROSS-REFERENCE ART COLLECTIONS [XRACs] AND DIGESTS
- Y02—TECHNOLOGIES OR APPLICATIONS FOR MITIGATION OR ADAPTATION AGAINST CLIMATE CHANGE
- Y02A—TECHNOLOGIES FOR ADAPTATION TO CLIMATE CHANGE
- Y02A40/00—Adaptation technologies in agriculture, forestry, livestock or agroalimentary production
- Y02A40/10—Adaptation technologies in agriculture, forestry, livestock or agroalimentary production in agriculture
Abstract
The invention provides a multispectral imaging method and device of an abdominal cavity endoscope, and relates to the technical field of image processing; the method comprises the following steps: transmitting detection light of various spectrum bands to a detection area, obtaining an endogenous light signal of the detection area under each spectrum band through a node of a distributed system, and obtaining an initial spectrum image under each spectrum band according to the endogenous light signal; acquiring pose data of a detection end of an abdominal cavity endoscope; obtaining an imaging position of each initial spectrum image in a detection area according to the pose data and each initial spectrum image; inputting the multispectral image, the pose data of the abdominal cavity endoscope and the imaging position corresponding to each initial spectral image into a three-dimensional convolutional neural network; obtaining structural data of a detection area under all spectrum bands; and obtaining a three-dimensional image of the detection area through a three-dimensional reconstruction algorithm. The invention has the beneficial effects that: imaging efficiency is improved and imaging results are improved.
Description
Technical Field
The invention relates to the technical field of image processing, in particular to a multispectral imaging method and device of an abdominal cavity endoscope.
Background
Laparoscopy is a minimally invasive surgical technique by inserting an elongated mirror into the abdominal cavity, and viewing and manipulating internal organs through the endoscope. Minimally invasive surgery typically uses an abdominal endoscope for diagnosis and detection of the lesion in order to observe the specific condition of the lesion inside.
In existing laparoscopic imaging, because the design and size of the laparoscopic instrument limits its field of view, only a small range of tissue areas are typically observed; moreover, the internal structure of the abdominal cavity is complex, tissues with different depths are densely distributed, and clear depth information is difficult to provide; in performing laparoscopic imaging, illumination using a light source is required, but in some cases there may be insufficient light source or uneven light distribution; will affect imaging quality.
Disclosure of Invention
The invention solves the problem of how to improve the imaging effect and imaging efficiency of the abdominal cavity endoscope to a certain extent.
In order to solve the problems, the invention provides a multispectral imaging method and a multispectral imaging device of an abdominal cavity endoscope.
In a first aspect, the present invention provides a method of laparoscopic multispectral imaging for a distributed system, the method comprising:
transmitting detection light of various spectrum bands to a detection area, and obtaining an endogenous optical signal of the detection area under each spectrum band through nodes of the distributed system, wherein each node correspondingly processes the endogenous optical signal of one spectrum band;
obtaining an initial spectrum image under each spectrum band according to the endogenous light signals;
acquiring pose data of a detection end of an abdominal cavity endoscope;
obtaining an imaging position of each initial spectrum image in the detection area according to the pose data and each initial spectrum image;
obtaining multispectral images under the spectrum bands corresponding to all the nodes according to the initial spectrum images corresponding to each spectrum band;
inputting the multispectral image, the pose data of the abdominal cavity endoscope and the imaging position corresponding to each initial spectral image into a three-dimensional convolutional neural network;
obtaining structural data of the detection area under all the spectrum bands through the three-dimensional convolutional neural network;
inputting the multispectral image, the pose data of the abdominal cavity endoscope and the imaging position corresponding to each initial spectral image into a three-dimensional convolutional neural network;
obtaining structural data of the detection region in all the spectrum bands through the three-dimensional convolutional neural network, wherein the structural data comprises the size, the shape and the texture of the detection region;
and obtaining a three-dimensional image of the detection area through a three-dimensional reconstruction algorithm according to the size, shape and texture of the detection area.
Optionally, the obtaining an initial spectrum image under each spectrum band according to the endogenous light signal includes:
obtaining the intensity value of the spectrum band corresponding to the endogenous optical signal according to the endogenous optical signal;
obtaining the region characteristics of the detection region under the spectrum wave band according to the intensity value of the spectrum wave band;
and obtaining the initial spectrum image under the spectrum band according to the region characteristics.
Optionally, the obtaining the region feature of the detection region under the spectrum band according to the intensity value of the spectrum band includes:
obtaining the distribution of the intensity values in the detection area according to the intensity values of the spectrum bands;
according to the distribution of the intensity values in the detection area, obtaining the boundary contour of the detection area through an edge detection algorithm;
obtaining structural data of the detection area in the spectrum band through morphological rules according to the distribution of the intensity values;
and obtaining the region characteristics of the detection region according to the boundary outline of the detection region and the structural data of the detection region in the spectrum band.
Optionally, the obtaining the region feature of the detection region under the spectrum band according to the intensity value of the spectrum band includes:
obtaining the distribution of the intensity values in the detection area according to the intensity values of the spectrum bands;
according to the distribution of the intensity values in the detection area, obtaining the boundary contour of the detection area through an edge detection algorithm;
obtaining structural data of the detection area in the spectrum band through morphological rules according to the distribution of the intensity values;
and obtaining the region characteristics of the detection region according to the boundary outline of the detection region and the structural data of the detection region in the spectrum band.
Optionally, the obtaining multispectral images under the spectrum bands corresponding to all the nodes according to the initial spectrum image corresponding to each spectrum band includes:
acquiring coordinates of pixel points of each initial spectrum image in the coordinate system;
overlapping the pixel points with the same coordinates in each initial spectrum image through a preset overlapping rule to obtain overlapping images of the initial spectrum images under all spectrum bands;
and taking the superimposed image as the multispectral image.
Optionally, the obtaining a three-dimensional image of the detection area according to the structural data of the detection area through a three-dimensional reconstruction algorithm includes:
obtaining a key structure of the detection area according to the size, the shape and the texture of the detection area, wherein the key structure comprises: corner features and texture features of the detection region;
according to the corner features and the texture features, three-dimensional point cloud data of the key structure are obtained through the three-dimensional reconstruction algorithm;
and obtaining the three-dimensional image of the detection area according to the three-dimensional point cloud data.
Optionally, the obtaining the three-dimensional image of the detection area according to the three-dimensional point cloud data includes:
dividing a plurality of point cloud areas according to the three-dimensional point cloud data, wherein all the point cloud areas form the three-dimensional point cloud data;
distributing the corresponding nodes of the distributed system for the point cloud area;
obtaining a voxel grid corresponding to the point cloud region according to the point cloud region through the node;
according to the voxel grid, obtaining a three-dimensional modeling of the point cloud area;
and obtaining the three-dimensional image of the detection area according to the three-dimensional modeling corresponding to the voxel grids of all the nodes.
In a second aspect, the present invention provides a laparoscopic multispectral imaging device for a distributed system, the laparoscopic multispectral imaging device comprising:
the optical signal detection unit is used for sending detection light of various spectrum bands to a detection area, and obtaining an endogenous optical signal of the detection area under each spectrum band through a node of the distributed system, wherein each node correspondingly processes the endogenous optical signal of one spectrum band;
the processing unit is used for obtaining an initial spectrum image under each spectrum band according to the endogenous light signals;
the gesture sensing unit is used for acquiring gesture data of a detection end of the abdominal cavity endoscope;
the processing unit is used for obtaining the imaging position of each initial spectrum image in the detection area according to the pose data and each initial spectrum image;
obtaining multispectral images under the spectrum bands corresponding to all the nodes according to the initial spectrum images corresponding to each spectrum band;
the three-dimensional establishing unit is used for inputting the multispectral image, the pose data of the abdominal cavity endoscope and the imaging position corresponding to each initial spectral image into a three-dimensional convolutional neural network;
obtaining structural data of the detection region in all the spectrum bands through the three-dimensional convolutional neural network, wherein the structural data comprises the size, the shape and the texture of the detection region;
and obtaining a three-dimensional image of the detection area through a three-dimensional reconstruction algorithm according to the size, shape and texture of the detection area.
Optionally, the laparoscopic multispectral imaging device comprises a memory, a processor and a computer program stored on the memory and capable of running on the processor, wherein the processor realizes the functions of the laparoscopic multispectral imaging device when executing the computer program.
Optionally, the laparoscopic multispectral imaging device has a computer program stored thereon, and when the computer program is executed by a processor, the functions of the laparoscopic multispectral imaging device are realized.
According to the method and the device for imaging the abdominal cavity endoscope in the multispectral manner, detection light of various spectral bands is utilized and is sent to the detection area through the nodes of the distributed system, each node processes an endogenous light signal of one spectral band, and the endogenous light signals of the detection area under each spectral band are obtained through simultaneous processing of the nodes, so that parallel processing of each spectral band is realized, the processing time of the whole endoscope imaging is shortened, and the imaging efficiency is improved. And then according to the endogenous optical signals, obtaining initial spectral images in each spectral band, combining pose data of an endoscope detection end and each initial spectral image, calculating imaging positions of each initial spectral image in a detection area, ensuring the alignment of images in different spectral bands in space, generating multispectral images in all spectral bands according to the initial spectral images corresponding to each spectral band, inputting the multispectral images, the pose data and the imaging positions of the initial spectral images into a three-dimensional convolutional neural network for training, finally obtaining three-dimensional images of the detection area, obtaining abundant structural information in different spectral bands, displaying in a three-dimensional form, and improving the imaging effect of the abdominal cavity endoscope.
Drawings
FIG. 1 is one of the flowcharts of the laparoscopic multispectral imaging method of the present invention;
FIG. 2 is a second flowchart of the method of the present invention for laparoscopic multispectral imaging;
FIG. 3 is a third flowchart of the method of laparoscopic multispectral imaging of the present invention;
FIG. 4 is a fourth flowchart of the laparoscopic multispectral imaging method of the present invention;
FIG. 5 is a fifth flowchart of the laparoscopic multispectral imaging method of the present invention;
FIG. 6 is a flow chart of a method of laparoscopic multispectral imaging of the present invention;
FIG. 7 is a flow chart of a method of laparoscopic multispectral imaging of the present invention;
fig. 8 is a block diagram showing the structure of the multi-spectral imaging apparatus for the abdominal cavity endoscope of the present invention.
Detailed Description
In order that the above objects, features and advantages of the invention will be readily understood, a more particular description of the invention will be rendered by reference to specific embodiments thereof which are illustrated in the appended drawings.
In a first aspect, as shown in connection with fig. 1, the present invention provides a method of laparoscopic multispectral imaging for a distributed system, the method comprising:
transmitting detection light of various spectrum bands to a detection area, and obtaining an endogenous optical signal of the detection area under each spectrum band through nodes of the distributed system, wherein each node correspondingly processes the endogenous optical signal of one spectrum band;
specifically, detection light of various spectrum bands is sent to a detection area, and an endogenous light signal of the detection area under each spectrum band is obtained through node processing of a distributed system, so that richer and more detailed spectrum information is obtained, and a more accurate data basis is provided for subsequent image processing and analysis.
Obtaining an initial spectrum image under each spectrum band according to the endogenous light signals;
specifically, by using the endogenous optical signals in each spectral band, an initial spectral image in the corresponding spectral band can be obtained, the initial spectral image reflects the characteristics of the endogenous optical signals corresponding to each spectral band, in the preferred embodiment of the invention, the step can use spectral analysis technology, such as chromatographic separation or optical filters, to separate or filter the interference of the endogenous optical signals in each spectral band from the interference of other spectral bands, thereby obtaining the pure optical signals in each spectral band, and more differentiated and rich image data can be obtained through the initial spectral image, and meanwhile, the initial spectral images can reflect the characteristics and the attributes of the detected areas in different spectral bands.
Acquiring pose data of a detection end of an abdominal cavity endoscope;
specifically, pose data of the detecting end of the abdominal cavity endoscope is obtained through the arranged sensor or system, wherein the pose data comprise information such as the position and the direction of the detecting end, and the pose data can be obtained through different sensors and technologies. In a preferred embodiment of the invention, the sensor can be fixed at the tail end of the detecting end of the endoscope or connected to the detecting end of the endoscope, and through the combined use of different sensors, the three-dimensional position and posture information of the endoscope can be obtained, so that an accurate basis is provided for the subsequent imaging position calculation.
Obtaining an imaging position of each initial spectrum image in the detection area according to the pose data and each initial spectrum image;
specifically, through pose data, the position and the direction of the endoscope detection end relative to a reference coordinate system can be obtained, and the position and the direction of each initial spectrum image in a three-dimensional space can be calculated by combining the initial spectrum images, and can be completed through calibration and calibration processes. Wherein more accurate pose data can be obtained by matching the raw readings of the sensor with the actual measurements.
Obtaining multispectral images under the spectrum bands corresponding to all the nodes according to the initial spectrum images corresponding to each spectrum band;
specifically, at each node, the initial spectrum image of the processed spectrum band is converted into a corresponding multispectral image, wherein the conversion can be realized by mapping the light intensity information in the initial spectrum image onto different bands, and a specific conversion method can select a proper spectrum decomposition or conversion technology according to application requirements and algorithms. The characteristics and the attributes of the region to be detected are more comprehensively analyzed through the multispectral images, the multispectral images can provide the spectral information of different wave bands including infrared, ultraviolet, visible light and the like, and the detection capability of objects and tissues is expanded.
Inputting the multispectral image, the pose data of the abdominal cavity endoscope and the imaging position corresponding to each initial spectral image into a three-dimensional convolutional neural network;
specifically, by taking a multispectral image as input, the neural network can learn and extract spectral features of different wavebands, and the three-dimensional context analysis and processing are carried out on the image by combining the pose data and the imaging position information of the abdominal cavity endoscope. Through neural networks, the multispectral images are subjected to tasks such as feature extraction, object classification, target detection, gesture estimation and the like, and image data are further optimized and analyzed, so that more accurate imaging and image analysis results are realized.
And obtaining structural data of the detection region in all the spectrum bands through the three-dimensional convolutional neural network, wherein the structural data comprises the size, the shape and the texture of the detection region.
Specifically, through training and optimization, the three-dimensional convolutional neural network can obtain structural data of a detection region in all spectral bands, in the training process, the neural network learns and extracts image features and predicted target parameters through iterative optimization loss functions, and by providing input data consisting of multispectral images, pose data and imaging positions for the network, the network can learn and acquire the structural information of the detection region in different spectral bands.
In some preferred embodiments, the three-dimensional convolutional neural network stops training the initial structural data in the spectral band when the value of the loss function of the three-dimensional convolutional neural network is in a balanced trend; specifically, when the values of the loss functions of the three-dimensional convolutional neural network tend to be balanced, training of the initial structural data in all spectral bands can be stopped, and at this time, the network has learned the best parameter and feature extraction capability. And taking the initial structure data as the structure data of the detection area under all spectral bands; in this embodiment, the structure data includes: the size, shape and texture of the detection area; specifically, the initial structural data acquired in the training process is used as structural data of the detection area under all spectrum bands, the structural data comprises information such as the size, the shape and the texture of the detection area, the information is used for describing and analyzing the characteristics and the attributes of the area to be detected, and the structural characteristics of the detection area can be obtained more accurately and comprehensively by combining the three-dimensional convolutional neural network and the initial structural data.
And obtaining a three-dimensional image of the detection area through a three-dimensional reconstruction algorithm according to the size, shape and texture of the detection area.
Specifically, the three-dimensional reconstruction algorithm can reconstruct the three-dimensional structure of the detection area by utilizing the characteristic information such as the size, the shape, the texture and the like of the detection area and the possible depth information, can perform modeling and rendering based on the data representation forms such as point cloud, voxels or grids, can acquire the accurate position and the shape of the detection area in space and the internal structure and the tissue distribution condition of the detection area through the three-dimensional reconstruction algorithm, and provides more comprehensive and three-dimensional image information.
According to the abdominal cavity endoscope multispectral imaging method, detection light of various spectral bands is utilized and sent to the detection area through the nodes of the distributed system, each node processes an endogenous light signal of one spectral band, and the endogenous light signal of the detection area under each spectral band is obtained through node processing, so that parallel processing of each spectral band is realized, the processing time of the whole endoscope imaging is shortened, and the imaging efficiency is improved. And then according to the endogenous optical signals, obtaining initial spectral images in each spectral band, combining pose data of an endoscope detection end and each initial spectral image, calculating imaging positions of each initial spectral image in a detection area, ensuring the alignment of images in different spectral bands in space, generating multispectral images in all spectral bands according to the initial spectral images corresponding to each spectral band, inputting the multispectral images, the pose data and the imaging positions of the initial spectral images into a three-dimensional convolutional neural network for training, finally obtaining three-dimensional images of the detection area, obtaining abundant structural information in different spectral bands, displaying in a three-dimensional form, and improving the imaging effect of the abdominal cavity endoscope.
Optionally, as shown in fig. 2, the obtaining an initial spectral image under each spectral band according to the endogenous optical signal includes:
obtaining the intensity value of the spectrum band corresponding to the endogenous optical signal according to the endogenous optical signal;
obtaining the region characteristics of the detection region under the spectrum wave band according to the intensity value of the spectrum wave band;
and obtaining the initial spectrum image under the spectrum band according to the region characteristics.
Specifically, the spectrum measuring equipment is used for measuring the endogenous optical signals and obtaining intensity values of corresponding spectrum bands, wherein the intensity values represent energy distribution conditions of the spectrum signals under different bands, the computer vision technology and the image processing algorithm are used for extracting the regional characteristics of the detection region under the spectrum bands according to the intensity values of the spectrum bands, and the characteristic difference of the detection region under the spectrum bands is deduced by analyzing the change of the intensity values of the bands. And combining the extracted regional characteristics with the intensity values of the spectrum bands, and fusing and superposing the regional characteristics and the spectrum intensity values to obtain an initial spectrum image under the corresponding spectrum bands. By converting the endogenous light signal into an initial spectral image, the surface features and tissue structure of the detection region can be observed and analyzed at different wavebands.
In this embodiment, by acquiring the intensity value of the spectrum band according to the endogenous optical signal, the region feature is extracted, and an initial spectrum image is generated, so that an accurate data basis is processed for the subsequent spectrum image.
Optionally, referring to fig. 3, the obtaining, according to the intensity value of the spectrum band, a region feature of the detection region under the spectrum band includes:
obtaining the distribution of the intensity values in the detection area according to the intensity values of the spectrum bands;
according to the distribution of the intensity values in the detection area, obtaining the boundary contour of the detection area through an edge detection algorithm;
obtaining structural data of the detection area in the spectrum band through morphological rules according to the distribution of the intensity values;
and obtaining the region characteristics of the detection region according to the boundary outline of the detection region and the structural data of the detection region in the spectrum band.
Specifically, by analyzing the intensity values of the spectral bands, the intensity distribution at each location in the detection area can be known, exhibiting differences in brightness of the target object, and possibly specific spectral features. By means of an edge detection algorithm, the boundary contour of the detection area can be determined from the change in the intensity values, wherein the edge contour reflects the shape and boundary information of the target object. By applying morphological rules, the structural data of the detection area under the spectrum band can be obtained according to the distribution of the intensity values, morphological operations can change the shape, the size, the texture and other characteristics of the target object, further the relation and the characteristics between the shape, the size and the texture can be extracted, and by combining the boundary contour of the detection area and the structural data under the spectrum band, the area characteristics of the detection area can be obtained more accurately and comprehensively, the characteristics comprise the shape, the size, the texture, the boundary attribute and the like of the target object, and the characteristics and the attributes of the detection area can be more comprehensively described and analyzed.
In this embodiment, the region characteristics of the detection region are obtained according to the intensity value of the spectrum band, so that information such as the surface morphology and the internal structure of the target object in the spectrum band is provided, and the application fields of spectrum image processing and image analysis are expanded.
Optionally, referring to fig. 4, the obtaining, according to the pose data and each initial spectrum image, an imaging position of each initial spectrum image in the detection area includes:
converting the pose data of the abdominal cavity endoscope into a coordinate system of the initial spectrum image to obtain the pose of the abdominal cavity endoscope in the coordinate system of the initial spectrum image;
obtaining the detection range of the endoscope in the detection area according to the pose of the abdominal cavity endoscope in the coordinate system of the initial spectrum image;
and taking the detection range as the imaging position of the initial spectrum image in the detection area.
Specifically, according to pose data of the laparoscope including position and pose information, transformation of a coordinate system can be performed, the pose data of the laparoscope can be transformed from the coordinate system of the laparoscope to the coordinate system of the initial spectral image, the transformation can be achieved through matrix transformation, the coordinate system of the laparoscope and the coordinate system of the initial spectral image are aligned, and according to the pose information of the laparoscope in the coordinate system of the initial spectral image, the position of the laparoscope on the initial spectral image can be determined. The detection range of the endoscope in the detection area is determined by mapping the field of view of the endoscope onto the initial spectral image. The detection range of the endoscope on the initial spectral image can be regarded as the imaging position of the initial spectral image in the detection region, and the region corresponding to the endoscope visual field in the initial spectral image is represented for further spectral analysis and image processing.
In this embodiment, according to pose data of the laparoscope including position and pose information, transformation of a coordinate system may be performed, the pose data of the laparoscope may be transformed from the coordinate system of the laparoscope to the coordinate system of the initial spectral image, the transformation may be performed by matrix transformation, the coordinate system of the laparoscope and the coordinate system of the initial spectral image may be aligned, and according to pose information of the laparoscope in the coordinate system of the initial spectral image, a position of the laparoscope on the initial spectral image may be determined. The detection range of the endoscope on the initial spectrum image can be regarded as the imaging position of the initial spectrum image in the detection region, and the region corresponding to the endoscope field of view in the initial spectrum image is shown for further spectrum analysis and image processing, so that the imaging effect of the abdominal cavity endoscope is improved.
Optionally, in combination with fig. 5, the obtaining multispectral images under the spectrum bands corresponding to all the nodes according to the initial spectrum image corresponding to each spectrum band includes:
acquiring coordinates of pixel points of each initial spectrum image in the coordinate system;
overlapping the pixel points with the same coordinates in each initial spectrum image through a preset overlapping rule to obtain overlapping images of the initial spectrum images under all spectrum bands;
and taking the superimposed image as the multispectral image.
Specifically, in a preferred embodiment of the present invention, for each initial spectral image, coordinates of each pixel point in a coordinate system may be obtained through an image processing method, positions of each pixel point in the coordinate system are determined by aligning the positions of the pixel points of the image with the coordinate system, for the initial spectral image in each spectral band, the pixel points in the same coordinate position are found, and the superimposition processing is performed according to a preset superimposition rule. In a preferred embodiment of the invention, the superposition may be performed using a weighted average of the pixel values. And combining the pixel points on the same coordinate position into one pixel through superposition processing to obtain a superposition image under the spectrum bands, and combining the superposition images under each spectrum band to form a multispectral image. The multispectral image contains pixel information under different spectral bands, and can provide richer spectral characteristics and color information.
In this embodiment, according to the initial spectral image of each spectral band, multispectral images of all nodes under the spectral bands are obtained, and image information of different bands is integrated, so as to provide more comprehensive and accurate spectral features.
Optionally, referring to fig. 6, the obtaining, by a three-dimensional reconstruction algorithm, a three-dimensional image of the detection area according to the structural data of the detection area includes:
obtaining a key structure of the detection area according to the size, the shape and the texture of the detection area, wherein the key structure comprises: corner features and texture features of the detection region;
according to the corner features and the texture features, three-dimensional point cloud data of the key structure are obtained through the three-dimensional reconstruction algorithm;
and obtaining the three-dimensional image of the detection area according to the three-dimensional point cloud data.
Specifically, key structural information is extracted by analyzing geometric attributes and texture features of the detection area, wherein the key structural information comprises angular point features and texture features of the detection area, the angular point features refer to positions with obvious angle changes in an image, and the texture features refer to areas with certain texture changes in the image. By extracting these structural information, the geometry and texture features of the detection region are more accurately described. And calculating a three-dimensional reconstruction algorithm by utilizing the corner features and the texture features, and deducing the positions of the corresponding three-dimensional points. And obtaining a three-dimensional point cloud data set by carrying out three-dimensional reconstruction on the key structure information, wherein each point represents a three-dimensional position in space. According to the three-dimensional point cloud data, voxel processing or a surface reconstruction algorithm can be performed to convert the point cloud data into a more visual three-dimensional image representation.
In this embodiment, key structure information is extracted by analyzing the size, shape and texture of the detection area, three-dimensional point cloud data of the key structure is obtained by a three-dimensional reconstruction algorithm, and then the three-dimensional point cloud data is converted into a three-dimensional image representation to obtain a three-dimensional image of the detection area. The obtained three-dimensional image can provide more comprehensive and accurate spatial information.
Optionally, referring to fig. 7, the obtaining the three-dimensional image of the detection area according to the three-dimensional point cloud data includes:
dividing a plurality of point cloud areas according to the three-dimensional point cloud data, wherein all the point cloud areas form the three-dimensional point cloud data;
distributing the corresponding nodes of the distributed system for the point cloud area;
obtaining a voxel grid corresponding to the point cloud region according to the point cloud region through the node;
according to the voxel grid, obtaining a three-dimensional modeling of the point cloud area;
and obtaining the three-dimensional image of the detection area according to the three-dimensional modeling corresponding to the voxel grids of all the nodes.
Specifically, the three-dimensional point cloud data is divided into a plurality of small point cloud areas according to the position information of the three-dimensional point cloud data, and the point cloud areas can be regular or defined according to the requirements of specific applications. Each point cloud region is associated with a node in the distributed system. Therefore, the calculation tasks can be distributed to different nodes for parallel processing, and the calculation efficiency and speed are improved. And on each node, carrying out voxelization according to the data of the point cloud area, wherein the voxelization is to convert the point cloud data into a three-dimensional grid representation with a voxel structure, carrying out a surface reconstruction algorithm through the voxel grid, converting the voxel grid into a three-dimensional modeling representation with geometrical information, and obtaining a three-dimensional image of the whole detection area by integrating and fusing the data of the three-dimensional modeling generated on each node.
In the embodiment, the distributed computing mode can improve the computing efficiency and speed, and can process the large-scale point cloud data.
In a second aspect, as shown in connection with fig. 8, the present invention provides a laparoscopic multispectral imaging device for a distributed system, the laparoscopic multispectral imaging device comprising:
the optical signal detection unit is used for sending detection light of various spectrum bands to a detection area, and obtaining an endogenous optical signal of the detection area under each spectrum band through a node of the distributed system, wherein each node correspondingly processes the endogenous optical signal of one spectrum band;
the processing unit is used for obtaining an initial spectrum image under each spectrum band according to the endogenous light signals;
the gesture sensing unit is used for acquiring gesture data of a detection end of the abdominal cavity endoscope;
the processing unit is used for obtaining the imaging position of each initial spectrum image in the detection area according to the pose data and each initial spectrum image;
obtaining multispectral images under the spectrum bands corresponding to all the nodes according to the initial spectrum images corresponding to each spectrum band;
the three-dimensional establishing unit is used for inputting the multispectral image, the pose data of the abdominal cavity endoscope and the imaging position corresponding to each initial spectral image into a three-dimensional convolutional neural network;
obtaining structural data of the detection region in all the spectrum bands through the three-dimensional convolutional neural network, wherein the structural data comprises the size, the shape and the texture of the detection region;
and obtaining a three-dimensional image of the detection area through a three-dimensional reconstruction algorithm according to the size, shape and texture of the detection area.
Optionally, the laparoscopic multispectral imaging device comprises a memory, a processor and a computer program stored on the memory and capable of running on the processor, wherein the processor realizes the functions of the laparoscopic multispectral imaging device when executing the computer program.
Optionally, the laparoscopic multispectral imaging device stores a computer program therein, and the computer program when executed by the processor realizes the functions of the laparoscopic multispectral imaging device.
The advantages of the device and the method for imaging the multispectral of the abdominal cavity endoscope are the same as those of the prior art, and are not repeated here.
Those skilled in the art will appreciate that all or part of the processes in the methods of the above embodiments may be implemented by a computer program for instructing relevant hardware, where the program may be stored in a non-volatile computer readable storage medium, and where the program, when executed, may include processes in the embodiments of the methods described above. Any reference to memory, storage, database, or other medium used in embodiments provided herein may include non-volatile and/or volatile memory. The nonvolatile memory can include Read Only Memory (ROM), programmable ROM (PROM), electrically Programmable ROM (EPROM), electrically Erasable Programmable ROM (EEPROM), or flash memory. Volatile memory can include Random Access Memory (RAM) or external cache memory. By way of illustration and not limitation, RAM is available in a variety of forms such as Static RAM (SRAM), dynamic RAM (DRAM), synchronous DRAM (SDRAM), double Data Rate SDRAM (DDRSDRAM), enhanced SDRAM (ESDRAM), synchronous Link DRAM (SLDRAM), memory bus direct RAM (RDRAM), direct memory bus dynamic RAM (DRDRAM), and memory bus dynamic RAM (RDRAM), among others.
It should be noted that in this document, relational terms such as "first" and "second" and the like are used solely to distinguish one entity or action from another entity or action without necessarily requiring or implying any actual such relationship or order between such entities or actions. Moreover, the terms "comprises," "comprising," or any other variation thereof, are intended to cover a non-exclusive inclusion, such that a process, method, article, or apparatus that comprises a list of elements does not include only those elements but may include other elements not expressly listed or inherent to such process, method, article, or apparatus. Without further limitation, an element defined by the phrase "comprising one … …" does not exclude the presence of other like elements in a process, method, article, or apparatus that comprises the element. The foregoing is only a specific embodiment of the invention to enable those skilled in the art to understand or practice the invention. Various modifications to these embodiments will be readily apparent to those skilled in the art, and the generic principles defined herein may be applied to other embodiments without departing from the spirit or scope of the invention. Thus, the present invention is not intended to be limited to the embodiments shown herein but is to be accorded the widest scope consistent with the principles and novel features disclosed herein.
Claims (8)
1. A laparoscopic multispectral imaging device for a distributed system, the laparoscopic multispectral imaging device comprising:
the optical signal detection unit is used for sending detection light of various spectrum bands to a detection area, and obtaining an endogenous optical signal of the detection area under each spectrum band through a node of the distributed system, wherein each node correspondingly processes the endogenous optical signal of one spectrum band;
the processing unit is used for obtaining an initial spectrum image under each spectrum band according to the endogenous light signals;
the gesture sensing unit is used for acquiring gesture data of a detection end of the abdominal cavity endoscope;
the processing unit is used for obtaining the imaging position of each initial spectrum image in the detection area according to the pose data and each initial spectrum image;
obtaining multispectral images under the spectrum bands corresponding to all the nodes according to the initial spectrum images corresponding to each spectrum band;
the three-dimensional establishing unit is used for inputting the multispectral image, the pose data of the abdominal cavity endoscope and the imaging position corresponding to each initial spectral image into a three-dimensional convolutional neural network;
obtaining structural data of the detection region in all the spectrum bands through the three-dimensional convolutional neural network, wherein the structural data comprises the size, the shape and the texture of the detection region;
according to the size, shape and texture of the detection area, a three-dimensional image of the detection area is obtained through a three-dimensional reconstruction algorithm; the method specifically comprises the following steps: obtaining a key structure of the detection area according to the size, the shape and the texture of the detection area, wherein the key structure comprises: corner features and texture features of the detection region;
according to the corner features and the texture features, three-dimensional point cloud data of the key structure are obtained through the three-dimensional reconstruction algorithm;
and obtaining the three-dimensional image of the detection area according to the three-dimensional point cloud data.
2. The laparoscopic multispectral imaging device according to claim 1, wherein the processing unit is configured to obtain an initial spectral image for each of the spectral bands from the endogenous light signals, comprising:
obtaining the intensity value of the spectrum band corresponding to the endogenous optical signal according to the endogenous optical signal;
obtaining the region characteristics of the detection region under the spectrum wave band according to the intensity value of the spectrum wave band;
and obtaining the initial spectrum image under the spectrum band according to the region characteristics.
3. The device according to claim 2, wherein the processing unit is configured to obtain, in particular, a region characteristic of the detection region in the spectral band according to the intensity values of the spectral band, and comprises:
obtaining the distribution of the intensity values in the detection area according to the intensity values of the spectrum bands;
according to the distribution of the intensity values in the detection area, obtaining the boundary contour of the detection area through an edge detection algorithm;
obtaining structural data of the detection area in the spectrum band through morphological rules according to the distribution of the intensity values;
and obtaining the region characteristics of the detection region according to the boundary outline of the detection region and the structural data of the detection region in the spectrum band.
4. The apparatus according to claim 1, wherein the processing unit is configured to obtain an imaging position of each of the initial spectral images in the detection area based on the pose data and each of the initial spectral images, and includes:
converting the pose data of the abdominal cavity endoscope into a coordinate system of the initial spectrum image to obtain the pose of the abdominal cavity endoscope in the coordinate system of the initial spectrum image;
obtaining the detection range of the endoscope in the detection area according to the pose of the abdominal cavity endoscope in the coordinate system of the initial spectrum image;
and taking the detection range as the imaging position of the initial spectrum image in the detection area.
5. The apparatus according to claim 4, wherein the processing unit is configured to obtain multispectral images in the spectral bands corresponding to all the nodes from the initial spectral image corresponding to each spectral band, and includes:
acquiring coordinates of pixel points of each initial spectrum image in the coordinate system;
overlapping the pixel points with the same coordinates in each initial spectrum image through a preset overlapping rule to obtain overlapping images of the initial spectrum images under all spectrum bands;
and taking the superimposed image as the multispectral image.
6. The apparatus according to claim 1, wherein the three-dimensional establishing unit is specifically configured to obtain the three-dimensional image of the detection area according to the three-dimensional point cloud data, and includes:
dividing a plurality of point cloud areas according to the three-dimensional point cloud data, wherein all the point cloud areas form the three-dimensional point cloud data;
distributing the corresponding nodes of the distributed system for the point cloud area;
obtaining a voxel grid corresponding to the point cloud region according to the point cloud region through the node;
according to the voxel grid, obtaining a three-dimensional modeling of the point cloud area;
and obtaining the three-dimensional image of the detection area according to the three-dimensional modeling corresponding to the voxel grids of all the nodes.
7. The laparoscopic multispectral imaging device according to any one of claims 1-6, comprising a memory, a processor and a computer program stored on the memory and executable on the processor, wherein the processor performs the functions of the laparoscopic multispectral imaging device according to any one of claims 1-6 when the computer program is executed by the processor.
8. The laparoscopic multispectral imaging device according to any one of claims 1-6, wherein a computer program is stored thereon, which, when executed by a processor, performs the functions of the laparoscopic multispectral imaging device according to any one of claims 1-6.
Priority Applications (1)
| Application Number | Priority Date | Filing Date | Title |
|---|---|---|---|
| CN202311486571.6A CN117204796B (en) | 2023-11-09 | 2023-11-09 | Multispectral imaging method and device of abdominal cavity endoscope |
Applications Claiming Priority (1)
| Application Number | Priority Date | Filing Date | Title |
|---|---|---|---|
| CN202311486571.6A CN117204796B (en) | 2023-11-09 | 2023-11-09 | Multispectral imaging method and device of abdominal cavity endoscope |
Publications (2)
| Publication Number | Publication Date |
|---|---|
| CN117204796A CN117204796A (en) | 2023-12-12 |
| CN117204796B true CN117204796B (en) | 2024-02-13 |
Family
ID=89037541
Family Applications (1)
| Application Number | Title | Priority Date | Filing Date |
|---|---|---|---|
| CN202311486571.6A Active CN117204796B (en) | 2023-11-09 | 2023-11-09 | Multispectral imaging method and device of abdominal cavity endoscope |
Country Status (1)
| Country | Link |
|---|---|
| CN (1) | CN117204796B (en) |
Families Citing this family (1)
| Publication number | Priority date | Publication date | Assignee | Title |
|---|---|---|---|---|
| CN117152362A (en) * | 2023-10-27 | 2023-12-01 | 深圳市中安视达科技有限公司 | Multi-path imaging method, device, equipment and storage medium for endoscope multi-spectrum |
Citations (9)
| Publication number | Priority date | Publication date | Assignee | Title |
|---|---|---|---|---|
| CN104083141A (en) * | 2013-11-21 | 2014-10-08 | 北京华科创智健康科技股份有限公司 | Multispectral combined output light source device and endoscope device and endoscope device |
| CN104814712A (en) * | 2013-11-07 | 2015-08-05 | 南京三维视嘉科技发展有限公司 | Three-dimensional endoscope and three-dimensional imaging method |
| CN106303279A (en) * | 2016-08-31 | 2017-01-04 | 北京数字精准医疗科技有限公司 | Multi-spectrum endoscopic automatic exposure formation method |
| CN107260117A (en) * | 2016-03-31 | 2017-10-20 | 柯惠有限合伙公司 | Chest endoscope for surface scan |
| EP4056123A1 (en) * | 2019-11-06 | 2022-09-14 | Daegu Gyeongbuk Institute Of Science and Technology | Three-dimensional diagnostic system |
| CN115245312A (en) * | 2021-04-27 | 2022-10-28 | 山东威高宏瑞医学科技有限公司 | Endoscope multispectral image processing system and processing and training method |
| CN116363199A (en) * | 2023-01-28 | 2023-06-30 | 北京理工大学 | Bronchoscope positioning method and device |
| CN116452752A (en) * | 2023-04-28 | 2023-07-18 | 重庆理工大学 | Intestinal wall reconstruction method combining monocular dense SLAM and residual error network |
| WO2023192306A1 (en) * | 2022-03-29 | 2023-10-05 | Activ Surgical, Inc. | Systems and methods for multispectral and mosaic imaging |
Family Cites Families (4)
| Publication number | Priority date | Publication date | Assignee | Title |
|---|---|---|---|---|
| DE102010009905A1 (en) * | 2010-03-02 | 2011-09-08 | Friedrich-Alexander-Universität Erlangen-Nürnberg | Method and device for acquiring information about the three-dimensional structure of the inner surface of a body cavity |
| US10198872B2 (en) * | 2015-08-10 | 2019-02-05 | The Board Of Trustees Of The Leland Stanford Junior University | 3D reconstruction and registration of endoscopic data |
| US11503987B2 (en) * | 2018-03-26 | 2022-11-22 | Intuitive Surgical Operations, Inc. | Plenoptic endoscope with fiber bundle |
| US20220012874A1 (en) * | 2018-07-31 | 2022-01-13 | Deutsches Krebsforschungszentrum Stiftung des öffentlichen Rechts | Method and system for augmented imaging using multispectral information |
-
2023
- 2023-11-09 CN CN202311486571.6A patent/CN117204796B/en active Active
Patent Citations (9)
| Publication number | Priority date | Publication date | Assignee | Title |
|---|---|---|---|---|
| CN104814712A (en) * | 2013-11-07 | 2015-08-05 | 南京三维视嘉科技发展有限公司 | Three-dimensional endoscope and three-dimensional imaging method |
| CN104083141A (en) * | 2013-11-21 | 2014-10-08 | 北京华科创智健康科技股份有限公司 | Multispectral combined output light source device and endoscope device and endoscope device |
| CN107260117A (en) * | 2016-03-31 | 2017-10-20 | 柯惠有限合伙公司 | Chest endoscope for surface scan |
| CN106303279A (en) * | 2016-08-31 | 2017-01-04 | 北京数字精准医疗科技有限公司 | Multi-spectrum endoscopic automatic exposure formation method |
| EP4056123A1 (en) * | 2019-11-06 | 2022-09-14 | Daegu Gyeongbuk Institute Of Science and Technology | Three-dimensional diagnostic system |
| CN115245312A (en) * | 2021-04-27 | 2022-10-28 | 山东威高宏瑞医学科技有限公司 | Endoscope multispectral image processing system and processing and training method |
| WO2023192306A1 (en) * | 2022-03-29 | 2023-10-05 | Activ Surgical, Inc. | Systems and methods for multispectral and mosaic imaging |
| CN116363199A (en) * | 2023-01-28 | 2023-06-30 | 北京理工大学 | Bronchoscope positioning method and device |
| CN116452752A (en) * | 2023-04-28 | 2023-07-18 | 重庆理工大学 | Intestinal wall reconstruction method combining monocular dense SLAM and residual error network |
Also Published As
| Publication number | Publication date |
|---|---|
| CN117204796A (en) | 2023-12-12 |
Similar Documents
| Publication | Publication Date | Title |
|---|---|---|
| CN111414798B (en) | Head posture detection method and system based on RGB-D image | |
| Aguilar et al. | A robust graph transformation matching for non-rigid registration | |
| US9599461B2 (en) | Surface data acquisition, storage, and assessment system | |
| CN117204796B (en) | Multispectral imaging method and device of abdominal cavity endoscope | |
| US20120020573A1 (en) | Image analysis systems using non-linear data processing techniques and methods using same | |
| JP2021530061A (en) | Image processing methods and their devices, electronic devices and computer-readable storage media | |
| CN109215079B (en) | Image processing method, surgical navigation device, electronic device, and storage medium | |
| CN109146948A (en) | The quantization of crop growing state phenotypic parameter and the correlation with yield analysis method of view-based access control model | |
| DE102006054822A1 (en) | Virtual biological object`s e.g. colon, characteristics paths e.g. prone position, regulating method for e.g. angioscopy, involves measuring correlation between object paths by minimizing energy function that includes error and switch terms | |
| EP3472803A1 (en) | Method and apparatus for estimating the value of a physical parameter in a biological tissue | |
| CN112232109A (en) | Living body face detection method and system | |
| CN106709967A (en) | Endoscopic imaging algorithm and control system | |
| CN111524224B (en) | Panoramic imaging method for surface temperature distribution of power transformer | |
| US20120263371A1 (en) | Method of image fusion | |
| US10401141B2 (en) | Method and apparatus for obtaining a three-dimensional map of tympanic membrane thickness | |
| Li et al. | Sublingual vein extraction algorithm based on hyperspectral tongue imaging technology | |
| WO2017086433A1 (en) | Medical image processing method, device, system, and program | |
| CN112215878B (en) | X-ray image registration method based on SURF feature points | |
| CN112261399B (en) | Capsule endoscope image three-dimensional reconstruction method, electronic device and readable storage medium | |
| CN109816665B (en) | Rapid segmentation method and device for optical coherence tomography image | |
| Li | Hyperspectral imaging technology used in tongue diagnosis | |
| Jourlin et al. | Asplünd׳ s metric defined in the Logarithmic Image Processing (LIP) framework: A new way to perform double-sided image probing for non-linear grayscale pattern matching | |
| CN112991186B (en) | Unmanned aerial vehicle large-field-of-view hyperspectral image generation method and system | |
| Gallo et al. | Breast shape parametrization through planar projections | |
| Rattanalappaiboon et al. | Fuzzy zoning for feature matching technique in 3D reconstruction of nasal endoscopic images |
Legal Events
| Date | Code | Title | Description |
|---|---|---|---|
| PB01 | Publication | ||
| PB01 | Publication | ||
| SE01 | Entry into force of request for substantive examination | ||
| SE01 | Entry into force of request for substantive examination | ||
| GR01 | Patent grant | ||
| GR01 | Patent grant |