CN109771052B

CN109771052B - Three-dimensional image establishing method and system based on multi-view imaging and multi-polarization state imaging

Info

Publication number: CN109771052B
Application number: CN201811625313.0A
Authority: CN
Inventors: 舒远; 李梓彤; 陈维宇; 刘楚明; 阮思纯; 徐炜文; 王星泽
Original assignee: Heren Technology Shenzhen Co ltd
Current assignee: Heren Technology Shenzhen Co ltd
Priority date: 2018-12-28
Filing date: 2018-12-28
Publication date: 2021-07-27
Anticipated expiration: 2038-12-28
Also published as: CN109771052A

Abstract

The application provides a three-dimensional image establishing method and a system based on multi-view imaging and multi-polarization state imaging, wherein the method comprises the following steps: the wide-spectrum light source generates wide-band wide-spectrum light, the spectrum imaging device collects a plurality of narrow-band spectrum images formed by reflection of the wide-spectrum light by the operation area in multiple angles, a primary three-dimensional reconstruction image of the operation area is constructed by utilizing a multi-view stereo vision algorithm, the multi-polarization state imaging device collects a plurality of narrow-band polarization images formed by reflection of the polarization light by the operation area, a body fluid covering image of a human organ in the operation area is obtained, and the processing system obtains a three-dimensional reconstruction image of the operation area according to the primary three-dimensional reconstruction image and the body fluid covering image of the human organ and outputs the three-dimensional reconstruction image to the display device.

Description

Three-dimensional image establishing method and system based on multi-view imaging and multi-polarization state imaging

Technical Field

The application relates to the field of electronics, in particular to a three-dimensional image establishing method and system based on multi-view imaging and multi-polarization imaging.

Background

With the increasing public health importance and the increasing medical level, surgical operations are also more common. In the surgical operation, the success rate and the effect of the operation are closely related to the accuracy of the operation, the traditional operation depends on the experience of a doctor, the operation tool and the position of a pathological change are visually judged, the accuracy is not high, and the requirement on the doctor is also high.

However, how to improve the precision of the surgical operation is a problem which is not solved so far.

Disclosure of Invention

The application provides a three-dimensional image establishing method and a three-dimensional image establishing system based on multi-view imaging and multi-polarization imaging, which can effectively improve the precision of a surgical operation.

In a first aspect, a three-dimensional image building method based on multi-view imaging and multi-polarization imaging is provided, the method comprising the following steps:

the broad spectrum light source generates broad spectrum light with a wide wave band, wherein the broad spectrum light is used for illuminating an operation area, the broad spectrum light comprises natural light and polarized light, and the operation area is a body area of a patient to be operated;

the spectral imaging device collects a plurality of narrow-band spectral images formed by the reflection of the broad-spectrum light rays by the operation area in a multi-angle mode, and a primary three-dimensional reconstruction image of the operation area is constructed by utilizing a multi-view stereo vision algorithm, wherein different narrow-band spectral images have different imaging narrow bands, and the number of the imaging narrow bands is more than 100;

collecting a plurality of narrow-band polarized images formed by reflecting the polarized light rays by the operation area by a multi-polarization-state imaging device to obtain a body fluid coverage image of a human organ in the operation area;

and the processing system obtains the three-dimensional reconstruction image of the operation area according to the primary three-dimensional reconstruction image and the body fluid coverage image of the human organ and outputs the three-dimensional reconstruction image to the display device.

Optionally, the method further comprises: the processing system performs real-time analysis on the body fluid component types and the organ types in the operation region by combining a classification model according to a plurality of narrow-band polarization images and outputs an analysis result, wherein the real-time analysis comprises organ type analysis, protein content analysis, crystal type analysis and crystal content analysis, and the classification model is a body fluid classification model and an organ classification model obtained by training a convolutional neural network by using sample data in advance;

determining whether the protein content or the crystal content in the analysis result is greater than a first threshold value or less than a second threshold value, wherein the first threshold value is greater than the second threshold value;

and sending out prompt information under the condition that the protein content or the crystal content in the analysis result is greater than the first threshold value or less than the second threshold value.

Optionally, after obtaining the three-dimensional reconstructed image of the surgical region, the method further comprises:

and rendering and reinforcing the target area of the three-dimensional reconstruction image, and outputting the reinforced three-dimensional reconstruction image, wherein the target area comprises a shielded area in the operation area and a specific area in the operation area.

Optionally, the rendering and enhancing the target region of the three-dimensional reconstructed image includes:

under the condition that the target area is a shielded area in the operation area, removing a part for shielding the target area, and rendering the target area which is not shielded; alternatively, the first and second electrodes may be,

and when the target area is a specific area in the operation area, improving the brightness of the specific area, and rendering the target area with the improved brightness.

Optionally, the target area is an area determined by the processing system according to position information sent by a surgical tool, wherein one or more of a gyroscope, an accelerometer and a distance sensor are arranged in the surgical tool, and the position information is calculated according to position data acquired by one or more of the gyroscope, the accelerometer and the distance sensor; alternatively, the first and second electrodes may be,

the target area is an area determined by the processing system based on the received user request.

In a second aspect, a three-dimensional image building system based on multi-view imaging and multi-polarization state imaging is provided, the system comprising:

a broad spectrum light source that generates broad spectrum light of a broad band, wherein the broad spectrum light is used to illuminate a surgical field, the broad spectrum light comprising natural light and polarized light, the surgical field being a body region of a patient being operated on;

the spectral imaging equipment is used for collecting a plurality of narrow-band spectral images formed by reflecting the broad-spectrum light rays by the operation area in a multi-angle mode, and constructing a primary three-dimensional reconstruction image of the operation area by utilizing a multi-view stereo vision algorithm, wherein different narrow-band spectral images have different imaging narrow bands, and the number of the imaging narrow bands is more than 100;

the multi-polarization-state imaging device is used for collecting a plurality of narrow-band polarization images formed by reflecting the broad-spectrum light rays by the operation area and obtaining a body fluid covering image of a human organ in the operation area;

and the processing system is used for obtaining the three-dimensional reconstruction image of the operation area according to the primary three-dimensional reconstruction image and the body fluid coverage image of the human organ and outputting the three-dimensional reconstruction image to display equipment.

Optionally, the processing system is further configured to perform, according to the plurality of narrow-band polarization images, an instant analysis on the body fluid component types and the organ types in the surgical region by combining a classification model, and output an analysis result, where the instant analysis includes an organ type analysis, a protein content analysis, a crystal type analysis, and a crystal content analysis, and the classification model is a body fluid classification model and an organ classification model obtained by training a convolutional neural network in advance by using sample data;

the processing system is further configured to determine whether the analysis result is greater than a first threshold or less than a second threshold, wherein the first threshold is greater than the second threshold;

the processing system is further configured to send out a prompt message when the analysis result is greater than the first threshold or less than the second threshold.

Optionally, the processing system is further configured to, after obtaining the three-dimensional reconstruction image of the operation region, perform rendering enhancement on the three-dimensional reconstruction image of the operation region, and output the enhanced three-dimensional reconstruction image, where the target region includes a blocked region in the operation region and a specific region in the operation region.

Optionally, the processing system is specifically configured to, in a case that the target area is an area that is occluded in the operation area, remove a portion that occludes the target area, and render an unobstructed target area; alternatively, the first and second electrodes may be,

the processing system is specifically used for improving the brightness of the specific area and rendering the target area with improved brightness under the condition that the target area is the specific area in the operation area.

According to the three-dimensional image establishing method and system based on multi-view imaging and multi-polarization-state imaging, broad-spectrum light rays with wide wave bands are generated through a broad-spectrum light source, spectrum imaging equipment collects a plurality of narrow-band spectrum images formed by reflecting the broad-spectrum light rays by an operation area in a multi-view stereoscopic vision algorithm, a primary three-dimensional reconstruction image of the operation area is constructed, multi-polarization-state imaging equipment collects a plurality of narrow-band polarization images formed by reflecting the polarized light rays by the operation area, body fluid covering images of human organs in the operation area are obtained, and a processing system obtains the three-dimensional reconstruction image of the operation area according to the primary three-dimensional reconstruction image and the body fluid covering images of the human organs and outputs the three-dimensional reconstruction image to display equipment. The three-dimensional image with higher accuracy is reconstructed by obtaining richer effective information according to the plurality of narrow-band spectral images and the plurality of narrow-band polarization images, and the reconstructed three-dimensional image can be rendered and enhanced according to the requirements of a doctor by a virtual enhancement technology, so that the doctor is guided or assisted to perform the operation correctly, and the accuracy of the operation is improved.

Drawings

Fig. 1 is a schematic flowchart of a three-dimensional image building method based on multi-view imaging and multi-polarization imaging according to the present application;

FIG. 2 is a schematic diagram of arrangement of spectral light sources in a three-dimensional image building method based on multi-view imaging and multi-polarization imaging provided by the present application;

FIG. 3 is a schematic diagram of a prompt message by way of augmented reality according to the present disclosure;

FIG. 4a is a reconstructed image of an operation region before augmented reality according to the present application;

FIG. 4b is a reconstructed image of an augmented reality surgical field according to the present application;

FIG. 5a is a reconstructed image of an alternate pre-augmented reality surgical field provided herein;

FIG. 5b is a reconstructed image of the surgical field after augmented reality as provided herein;

fig. 6 is a schematic structural diagram of a three-dimensional image creating system based on multi-view imaging and multi-polarization imaging provided by the present application.

DETAILED DESCRIPTION OF EMBODIMENT (S) OF INVENTION

Referring to fig. 1, fig. 1 is a schematic flowchart of a three-dimensional image establishing method based on multi-view imaging and multi-polarization imaging according to an embodiment of the present application. The three-dimensional image establishing method based on multi-view imaging and multi-polarization imaging comprises the following steps:

s101: the broad spectrum light source generates broad spectrum light of a broad wavelength band, wherein the broad spectrum light is used to illuminate a surgical field, the broad spectrum light comprises natural light and polarized light, and the surgical field is a body region where a patient is operated on.

In a specific embodiment of the present application, a broad spectrum light source is used to generate broad spectrum light over a broad band. The broad spectrum light source may produce wavelengths in a range extending from infrared to ultraviolet, from infrared to visible, or from visible to ultraviolet, and the like. In practice, the number of broad spectrum light sources may be one or more. In one embodiment, the broad spectrum light generated by the plurality of broad spectrum light sources has the same wavelength range, for example, the first broad spectrum light source generates broad spectrum light with a wavelength range of 350 nm to 850 nm, the second broad spectrum light source generates broad spectrum light with a wavelength range of 350 nm to 850 nm, and so on. In another embodiment, the broad spectrum light sources generate different wavelength ranges of the broad spectrum light, for example, the first broad spectrum light source generates broad spectrum light with a wavelength range of 350 nm to 550 nm, the second broad spectrum light source generates broad spectrum light with a wavelength range of 550 nm to 850 nm, and so on.

In the above examples, the broad spectrum light generated by the broad spectrum light source is illustrated as a continuous broad band, and in practical applications, the broad spectrum light generated by the broad spectrum light source may also be a discontinuous broad band, for example, the broad spectrum light generated by the first broad spectrum light source includes two bands, one band of the broad spectrum light has a wavelength range of 350 nm to 450 nm, and the other band of the broad spectrum light has a wavelength range of 650 nm to 850 nm, and the like, and is not limited herein.

In a specific embodiment of the present application, the polarized light is obtained by passing a broad spectrum of light through a polarizer, that is, a polarizer is disposed below a portion of the broad spectrum light source for obtaining the polarized light, and a polarizer is not disposed below another portion of the broad spectrum light source for obtaining the natural light.

In a specific embodiment of the present application, the arrangement of the broad spectrum light sources may be horizontal, vertical, or oblique, or a combination of multiple arrangements, for example, fig. 2 is an arrangement of the broad spectrum light sources provided in this application, where a group a of light sources have polarizers disposed below them for obtaining polarized light, a group B of light sources have no polarizers below them, a1 and a B1 of light sources are vertical irradiation light source arrays, a2, B2, A3, and B3 are oblique light sources respectively disposed at two sides of the a1 and B1 light sources and form an angle θ with the horizontal direction, it can be understood that the arrangement of the light sources shown in fig. 2 is beneficial to fully mining biological characteristic information due to different collected biological tissue characteristic information under different angles of light beams, and it should be understood that the arrangement of the broad spectrum light sources shown in fig. 2 is only used for illustration, and not to be limited in particular, the broad spectrum light source of the present application may have more vertical light source arrays and tilted angle light source arrays.

In a specific embodiment of the present application, the operation area is a body area of a patient to be operated on, and not an entire operation table, and the operation area may include: a surgical site at which the patient is operated, surgical tools placed at the surgical site, and surgical markers. The operation position can be the position needing operation on the patient and the position near the position needing operation. The surgical tool may be a scalpel, a pair of surgical scissors, a surgical retractor, a distractor, and the like. The surgical marker may be a surgical marker that facilitates positioning and is not particularly limited herein.

S102: the spectral imaging device collects a plurality of narrow-band spectral images formed by reflecting the broad-spectrum light rays by the operation area in a multi-angle mode, and a primary three-dimensional reconstruction image of the operation area is constructed by utilizing a multi-view stereo vision algorithm, wherein different narrow-band spectral images have different imaging narrow bands, and the number of the imaging narrow bands is more than 100.

In particular embodiments of the present application, a spectral imaging device may form a plurality of narrow-band spectral images. The spectral imaging device may be a plurality of multi-view spectral imaging devices arranged at multiple angles, or may be a combination of a hyperspectral imaging device and a depth camera, or may be a hyperspectral imaging device or a hyperspectral imaging device, and the like, and the application is not limited specifically. The hyperspectral imaging equipment simultaneously images the operation area by hundreds to thousands of continuous and subdivided spectral wave bands, and the hyperspectral imaging equipment simultaneously images the operation area by thousands to tens of thousands of continuous and subdivided spectral wave bands. Of course, the spectral imaging apparatus may also simultaneously image the surgical region with more continuous and subdivided spectral bands, such as tens of thousands to hundreds of thousands of continuous and subdivided spectral bands, and the like, and is not particularly limited herein.

In the specific implementation manner of the application, the spectral imaging device can identify and classify the targets by using the spectral difference of the surface components of the object, that is, different narrow-band spectral images have different imaging narrow bands, and the narrow-band spectral images obtained by shooting at different angles are integrated to obtain a wider field of view, so that richer information can be obtained, and then the multi-view stereo vision algorithm is used to realize three-dimensional reconstruction. For example, human skin can better reflect a narrow imaging band, human muscle can better reflect b narrow imaging band, human blood vessel can better reflect c narrow imaging band, and human skeleton can better reflect d narrow imaging band. Therefore, an A narrow-band spectral image reflecting the skin condition of the human body can be obtained according to the a imaging narrow band, a B narrow-band spectral image reflecting the muscle condition of the human body can be obtained according to the B imaging narrow band, a C narrow-band spectral image reflecting the blood vessel condition of the human body can be obtained according to the C imaging narrow band, and a D narrow-band spectral image reflecting the bone condition of the human body can be obtained according to the D imaging narrow band. It can be understood that when the subdivision degree of the spectral band is higher, the visual field of the spectral imaging device is wider, the information obtained by the spectral generation device is natural and richer, the obtained human body information is more accurate, and a more accurate three-dimensional reconstruction model is further realized.

Preferably, the spectral imaging device may perform three-dimensional reconstruction of the operation region by using a binocular stereo vision matching (SAD) algorithm, that is, narrow-band spectral images of different angles obtained by using the binocular hyperspectral imaging device, and summing Absolute Differences between values corresponding to pixels in each image to evaluate similarity of the narrow-band spectral images of different angles, so as to obtain the disparity map. After the parallax information is obtained, the depth information and the three-dimensional information of the original image can be obtained according to the projection model. Certainly, the spectral imaging apparatus may also use other multi-view stereo vision algorithms such as a Block Matching (BM) algorithm, a Semi-Global Block Matching (SGM) algorithm, and the like to perform preliminary three-dimensional reconstruction of the operation region, which is not specifically limited in this application.

S103: and acquiring a plurality of narrow-band polarized images formed by reflecting the polarized light rays by the operation area by the multi-polarization-state imaging equipment, and acquiring a body fluid coverage image of the human organ in the operation area.

In particular embodiments of the present application, a multi-polarization state imaging device may form multiple narrow-band polarization images. The multi-polarization imaging device may be one or more than one, and the multi-polarization imaging device may be an orthogonal multi-polarization imaging device or an interference multi-polarization imaging device, or the like. The orthogonal multi-polarization imaging device simultaneously images the operation area by hundreds to thousands of continuous and subdivided spectral wave bands, and the interference multi-polarization imaging device simultaneously images the operation area by thousands to tens of thousands of continuous and subdivided spectral wave bands. Of course, the multi-polarization imaging device may also simultaneously image the surgical area with more continuous and subdivided polarized light bands, for example, tens of thousands to hundreds of thousands of continuous and subdivided polarized light bands, and the like, and is not limited herein.

In a specific embodiment of the present application, the imaging narrow band corresponding to the narrow-band polarization image and the imaging narrow band corresponding to the narrow-band spectral image may be the same, for example: the blood vessel of the human body can better reflect the C imaging narrow band, so that a C narrow band spectrum image and a C narrow band polarization image reflecting the condition of the blood vessel of the human body can be obtained according to the C imaging narrow band. The imaging narrow band corresponding to the narrow band polarized image may also be different from the imaging narrow band corresponding to the narrow band spectral image, for example: the blood vessels of the human body can better reflect the C1 imaging narrow wave band, and better reflect the C2 polarization imaging narrow wave band, so that a C1 narrow wave band spectrum image reflecting the blood vessel condition of the human body can be obtained according to the C1 imaging narrow wave band, and a C2 narrow wave band polarization image reflecting the blood vessel condition of the human body can be obtained according to the C2 polarization imaging narrow wave band. It is to be understood that the above examples are illustrative only and are not to be construed as limiting in any way.

In the specific implementation manner of the application, the multi-polarization-state imaging device can obtain the polarization characteristic parameters of the operation area according to the polarization angle, the polarization degree and the ellipticity angle of the light beams in the multiple narrow-band polarization images, and provide signs of organ surface roughness, texture trend, surface orientation, water content and the like of the operation area. Therefore, the three-dimensional reconstruction of the operation area is carried out by combining the spectral imaging equipment and the multi-polarization state imaging equipment, and more complete and reliable information which cannot be extracted by human eyes can be provided for doctors. That is to say, the spectral imaging device may extract the features of the whole surgical region, and the primary three-dimensional reconstruction of the surgical region is implemented by using a multi-view stereo vision algorithm, and the multi-polarization imaging device may extract the features of the contours and surfaces of the organ surface roughness, the muscle texture trend, the organ surface covered liquid type, and the like, which cannot be displayed by the spectral imaging device, to obtain the image of the surface roughness or the body fluid covered condition of the human organ clearly described in the surgical region.

S104: and the processing system obtains the three-dimensional reconstruction image of the operation area according to the primary three-dimensional reconstruction image and the body fluid coverage image of the human organ and outputs the three-dimensional reconstruction image to the display device.

In an embodiment of the application, a processing system constructs a three-dimensional image of the surgical field from the plurality of narrowband spectral images and the plurality of narrowband polarization images. That is, the processing system integrates the information respectively extracted by the narrow-band spectral image and the narrow-band polarization image, and establishes a detailed and rich operation area three-dimensional information map. For example: the processing system may reconstruct a three-dimensional image a of the skin from the a narrow band spectral images at different angles, reconstructing a three-dimensional image B of muscles according to the B narrow-band spectral images at different angles, reconstructing a three-dimensional image C of blood vessels according to the C narrow-band spectral images at different angles, reconstructing a three-dimensional image D of the skeleton according to the D narrow-band spectral images at different angles, reconstructing a three-dimensional image E of the microvasculature according to the E narrow-band polarized images at different angles, reconstructing a three-dimensional image F of the covering liquid on the surface of each organ according to the F narrow-band polarization images at different angles, fusing the three-dimensional image A, the three-dimensional image B, the three-dimensional image C, the three-dimensional image D, the three-dimensional image E and the three-dimensional image F, a detailed and real three-dimensional image of the surgical area can be obtained which simultaneously reflects the roughness of the skin, muscle, blood vessels, bones, micro-vessels and organs.

In a specific embodiment of the present application, the method further comprises: the processing system performs real-time analysis on the body fluid component types and the organ types in the operation region by combining a classification model according to a plurality of narrow-band polarization images and outputs an analysis result, wherein the real-time analysis comprises protein type analysis, protein content analysis, crystal type analysis and crystal content analysis, and the classification model is a body fluid classification model and an organ classification model obtained by training a convolutional neural network by using sample data in advance; determining whether the protein content or the crystal content in the analysis result is greater than a first threshold value or less than a second threshold value, wherein the first threshold value is greater than the second threshold value; and sending out prompt information under the condition that the protein content or the crystal content in the analysis result is greater than the first threshold value or less than the second threshold value. The body fluid classification model and the organ classification model can be trained classification models obtained by training a convolutional neural network by using a large number of body fluid picture samples and organ samples marked with prediction results in advance, the body fluid classification models can obtain body fluid classification results according to narrow-band polarization images containing organ surface water content characteristics obtained by a multi-polarization state imaging system, the organ classification models can obtain organ classification results according to narrow-band polarization images containing organ surface contour characteristics obtained by the multi-polarization state imaging system, the organ classification results are output to reality enhancing or mixed reality equipment worn by a doctor, and more information is provided for the doctor in the operation process so as to further judge the disease condition and perform the operation. Wherein, the prompt message can be one of sound, light or text. For example, as shown in fig. 3, a text can be added to the three-dimensional image by an augmented reality or mixed reality method to remind the doctor about the abnormal hemoglobin content in the blood of the patient during the operation. Moreover, the multi-polarization-state imaging device can also perform judgment of organ classification by combining a neural network model according to polarized light information, and can also add characters on a three-dimensional image by a method of augmented reality or mixed reality to remind a doctor according to a judgment result, which is not repeated in the application.

In a specific embodiment of the present application, the processing system constructing the three-dimensional image of the surgical region according to the plurality of narrowband spectral images and the plurality of narrowband polarization images may specifically be: the processing system positions and segments the surgical site, the surgical tool and the surgical marker to obtain a positioning segmentation result, determines relative positions among the surgical site, the surgical tool and the surgical marker according to the positioning segmentation result, and reconstructs a three-dimensional image according to the relative positions among the surgical site, the surgical tool and the surgical marker. It will be appreciated that a greater abundance of information is available from the plurality of narrowband spectral images and the plurality of narrowband polarization images, and thus, the surgical site, the surgical tool, and the surgical marker may be more accurately located and segmented.

In a specific embodiment of the present application, the processing system may specifically locate and segment the surgical site, the surgical tool, and the surgical marker to obtain a location segmentation result by: after obtaining the plurality of narrowband spectral images and the plurality of narrowband polarization images, the processing system may extract texture features of the plurality of narrowband spectral images and the plurality of narrowband polarization images with a feature extraction algorithm (e.g., a convolutional neural network). Then, the mark made before the operation, the operation tool and the operation related area are accurately positioned or segmented by combining the image recognition algorithm (such as the convolutional neural network) and the motion tracking algorithm, so that the positioning segmentation result is obtained.

In a specific embodiment of the present application, after obtaining the three-dimensional reconstructed image of the surgical field, the method further comprises: and rendering and reinforcing the target area of the three-dimensional reconstruction image, and outputting the reinforced three-dimensional reconstruction image, wherein the target area comprises a shielded area in the operation area and a specific area in the operation area. The rendering and strengthening of the target area of the three-dimensional reconstruction image comprises the following steps: under the condition that the target area is a shielded area in the operation area, removing a part for shielding the target area, and rendering the target area which is not shielded; or, when the target area is a specific area in the operation area, the brightness of the specific area is increased, and the target area with the increased brightness is rendered.

Preferably, the rendered target region which is not occluded may be obtained by using a trained occlusion supplementary model obtained by training a symmetric convolutional neural network (symmet) using a large number of binocular images and images of the occlusion objects in the binocular images as samples, the occlusion supplementary model learns occlusion features in the large number of binocular images, and a network model which can predict the occlusion problem of the image of the multi-view spectral imaging apparatus by a bilateral symmetric structure. The rendering and enhancing result by using the occlusion supplementary model can bring more operation information to the operation of the doctor, for example, when a certain operation region is as shown in fig. 4a, and when the target region is the occluded region in the operation region, the doctor can see the rendered and enhanced three-dimensional reconstruction image as shown in fig. 4b by using Augmented Reality (AR) glasses, and perform the operation according to the enhanced three-dimensional reconstruction image. It is understood that the doctor can only perform the operation according to experience before the rendering and strengthening of the target area are not performed, and after the rendering and strengthening of the target area are performed, the doctor can use the AR glasses to assist or guide the operation according to the rendered target area image which is not blocked. It should be understood that the above example is only used for illustration, and the rendering of the occlusion region can also be obtained by using other algorithms, for example, using a Left-right cross-checking (LRC) algorithm to infer the occlusion position through the pre-computed parallax result, and the present application is not limited in particular.

Preferably, the target region after brightness enhancement is rendered may be obtained by using a trained brightness supplementary model, which is obtained by training a convolutional neural network by using operation region sample pictures before and after brightness enhancement, and the result after the brightness supplementary model is rendered and enhanced may bring more operation information for the operation of the doctor, for example, as shown in fig. 5a, in the case that the target region is a specific region in the operation region, for example, some lesion regions or shadow regions, which cannot be clearly seen by the doctor, the doctor may see the rendered and enhanced three-dimensional reconstructed image shown in fig. 5b after using AR glasses, and perform the operation according to the enhanced three-dimensional reconstructed image. It can be understood that the doctor can only perform the operation according to experience before the rendering and strengthening of the target area are not performed, and after the brightness of the target area is increased, the doctor can use the AR glasses to assist or guide the operation according to the rendered brightness-increased target area image.

In a specific embodiment of the present application, the target area is an area determined by a processing system according to position information sent by a surgical tool, wherein one or more of a gyroscope, an accelerometer, and a distance sensor are disposed in the surgical tool, and the position information is calculated according to position data acquired by the gyroscope, the accelerometer, and the one or more of the distance sensor. For example: when a doctor scalpel is placed in a certain lesion area, the AR glasses of the doctor automatically display the three-dimensional reconstruction image with the highlighted lesion area. It is to be understood that the above examples are illustrative only and are not to be construed as limiting in any way.

In a specific embodiment of the present application, the target area may also be an area determined by the processing system according to a received user request. For example: a doctor needs to check a certain blocked target area in the operation process, at the moment, the doctor can select the blocked target area needing to be checked by using AR equipment or a doctor assistant through using other terminal equipment frames connected with the processing system, the processing system determines the target area according to the received target area information, and the unblocked target area image which is rendered by the doctor is input into AR glasses of the doctor. It is to be understood that the above examples are illustrative only and are not to be construed as limiting in any way.

In a specific embodiment of the present application, the output of the enhanced three-dimensional reconstructed image may be output to a display screen of a head-mounted display device worn by a doctor, such as AR glasses, Mixed Reality (MR) glasses, and the like; or output to a terminal device display screen, such as a computer display screen, a flat panel display screen, etc.; and the surgical guide or the auxiliary surgical system can be output to a mechanical surgical system to provide surgical guidance or assistance for the mechanical arm to perform the surgery.

In the above scheme, a broad-spectrum light source generates broad-spectrum light with a wide band, a spectral imaging device collects a plurality of narrow-band spectral images formed by reflecting the broad-spectrum light by the operation area in multiple angles, a primary three-dimensional reconstruction image of the operation area is constructed by using a multi-view stereo vision algorithm, a multi-polarization state imaging device collects a plurality of narrow-band polarization images formed by reflecting the polarized light by the operation area, and a body fluid covering image of a human organ in the operation area is obtained, so that a processing system obtains the three-dimensional reconstruction image of the operation area according to the primary three-dimensional reconstruction image and the body fluid covering image of the human organ, and outputs the three-dimensional reconstruction image to a display device. According to the method, richer effective information can be obtained according to the plurality of narrow-band spectral images and the plurality of narrow-band polarization images, so that a three-dimensional image with higher accuracy is reconstructed, and the reconstructed three-dimensional image can be rendered and enhanced according to the requirements of a doctor through a virtual enhancement technology, so that the doctor is guided or assisted to perform the operation correctly, and the accuracy of the operation is improved.

Referring to fig. 6, fig. 6 is a schematic structural diagram of a three-dimensional image creating system based on multi-view imaging and multi-polarization imaging according to the present application. In the embodiment of the present application, a three-dimensional image creating system based on multi-view imaging and multi-polarization imaging includes: a broad spectrum light source 110, a spectral imaging device 120, a multi-polarization state imaging device 130, and a processing system 140.

The broad spectrum light source 110 is used for generating broad spectrum light of a wide wavelength band, wherein the broad spectrum light is used for illuminating a surgical area, the broad spectrum light comprises natural light and polarized light, and the surgical area is a body area of a patient to be operated on;

the spectral imaging device 120 is configured to collect multiple narrow-band spectral images formed by the reflection of the broad-spectrum light by the surgical area in multiple angles, and construct a primary three-dimensional reconstructed image of the surgical area by using a multi-view stereo vision algorithm, where different narrow-band spectral images have different imaging narrow bands, and the number of the imaging narrow bands is greater than 100;

the multi-polarization state imaging device 130 is configured to collect a plurality of narrow-band polarization images formed by the reflection of the broad-spectrum light by the operation region, and obtain a body fluid coverage image of a human organ in the operation region;

the processing system 140 is configured to obtain a three-dimensional reconstructed image of the operation area according to the primary three-dimensional reconstructed image and the body fluid covered image of the human body organ, and output the three-dimensional reconstructed image to a display device. .

In a specific embodiment of the present application, broad spectrum light source 110 is used to generate broad spectrum light over a broad band. The broad spectrum light source may produce wavelengths in a range extending from infrared to ultraviolet, from infrared to visible, or from visible to ultraviolet, and the like. In practical applications, the number of the broad spectrum light sources 110 may be one or more. In one embodiment, the broad spectrum light sources 110 generate broad spectrum light with the same wavelength range, for example, the first broad spectrum light source generates broad spectrum light with a wavelength range of 350 nm to 850 nm, the second broad spectrum light source generates broad spectrum light with a wavelength range of 350 nm to 850 nm, and so on. In another embodiment, the broad spectrum light sources generate different wavelength ranges of the broad spectrum light, for example, the first broad spectrum light source generates broad spectrum light with a wavelength range of 350 nm to 550 nm, the second broad spectrum light source generates broad spectrum light with a wavelength range of 550 nm to 850 nm, and so on.

In the above examples, the broad spectrum light generated by the broad spectrum light source 110 is illustrated as a continuous broad band, and in practical applications, the broad spectrum light generated by the broad spectrum light source 110 may also be a discontinuous broad band, for example, the broad spectrum light generated by the first broad spectrum light source includes two bands, one band of the broad spectrum light has a wavelength range of 350 nm to 450 nm, and the other band of the broad spectrum light has a wavelength range of 650 nm to 850 nm, and the like, and is not limited herein.

In a specific embodiment of the present application, the polarized light is obtained by passing a broad spectrum of light through a polarizer, that is, a polarizer is disposed below a portion of the broad spectrum light sources 110 for obtaining the polarized light, and a polarizer is not disposed below another portion of the broad spectrum light sources 110 for obtaining the natural light, it is understood that the polarized light is used for collecting the polarized image by the multi-polarization state imaging device, and the natural light is used for collecting the spectral image by the spectral imaging device.

In a specific embodiment of the present application, the arrangement of the broad spectrum light sources 110 may be a horizontal arrangement, a vertical arrangement, an oblique arrangement, or a combination of multiple arrangements, for example, fig. 2 is an arrangement of the broad spectrum light sources provided in this application, where a group a of light sources have polarizers disposed below them for obtaining polarized light, a group B of light sources have no polarizers below them, a1 and a B1 of light sources are vertical irradiation light source arrays, a2, a2, A3 and A3 are oblique light sources respectively disposed at two sides of the a1 and B1 of light sources and form an angle θ with the horizontal direction, it can be understood that the arrangement of the light sources shown in fig. 2 is beneficial to fully mining biological characteristic information due to different collected biological tissue characteristic information under different angles of light beams, and it should be understood that the arrangement of the broad spectrum light sources shown in fig. 2 is only used for illustration, and not to be limited in particular, the broad spectrum light source of the present application may have more vertical light source arrays and tilted angle light source arrays.

In particular embodiments of the present application, spectral imaging device 120 may form a plurality of narrow-band spectral images. The spectral imaging device 120 may be a plurality of multi-view spectral imaging devices arranged at multiple angles, or may be a combination of a hyperspectral imaging device and a depth camera, and the spectral imaging device 120 may also be a hyperspectral imaging device or a hyperspectral imaging device, and the like, which is not limited in this application. The hyperspectral imaging equipment simultaneously images the operation area by hundreds to thousands of continuous and subdivided spectral wave bands, and the hyperspectral imaging equipment simultaneously images the operation area by thousands to tens of thousands of continuous and subdivided spectral wave bands. Of course, the spectral imaging apparatus may also simultaneously image the surgical region with more continuous and subdivided spectral bands, such as tens of thousands to hundreds of thousands of continuous and subdivided spectral bands, and the like, and is not particularly limited herein.

In the specific implementation manner of the application, the spectral imaging device can identify and classify the targets by using the spectral difference of the surface components of the object, that is, different narrow-band spectral images have different imaging narrow bands, and the narrow-band spectral images obtained by shooting at different angles are integrated to obtain a wider field of view, so that richer information can be obtained, and then the multi-view stereo vision algorithm is used to realize three-dimensional reconstruction. For example, human skin can better reflect a narrow imaging band, human muscle can better reflect b narrow imaging band, human blood vessel can better reflect c narrow imaging band, and human skeleton can better reflect d narrow imaging band. Therefore, an A narrow-band spectral image reflecting the skin condition of the human body can be obtained according to the a imaging narrow band, a B narrow-band spectral image reflecting the muscle condition of the human body can be obtained according to the B imaging narrow band, a C narrow-band spectral image reflecting the blood vessel condition of the human body can be obtained according to the C imaging narrow band, and a D narrow-band spectral image reflecting the bone condition of the human body can be obtained according to the D imaging narrow band. It can be understood that, when the spectral band is subdivided to a higher degree, the field of view of the spectral imaging device 120 is wider, the information obtained by the spectral generation device 120 is naturally richer, the obtained human body information is more accurate, and thus a more accurate three-dimensional reconstruction model is realized.

In particular embodiments of the present application, the multi-polarization state imaging device 130 may form a plurality of narrow-band polarization images. Among them, the multi-polarization state imaging device 130 may be one or more, and the multi-polarization state imaging device 130 may be an orthogonal multi-polarization state imaging device or an interference multi-polarization state imaging device, or the like. The orthogonal multi-polarization imaging device simultaneously images the operation area by hundreds to thousands of continuous and subdivided spectral wave bands, and the interference multi-polarization imaging device simultaneously images the operation area by thousands to tens of thousands of continuous and subdivided spectral wave bands. Of course, the multi-polarization imaging device may also simultaneously image the surgical area with more continuous and subdivided polarized light bands, for example, tens of thousands to hundreds of thousands of continuous and subdivided polarized light bands, and the like, and is not limited herein.

In a specific embodiment of the present application, the multi-polarization imaging device 130 may obtain polarization characteristic parameters of the surgical area according to the polarization angle, the polarization degree, and the ellipticity angle of the light beam in the multiple narrow-band polarization images, for example, signs of organ surface roughness, texture trend, surface orientation, water content, etc. of the surgical area are provided, and compared with the spectral imaging device 120, the multi-polarization imaging device 130 may provide more accurate information for organ contour and surface orientation identification, highlight the contrast of the organ in the surgical area with the background in the picture, and further improve the accuracy of the subsequent identification with the organ classification model and the body fluid classification model. Therefore, the three-dimensional reconstruction of the operation region by combining the spectral imaging device 120 and the multi-polarization state imaging device 130 can provide more complete and reliable information which cannot be extracted by human eyes for doctors. That is, the spectral imaging device 120 may extract features of the entire surgical area, and use a multi-view stereo vision algorithm to achieve the primary three-dimensional reconstruction of the surgical area, and the multi-polarization imaging device 130 may extract features of contours and surface orientations, such as organ surface roughness, muscle texture trend, and fluid type covered on the organ surface, which cannot be displayed by the spectral imaging device, to obtain an image of the surface roughness or fluid covering condition of the human organ clearly described in the surgical area.

In an embodiment of the present application, the processing system 140 constructs a three-dimensional image of the surgical field from the plurality of narrowband spectral images and the plurality of narrowband polarization images. That is, the processing system integrates the information respectively extracted by the narrow-band spectral image and the narrow-band polarization image, and establishes a detailed and rich operation area three-dimensional information map. For example: the processing system may reconstruct a three-dimensional image a of the skin from the a narrow band spectral images at different angles, reconstructing a three-dimensional image B of muscles according to the B narrow-band spectral images at different angles, reconstructing a three-dimensional image C of blood vessels according to the C narrow-band spectral images at different angles, reconstructing a three-dimensional image D of the skeleton according to the D narrow-band spectral images at different angles, reconstructing a three-dimensional image E of the microvasculature according to the E narrow-band polarized images at different angles, reconstructing a three-dimensional image F of the covering liquid on the surface of each organ according to the F narrow-band polarization images at different angles, fusing the three-dimensional image A, the three-dimensional image B, the three-dimensional image C, the three-dimensional image D, the three-dimensional image E and the three-dimensional image F, a detailed and real three-dimensional image of the surgical area can be obtained which simultaneously reflects the roughness of the skin, muscle, blood vessels, bones, micro-vessels and organs.

In a specific embodiment of the present application, the processing system 140 is further configured to perform an instant analysis on the body fluid component types and the organ types in the surgical field according to a plurality of narrow-band polarization images by combining a classification model, and output an analysis result, where the instant analysis includes an organ type analysis, a protein content analysis, a crystal type analysis, and a crystal content analysis, and the classification model is a body fluid classification model and an organ classification model obtained by training a convolutional neural network in advance by using sample data; the processing system is further configured to determine whether the analysis result is greater than a first threshold or less than a second threshold, wherein the first threshold is greater than the second threshold; the processing system is further configured to send out a prompt message when the analysis result is greater than the first threshold or less than the second threshold. The body fluid classification model and the organ classification model can be trained classification models obtained by training a convolutional neural network by using a large number of body fluid picture samples and organ samples marked with prediction results in advance, the body fluid classification models can obtain body fluid classification results according to narrow-band polarization images containing organ surface water content characteristics obtained by a multi-polarization state imaging system, the organ classification models can obtain organ classification results according to narrow-band polarization images containing organ surface contour characteristics obtained by the multi-polarization state imaging system, the organ classification results are output to reality enhancing or mixed reality equipment worn by a doctor, and more information is provided for the doctor in the operation process so as to further judge the disease condition and perform the operation. Wherein, the prompt message can be one of sound, light or text. For example, as shown in fig. 3, a text can be added to the three-dimensional image by an augmented reality method to remind the doctor of the abnormal hemoglobin content in the blood of the patient during the operation. Moreover, the multi-polarization-state imaging device can also perform judgment of organ classification by combining a neural network model according to polarized light information, and can also add characters on a three-dimensional image by a method of augmented reality or mixed reality to remind a doctor, which is not repeated in the application.

In a specific embodiment of the present application, the processing system 140 may specifically construct a three-dimensional image of the surgical region according to the plurality of narrowband spectral images and the plurality of narrowband polarization images by: the processing system positions and segments the surgical site, the surgical tool and the surgical marker to obtain a positioning segmentation result, determines relative positions among the surgical site, the surgical tool and the surgical marker according to the positioning segmentation result, and reconstructs a three-dimensional image according to the relative positions among the surgical site, the surgical tool and the surgical marker. It will be appreciated that a greater abundance of information is available from the plurality of narrowband spectral images and the plurality of narrowband polarization images, and thus, the surgical site, the surgical tool, and the surgical marker may be more accurately located and segmented.

In a specific embodiment of the present application, the processing system 140 may specifically perform positioning and segmentation on the surgical site, the surgical tool, and the surgical marker to obtain a positioning segmentation result by: after obtaining the plurality of narrowband spectral images and the plurality of narrowband polarization images, the processing system may extract texture features of the plurality of narrowband spectral images and the plurality of narrowband polarization images with a feature extraction algorithm (e.g., a convolutional neural network). Then, the mark made before the operation, the operation tool and the operation related area are accurately positioned or segmented by combining the image recognition algorithm (such as the convolutional neural network) and the motion tracking algorithm, so that the positioning segmentation result is obtained.

In a specific embodiment of the present application, optionally, the processing system is further configured to, after obtaining the three-dimensional reconstruction image of the operation region, perform rendering enhancement on the three-dimensional reconstruction image in a target region, and output the enhanced three-dimensional reconstruction image, where the target region includes a blocked region in the operation region and a specific region in the operation region. The processing system is specifically used for removing a part for shielding the target area and rendering the target area which is not shielded under the condition that the target area is the shielded area in the operation area; or, the processing system is specifically configured to, when the target area is a specific area in the operation area, boost the brightness of the specific area, and render the target area with the boosted brightness.

Preferably, the rendered target region which is not occluded may be obtained by using a trained occlusion supplementary model obtained by training a symmetric convolutional neural network (symmet) using a large number of binocular images and images of the occlusion objects in the binocular images as samples, the occlusion supplementary model learns occlusion features in the large number of binocular images, and a network model which can predict the occlusion problem of the image of the multi-view spectral imaging apparatus by a bilateral symmetric structure. The enhanced result rendered by using the occlusion supplementary model can bring more operation information to the operation of the doctor, for example, when a certain operation region is as shown in fig. 4a, and when the target region is the occluded region in the operation region, the doctor can see the enhanced three-dimensional reconstruction image rendered as shown in fig. 4b by using reality enhancement (AR) glasses, and perform the operation according to the enhanced three-dimensional reconstruction image. It is understood that the doctor can only perform the operation according to experience before the rendering and strengthening of the target area are not performed, and after the rendering and strengthening of the target area are performed, the doctor can use the AR glasses to assist or guide the operation according to the rendered target area image which is not blocked. It should be understood that the above example is only used for illustration, and the rendering of the occlusion region can also be obtained by using other algorithms, for example, using a Left-right cross-checking (LRC) algorithm to infer the occlusion position through the pre-computed parallax result, and the present application is not limited in particular.

In a specific embodiment of the present application, the target area is an area determined by the processing system 140 according to position information sent by a surgical tool, wherein one or more of a gyroscope, an accelerometer, and a distance sensor are disposed in the surgical tool, and the position information is calculated according to position data collected by one or more of the gyroscope, the accelerometer, and the distance sensor. For example: when a doctor scalpel is placed in a certain lesion area, the AR glasses of the doctor automatically display the three-dimensional reconstruction image with the highlighted lesion area. It is to be understood that the above examples are illustrative only and are not to be construed as limiting in any way.

In a specific embodiment of the present application, the target area may also be an area determined by the processing system 140 according to the received user request. For example: a doctor needs to check a certain blocked target area in the operation process, at the moment, the doctor can select the blocked target area needing to be checked by using AR equipment or a doctor assistant through using other terminal equipment frames connected with the processing system, the processing system determines the target area according to the received target area information, and the unblocked target area image which is rendered by the doctor is input into AR glasses of the doctor. It is to be understood that the above examples are illustrative only and are not to be construed as limiting in any way.

In a specific embodiment of the present application, the processing system 140 outputs the enhanced three-dimensional reconstructed image to a head-mounted display device display screen (e.g., AR glasses, MR glasses, etc.) worn by the doctor, a terminal device display screen (e.g., a computer display screen, a flat panel display screen, etc.), or a mechanical surgical system to provide surgical guidance or assistance for the mechanical arm to perform the surgery.

In the system, a broad-spectrum light source generates broad-spectrum light with a wide band, a spectral imaging device collects a plurality of narrow-band spectral images formed by reflecting the broad-spectrum light by an operation area in a multi-angle mode, a primary three-dimensional reconstruction image of the operation area is constructed by utilizing a multi-view stereo vision algorithm, a multi-polarization-state imaging device collects a plurality of narrow-band polarization images formed by reflecting the polarized light by the operation area, a body fluid covering image of a human organ in the operation area is obtained, and a processing system obtains the three-dimensional reconstruction image of the operation area according to the primary three-dimensional reconstruction image and the body fluid covering image of the human organ and outputs the three-dimensional reconstruction image to a display device. Through the system, richer effective information can be obtained according to the plurality of narrow-band spectral images and the plurality of narrow-band polarization images, so that a three-dimensional image with higher accuracy is reconstructed, and the reconstructed three-dimensional image is rendered and enhanced through a virtual enhancement technology according to the requirements of a doctor, so that the doctor is guided or assisted to perform the operation correctly, and the accuracy of the operation is improved.

In the several embodiments provided in the present application, it should be understood that the disclosed system, terminal and method can be implemented in other manners. For example, the above-described apparatus embodiments are merely illustrative, and for example, the division of the units is only one logical division, and other divisions may be realized in practice, for example, a plurality of units or components may be combined or integrated into another system, or some features may be omitted, or not executed. In addition, the shown or discussed mutual coupling or direct coupling or communication connection may be an indirect coupling or communication connection through some interfaces, devices or units, and may also be an electric, mechanical or other form of connection.

The units described as separate parts may or may not be physically separate, and parts displayed as units may or may not be physical units, may be located in one place, or may be distributed on a plurality of network units. Some or all of the units can be selected according to actual needs to achieve the purpose of the solution of the embodiment of the present invention.

In addition, functional units in the embodiments of the present invention may be integrated into one processing unit, or each unit may exist alone physically, or two or more units are integrated into one unit. The integrated unit can be realized in a form of hardware, and can also be realized in a form of a software functional unit.

The integrated unit, if implemented in the form of a software functional unit and sold or used as a stand-alone product, may be stored in a computer readable storage medium. Based on such understanding, the technical solution of the present invention essentially or partially contributes to the prior art, or all or part of the technical solution can be embodied in the form of a software product stored in a storage medium and including instructions for causing a computer device (which may be a personal computer, a server, or a network device) to execute all or part of the steps of the method according to the embodiments of the present invention. And the aforementioned storage medium includes: a U-disk, a removable hard disk, a Read-Only Memory (ROM), a Random Access Memory (RAM), a magnetic disk or an optical disk, and other various media capable of storing program codes. While the invention has been described with reference to specific embodiments, the invention is not limited thereto, and various equivalent modifications and substitutions can be easily made by those skilled in the art within the technical scope of the invention. Therefore, the protection scope of the present invention shall be subject to the protection scope of the claims.

Claims

1. A three-dimensional image establishing method based on multi-view imaging and multi-polarization state imaging is characterized by comprising the following steps:

2. The method of claim 1, further comprising:

the processing system performs real-time analysis on the body fluid component types and the organ types in the operation region by combining a classification model according to a plurality of narrow-band polarization images and outputs an analysis result, wherein the real-time analysis comprises organ type analysis, protein content analysis, crystal type analysis and crystal content analysis, and the classification model is a body fluid classification model and an organ classification model obtained by training a convolutional neural network by using sample data in advance;

3. The method of claim 1, wherein after obtaining the three-dimensional reconstructed image of the surgical region, the method further comprises:

4. The method of claim 3, wherein rendering the three-dimensional reconstructed image to the target region comprises:

5. The method of claim 4,

the target area is an area determined by a processing system according to position information sent by a surgical tool, wherein one or more of a gyroscope, an accelerometer and a distance sensor are arranged in the surgical tool, and the position information is calculated according to position data acquired by one or more of the gyroscope, the accelerometer and the distance sensor; alternatively, the first and second electrodes may be,

6. A three-dimensional image building system based on multi-view imaging and multi-polarization state imaging is characterized by comprising:

7. The system of claim 6,

the processing system is further used for performing instant analysis on the body fluid component types and the organ types in the operation region by combining a classification model according to the plurality of narrow-band polarization images and outputting an analysis result, wherein the instant analysis comprises organ type analysis, protein content analysis, crystal type analysis and crystal content analysis, and the classification model is a body fluid classification model and an organ classification model obtained by training a convolutional neural network by using sample data in advance;

the processing system is further configured to determine whether a protein content or a crystalline content in the analysis result is greater than a first threshold or less than a second threshold, wherein the first threshold is greater than the second threshold;

the processing system is further used for sending out prompt information when the protein content or the crystallization content in the analysis result is larger than the first threshold value or smaller than the second threshold value.

8. The system of claim 6, wherein the processing system is further configured to, after obtaining the three-dimensional reconstruction image of the operation region, perform rendering enhancement on the three-dimensional reconstruction image of the operation region, and output an enhanced three-dimensional reconstruction image, wherein the target region includes an occluded region in the operation region and a specific region in the operation region.

9. The system of claim 8,

the processing system is specifically used for removing a part for shielding the target area and rendering the target area which is not shielded under the condition that the target area is the shielded area in the operation area; alternatively, the first and second electrodes may be,

10. The system of claim 9,