CN117710574A - Image reconstruction method, device, system, electronic equipment and storage medium - Google Patents
Image reconstruction method, device, system, electronic equipment and storage medium Download PDFInfo
- Publication number
- CN117710574A CN117710574A CN202211088631.4A CN202211088631A CN117710574A CN 117710574 A CN117710574 A CN 117710574A CN 202211088631 A CN202211088631 A CN 202211088631A CN 117710574 A CN117710574 A CN 117710574A
- Authority
- CN
- China
- Prior art keywords
- image
- dimensional
- detected object
- reconstruction
- dimensional transmission
- Prior art date
- Legal status (The legal status is an assumption and is not a legal conclusion. Google has not performed a legal analysis and makes no representation as to the accuracy of the status listed.)
- Pending
Links
- 238000000034 method Methods 0.000 title claims abstract description 106
- 230000005540 biological transmission Effects 0.000 claims abstract description 322
- 238000001514 detection method Methods 0.000 claims abstract description 43
- 238000003384 imaging method Methods 0.000 claims abstract description 31
- 230000008569 process Effects 0.000 claims description 55
- 238000012545 processing Methods 0.000 claims description 22
- 238000004891 communication Methods 0.000 claims description 18
- 239000011159 matrix material Substances 0.000 claims description 18
- 238000004590 computer program Methods 0.000 claims description 12
- 238000006243 chemical reaction Methods 0.000 claims description 7
- 238000002591 computed tomography Methods 0.000 description 31
- 238000004364 calculation method Methods 0.000 description 12
- 229910000679 solder Inorganic materials 0.000 description 11
- 238000004422 calculation algorithm Methods 0.000 description 9
- 230000007547 defect Effects 0.000 description 7
- 238000005516 engineering process Methods 0.000 description 7
- 238000010586 diagram Methods 0.000 description 6
- 230000000694 effects Effects 0.000 description 6
- 230000006870 function Effects 0.000 description 6
- 230000005855 radiation Effects 0.000 description 5
- 230000009471 action Effects 0.000 description 3
- 230000007812 deficiency Effects 0.000 description 3
- 238000003466 welding Methods 0.000 description 3
- 238000012217 deletion Methods 0.000 description 2
- 230000037430 deletion Effects 0.000 description 2
- 238000006073 displacement reaction Methods 0.000 description 2
- 238000013507 mapping Methods 0.000 description 2
- 230000003287 optical effect Effects 0.000 description 2
- 230000002093 peripheral effect Effects 0.000 description 2
- 230000002441 reversible effect Effects 0.000 description 2
- 238000004088 simulation Methods 0.000 description 2
- 238000003325 tomography Methods 0.000 description 2
- 238000003491 array Methods 0.000 description 1
- 230000009286 beneficial effect Effects 0.000 description 1
- 238000002247 constant time method Methods 0.000 description 1
- 238000013500 data storage Methods 0.000 description 1
- 238000013135 deep learning Methods 0.000 description 1
- 238000000802 evaporation-induced self-assembly Methods 0.000 description 1
- 230000006872 improvement Effects 0.000 description 1
- 238000007689 inspection Methods 0.000 description 1
- 230000010354 integration Effects 0.000 description 1
- 230000004807 localization Effects 0.000 description 1
- 230000004048 modification Effects 0.000 description 1
- 238000012986 modification Methods 0.000 description 1
- 239000013307 optical fiber Substances 0.000 description 1
- 238000005457 optimization Methods 0.000 description 1
- 230000000452 restraining effect Effects 0.000 description 1
- 239000004065 semiconductor Substances 0.000 description 1
- 239000007787 solid Substances 0.000 description 1
- 230000001360 synchronised effect Effects 0.000 description 1
Abstract
The embodiment of the invention provides an image reconstruction method, an image reconstruction device, an image reconstruction system, electronic equipment and a storage medium, wherein the image reconstruction method comprises the following steps: acquiring a two-dimensional transmission image obtained by carrying out X-ray scanning on a detected object based on X-ray acquisition equipment; determining a first image corresponding to the two-dimensional transmission image, wherein the first image is an image of a detected object obtained through an acquisition device with an imaging principle different from that of an X-ray acquisition device; performing first boundary constraint on the two-dimensional transmission image by using the first image to obtain a preprocessed two-dimensional transmission image, wherein the first boundary constraint is used for removing pixel points which do not belong to the object range of the detected object in the two-dimensional transmission image; and carrying out three-dimensional reconstruction based on the preprocessed two-dimensional transmission image to obtain a three-dimensional detection image of the detected object. The image quality of the three-dimensional detection image of the detected object obtained by carrying out three-dimensional iterative reconstruction on the basis of the preprocessed two-dimensional transmission image is greatly improved.
Description
Technical Field
The present invention relates to the field of image reconstruction technologies, and in particular, to an image reconstruction method, apparatus, system, electronic device, and storage medium.
Background
In the field of industrial inspection, electronic devices are often inspected for defects using X-rays. For example, BGA (Ball Grid Array) solder defects, including solder Ball attach, solder Ball loss, solder Ball displacement, solder Ball voids, dummy solder joints, and pillow effects, can severely affect circuit reliability of a circuit board. Some defects can affect the bonding immediately after the bonding, such as short circuit formed by solder ball bonding; some defects only affect the use process of the circuit board, for example, solder balls with pillow effect are easy to crack at the pillow to form virtual solder joints in use. Two-dimensional X-rays can detect defects such as solder ball continuous welding, loss, displacement, cavity and the like, but the detection capability of the solder ball continuous welding and the pillow effect is weak, and three-dimensional tomography can enable the X-ray detection to cover all common defects of BGA welding.
Plane CT (computed tomography ) is a main means for realizing three-dimensional tomography of an electronic device, and the plane CT reconstructs a three-dimensional image of the electronic device by acquiring multi-angle X-ray data of the electronic device and by adopting an iterative reconstruction algorithm and enhancement methods such as sparse reconstruction, dictionary learning, deep learning and the like, so as to analyze whether defects exist. However, the X-ray data obtained by the conventional planar CT method is incomplete, and the quality of the three-dimensional image obtained by reconstruction is low due to the problems of data missing and inaccuracy.
Disclosure of Invention
The embodiment of the invention aims to provide an image reconstruction method, an image reconstruction device, an image reconstruction system, electronic equipment and a storage medium, so as to improve the image quality of a three-dimensional reconstructed image. The specific technical scheme is as follows:
in a first aspect, an embodiment of the present invention provides an image reconstruction method, including:
acquiring a two-dimensional transmission image obtained by carrying out X-ray scanning on a detected object based on X-ray acquisition equipment;
determining a first image corresponding to the two-dimensional transmission image, wherein the first image is an image of the detected object obtained through an acquisition device with an imaging principle different from that of the X-ray acquisition device;
performing first boundary constraint on the two-dimensional transmission image by using the first image to obtain a preprocessed two-dimensional transmission image, wherein the first boundary constraint is used for removing pixel points which do not belong to the object range of the detected object in the two-dimensional transmission image;
and carrying out three-dimensional reconstruction based on the preprocessed two-dimensional transmission image to obtain a three-dimensional detection image of the detected object.
In a second aspect, an embodiment of the present invention provides an image reconstruction apparatus, including:
The image acquisition module is used for acquiring a two-dimensional transmission image obtained by carrying out X-ray scanning on the detected object based on the X-ray acquisition equipment;
the projection module is used for determining a first image corresponding to the two-dimensional transmission image, wherein the first image is an image of the detected object obtained through an acquisition device with an imaging principle different from that of the X-ray acquisition device;
the first boundary constraint module is used for carrying out first boundary constraint on the two-dimensional transmission image by utilizing the first image to obtain a preprocessed two-dimensional transmission image, wherein the first boundary constraint module is used for removing pixel points which do not belong to the object range of the detected object in the two-dimensional transmission image;
and the reconstruction module is used for carrying out three-dimensional reconstruction based on the preprocessed two-dimensional transmission image to obtain a three-dimensional detection image of the detected object.
In a third aspect, an embodiment of the present invention provides an image reconstruction system, the system including a target acquisition device, an X-ray acquisition device, and an image processing device, wherein:
the X-ray acquisition equipment is used for acquiring a two-dimensional transmission image obtained by carrying out X-ray scanning on the detected object based on the X-ray acquisition equipment;
The image processing device is used for determining a first image corresponding to the two-dimensional transmission image; performing first boundary constraint on the two-dimensional transmission image by using the first image to obtain a preprocessed two-dimensional transmission image; and carrying out three-dimensional reconstruction based on the preprocessed two-dimensional transmission image to obtain a three-dimensional detection image of the detected object, wherein the first image is an image of the detected object obtained by a target acquisition device with an imaging principle different from that of the X-ray acquisition device, and the first boundary constraint is used for removing pixel points which do not belong to the object range of the detected object in the two-dimensional transmission image. .
In a fourth aspect, an embodiment of the present invention provides an electronic device, including a processor, a communication interface, a memory, and a communication bus, where the processor, the communication interface, and the memory complete communication with each other through the communication bus;
a memory for storing a computer program;
a processor for implementing the method steps of any of the above first aspects when executing a program stored on a memory.
In a fifth aspect, embodiments of the present invention provide a computer-readable storage medium having a computer program stored therein, which when executed by a processor, implements the method steps of any of the first aspects described above.
The embodiment of the invention has the beneficial effects that:
in the scheme provided by the embodiment of the invention, the electronic device can perform X-ray scanning on the detected object based on the X-ray acquisition device to obtain the two-dimensional transmission image, so as to determine the first image corresponding to the two-dimensional transmission image, wherein the first image is an image of the detected object obtained by the acquisition device with the imaging principle different from that of the X-ray acquisition device, the information of other dimensions of the detected object, such as external contour image information, can be identified, and further, the first boundary constraint can be performed on the two-dimensional transmission image by using the first image, so that pixel points which do not belong to the object external contour range identified by the corresponding first image in the two-dimensional transmission image are removed, namely pixel points which do not belong to the object range of the detected object are removed, and the second image information of the detected object is added, so that the preprocessed two-dimensional transmission image is more accurate, the problem of inaccurate three-dimensional reconstruction caused by data deletion and inaccuracy of the two-dimensional transmission image can be made up, and the image quality of the three-dimensional detection image of the detected object obtained by performing three-dimensional iterative reconstruction on the preprocessed two-dimensional transmission image is greatly improved. Meanwhile, the calculation of invalid pixel points, namely removed pixel points, is avoided, and the calculation efficiency of a reconstruction algorithm is improved. Of course, it is not necessary for any one product or method of practicing the invention to achieve all of the advantages set forth above at the same time.
Drawings
In order to more clearly illustrate the embodiments of the invention or the technical solutions in the prior art, the drawings used in the embodiments or the description of the prior art will be briefly described below, and it is obvious that the drawings in the following description are only some embodiments of the invention, and other embodiments may be obtained according to these drawings to those skilled in the art.
FIG. 1 is a flowchart of an image reconstruction method according to an embodiment of the present invention;
FIG. 2 (a) is a schematic diagram of X-ray data acquisition of a circuit board based on planar CT of the embodiment shown in FIG. 1;
FIG. 2 (b) is another schematic diagram of X-ray data acquisition of a circuit board based on planar CT of the embodiment shown in FIG. 1;
FIG. 2 (c) is a further schematic diagram of X-ray data acquisition of a circuit board based on planar CT of the embodiment shown in FIG. 1;
FIG. 3 is a specific flowchart of step S104 in the embodiment shown in FIG. 1;
FIG. 4 is a specific flowchart of step S303 in the embodiment shown in FIG. 3;
FIG. 5 is a schematic flow chart of a three-dimensional reconstruction method based on the embodiment shown in FIG. 4;
FIG. 6 is a flow chart of a boundary constraint based on the embodiment of FIG. 1;
FIG. 7 is another specific flowchart of step S303 in the embodiment shown in FIG. 3;
FIG. 8 is a flow chart of an image reconstruction method based on the embodiment shown in FIG. 1;
fig. 9 is a schematic structural diagram of an image reconstruction device according to an embodiment of the present invention;
FIG. 10 is a schematic diagram of an image reconstruction system according to an embodiment of the present invention;
fig. 11 is a schematic structural diagram of an electronic device according to an embodiment of the present invention.
Detailed Description
The following description of the embodiments of the present invention will be made clearly and completely with reference to the accompanying drawings, in which it is apparent that the embodiments described are only some embodiments of the present invention, but not all embodiments. Based on the embodiments of the present invention, all other embodiments obtained by the person skilled in the art based on the present invention are included in the scope of protection of the present invention.
The following describes terms related to the embodiments of the present invention:
plane CT is an X-ray acquisition device commonly used for circuit board detection, and can reconstruct three-dimensional images of the circuit board by acquiring multi-angle X-ray data of the circuit board. Planar CT may use similar components to conventional CT including radiation sources, flat panel detectors, etc. The principle is as follows: the X-ray source emits X-rays, and the emitted X-rays pass through the circuit board at a certain angle and are received by the flat panel detector, so that X-ray data of the angle can be obtained.
The depth camera is a camera that can measure a point-to-lens distance of the object surface, and a pixel value of a pixel point in an image captured by the depth camera can represent depth information of the point.
Multi-View stereovision (MVS), for short, is a technique that reconstructs three-dimensional shapes of an object surface using multiple visible light images of known angles acquired for the same object.
Synchronous positioning and mapping (Simultaneous Localization and Mapping), abbreviated as SLAM, is a technology for constructing a three-dimensional structure of an object by utilizing known multi-angle two-dimensional images of the object or depth information of pixel points in depth images.
The structure registration is a technology for uniformly reconstructing a coordinate system of the obtained three-dimensional image by utilizing scanning parameters of planar CT scanning and scanning parameters of depth camera/visible light camera scanning.
The image registration is a technology for performing feature matching on a three-dimensional image reconstructed by a plane CT and a three-dimensional image reconstructed by a depth camera/visible light image, and unifying coordinate systems of the three-dimensional images obtained by reconstruction.
In order to improve the image quality of the three-dimensional reconstructed image, the embodiment of the invention provides an image reconstruction method, an apparatus, a system, an electronic device, a computer readable storage medium and a computer program product, and the image reconstruction method provided by the embodiment of the invention is first described below.
The image reconstruction method provided by the embodiment of the invention can be applied to any electronic equipment needing to reconstruct a three-dimensional image. For example, an image processing apparatus in an image reconstruction system, a processor of a planar CT apparatus in the field of industrial detection, a controller, a server for reconstructing a three-dimensional image, and the like may be mentioned, and are not particularly limited herein. For clarity of description, hereinafter, referred to as an electronic device.
As shown in fig. 1, an image reconstruction method, the method comprising:
s101, acquiring a two-dimensional transmission image obtained by carrying out X-ray scanning on a detected object based on an X-ray acquisition device.
S102, determining a first image corresponding to the two-dimensional transmission image.
The first image is an image of the detected object obtained through an acquisition device with an imaging principle different from that of the X-ray acquisition device.
And S103, performing first boundary constraint on the two-dimensional transmission image by using the first image to obtain a preprocessed two-dimensional transmission image.
The first boundary constraint is used for removing pixel points which do not belong to the object range of the detected object in the two-dimensional transmission image.
And S104, carrying out three-dimensional reconstruction based on the preprocessed two-dimensional transmission image to obtain a three-dimensional detection image of the detected object.
In the scheme provided by the embodiment of the invention, the electronic device can determine the first image corresponding to the two-dimensional transmission image based on the two-dimensional transmission image obtained by carrying out X-ray scanning on the detected object by the X-ray acquisition device, wherein the first image is the image of the detected object obtained by the acquisition device with the imaging principle different from that of the X-ray acquisition device, the information of other dimensions of the detected object, such as external contour image information, can be identified, and further, the first boundary constraint can be carried out on the two-dimensional transmission image by utilizing the first image, so that pixel points which do not belong to the external contour range of the detected object and are identified by the corresponding first image in the two-dimensional transmission image are removed, namely pixel points which do not belong to the object range of the detected object are removed, and the second image information of the detected object is added, so that the preprocessed two-dimensional transmission image is more accurate, the problem of three-dimensional reconstruction inaccuracy caused by the data deficiency and inaccuracy of the two-dimensional transmission image can be made up, and the image quality of the three-dimensional detection image of the detected object obtained by carrying out three-dimensional reconstruction based on the preprocessed two-dimensional transmission image is greatly improved. Meanwhile, the calculation of invalid pixel points, namely removed pixel points, is avoided, and the calculation efficiency of a reconstruction algorithm is improved.
In the current method for reconstructing a three-dimensional image by planar CT scanning, the scanning mode of planar CT may cause incomplete acquired X-ray data, and the lack of data and inaccuracy may cause lower image quality of the reconstructed three-dimensional image. In order to improve the image quality of the three-dimensional reconstructed image, in the scheme provided by the embodiment of the invention, the first image of the detected object can be acquired, wherein the first image is an image of the detected object obtained through an acquisition device with an imaging principle different from that of an X-ray acquisition device, and can identify information of other dimensions of the detected object, such as external contour image information, and the first image is used as supplementary data of X-ray data and constrains the X-ray data. The acquisition device different from the imaging principle of the X-ray acquisition device may be an optical acquisition device capable of acquiring an external contour image of the detected object, for example, an RGB camera, a depth camera, or the like.
In the above step S101, the electronic device may acquire a two-dimensional transmission image obtained by performing X-ray scanning on the object to be detected based on the X-ray acquisition device. The two-dimensional transmission image is a two-dimensional transmission image of the detected object obtained by scanning from one or more angles based on the X-ray acquisition device. The X-ray acquisition device may comprise a source and a detector, the source emitting X-rays, the emitted X-rays passing through the object under examination at an angle, being received by the detector, whereby the electronic device may obtain a two-dimensional transmission image of the angle. Further, by changing the angle at which the X-rays pass through the object to be inspected, the electronic device can acquire two-dimensional transmission images of the object to be inspected acquired from a plurality of angles.
For example, the X-ray acquisition device may be a planar CT that includes a radiation source and a flat panel detector, the object to be detected may be a circuit board, and the planar CT may acquire a two-dimensional transmission image of the circuit board using a plurality of scanning modes. One way of acquiring X-ray data of a circuit board by planar CT may be as shown in fig. 2 (a), wherein the radiation source 201 and the flat panel detector 203 are kept fixed, and the circuit board 202 rotates in a preset direction, so that the flat panel detector 203 can acquire X-ray data of a plurality of angles of the circuit board 202.
For another example, a manner of acquiring X-ray data of the circuit board by planar CT may be as shown in fig. 2 (b), where the radiation source 204 is kept fixed, and the circuit board 205 and the flat panel detector 206 rotate in the same direction according to a preset direction, a rotation radius, and a rotation speed, so that the flat panel detector 206 may acquire X-ray data of multiple angles of the circuit board 205.
For another example, a manner of acquiring X-ray data of the circuit board by planar CT may be as shown in fig. 2 (c), where the circuit board 208 is kept fixed, and the radiation source 207 and the flat panel detector 209 rotate in the same direction according to a preset direction, a rotation radius, and a rotation speed, so that the flat panel detector 209 may also acquire X-ray data of multiple angles of the circuit board 208. Based on X-ray data of a plurality of angles acquired by the flat panel detector, the planar CT can obtain two-dimensional transmission images of the circuit board of the plurality of angles.
Further, in the step S102, the electronic device may determine the first image corresponding to the two-dimensional transmission image, and the first image is an image of the detected object obtained by an acquisition device having a different imaging principle from that of the X-ray acquisition device, for example, a visible light image. Which may identify characteristics of the detected object from other dimensions, such as external profile information, etc.
In order to improve the image quality of the three-dimensional reconstructed image, the electronic device may perform a first boundary constraint on the two-dimensional transmission image by using the first image, so as to obtain a corresponding preprocessed two-dimensional transmission image. The first boundary constraint may remove pixels in the two-dimensional transmission image that do not belong to the object range of the detected object.
As an embodiment, the first image is an image simulating an imaging effect of the projection of the outer contour of the detected object onto the detector of the X-ray acquisition device, and the object range of the projection of the outer contour of the detected object can be identified. Then, the first image is used for carrying out first boundary constraint on the two-dimensional transmission image, so that the pixel points which do not belong to the object range identified by the corresponding first image in the two-dimensional transmission image can be removed, namely, the pixel points which do not belong to the object range of the detected object in the two-dimensional transmission image can be removed.
In one embodiment, the pixel values in the first image may be set to: the pixel value in the object range projected by the outer contour of the detected object is a first preset value, and the pixel value outside the object range is a second preset value. For example, the first preset value may be set to 1 and the second preset value to 0, which is not particularly limited herein. Therefore, in the process of the first boundary constraint, the region corresponding to the first preset value in the two-dimensional transmission image can not be changed, and the pixel value of the region corresponding to the second preset value in the two-dimensional transmission image is changed into the background pixel value of the image, so that the pixels in the object range are distinguished from the pixels out of the object range.
The electronic equipment adds second image information of the detected object in the process of restraining the first boundary, so that the influence caused by data deletion of the two-dimensional transmission image is reduced. The preprocessed two-dimensional transmission image can accurately identify the object range of the detected object, and compared with the two-dimensional transmission image which is not subjected to the first boundary constraint, the three-dimensional reconstruction is performed based on the preprocessed two-dimensional transmission image, so that the image quality of the obtained three-dimensional detection image of the detected object is greatly improved.
Further, in the step S104, the electronic device may perform three-dimensional reconstruction based on the preprocessed two-dimensional transmission image, to obtain a three-dimensional detection image of the detected object. The electronic equipment can reconstruct the three-dimensional detection image of the detected object by taking the preprocessed two-dimensional transmission image as original data, and in the reconstruction process, the electronic equipment can continuously improve the accuracy of the reconstructed three-dimensional detection image of the detected object by using an iterative algorithm until the reconstructed three-dimensional detection image meets the preset accuracy requirement.
As shown in fig. 3, the step of performing three-dimensional iterative reconstruction based on the preprocessed two-dimensional transmission image to obtain the three-dimensional detection image of the detected object may include:
s301, acquiring an initialized three-dimensional reconstruction image as a target three-dimensional reconstruction image.
The electronic device can acquire an initialized three-dimensional reconstruction image as a target three-dimensional reconstruction image, and further can restrict and update the target three-dimensional reconstruction image in the process of reconstructing the three-dimensional detection image of the detected object so as to gradually improve the accuracy of the reconstructed three-dimensional detection image.
In one embodiment, the initialized three-dimensional reconstructed image may be a three-dimensional reconstructed image obtained by random initialization; in another embodiment, the electronic device may reconstruct an initialized three-dimensional reconstructed image based on the plurality of two-dimensional transmission images based on the two-dimensional transmission images of the detected object acquired from the plurality of angles by the X-ray acquisition device; in another embodiment, the electronic device may reconstruct an initialized three-dimensional reconstructed image based on the preprocessed two-dimensional transmission image obtained by the first boundary constraint using the first image, which is not particularly limited herein.
S302, according to scanning parameters adopted in an X-ray scanning process, a two-dimensional transmission projection image of the target three-dimensional reconstruction image projected onto a detector of the X-ray acquisition device is determined.
Because the target three-dimensional reconstruction image is an initialized three-dimensional reconstruction image acquired by the electronic equipment, the image quality and the accuracy are relatively low; the preprocessed two-dimensional transmission image is obtained through the first boundary constraint, and pixel points which do not belong to the object range of the detected object are removed, so that the image quality and accuracy are high. Thus, in the process of three-dimensional iterative reconstruction, the target three-dimensional reconstructed image can be continuously optimized by using the preprocessed two-dimensional transmission image.
The preprocessed two-dimensional transmission image is a two-dimensional image, and the target three-dimensional reconstruction image is a three-dimensional image, so that a two-dimensional transmission projection image corresponding to the target three-dimensional reconstruction image can be acquired by using a method of simulating projection, and the difference between the two-dimensional transmission projection image and the preprocessed two-dimensional transmission image can be determined.
The electronic device can determine a two-dimensional transmission projection image of the three-dimensional reconstruction image of the target projected onto the detector of the X-ray acquisition device according to scanning parameters adopted in the X-ray scanning process, such as an acquisition angle, an imaging distance, imaging parameters of the X-ray device and the like. The two-dimensional transmission projection image can be used for identifying the two-dimensional transmission image obtained by adopting the X-ray acquisition device to perform X-ray scanning on the detected object represented by the target three-dimensional reconstruction image.
S303, updating the target three-dimensional reconstruction image based on the difference between the two-dimensional transmission projection image and the preprocessed two-dimensional transmission image to obtain an updated three-dimensional reconstruction image;
since the two-dimensional transmission projection image is a two-dimensional transmission projection image projected onto the detector of the X-ray acquisition device by using the target three-dimensional reconstruction image obtained by using the method of analog projection, if the target three-dimensional reconstruction image at the moment is very accurate, the difference between the corresponding two-dimensional transmission projection image and the preprocessed two-dimensional transmission image is very small, so that the difference between the two-dimensional transmission projection image and the preprocessed two-dimensional transmission image can represent the accuracy of the target three-dimensional reconstruction image. The electronic device can update the target three-dimensional reconstruction image according to the difference between the two-dimensional transmission projection image and the preprocessed two-dimensional transmission image, and an updated three-dimensional reconstruction image is obtained.
S304, performing second boundary constraint on the updated three-dimensional reconstruction image by using the second image to obtain a constrained three-dimensional reconstruction image.
The second image is an image of the detected object acquired by an acquisition device with an imaging principle different from that of the X-ray acquisition device, and the second image comprises three-dimensional information of the detected object. For example, the acquisition device different from the imaging principle of the X-ray acquisition device may be an acquisition device capable of acquiring three-dimensional information including the object to be detected, such as a depth camera.
And using the second image, the electronic equipment can carry out second boundary constraint on the updated three-dimensional reconstruction image to obtain a constrained three-dimensional reconstruction image. The second boundary constraint is used for removing pixel points which do not belong to the object range of the detected object in the updated three-dimensional reconstruction image. Because the second image comprises the three-dimensional information of the detected object, the second boundary constraint is carried out on the updated three-dimensional reconstruction image by using the second image, so that the three-dimensional information which is different from the range of the detected object represented by the second image in the updated three-dimensional reconstruction image can be removed, namely, the pixel points which do not belong to the object range of the detected object in the updated three-dimensional reconstruction image are removed, and the updated three-dimensional reconstruction image is more accurate. Not only accelerating the convergence rate of the iterative algorithm of the three-dimensional reconstruction, but also removing partial pixel points and improving the efficiency of the three-dimensional reconstruction.
In this embodiment, the electronic device may perform the second boundary constraint on the updated three-dimensional reconstruction image by using the second image in the three-dimensional reconstruction process, to obtain the constrained three-dimensional reconstruction image, and may remove the pixel points in the updated three-dimensional reconstruction image that do not belong to the object range of the detected object, so that the updated three-dimensional reconstruction image is more accurate. Not only accelerating the convergence rate of the iterative algorithm of the three-dimensional reconstruction, but also removing partial pixel points and improving the efficiency of the three-dimensional reconstruction.
As shown in fig. 4, the step of updating the target three-dimensional reconstructed image based on the difference between the two-dimensional transmission projection image and the preprocessed two-dimensional transmission image may include:
s401, determining a three-dimensional reconstruction residual error based on the difference between the two-dimensional transmission projection image and the preprocessed two-dimensional transmission image; if the three-dimensional reconstruction residual is greater than a preset threshold, executing step S402; if the three-dimensional reconstruction residual is not greater than the preset threshold, step S403 is performed.
The two-dimensional transmission projection image and the preprocessed two-dimensional transmission image are determined, and the electronic device can determine a three-dimensional reconstruction residual error based on the difference between the two-dimensional transmission projection image and the preprocessed two-dimensional transmission image, wherein the three-dimensional reconstruction residual error is used for identifying the accuracy of the target three-dimensional reconstruction image. In one embodiment, since the two-dimensional transmission projection image and the preprocessed two-dimensional transmission image are two-dimensional images, pixel values corresponding to the same pixel positions in the two images can be subtracted to obtain a residual image, and the residual image is the three-dimensional reconstruction residual.
In another embodiment, the preprocessed two-dimensional transmission image may be a plurality of images acquired from a plurality of angles, the corresponding two-dimensional transmission projection image is also a plurality of images, the electronic device may calculate a difference value between the two-dimensional transmission projection image of each acquired angle and the corresponding preprocessed two-dimensional transmission image, and further may calculate a sum or an average value of the difference values corresponding to the plurality of angles, as a three-dimensional reconstruction residual; or according to the preset weight corresponding to each angle, calculating the weighted sum value of the difference values corresponding to the angles to be used as a three-dimensional reconstruction residual error.
And further, the three-dimensional reconstruction residual error can be compared with a preset residual error threshold, and the preset threshold can be set based on the accuracy and the image quality requirement of the three-dimensional detection image, which is not particularly limited herein. If the three-dimensional reconstruction residual error is greater than the preset threshold value, the difference between the two-dimensional transmission projection image and the preprocessed two-dimensional transmission image is larger, at this time, the accuracy of the target three-dimensional reconstruction image is lower, the picture quality is not high enough, further optimization is needed, and the step S402 can be continuously executed.
If the three-dimensional reconstruction residual error is not greater than the preset threshold value, it indicates that the difference between the two-dimensional transmission projection image and the preprocessed two-dimensional transmission image is smaller, at this time, the accuracy of the target three-dimensional reconstruction image is higher, the picture quality is high, and the target three-dimensional reconstruction image can be used as a three-dimensional detection image of the detected object, that is, step S403 is executed.
And S402, carrying out back propagation based on the three-dimensional reconstruction residual error, and updating the target three-dimensional reconstruction image to obtain an updated three-dimensional reconstruction image.
If the three-dimensional reconstruction residual error is larger than a preset threshold value, the target three-dimensional reconstruction image needs to be further optimized, the electronic equipment can perform back propagation based on the three-dimensional reconstruction residual error, update the target three-dimensional reconstruction image, and compared with the initialized three-dimensional reconstruction image, the updated target three-dimensional reconstruction image is more accurate. The method for back propagation based on the three-dimensional reconstruction residual error can be to calculate the deviation of a preset residual error function according to the three-dimensional reconstruction residual error, further determine the direction of the maximum decrease of the preset residual error function value based on the deviation calculation result, and adjust the target three-dimensional reconstruction image according to the direction so as to achieve the required accuracy at the highest speed.
In this case, after the step of performing the second boundary constraint on the updated three-dimensional reconstructed image by using the second image to obtain a constrained three-dimensional reconstructed image, the method may further include:
and taking the constrained three-dimensional reconstruction image as a target three-dimensional reconstruction image, and returning to the step of determining a two-dimensional transmission projection image of the target three-dimensional reconstruction image projected onto a detector of the X-ray acquisition equipment according to the scanning parameters adopted in the X-ray scanning process until the three-dimensional reconstruction residual error is not larger than the preset threshold value, so as to obtain a three-dimensional detection image of the detected object.
Furthermore, the electronic device may take the constrained target three-dimensional reconstructed image as a target three-dimensional reconstructed image, and return to the above step of determining a two-dimensional transmission projection image of the target three-dimensional reconstructed image projected onto the detector of the X-ray acquisition device according to the scanning parameters adopted in the X-ray scanning process. And the electronic equipment can optimize the target three-dimensional reconstruction image again through the preprocessed two-dimensional transmission image with high accuracy, and the steps are circularly executed, so that the accuracy and the image quality of the target three-dimensional reconstruction image are continuously improved until the three-dimensional reconstruction residual error is not more than the preset threshold value, and the three-dimensional detection image of the detected object can be obtained.
S403, obtaining a three-dimensional detection image of the detected object.
If the three-dimensional reconstruction residual error is not greater than the preset threshold value, the accuracy and the image quality of the target three-dimensional reconstruction image at the moment are higher, and the requirements of the image quality and the accuracy are met, so that the target three-dimensional reconstruction image can be used as a three-dimensional detection image of a detected object.
In one embodiment, the electronic device may perform three-dimensional iterative reconstruction on the target three-dimensional reconstruction image of the detected object through the flowchart shown in fig. 5, where the three-dimensional iterative reconstruction may include the following steps A1-a10:
A1: and initializing three-dimensional reconstruction.
The electronic device may begin initializing the three-dimensional reconstruction, such as randomly initializing the three-dimensional reconstruction; or based on the two-dimensional transmission images of the detected object acquired from a plurality of angles by the X-ray acquisition equipment, carrying out initialization three-dimensional reconstruction; or performing an initialization three-dimensional reconstruction based on the preprocessed two-dimensional transmission image obtained by performing the first boundary constraint by using the first image.
A2: and acquiring a three-dimensional reconstruction image of the target.
Based on the initialized three-dimensional reconstruction, the electronic device may acquire the initialized three-dimensional reconstruction image as a target three-dimensional reconstruction image.
A3: a two-dimensional transmission projection image is acquired.
The electronic device can determine a two-dimensional transmission projection image obtained by projecting the three-dimensional reconstruction image of the target onto the detector of the X-ray acquisition device according to the scanning parameters corresponding to the two-dimensional transmission image.
A4: and acquiring a preprocessed two-dimensional transmission image.
A5: and determining a three-dimensional reconstruction residual error.
The electronic device may subtract the preprocessed two-dimensional transmission image from the pixel values with the same pixel positions in the two-dimensional transmission projection image to obtain a residual image, which is used as a three-dimensional reconstruction residual.
A6: and determining whether the three-dimensional reconstruction residual is greater than a preset threshold.
The electronic equipment can judge whether the three-dimensional reconstruction residual is larger than a preset threshold value, and if not, the electronic equipment can execute the step A7; if it is greater than the preset threshold, the electronic device may perform step A8.
A7: and outputting a reconstruction result.
The three-dimensional reconstruction residual error is not larger than a preset threshold value, and the electronic equipment can output the target three-dimensional reconstruction image as a three-dimensional detection image of the detected object.
A8: the three-dimensional reconstruction residual is back-propagated.
The three-dimensional reconstruction residual is greater than a preset threshold, and the electronic device can perform back propagation based on the three-dimensional reconstruction residual.
A9: and updating the three-dimensional reconstruction image of the target.
The electronic device can update the target three-dimensional reconstruction image based on the three-dimensional reconstruction residual error to obtain an updated three-dimensional reconstruction image.
A10: a second image is acquired.
The electronic equipment can acquire a second image of the detected object corresponding to the target three-dimensional reconstruction image, and carry out second boundary constraint on the target three-dimensional reconstruction image, so that the constrained three-dimensional reconstruction image is used as the target three-dimensional reconstruction image. And returning to the step A2, continuing to update and optimize the target three-dimensional reconstruction image until the three-dimensional reconstruction residual error corresponding to the target three-dimensional reconstruction image is not greater than a preset threshold value, and outputting the target three-dimensional reconstruction image at the moment as a three-dimensional detection image of the detected object.
In this embodiment, the electronic device may perform back propagation based on the three-dimensional reconstruction residual error, and continuously update the target three-dimensional reconstruction image until the three-dimensional reconstruction residual error is not greater than a preset threshold value, so as to obtain a three-dimensional detection image of the detected object. The target three-dimensional reconstruction image is continuously optimized by utilizing the preprocessed two-dimensional transmission image with higher accuracy and image quality, so that the image quality of the target three-dimensional reconstruction image is continuously improved in the three-dimensional reconstruction process, and the image quality of the three-dimensional detection image of the detected object obtained by reconstruction is greatly improved.
As an implementation manner of the embodiment of the present invention, as shown in fig. 6, the above-mentioned boundary constraint manner may include:
s601, setting a pixel value of a region corresponding to the detected object in a target image as a first target value, and setting pixel values of other regions except the detected object in the target image as a second target value to obtain a constraint image template;
when the boundary constraint is a first boundary constraint, the target image is the first image, and when the boundary constraint is a second boundary constraint, the target image is the second image.
When the boundary constraint is a first boundary constraint, the target image is a first image, which may be an image of the detected object acquired by the visible light camera, and the first image may include two-dimensional information of the detected object, so the first boundary constraint may be referred to as a two-dimensional boundary constraint for performing boundary constraint on the two-dimensional transmission image.
The electronic device may set the pixel value of the region corresponding to the detected object in the first image as a first target value, and set the pixel values of the remaining regions except for the detected object as a second target value, so that the obtained constraint image template may accurately distinguish the region corresponding to the detected object from the background region from the two-dimensional angle.
Similarly, when the boundary constraint is the second boundary constraint, the target image is the second image, which may be an image of the detected object acquired by the depth camera, and the second image may include three-dimensional information of the detected object, so that the second boundary constraint may be referred to as a three-dimensional boundary constraint, and is used for performing the boundary constraint on the updated three-dimensional reconstructed image.
The electronic device may set the pixel value of the region corresponding to the detected object in the second image as the first target value, and set the pixel value of the remaining regions except for the detected object as the second target value, so that the obtained constraint image template may accurately distinguish the region corresponding to the detected object from the background region from the three-dimensional angle.
The first target value and the second target value may be set to different pixel values, and may be different between a region corresponding to the detected object and a background region, which is not particularly limited herein.
S602, based on the constraint image template, pixel values of pixel points corresponding to detected objects in an image to be constrained are kept unchanged, and pixel values of pixel points corresponding to other areas except the detected objects in the image to be constrained are determined to be a third target value, so that a constrained image is obtained;
when the boundary constraint is a first boundary constraint, the image to be constrained is the two-dimensional transmission image, the constrained image is the preprocessed two-dimensional transmission image, or when the boundary constraint is a second boundary constraint, the image to be constrained is the updated three-dimensional reconstruction image, the constrained image is the constrained three-dimensional reconstruction image, and the constrained image template and the image to be constrained show the same detected object.
The electronic device determines a constraint image template, based on the constraint image template, the pixel values of the pixel points corresponding to the detected object in the image to be constrained can be kept unchanged, and the pixel values of the pixel points corresponding to the rest areas except the detected object in the image to be constrained are set as a third target value, so that the constrained image is obtained.
When the boundary constraint is the first boundary constraint, the image to be constrained is a two-dimensional transmission image, and the electronic device may set the pixel values of the pixel points corresponding to the pixel points in the first image as the first target value in the two-dimensional transmission image to be unchanged, and set the pixel values of the remaining pixel points as the third target value. Since the pixel point in the first image with the pixel value of the first target value is the pixel point corresponding to the detected object, the pixel point in the constrained image, i.e. the preprocessed two-dimensional transmission image, which does not belong to the object range of the detected object is set as the third target value, and can be accurately distinguished from the pixel point in the object range of the detected object.
Similarly, when the boundary constraint is the second boundary constraint, the image to be constrained is an updated three-dimensional reconstructed image, and the electronic device may set the pixel values of the pixel points corresponding to the pixel points in the second image, where the pixel values in the second image are the first target values, to be the third target values. Because the pixel point with the pixel value of the second image being the first target value is the pixel point corresponding to the detected object, the pixel point in the constrained image, i.e. the object range of the constrained three-dimensional reconstruction image, which does not belong to the detected object is set as the third target value, and can be accurately distinguished from the pixel point in the object range of the detected object.
The third target value may be a pixel value different from both the first target value and the second target value, and may be the same as the second target value, as long as it is different from the first target value. For example, the first target value may be 1, the second target value and the third target value may be 0, etc., and are not particularly limited herein.
In this embodiment, the electronic device may set the pixel value of the region corresponding to the detected object in the target image to a first target value, and set the pixel value of the remaining region excluding the detected object in the target image to a second target value, to obtain the constraint image template. And processing pixel values of the image to be constrained based on the constraint image template to obtain a constrained image. Therefore, in the process of the first boundary constraint or the second boundary constraint, the boundary constraint is accurately carried out, the pixel points which do not belong to the object range of the detected object in the image to be constrained are accurately removed, the number of the pixel points which participate in the subsequent three-dimensional reconstruction calculation is reduced, the three-dimensional reconstruction time is greatly reduced, and the three-dimensional reconstruction efficiency is improved.
As an implementation manner of the embodiment of the present invention, based on the constraint image template, the step of determining, as a third target value, the pixel values of the pixel points corresponding to the remaining areas except for the detected object in the image to be constrained, where the pixel values of the pixel points corresponding to the detected object in the image to be constrained are kept unchanged may include:
When the boundary constraint is the first boundary constraint, calculating the pixel value of each pixel point in the preprocessed two-dimensional transmission image according to the following formula:
A M (i，j)＝A(i，j)×M(i，j)
wherein A is M And (i, j) is the pixel value of the pixel point with the coordinate of (i, j) in the preprocessed two-dimensional transmission image, A (i, j) is the pixel value of the pixel point with the coordinate of (i, j) in the two-dimensional transmission image, and M (i, j) is the pixel value of the pixel point with the coordinate of (i, j) in the constraint image template corresponding to the two-dimensional transmission image.
When the boundary constraint is the first boundary constraint, as an embodiment, the first target value may be set to 1 and the second target value may be set to 0. The constraint image template is a binary image, only one connected domain exists in the binary image, the pixel value of the pixel point in the connected domain is 1, and the pixel value of the other parts is 0.
When the electronic device calculates the pixel value of each pixel point in the preprocessed two-dimensional transmission image according to the above formula, the pixel value of the pixel point in the preprocessed two-dimensional transmission image corresponding to the pixel point with the pixel value of the first target value in the constraint image template may be kept unchanged, and the pixel values of the other pixel points in the preprocessed two-dimensional transmission image, that is, the pixel points outside the object range of the detected object, may be changed, for example, become 0. Therefore, pixel points which do not belong to the object range of the detected object in the two-dimensional transmission image are removed, and the area corresponding to the detected object in the preprocessed two-dimensional transmission image can be more accurately distinguished from other areas except the detected object.
When the boundary constraint is the second boundary constraint, calculating pixel values of all pixel points in the constrained three-dimensional reconstruction image according to the following formula:
A M (i，j，k)＝A(i，j，k)×M(i，j，k)
wherein A is M (i, j, k) is the pixel value of the pixel point with the coordinate of (i, j, k) in the constrained three-dimensional reconstruction image, A (i, j, k) is the pixel value of the pixel point with the coordinate of (i, j, k) in the updated three-dimensional reconstruction image, and M (i, j, k) is the pixel value of the pixel point with the coordinate of (i, j, k) in the constrained image template corresponding to the second image.
When the boundary constraint is the second boundary constraint, as an embodiment, the first target value may be set to 1 and the second target value may be set to 0. When the electronic device calculates the pixel value of each pixel point in the constrained three-dimensional reconstruction image according to the formula, the pixel value of the pixel point in the constrained three-dimensional reconstruction image corresponding to the pixel point with the pixel value of the first target value in the constrained image template can be kept unchanged, and the pixel values of the other pixel points in the constrained three-dimensional reconstruction image, namely, the pixel points outside the object range of the detected object, are changed, for example, are changed to 0. Therefore, pixel points which do not belong to the object range of the detected object in the updated three-dimensional reconstruction image are removed, and the region corresponding to the detected object in the constrained three-dimensional reconstruction image and other regions except the detected object can be more accurately distinguished.
In this embodiment, when the electronic device calculates the pixel value of each pixel point in the preprocessed two-dimensional transmission image according to the above formula, the pixel point outside the object range of the detected object can be accurately removed; when the pixel value of each pixel point in the constrained three-dimensional reconstruction image is calculated through the formula, the pixel points outside the object range of the detected object can be accurately removed. Therefore, partial pixel points are removed more accurately and rapidly, the accuracy of three-dimensional reconstruction is improved, the time of three-dimensional iterative reconstruction is reduced, and the efficiency of three-dimensional iterative reconstruction is improved.
As an implementation manner of the embodiment of the present invention, the step of determining the two-dimensional transmission projection image of the target three-dimensional reconstruction image projected onto the detector of the X-ray acquisition device according to the scanning parameters adopted in the X-ray scanning process may include:
arranging pixel values of the target three-dimensional reconstructed image into one-dimensional image data; determining a system matrix based on the arrangement mode corresponding to the one-dimensional image data and scanning parameters adopted in the X-ray scanning process; and calculating a two-dimensional transmission projection image corresponding to the target three-dimensional reconstruction image according to the system matrix and the one-dimensional image data.
Because the target three-dimensional reconstruction image is a three-dimensional array, the target three-dimensional reconstruction image is simulated and projected onto a detector of the X-ray acquisition equipment for convenience, and a two-dimensional transmission projection image is obtained. In this embodiment, the electronic device may first arrange the pixel values of the target three-dimensional reconstructed image into one-dimensional image data according to a preset arrangement manner, for example, when the target three-dimensional reconstructed image is a three-dimensional array I, the corresponding one-dimensional image data may be a one-dimensional array I * One-dimensional array I * The three-dimensional array I is obtained by arranging the three-dimensional array I according to a preset arrangement mode.
The preset arrangement manner may be set according to actual requirements, for example, the arrangement may be performed according to a row order from the first pixel point in the first row, or the arrangement may be performed according to a column order from the first pixel point in the first column, which is not limited herein.
Next, the electronic device may determine a corresponding system matrix based on the arrangement corresponding to the one-dimensional image data and the scanning parameters employed in the X-ray scanning process. Wherein the system matrix is used for identifying a coordinate conversion relation between the one-dimensional image data and the two-dimensional transmission projection image.
Because the one-dimensional image data is obtained according to the arrangement mode, the arrangement mode identifies the coordinate conversion relation between the one-dimensional image data and the three-dimensional image data, and the scanning parameters adopted in the X-ray scanning process can identify the coordinate conversion relation between the three-dimensional image data and the two-dimensional transmission projection image, the electronic equipment can calculate and obtain a system matrix capable of identifying the coordinate conversion relation between the one-dimensional image data and the two-dimensional transmission projection image according to the arrangement mode and the scanning parameters corresponding to the two-dimensional transmission image. When the arrangement mode is changed or the scanning parameters corresponding to the two-dimensional transmission images are different, the corresponding system matrix is also changed.
Because the system matrix can identify the coordinate conversion relation between the one-dimensional image data and the two-dimensional transmission projection image, the electronic equipment can calculate and obtain the two-dimensional transmission projection image corresponding to the target three-dimensional reconstruction image according to the system matrix and the one-dimensional image data. In one embodiment, the target three-dimensional reconstructed image I is arranged according to a preset arrangement mode to obtain one-dimensional image data I * The electronic device may determine that the corresponding system matrix is Φ. The electronic device may calculate the two-dimensional transmission projection image p corresponding to the two-dimensional transmission image according to the following formula:
ΦI * ＝p
Wherein I is * And phi is a system matrix corresponding to the two-dimensional transmission image.
In this case, as shown in fig. 7, the step of updating the target three-dimensional reconstructed image based on the difference between the two-dimensional transmission projection image and the preprocessed two-dimensional transmission image to obtain an updated three-dimensional reconstructed image may include:
and S701, updating the one-dimensional image data based on the difference between the two-dimensional transmission projection image and the preprocessed two-dimensional transmission image to obtain updated one-dimensional image data.
For the above case where the pixel values of the target three-dimensional reconstructed image are arranged as one-dimensional image data, the electronic device may update the one-dimensional image data based on the difference between the two-dimensional transmission projection image and the preprocessed two-dimensional transmission image, thereby obtaining updated one-dimensional image data. Specifically, the electronic device may calculate a three-dimensional reconstruction residual according to a difference between the two-dimensional transmission projection image and the preprocessed two-dimensional transmission image, for example, the two-dimensional transmission projection image may be p, and the preprocessed two-dimensional transmission image may be p 1 Furthermore, the difference between the two-dimensional transmission projection image and the preprocessed two-dimensional transmission image can be calculated, namely, p-p1=Δp is calculated, and Δp is the three-dimensional reconstruction residual error.
Therefore, if the three-dimensional reconstruction residual error is larger than a preset threshold value, the three-dimensional reconstruction residual error can be back-propagated, and the one-dimensional image data is updated to obtain updated one-dimensional image data. For example, when the three-dimensional reconstruction residual error is Δp and the one-dimensional image data is I, the updated one-dimensional image data I 'can be obtained by calculating the following formula' * ：
I′ * ＝I * -αψ(Δp)
Where α is an iteration step length, which may be set based on an actual requirement of three-dimensional iterative reconstruction, and ψ (·) represents a function for calculating a three-dimensional reconstruction residual Δp, which may be obtained by using a plurality of calculation methods, for example, a function ψ (Δp) of the three-dimensional reconstruction residual Δp may be calculated by the following formula:
ψ(Δp)＝2Φ T Δp
wherein phi is T Is the transpose of the system matrix Φ.
Therefore, the electronic equipment can rapidly determine the direction of three-dimensional iteration through a reverse propagation mode such as a bias derivative based on the function of the three-dimensional reconstruction residual error delta p, and update the one-dimensional image data by adopting a gradient descent mode to obtain updated one-dimensional image data.
S702, arranging the updated one-dimensional image data according to the inverse operation of the arrangement mode to obtain an updated three-dimensional reconstruction image.
The electronic device updates the one-dimensional image data, and after the updated one-dimensional image data is obtained, the updated one-dimensional image data can be arranged according to the inverse operation of the arrangement mode, so that an updated three-dimensional reconstruction image can be obtained. For example, the updated one-dimensional image data is a one-dimensional array I' * The electronic device can operate the one-dimensional array I 'according to the reverse of the arrangement mode' * Is rearranged into a three-dimensional array I 'so that the three-dimensional array I' can be used as an updated three-dimensional reconstructed image.
Since the arrangement of the updated one-dimensional image data is performed according to the inverse operation of the arrangement mode adopted by arranging the pixel values of the target three-dimensional reconstructed image into the one-dimensional image data, the three-dimensional image arranged according to the inverse operation is consistent with the data structure of the target three-dimensional reconstructed image, and the updated one-dimensional image data can be accurately restored to the three-dimensional image, so that the updated three-dimensional reconstructed image can be obtained.
In this embodiment, the electronic device may convert the target three-dimensional reconstructed image into one-dimensional image data, and after obtaining updated one-dimensional image data, arrange the updated one-dimensional image data based on the inverse operation of the arrangement mode, to obtain an updated three-dimensional reconstructed image. Thereby, the calculation can be conveniently performed, and the accuracy of the updated three-dimensional reconstruction image can be improved.
As an implementation manner of the embodiment of the present invention, the step of determining the first image corresponding to the two-dimensional transmission image may include:
acquiring a second image of the detected object; and based on scanning parameters adopted in the X-ray scanning process, projecting the second image onto a detector of the X-ray acquisition equipment to obtain a first image corresponding to the two-dimensional transmission image.
The second image may be an image of the detected object obtained based on the visible light collecting device. In one embodiment, the visible light collection device may be a visible light camera capable of collecting two-dimensional images, and the visible light camera may scan the detected object from a plurality of angles, thereby obtaining visible light images of the detected object collected from a plurality of angles. The electronic device may construct a second image of the detected object through MVS technology or SLAM technology based on the visible light images acquired at the plurality of angles.
In another embodiment, the visible light collecting device may be a depth camera, and since the depth camera may determine a pixel value of a pixel in the depth image based on a distance between a point on the surface of the object and the lens, the pixel value of the pixel may include depth information corresponding to the pixel, that is, three-dimensional information of the detected object, in the depth image of the detected object obtained by scanning the detected object by the depth camera. The electronic device may construct a second image of the detected object based on depth information of the pixels in the plurality of depth images.
For example, when the detected object is a circuit board and the circuit board is scanned by using the depth camera, the front and back sides of the circuit board can be scanned, so that the electronic device can construct and obtain the second image of the circuit board based on the depth image of the front side of the circuit board and the depth image of the back side of the circuit board.
Furthermore, the electronic device may project the second image onto the detector of the X-ray acquisition device based on the scanning parameters adopted in the X-ray scanning process, so as to obtain a first image corresponding to the two-dimensional transmission image. The second image is an image comprising three-dimensional information of the detected object, in order to carry out boundary constraint on the two-dimensional transmission image by utilizing the second image, the three-dimensional image is required to be projected into the two-dimensional image, and meanwhile, under the condition that the two-dimensional image obtained by projection is consistent with the scanning parameters corresponding to the two-dimensional transmission image, the two-dimensional image obtained by projection has constraint significance on the two-dimensional transmission image, so that the electronic equipment can project the second image onto a detector of the X-ray acquisition equipment based on the scanning parameters adopted in the X-ray scanning process to obtain a first image corresponding to the two-dimensional transmission image, and the scanning parameters corresponding to the two-dimensional transmission image of the scanning parameters corresponding to the first image are consistent.
Since the imaging principles of the X-ray acquisition device and the acquisition device acquiring the second image are different, the internal parameters of both are different from the external parameters. In order to ensure the accuracy of the projection of the second image onto the detector of the X-ray acquisition device, the method may further comprise, before the step of projecting the second image onto the detector of the X-ray acquisition device based on the scanning parameters employed in the X-ray scanning process described above:
acquiring a three-dimensional transmission image of the detected object; and registering the second image to a coordinate system corresponding to the three-dimensional transmission image based on scanning parameters adopted in the X-ray scanning process and scanning parameters corresponding to the second image, so as to obtain a registered second image.
The electronic device may acquire a three-dimensional transmission image of the detected object, wherein the three-dimensional transmission image is reconstructed based on the two-dimensional transmission image. Specifically, the electronic device may acquire a plurality of two-dimensional transmission images of the detected object acquired from a plurality of angles by the X-ray acquisition device, and further perform three-dimensional reconstruction based on the plurality of two-dimensional transmission images, to obtain a three-dimensional transmission image of the detected object.
Because the imaging principles of the X-ray acquisition device and the imaging principles of the acquisition device for acquiring the second image are different, the internal parameters of the two are different from the external parameters, and the coordinate systems corresponding to the second image and the three-dimensional transmission image are different, in order to unify the coordinate systems of the second image and the three-dimensional transmission image and ensure the smooth proceeding of the subsequent processing, the electronic device can register the second image to the coordinate system corresponding to the three-dimensional transmission image based on the scanning parameters adopted in the X-ray scanning process and the scanning parameters corresponding to the second image, so as to obtain the registered second image.
The electronic device may adopt a structural registration manner, and register the second image to a coordinate system corresponding to the X-ray acquisition device based on parameters adopted in the X-ray scanning process and scanning parameters corresponding to the second image, that is, project the coordinate system of the second image onto the coordinate system corresponding to the X-ray acquisition device.
In one embodiment, the second image may be feature matched with the three-dimensional transmission image based on image features in the second image and the three-dimensional transmission image. For example, when the detected object is a circuit board, the second image and the three-dimensional transmission image may include image features corresponding to circuit elements such as capacitance, inductance and resistance in the circuit board, so that the second image may be registered to a coordinate system where the three-dimensional transmission image is located based on the image features corresponding to each other in the three-dimensional image, thereby further improving registration accuracy of the coordinate systems of the second image and the three-dimensional transmission image.
Accordingly, the step of projecting the second image onto the detector of the X-ray acquisition device based on the scanning parameters adopted in the X-ray scanning process may include:
and projecting the registered second image onto a detector of the X-ray acquisition device based on scanning parameters adopted in the X-ray scanning process.
The coordinate system corresponding to the registered second image is consistent with the coordinate system of the X-ray acquisition device, so that the electronic device can simulate and project the registered second image onto a detector of the X-ray acquisition device according to the scanning parameters adopted in the X-ray scanning process, and a first image corresponding to the two-dimensional transmission image is obtained.
For example, the scan parameters may include an acquisition angle, an imaging distance, and the like of the two-dimensional transmission image, and the electronic device may perform analog projection on the registered second image onto a detector of the X-ray acquisition device according to the acquisition angle, the imaging distance, and the like, to obtain the first image. In this way, the obtained first image can simulate the imaging effect of the external contour of the detected object projected onto the detector of the X-ray acquisition device according to the acquisition angle, the imaging distance and the like, so that the first image can be utilized to accurately carry out first boundary constraint on the two-dimensional transmission image.
In one implementation manner, the three-dimensional reconstruction method provided by the embodiment of the present invention may be applied to an image processing system shown in fig. 8, where the image processing system includes a visible light/depth camera module, a plane CT module, and a data processing module, where the visible light/depth camera module is configured to process a visible light image acquired by a visible light acquisition device, where the visible light acquisition device is a visible light camera or a depth camera. The plane CT module is used for processing the X-ray image acquired by the X-ray acquisition equipment, wherein the X-ray acquisition equipment is plane CT equipment. The data processing module is used for carrying out boundary constraint, three-dimensional reconstruction and other processing on the images processed by the visible light/depth camera module and the plane CT module to obtain a three-dimensional detection image of the detected object.
The visible light/depth camera module acquires a visible light image, namely a visible light image or a depth image of the detected object through a multi-angle scanning mode, and reconstructs a second image of the detected object based on MVS or SLAM technology. The plane CT module acquires a two-dimensional transmission image of the detected object through a multi-angle scanning mode, and performs three-dimensional reconstruction (CT pre-reconstruction) to obtain the three-dimensional transmission image of the detected object. The visible light/depth camera module and the planar CT module perform structural registration and image registration on the second image and the three-dimensional transmission image, so that the second image is registered to a coordinate system corresponding to the planar CT device.
The data processing module can project the second image registered in the coordinate system corresponding to the planar CT equipment onto the planar detector of the planar CT equipment according to the acquisition parameters corresponding to the two-dimensional transmission image, so as to obtain a first image. And performing first boundary constraint on the two-dimensional transmission image by using the first image to obtain a corresponding preprocessed two-dimensional transmission image. The image processing module can further take the preprocessed two-dimensional transmission image as original data, take a second image registered in a coordinate system corresponding to the plane CT equipment as priori information, namely, carry out second boundary constraint by using the second image, and carry out three-dimensional reconstruction by an iterative reconstruction algorithm, wherein a reconstruction result is a three-dimensional detection image of the detected object.
In this embodiment, the electronic device may acquire a three-dimensional transmission image of the detected object, register the second image to a coordinate system corresponding to the three-dimensional transmission image based on a scanning parameter adopted in the X-ray scanning process and a scanning parameter corresponding to the second image, and obtain a registered second image, and further, project the registered second image onto a detector of the X-ray acquisition device based on the scanning parameter adopted in the X-ray scanning process, so as to obtain the first image. Therefore, the first image is more accurate, the accuracy of the first boundary constraint based on the first image can be guaranteed, and the accuracy of three-dimensional reconstruction is further improved.
As an implementation of the embodiment of the present invention, the two-dimensional transmission image may include X-ray images of the detected object acquired by the X-ray acquisition device from a plurality of angles. Correspondingly, the step of determining the first image corresponding to the two-dimensional transmission image based on the scanning parameters adopted in the X-ray scanning process may include:
and aiming at each two-dimensional transmission image, determining a first image projected onto a detector of the X-ray acquisition device by the second image according to scanning parameters adopted in the X-ray scanning process corresponding to the two-dimensional transmission image.
In order to ensure that each X-ray image of the detected object acquired from multiple angles can be accurately bounded, for each two-dimensional transmission image, the electronic device can determine, according to scanning parameters adopted in an X-ray scanning process corresponding to the two-dimensional transmission image, a first image of the second image projected onto a detector of the X-ray acquisition device. The specific projection manner has been described in detail in the above embodiments, and will not be described in detail here.
In the case that the two-dimensional transmission image includes an X-ray image of the detected object acquired from a plurality of angles by the X-ray acquisition device, in the process of performing first boundary constraint on the two-dimensional transmission image by using the first image to obtain a preprocessed two-dimensional transmission image, the electronic device performs first boundary constraint on the two-dimensional transmission image by using the first image corresponding to the two-dimensional transmission image for each two-dimensional transmission image, so as to obtain preprocessed two-dimensional transmission images corresponding to the two-dimensional transmission image, so that the preprocessed two-dimensional transmission images are multiple.
The subsequent processing of the two-dimensional transmission image and the preprocessed two-dimensional transmission image may be respectively processing each image, for example, when determining the two-dimensional transmission projection image, the electronic device may determine, according to the scanning parameters corresponding to each two-dimensional transmission image, that the target three-dimensional reconstruction image is projected onto the detector of the X-ray acquisition device according to the scanning parameters corresponding to the two-dimensional transmission image, so as to obtain the two-dimensional transmission projection image.
In this embodiment, the two-dimensional transmission images may include X-ray images of the detected object acquired by the X-ray acquisition device from multiple angles, and then, for each two-dimensional transmission image, the electronic device may determine a corresponding first image, so as to perform a first boundary constraint on each two-dimensional transmission image in a manner described in the foregoing embodiment, and perform a subsequent three-dimensional reconstruction process. In this way, each preprocessed two-dimensional transmission image is more accurate, the quality of the three-dimensional detection image of the detected object obtained by three-dimensional reconstruction based on the preprocessed two-dimensional transmission images is also better, and the three-dimensional reconstruction effect is better.
Corresponding to the above image reconstruction method, the embodiment of the present invention further provides an image reconstruction device, and the image reconstruction device provided by the embodiment of the present invention is described below.
As shown in fig. 9, an image reconstruction apparatus, the apparatus comprising:
an image acquisition module 901, configured to acquire a two-dimensional transmission image obtained by performing X-ray scanning on a detected object based on an X-ray acquisition device;
a projection module 902, configured to determine a first image corresponding to the two-dimensional transmission image based on a scanning parameter adopted in an X-ray scanning process;
the first image is an image of the detected object obtained through an acquisition device with an imaging principle different from that of the X-ray acquisition device.
A first boundary constraint module 903, configured to perform a first boundary constraint on the two-dimensional transmission image by using the first image, so as to obtain a preprocessed two-dimensional transmission image;
the first boundary constraint is used for removing pixel points which do not belong to the object range of the detected object in the two-dimensional transmission image.
And a reconstruction module 904, configured to perform three-dimensional reconstruction based on the preprocessed two-dimensional transmission image, so as to obtain a three-dimensional detection image of the detected object.
In the scheme provided by the embodiment of the invention, the electronic device can determine the first image corresponding to the two-dimensional transmission image based on the two-dimensional transmission image obtained by carrying out X-ray scanning on the detected object by the X-ray acquisition device, wherein the first image is the image of the detected object obtained by the acquisition device with the imaging principle different from that of the X-ray acquisition device, the information of other dimensions of the detected object, such as external contour image information, can be identified, and further, the first boundary constraint can be carried out on the two-dimensional transmission image by utilizing the first image, so that pixel points which do not belong to the external contour range of the detected object and are identified by the corresponding first image in the two-dimensional transmission image are removed, namely pixel points which do not belong to the object range of the detected object are removed, and the second image information of the detected object is added, so that the preprocessed two-dimensional transmission image is more accurate, the problem of three-dimensional reconstruction inaccuracy caused by the data deficiency and inaccuracy of the two-dimensional transmission image can be made up, and the image quality of the three-dimensional detection image of the detected object obtained by carrying out three-dimensional reconstruction based on the preprocessed two-dimensional transmission image is greatly improved. Meanwhile, the calculation of invalid pixel points, namely removed pixel points, is avoided, and the calculation efficiency of a reconstruction algorithm is improved.
As an implementation of the embodiment of the present invention, the reconstruction module 904 may include:
and the initialization unit is used for acquiring the initialized three-dimensional reconstruction image as a target three-dimensional reconstruction image.
And the simulation projection unit is used for determining a two-dimensional transmission projection image of the target three-dimensional reconstruction image projected onto the detector of the X-ray acquisition device according to the scanning parameters adopted in the X-ray scanning process.
An image updating unit, configured to update the target three-dimensional reconstruction image based on a difference between the two-dimensional transmission projection image and the preprocessed two-dimensional transmission image, to obtain an updated three-dimensional reconstruction image;
and the second boundary constraint unit is used for carrying out second boundary constraint on the updated three-dimensional reconstruction image by utilizing the second image to obtain a constrained three-dimensional reconstruction image.
The second boundary constraint is used for removing pixel points which do not belong to the object range of the detected object in the updated three-dimensional reconstruction image, the second image is an image of the detected object acquired by acquisition equipment with an imaging principle different from that of the X-ray acquisition equipment, and the second image comprises three-dimensional information of the detected object.
As an implementation manner of the embodiment of the present invention, the image updating unit includes:
a residual determination subunit, configured to determine a three-dimensional reconstruction residual based on a difference between the two-dimensional transmission projection image and the preprocessed two-dimensional transmission image;
wherein the three-dimensional reconstruction residual is used to identify the accuracy of the target three-dimensional reconstruction image.
An image updating subunit, configured to update the target three-dimensional reconstructed image based on the three-dimensional reconstructed residual error that is counter-propagated if the three-dimensional reconstructed residual error is greater than a preset threshold value, so as to obtain an updated three-dimensional reconstructed image;
the reconstruction module 904 may further include:
and the triggering unit is used for carrying out second boundary constraint on the updated three-dimensional reconstruction image by using the second image, taking the constrained three-dimensional reconstruction image as a target three-dimensional reconstruction image after obtaining the constrained three-dimensional reconstruction image, and triggering the simulated projection unit until the three-dimensional reconstruction residual error is not more than the preset threshold value, so as to obtain a three-dimensional detection image of the detected object.
As an implementation manner of the embodiment of the present invention, the apparatus may include a boundary constraint module, and the boundary constraint module may include:
A constraint image template unit, configured to set a pixel value of an area corresponding to the detected object in a target image as a first target value, and set a pixel value of the rest areas except the detected object in the target image as a second target value, so as to obtain a constraint image template;
when the boundary constraint is a first boundary constraint, the target image is the first image, or when the boundary constraint is a second boundary constraint, the target image is the second image.
The boundary constraint unit is used for keeping the pixel values of the pixel points corresponding to the detected object in the image to be constrained unchanged based on the constraint image template, determining the pixel values of the pixel points corresponding to the other areas except the detected object in the image to be constrained as a third target value, and obtaining a constrained image;
when the boundary constraint is a first boundary constraint, the image to be constrained is the two-dimensional transmission image, the constrained image is the preprocessed two-dimensional transmission image, and when the boundary constraint is a second boundary constraint, the image to be constrained is the updated three-dimensional reconstruction image, and the constrained image is the constrained three-dimensional reconstruction image.
As one implementation of the embodiment of the present invention, the boundary constraint unit may include:
the first constraint subunit is configured to calculate, when the boundary constraint is the first boundary constraint, a pixel value of each pixel point in the preprocessed two-dimensional transmission image according to the following formula:
A M (i，j)＝A(i，j)×M(i，j)
wherein A is M (i, j) is the pixel value of the pixel point with the coordinates (i, j) in the preprocessed two-dimensional transmission image, and A (i,j) And M (i, j) is the pixel value of the pixel point with the coordinate (i, j) in the constraint image template corresponding to the two-dimensional transmission image.
And the second constraint subunit is used for calculating the pixel value of each pixel point in the constrained three-dimensional reconstruction image according to the following formula when the boundary constraint is the second boundary constraint:
A M (i，j，k)＝A(i，j，k)×M(i，j，k)
wherein A is M (i, j, k) is the pixel value of the pixel point with the coordinate of (i, j, k) in the constrained three-dimensional reconstruction image, A (i, j, k) is the pixel value of the pixel point with the coordinate of (i, j, k) in the updated three-dimensional reconstruction image, and M (i, j, k) is the pixel value of the pixel point with the coordinate of (i, j, k) in the constrained image template corresponding to the second image.
As one implementation of the embodiment of the present invention, the above-mentioned analog projection unit may include:
a first arrangement subunit, configured to arrange pixel values of the target three-dimensional reconstructed image into one-dimensional image data.
A system matrix determining subunit, configured to determine a corresponding system matrix based on the arrangement mode corresponding to the one-dimensional image data and a scanning parameter adopted in the X-ray scanning process;
wherein the system matrix is used for identifying a coordinate conversion relation between the one-dimensional image data and the two-dimensional transmission projection image.
The simulation projection subunit is used for calculating a two-dimensional transmission projection image corresponding to the target three-dimensional reconstruction image according to the system matrix and the one-dimensional image data;
the image update unit may include:
and the data updating subunit is used for updating the one-dimensional image data based on the difference between the two-dimensional transmission projection image and the preprocessed two-dimensional transmission image to obtain updated one-dimensional image data.
And the second arrangement subunit is used for arranging the updated one-dimensional image data according to the inverse operation of the arrangement mode to obtain an updated three-dimensional reconstruction image.
As an implementation of the embodiment of the present invention, the projection module 902 may include:
a second image acquisition unit configured to acquire a second image of the detected object;
and the first image projection unit is used for projecting the second image onto a detector of the X-ray acquisition equipment based on scanning parameters adopted in the X-ray scanning process to obtain a first image corresponding to the two-dimensional transmission image.
As an implementation manner of the embodiment of the present invention, the foregoing apparatus may further include:
the three-dimensional transmission image acquisition module is used for acquiring a three-dimensional transmission image of the detected object before the second image is projected onto a detector of the X-ray acquisition device based on scanning parameters adopted in an X-ray scanning process;
the three-dimensional transmission image is reconstructed based on the two-dimensional transmission image.
The image registration module is used for registering the second image to a coordinate system corresponding to the three-dimensional transmission image based on scanning parameters adopted in the X-ray scanning process and scanning parameters corresponding to the second image, so as to obtain a registered second image;
the first image projection unit may include:
And the image projection subunit is used for projecting the registered second image onto a detector of the X-ray acquisition device based on scanning parameters adopted in the X-ray scanning process.
As one implementation of the embodiment of the present invention, the above two-dimensional transmission image may include X-ray images of the detected object acquired by the X-ray acquisition device from a plurality of angles;
the projection module 902 may include:
and the second image projection unit is used for determining a first image projected onto a detector of the X-ray acquisition device by the second image according to the scanning parameters adopted in the X-ray scanning process corresponding to each two-dimensional transmission image.
Corresponding to the image reconstruction method, the embodiment of the invention also provides an image reconstruction system, and the image reconstruction system provided by the embodiment of the invention is described below.
As shown in fig. 10, an image reconstruction system, the system comprising: an object acquisition apparatus 1001, an X-ray acquisition apparatus 1002, and an image processing apparatus 1003, wherein:
the X-ray acquisition device 1002 is configured to acquire a two-dimensional transmission image obtained by performing X-ray scanning on the detected object based on the X-ray acquisition device.
The image processing apparatus 1003 is configured to determine a first image corresponding to the two-dimensional transmission image; performing first boundary constraint on the two-dimensional transmission image by using the first image to obtain a preprocessed two-dimensional transmission image; and carrying out three-dimensional reconstruction based on the preprocessed two-dimensional transmission image to obtain a three-dimensional detection image of the detected object.
The first image is an image of the detected object obtained by a target acquisition device 1001 with an imaging principle different from that of the X-ray acquisition device, and the first boundary constraint is used for removing pixels in the two-dimensional transmission image, which do not belong to an object range of the detected object.
In the scheme provided by the embodiment of the invention, the image processing device can determine the first image corresponding to the two-dimensional transmission image based on the two-dimensional transmission image obtained by carrying out X-ray scanning on the detected object by the X-ray acquisition device, wherein the first image is the image of the detected object obtained by the acquisition device with the imaging principle different from that of the X-ray acquisition device, can identify information of other dimensions of the detected object, such as external contour image information, and further can carry out first boundary constraint on the two-dimensional transmission image by using the first image, thereby removing pixel points which do not belong to the external contour range of the detected object and are identified by the corresponding first image in the two-dimensional transmission image, namely removing pixel points which do not belong to the object range of the detected object, and adding the second image information of the detected object, so that the preprocessed two-dimensional transmission image is more accurate, the problem of three-dimensional reconstruction inaccuracy caused by the data deficiency and inaccuracy of the two-dimensional transmission image can be made up, and the image quality of the three-dimensional detection image of the detected object obtained by carrying out three-dimensional reconstruction on the preprocessed two-dimensional transmission image is greatly improved. Meanwhile, the calculation of invalid pixel points, namely removed pixel points, is avoided, and the calculation efficiency of a reconstruction algorithm is improved.
The embodiment of the present invention further provides an electronic device, as shown in fig. 11, including a processor 1101, a communication interface 1102, a memory 1103 and a communication bus 1104, where the processor 1101, the communication interface 1102 and the memory 1103 complete communication with each other through the communication bus 1104,
a memory 1103 for storing a computer program;
the processor 1101 is configured to implement the method steps described in any of the foregoing embodiments when executing a program stored in the memory 1103.
The communication bus mentioned above for the electronic devices may be a peripheral component interconnect standard (Peripheral Component Interconnect, PCI) bus or an extended industry standard architecture (Extended Industry Standard Architecture, EISA) bus, etc. The communication bus may be classified as an address bus, a data bus, a control bus, or the like. For ease of illustration, the figures are shown with only one bold line, but not with only one bus or one type of bus.
The communication interface is used for communication between the electronic device and other devices.
The Memory may include random access Memory (Random Access Memory, RAM) or may include Non-Volatile Memory (NVM), such as at least one disk Memory. Optionally, the memory may also be at least one memory device located remotely from the aforementioned processor.
The processor may be a general-purpose processor, including a central processing unit (Central Processing Unit, CPU), a network processor (Network Processor, NP), etc.; but also digital signal processors (Digital Signal Processor, DSP), application specific integrated circuits (Application Specific Integrated Circuit, ASIC), field programmable gate arrays (Field-Programmable Gate Array, FPGA) or other programmable logic devices, discrete gate or transistor logic devices, discrete hardware components.
In a further embodiment of the present invention, a computer readable storage medium is also provided, in which a computer program is stored, which computer program, when being executed by a processor, implements the steps of the method according to any of the embodiments described above.
In a further embodiment of the present invention, a computer program product comprising instructions is also provided, which when run on a computer causes the computer to perform the method steps of any of the embodiments described above.
In the above embodiments, it may be implemented in whole or in part by software, hardware, firmware, or any combination thereof. When implemented in software, may be implemented in whole or in part in the form of a computer program product. The computer program product includes one or more computer instructions. When loaded and executed on a computer, produces a flow or function in accordance with embodiments of the present invention, in whole or in part. The computer may be a general purpose computer, a special purpose computer, a computer network, or other programmable apparatus. The computer instructions may be stored in or transmitted from one computer-readable storage medium to another, for example, by wired (e.g., coaxial cable, optical fiber, digital Subscriber Line (DSL)), or wireless (e.g., infrared, wireless, microwave, etc.). The computer readable storage medium may be any available medium that can be accessed by a computer or a data storage device such as a server, data center, etc. that contains an integration of one or more available media. The usable medium may be a magnetic medium (e.g., floppy Disk, hard Disk, magnetic tape), an optical medium (e.g., DVD), or a semiconductor medium (e.g., solid State Disk (SSD)), etc.
It is noted that relational terms such as first and second, and the like are used solely to distinguish one entity or action from another entity or action without necessarily requiring or implying any actual such relationship or order between such entities or actions. Moreover, the terms "comprises," "comprising," or any other variation thereof, are intended to cover a non-exclusive inclusion, such that a process, method, article, or apparatus that comprises a list of elements does not include only those elements but may include other elements not expressly listed or inherent to such process, method, article, or apparatus. Without further limitation, an element defined by the phrase "comprising one … …" does not exclude the presence of other like elements in a process, method, article, or apparatus that comprises the element.
In this specification, each embodiment is described in a related manner, and identical and similar parts of each embodiment are all referred to each other, and each embodiment mainly describes differences from other embodiments. In particular, for apparatus, systems, electronic devices, computer readable storage media, and computer program product embodiments, the description is relatively simple as it is substantially similar to method embodiments, as relevant points are found in the partial description of method embodiments.
The foregoing description is only of the preferred embodiments of the present invention and is not intended to limit the scope of the present invention. Any modification, equivalent replacement, improvement, etc. made within the spirit and principle of the present invention are included in the protection scope of the present invention.
Claims (13)
1. A method of image reconstruction, the method comprising:
acquiring a two-dimensional transmission image obtained by carrying out X-ray scanning on a detected object based on X-ray acquisition equipment;
determining a first image corresponding to the two-dimensional transmission image, wherein the first image is an image of the detected object obtained through an acquisition device with an imaging principle different from that of the X-ray acquisition device;
performing first boundary constraint on the two-dimensional transmission image by using the first image to obtain a preprocessed two-dimensional transmission image, wherein the first boundary constraint is used for removing pixel points which do not belong to the object range of the detected object in the two-dimensional transmission image;
and carrying out three-dimensional reconstruction based on the preprocessed two-dimensional transmission image to obtain a three-dimensional detection image of the detected object.
2. The method according to claim 1, wherein the step of performing three-dimensional reconstruction based on the preprocessed two-dimensional transmission image to obtain a three-dimensional detection image of the detected object comprises:
Acquiring an initialized three-dimensional reconstruction image as a target three-dimensional reconstruction image;
according to scanning parameters adopted in the X-ray scanning process, determining a two-dimensional transmission projection image of the three-dimensional reconstruction image of the target projected onto a detector of the X-ray acquisition equipment;
updating the target three-dimensional reconstruction image based on the difference between the two-dimensional transmission projection image and the preprocessed two-dimensional transmission image to obtain an updated three-dimensional reconstruction image;
and performing second boundary constraint on the updated three-dimensional reconstruction image to obtain a constrained three-dimensional reconstruction image, wherein the second boundary constraint is used for removing pixel points which do not belong to the object range of the detected object in the updated three-dimensional reconstruction image, the second image is an image of the detected object acquired by acquisition equipment with an imaging principle different from that of the X-ray acquisition equipment, and the second image comprises three-dimensional information of the detected object.
3. The method of claim 2, wherein the step of updating the target three-dimensional reconstructed image based on a difference between the two-dimensional transmission projection image and the preprocessed two-dimensional transmission image comprises:
Determining a three-dimensional reconstruction residual error based on a difference between the two-dimensional transmission projection image and the preprocessed two-dimensional transmission image, wherein the three-dimensional reconstruction residual error is used for identifying the accuracy of the target three-dimensional reconstruction image;
if the three-dimensional reconstruction residual error is larger than a preset threshold value, back propagation is carried out based on the three-dimensional reconstruction residual error, and the target three-dimensional reconstruction image is updated to obtain an updated three-dimensional reconstruction image;
after the step of using the second image to perform a second boundary constraint on the updated three-dimensional reconstructed image to obtain a constrained three-dimensional reconstructed image, the method further includes:
and taking the constrained three-dimensional reconstruction image as a target three-dimensional reconstruction image, and returning to the step of determining a two-dimensional transmission projection image of the target three-dimensional reconstruction image projected onto a detector of the X-ray acquisition equipment according to scanning parameters adopted in the X-ray scanning process until the three-dimensional reconstruction residual error is not larger than the preset threshold value, so as to obtain a three-dimensional detection image of the detected object.
4. The method according to claim 2, characterized in that the manner of boundary constraint comprises:
Setting a pixel value of a region corresponding to the detected object in a target image as a first target value, and setting pixel values of other regions except the detected object in the target image as a second target value to obtain a constraint image template, wherein when the boundary constraint is a first boundary constraint, the target image is the first image, or when the boundary constraint is a second boundary constraint, the target image is the second image;
based on the constraint image template, keeping the pixel values of the pixel points corresponding to the detected object in the image to be constrained unchanged, and determining the pixel values of the pixel points corresponding to the other areas except the detected object in the image to be constrained as a third target value to obtain a constrained image; when the boundary constraint is a first boundary constraint, the image to be constrained is the two-dimensional transmission image, the constrained image is the preprocessed two-dimensional transmission image, or when the boundary constraint is a second boundary constraint, the image to be constrained is the updated three-dimensional reconstruction image, the constrained image is the constrained three-dimensional reconstruction image, and the constrained image template and the image to be constrained show the same detected object.
5. The method according to claim 4, wherein the step of determining, based on the constraint image template, the pixel values of the pixels corresponding to the detected object in the image to be constrained and the pixel values of the pixels corresponding to the remaining areas except the detected object in the image to be constrained as the third target value includes:
when the boundary constraint is the first boundary constraint, calculating the pixel value of each pixel point in the preprocessed two-dimensional transmission image according to the following formula:
A M (i，j)＝A(i，j)×M(i，j)
wherein A is M (i, j) is the pixel value of the pixel point with the coordinates of (i, j) in the preprocessed two-dimensional transmission image, A (i, j) is the pixel value of the pixel point with the coordinates of (i, j) in the two-dimensional transmission image, and M (i, j) is the pixel value of the pixel point with the coordinates of (i, j) in a constraint image template corresponding to the two-dimensional transmission image;
when the boundary constraint is the second boundary constraint, calculating pixel values of all pixel points in the constrained three-dimensional reconstruction image according to the following formula:
A M (i，j，k)＝A(i，j，k)×M(i，j，k)
wherein A is M (i, j, k) is in the constrained three-dimensional reconstructed imageAnd (3) the pixel value of the pixel point with the coordinates of (i, j, k), wherein A (i, j, k) is the pixel value of the pixel point with the coordinates of (i, j, k) in the updated three-dimensional reconstruction image, and M (i, j, k) is the pixel value of the pixel point with the coordinates of (i, j, k) in the constraint image template corresponding to the second image.
6. The method according to claim 2, wherein the step of determining a two-dimensional transmission projection image of the three-dimensional reconstruction image of the object onto a detector of the X-ray acquisition device based on scanning parameters employed during the X-ray scanning comprises:
arranging pixel values of the target three-dimensional reconstructed image into one-dimensional image data;
determining a system matrix based on an arrangement mode corresponding to the one-dimensional image data and scanning parameters adopted in an X-ray scanning process, wherein the system matrix is used for identifying a coordinate conversion relation between the one-dimensional image data and the two-dimensional transmission projection image;
according to the system matrix and the one-dimensional image data, calculating a two-dimensional transmission projection image corresponding to the target three-dimensional reconstruction image;
the step of updating the target three-dimensional reconstruction image based on the difference between the two-dimensional transmission projection image and the preprocessed two-dimensional transmission image to obtain an updated three-dimensional reconstruction image comprises the following steps:
updating the one-dimensional image data based on the difference between the two-dimensional transmission projection image and the preprocessed two-dimensional transmission image to obtain updated one-dimensional image data;
And arranging the updated one-dimensional image data according to the inverse operation of the arrangement mode to obtain an updated three-dimensional reconstruction image.
7. The method of any of claims 1-6, wherein the step of determining the first image corresponding to the two-dimensional transmission image comprises:
acquiring a second image of the detected object;
and based on scanning parameters adopted in the X-ray scanning process, projecting the second image onto a detector of the X-ray acquisition equipment to obtain a first image corresponding to the two-dimensional transmission image.
8. The method of claim 7, wherein prior to the step of projecting the second image onto a detector of the X-ray acquisition device based on scanning parameters employed in the X-ray scanning process, the method further comprises:
acquiring a three-dimensional transmission image of the detected object, wherein the three-dimensional transmission image is reconstructed based on the two-dimensional transmission image;
registering the second image to a coordinate system corresponding to the three-dimensional transmission image based on scanning parameters adopted in the X-ray scanning process and scanning parameters corresponding to the second image to obtain a registered second image;
The step of projecting the second image onto a detector of the X-ray acquisition device based on scanning parameters employed in the X-ray scanning process comprises:
and projecting the registered second image onto a detector of the X-ray acquisition device based on scanning parameters adopted in the X-ray scanning process.
9. The method of any of claims 1-6, wherein the two-dimensional transmission image comprises X-ray images of the detected object acquired by the X-ray acquisition device from a plurality of angles;
the step of determining the first image corresponding to the two-dimensional transmission image comprises the following steps:
and aiming at each two-dimensional transmission image, determining a first image projected onto a detector of the X-ray acquisition device by the second image according to scanning parameters adopted in the X-ray scanning process corresponding to the two-dimensional transmission image.
10. An image reconstruction apparatus, the apparatus comprising:
the image acquisition module is used for acquiring a two-dimensional transmission image obtained by carrying out X-ray scanning on the detected object based on the X-ray acquisition equipment;
the projection module is used for determining a first image corresponding to the two-dimensional transmission image, wherein the first image is an image of the detected object obtained through an acquisition device with an imaging principle different from that of the X-ray acquisition device;
The first boundary constraint module is used for carrying out first boundary constraint on the two-dimensional transmission image by utilizing the first image to obtain a preprocessed two-dimensional transmission image, wherein the first boundary constraint module is used for removing pixel points which do not belong to the object range of the detected object in the two-dimensional transmission image;
and the reconstruction module is used for carrying out three-dimensional reconstruction based on the preprocessed two-dimensional transmission image to obtain a three-dimensional detection image of the detected object.
11. An image reconstruction system, comprising a target acquisition device, an X-ray acquisition device, and an image processing device, wherein:
the X-ray acquisition equipment is used for acquiring a two-dimensional transmission image obtained by carrying out X-ray scanning on the detected object based on the X-ray acquisition equipment;
the image processing device is used for determining a first image corresponding to the two-dimensional transmission image; performing first boundary constraint on the two-dimensional transmission image by using the first image to obtain a preprocessed two-dimensional transmission image; and carrying out three-dimensional reconstruction based on the preprocessed two-dimensional transmission image to obtain a three-dimensional detection image of the detected object, wherein the first image is an image of the detected object obtained by a target acquisition device with an imaging principle different from that of the X-ray acquisition device, and the first boundary constraint is used for removing pixel points which do not belong to the object range of the detected object in the two-dimensional transmission image.
12. The electronic equipment is characterized by comprising a processor, a communication interface, a memory and a communication bus, wherein the processor, the communication interface and the memory are communicated with each other through the communication bus;
a memory for storing a computer program;
a processor for carrying out the method steps of any one of claims 1-9 when executing a program stored on a memory.
13. A computer-readable storage medium, characterized in that the computer-readable storage medium has stored therein a computer program which, when executed by a processor, implements the method steps of any of claims 1-9.
Priority Applications (1)
| Application Number | Priority Date | Filing Date | Title |
|---|---|---|---|
| CN202211088631.4A CN117710574A (en) | 2022-09-07 | 2022-09-07 | Image reconstruction method, device, system, electronic equipment and storage medium |
Applications Claiming Priority (1)
| Application Number | Priority Date | Filing Date | Title |
|---|---|---|---|
| CN202211088631.4A CN117710574A (en) | 2022-09-07 | 2022-09-07 | Image reconstruction method, device, system, electronic equipment and storage medium |
Publications (1)
| Publication Number | Publication Date |
|---|---|
| CN117710574A true CN117710574A (en) | 2024-03-15 |
Family
ID=90159376
Family Applications (1)
| Application Number | Title | Priority Date | Filing Date |
|---|---|---|---|
| CN202211088631.4A Pending CN117710574A (en) | 2022-09-07 | 2022-09-07 | Image reconstruction method, device, system, electronic equipment and storage medium |
Country Status (1)
| Country | Link |
|---|---|
| CN (1) | CN117710574A (en) |
-
2022
- 2022-09-07 CN CN202211088631.4A patent/CN117710574A/en active Pending
Similar Documents
| Publication | Publication Date | Title |
|---|---|---|
| JP7222582B2 (en) | Method for structure-from-motion (SfM) processing in computer vision systems | |
| US11953451B2 (en) | Item inspection by dynamic selection of projection angle | |
| WO2023082306A1 (en) | Image processing method and apparatus, and electronic device and computer-readable storage medium | |
| CN112686961A (en) | Method and device for correcting calibration parameters of depth camera | |
| KR20190123865A (en) | Calibration method of x-ray apparatus and calibration apparatus for the same | |
| US9857163B2 (en) | Parametric control of object scanning | |
| KR101621815B1 (en) | Computed tomography image processing apparatus and method of generating three-dimensional reconstructed image | |
| CN117710574A (en) | Image reconstruction method, device, system, electronic equipment and storage medium | |
| Sebar et al. | A trustable 3D photogrammetry approach for cultural heritage | |
| US11935228B2 (en) | Method to acquire a 3D image of a sample structure | |
| CN114966578A (en) | Radar external parameter calibration method and device based on shooting equipment and computer equipment | |
| TW201922178A (en) | Projection matching method for X-ray tomography | |
| US20210103158A1 (en) | Individual Channel Characterization of Collimator | |
| Lourenço et al. | Extrinsic calibration of 2D laser range finders using planar features | |
| Eichhardt et al. | Improvement of camera calibration using surface normals | |
| Dias et al. | 3d reconstruction with low resolution, small baseline and high radial distortion stereo images | |
| CN113487626B (en) | Mirror image identification method and device, electronic equipment and storage medium | |
| CN112750152B (en) | Image registration method and device, computer equipment and storage medium | |
| CN114877826B (en) | Binocular stereo matching three-dimensional measurement method, system and storage medium | |
| Hu et al. | Minimal vision system for linear camera self-calibration and 3D reconstruction using translation platform | |
| Govorov et al. | Study of digital camera calibration on a flat test object | |
| Bevilacqua et al. | Self-calibration for a camera-projector pair | |
| Lehr et al. | 3D reconstruction of volume defects from few X-ray images | |
| Kim | Study on Comparative Analysis of Camera Calibration. | |
| CN115560687A (en) | Method and device for measuring size of drilling bucket and rotary drilling rig |
Legal Events
| Date | Code | Title | Description |
|---|---|---|---|
| PB01 | Publication | ||
| PB01 | Publication | ||
| SE01 | Entry into force of request for substantive examination | ||
| SE01 | Entry into force of request for substantive examination |