CN111161407A - Underwater three-dimensional reconstruction system and method - Google Patents
Underwater three-dimensional reconstruction system and method Download PDFInfo
- Publication number
- CN111161407A CN111161407A CN201911369346.8A CN201911369346A CN111161407A CN 111161407 A CN111161407 A CN 111161407A CN 201911369346 A CN201911369346 A CN 201911369346A CN 111161407 A CN111161407 A CN 111161407A
- Authority
- CN
- China
- Prior art keywords
- image
- underwater
- dimensional
- module
- dimensional reconstruction
- Prior art date
- Legal status (The legal status is an assumption and is not a legal conclusion. Google has not performed a legal analysis and makes no representation as to the accuracy of the status listed.)
- Withdrawn
Links
Images
Classifications
-
- G—PHYSICS
- G06—COMPUTING; CALCULATING OR COUNTING
- G06T—IMAGE DATA PROCESSING OR GENERATION, IN GENERAL
- G06T17/00—Three dimensional [3D] modelling, e.g. data description of 3D objects
-
- G—PHYSICS
- G06—COMPUTING; CALCULATING OR COUNTING
- G06T—IMAGE DATA PROCESSING OR GENERATION, IN GENERAL
- G06T15/00—3D [Three Dimensional] image rendering
- G06T15/005—General purpose rendering architectures
-
- G—PHYSICS
- G06—COMPUTING; CALCULATING OR COUNTING
- G06T—IMAGE DATA PROCESSING OR GENERATION, IN GENERAL
- G06T9/00—Image coding
Landscapes
- Engineering & Computer Science (AREA)
- Physics & Mathematics (AREA)
- General Physics & Mathematics (AREA)
- Theoretical Computer Science (AREA)
- Computer Graphics (AREA)
- Geometry (AREA)
- Software Systems (AREA)
- Multimedia (AREA)
- Image Processing (AREA)
Abstract
The invention discloses an underwater three-dimensional reconstruction system and method, and relates to the field of underwater detection. The system comprises: the system comprises an embedded processing platform arranged underwater, a binocular camera arranged underwater and an industrial personal computer arranged on water; the binocular camera is used for acquiring underwater images and sending the underwater images to the embedded processing platform; the embedded processing platform is used for compressing and coding the underwater image and sending the compressed and coded underwater image to the industrial personal computer; the industrial personal computer is used for decoding the underwater image, processing the stereo image and then performing three-dimensional reconstruction according to the underwater image processed by the stereo image. By the aid of the system, finer underwater scene three-dimensional reconstruction is realized, three-dimensional reconstruction resolution is greatly improved, and the system has great advantages in visualization.
Description
Technical Field
The invention relates to the field of underwater detection, in particular to an underwater three-dimensional reconstruction system and method.
Background
With the further demand and development of ocean resources, the underwater detection technology is also rapidly developed. The underwater three-dimensional reconstruction has wide requirements in the fields of underwater monitoring, underwater operation and the like. The current mainstream underwater three-dimensional reconstruction technology mainly adopts a sonar imaging technology, obtains the distance of a target through sound reflection, obtains the distance information of a space through modes such as scanning and array, and maps a two-dimensional image to a three-dimensional space to finally obtain a three-dimensional model of the target. However, the space resolution of sonar is low, so that the reconstructed three-dimensional model is not ideal in effect, the surface of the model is rough, the texture information of the surface is lost, and the model is inconvenient to use.
Disclosure of Invention
The invention aims to solve the technical problem of the prior art and provides an underwater three-dimensional reconstruction system and method.
The technical scheme for solving the technical problems is as follows:
an underwater three-dimensional reconstruction system comprising: the system comprises an embedded processing platform arranged underwater, a binocular camera arranged underwater and an industrial personal computer arranged on water;
the binocular camera is used for acquiring underwater images and sending the underwater images to the embedded processing platform;
the embedded processing platform is used for compressing and coding the underwater image and sending the compressed and coded underwater image to the industrial personal computer;
the industrial personal computer is used for decoding the underwater image, processing the stereo image and then performing three-dimensional reconstruction according to the underwater image processed by the stereo image.
The invention has the beneficial effects that: this scheme is through two mesh camera collection images under water, embedded processing platform is right again image under water carries out the compression and compiles, and send the image under water after the compression coding for the industrial computer, the industrial computer is to image decoding under water, three-dimensional image processing, carry out the three-dimensional reconstruction of image under water again, system through this scheme, realize more meticulous scene three-dimensional reconstruction under water, compare the scheme of prior art sonar formation of image, three-dimensional resolution ratio has been improved greatly to rebuild, and there is huge advantage in visualization, simultaneously, the result of three-dimensional reconstruction can directly be seen in real-time three-dimensional reconstruction, make operating personnel can assess the reconstruction effect in real time, the remote operation of being convenient for.
On the basis of the technical scheme, the invention can be further improved as follows.
Further, the industrial computer specifically includes: the device comprises a correction module, a stereo matching module and a depth calculation module;
the correction module is used for correcting the underwater image according to the two camera parameters calibrated by the binocular camera;
the stereo matching module is used for calculating a parallax image which is matched with the corrected underwater image and contains the characteristic points by using the CUDA;
the depth calculation module is used for selecting the feature points according to a preset rule, obtaining the depth values of the selected feature points through a triangulation principle, and finishing the three-dimensional image processing of the underwater image.
The beneficial effect of adopting the further scheme is that: the underwater image is corrected by the image correction module according to the two camera parameters calibrated by the binocular camera, so that the rapid stereo matching is conveniently realized; the stereo matching calculation process is accelerated by adopting CUDA (compute unified device architecture), so that the real-time performance of the system and the operation amount in the stereo matching process are improved; more feature points are selected in the area with larger depth change, less feature points are selected in the area with smaller depth change, the accuracy of feature point extraction is improved, and the depth value corresponding to the feature points can be obtained through the triangulation principle.
Further, the industrial computer still specifically includes: the device comprises an image enhancement module and a parallax restoration module;
the image enhancement module is used for carrying out image enhancement on the corrected underwater image by adopting a method of combining homomorphic filtering and histogram equalization according to the characteristics of underwater image imaging;
and the parallax repairing module is used for repairing the time difference image by adopting a bilinear interpolation and morphological smoothing mode on the parallax image before the depth calculation.
The beneficial effect of adopting the further scheme is that: the method combines homomorphic filtering and histogram equalization through an image enhancement module, then performs image enhancement processing on the acquired image according to a constrained least square filtering algorithm, so as to realize defogging and solve the problem of uneven illumination, reduce the blurring phenomenon caused by the image, and add an image enhancement and recovery method aiming at the imaging characteristics of the underwater image before the stereo matching of the image, thereby improving the accuracy of the stereo matching.
Further, the industrial computer specifically still includes: the device comprises a three-dimensional mapping module, a data fusion module, a triangulation module and a texture rendering module;
the three-dimensional mapping module is used for mapping the feature points to a three-dimensional space according to the two-dimensional coordinates of the feature points and the camera parameters after the depth calculation to obtain a three-dimensional point cloud;
the data fusion module is used for updating the three-dimensional point cloud correspondingly according to the image frame of the underwater image until the data fusion is completed;
the triangulation module is used for triangulating the three-dimensional point cloud by adopting a local Delaunay grid boundary construction method after data fusion is completed, and connecting the divided three-dimensional point sets to form a triangular network so as to obtain topological structure information of the object surface of the underwater image;
the texture rendering module is used for performing mapping rendering on the triangular network to complete three-dimensional reconstruction of the underwater image.
The beneficial effect of adopting the further scheme is that: mapping the two-dimensional coordinates of each characteristic point to a three-dimensional space through a three-dimensional mapping module to obtain a three-dimensional point cloud corresponding to the characteristic points, so that the two-dimensional coordinates are mapped to an image of the three-dimensional space; the data fusion module is used for carrying out interpolation processing on repeated point cloud data by taking a coordinate system when the first frame of image is imaged as a reference and converting subsequent images into the coordinate system, and deleting redundancy to realize the elimination of the repeated point cloud data; the triangulation module adopts a local Delaunay grid boundary construction method to connect the three-dimensional point sets into a triangular network to obtain topological structure information of the surface of the object so as to facilitate visual mapping; the texture rendering module maps texture pictures inside the two-dimensional image corresponding to the three vertexes of the triangle to the triangular area through an OpenGL library, so that texture rendering of the whole model is achieved, and the visualization effect is improved.
Further, the industrial personal computer also comprises a GPU with a CUDA core and the computing capacity of more than 7.0.
The beneficial effect of adopting the further scheme is that: the operation is accelerated by adding the high-performance GPU, so that the real-time performance of the whole system is improved, and the operation of personnel is facilitated.
Another technical solution of the present invention for solving the above technical problems is as follows:
an underwater three-dimensional reconstruction method, comprising:
s1, collecting underwater images and sending the underwater images to the embedded processing platform;
s2, carrying out compression coding on the underwater image, and sending the compressed and coded underwater image to the industrial personal computer;
and S3, decoding the underwater image, processing the stereo image, and performing three-dimensional reconstruction according to the underwater image processed by the stereo image.
The invention has the beneficial effects that: this scheme is through two mesh camera collection images under water, embedded processing platform is right again image under water carries out the compression and compiles, and send the image under water after the compression coding for the industrial computer, the industrial computer is to image decoding under water, three-dimensional image processing, carry out the three-dimensional reconstruction of image under water again, system through this scheme, realize more meticulous scene three-dimensional reconstruction under water, compare the scheme of prior art sonar formation of image, three-dimensional resolution ratio has been improved greatly to rebuild, and there is huge advantage in visualization, simultaneously, the result of three-dimensional reconstruction can directly be seen in real-time three-dimensional reconstruction, make operating personnel can assess the reconstruction effect in real time, the remote operation of being convenient for.
On the basis of the technical scheme, the invention can be further improved as follows.
Further, the image processing specifically includes:
correcting the underwater image according to the two camera parameters calibrated by the binocular camera;
calculating a parallax image which is matched with the corrected underwater image and contains the characteristic points by using the CUDA;
and selecting the feature points according to a preset rule, and obtaining the depth values of the selected feature points through a triangulation principle to finish the three-dimensional image processing of the underwater image.
The beneficial effect of adopting the further scheme is that: the underwater image is corrected by the image correction module according to the two camera parameters calibrated by the binocular camera, so that the rapid stereo matching is conveniently realized; the stereo matching calculation process is accelerated by adopting CUDA (compute unified device architecture), so that the real-time performance of the system and the operation amount in the stereo matching process are improved; more feature points are selected in the area with larger depth change, less feature points are selected in the area with smaller depth change, the accuracy of feature point extraction is improved, and the depth value corresponding to the feature points can be obtained through the triangulation principle.
Further, the image processing further specifically includes:
according to the characteristics of underwater image imaging, image enhancement is carried out on the corrected underwater image by adopting a method of combining homomorphic filtering and histogram equalization;
and repairing the time difference image by adopting a bilinear interpolation and morphological smoothing mode on the parallax image before the depth calculation.
The beneficial effect of adopting the further scheme is that: the method combines homomorphic filtering and histogram equalization through an image enhancement module, then performs image enhancement processing on the acquired image according to a constrained least square filtering algorithm, so as to realize defogging and solve the problem of uneven illumination, reduce the blurring phenomenon caused by the image, and add an image enhancement and recovery method aiming at the imaging characteristics of the underwater image before the stereo matching of the image, thereby improving the accuracy of the stereo matching.
Further, the three-dimensional reconstruction specifically includes:
mapping the feature points to a three-dimensional space according to the two-dimensional coordinates of the feature points and the camera parameters after the depth calculation to obtain three-dimensional point cloud;
updating the three-dimensional point cloud correspondingly according to the image frame update of the underwater image until data fusion is completed;
after data fusion is completed, triangularization subdivision is carried out on the three-dimensional point cloud by adopting a local Delaunay grid boundary construction method, the divided three-dimensional point sets are connected to form a triangular network, and topological structure information of the object surface of the underwater image is obtained;
and performing mapping rendering on the triangular network to finish the three-dimensional reconstruction of the underwater image.
The beneficial effect of adopting the further scheme is that: mapping the two-dimensional coordinates of each characteristic point to a three-dimensional space through a three-dimensional mapping module to obtain a three-dimensional point cloud corresponding to the characteristic points, so that the two-dimensional coordinates are mapped to an image of the three-dimensional space; the data fusion module is used for carrying out interpolation processing on repeated point cloud data by taking a coordinate system when the first frame of image is imaged as a reference and converting subsequent images into the coordinate system, and deleting redundancy to realize the elimination of the repeated point cloud data; the triangulation module adopts a local Delaunay grid boundary construction method to connect the three-dimensional point sets into a triangular network to obtain topological structure information of the surface of the object so as to facilitate visual mapping; the texture rendering module maps texture pictures inside the two-dimensional image corresponding to the three vertexes of the triangle to the triangular area through an OpenGL library, so that texture rendering of the whole model is achieved, and the visualization effect is improved.
Further, the industrial personal computer also comprises a GPU with a CUDA core and the computing capacity of more than 7.0.
The beneficial effect of adopting the further scheme is that: the operation is accelerated by adding the high-performance GPU, so that the real-time performance of the whole system is improved, and the operation of personnel is facilitated.
Advantages of additional aspects of the invention will be set forth in part in the description which follows, and in part will be obvious from the description, or may be learned by practice of the invention.
Drawings
Fig. 1 is a block diagram of an underwater three-dimensional reconstruction system according to an embodiment of the present invention;
FIG. 2 is a hardware connection diagram of a three-dimensional reconstruction system according to an embodiment of the present invention;
fig. 3 is a software architecture block diagram of an underwater three-dimensional reconstruction system provided in an embodiment of the present invention;
FIG. 4 is a hardware connection diagram of a three-dimensional reconstruction system according to another embodiment of the present invention;
fig. 5 is a software architecture block diagram of an underwater three-dimensional reconstruction system according to another embodiment of the present invention;
fig. 6 is a schematic flow chart of an underwater three-dimensional reconstruction method according to an embodiment of the present invention.
Detailed Description
The principles and features of this invention are described below in conjunction with the following drawings, which are set forth to illustrate, but are not to be construed to limit the scope of the invention.
As shown in fig. 1, an underwater three-dimensional reconstruction system provided in an embodiment of the present invention includes: the system comprises an embedded processing platform 11 arranged underwater, a binocular camera 12 arranged underwater and an industrial personal computer 13 arranged on water;
in some embodiments, the underwater three-dimensional reconstruction system hardware, as shown in fig. 2, mainly includes a binocular camera 12, an embedded processing platform 11, a transmission cable, an industrial personal computer 13, a power box 14, and the like; the underwater three-dimensional reconstruction system mainly comprises an underwater part and an overwater part. The underwater part has the main functions of acquiring and coding underwater images, and the overwater part has the main functions of completing power supply of the underwater part, enhancement processing of the underwater images and real-time three-dimensional reconstruction. The transmission line is responsible for connecting the underwater part and the overwater part together to complete network data transmission and power transmission.
In some embodiments, different data and power schemes may be selected for different depths of operation. When the water depth is within 300 meters, adopting a power carrier transmission scheme; when the water depth exceeds 300 meters, the power carrier is difficult to meet the requirements, and a transmission scheme of optical fibers and high-voltage cables is adopted to meet the transmission requirements of data and power supply. In the use process, if the power supply and data interface is compatible with the underwater mobile platform, the transmission line part can directly utilize a transmission cable of the underwater mobile platform carrying the system so as to reduce the redundancy of the system.
The binocular camera 12 is used for collecting underwater images and sending the underwater images to the embedded processing platform 11;
the embedded processing platform 11 is used for compressing and encoding the underwater image and sending the compressed and encoded underwater image to the industrial personal computer 13;
it should be noted that the underwater part is composed of a binocular camera 12 and an embedded processing platform 11. The binocular camera 12 mainly completes the acquisition of underwater images, and adopts a fixed base line, and as the underwater visible distance generally does not exceed 5 meters, the base line of the binocular camera 12 cannot be too large, and is preferably within 10 cm. The binocular camera 12 needs to ensure that the two output camera images are always in synchronization. The binocular camera 12 can support high resolution (1920 × 1080) and high frame rate (30fps) outputs, and is connected to the embedded processing platform 11 through a USB interface. The embedded processing platform 11 mainly completes compression and encoding of binocular images. In some embodiments, the embedded processing platform 11 may be a processor platform with a linux system and a hardware codec accelerator, such as Jetson Nano, Jetson TX2, and the like. The images of the binocular camera 12 are received through the USB interface, and are transmitted to the industrial personal computer 13 for processing through the network interface after being compressed and coded.
The industrial personal computer 13 is used for decoding the underwater image, performing three-dimensional image processing, and performing three-dimensional reconstruction according to the underwater image after the three-dimensional image processing.
It should be noted that the above-water part may be composed of an industrial personal computer 13 and a power box 14. The power box 14 is responsible for power supply of the industrial personal computer 13 and the underwater part. The industrial personal computer 13 is responsible for receiving the coded images uploaded by the underwater part, and processing and three-dimensional reconstruction work of the underwater model are carried out after decoding. Because the image processing and three-dimensional reconstruction have a large amount of calculation, the industrial personal computer 13 needs to install a graphics card with a strong calculation performance, and can adopt Nvidia GPUs with a CUDA core and a calculation capability of more than 7.0, such as GTX 1080Ti, RTX 2080 and the like, so as to meet the requirements of real-time three-dimensional reconstruction.
The underwater three-dimensional reconstruction system can comprise the following functional modules, and the software block diagram of each functional module is shown in fig. 3:
an image acquisition module: operating on an embedded processing platform 11. Parameters of the binocular camera 12, which may include a frame rate, a resolution, etc., are set by the V4L2, and images of the binocular camera 12 are read by the OpenCV through the USB interface and sent to the image compression module.
An image compression module: operating on an embedded processing platform 11. And calling a coding accelerator on the embedded processing platform 11 to realize the coding of the acquired image, wherein the coding adopts an H.264 image compression algorithm to reduce the transmission bandwidth, and the coded video code stream is transmitted to the industrial personal computer 13 through a network interface.
An image decoding module: operating on an industrial control computer 13. And receiving the video code stream through a network interface, and decoding and recovering the binocular image.
An image correction module: operating on an industrial control computer 13. The images of the binocular camera 12 need to be corrected for ease of use during use. The calibration requires calibration of the binocular camera 12, which is performed once after the camera is installed. The calibration can be carried out by adopting a Stereo Camera calibration tool kit of MatLab, the respective internal parameters and lens distortion parameters of the two cameras of the binocular Camera 12 are obtained after the calibration, and the correction of the image of the binocular Camera 12 can be realized through the parameters, so that the same point of the corrected image falls on the same line of the two images, and the rapid Stereo matching is convenient to realize.
An image enhancement module: operating on an industrial control computer 13. Underwater camera imaging can be affected by scattering and attenuated absorption of light. The forward scattering of light can cause the image to have a fogging effect, the backward scattering can cause the image to be blurred, and the light attenuation can cause uneven illumination and color distortion. Therefore, different from binocular vision three-dimensional reconstruction on the ground, the underwater binocular three-dimensional reconstruction needs image enhancement processing on the acquired image so as to reduce the influence on subsequent matching.
According to the characteristics of underwater image imaging, a method combining homomorphic filtering and histogram equalization is adopted to achieve defogging and solve the problem of uneven illumination, and a constrained least square filtering algorithm is adopted to reduce the blurring phenomenon caused by images.
A stereo matching module: operating on an industrial control computer 13. The stereo matching module is used for calculating matched characteristic points of the left camera and the right camera, and after image correction, the same point can be on the same line in the left image and the right image, so that the matching point can be found by searching along the same line. Because a large amount of operations are needed in the stereo matching process, in order to improve the real-time performance of the system, the stereo matching calculation process is accelerated by adopting the CUDA.
A parallax repairing module: operating on an industrial control computer 13. In the stereo matching process, the parallax is difficult to accurately calculate in the area with smooth texture in the image and the area with different shelters in the left image and the right image, and in addition, the parallax image obtained by stereo matching is influenced by noise in the matching process, so that the distortion phenomenon inevitably exists in the parallax image obtained by stereo matching. In order to reduce the error introduced by the part, the parallax image can be repaired by adopting a bilinear interpolation and morphological smoothing mode.
A depth calculation module: operating on an industrial control computer 13. Firstly, selecting feature points from the parallax image, wherein the selection criterion is to select more feature points in a region with larger depth change and select less feature points in a region with smaller depth change. After the feature points are selected, the depth values corresponding to the feature points can be obtained through the triangulation principle.
A three-dimensional mapping module: operating on an industrial control computer 13. After the depth value is calculated, the two-dimensional coordinates of each feature point can be mapped to a three-dimensional space through the coordinates of the feature points in the image and the camera parameters, and a three-dimensional point cloud corresponding to the feature points is obtained.
A data fusion module: operating on an industrial control computer 13. During the working process, the binocular camera 12 needs to be moved continuously, and the generated three-dimensional point cloud also needs to be updated in time along with the new image frame. And (3) taking a coordinate system when the first frame of image is imaged as a reference, transforming the subsequent images to the coordinate system, carrying out interpolation processing on repeated point cloud data, deleting redundancy, and adding new point cloud data.
A triangulation module: operating on an industrial control computer 13. After the data fusion is completed, triangularization subdivision needs to be performed on the obtained three-dimensional point cloud so as to facilitate visualization of a map. The triangulation adopts a local Delaunay grid boundary construction method to connect the three-dimensional point sets into a triangular network, so that the topological structure information of the surface of the object is obtained.
A texture rendering module: operating on an industrial control computer 13. The last step is to perform mapping rendering on the triangular network formed by the three-dimensional point set. The OpenGL library is adopted for map rendering, and each small triangle obtained by triangulation in the previous step can map texture pictures inside the two-dimensional image corresponding to three vertexes of the triangle to the triangle area through the OpenGL library, so that texture rendering of the whole model is achieved, and the visualization effect is improved. The three-dimensional model after texture rendering can be directly displayed on a screen, so that the modeling effect can be conveniently evaluated in real time, and corresponding operation is carried out.
This scheme is through two mesh camera 12 collection images under water, embedded processing platform 11 carries out the compression coding to images under water again, and send the images under water after the compression coding for industrial computer 13, industrial computer 13 is to image decoding under water, three-dimensional image processing, carry out the three-dimensional reconstruction of images under water again, system through this scheme, realize more meticulous scene three-dimensional reconstruction under water, the scheme of sonar formation of image of prior art compares, rebuild three-dimensional resolution ratio has been improved greatly, and there is huge advantage in visualization, and simultaneously, real-time three-dimensional reconstruction can directly see three-dimensional reconstruction's result, make operating personnel can assess the reconstruction effect in real time, the remote operation of being convenient for.
Preferably, in any of the embodiments, the industrial personal computer 13 specifically includes: the device comprises a correction module, a stereo matching module and a depth calculation module;
the correction module is used for correcting the underwater image according to the two camera parameters calibrated by the binocular camera 12;
the stereo matching module is used for calculating a parallax image which is matched with the corrected underwater image and contains the characteristic points by adopting the CUDA;
the depth calculation module is used for selecting the feature points according to a preset rule, obtaining the depth values of the selected feature points through a triangulation principle, and finishing the three-dimensional image processing of the underwater image.
The underwater images are corrected by the image correction module according to the two camera parameters calibrated by the binocular camera 12, so that the rapid stereo matching is conveniently realized; the stereo matching calculation process is accelerated by adopting CUDA (compute unified device architecture), so that the real-time performance of the system and the operation amount in the stereo matching process are improved; more feature points are selected in the area with larger depth change, less feature points are selected in the area with smaller depth change, the accuracy of feature point extraction is improved, and the depth value corresponding to the feature points can be obtained through the triangulation principle.
Preferably, in any of the embodiments, the industrial personal computer 13 further specifically includes: the device comprises an image enhancement module and a parallax restoration module;
the image enhancement module is used for enabling the underwater camera imaging to be influenced by light scattering, attenuation and absorption according to the characteristics of the underwater image imaging. The forward scattering of light can cause the image to have a fogging effect, the backward scattering can cause the image to be blurred, and the light attenuation can cause uneven illumination and color distortion. Therefore, different from binocular vision three-dimensional reconstruction on the ground, the underwater binocular three-dimensional reconstruction needs image enhancement processing on the acquired image so as to reduce the influence on subsequent matching. Carrying out image enhancement on the corrected underwater image by adopting a method combining homomorphic filtering and histogram equalization;
and the parallax repairing module is used for repairing the time difference image by adopting a bilinear interpolation and morphological smoothing mode on the parallax image before the depth calculation.
The method combines homomorphic filtering and histogram equalization through an image enhancement module, then performs image enhancement processing on the acquired image according to a constrained least square filtering algorithm, so as to realize defogging and solve the problem of uneven illumination, reduce the blurring phenomenon caused by the image, and add an image enhancement and recovery method aiming at the imaging characteristics of the underwater image before the stereo matching of the image, thereby improving the accuracy of the stereo matching.
Preferably, in any of the embodiments, the industrial personal computer 13 further includes: the device comprises a three-dimensional mapping module, a data fusion module, a triangulation module and a texture rendering module;
the three-dimensional mapping module is used for mapping the feature points to a three-dimensional space according to the two-dimensional coordinates of the feature points after the depth calculation and the camera parameters to obtain a three-dimensional point cloud;
the data fusion module is used for updating the three-dimensional point cloud correspondingly according to the image frame of the underwater image until the data fusion is completed;
the triangulation module is used for triangulating the three-dimensional point cloud by adopting a local Delaunay grid boundary construction method after data fusion is completed, and connecting the divided three-dimensional point sets to form a triangular network to obtain topological structure information of the object surface of the underwater image;
the texture rendering module is used for performing mapping rendering on the triangular network to complete three-dimensional reconstruction of the underwater image.
Mapping the two-dimensional coordinates of each characteristic point to a three-dimensional space through a three-dimensional mapping module to obtain a three-dimensional point cloud corresponding to the characteristic points, so that the two-dimensional coordinates are mapped to an image of the three-dimensional space; the data fusion module is used for carrying out interpolation processing on repeated point cloud data by taking a coordinate system when the first frame of image is imaged as a reference and converting subsequent images into the coordinate system, and deleting redundancy to realize the elimination of the repeated point cloud data; the triangulation module adopts a local Delaunay grid boundary construction method to connect the three-dimensional point sets into a triangular network to obtain topological structure information of the surface of the object so as to facilitate visual mapping; the texture rendering module maps texture pictures inside the two-dimensional image corresponding to the three vertexes of the triangle to the triangular area through an OpenGL library, so that texture rendering of the whole model is achieved, and the visualization effect is improved.
Preferably, in any of the above embodiments, the industrial personal computer 13 further comprises a GPU having a CUDA core and a computing power above 7.0.
The operation is accelerated by adding the high-performance GPU, so that the real-time performance of the whole system is improved, and the operation of personnel is facilitated.
In other embodiments provided by the present invention, as shown in fig. 4, the embedded processing platform 11 may be replaced by a high performance computing platform with a CUDA core, such as Nvidia Xavier, and the industrial personal computer 13 will not need to be configured with a high performance computing graphics card. The main image processing and three-dimensional reconstruction tasks are completed on the embedded processing platform 11, and the industrial personal computer 13 mainly completes the tasks of image decoding and display.
The software block diagram of the underwater three-dimensional reconstruction system is shown in fig. 5, and software functional modules from image acquisition to texture rendering work in an embedded processing platform 11. After texture rendering is completed, the interface image is compressed and encoded, and then transmitted to the industrial personal computer 13 through the transmission line, and then decoded and displayed by the industrial personal computer 13.
Compared with the previous scheme, the scheme has the advantages that the influence of image compression on detail loss caused by stereo matching is avoided, meanwhile, the dependence on the industrial personal computer 13 is reduced, the industrial personal computer can be conveniently integrated with an upper computer of an underwater mobile platform, the defects that operation processing is concentrated on the embedded processing platform 11, the performance of the embedded processing platform 11 is weak, and the real-time performance of the system can be reduced.
In another embodiment provided by the present invention, as shown in fig. 6, an underwater three-dimensional reconstruction method includes: s1, collecting underwater images and sending the underwater images to the embedded processing platform 11;
s2, compressing and encoding the underwater image, and sending the compressed and encoded underwater image to the industrial personal computer 13;
and S3, decoding the underwater image, processing the stereo image, and performing three-dimensional reconstruction according to the underwater image processed by the stereo image.
This scheme is through two mesh camera 12 collection images under water, embedded processing platform 11 carries out the compression coding to images under water again, and send the images under water after the compression coding for industrial computer 13, industrial computer 13 is to image decoding under water, three-dimensional image processing, carry out the three-dimensional reconstruction of images under water again, system through this scheme, realize more meticulous scene three-dimensional reconstruction under water, the scheme of sonar formation of image of prior art compares, rebuild three-dimensional resolution ratio has been improved greatly, and there is huge advantage in visualization, and simultaneously, real-time three-dimensional reconstruction can directly see three-dimensional reconstruction's result, make operating personnel can assess the reconstruction effect in real time, the remote operation of being convenient for.
Preferably, in any of the above embodiments, the image processing specifically includes:
correcting the underwater image according to the two camera parameters calibrated by the binocular camera 12;
calculating a parallax image which is matched with the corrected underwater image and contains the characteristic points by using the CUDA;
and selecting the feature points according to a preset rule, and obtaining the depth values of the selected feature points through a triangulation principle to finish the three-dimensional image processing of the underwater image.
The underwater images are corrected by the image correction module according to the two camera parameters calibrated by the binocular camera 12, so that the rapid stereo matching is conveniently realized; the stereo matching calculation process is accelerated by adopting CUDA (compute unified device architecture), so that the real-time performance of the system and the operation amount in the stereo matching process are improved; more feature points are selected in the area with larger depth change, less feature points are selected in the area with smaller depth change, the accuracy of feature point extraction is improved, and the depth value corresponding to the feature points can be obtained through the triangulation principle.
Preferably, in any of the above embodiments, the image processing further specifically includes:
according to the characteristics of underwater image imaging, image enhancement is carried out on the corrected underwater image by adopting a method of combining homomorphic filtering and histogram equalization;
and repairing the time difference image by adopting a bilinear interpolation and morphological smoothing mode on the parallax image before the depth calculation.
The method combines homomorphic filtering and histogram equalization through an image enhancement module, then performs image enhancement processing on the acquired image according to a constrained least square filtering algorithm, so as to realize defogging and solve the problem of uneven illumination, reduce the blurring phenomenon caused by the image, and add an image enhancement and recovery method aiming at the imaging characteristics of the underwater image before the stereo matching of the image, thereby improving the accuracy of the stereo matching.
Preferably, in any of the above embodiments, the three-dimensional reconstruction specifically includes:
mapping the feature points to a three-dimensional space according to the two-dimensional coordinates of the feature points and the camera parameters after the depth calculation to obtain three-dimensional point cloud;
updating the three-dimensional point cloud correspondingly according to the image frame update of the underwater image until the data fusion is completed;
after data fusion is completed, triangularization subdivision is carried out on the three-dimensional point cloud by adopting a local Delaunay grid boundary construction method, the divided three-dimensional point sets are connected to form a triangular network, and topological structure information of the object surface of the underwater image is obtained;
and (4) performing mapping rendering on the triangular network to complete three-dimensional reconstruction of the underwater image.
Mapping the two-dimensional coordinates of each characteristic point to a three-dimensional space through a three-dimensional mapping module to obtain a three-dimensional point cloud corresponding to the characteristic points, so that the two-dimensional coordinates are mapped to an image of the three-dimensional space; the data fusion module is used for carrying out interpolation processing on repeated point cloud data by taking a coordinate system when the first frame of image is imaged as a reference and converting subsequent images into the coordinate system, and deleting redundancy to realize the elimination of the repeated point cloud data; the triangulation module adopts a local Delaunay grid boundary construction method to connect the three-dimensional point sets into a triangular network to obtain topological structure information of the surface of the object so as to facilitate visual mapping; the texture rendering module maps texture pictures inside the two-dimensional image corresponding to the three vertexes of the triangle to the triangular area through an OpenGL library, so that texture rendering of the whole model is achieved, and the visualization effect is improved.
Preferably, in any of the above embodiments, the industrial personal computer 13 further comprises a GPU having a CUDA core and a computing power above 7.0.
The operation is accelerated by adding the high-performance GPU, so that the real-time performance of the whole system is improved, and the operation of personnel is facilitated.
It is understood that some or all of the alternative embodiments described above may be included in some embodiments.
It should be noted that the above embodiments are product embodiments corresponding to the previous method embodiments, and for the description of each optional implementation in the product embodiments, reference may be made to corresponding descriptions in the above method embodiments, and details are not described here again.
The reader should understand that in the description of this specification, reference to the description of the terms "one embodiment," "some embodiments," "an example," "a specific example," or "some examples," etc., means that a particular feature, structure, material, or characteristic described in connection with the embodiment or example is included in at least one embodiment or example of the invention. In this specification, the schematic representations of the terms used above are not necessarily intended to refer to the same embodiment or example. Furthermore, the particular features, structures, materials, or characteristics described may be combined in any suitable manner in any one or more embodiments or examples. Furthermore, various embodiments or examples and features of different embodiments or examples described in this specification can be combined and combined by one skilled in the art without contradiction.
In the several embodiments provided in the present application, it should be understood that the disclosed apparatus and method may be implemented in other ways. For example, the above-described method embodiments are merely illustrative, and for example, the division of steps into only one logical functional division may be implemented in practice in another way, for example, multiple steps may be combined or integrated into another step, or some features may be omitted, or not implemented.
The above method, if implemented in the form of software functional units and sold or used as a stand-alone product, may be stored in a computer readable storage medium. Based on such understanding, the technical solution of the present invention essentially or partially contributes to the prior art, or all or part of the technical solution can be embodied in the form of a software product stored in a storage medium and including instructions for causing a computer device (which may be a personal computer, a server, or a network device) to execute all or part of the steps of the method according to the embodiments of the present invention. And the aforementioned storage medium includes: various media capable of storing program codes, such as a usb disk, a removable hard disk, a Read-only memory (ROM), a Random Access Memory (RAM), a magnetic disk, or an optical disk.
While the invention has been described with reference to specific embodiments, it will be understood by those skilled in the art that various changes in form and details may be made therein without departing from the spirit and scope of the invention as defined by the appended claims. Therefore, the protection scope of the present invention shall be subject to the protection scope of the claims.
Claims (10)
1. An underwater three-dimensional reconstruction system, comprising: the system comprises an embedded processing platform arranged underwater, a binocular camera arranged underwater and an industrial personal computer arranged on water;
the binocular camera is used for acquiring underwater images and sending the underwater images to the embedded processing platform;
the embedded processing platform is used for compressing and coding the underwater image and sending the compressed and coded underwater image to the industrial personal computer;
the industrial personal computer is used for decoding the underwater image, processing the stereo image and then performing three-dimensional reconstruction according to the underwater image processed by the stereo image.
2. The underwater three-dimensional reconstruction system according to claim 1, wherein the industrial personal computer specifically comprises: the device comprises a correction module, a stereo matching module and a depth calculation module;
the correction module is used for correcting the underwater image according to the two camera parameters calibrated by the binocular camera;
the stereo matching module is used for calculating a parallax image which is matched with the corrected underwater image and contains the characteristic points by using the CUDA;
the depth calculation module is used for selecting the feature points according to a preset rule, obtaining the depth values of the selected feature points through a triangulation principle, and finishing the three-dimensional image processing of the underwater image.
3. The underwater three-dimensional reconstruction system according to claim 2, wherein the industrial personal computer further comprises: the device comprises an image enhancement module and a parallax restoration module;
the image enhancement module is used for carrying out image enhancement on the corrected underwater image by adopting a method of combining homomorphic filtering and histogram equalization according to the characteristics of underwater image imaging;
and the parallax repairing module is used for repairing the time difference image by adopting a bilinear interpolation and morphological smoothing mode on the parallax image before the depth calculation.
4. The underwater three-dimensional reconstruction system according to claim 2 or 3, wherein the industrial personal computer further comprises: the device comprises a three-dimensional mapping module, a data fusion module, a triangulation module and a texture rendering module;
the three-dimensional mapping module is used for mapping the feature points to a three-dimensional space according to the two-dimensional coordinates of the feature points and the camera parameters after the depth calculation to obtain a three-dimensional point cloud;
the data fusion module is used for updating the three-dimensional point cloud correspondingly according to the image frame of the underwater image until the data fusion is completed;
the triangulation module is used for triangulating the three-dimensional point cloud by adopting a local Delaunay grid boundary construction method after data fusion is completed, and connecting the divided three-dimensional point sets to form a triangular network so as to obtain topological structure information of the object surface of the underwater image;
the texture rendering module is used for performing mapping rendering on the triangular network to complete three-dimensional reconstruction of the underwater image.
5. The underwater three-dimensional reconstruction system of claim 4, wherein the industrial personal computer further comprises a GPU having a CUDA core and a computing power of 7.0 or more.
6. An underwater three-dimensional reconstruction method, comprising:
s1, collecting underwater images and sending the underwater images to the embedded processing platform;
s2, carrying out compression coding on the underwater image, and sending the compressed and coded underwater image to the industrial personal computer;
and S3, decoding the underwater image, processing the stereo image, and performing three-dimensional reconstruction according to the underwater image processed by the stereo image.
7. The underwater three-dimensional reconstruction method according to claim 6, wherein the image processing specifically comprises:
correcting the underwater image according to the two camera parameters calibrated by the binocular camera;
calculating a parallax image which is matched with the corrected underwater image and contains the characteristic points by using the CUDA;
and selecting the feature points according to a preset rule, and obtaining the depth values of the selected feature points through a triangulation principle to finish the three-dimensional image processing of the underwater image.
8. The underwater three-dimensional reconstruction method according to claim 7, wherein the image processing further specifically comprises:
according to the characteristics of underwater image imaging, image enhancement is carried out on the corrected underwater image by adopting a method of combining homomorphic filtering and histogram equalization;
and repairing the time difference image by adopting a bilinear interpolation and morphological smoothing mode on the parallax image before the depth calculation.
9. The underwater three-dimensional reconstruction method according to claim 7 or 8, wherein the three-dimensional reconstruction specifically comprises:
mapping the feature points to a three-dimensional space according to the two-dimensional coordinates of the feature points and the camera parameters after the depth calculation to obtain three-dimensional point cloud;
updating the three-dimensional point cloud correspondingly according to the image frame update of the underwater image until data fusion is completed;
after data fusion is completed, triangularization subdivision is carried out on the three-dimensional point cloud by adopting a local Delaunay grid boundary construction method, the divided three-dimensional point sets are connected to form a triangular network, and topological structure information of the object surface of the underwater image is obtained;
and performing mapping rendering on the triangular network to finish the three-dimensional reconstruction of the underwater image.
10. The underwater three-dimensional reconstruction method according to claim 9, wherein the industrial personal computer further comprises a GPU having a CUDA core and a computational capability of 7.0 or more.
Priority Applications (1)
| Application Number | Priority Date | Filing Date | Title |
|---|---|---|---|
| CN201911369346.8A CN111161407A (en) | 2019-12-26 | 2019-12-26 | Underwater three-dimensional reconstruction system and method |
Applications Claiming Priority (1)
| Application Number | Priority Date | Filing Date | Title |
|---|---|---|---|
| CN201911369346.8A CN111161407A (en) | 2019-12-26 | 2019-12-26 | Underwater three-dimensional reconstruction system and method |
Publications (1)
| Publication Number | Publication Date |
|---|---|
| CN111161407A true CN111161407A (en) | 2020-05-15 |
Family
ID=70558321
Family Applications (1)
| Application Number | Title | Priority Date | Filing Date |
|---|---|---|---|
| CN201911369346.8A Withdrawn CN111161407A (en) | 2019-12-26 | 2019-12-26 | Underwater three-dimensional reconstruction system and method |
Country Status (1)
| Country | Link |
|---|---|
| CN (1) | CN111161407A (en) |
Cited By (3)
| Publication number | Priority date | Publication date | Assignee | Title |
|---|---|---|---|---|
| CN111990314A (en) * | 2020-08-25 | 2020-11-27 | 中国水产科学研究院渔业机械仪器研究所 | System and method for quantitative observation of fish behaviors |
| CN113507557A (en) * | 2021-09-06 | 2021-10-15 | 大连理工江苏研究院有限公司 | Underwater image transmission and processing system and method |
| CN114923665A (en) * | 2022-05-27 | 2022-08-19 | 上海交通大学 | Image reconstruction method and image reconstruction test system for wave three-dimensional height field |
-
2019
- 2019-12-26 CN CN201911369346.8A patent/CN111161407A/en not_active Withdrawn
Cited By (4)
| Publication number | Priority date | Publication date | Assignee | Title |
|---|---|---|---|---|
| CN111990314A (en) * | 2020-08-25 | 2020-11-27 | 中国水产科学研究院渔业机械仪器研究所 | System and method for quantitative observation of fish behaviors |
| CN113507557A (en) * | 2021-09-06 | 2021-10-15 | 大连理工江苏研究院有限公司 | Underwater image transmission and processing system and method |
| CN113507557B (en) * | 2021-09-06 | 2021-12-10 | 大连理工江苏研究院有限公司 | Underwater image transmission and processing system and method |
| CN114923665A (en) * | 2022-05-27 | 2022-08-19 | 上海交通大学 | Image reconstruction method and image reconstruction test system for wave three-dimensional height field |
Similar Documents
| Publication | Publication Date | Title |
|---|---|---|
| CN110335343B (en) | Human body three-dimensional reconstruction method and device based on RGBD single-view-angle image | |
| JP4052331B2 (en) | Virtual viewpoint image generation method, three-dimensional image display method and apparatus | |
| US8860712B2 (en) | System and method for processing video images | |
| US7102683B2 (en) | Single lens 3D camera | |
| US6862024B2 (en) | Enhancing textured range images using a 2D editor | |
| CN111161407A (en) | Underwater three-dimensional reconstruction system and method | |
| US20120032948A1 (en) | System and method for processing video images for camera recreation | |
| CN108648264B (en) | Underwater scene reconstruction method based on motion recovery and storage medium | |
| CN103530907B (en) | Complicated three-dimensional model drawing method based on images | |
| US6963671B2 (en) | Method for determining distances to a surface from a range image | |
| CN111612878A (en) | Method and device for making static photo into three-dimensional effect video | |
| CN114863038A (en) | Real-time dynamic free visual angle synthesis method and device based on explicit geometric deformation | |
| CN115222889A (en) | 3D reconstruction method and device based on multi-view image and related equipment | |
| CN113379901A (en) | Method and system for establishing house live-action three-dimension by utilizing public self-photographing panoramic data | |
| Bleyer et al. | Temporally consistent disparity maps from uncalibrated stereo videos | |
| CN109461197B (en) | Cloud real-time drawing optimization method based on spherical UV and re-projection | |
| US6792205B2 (en) | Method for generating a textured range image | |
| CN115100337A (en) | Whole body portrait video relighting method and device based on convolutional neural network | |
| CN115359173A (en) | Virtual multi-view video generation method and device, electronic equipment and storage medium | |
| CN116342804A (en) | Outdoor scene three-dimensional reconstruction method and device, electronic equipment and storage medium | |
| Gu et al. | Ue4-nerf: Neural radiance field for real-time rendering of large-scale scene | |
| US20240087231A1 (en) | Method, apparatus, computer device and storage medium for three-dimensional reconstruction of indoor structure | |
| CN111179210B (en) | Face texture map generation method and system and electronic equipment | |
| CN112102504A (en) | Three-dimensional scene and two-dimensional image mixing method based on mixed reality | |
| CN116385577A (en) | Virtual viewpoint image generation method and device |
Legal Events
| Date | Code | Title | Description |
|---|---|---|---|
| PB01 | Publication | ||
| PB01 | Publication | ||
| SE01 | Entry into force of request for substantive examination | ||
| SE01 | Entry into force of request for substantive examination | ||
| WW01 | Invention patent application withdrawn after publication |
Application publication date: 20200515 |
|
| WW01 | Invention patent application withdrawn after publication |