CN115330846A - Three-dimensional digital holography-based pavement disease image fusion data processing method - Google Patents
Three-dimensional digital holography-based pavement disease image fusion data processing method Download PDFInfo
- Publication number
- CN115330846A CN115330846A CN202210983139.7A CN202210983139A CN115330846A CN 115330846 A CN115330846 A CN 115330846A CN 202210983139 A CN202210983139 A CN 202210983139A CN 115330846 A CN115330846 A CN 115330846A
- Authority
- CN
- China
- Prior art keywords
- disease
- dimensional
- data
- fusion
- image
- Prior art date
- Legal status (The legal status is an assumption and is not a legal conclusion. Google has not performed a legal analysis and makes no representation as to the accuracy of the status listed.)
- Pending
Links
- 201000010099 disease Diseases 0.000 title claims abstract description 152
- 208000037265 diseases, disorders, signs and symptoms Diseases 0.000 title claims abstract description 152
- 230000004927 fusion Effects 0.000 title claims abstract description 52
- 238000001093 holography Methods 0.000 title claims abstract description 12
- 238000003672 processing method Methods 0.000 title claims abstract description 12
- 238000012549 training Methods 0.000 claims abstract description 27
- 238000000034 method Methods 0.000 claims abstract description 25
- 238000012360 testing method Methods 0.000 claims abstract description 17
- 238000012937 correction Methods 0.000 claims abstract description 3
- 238000004088 simulation Methods 0.000 claims abstract description 3
- 239000011159 matrix material Substances 0.000 claims description 53
- 230000009466 transformation Effects 0.000 claims description 21
- 230000009467 reduction Effects 0.000 claims description 8
- 230000008859 change Effects 0.000 claims description 6
- 238000012545 processing Methods 0.000 claims description 6
- 238000012795 verification Methods 0.000 claims description 6
- 238000003062 neural network model Methods 0.000 claims description 5
- 238000007781 pre-processing Methods 0.000 claims description 4
- 230000008569 process Effects 0.000 claims description 4
- 238000012935 Averaging Methods 0.000 claims description 3
- 230000003190 augmentative effect Effects 0.000 claims description 3
- 238000006243 chemical reaction Methods 0.000 claims description 3
- 210000001503 joint Anatomy 0.000 claims description 3
- 238000010606 normalization Methods 0.000 claims description 3
- 230000003287 optical effect Effects 0.000 claims description 3
- 238000005457 optimization Methods 0.000 claims description 3
- 238000011946 reduction process Methods 0.000 claims description 3
- 230000011218 segmentation Effects 0.000 claims description 3
- 238000007499 fusion processing Methods 0.000 claims description 2
- 238000001514 detection method Methods 0.000 abstract description 10
- 230000007547 defect Effects 0.000 description 4
- 238000000605 extraction Methods 0.000 description 2
- 238000003703 image analysis method Methods 0.000 description 2
- 238000013135 deep learning Methods 0.000 description 1
- 238000012423 maintenance Methods 0.000 description 1
- 238000005259 measurement Methods 0.000 description 1
- 238000012986 modification Methods 0.000 description 1
- 230000004048 modification Effects 0.000 description 1
- 238000004445 quantitative analysis Methods 0.000 description 1
- 238000011160 research Methods 0.000 description 1
- 230000001360 synchronised effect Effects 0.000 description 1
Images
Classifications
-
- G—PHYSICS
- G06—COMPUTING; CALCULATING OR COUNTING
- G06T—IMAGE DATA PROCESSING OR GENERATION, IN GENERAL
- G06T7/00—Image analysis
- G06T7/50—Depth or shape recovery
- G06T7/521—Depth or shape recovery from laser ranging, e.g. using interferometry; from the projection of structured light
-
- G—PHYSICS
- G06—COMPUTING; CALCULATING OR COUNTING
- G06T—IMAGE DATA PROCESSING OR GENERATION, IN GENERAL
- G06T3/00—Geometric image transformations in the plane of the image
- G06T3/40—Scaling of whole images or parts thereof, e.g. expanding or contracting
- G06T3/4038—Image mosaicing, e.g. composing plane images from plane sub-images
-
- G—PHYSICS
- G06—COMPUTING; CALCULATING OR COUNTING
- G06T—IMAGE DATA PROCESSING OR GENERATION, IN GENERAL
- G06T5/00—Image enhancement or restoration
- G06T5/70—Denoising; Smoothing
-
- G—PHYSICS
- G06—COMPUTING; CALCULATING OR COUNTING
- G06V—IMAGE OR VIDEO RECOGNITION OR UNDERSTANDING
- G06V10/00—Arrangements for image or video recognition or understanding
- G06V10/20—Image preprocessing
- G06V10/30—Noise filtering
-
- G—PHYSICS
- G06—COMPUTING; CALCULATING OR COUNTING
- G06V—IMAGE OR VIDEO RECOGNITION OR UNDERSTANDING
- G06V10/00—Arrangements for image or video recognition or understanding
- G06V10/70—Arrangements for image or video recognition or understanding using pattern recognition or machine learning
- G06V10/77—Processing image or video features in feature spaces; using data integration or data reduction, e.g. principal component analysis [PCA] or independent component analysis [ICA] or self-organising maps [SOM]; Blind source separation
- G06V10/80—Fusion, i.e. combining data from various sources at the sensor level, preprocessing level, feature extraction level or classification level
- G06V10/806—Fusion, i.e. combining data from various sources at the sensor level, preprocessing level, feature extraction level or classification level of extracted features
-
- G—PHYSICS
- G06—COMPUTING; CALCULATING OR COUNTING
- G06T—IMAGE DATA PROCESSING OR GENERATION, IN GENERAL
- G06T2200/00—Indexing scheme for image data processing or generation, in general
- G06T2200/32—Indexing scheme for image data processing or generation, in general involving image mosaicing
-
- G—PHYSICS
- G06—COMPUTING; CALCULATING OR COUNTING
- G06T—IMAGE DATA PROCESSING OR GENERATION, IN GENERAL
- G06T2207/00—Indexing scheme for image analysis or image enhancement
- G06T2207/10—Image acquisition modality
- G06T2207/10004—Still image; Photographic image
-
- G—PHYSICS
- G06—COMPUTING; CALCULATING OR COUNTING
- G06T—IMAGE DATA PROCESSING OR GENERATION, IN GENERAL
- G06T2207/00—Indexing scheme for image analysis or image enhancement
- G06T2207/10—Image acquisition modality
- G06T2207/10028—Range image; Depth image; 3D point clouds
-
- G—PHYSICS
- G06—COMPUTING; CALCULATING OR COUNTING
- G06T—IMAGE DATA PROCESSING OR GENERATION, IN GENERAL
- G06T2207/00—Indexing scheme for image analysis or image enhancement
- G06T2207/20—Special algorithmic details
- G06T2207/20212—Image combination
- G06T2207/20221—Image fusion; Image merging
-
- G—PHYSICS
- G06—COMPUTING; CALCULATING OR COUNTING
- G06T—IMAGE DATA PROCESSING OR GENERATION, IN GENERAL
- G06T2207/00—Indexing scheme for image analysis or image enhancement
- G06T2207/30—Subject of image; Context of image processing
- G06T2207/30248—Vehicle exterior or interior
- G06T2207/30252—Vehicle exterior; Vicinity of vehicle
Landscapes
- Engineering & Computer Science (AREA)
- Theoretical Computer Science (AREA)
- Physics & Mathematics (AREA)
- General Physics & Mathematics (AREA)
- Computer Vision & Pattern Recognition (AREA)
- Multimedia (AREA)
- Computing Systems (AREA)
- General Health & Medical Sciences (AREA)
- Medical Informatics (AREA)
- Software Systems (AREA)
- Evolutionary Computation (AREA)
- Databases & Information Systems (AREA)
- Artificial Intelligence (AREA)
- Health & Medical Sciences (AREA)
- Optics & Photonics (AREA)
- Image Analysis (AREA)
Abstract
The invention discloses a data processing method for three-dimensional digital holography-based pavement disease image fusion, which comprises the steps of generating a digital interference fringe image through a laser interferometer, projecting the digital interference fringe image onto a pavement, and collecting the pavement to obtain an original image; simulating, reducing and reconstructing by using a computer to obtain a three-dimensional holographic disease map; obtaining a two-dimensional disease map by adopting Fourier transform; fusing the three-dimensional holographic disease image and the two-dimensional disease image to generate a fused disease image; and carrying out test simulation correction on the fusion model by adopting the specified data to obtain a final training model. According to the method, the three-dimensional disease graph and the two-dimensional disease graph can be obtained according to the original hologram, the left disease graph, the right disease graph, the front disease graph and the rear disease graph are spliced to form the multi-size disease database through the least square method, the data accuracy is greatly improved through the combination of the two-dimensional disease graph data and the three-dimensional disease graph data, the image quality is improved, the disease detection types cover various diseases of the pavement, and the pavement diseases can be conveniently and automatically identified by equipment.
Description
Technical Field
The invention relates to the field of pavement maintenance data processing, in particular to a data processing method for pavement disease image fusion based on three-dimensional digital holography.
Background
Since the first pavement rapid detection system in the 70 s of the 20 th century appeared, many scholars and researchers at home and abroad began to focus on the research on pavement disease image processing and automatic identification. The method generally comprises the steps of firstly carrying out preprocessing work such as image enhancement and the like on a two-dimensional road image acquired by detection equipment, then segmenting and extracting target features through various methods such as a threshold value and the like, and finally identifying and classifying. The image analysis method can make specific quantitative analysis on the segmented disease target, but as the pavement disease image data is used as a nonlinear detection target, the image analysis method mainly faces the following challenges: the background is complex and changeable, the speckle noise is strong, the signal-to-noise ratio of the target is low, the contrast between the target and the background is low, and the spatial continuity of the target pixel is poor. Although researchers have proposed many different treatment methods, there is no general effective method for automatically identifying various pavement diseases (cracks and deformations), and at present, the method is mainly based on manual identification of disease images, and the labor cost is high and the efficiency is low. The three-dimensional point cloud data generated by the three-dimensional laser detection technology is huge, and intelligent identification is difficult to achieve. When the existing deep learning network is used for training, due to the problems of disease data volume and precision, unification is difficult to achieve through multi-data fusion.
Therefore, how to combine with a new technology to solve the problems of quality, precision and automatic identification of pavement diseases and image data, and establishing a multi-size disease database has become a technical problem which needs to be solved urgently.
Disclosure of Invention
Aiming at the problems, the invention aims to provide a data processing method for road surface disease image fusion based on three-dimensional digital holography.
In order to realize the technical purpose, the scheme of the invention is as follows: the data processing method for the pavement disease image fusion based on the three-dimensional digital holography comprises the following specific steps:
s1, generating a digital interference fringe image through a laser interferometer in three-dimensional digital holographic vehicle-mounted acquisition equipment, projecting the digital interference fringe image onto a road surface, acquiring the road surface by using at least two industrial cameras to obtain an original image, and cutting the original image according to an effective acquisition range of the industrial cameras to obtain an original hologram;
s2, simulating an optical diffraction process by using a computer to realize holographic reconstruction and processing reduction reconstruction of the pavement to obtain a three-dimensional holographic disease map with pavement depth data;
s3, removing interference fringe information from the original hologram by adopting Fourier multi-level matrix transformation and Fourier multi-level matrix inverse transformation, thereby obtaining a two-dimensional disease map;
s4, performing homologous data fusion on the three-dimensional holographic disease image and the two-dimensional disease image generated by the same original hologram to generate a fused disease image;
s5, respectively carrying out multi-size fusion on the three-dimensional holographic disease images and the two-dimensional disease images obtained by the at least two industrial cameras by adopting a least square fitting method to obtain multi-size pavement data sets; and carrying out noise reduction treatment on the small-size pavement data generated in the first step, and cutting off edge noise of each edge.
S6: and performing data fusion on the multi-size pavement data set to generate a three-dimensional holographic disease database, a two-dimensional disease database and a fusion disease database, substituting the built neural network model for training to obtain a fusion model, and performing test simulation correction on the fusion model by adopting specified data to obtain a final training model.
Preferably, in step S2, the original hologram includes light intensity information of the road surface with digital interference fringes and phase information of light waves, three-dimensional reconstruction is performed by using fast fourier transform according to a relationship between the light intensity and an electromagnetic equation to obtain a phase difference, and the whole depth information of the original hologram of the road surface is restored by calculating continuous phase difference changes, so as to obtain a three-dimensional holographic disease map with road surface depth data;
the specific digital holographic three-dimensional reduction process comprises the following steps:
s21, the electromagnetic energy equation of any pixel point (x, y) in the original hologram is as follows:
E(x，y)＝A(x，y)exp(iφ(x，y))
wherein phi (x, y) is phase information of any pixel point (x, y), and A (x, y) is light intensity information of any pixel point (x, y);
s22, the light intensity equation of any pixel point (x, y) of the original hologram is as follows:
I(x，y)＝A(x，y) 2 ＝I(x，y)＝|E(x，y)| 2 ＝E(x，y)E(x，y) * ；
s23, a Fresnel digital diffraction transformation formula:
wherein: Γ (ξ, β) is the light intensity and phase of the original hologram, λ is the wavelength of the light source, i represents complex number, d is the perpendicular distance from the laser interferometer to the camera aperture plane, ρ is the diffraction distance from the camera aperture to the laser interferometer plane;
s24, fourier transformation is equivalent to Fresnel change, three-dimensional holographic phase information of any pixel point (x, y) of the original hologram is obtained through formula conversion, the phase difference of the whole original hologram is obtained through continuous Fourier change, namely the depth information of all pixel points of the original hologram, and therefore the three-dimensional holographic disease map of the pavement is obtained through three-dimensional reduction:
I(x，y)＝|Γ(ξ，β)| 2
preferably, in step S3, the road information of the original hologram itself is obtained, and a fourier multilevel formula of light intensity a (x, y) of any pixel point (x, y) in the original hologram:
according to a Fourier multi-stage formula, deleting zero order and a order of Fourier in the Fourier multi-stage formula 0 +a 1 (x，y) 1 Carrying out Fourier multi-stage inverse transformation on the data to obtain any pixel point (x, y) two-dimensional pixel in the original hologram for eliminating interference fringe information;
and eliminating digital interference fringes by performing Fourier multilevel matrix transformation and Fourier multilevel matrix inverse transformation on all pixel points of the original hologram to obtain a two-dimensional road surface disease image.
Preferably, in step S4, the three-dimensional hologram defect map and the two-dimensional hologram defect map generated by the same original hologram are subjected to pixel-level data fusion;
the specific data fusion process is as follows: the image of the original hologram is represented as matrix data, a three-dimensional holographic disease map generation matrix A and a two-dimensional disease map generation matrix B, the matrixes can be represented as superposition of a low-rank matrix and a sparse matrix under an optimization criterion, a matrix C is obtained by respectively adding the matrix A and the matrix B, an augmented Lagrange multiplier method is carried out on the matrix C to solve a low-rank matrix E sparse matrix F, the matrix E and the matrix F are respectively superposed and converted to generate a graph, and finally the fused disease map is obtained.
Preferably, in step S5, the three-dimensional holographic disease image, the two-dimensional disease image and the fused disease image acquired by different industrial cameras are stitched to obtain a road surface image;
obtaining an optimal fitting plane ax + by + cz + d =0 of the road surface map to be spliced by adopting a least square method, rotating the road surface map of the other industrial camera by taking the fitting plane of one industrial camera as a reference, enabling the road surface maps on the left side and the right side to be completely positioned on the same reference plane, and carrying out seamless splicing to generate a left spliced disease map and a right spliced disease map;
and in the same way, the left and right spliced disease images of two different industrial cameras are spliced in front and back, the deviation angle of two plane spaces is obtained according to the space data of the fitting plane, the left and right spliced images in the front are selected as reference planes, the images to be spliced are subjected to three-dimensional space rotation, the three-dimensional data matrix to be spliced is subjected to space butt joint, seamless splicing is achieved, and the front and back splicing can generate the multi-size pavement map.
Preferably, in step S6, the three-dimensional hologram defect map and the two-dimensional defect map with different sizes are respectively subjected to feature engineering;
firstly, marking partial data, preprocessing the data through normalization and binarization to generate a three-dimensional holographic disease database, a two-dimensional disease database, a three-dimensional hologram and a fusion disease database, carrying out data segmentation on the three databases to generate three corresponding training sets, verification sets and test sets, respectively substituting the three training sets, the verification sets and the test sets into a built neural network model to carry out iterative training to obtain three training models, respectively obtaining two fusion models through a training model weighted voting fusion mode and a averaging fusion mode, importing the fusion models, testing the three test sets, and selecting a result model meeting requirements to export the training models.
The method has the advantages that the digital holographic vehicle-mounted detection equipment provides a large-scale pavement detection rapid technology, the three-dimensional disease graph and the two-dimensional disease graph are obtained after interference fringes are removed through reconstruction according to the collected original hologram, the left and right and front and rear disease graphs are spliced to form the multi-size disease database through the least square method, the data accuracy is greatly improved through the combination of the two-dimensional disease graph data and the three-dimensional disease graph data, the image quality is improved, various diseases (cracks, deformation and the like) of the pavement are covered by the disease detection types, and the pavement diseases can be automatically identified conveniently by the equipment.
Drawings
In order to more clearly illustrate the embodiments of the present invention or the technical solutions in the prior art, the drawings used in the description of the embodiments or the prior art will be briefly introduced below, and it is obvious that the drawings in the following description are some embodiments of the present invention, and for those skilled in the art, other drawings can be obtained according to these drawings without creative efforts.
FIG. 1 is a flow chart of the present application;
FIG. 2 is a road surface original hologram (2*1 m) collected by a single industrial camera in the present application;
FIG. 3 is a three-dimensional holographic disease map (2*1 meters) reduced by a single industrial camera in the present application;
FIG. 4 is a two-dimensional disease map of a single industrial camera de-striping in the present application (2*1 meters);
FIG. 5 is a single industrial camera fusion disease map (2*1 meters) in the present application;
FIG. 6 is a multi-size mosaic of left and right industrial camera three-dimensional holograms in the present application (4*1 meters);
FIG. 7 is a multi-scale mosaic of left and right industrial camera fusion disease maps (4*1 meters) in the present application;
FIG. 8 is a multi-size mosaic of a two-dimensional disease map for striping of left and right industrial cameras (4*1 meters) in the present application;
fig. 9 is a three-dimensional holographic disease map mosaic of the left and right industrial cameras (4 × 10 m) in the present application;
fig. 10 is a three-dimensional holographic data mosaic display of the left and right industrial cameras of the present application (4 x 10 meters);
fig. 11 is a three-dimensional point cloud data mosaic display of the left and right industrial cameras of the present application (4 × 10 m);
fig. 12 shows the disease identification result in the present application.
Detailed Description
To make the objects, technical solutions and advantages of the embodiments of the present invention clearer, the technical solutions of the embodiments of the present invention will be described clearly and completely with reference to fig. 1 to 12 in the embodiments of the present invention, and it is obvious that the described embodiments are some, but not all, embodiments of the present invention. All other embodiments, which can be derived by a person skilled in the art from the embodiments given herein without making any creative effort, shall fall within the protection scope of the present invention.
In order to better explain the working process of the present invention, the following description explains the principle of implementing the present invention.
The two-dimensional image splicing is easily influenced by ambient light and the characteristics of a shot road surface, the splicing seams of the images are obvious, and the brightness of the spliced images is uneven; in addition, the two-dimensional image has no depth information, and the deformation diseases of the road surface are difficult to judge: pit, sink, hug, etc. The common method for three-dimensional point cloud splicing comprises the following steps: a splicing method relying on external feature point auxiliary information and a splicing method relying on repeated measurement of overlapped data are not suitable for pavement diseases, and rapid detection of vehicle-mounted equipment is needed for pavement disease detection; in addition, the point cloud data is difficult to splice in a large range, and generally only can be displayed through sparse point cloud data, but the data precision and quality are greatly reduced. The fusion of the two-dimensional image and the three-dimensional data is basically only simple data superposition or characteristic fusion due to different sources of the data, but the road surface is difficult to have reference characteristics, and even if the road surface is diseased, the characteristic extraction is the problem to be solved.
Therefore, in order to solve the problem of homology of two-dimensional data and three-dimensional data, improve the quality and precision of the collected images of the road surface diseases, reduce the data volume and realize automatic identification, the invention mainly aims at establishing a multi-scale disease database and multi-model fusion for automatic identification of the road surface diseases.
The first embodiment is as follows:
the method comprises the following steps: in order to ensure the collection of single lane width data (3.75 meters), a digital interference fringe image is generated by a laser interferometer and projected onto a road surface (the coverage width is 4.2 meters), at least two industrial cameras are used for collecting the road surface to obtain an original image, the original image is cut according to the collection effective range of the two industrial cameras to obtain an original hologram, and the image 2 is a road surface original hologram (2*1 meters) respectively collected by a left camera and a right camera.
Step two: in order to obtain high-precision three-dimensional information, the holographic reconstruction of the road surface and the restoration reconstruction of the original hologram are realized by simulating an optical diffraction process by a computer. Referring to fig. 3, the left and right single-camera restored three-dimensional holographic disease maps (2*1 meters). The original hologram comprises light intensity information of the road surface with digital interference fringes and phase information of light waves, phase difference is obtained by three-dimensional reconstruction through Fourier fast transformation according to the relation between the light intensity and an electromagnetic equation, and the whole depth information of the original hologram of the road surface is restored by calculating continuous phase difference change, so that the three-dimensional holographic disease map with the road surface depth data is obtained. The specific digital holographic three-dimensional reduction process comprises the following steps:
the electromagnetic energy equation of any pixel point (x, y) in the original hologram is as follows:
E(x，y)＝A(x，y)exp(iφ(x，y))
wherein phi (x, y) is phase information of any pixel point (x, y), and A (x, y) is light intensity information of any pixel point (x, y);
the light intensity equation of any pixel point (x, y) of the original hologram is as follows:
I(x，y)＝A(x，y) 2 ＝I(x，y)＝|E(x，y)| 2 ＝E(x，y)E(x，y) * ；
fresnel digital diffraction transformation formula:
wherein: Γ (ξ, β) is the light intensity and phase of the original hologram, λ is the wavelength of the light source, i represents complex number, d is the perpendicular distance from the laser interferometer to the camera aperture plane, ρ is the diffraction distance from the camera aperture to the laser interferometer plane;
fourier transformation is equivalent to Fresnel variation, three-dimensional holographic phase information of any pixel point (x, y) of the original hologram is obtained through formula conversion, and the phase difference of the whole original hologram, namely the depth information of all pixel points of the original hologram, is obtained through continuous Fourier variation, so that the three-dimensional holographic disease map of the pavement is obtained through three-dimensional reduction:
I(x，y)＝|Γ(ξ，β)| 2
step three: the original hologram contains road surface information, and a homologous two-dimensional disease image of the road surface disease can be obtained by removing interference fringes, and reference is made to fig. 4: and respectively removing the two-dimensional disease images of the stripes by the left and right single cameras.
Fourier multistage formula of light intensity A (x, y) of any pixel point (x, y) in original hologram:
removing Fourier zero order and first order a 0 +a 1 (x，y) 1 Carrying out Fourier multistage inverse transformation on the data to obtain two-dimensional pixels of any pixel points (x, y) in the original hologram without interference fringe information; and eliminating digital interference fringes by performing Fourier multilevel matrix transformation and Fourier multilevel matrix inverse transformation on all pixel points of the original hologram to obtain a two-dimensional road surface disease image.
Step four: the three-dimensional holographic disease image contains depth information, and deformation diseases of the pavement can be analyzed; the two-dimensional disease map has pavement texture characteristics and can be used for analyzing cracks of a pavement. However, both information are missing, and in order to extract more detailed disease features, it is necessary to perform pixel-level matrix data fusion on two homologous data, namely a three-dimensional holographic disease map and a two-dimensional disease map. Therefore, a matrix A is generated for the three-dimensional holographic disease map, a matrix B is generated for the two-dimensional disease map, the matrix can be represented as superposition of a low-rank matrix and a sparse matrix under an optimization criterion, a matrix C is obtained by respectively adding the matrix A and the matrix B, an augmented Lagrange multiplier method is carried out on the matrix C to solve a low-rank matrix E sparse matrix F, the matrix E and the matrix F are respectively superposed and converted to generate a graph, and finally a fused disease map is obtained, wherein the result refers to fig. 5.
Step five: because the coverage range of some road surface diseases is larger and exceeds the effective shooting range of a single camera, in order to identify larger diseases, pictures shot by different cameras need to be spliced left and right and front and back to generate a spliced disease picture with larger size. The three-dimensional digital holographic vehicle-mounted acquisition equipment adopts a synchronous shooting mode for different industrial cameras, firstly carries out noise reduction processing on the small-size pavement original hologram data generated in the first step, cuts off edge noise on each side, secondly obtains an optimal fitting plane ax + by + cz + d =0 of a pavement map to be spliced by adopting a least square method, rotates the pavement map of the other industrial camera by taking the fitting plane of one industrial camera as a reference, so that the pavement maps on the left side and the right side are completely positioned on the same reference plane, and carries out seamless splicing to generate left and right spliced disease maps (the result refers to a multi-size spliced map (4*1 m) of the three-dimensional holographic disease map of the left and the right industrial cameras of a picture 6, a multi-size spliced map (4*1 m) of the fused disease map of the left and the right industrial cameras of a picture 8 and a multi-size spliced map (4*1 m)) of the two-dimensional disease map with stripes removed; and similarly, splicing the left and right spliced disease patterns of two different industrial cameras back and forth, solving the deviation angle of two plane spaces according to the space data of the fitting plane, selecting the front left and right spliced images as a reference plane, performing three-dimensional space rotation on the images to be spliced, and finally performing spatial butt joint on the three-dimensional data matrix to be spliced to achieve seamless splicing, and splicing the front and back to generate the multi-size pavement map (the result refers to the spliced three-dimensional holographic disease pattern (4 x 10 m) of the left and right cameras in the figure 9). A multi-dimensional pavement data set is obtained. In addition, for convenience of presentation, a three-dimensional hologram and a three-dimensional mosaic may be three-dimensionally displayed, referring to fig. 10 and 11.
Step six: and establishing a multi-size pavement disease data set, and respectively performing characteristic engineering on disease databases of different sizes. Firstly, marking partial data, preprocessing the data through normalization and binarization to generate a three-dimensional holographic disease database, a two-dimensional disease database, a three-dimensional hologram and a fusion disease database, carrying out data segmentation on the three databases to generate three corresponding training sets, verification sets and test sets, respectively substituting the three training sets, the verification sets and the test sets into the built neural network model to carry out repeated iterative training to obtain three training models, respectively obtaining two fusion models through a training model weighted voting fusion mode and a averaging fusion mode, importing the fusion models, testing the three test sets, and selecting a result model meeting requirements to export the training model.
In order to verify the validity and authenticity of a multi-scale disease training result, feature extraction and model fusion are carried out according to existing training data, a training model is derived, and test disease graphs of different scales are tested. Referring to the disease identification result of FIG. 12, it is shown that cracks (located in the effective range of the left camera) are well identified on the test disease graphs of different scales (12 a: left camera three-dimensional hologram (2*1 m), 12b: left camera fused disease graph (2*1 m), 12c: left and right camera three-dimensional hologram multi-size mosaic (4*1 m) and 12d: left and right camera fused disease graph multi-size mosaic (4*1 m)).
The above description is only a preferred embodiment of the present invention, and should not be taken as limiting the invention, and any minor modifications, equivalents and improvements made on the above embodiment according to the technical spirit of the present invention should be included in the protection scope of the technical solution of the present invention.
Claims (6)
1. The data processing method for the pavement disease image fusion based on the three-dimensional digital holography is characterized by comprising the following specific steps of:
s1, generating a digital interference fringe image through a laser interferometer in three-dimensional digital holographic vehicle-mounted acquisition equipment, projecting the digital interference fringe image onto a road surface, acquiring the road surface by using at least two industrial cameras to obtain an original image, and cutting the original image according to an effective acquisition range of the industrial cameras to obtain an original hologram;
s2, simulating an optical diffraction process by using a computer to realize holographic reconstruction and processing reduction reconstruction of the pavement to obtain a three-dimensional holographic disease map with pavement depth data;
s3, removing interference fringe information from the original hologram by adopting Fourier multi-level matrix transformation and Fourier multi-level matrix inverse transformation, thereby obtaining a two-dimensional disease map;
s4, carrying out homologous data fusion on the three-dimensional holographic disease image and the two-dimensional disease image generated by the same original hologram to generate a fused disease image;
s5, respectively carrying out multi-size fusion on the three-dimensional holographic disease images and the two-dimensional disease images obtained by the at least two industrial cameras by adopting a least square fitting method to obtain a multi-size pavement data set; carrying out noise reduction processing on the small-size pavement data generated in the first step, and cutting off edge noise of each edge;
s6: and performing data fusion on the pavement data sets with multiple sizes to generate a three-dimensional holographic disease database, a two-dimensional disease database and a fusion disease database, substituting the three-dimensional holographic disease database, the two-dimensional disease database and the fusion disease database into the built neural network model for training to obtain a fusion model, and performing test simulation correction on the fusion model by adopting specified data to obtain a final training model.
2. The data processing method for three-dimensional digital holography based pavement disease image fusion as claimed in claim 1, wherein: in the step S2, the original hologram comprises light intensity information with digital interference fringes on the road surface and phase information of light waves, three-dimensional reconstruction is carried out by adopting Fourier fast transformation according to the relation between the light intensity and an electromagnetic equation to obtain a phase difference, and the whole depth information of the original hologram of the road surface is restored by calculating continuous phase difference change to obtain a three-dimensional holographic disease map with road surface depth data;
the specific digital holographic three-dimensional reduction process comprises the following steps:
s21, an electromagnetic energy equation of any pixel point (x, y) in the original hologram is as follows:
E(x,y)＝A(x,y)exp(iφ(x,y))
wherein phi (x, y) is phase information of any pixel point (x, y), and A (x, y) is light intensity information of any pixel point (x, y);
s22, the light intensity equation of any pixel point (x, y) of the original hologram is as follows:
I(x,y)＝A(x,y) 2 ＝I(x,y)＝E(x,y) 2 ＝E(x,y)E(x,y)*；
s23, a Fresnel digital diffraction transformation formula:
wherein: Γ (ξ, β) is the light intensity and phase of the original hologram, λ is the wavelength of the light source, i represents complex number, d is the perpendicular distance from the laser interferometer to the camera aperture plane, ρ is the diffraction distance from the camera aperture to the laser interferometer plane;
s24, fourier transformation is equivalent to Fresnel change, three-dimensional holographic phase information of any pixel point (x, y) of the original hologram is obtained through formula conversion, the phase difference of the whole original hologram is obtained through continuous Fourier change, namely the depth information of all pixel points of the original hologram, and therefore the three-dimensional holographic disease image of the pavement is obtained through three-dimensional reduction:
I(x,y)＝|Γ(ξ，β)| 2
3. the data processing method for three-dimensional digital holography based pavement disease image fusion as claimed in claim 1, wherein: in step S3, obtaining the road information of the original hologram, a fourier multilevel formula of the light intensity a (x, y) of any pixel point (x, y) in the original hologram:
according to a Fourier multi-stage formula, deleting zero order and a order of Fourier in the Fourier multi-stage formula 0 +a 1 (x,y) 1 Carrying out Fourier multistage inverse transformation on the data to obtain two-dimensional pixels of any pixel points (x, y) in the original hologram without interference fringe information;
and eliminating digital interference fringes by performing Fourier multilevel matrix transformation and Fourier multilevel matrix inverse transformation on all pixel points of the original hologram to obtain a two-dimensional road surface disease image.
4. The data processing method for three-dimensional digital holography based pavement disease image fusion as claimed in claim 1, wherein: in the step S4, performing pixel-level data fusion on a three-dimensional holographic disease image and a two-dimensional disease image generated by the same original hologram;
the specific data fusion process is as follows: the image of the original hologram is represented as matrix data, a three-dimensional holographic disease map generation matrix A and a two-dimensional disease map generation matrix B, the matrixes can be represented as superposition of a low-rank matrix and a sparse matrix under an optimization criterion, a matrix C is obtained by respectively adding the matrix A and the matrix B, an augmented Lagrange multiplier method is carried out on the matrix C to solve a low-rank matrix E sparse matrix F, the matrix E and the matrix F are respectively superposed and converted to generate a graph, and finally the fused disease map is obtained.
5. The data processing method for three-dimensional digital holography based pavement disease image fusion as claimed in claim 1, wherein: in the step S5, splicing and stitching three-dimensional holographic disease images, two-dimensional disease images and fusion disease images acquired by different industrial cameras to obtain a pavement map;
obtaining an optimal fitting plane ax + by + cz + d =0 of the road surface map to be spliced by adopting a least square method, rotating the road surface map of the other industrial camera by taking the fitting plane of one industrial camera as a reference, enabling the road surface maps on the left side and the right side to be completely positioned on the same reference plane, and carrying out seamless splicing to generate a left spliced disease map and a right spliced disease map;
splicing left and right spliced disease images of two different industrial cameras front and back, solving the deviation angle of two plane spaces according to the space data of a fitting plane, selecting the front left and right spliced images as a reference plane, performing three-dimensional space rotation on the images to be spliced, finally performing space butt joint on the three-dimensional data matrix to be spliced, achieving seamless splicing, and splicing front and back to generate a multi-size pavement map.
6. The data processing method for three-dimensional digital holography based pavement disease image fusion as claimed in claim 1, wherein: in step S6, respectively performing characteristic engineering on the three-dimensional holographic disease images and the two-dimensional disease images with different sizes;
firstly, marking partial data, preprocessing the data through normalization and binarization to generate a three-dimensional holographic disease database, a two-dimensional disease database, a three-dimensional hologram and a fusion disease database, carrying out data segmentation on the three databases to generate three corresponding training sets, verification sets and test sets, respectively substituting the three training sets, the verification sets and the test sets into a built neural network model to carry out iterative training to obtain three training models, respectively obtaining two fusion models through a training model weighted voting fusion mode and a averaging fusion mode, importing the fusion models, testing the three test sets, and selecting a result model meeting requirements to export the training models.
Priority Applications (1)
| Application Number | Priority Date | Filing Date | Title |
|---|---|---|---|
| CN202210983139.7A CN115330846A (en) | 2022-08-16 | 2022-08-16 | Three-dimensional digital holography-based pavement disease image fusion data processing method |
Applications Claiming Priority (1)
| Application Number | Priority Date | Filing Date | Title |
|---|---|---|---|
| CN202210983139.7A CN115330846A (en) | 2022-08-16 | 2022-08-16 | Three-dimensional digital holography-based pavement disease image fusion data processing method |
Publications (1)
| Publication Number | Publication Date |
|---|---|
| CN115330846A true CN115330846A (en) | 2022-11-11 |
Family
ID=83923061
Family Applications (1)
| Application Number | Title | Priority Date | Filing Date |
|---|---|---|---|
| CN202210983139.7A Pending CN115330846A (en) | 2022-08-16 | 2022-08-16 | Three-dimensional digital holography-based pavement disease image fusion data processing method |
Country Status (1)
| Country | Link |
|---|---|
| CN (1) | CN115330846A (en) |
Cited By (2)
| Publication number | Priority date | Publication date | Assignee | Title |
|---|---|---|---|---|
| CN116756396A (en) * | 2023-06-29 | 2023-09-15 | 广东齐峰信息科技有限公司 | Digital archive management system and method based on knowledge graph |
| CN117315164A (en) * | 2023-11-28 | 2023-12-29 | 虚拟现实(深圳)智能科技有限公司 | Optical waveguide holographic display method, device, equipment and storage medium |
-
2022
- 2022-08-16 CN CN202210983139.7A patent/CN115330846A/en active Pending
Cited By (4)
| Publication number | Priority date | Publication date | Assignee | Title |
|---|---|---|---|---|
| CN116756396A (en) * | 2023-06-29 | 2023-09-15 | 广东齐峰信息科技有限公司 | Digital archive management system and method based on knowledge graph |
| CN116756396B (en) * | 2023-06-29 | 2023-12-22 | 广东齐峰信息科技有限公司 | Digital archive management system and method based on knowledge graph |
| CN117315164A (en) * | 2023-11-28 | 2023-12-29 | 虚拟现实(深圳)智能科技有限公司 | Optical waveguide holographic display method, device, equipment and storage medium |
| CN117315164B (en) * | 2023-11-28 | 2024-02-23 | 虚拟现实(深圳)智能科技有限公司 | Optical waveguide holographic display method, device, equipment and storage medium |
Similar Documents
| Publication | Publication Date | Title |
|---|---|---|
| CN111063021B (en) | Method and device for establishing three-dimensional reconstruction model of space moving target | |
| CN115330846A (en) | Three-dimensional digital holography-based pavement disease image fusion data processing method | |
| Sweeney et al. | Large scale sfm with the distributed camera model | |
| CN103763543B (en) | The acquisition method of resultant hologram | |
| CN114004938B (en) | Urban scene reconstruction method and device based on mass data | |
| CN114359509A (en) | Multi-view natural scene reconstruction method based on deep learning | |
| CN103941569A (en) | Method for LED multi-angle holographic three-dimensional data representation | |
| Wang et al. | Three-dimensional reconstruction of dilute bubbly flow field with light-field images based on deep learning method | |
| Wang et al. | A novel three-dimensional surface reconstruction method for the complex fabrics based on the MVS | |
| Hassner et al. | SIFTing through scales | |
| Tyle_ek et al. | Refinement of surface mesh for accurate multi-view reconstruction | |
| Wang et al. | Accurate 3D reconstruction of single-frame speckle-encoded textureless surfaces based on densely connected stereo matching network | |
| CN110580468A (en) | Single-wood structure parameter extraction method based on image matching point cloud | |
| CN113963107B (en) | Binocular vision-based large-scale target three-dimensional reconstruction method and system | |
| Yao et al. | Research on three-dimensional model reconstruction of slope erosion based on sequence images | |
| CN113129348B (en) | Monocular vision-based three-dimensional reconstruction method for vehicle target in road scene | |
| Liu et al. | Benchmarking large-scale multi-view 3D reconstruction using realistic synthetic images | |
| Jiale et al. | Non-textured objects visual SLAM using pola-rization 3D reconstruction | |
| Pang et al. | Dynamic holographic imaging of real-life scene | |
| Ruotsala | Digital Close-Range Photogrammetry–A Modern Method to Document Forensic Mass Graves | |
| Lyra et al. | Development of an efficient 3D reconstruction solution from permissive open-source code | |
| Ma et al. | A Deep Learning-Based Hyperspectral Keypoint Representation Method and Its Application for 3D Reconstruction | |
| Li et al. | Toward Cooperative 3D Object Reconstruction with Multi-agent | |
| Wen et al. | 3D inspection technology combining passive stereo matching and active structured light for steel plate surface sample | |
| Wang et al. | High-speed 3D shape measurement from noisy fringe images using deep learning |
Legal Events
| Date | Code | Title | Description |
|---|---|---|---|
| PB01 | Publication | ||
| PB01 | Publication | ||
| SE01 | Entry into force of request for substantive examination | ||
| SE01 | Entry into force of request for substantive examination |