CN111063021A

CN111063021A - Method and device for establishing three-dimensional reconstruction model of space moving target

Info

Publication number: CN111063021A
Application number: CN201911146834.2A
Authority: CN
Inventors: 潘泉; 周康博; 侯晓磊; 刘勇
Original assignee: Northwestern Polytechnical University
Current assignee: Northwestern Polytechnical University
Priority date: 2019-11-21
Filing date: 2019-11-21
Publication date: 2020-04-24
Anticipated expiration: 2039-11-21
Also published as: CN111063021B

Abstract

The invention discloses a method and a device for establishing a three-dimensional reconstruction model of a space moving target, which are used for acquiring a space monocular image sequence of a space fragment or a failure satellite; training a spatial monocular image sequence by adopting a trained generation countermeasure network to eliminate the flare and shadow of each spatial monocular image in the spatial monocular image sequence and generate a first image sequence; extracting a characteristic value matching relation between each image and other images in the first image sequence by a scale invariant characteristic transformation method; establishing a three-dimensional model sparse point cloud of a space debris or a failure satellite according to the characteristic value matching relation; generating a three-dimensional model grid model according to the three-dimensional model sparse point cloud, and sequentially carrying out grid filtering and texturing on the three-dimensional model grid model to obtain a three-dimensional reconstruction model of a space debris or a failure satellite; the method and the device can solve the problem of distortion of the three-dimensional reconstruction model when the three-dimensional reconstruction model of the space moving target is reconstructed according to the space monocular image sequence.

Description

Method and device for establishing three-dimensional reconstruction model of space moving target

[ technical field ] A method for producing a semiconductor device

The invention belongs to the technical field of three-dimensional reconstruction of moving objects in a cosmic space, and particularly relates to a method for establishing a three-dimensional reconstruction model of a space moving object.

[ background of the invention ]

In space missions, in-orbit services are often required for satellites or space station equipment, such as docking of space stations, recovery of failed satellites, and removal of space debris. With the development of aerospace technologies of countries around the world, more than 6000 spacecrafts launched by human beings are developed so far, only a small part of the spacecrafts enter the atmosphere and are destroyed by themselves, and most satellites or other artificial celestial bodies still remain in the space orbit as space garbage.

The existing mode for removing space garbage generally adopts a primary and secondary satellite to remove a target, but when a large satellite is used for removing space debris or a failed satellite, the problems of fuel supply, orbit conversion and the like are involved, and a large amount of energy and time are wasted. Therefore, the space debris or the failed satellite is observed in advance by the aid of the sub-satellite to obtain an image, and reconstruction is performed according to observation information to provide prior information for next-step removal of the space debris or recovery of the failed satellite by the aid of the mother satellite.

However, since the images obtained by pre-observation are affected by the target material and the space environment, the reconstruction model may be distorted in the reconstruction process, the accuracy of the reconstruction model may be reduced, and the removal of space debris and the recovery plan of the failed satellite may be affected.

[ summary of the invention ]

The invention aims to provide a method and a device for establishing a three-dimensional reconstruction model of a space moving object, which aim to solve the problem of distortion of the three-dimensional reconstruction model when the three-dimensional reconstruction model of the space moving object is reconstructed according to a space monocular image sequence.

The invention adopts the following technical scheme: a method for building a three-dimensional reconstruction model of a space moving object comprises the following steps:

acquiring a space monocular image sequence of a space debris or a failure satellite;

training a spatial monocular image sequence by adopting a trained generation countermeasure network to eliminate the flare and shadow of each spatial monocular image in the spatial monocular image sequence and generate a first image sequence;

extracting a characteristic value matching relation between each image and other images in the first image sequence by a scale invariant characteristic transformation method;

establishing a three-dimensional model sparse point cloud of a space debris or a failure satellite according to the characteristic value matching relation;

and generating a three-dimensional model grid model according to the three-dimensional model sparse point cloud, and sequentially carrying out grid filtering and texturing on the three-dimensional model grid model to obtain a three-dimensional reconstruction model of the space debris or the failure satellite.

Further, after obtaining the three-dimensional reconstruction model of the space debris or the failed satellite, the method further includes:

generating a second image sequence according to the three-dimensional reconstruction model; the second image sequence corresponds to the images in the space monocular image sequence and is equal in number, and the two corresponding images have the same shooting angle with respect to the space debris or the failed satellite;

calculating the defect degree of each image in the second image sequence;

when the defect degree is larger than a first preset threshold value, acquiring a space monocular image corresponding to a defect image in the second image sequence in the space monocular image sequence;

repairing the corresponding space monocular image by adopting the trained generated confrontation network to obtain a repaired space monocular image sequence;

and replacing the space monocular image sequence with the repaired space monocular image sequence, and continuing to execute.

Further, when the second image sequence is generated according to the three-dimensional reconstruction model, the second image sequence is generated by respectively adopting a pure black background and a pure white background.

Further, the method for generating each image in the second image sequence is as follows:

determining a sampling frame of the two-dimensional image;

sequentially carrying out sampling scanning on the three-dimensional reconstruction model by using a sampling frame to generate a plurality of fragment images;

and recombining the plurality of fragment images according to the sampling scanning sequence to obtain a sampling image.

Further, calculating the defect degree of each image in the second image sequence comprises:

acquiring a pure black background image and a pure white background image;

converting the pure black background image and the pure white background image into gray level images;

generating a vacancy distinguishing matrix according to the gray value of each pixel point in the gray image of the pure white background image and the gray value of each pixel point in the gray image of the pure black background image;

and generating the defect degree of each image according to the vacancy distinguishing matrix.

Further, the repairing of the space monocular image corresponding to the trained generated countermeasure network includes:

segmenting the vacancy distinguishing matrix according to the plurality of fragment images to obtain a plurality of fragment matrixes;

calculating the sum value of each element in each fragment matrix;

marking the image area corresponding to the fragment matrix with the sum value larger than a second preset threshold value as an area to be repaired;

and repairing the area to be repaired by adopting a trained pair of generated confrontation networks.

The other technical scheme of the invention is as follows: an apparatus for modeling a three-dimensional reconstruction of a spatially moving object, comprising:

the acquisition module is used for acquiring a space monocular image sequence of a space debris or a failure satellite;

the training module is used for training the spatial monocular image sequence by adopting a trained generation countermeasure network so as to eliminate the flare and the shadow of each spatial monocular image in the spatial monocular image sequence and generate a first image sequence;

the matching module is used for extracting a characteristic value matching relation between each image and other images in the first image sequence by a scale-invariant characteristic transformation method;

the first construction module is used for constructing a three-dimensional model sparse point cloud of the space debris or the failure satellite according to the characteristic value matching relation;

the first generation module is used for generating a three-dimensional model grid model according to the three-dimensional model sparse point cloud, and sequentially carrying out grid filtering and texturing on the three-dimensional model grid model to obtain a three-dimensional reconstruction model of a space debris or a failure satellite.

Further, still include:

the second generation module is used for generating a second image sequence according to the three-dimensional reconstruction model after the three-dimensional reconstruction model of the space debris or the failure satellite is obtained; the second image sequence corresponds to the images in the space monocular image sequence and is equal in number, and the two corresponding images have the same shooting angle with respect to the space debris or the failed satellite;

the calculation module is used for calculating the defect degree of each image in the second image sequence;

the judging module is used for acquiring a space monocular image corresponding to a defective image in the second image sequence in the space monocular image sequence when the defect degree is larger than a first preset threshold value;

the repairing module is used for repairing the space monocular image corresponding to the trained generation countermeasure network to obtain a repaired space monocular image sequence;

and the replacing module is used for replacing the space monocular image sequence with the repaired space monocular image sequence and continuing to execute the operation.

Further, the second generation module further comprises a pure black background generation submodule and a pure white background generation submodule;

and the pure black background generation submodule and the pure white background generation submodule are used for generating a second image sequence by respectively adopting a pure black background and a pure white background when generating the second image sequence according to the three-dimensional reconstruction model.

The other technical scheme of the invention is as follows: a three-dimensional reconstruction modeling device of a space moving object comprises a memory, a processor and a computer program which is stored in the memory and can run on the processor, wherein the processor executes the computer program to realize the three-dimensional reconstruction modeling method of the space moving object.

The invention has the beneficial effects that: the method and the device have the advantages that the anti-network is generated to repair the spatial monocular image sequence, so that the effects of flare and shadow caused by the special spatial material and the special illumination environment can be eliminated, the repaired image sequence is subjected to three-dimensional reconstruction, the task requirement of the three-dimensional reconstruction of the special spatial environment can be met, the three-dimensional model distortion rate is reduced, and the better reconstruction model precision is achieved.

[ description of the drawings ]

FIG. 1 is a flow chart of one embodiment of the present application;

FIG. 2 is a flow chart of another embodiment of the present application;

FIG. 3 is a flow chart of a process of generating a first sequence of images in an embodiment of the present application;

FIG. 4 is a schematic diagram illustrating the difference between the extreme points in discrete space and continuous space in the embodiment of the present application;

FIG. 5 is a graph of an image gradient transformation descriptor in an embodiment of the present application;

FIG. 6 is a flow chart of a recovery structure in motion in an embodiment of the present application;

FIG. 7 is a diagram illustrating a reprojection error according to an embodiment of the present application;

fig. 8 is a schematic diagram of a structure recovery method in incremental movement according to an embodiment of the present application.

[ detailed description ] embodiments

The present invention will be described in detail below with reference to the accompanying drawings and specific embodiments.

The method for establishing the three-dimensional reconstruction model of the spatial moving object provided by the embodiment of the present application may be applied to various intelligent devices, for example, a tablet computer, a notebook computer, an ultra-mobile personal computer (UMPC), a netbook, and other terminal devices.

The embodiment of the application provides a method for establishing a three-dimensional reconstruction model of a space moving target, which comprises the steps of acquiring a space monocular image sequence of space fragments or invalid satellites as shown in figure 1; training a spatial monocular image sequence by adopting a trained generation countermeasure network to eliminate the flare and shadow of each spatial monocular image in the spatial monocular image sequence and generate a first image sequence; extracting a characteristic value matching relation between each image and other images in the first image sequence by a scale invariant characteristic transformation method; establishing a three-dimensional model sparse point cloud of a space debris or a failure satellite according to the characteristic value matching relation; and generating a three-dimensional model grid model according to the three-dimensional model sparse point cloud, and sequentially carrying out grid filtering and texturing on the three-dimensional model grid model to obtain a three-dimensional reconstruction model of the space debris or the failure satellite.

In the embodiment of the application, a generated countermeasure network (GAN) is used for repairing an input image of unordered time (namely, an image in a spatial monocular image sequence), the influence of flare and shadow on a visual image is fully eliminated, an output result of the generated countermeasure network is used as an image input of three-dimensional reconstruction, coordinates of image Feature points are extracted by using a Scale-Invariant Feature Transform (SIFT) method, three-dimensional information (namely sparse point cloud) of a space fragment or a failed satellite is calculated by using a recovery Structure from Motion (SfM) method in Motion, a dense reconstruction grid model is performed by using a Multi-View Stereo (MVS) method, and grid filtering and processing are sequentially performed on the dense grid model to obtain the three-dimensional reconstruction model. In the embodiment of the application, the generation of the countermeasure network is a generation countermeasure network model generated by adopting a method for generating the countermeasure network for training.

The method and the device have the advantages that the anti-network is generated to repair the spatial monocular image sequence, so that the effects of flare and shadow caused by the special spatial material and the special illumination environment can be eliminated, the repaired image sequence is subjected to three-dimensional reconstruction, the task requirement of the three-dimensional reconstruction of the special spatial environment can be met, the three-dimensional model distortion rate is reduced, and the better reconstruction model precision is achieved.

In addition, the images in the spatial monocular image sequence adopted by the application can be acquired by a monocular camera, and the image sequence can be ordered or unordered. Compared with a laser radar and a double-sided camera, the monocular camera has the characteristics of small volume, high reliability and low cost, can be better suitable for being carried by a microsatellite, namely suitable for observing space fragments or invalid satellites and acquiring images by the sub-satellites in the primary and secondary satellites.

When the step is carried out, a darkroom is used for simulating a space environment, and sunlight is simulated through a parallel light source to collect images.

The generation of the countermeasure network is generated by utilizing the training of the existing occlusion data set, firstly, the images are randomly occluded by the occlusion data set, the generation of the countermeasure network is trained to generate an occlusion part, the generated occlusion part is compared with the data of the occluded part of the original image, and the error between the generated part and the original image is continuously reduced through a loss function so as to finish the training of the generation of the countermeasure network. The use is only allowed when the generated generation countermeasure network achieves a good accuracy in the test set to ensure that the glare and shadows in the spatial monocular image can be well removed.

The training process based on generation of the countermeasure network satisfies:

wherein G (z) represents a generative model, D (x) represents a discriminant model, p_rRepresenting the distribution of real images, p_gRepresenting the distribution of the generated image. The generation model g (z) is used to generate an image, and the discrimination model d (x) is used to discriminate the probability that the input data x is from a real image. The training of the discriminant model is supervised two-classification, the training of the generative model is the series training of the generative model and the discriminant model, and the results are converged by the alternate training of the generator and the discriminant, it is noted that the parameters of the discriminant model cannot be changed when the generators are trained in series. Iteratively updating G to maximize the probability that the arbiter will make the wrong classification, and iteratively updating D to maximize the probability that the arbiter will make the correct classification。

In the initial stage of training, it is impossible to provide enough gradient to train D, because the noise generated by the generator G is far from the true value, the discriminator D can accurately discriminate the true and false samples, the log (1-D (G (z)) tends to saturate, there is not enough gradient to minimize the log (1-D (G (z)), and the generating network should be trained to maximize logd (x). The objective function gives stronger gradient information in the early training phase and has the same effect on the training of G and D.

The generation network G designed by the invention mainly comprises two parts, wherein the first part is a line generation network G_lMainly responsible for generating image lines according to the original image and filling lines of the image shielding part, and the other part is an image generation network G_pAnd the system is mainly responsible for filling, repairing and coloring the lines generated by the line generation network. Let X denote the unoccluded line data and N denote whether the image data is occluded by the occlusion data set, denoted as

Y_tFor a line representation of an image, the lines generated by the line generating network may be expressed as

Wherein the content of the first and second substances,

⊙ is a Hadamard product in the graph generation network, again with the unobstructed line data x as input, the image generated by the graph generation network can be represented as:

the arbiter uses a markov arbiter as a 70 × 70 PatchGAN network. In a traditional discriminator, a general generation countermeasure network can only evaluate whether the whole picture is true or false, obviously, the evaluation will affect the discrimination of some high-resolution, high-definition and large-size pictures, the PatchGAN will give a matrix discrimination of 70 × 70 for each picture, each element of the matrix represents true or false of a receptive field, the precision of the evaluation of the whole picture is improved, the picture repairing error is reduced, and meanwhile, a more accurate result can be provided for the input of the three-dimensional reconstructed picture.

The loss function of the discriminator is

Where x is the observed data, y is the output data, and z is the random noise vector. Fig. 3 is a schematic diagram of an image restoration process according to an embodiment of the present application.

Conventional GAN-based image restoration uses only one GAN; the method of the embodiment adopts a contour generation GAN and a pix2pix network (a GAN network for generating images based on the contours generated by the GAN), and enables the image generation result to have better shape contour information of the observed target (dead satellites or fragments) by combining 2 GAN networks.

In this embodiment, the generated countermeasure network by using pre-training is a generated countermeasure network obtained by training using a large number of images of artificial celestial bodies such as satellites and fragments in order to accelerate the training process and to converge the network quickly.

And taking the first image sequence after the generation of the countermeasure network processing as an input image sequence of a three-dimensional reconstruction part, extracting the image feature points of each image in the sequence by using a scale-invariant feature transformation method, and matching one image of the input image sequence with the feature points of other images.

Obtaining image scale space by Gaussian blur image filter, using N-dimensional space Gaussian function

It is known that the larger the standard deviation σ of the normal distribution is, the more blurred the image is, where r is a blur radius, and when the size of the two-dimensional template is mxn, the gaussian calculation formula corresponding to the element (x, y) on the template is as follows

Two-dimensional Gaussian blur results of the image can be obtained.

In order to prevent the black edge effect caused by excessive loss of the image edge pixels, the image needs to be subjected to separation Gaussian blur processing, and the operation can reduce the amount of Gaussian kernel and convolution operation at the same time. Due to the separability of the gaussian blur, the effect obtained by using the two-dimensional matrix transformation can be equivalently obtained by respectively performing the one-dimensional gaussian matrix transformation in the horizontal direction and the vertical direction, so that the black edges generated by performing the two-dimensional gaussian matrix operation on the image can be offset by two times of one-dimensional gaussian convolution.

Continuously reducing the price of the image without the flare and the shadow, sampling and applying Gaussian filtering to the image without the flare and the shadow, carrying out Gaussian blur with different parameters on each layer of image to obtain a Gaussian pyramid, wherein the layer number of the Gaussian pyramid is determined by the dimension of the image without the flare and the shadow and the dimension of the tip image, and the layer number c is log₂[min(m,n)]-t, where t ∈ [0, log2[ min (m, n) ]]]M and n represent the dimension of the original image, t represents the logarithm value of the minimum dimension of the tip image, each layer of the Gaussian golden sub-tower is composed of a plurality of images obtained by adding Gaussian blur of different parameters to the image after removing flare and shadow, and the plurality of images are called as a group.

Comparing the maximum and minimum values of the scale-normalized gaussian laplacian function or gaussian difference function with other feature extraction functions can generate the most stable image features. The difference is made between each adjacent layer of the gaussian pyramid to obtain a gaussian difference pyramid, the monitoring points of the gaussian difference space are compared with the adjacent points and the adjacent layer points to preliminarily obtain discrete space extreme points of the gaussian difference function, and a schematic diagram of the discrete space extreme points and the continuous space extreme points is shown in fig. 4. And performing sub-pixel interpolation in the discrete space points to obtain continuous space extreme points. In order to improve the stability of the key points, curve fitting needs to be carried out on a Gaussian difference function in a scale space, and the fitting function of the Gaussian difference function in the scale space is

Where X ═ (X, y, σ)^TAnd σ is a scale space coordinate, and (x, y) represents the position of an image pixel. By deriving the fitting function and making it zero, the offset of the extreme point can be calculatedIs composed of

The corresponding extreme point equation has a value of

Wherein the content of the first and second substances,

when offset from the center of interpolation

At x, y, σ greater than 0.5, the keypoint location should be changed and re-interpolated at the new location until convergence, and when the iteration range or image boundary is exceeded, the interpolation should be terminated and the point culled.

In addition, the gaussian difference operator generates a strong edge response at the boundary of the image, that is, a large principal curvature exists near a pixel point crossing the edge, and a small principal curvature exists at an adjacent pixel point perpendicular to the edge, so that the pixel point of the edge response should be eliminated. Passing a 2 x2 hessian matrix

The principal curvature can be determined, the eigenvalues of H α and β being the gradients in the x and y directions, the trace tr (H) T of the matrix_xx+T_yyα + β, determinant det (h) T of matrix_xxT_yy-(T_xy)²αβ, when α is r β and r is greater than 1

Let α be the maximum eigenvalue, β be the minimum eigenvalue,

at α - β, it is known that the α and β values can represent gradient values in a certain direction, the larger the ratio, the larger the gradient value in the direction, and the smaller the ratio, the smaller the gradient value, and the edge feature is metThe ratio is smaller than a threshold value to determine whether the principal curvature is below the threshold value r. Namely:

then obtaining the direction information of the key points, and for the key points obtained by the method, the mode of the gradient distribution characteristics of the pixels in the 3 sigma range is as follows:

the gradient characteristic distribution direction is as follows:

θ(x,y)＝tan^-1((L(x,y+1)-L(x,y-1))/(L(x+1,y)-L(x-1,y)))，

the descriptor is established to describe the feature points, the steps are as shown in fig. 5, and the descriptor is selected to be free from the influence of external environment changes such as illumination and visual angle, and has stronger robustness. The descriptor contains key point information and information of other pixel points acting on the key point, and in addition, each descriptor has higher specificity, so that the probability of correct feature matching can be effectively improved. In order to ensure the rotation invariance, firstly, the coordinate axis direction is consistent with the main direction of a key point, secondly, the key point is used as the center, all pixels in a window are selected by taking 16 multiplied by 16 as the window size, the whole window is divided into 16 subblocks of 4 multiplied by 4, each subblock comprises a pixel point, finally, the modulus and the direction of each pixel point are calculated, gradient values in a subarea are distributed to 8 directions, the weight of each subblock is calculated, each characteristic value comprises 16 multiplied by 8 to 128 dimensions, the 128-dimensional characteristic vector can be used as a descriptor, and the characteristic points with unchanged scale and changed features of the image and description information thereof can be obtained through the steps.

And taking the matching relation obtained in the scale invariant feature transformation method as three-dimensional reconstruction input, and performing sparse reconstruction by a structure recovery method in motion to obtain the sparse point cloud of the three-dimensional reconstruction model.

The structure recovery method in motion can be regarded as a problem of estimating camera parameters and three-dimensional point positions, and the steps can be divided into:

(1) matching the detected feature points of every two pictures, and only reserving feature matching pairs which can meet geometric constraint;

(2) restoring internal and external parameters of the camera by a scale invariant feature transformation method;

(3) recovering the depth information of the three-dimensional points by using a triangulation method;

(4) and solving by adopting an optimization algorithm of nonlinear least squares, such as a beam adjustment method.

As shown in fig. 6, the description of the feature point is a 128-dimensional vector containing gradient information of a pixel in the neighborhood of the key point, so that performing feature matching can be regarded as finding the key point information closest to the currently-determined key point in two pictures, and calculating whether the key point information matches the currently-determined key point, and considering a large calculation amount caused by a high-dimensional feature point extracted by the feature transformation with unchanged scale, the nearest neighbor search algorithm adopted in the embodiment of the present application is a BBF algorithm based on a k-d tree (k-dimensional tree).

The k-d tree can be regarded as a binary search tree in a k-dimensional space, in the one-dimensional binary search tree, the numerical value of any node in the left sub-tree of the root node is smaller than that of the root node, the numerical value of any node in the right sub-tree of the root node is larger than that of the root node, and any left and right sub-trees are binary search trees. For k-dimensional space, a certain reference dimension d is selected₀Comparing the k-dimensional data with the data at d₀The size relationship of the data in dimension, i.e. one perpendicular to d₀The hyperplane divides the k-dimensional space into two parts, wherein the data on one side of the hyperplane are all larger than d₀The value on the dimension is less than d₀The value in the dimension. The dimensionality is continuously and recursively divided in the direction of the maximum variance until the dimensionality is divided to leaf nodes of data in the k-dimensional space, a k-d tree can be established, data information of the k-d tree is stored only in the leaf nodes, and the root nodes and the middle nodes are used for storing data dividing information.

And (3) utilizing a BBF algorithm to carry out a nearest neighbor search step:

(1) for the data set S, the target is soughtData U is firstly judged from the root node to divide dimension d₀The value on the data set S is S₀When U (k) < s₀When U (k) > s, then enter the left sub-tree₀And if so, entering the right subtree for access. When U (k) reaches the leaf node, calculating the distance between the leaf node data and U (k) and storing the minimum distance, and marking the data point as the current closest point P_nThe distance is recorded as "current minimum distance" D_n；

(2) Backtracking upwards, and calculating whether the leaf nodes which are not visited have the distance D less than the current minimum distance_nShorter distance, when there is data on a leaf node under the same parent node as U (k) and the distance between U (k) and U (k) is less than D_nThen, the branch is considered to have data closer to U (k), and P is updated_nAnd D_nIf no closer data point is found, the backtracking is continued until a closer point is found or a set maximum backtracking number is reached.

When the number of the feature matching pairs obtained by the two pictures by using a BBF algorithm based on a k-d tree is not less than 16, the two pictures can be used as initial selection images of a recovery structure method in motion. However, since some feature points may not meet objective physical conditions in an actual scene, epipolar geometric calculation should be performed on the feature matching points, and the F matrix is a connection matrix of pixel coordinates of two pictures, and each matching pair pixel coordinate should satisfy the requirement

Wherein the contact matrix F contains camera reference information. And solving the F matrix value, wherein the value contains a large amount of noise information, the F matrix value needs to be filtered by a random sampling consistency algorithm, the basic matrix is solved by using an 8-point method in each iteration, and the matching pairs which do not meet the basic matrix are removed to obtain the final image matching pairs. Let the matching feature of the ith graph be f_iThen the corresponding feature point { f }₁,f₂,f₃...f_nA track is formed, and so on, other eligible tracks can be found, each containing the same feature on the corresponding imageThe pixels are typically key nodes or edge contours of the image.

The parameters in the camera are described by a focal length f and two radial distortion parameters k₁，k₂The description of the camera external parameters uses a 3 × 3 rotation matrix R and a 1 × 3 translation vector. In the actual scene, a point X in the three-dimensional space is arbitrarily taken_jThen, through calculation of the projection equation, the two-dimensional image can be projected on the two-dimensional image, but the point calculated through the projection equation has an error from the true value of the point on the image, and the error is the distance between the projection point and the true point. For m tracks of multiple pictures generated by n view angles, the target optimization equation of the projection error is as follows:

wherein, w_ijTo observe the weight, can be expressed as

||q_ij-P(C_i,X_j)||²The reprojection error for the trajectory j observed in camera i. The scale invariant feature transform method is to optimize the function g (C, X) by using a beam adjustment method by finding appropriate camera scene parameters. The reprojection error is shown in fig. 7.

When the initial image is selected by the scale-invariant feature change method, the image is ensured to have enough matching points and a camera center far enough, otherwise, the beam adjustment method can enter local optimization and cannot obtain a global optimal solution. And then, estimating the external parameters of the initialized matched pairs by adopting a 5-point method, and providing initialized three-dimensional points by triangularization of the track. When two pictures are initialized, rough reconstruction is carried out by using a sparse beam adjustment method, beam adjustment can be continuously carried out by continuously increasing three-dimensional points of the camera until no enough three-dimensional points can be continuously added, and sparse three-dimensional point cloud reconstruction can be obtained at the moment.

And (3) using the three-dimensional reconstruction model sparse point cloud obtained by the motion recovery structure method as input, and performing dense reconstruction by using a multi-view stereoscopic vision method to obtain a three-dimensional model grid model.

Because the scale invariant feature transformation method is a three-dimensional reconstruction algorithm obtained by matching feature points, for each picture, the proportion of the feature points in the pixels is very low in the whole situation, so that dense three-dimensional reconstruction cannot be performed only through the feature points. Therefore, generating a dense reconstruction of multiple views from a three-dimensional sparse point cloud extracted by a scale-invariant feature transform method requires finding the projection of any point on all valid and viewable images of that point in all valid images.

The method and the device adopt a multi-view stereo matching algorithm based on depth map fusion, and are characterized in that redundant recovery point clouds can be obtained, the reconstruction precision is high, the robustness is good, and the method and the device are very suitable for three-dimensional reconstruction of special space environments. The method comprises the following specific steps:

(1) a depth map is reconstructed for each input view. The depth of the image is calculated using a very simple and efficient but robust window matching view and a small number of neighboring views and the confidence value of each pixel is estimated, using only the points with higher confidence in the stitching model process.

Assume that the input is a set of views V ═ V of the object₀,...,V_n-1Camera parameters and an approximate bounding box or volume containing the object. Selecting a set of k neighboring views for each reference view R e V

And correlated with R by robust window matching. For each pixel p, it travels along the reflected ray within the bounding volume of the object. For each depth value d, the generated three-dimensional position is re-projected into all views in C. Calculating normalized cross-correlation values NCC (R, C) for an m window centered on p and a corresponding window centered on projection_jD), wherein each view C_jThere is sub-pixel accuracy.

If the NCC value is greater than the threshold value in at least two graphs in the C set, the depth d is considered to be valid, and the set formed by all graphs with the NCC greater than the threshold value is recorded as C_v(d) In that respect For an effective depth d, the correlation value corr (d) is calculated as C_v(d) The NCC mean value of all images in (1)

Wherein | | | C_v(d) Calculated | |, C_v(d) For each pixel p in R, the depth is chosen to be the value of d that maximizes corr (d), otherwise there is no valid value of d, and the confidence of each recovered depth value is expressed as

Where φ represents a threshold value, which can be set according to actual use conditions, and is generally set to about 0.6.

(2) The results are stitched into the reconstructed model. In this step, the result set of confidence weighted depth maps is merged using a volumetric method, which outputs a triangular mesh with confidence for each vertex. After obtaining an incomplete set of depth maps, they may be merged into a single surface mesh representation using a volumetric method that converts each depth map into weighted signed distance volumes, sums the volumes, and extracts the surface set to zero. After the multi-view dense reconstruction is performed on the image, the reconstruction model can be more complete only by filtering and texturing the reconstruction result, and it should be noted that in consideration of the task requirements of the three-dimensional reconstruction, the pose estimation and capture are performed on the failed satellite or the fragment through the three-dimensional reconstruction, that is, the task has higher requirements on the shape of the observed object and has no requirements on the visual appearance, the filtering processing is only performed on the reconstruction grid instead of texturing mapping, but the picture input in the resampling process is considered, so the texturing step cannot be omitted.

And sequentially carrying out grid filtering and texturing on the three-dimensional model grid model to obtain a three-dimensional reconstruction model.

After a complete dense grid model is obtained by a multi-view reconstruction method, a large amount of noise still exists in the grid model, so that the grid needs to be smoothed by a grid denoising algorithm. The embodiment of the application adopts a bilateral filtering algorithm to perform denoising and smoothing.

For image i (u), bilateral filtering is defined as:

where n (u) is the neighborhood of vertex u ═ x, y, and the spatial domain kernel is expressed as a standard deviation σ_cStandard gaussian filtering of

Value domain kernel is expressed as a standard deviation of σ_sStandard gaussian filtering of

One filtering iteration is completed after updating the values of all the vertices in the mesh.

The texturing process first entails determining the visibility of faces in the input image, where a pair-wise Markov random field energy formula is used to compute a label l that will look at l_iAssigned to each mesh plane F_iUsed as texture:

wherein the data item E_dataTend to use good input images to texture faces; smoothness term E_smoothMinimizing visibility of the different picture textured stitching seams; the minimization of E (l) is achieved by graph cut and alpha expansion algorithms.

After the label l is obtained by minimizing the Markov random field energy formula, the color block color also needs to be adjusted. First, it should be guaranteed that each mesh vertex belongs to only one texture block. Thus, each vertex at the seam needs to be duplicated into two vertices, i.e.Texture block vertex v on the left side of the seam_leftAnd texture Block vertex v on the right side of the seam_rightWhen each vertex has only one color f before adjustment_v. Next, an accumulated correction value g is calculated for each vertex by minimizing the following expression_v：

Wherein v includes v_leftAnd v_right，v_iAnd v_jAdjacent and in the same texture block, λ is a parameter to be constant. The first term in the above formula ensures the left side of the seam

To the right side of the joint

The adjusted colors are as similar as possible and the second term is used to minimize the difference in adjustment between adjacent vertices within the same texture block, which facilitates gradual adjustment within a texture block. When finding the optimal g for all vertices_vThen, the correction value of each texel is determined from the g of the vertices around the texel through the centroid coordinate system_vAnd (4) inserting. Finally, the correction values are added to the input image, the texture blocks are packed into a texture map set, and the texture coordinates are attached to the vertices.

In another embodiment of the present application, after obtaining the three-dimensional reconstruction model of the space debris or the failed satellite, considering the complexity of the space environment and the uncertainty of the motion of the observed object, such as incomplete model reconstruction caused by environmental factors such as long distance, partial occlusion, local flare or too strong shadow interference, the present embodiment adds a prediction feedback repair step according to the actual situation.

For the three-dimensional reconstruction of the general method, if the accuracy of the observed image is to be increased, the number of input pictures must be increased, but the embodiment is based on the image restoration technology, so the number of input pictures can be increased by using the restoration algorithm on the premise of not increasing the number of taken pictures. In addition, considering the complexity of the actual situation, for a satellite or a spatial fragment body rotating around a single axis in space, more useful picture data cannot be acquired within a specific time, so that the prediction feedback patching of an invisible area or an unfinished reconstruction part by using the image processing method is more practical. The specific scheme is as follows:

generating a second image sequence according to the three-dimensional reconstruction model; the second image sequence corresponds to the images in the space monocular image sequence and is equal in number, and the two corresponding images have the same shooting angle with respect to the space debris or the failed satellite.

And when the second image sequence is generated according to the three-dimensional reconstruction model, generating the second image sequence by respectively adopting a pure black background and a pure white background. The generation method of each image in the second image sequence comprises the following steps: determining a sampling frame of the two-dimensional image; sequentially carrying out sampling scanning on the three-dimensional reconstruction model by using a sampling frame to generate a plurality of fragment images; and recombining the plurality of fragment images according to the sampling scanning sequence to obtain a sampling image.

Calculating the defect degree of each image in the second image sequence; the method specifically comprises the following steps: acquiring a pure black background image and a pure white background image; converting the pure black background image and the pure white background image into gray level images; generating a vacancy distinguishing matrix according to the gray value of each pixel point in the gray image of the pure white background image and the gray value of each pixel point in the gray image of the pure black background image; and generating the defect degree of each image according to the vacancy distinguishing matrix.

When the defect degree is larger than a first preset threshold value, acquiring a space monocular image corresponding to a defect image in the second image sequence in the space monocular image sequence; and repairing the corresponding space monocular image by adopting the trained generated confrontation network to obtain a repaired space monocular image sequence. And replacing the space monocular image sequence with the repaired space monocular image sequence, and continuing to execute. The replacement process here is to select a defective monocular image in the original spatial monocular image sequence, to repair the designated part of the defective monocular image by generating an antagonistic network, and to insert the repaired image into the spatial monocular image sequence after the repair, at this time, the spatial monocular image with the defect will be deleted from the sequence.

The repairing process specifically comprises the following steps: segmenting the vacancy distinguishing matrix according to the plurality of fragment images to obtain a plurality of fragment matrixes; calculating the sum value of each element in each fragment matrix; marking the image area corresponding to the fragment matrix with the sum value larger than a second preset threshold value as an area to be repaired; and repairing the area to be repaired by adopting the trained generated confrontation network pair.

According to the above method, in the present embodiment, as shown in fig. 2, the secondary reconstruction process is specifically described as follows:

(1) and (3) carrying out re-acquisition of converting the three-dimensional model into a two-dimensional picture on the model by segmenting the reconstructed model, and respectively carrying out two-time acquisition on the image by using a pure black background and a pure white background.

(2) And selecting a sampling kernel with a proper size, and rescanning and sampling the picture (the picture of the previous step) in a convolution-like mode.

(3) Calculating the vacancy or fracture area of the image region acquired by the sampling core, and repairing the region by using a generated countermeasure network when the fracture and defect area in the region is larger than a certain threshold value;

(4) rotating a specific angle, and repeating the collecting and repairing steps;

(5) feeding the restored image back to the sampling picture set, and reconstructing the image set by an incremental motion restoration structure method, wherein fig. 8 is a schematic diagram of the incremental motion restoration structure method.

The purpose of segmenting the model in the step (1) is to prevent the points on other surfaces from interfering with the vacancy on the sampling plane when generating the two-dimensional picture, because when the incomplete three-dimensional plane regenerates the two-dimensional image, the vacancy in front is filled by the complete model in the back, and the acquisition in the step (3) is interfered. The size of the sampling core in the step (2) is adaptive to the picture, the threshold value and the task requirement.

The selection of the threshold value is related to the integrity evaluation of image reconstruction, so that the threshold value is the key of the whole feedback restoration link, and after the threshold value gamma is determined according to task requirements, whether the image needs to be repaired can be judged. The size of the resampled picture is set to be a multiplied by b dimension, firstly, the sampled RGB picture is converted into a gray image, and the formula is as follows:

wherein, I (x, y) represents the gray value of a pixel point on the generated gray image, I_R(x,y)、I_G(x, y) and I_B(x, y) represent the values of the original RGB image at R, G, B, respectively. In addition, in the two acquisitions, because a pure white background and a pure black background are adopted, the gray values of the two images in the vacant part have jump, when white is used, the gray value is 255, and when black is used, the gray value is 0. The gray value of the black background resampling image is subtracted from the white background resampling image to obtain a vacancy distinguishing matrix

The size of the three-dimensional model is the same as that of the resampled picture, the elements are 0 and 255, and the part with the value of 255 is the vacant part of the three-dimensional model. For convenience of calculation, the matrix is aligned

And (3) carrying out normalization treatment:

and obtaining a normalized vacancy judging matrix P.

The size of the sampling kernel is taken as i x i, expressed as

Marking the I multiplied by I order discrimination matrix corresponding to the sampling kernel in the image I as P_kIt can be known that

The image inpainting discriminant can be written as:

wherein ⊙ is Hadamard product,

represents P_k⊙C_kThe sum of the elements in the result matrix of (a),

and the sum of the normalized values which represent the restoration needs to be carried out in the whole collection picture, and when the sum is larger than the threshold value gamma, the picture needs to be restored. In addition, if the size i of the kernel is set to the pixel size of the whole picture, the picture can be directly judged, and in this case, the picture can be judged

If the picture needs to be repaired, the specific position needed by the image needs to be determined, and the position determination can be performed through sampling of the sampling kernel. Firstly, the method is approximately the same as the whole image judging flow, and the sampling core is used for scanning and calculating the gray value of the image

Determining a region determination threshold γ for the determination region of each kernel_kWhen is coming into contact with

Then, the position record of the judgment area is stored as an area t_jWherein j is 1,2, and n is a storage area number, and after the scanning of the whole picture is finished, the recorded area t is subjected to scanning_jSet of constituents T_jRepair one by one, where t_j∈T_j。

When the size of the sampling kernel and the size of the effective area of the picture are not in integral multiple relation, a 0 complementing method can be adopted, and the part beyond the boundary is filled with 0, because 0 in the matrix indicates that no vacancy exists in the image, the part beyond the boundary can be regarded as no image hole by the algorithm through the 0 complementing method, and therefore the weight at the boundary is reduced.

In this embodiment, the apparatus further includes:

the second generation module also comprises a pure black background generation submodule and a pure white background generation submodule; and the pure black background generation submodule and the pure white background generation submodule are used for generating a second image sequence by respectively adopting a pure black background and a pure white background when generating the second image sequence according to the three-dimensional reconstruction model.

Another embodiment of the invention: a three-dimensional reconstruction modeling device of a space moving object comprises a memory, a processor and a computer program which is stored in the memory and can run on the processor, wherein the processor executes the computer program to realize the three-dimensional reconstruction modeling method of the space moving object.

A three-dimensional reconstruction model establishing device of a space moving object can be computing devices such as a desktop computer, a notebook computer, a palm computer and a cloud server. The three-dimensional reconstruction modeling device for the space moving object can comprise, but is not limited to, a processor and a memory. It will be appreciated by those skilled in the art that the above description is merely an example of a three-dimensional reconstruction modeling apparatus for a spatially moving object and does not constitute a limitation of a three-dimensional reconstruction modeling apparatus for a spatially moving object, and may include more or less components than those shown, or some components in combination, or different components, such as an input-output device, a network access device, etc.

The Processor may be a Central Processing Unit (CPU), or other general purpose Processor, a Digital Signal Processor (DSP), an Application Specific Integrated Circuit (ASIC), an off-the-shelf Programmable Gate Array (FPGA) or other Programmable logic device, discrete Gate or transistor logic, discrete hardware components, etc. A general purpose processor may be a microprocessor or the processor may be any conventional processor or the like.

The memory may in some embodiments be an internal storage unit of a three-dimensional reconstruction modeling apparatus for a spatially moving object, such as a hard disk or a memory of a three-dimensional reconstruction modeling apparatus for a spatially moving object. The memory may be an external storage device of the three-dimensional reconstruction model building device of the space moving object in other embodiments, such as a plug-in hard disk, a Smart Memory Card (SMC), a Secure Digital (SD) Card, a Flash memory Card (Flash Card), and the like provided on the three-dimensional reconstruction model building device of the space moving object. Further, the memory may also include both an internal storage unit and an external storage device of a three-dimensional reconstruction modeling apparatus for a spatially moving object. The memory is used for storing an operating system, application programs, a BootLoader (BootLoader), data, and other programs, such as program codes of computer programs. The memory may also be used to temporarily store data that has been output or is to be output.

The embodiments of the present application further provide a computer-readable storage medium, where a computer program is stored, and when the computer program is executed by a processor, the steps in the above-mentioned method embodiments may be implemented.

The embodiments of the present application further provide a computer program product, which when running on a three-dimensional reconstruction model building apparatus for a spatial moving object, enables the three-dimensional reconstruction model building apparatus for the spatial moving object to implement the steps in the above-mentioned method embodiments when executed.

The integrated unit, if implemented in the form of a software functional unit and sold or used as a stand-alone product, may be stored in a computer readable storage medium. Based on such understanding, all or part of the processes in the methods of the embodiments described above can be implemented by a computer program, which can be stored in a computer-readable storage medium and can implement the steps of the embodiments of the methods described above when the computer program is executed by a processor. Wherein the computer program comprises computer program code, which may be in the form of source code, object code, an executable file or some intermediate form, etc. The computer readable medium may include at least: any entity or apparatus capable of carrying computer program code to a terminal device, recording medium, computer Memory, Read-Only Memory (ROM), Random-Access Memory (RAM), electrical carrier wave signals, telecommunications signals, and software distribution medium. Such as a usb-disk, a removable hard disk, a magnetic or optical disk, etc. In certain jurisdictions, computer-readable media may not be an electrical carrier signal or a telecommunications signal in accordance with legislative and patent practice.

In the above embodiments, the descriptions of the respective embodiments have respective emphasis, and reference may be made to the related descriptions of other embodiments for parts that are not described or illustrated in a certain embodiment.

Those of ordinary skill in the art will appreciate that the various illustrative modules and algorithm steps described in connection with the embodiments disclosed herein may be implemented as electronic hardware or combinations of computer software and electronic hardware. Whether such functionality is implemented as hardware or software depends upon the particular application and design constraints imposed on the implementation. Skilled artisans may implement the described functionality in varying ways for each particular application, but such implementation decisions should not be interpreted as causing a departure from the scope of the present application.

In the embodiments provided in the present application, it should be understood that the disclosed apparatus/device and method may be implemented in other ways. For example, the above-described apparatus/device embodiments are merely illustrative, and for example, a division of modules is merely a division of logical functions, and an actual implementation may have another division, for example, a plurality of modules may be combined or integrated into another system, or some features may be omitted, or not executed.

Modules described as separate components may or may not be physically separate, and modules may or may not be physical units, i.e., may be located in one place, or may be distributed over a plurality of network modules. Some or all of the modules may be selected according to actual needs to achieve the purpose of the solution of the present embodiment.

Claims

1. A method for establishing a three-dimensional reconstruction model of a space moving object is characterized by comprising the following steps:

training the spatial monocular image sequence by adopting a trained generation countermeasure network so as to eliminate the flare and the shadow of each spatial monocular image in the spatial monocular image sequence and generate a first image sequence;

extracting a characteristic value matching relation between each image and other images in the first image sequence by a scale-invariant characteristic transformation method;

2. The method for building a three-dimensional reconstruction model of a space moving object according to claim 1, wherein after obtaining the three-dimensional reconstruction model of the space debris or the failed satellite, the method further comprises:

generating a second image sequence according to the three-dimensional reconstruction model; the second image sequence corresponds to the images in the space monocular image sequence and is equal in number, and the corresponding images are the same in shooting angle of the space debris or the failed satellite;

calculating the defect degree of each image in the second image sequence;

adopting a trained generated confrontation network to repair the corresponding space monocular image to obtain a repaired space monocular image sequence;

3. The method as claimed in claim 2, wherein the generating of the second image sequence based on the three-dimensional reconstructed model is performed by using a black background and a white background.

4. A method for modeling a three-dimensional reconstruction of an object with spatial motion according to claim 2 or 3, wherein each image of said second sequence of images is generated by:

determining a sampling frame of the two-dimensional image;

sequentially carrying out sampling scanning on the three-dimensional reconstruction model by using the sampling frame to generate a plurality of fragment images;

5. The method of claim 4, wherein calculating the defect level of each image in the second sequence of images comprises:

acquiring a pure black background image and a pure white background image;

6. The method as claimed in claim 5, wherein the repairing the corresponding spatial monocular image using the trained generative confrontation network comprises:

calculating the sum value of each element in each fragment matrix;

and repairing the area to be repaired by adopting the trained pair of generated confrontation networks.

7. An apparatus for modeling a three-dimensional reconstruction of a spatially moving object, comprising:

and the first generation module is used for generating a three-dimensional model grid model according to the three-dimensional model sparse point cloud, and sequentially carrying out grid filtering and texturing on the three-dimensional model grid model to obtain a three-dimensional reconstruction model of a space debris or a failure satellite.

8. The apparatus for modeling a three-dimensional reconstruction of an object moving in space of claim 7, further comprising:

the second generation module is used for generating a second image sequence according to the three-dimensional reconstruction model after the three-dimensional reconstruction model of the space debris or the failure satellite is obtained; the second image sequence corresponds to the images in the space monocular image sequence and is equal in number, and the corresponding images are the same in shooting angle of the space debris or the failed satellite;

the restoration module is used for restoring the corresponding space monocular image by adopting the trained generation countermeasure network to obtain a restored space monocular image sequence;

and the replacing module is used for replacing the space monocular image sequence by adopting the repaired space monocular image sequence and continuing to execute.

9. The apparatus according to claim 8, wherein the second generating module further comprises a pure black background generating sub-module and a pure white background generating sub-module;

10. An apparatus for three-dimensional reconstruction modeling of a spatially moving object, comprising a memory, a processor and a computer program stored in the memory and executable on the processor, characterized in that the processor implements a method for three-dimensional reconstruction modeling of a spatially moving object according to any of claims 1 to 6 when executing the computer program.