CN113099210A - Three-dimensional image restoration method and device, computer equipment and storage medium - Google Patents
Three-dimensional image restoration method and device, computer equipment and storage medium Download PDFInfo
- Publication number
- CN113099210A CN113099210A CN202110349391.8A CN202110349391A CN113099210A CN 113099210 A CN113099210 A CN 113099210A CN 202110349391 A CN202110349391 A CN 202110349391A CN 113099210 A CN113099210 A CN 113099210A
- Authority
- CN
- China
- Prior art keywords
- dimensional image
- image
- dimensional
- restoration
- neural network
- Prior art date
- Legal status (The legal status is an assumption and is not a legal conclusion. Google has not performed a legal analysis and makes no representation as to the accuracy of the status listed.)
- Granted
Links
- 238000000034 method Methods 0.000 title claims abstract description 45
- 238000013528 artificial neural network Methods 0.000 claims abstract description 58
- 238000012545 processing Methods 0.000 claims abstract description 36
- 238000004590 computer program Methods 0.000 claims description 25
- 230000009467 reduction Effects 0.000 claims description 10
- 238000004422 calculation algorithm Methods 0.000 description 9
- 238000012549 training Methods 0.000 description 9
- 238000010586 diagram Methods 0.000 description 8
- 230000008569 process Effects 0.000 description 6
- 238000004891 communication Methods 0.000 description 5
- 239000002131 composite material Substances 0.000 description 4
- 238000005516 engineering process Methods 0.000 description 4
- 238000011084 recovery Methods 0.000 description 4
- 238000005259 measurement Methods 0.000 description 3
- 238000011161 development Methods 0.000 description 2
- 230000000903 blocking effect Effects 0.000 description 1
- 238000004364 calculation method Methods 0.000 description 1
- 238000001514 detection method Methods 0.000 description 1
- 238000000605 extraction Methods 0.000 description 1
- 238000001914 filtration Methods 0.000 description 1
- 230000003993 interaction Effects 0.000 description 1
- 239000004973 liquid crystal related substance Substances 0.000 description 1
- 238000012986 modification Methods 0.000 description 1
- 230000004048 modification Effects 0.000 description 1
- 238000003062 neural network model Methods 0.000 description 1
- 230000003287 optical effect Effects 0.000 description 1
- 238000005457 optimization Methods 0.000 description 1
- 230000000306 recurrent effect Effects 0.000 description 1
- 230000003068 static effect Effects 0.000 description 1
Images
Classifications
-
- H—ELECTRICITY
- H04—ELECTRIC COMMUNICATION TECHNIQUE
- H04N—PICTORIAL COMMUNICATION, e.g. TELEVISION
- H04N13/00—Stereoscopic video systems; Multi-view video systems; Details thereof
- H04N13/20—Image signal generators
- H04N13/275—Image signal generators from 3D object models, e.g. computer-generated stereoscopic image signals
-
- H—ELECTRICITY
- H04—ELECTRIC COMMUNICATION TECHNIQUE
- H04N—PICTORIAL COMMUNICATION, e.g. TELEVISION
- H04N13/00—Stereoscopic video systems; Multi-view video systems; Details thereof
- H04N13/10—Processing, recording or transmission of stereoscopic or multi-view image signals
-
- H—ELECTRICITY
- H04—ELECTRIC COMMUNICATION TECHNIQUE
- H04N—PICTORIAL COMMUNICATION, e.g. TELEVISION
- H04N13/00—Stereoscopic video systems; Multi-view video systems; Details thereof
- H04N13/20—Image signal generators
- H04N13/204—Image signal generators using stereoscopic image cameras
Landscapes
- Engineering & Computer Science (AREA)
- Multimedia (AREA)
- Signal Processing (AREA)
- Image Analysis (AREA)
- Image Processing (AREA)
Abstract
The application relates to a three-dimensional image restoration method, a three-dimensional image restoration device, computer equipment and a storage medium. The method comprises the following steps: by acquiring a first three-dimensional image; inputting the first three-dimensional image into a preset image restoration neural network for restoration processing to obtain a second three-dimensional image; then judging whether the first three-dimensional image has an occluded part or a shadow part according to the first three-dimensional image and the second three-dimensional image; and if the first three-dimensional image has the blocked part or the shadow part, outputting a second three-dimensional image. By adopting the method, the three-dimensional image of the target can be restored, whether the occlusion or the shadow exists in the real three-dimensional image is judged, and then when the occlusion or the shadow exists in the real three-dimensional image is judged, the three-dimensional image is selected to be outputted and restored according to the judgment result, so that the aim of improving the three-dimensional image restoration precision is fulfilled.
Description
Technical Field
The present application relates to the field of three-dimensional image processing technologies, and in particular, to a three-dimensional image restoration method and apparatus, a computer device, and a storage medium.
Background
With the development of hardware such as depth cameras, laser sensors, image processors and the like and the development of depth neural networks, three-dimensional vision technology has been widely applied to assisting intelligent driving of automobiles, service robots and industrial robots. Because these robot systems are all in a dynamic scene, these systems often suffer from problems of occlusion, out-of-view range and light variation during the moving process, and the occlusion may cause the loss of target image information, which may cause errors or failures in the subsequent identification, detection and positioning; beyond the field of view also causes loss of image information; light variations can introduce errors into the target image information.
In the conventional technology, the missing contour of a three-dimensional object model is usually repaired by using an Euler circular curve method, and the method mainly aims at repairing contour information and cannot solve the problem of low image restoration precision caused by shielding.
Disclosure of Invention
In view of the above, it is necessary to provide a three-dimensional image restoration method, an apparatus, a computer device, and a storage medium capable of improving image restoration accuracy.
A three-dimensional image restoration method comprises the following steps:
acquiring a first three-dimensional image;
inputting the first three-dimensional image into a preset image restoration neural network for restoration processing to obtain a second three-dimensional image;
judging whether the first three-dimensional image has an occluded part or a shadow part according to the first three-dimensional image and the second three-dimensional image;
and if the first three-dimensional image has the blocked part or the shadow part, outputting a second three-dimensional image.
In one embodiment, acquiring a first three-dimensional image comprises:
acquiring an image of a photographic subject;
and carrying out three-dimensional reconstruction on the image of the shot object to obtain a first three-dimensional image.
In one embodiment, three-dimensionally reconstructing an image of a photographic subject to obtain a first three-dimensional image includes:
acquiring three-dimensional point cloud information of a shooting object according to an image of the shooting object;
and performing three-dimensional reconstruction on the image of the shot object according to the three-dimensional point cloud information of the shot object to obtain a first three-dimensional image.
In one embodiment, the preset image restoration neural network is obtained by the following method:
obtaining an initial sample image, cutting a random area of the initial sample image, and assigning the cut area to be black to obtain a sample image;
inputting a sample image into an initial image restoration neural network, and processing the sample image through a convolution attention layer, a three-dimensional convolution network layer and a three-dimensional deconvolution network layer to obtain a restored sample image;
and adjusting the network weight of the initial image restoration neural network according to the restoration sample image and the initial sample image to obtain a preset image restoration neural network.
In one embodiment, determining whether the first three-dimensional image has an occluded part or a shadow part according to the first three-dimensional image and the second three-dimensional image comprises:
respectively acquiring related information of the first three-dimensional image and the second three-dimensional image, wherein the related information is color information or histogram information;
obtaining a relevant information difference between the first three-dimensional image and the second three-dimensional image according to the relevant information of the first three-dimensional image and the second three-dimensional image;
and judging whether the first three-dimensional image has an occluded part or a shadow part according to the related information difference between the first three-dimensional image and the second three-dimensional image.
In one embodiment, the determining whether the first three-dimensional image has an occluded part or a shadow part according to the difference between the related information of the first three-dimensional image and the second three-dimensional image includes:
acquiring a related information difference range threshold corresponding to related information according to the type of the related information;
if the relevant information difference between the first three-dimensional image and the second three-dimensional image is in the relevant information difference range threshold value, judging that an occluded part or a shadow part does not exist in the three-dimensional image;
and if the relevant information difference between the first three-dimensional image and the second three-dimensional image is not in the relevant information difference range threshold value, judging that the blocked part or the shadow part exists in the three-dimensional image.
In one embodiment, the method further comprises:
and if the three-dimensional image has no shielded part and no shadow part, outputting the three-dimensional image.
A three-dimensional image restoration apparatus, the apparatus comprising:
the image acquisition module is used for acquiring a first three-dimensional image;
the image restoration module is used for inputting the first three-dimensional image into a preset image restoration neural network for restoration processing to obtain a second three-dimensional image;
the image judging module is used for judging whether the first three-dimensional image has a blocked part or a shadow part according to the first three-dimensional image and the second three-dimensional image;
and the image output module is used for outputting a second three-dimensional image if the first three-dimensional image has a blocked part or a shadow part.
A computer device comprising a memory and a processor, the memory storing a computer program, the processor implementing the following steps when executing the computer program:
acquiring a first three-dimensional image;
inputting the first three-dimensional image into a preset image restoration neural network for restoration processing to obtain a second three-dimensional image;
judging whether the first three-dimensional image has an occluded part or a shadow part according to the first three-dimensional image and the second three-dimensional image;
and if the first three-dimensional image has the blocked part or the shadow part, outputting a second three-dimensional image.
A computer-readable storage medium, on which a computer program is stored which, when executed by a processor, carries out the steps of:
acquiring a first three-dimensional image;
inputting the first three-dimensional image into a preset image restoration neural network for restoration processing to obtain a second three-dimensional image;
judging whether the first three-dimensional image has an occluded part or a shadow part according to the first three-dimensional image and the second three-dimensional image;
and if the first three-dimensional image has the blocked part or the shadow part, outputting a second three-dimensional image.
The three-dimensional image restoration method, the three-dimensional image restoration device, the computer equipment and the storage medium acquire the first three-dimensional image; inputting the first three-dimensional image into a preset image restoration neural network for restoration processing to obtain a second three-dimensional image; then judging whether the first three-dimensional image has an occluded part or a shadow part according to the first three-dimensional image and the second three-dimensional image; and if the first three-dimensional image has an occluded part or a shadow part, outputting the second three-dimensional image. According to the scheme, the three-dimensional image of the target can be restored, whether the real three-dimensional image has the shielding or the shadow or not is judged, and then when the real three-dimensional image has the shielding or the shadow, the restored three-dimensional image is selected to be output according to the judgment result, so that the purpose of improving the three-dimensional image restoration precision is achieved.
Drawings
FIG. 1 is a schematic flow chart of a three-dimensional image restoration method according to an embodiment;
FIG. 2 is a schematic flow chart illustrating the acquisition of a three-dimensional image according to one embodiment;
FIG. 3 is a schematic diagram illustrating a process of establishing a predetermined image recurrent neural network according to an embodiment;
FIG. 4 is a schematic diagram illustrating an embodiment of a predetermined image reduction neural network;
FIG. 5 is a flow diagram illustrating the determination of whether an occluded or shadowed portion exists in one embodiment;
FIG. 6 is a schematic flow chart illustrating the continuous generation of a restored three-dimensional image according to an embodiment;
FIG. 7 is a schematic illustration of a three-dimensional image reconstruction scenario in an industrial environment, in accordance with an embodiment;
FIG. 8 is a block diagram illustrating an exemplary embodiment of a three-dimensional image restoration apparatus;
FIG. 9 is a diagram illustrating an internal structure of a computer device according to an embodiment.
Detailed Description
In order to make the objects, technical solutions and advantages of the present application more apparent, the present application is described in further detail below with reference to the accompanying drawings and embodiments. It should be understood that the specific embodiments described herein are merely illustrative of the present application and are not intended to limit the present application.
In an embodiment, as shown in fig. 1, a three-dimensional image restoration method is provided, and this embodiment is illustrated by applying the method to a terminal, and it is to be understood that the method may also be applied to a server, and may also be applied to a system including a terminal and a server, and is implemented by interaction between the terminal and the server. In this embodiment, the method includes the steps of:
step 102, a first three-dimensional image is acquired.
Specifically, the processor acquires an image and establishes a first three-dimensional image for the image, wherein the first three-dimensional image is a real three-dimensional image. In computer vision, creating a three-dimensional image refers to the process of reconstructing three-dimensional information from single-view or multi-view images. The information of a single view is incomplete, so empirical knowledge is required for three-dimensional reconstruction from a single view. The method is that the camera is calibrated, namely the relation between the image coordinate system of the camera and the world coordinate system is calculated, and then three-dimensional information is reconstructed by utilizing the information in a plurality of two-dimensional images.
And 104, inputting the first three-dimensional image into a preset image reduction neural network for reduction processing to obtain a second three-dimensional image.
Specifically, the real three-dimensional image is input into a preset image restoration neural network, and the preset image restoration neural network outputs a second three-dimensional image, wherein the second three-dimensional image is a restored three-dimensional image.
The preset image recovery neural network is obtained by training according to a training set extracted from a sample three-dimensional image, the more samples of the training set are, the higher the precision of the preset image recovery neural network after training is completed is, and the more accurate the recovery three-dimensional image output by the preset image recovery neural network is. The preset image reduction neural network can be formed by a single neural network, such as a three-dimensional convolution network; or a composite neural network composed of a plurality of neural networks, such as a composite neural network composed of a plurality of three-dimensional convolution networks and a three-dimensional deconvolution network.
And 106, judging whether the first three-dimensional image has an occluded part or a shadow part according to the first three-dimensional image and the second three-dimensional image.
Specifically, the restored three-dimensional image and the real three-dimensional image are compared, and whether a blocked part or a shadow part exists in the third real three-dimensional image is judged by adopting a blocking judgment algorithm.
At present, various occlusion determination algorithms exist, such as an iteration number determination method: estimating the position of the current frame image in the next frame by using a Kalman filtering algorithm to obtain an estimated position, setting a tracking window at the estimated position, searching the position which is most similar to a missile target template in the tracking window by using a Mean Shift algorithm as the accurate position of the missile target, judging whether the target is shielded or lost according to the iteration times of the Mean Shift algorithm, limiting the iteration times of the algorithm, and if the target is found in the limited times, directly giving the position without loss; if the target is not found within the limited times, the shielding or the loss occurs, and the window width is expanded for searching again. Also for example, similarity determination method: for the k frame image, a threshold value Th is given, if the Bhattacharyya coefficient rho (yk) is larger than or equal to Th, no occlusion exists, and normal tracking is carried out; otherwise if rho (yk) < Th, it means occlusion; also for example, residual decision methods: in the current frame, whether the occlusion with large proportion appears is judged according to the size of the residual error between the estimated value of the Kalman filter about the target position and the measured value of the Kalman filter obtained by the Mean Shift algorithm. In this embodiment, a suitable occlusion determination algorithm is selected according to different implementation scenarios.
And step 108, outputting a second three-dimensional image if the first three-dimensional image has a blocked part or a shadow part.
Specifically, if there is an occluded part or a shadow part in the real three-dimensional image, the real three-dimensional image needs to be restored, and a three-dimensional image without occlusion and shadow is generated, so that the restored three-dimensional image is output. And if the blocked part or the shadow part does not exist in the real three-dimensional image, directly outputting the real three-dimensional image without restoring the real three-dimensional image.
The three-dimensional image restoration method comprises the steps of obtaining a first three-dimensional image; inputting the first three-dimensional image into a preset image restoration neural network for restoration processing to obtain a second three-dimensional image; then judging whether the first three-dimensional image has an occluded part or a shadow part according to the first three-dimensional image and the second three-dimensional image; and if the first three-dimensional image has the blocked part or the shadow part, outputting a second three-dimensional image. According to the scheme, the three-dimensional image of the target can be restored, whether the real three-dimensional image has the shielding or the shadow or not is judged, and then when the real three-dimensional image has the shielding or the shadow, the restored three-dimensional image is selected to be output according to the judgment result, so that the purpose of improving the three-dimensional image restoration precision is achieved.
In one embodiment, acquiring a first three-dimensional image comprises: acquiring an image of a photographic subject; and carrying out three-dimensional reconstruction on the image of the shot object to obtain a first three-dimensional image.
Further, three-dimensionally reconstructing an image of the object to be photographed to obtain a first three-dimensional image includes: acquiring three-dimensional point cloud information of a shooting object according to an image of the shooting object; and performing three-dimensional reconstruction on the image of the shot object according to the three-dimensional point cloud information of the shot object to obtain a first three-dimensional image.
In one embodiment, as shown in FIG. 2, acquiring a first three-dimensional image includes:
step 202, acquiring an image of a photographic subject.
Specifically, images at successive moments are acquired of the photographic object by the image acquisition device. The image acquisition device may be a camera or a camera. The camera is carried on a robot system, the robot system completes multi-target recognition through information collected by the camera, and the robot system can refer to intelligent driving automobiles, service robots, industrial robots and the like; the photographic subject may refer to an automobile, a person, an industrial part, or the like. The lighting conditions, the geometric characteristics of the camera, etc. have a great influence on the subsequent image processing.
And 204, acquiring three-dimensional point cloud information of the shooting object according to the image of the shooting object.
Specifically, the three-dimensional point cloud information is also called three-dimensional point cloud data. In some cases, the three-dimensional point cloud information also includes color image information.
And step 206, performing three-dimensional reconstruction on the image of the shot object according to the three-dimensional point cloud information of the shot object to obtain a first three-dimensional image.
Specifically, three-dimensional reconstruction is performed on the shot object based on three-dimensional point cloud information of the shot object at different moments, and global point cloud information is obtained. And then selecting a local area with an overlapping area in an area corresponding to the global point cloud information for measurement, acquiring the local point cloud information, registering and updating the global point cloud information, repeating the process until the measurement of all the surface areas is completed, and finally performing global optimization processing on the updated global point cloud data after the measurement is completed to obtain a point cloud model.
Optionally, the three-dimensional reconstruction is performed according to the image of the shot object, and the obtaining of the reconstructed three-dimensional image generally includes camera calibration, feature extraction, stereo matching and three-dimensional reconstruction.
In one embodiment, as shown in fig. 3, the preset image restoration neural network is obtained by:
step 302, obtaining an initial sample image, performing random area clipping on the initial sample image, and assigning the clipped area to black to obtain a sample image.
Specifically, a training set is obtained, wherein the training set at least comprises a sample image subjected to random area clipping.
And step 304, inputting the sample image into an initial image restoration neural network, and processing the sample image through a convolution attention layer, a three-dimensional convolution network layer and a three-dimensional deconvolution network layer to obtain a restored sample image.
Specifically, the image restoration neural network may be formed by a single neural network model, such as a three-dimensional convolution network; or a composite neural network composed of a plurality of neural networks, such as a composite neural network composed of a plurality of three-dimensional convolution networks and a three-dimensional deconvolution network.
And step 306, adjusting the network weight of the initial image restoration neural network according to the restoration sample image and the initial sample image to obtain a preset image restoration neural network.
Specifically, the image restoration neural network is obtained by training according to a training set extracted from the sample image.
For example, the image restoration neural network comprises an input layer, at least one attention layer, at least one three-dimensional convolution network layer, at least one three-dimensional deconvolution network layer and an output layer which are sequentially connected in series; the attention layer is used for performing convolution attention calculation on the received image features to obtain the attention map image features. And the three-dimensional convolution network layer and the three-dimensional deconvolution network layer are used for carrying out three-dimensional image restoration according to the received image characteristics.
As shown in fig. 4, which is a network topology diagram of an image restoration neural network in an embodiment, the image restoration neural network in fig. 4 sequentially connects in series an Input layer Input, a convolutional attention layer ConvAttention, a three-dimensional convolutional network layer Conv3D, a convolutional attention layer ConvAttention, 3 three-dimensional convolutional network layers Conv3D, 2 three-dimensional deconvolution network layers Conv3DT, a convolutional attention layer ConvAttention, a three-dimensional deconvolution network layer Conv3DT, a convolutional attention layer ConvAttention, 2 three-dimensional deconvolution network layers Conv3DT, and an Output layer Output. Fig. 4 includes the basic configuration of the network and the size of the processed tensor (the processed output of each layer in the network becomes the tensor). "2 × 64" means that the size of the convolution kernel is 2 × 2, the step size is 2, the number of convolution kernels is 64, and zero padding is required for each layer. For example, "[ l/2, w/2, h/2,64 ]" refers to the size of the processed tensor. The structure of the combination of the three-dimensional convolution network layer and the three-dimensional deconvolution network layer is a time sequence structure, so that the motion time sequence relation of the target three-dimensional image can be better learned. The convolution attention layer can promote the network to pay more attention to the main part of the three-dimensional image, and the network learning and three-dimensional image restoring capability is improved.
For example, assume that 1 image is input, the size of which is [ l, w, h, c ], l, w, h, c refer to length, width, height, and the number of channels, respectively. Obtaining tensor size [ l, w, h, c ] after connection and ConvAttention processing, obtaining tensor size [ l/2, w/2, h/2,64] after Conv3D processing with 64 Conv 2 convolution kernel step length of 2, obtaining tensor size [ l/4, w/4, h/2,64], obtaining tensor size [ l/8, w/8, h/38764 ] after Conv3D processing with ConvAttention processing and 128 convolution kernels, obtaining tensor size [ l/16, w/16, h/16,512] after Conv3D processing with 2 convolution kernel step length of 256 2, obtaining tensor size [ l/8, w/8, h/8,256] after Conv3D processing with 2 convolution kernel step length of 512, obtaining tensor size [ l/16, w/16, h/16,512] after Conv3DT processing with 2 kernel step length of 1024 2, obtaining tensor size [ l/8/368/w/78 ], the tensor size is [ l/4, w/4, h/4,512] obtained after being processed by Conv3DT with 512 2 × 2 convolution kernel step length of 2, the tensor size is [ l/2, w/2, h/2,256] obtained after Conv Attenttion processing and Conv3DT processing of 256 convolution kernels, the tensor size is [ l, w, h,128] obtained after being processed by Conv Attenttion processing and Conv3DT processing of 128 convolution kernels, the tensor size is [ l, w, h,128] obtained after being processed by c Conv3DT with 2 × 2 convolution kernel step length of 2, the tensor size is [ l, w, h, c ], and the tensor is output.
In one embodiment, the determining whether the first three-dimensional image has the occluded part or the shadow part according to the first three-dimensional image and the second three-dimensional image comprises: respectively acquiring related information of the first three-dimensional image and the second three-dimensional image, wherein the related information is color information or histogram information; obtaining a relevant information difference between the first three-dimensional image and the second three-dimensional image according to the relevant information of the first three-dimensional image and the second three-dimensional image; and judging whether the first three-dimensional image has an occluded part or a shadow part according to the related information difference between the first three-dimensional image and the second three-dimensional image.
Further, judging whether the first three-dimensional image has an occluded part or a shadow part according to the difference between the related information of the first three-dimensional image and the related information of the second three-dimensional image, includes: acquiring a related information difference range threshold corresponding to related information according to the type of the related information; if the relevant information difference between the first three-dimensional image and the second three-dimensional image is in the relevant information difference range threshold value, judging that an occluded part or a shadow part does not exist in the three-dimensional image; and if the relevant information difference between the first three-dimensional image and the second three-dimensional image is not in the relevant information difference range threshold value, judging that the blocked part or the shadow part exists in the three-dimensional image.
In one embodiment, as shown in fig. 5, the determining whether the first three-dimensional image has the occluded part or the shadow part according to the first three-dimensional image and the second three-dimensional image includes:
step 502, obtaining relevant information of the first three-dimensional image and the second three-dimensional image respectively, wherein the relevant information is color information or histogram information or other information.
Specifically, color information or histogram information of the real three-dimensional image and the restored three-dimensional image is acquired respectively.
And 504, obtaining the relevant information difference between the first three-dimensional image and the second three-dimensional image according to the relevant information of the first three-dimensional image and the second three-dimensional image.
Specifically, the color information difference or histogram information difference between the real three-dimensional image and the restored three-dimensional image is obtained according to the color information or histogram information of the real three-dimensional image and the restored three-dimensional image.
Step 506, according to the type of the related information, obtaining a related information difference range threshold corresponding to the related information.
Specifically, different related information difference thresholds are set according to the characteristics of the color information and the histogram information, if the color information is currently acquired, a color information difference threshold ± M% is set, and if the histogram information is currently acquired, a histogram information difference threshold ± N% is set, where M and N are not less than 3 in a normal case.
Step 508, if the relevant information difference between the first three-dimensional image and the second three-dimensional image is in the relevant information difference range threshold, determining that the three-dimensional image does not have an occluded part or a shadow part; and if the relevant information difference between the first three-dimensional image and the second three-dimensional image is not in the relevant information difference range threshold value, judging that the blocked part or the shadow part exists in the three-dimensional image.
Specifically, for example, if the color information difference between the real three-dimensional image and the restored three-dimensional image is within the color information difference threshold range, it is determined that there is no occluded part or shadow part in the real three-dimensional image; and if the color information difference between the real three-dimensional image and the restored three-dimensional image is not in the color information difference threshold range, judging that the blocked part or the shadow part exists in the real three-dimensional image.
In one embodiment, the method further comprises: and if the three-dimensional image has no shielded part and no shadow part, outputting the three-dimensional image.
Specifically, if the real three-dimensional image does not have the occluded part and the shadow part, the original image does not need to be restored, and therefore the real three-dimensional image is directly output.
In one embodiment, as shown in FIG. 6, there is provided a three-dimensional image restoration method, exemplified as applied to an industrial environment as shown in FIG. 7, comprising: an image of a workpiece on a conveyor belt is acquired. And performing three-dimensional reconstruction on the image of the workpiece to obtain a real three-dimensional image. Acquiring a real three-dimensional image, inputting the real three-dimensional image into a preset image reduction neural network for reduction processing to obtain a reduced three-dimensional image; judging whether the real three-dimensional image has a blocked part or a shadow part according to the real three-dimensional image and the restored three-dimensional image; and if the real three-dimensional image has a blocked part or a shadow part, outputting and restoring the three-dimensional image.
Specifically, the conveyor belt starts to run, and the automation device starts to place the workpiece on the conveyor belt. And starting the three-dimensional image restoration of the workpiece, acquiring the three-dimensional point cloud of the workpiece on the conveyor belt by using a three-dimensional camera on the industrial robot, and performing three-dimensional reconstruction on the three-dimensional point cloud to obtain a three-dimensional image i. And inputting the three-dimensional image i into a preset image restoration neural network to obtain a restored three-dimensional image i ', comparing i and i' according to a shielding judgment algorithm to judge whether the current workpiece state is in a shielding state, comparing color information of i and i ', if the difference value of the color information of i and i' is within the range of +/-M% (M is more than or equal to 3), determining that the workpiece is not in the shielding state, and outputting i. If the difference ratio of the color information of the two is in the range of +/-M% (M is larger than or equal to 3), the workpiece is considered to be in a shielding state, and i' is output. And judging whether to continue image acquisition, acquiring the next image if necessary, performing the restoration processing on the next image, and stopping the restoration of the three-dimensional image of the workpiece if not necessary.
In the embodiment, a real three-dimensional image is obtained; inputting the real three-dimensional image into a preset image restoration neural network for restoration processing to obtain a restored three-dimensional image; then judging whether the real three-dimensional image has a blocked part or a shadow part according to the real three-dimensional image and the restored three-dimensional image; and if the real three-dimensional image has a blocked part or a shadow part, outputting and restoring the three-dimensional image. According to the scheme, the three-dimensional image of the target can be restored, whether the real three-dimensional image has the shielding or the shadow or not is judged, and then when the real three-dimensional image has the shielding or the shadow, the restored three-dimensional image is selected to be output according to the judgment result, so that the purpose of improving the three-dimensional image restoration precision is achieved.
It should be understood that although the various steps in the flowcharts of fig. 1-3, 5-6 are shown in order as indicated by the arrows, the steps are not necessarily performed in order as indicated by the arrows. The steps are not performed in the exact order shown and described, and may be performed in other orders, unless explicitly stated otherwise. Moreover, at least some of the steps in fig. 1-3 and 5-6 may include multiple steps or multiple stages, which are not necessarily performed at the same time, but may be performed at different times, which are not necessarily performed in sequence, but may be performed alternately or alternately with other steps or at least some of the other steps.
In one embodiment, as shown in fig. 8, there is provided a three-dimensional image restoration apparatus 800 including: an image acquisition module 801, an image restoration module 802, an image judgment module 803 and an image output module 804,
wherein:
an image obtaining module 801, configured to obtain a first three-dimensional image.
The image restoration module 802 is configured to input the first three-dimensional image into a preset image restoration neural network for restoration processing, so as to obtain a second three-dimensional image.
And an image determining module 803, configured to determine whether an occluded part or a shadow part exists in the first three-dimensional image according to the first three-dimensional image and the second three-dimensional image.
The image output module 804 is configured to output a second three-dimensional image if the first three-dimensional image has an occluded part or a shadow part.
In one embodiment, the image acquisition module 801 includes:
and the image shooting submodule is used for acquiring an image of a shooting object.
And the three-dimensional reconstruction submodule is used for performing three-dimensional reconstruction on the image of the shot object to obtain a first three-dimensional image.
In one embodiment, the three-dimensional reconstruction sub-module further comprises:
and the three-dimensional information acquisition unit is used for acquiring the three-dimensional point cloud information of the shooting object according to the image of the shooting object.
And the three-dimensional image reconstruction unit is used for performing three-dimensional reconstruction on the image of the shot object according to the three-dimensional point cloud information of the shot object to obtain a first three-dimensional image.
In one embodiment, the image restoration module 802 includes:
and the sample obtaining submodule is used for obtaining an initial sample image, cutting a random area of the initial sample image, and assigning the cut area to be black to obtain the sample image.
And the model training submodule is used for inputting the sample image into an initial image reduction neural network and obtaining a reduced sample image through the processing of the convolution attention layer, the three-dimensional convolution network layer and the three-dimensional deconvolution network layer.
And the model generation submodule is used for adjusting the network weight of the initial image restoration neural network according to the restoration sample image and the initial sample image to obtain a preset image restoration neural network.
In one embodiment, the image determination module 803 includes:
and the related information acquisition sub-module is used for respectively acquiring related information of the first three-dimensional image and the second three-dimensional image, and the related information is color information or histogram information.
And the information difference obtaining submodule is used for obtaining the relevant information difference of the first three-dimensional image and the second three-dimensional image according to the relevant information of the first three-dimensional image and the second three-dimensional image.
And the information difference judging submodule is used for judging whether the first three-dimensional image has an occluded part or a shadow part according to the relevant information difference between the first three-dimensional image and the second three-dimensional image.
In one embodiment, the information difference determination sub-module further includes:
and the range threshold setting unit is used for acquiring a related information difference range threshold corresponding to the related information according to the type of the related information.
The range threshold value judging unit is used for judging that the blocked part or the shadow part does not exist in the three-dimensional image if the relevant information difference between the first three-dimensional image and the second three-dimensional image is in the relevant information difference range threshold value; and if the relevant information difference between the first three-dimensional image and the second three-dimensional image is not in the relevant information difference range threshold value, judging that the blocked part or the shadow part exists in the three-dimensional image.
In one embodiment, the image output module 804 is further configured to output the three-dimensional image if the three-dimensional image has no occluded part and no shadow part.
For specific limitations of the three-dimensional image restoration device, reference may be made to the above limitations of the three-dimensional image restoration method, which are not described herein again. The modules in the three-dimensional image restoration apparatus can be wholly or partially implemented by software, hardware and a combination thereof. The modules can be embedded in a hardware form or independent from a processor in the computer device, and can also be stored in a memory in the computer device in a software form, so that the processor can call and execute operations corresponding to the modules.
In one embodiment, a computer device is provided, which may be a terminal, and its internal structure diagram may be as shown in fig. 9. The computer device includes a processor, a memory, a communication interface, a display screen, and an input device connected by a system bus. Wherein the processor of the computer device is configured to provide computing and control capabilities. The memory of the computer device comprises a nonvolatile storage medium and an internal memory. The non-volatile storage medium stores an operating system and a computer program. The internal memory provides an environment for the operation of an operating system and computer programs in the non-volatile storage medium. The communication interface of the computer device is used for carrying out wired or wireless communication with an external terminal, and the wireless communication can be realized through WIFI, an operator network, NFC (near field communication) or other technologies. The computer program is executed by a processor to implement a three-dimensional image restoration method. The display screen of the computer equipment can be a liquid crystal display screen or an electronic ink display screen, and the input device of the computer equipment can be a touch layer covered on the display screen, a key, a track ball or a touch pad arranged on the shell of the computer equipment, an external keyboard, a touch pad or a mouse and the like.
Those skilled in the art will appreciate that the architecture shown in fig. 9 is merely a block diagram of some of the structures associated with the disclosed aspects and is not intended to limit the computing devices to which the disclosed aspects apply, as particular computing devices may include more or less components than those shown, or may combine certain components, or have a different arrangement of components.
In one embodiment, a computer device is provided, comprising a memory and a processor, the memory having a computer program stored therein, the processor implementing the following steps when executing the computer program:
acquiring a first three-dimensional image;
inputting the first three-dimensional image into a preset image restoration neural network for restoration processing to obtain a second three-dimensional image;
judging whether the first three-dimensional image has an occluded part or a shadow part according to the first three-dimensional image and the second three-dimensional image;
and if the first three-dimensional image has the blocked part or the shadow part, outputting a second three-dimensional image.
In one embodiment, the processor, when executing the computer program, further performs the steps of:
acquiring an image of a photographic subject;
and carrying out three-dimensional reconstruction on the image of the shot object to obtain a first three-dimensional image.
In one embodiment, the processor, when executing the computer program, further performs the steps of:
acquiring three-dimensional point cloud information of a shooting object according to an image of the shooting object;
and performing three-dimensional reconstruction on the image of the shot object according to the three-dimensional point cloud information of the shot object to obtain a first three-dimensional image.
In one embodiment, the processor, when executing the computer program, further performs the steps of:
obtaining an initial sample image, cutting a random area of the initial sample image, and assigning the cut area to be black to obtain a sample image;
inputting a sample image into an initial image restoration neural network, and processing the sample image through a convolution attention layer, a three-dimensional convolution network layer and a three-dimensional deconvolution network layer to obtain a restored sample image;
and adjusting the network weight of the initial image restoration neural network according to the restoration sample image and the initial sample image to obtain a preset image restoration neural network.
In one embodiment, the processor, when executing the computer program, further performs the steps of:
respectively acquiring related information of the first three-dimensional image and the second three-dimensional image, wherein the related information is color information or histogram information;
obtaining a relevant information difference between the first three-dimensional image and the second three-dimensional image according to the relevant information of the first three-dimensional image and the second three-dimensional image;
and judging whether the first three-dimensional image has an occluded part or a shadow part according to the related information difference between the first three-dimensional image and the second three-dimensional image.
In one embodiment, the processor, when executing the computer program, further performs the steps of:
acquiring a related information difference range threshold corresponding to related information according to the type of the related information;
if the relevant information difference between the first three-dimensional image and the second three-dimensional image is in the relevant information difference range threshold value, judging that an occluded part or a shadow part does not exist in the three-dimensional image;
and if the relevant information difference between the first three-dimensional image and the second three-dimensional image is not in the relevant information difference range threshold value, judging that the blocked part or the shadow part exists in the three-dimensional image.
In one embodiment, the processor, when executing the computer program, further performs the steps of:
and if the three-dimensional image has no shielded part and no shadow part, outputting the three-dimensional image.
In one embodiment, a computer-readable storage medium is provided, having a computer program stored thereon, which when executed by a processor, performs the steps of:
acquiring a first three-dimensional image;
inputting the first three-dimensional image into a preset image restoration neural network for restoration processing to obtain a second three-dimensional image;
judging whether the first three-dimensional image has an occluded part or a shadow part according to the first three-dimensional image and the second three-dimensional image;
and if the first three-dimensional image has the blocked part or the shadow part, outputting a second three-dimensional image.
In one embodiment, the computer program when executed by the processor further performs the steps of:
acquiring an image of a photographic subject;
and carrying out three-dimensional reconstruction on the image of the shot object to obtain a first three-dimensional image.
In one embodiment, the computer program when executed by the processor further performs the steps of:
acquiring three-dimensional point cloud information of a shooting object according to an image of the shooting object;
and performing three-dimensional reconstruction on the image of the shot object according to the three-dimensional point cloud information of the shot object to obtain a first three-dimensional image.
In one embodiment, the computer program when executed by the processor further performs the steps of:
obtaining an initial sample image, cutting a random area of the initial sample image, and assigning the cut area to be black to obtain a sample image;
inputting a sample image into an initial image restoration neural network, and processing the sample image through a convolution attention layer, a three-dimensional convolution network layer and a three-dimensional deconvolution network layer to obtain a restored sample image;
and adjusting the network weight of the initial image restoration neural network according to the restoration sample image and the initial sample image to obtain a preset image restoration neural network.
In one embodiment, the computer program when executed by the processor further performs the steps of:
respectively acquiring related information of the first three-dimensional image and the second three-dimensional image, wherein the related information is color information or histogram information;
obtaining a relevant information difference between the first three-dimensional image and the second three-dimensional image according to the relevant information of the first three-dimensional image and the second three-dimensional image;
and judging whether the first three-dimensional image has an occluded part or a shadow part according to the related information difference between the first three-dimensional image and the second three-dimensional image.
In one embodiment, the computer program when executed by the processor further performs the steps of:
acquiring a related information difference range threshold corresponding to related information according to the type of the related information;
if the relevant information difference between the first three-dimensional image and the second three-dimensional image is in the relevant information difference range threshold value, judging that an occluded part or a shadow part does not exist in the three-dimensional image;
and if the relevant information difference between the first three-dimensional image and the second three-dimensional image is not in the relevant information difference range threshold value, judging that the blocked part or the shadow part exists in the three-dimensional image.
In one embodiment, the computer program when executed by the processor further performs the steps of:
and if the three-dimensional image has no shielded part and no shadow part, outputting the three-dimensional image.
It will be understood by those skilled in the art that all or part of the processes of the methods of the embodiments described above can be implemented by hardware instructions of a computer program, which can be stored in a non-volatile computer-readable storage medium, and when executed, can include the processes of the embodiments of the methods described above. Any reference to memory, storage, database or other medium used in the embodiments provided herein can include at least one of non-volatile and volatile memory. Non-volatile Memory may include Read-Only Memory (ROM), magnetic tape, floppy disk, flash Memory, optical storage, or the like. Volatile Memory can include Random Access Memory (RAM) or external cache Memory. By way of illustration and not limitation, RAM can take many forms, such as Static Random Access Memory (SRAM) or Dynamic Random Access Memory (DRAM), among others.
The technical features of the above embodiments can be arbitrarily combined, and for the sake of brevity, all possible combinations of the technical features in the above embodiments are not described, but should be considered as the scope of the present specification as long as there is no contradiction between the combinations of the technical features.
The above-mentioned embodiments only express several embodiments of the present application, and the description thereof is more specific and detailed, but not construed as limiting the scope of the invention. It should be noted that, for a person skilled in the art, several variations and modifications can be made without departing from the concept of the present application, which falls within the scope of protection of the present application. Therefore, the protection scope of the present patent shall be subject to the appended claims.
Claims (10)
1. A method for three-dimensional image restoration, the method comprising:
acquiring a first three-dimensional image;
inputting the first three-dimensional image into a preset image restoration neural network for restoration processing to obtain a second three-dimensional image;
judging whether the first three-dimensional image has an occluded part or a shadow part according to the first three-dimensional image and the second three-dimensional image;
and if the first three-dimensional image has an occluded part or a shadow part, outputting the second three-dimensional image.
2. The method of claim 1, wherein said acquiring a first three-dimensional image comprises:
acquiring an image of a photographic subject;
and performing three-dimensional reconstruction on the image of the shot object to obtain a first three-dimensional image.
3. The method of claim 2, wherein the three-dimensional reconstruction of the image of the photographic subject to obtain a first three-dimensional image comprises:
acquiring three-dimensional point cloud information of the shot object according to the image of the shot object;
and performing three-dimensional reconstruction on the image of the shot object according to the three-dimensional point cloud information of the shot object to obtain a first three-dimensional image.
4. The method of claim 1, wherein the predetermined image restoration neural network is obtained by:
obtaining an initial sample image, cutting a random area of the initial sample image, and assigning the cut area to be black to obtain a sample image;
inputting the sample image into an initial image restoration neural network, and processing the sample image through a convolution attention layer, a three-dimensional convolution network layer and a three-dimensional deconvolution network layer to obtain a restored sample image;
and adjusting the network weight of the initial image reduction neural network according to the reduction sample image and the initial sample image to obtain a preset image reduction neural network.
5. The method according to claim 1, wherein said determining whether the first three-dimensional image has an occluded part or a shadow part according to the first three-dimensional image and the second three-dimensional image comprises:
respectively acquiring related information of the first three-dimensional image and the second three-dimensional image, wherein the related information is color information or histogram information;
obtaining a relevant information difference between the first three-dimensional image and the second three-dimensional image according to the relevant information of the first three-dimensional image and the second three-dimensional image;
and judging whether the first three-dimensional image has an occluded part or a shadow part according to the related information difference between the first three-dimensional image and the second three-dimensional image.
6. The method according to claim 5, wherein the determining whether the first three-dimensional image has the occluded part or the shadow part according to the difference between the related information of the first three-dimensional image and the second three-dimensional image comprises:
acquiring a related information difference range threshold corresponding to the related information according to the type of the related information;
if the relevant information difference between the first three-dimensional image and the second three-dimensional image is within the relevant information difference range threshold, judging that an occluded part or a shadow part does not exist in the three-dimensional image;
and if the relevant information difference between the first three-dimensional image and the second three-dimensional image is not within the relevant information difference range threshold, judging that an occluded part or a shadow part exists in the three-dimensional image.
7. The method of claim 1, further comprising:
and if the three-dimensional image has no shielded part and no shadow part, outputting the three-dimensional image.
8. A three-dimensional image restoration apparatus, comprising:
the image acquisition module is used for acquiring a first three-dimensional image;
the image restoration module is used for inputting the first three-dimensional image into a preset image restoration neural network for restoration processing to obtain a second three-dimensional image;
the image judging module is used for judging whether the first three-dimensional image has an occluded part or a shadow part according to the first three-dimensional image and the second three-dimensional image;
and the image output module is used for outputting the second three-dimensional image if the first three-dimensional image has a blocked part or a shadow part.
9. A computer device comprising a memory and a processor, the memory storing a computer program, characterized in that the processor, when executing the computer program, implements the steps of the method of any of claims 1 to 7.
10. A computer-readable storage medium, on which a computer program is stored, which, when being executed by a processor, carries out the steps of the method of any one of claims 1 to 7.
Priority Applications (1)
| Application Number | Priority Date | Filing Date | Title |
|---|---|---|---|
| CN202110349391.8A CN113099210B (en) | 2021-03-31 | 2021-03-31 | Three-dimensional image restoration method and device, computer equipment and storage medium |
Applications Claiming Priority (1)
| Application Number | Priority Date | Filing Date | Title |
|---|---|---|---|
| CN202110349391.8A CN113099210B (en) | 2021-03-31 | 2021-03-31 | Three-dimensional image restoration method and device, computer equipment and storage medium |
Publications (2)
| Publication Number | Publication Date |
|---|---|
| CN113099210A true CN113099210A (en) | 2021-07-09 |
| CN113099210B CN113099210B (en) | 2023-01-10 |
Family
ID=76672102
Family Applications (1)
| Application Number | Title | Priority Date | Filing Date |
|---|---|---|---|
| CN202110349391.8A Expired - Fee Related CN113099210B (en) | 2021-03-31 | 2021-03-31 | Three-dimensional image restoration method and device, computer equipment and storage medium |
Country Status (1)
| Country | Link |
|---|---|
| CN (1) | CN113099210B (en) |
Citations (7)
| Publication number | Priority date | Publication date | Assignee | Title |
|---|---|---|---|---|
| JPH05181980A (en) * | 1991-10-30 | 1993-07-23 | Sony Corp | Method and device for three-dimensional shape restoration |
| WO2012053521A1 (en) * | 2010-10-18 | 2012-04-26 | 株式会社トプコン | Optical information processing device, optical information processing method, optical information processing system, and optical information processing program |
| CN110033487A (en) * | 2019-02-25 | 2019-07-19 | 上海交通大学 | Vegetables and fruits collecting method is blocked based on depth association perception algorithm |
| US20190355093A1 (en) * | 2018-05-17 | 2019-11-21 | Korea Advanced Institute Of Science And Technology | Image processing apparatus using neural network and method performed by image processing apparatus |
| CN110766646A (en) * | 2018-07-26 | 2020-02-07 | 北京京东尚科信息技术有限公司 | Display rack shielding detection method and device and storage medium |
| CN111079545A (en) * | 2019-11-21 | 2020-04-28 | 上海工程技术大学 | Three-dimensional target detection method and system based on image restoration |
| CN112530027A (en) * | 2020-12-11 | 2021-03-19 | 北京奇艺世纪科技有限公司 | Three-dimensional point cloud repairing method and device and electronic equipment |
-
2021
- 2021-03-31 CN CN202110349391.8A patent/CN113099210B/en not_active Expired - Fee Related
Patent Citations (7)
| Publication number | Priority date | Publication date | Assignee | Title |
|---|---|---|---|---|
| JPH05181980A (en) * | 1991-10-30 | 1993-07-23 | Sony Corp | Method and device for three-dimensional shape restoration |
| WO2012053521A1 (en) * | 2010-10-18 | 2012-04-26 | 株式会社トプコン | Optical information processing device, optical information processing method, optical information processing system, and optical information processing program |
| US20190355093A1 (en) * | 2018-05-17 | 2019-11-21 | Korea Advanced Institute Of Science And Technology | Image processing apparatus using neural network and method performed by image processing apparatus |
| CN110766646A (en) * | 2018-07-26 | 2020-02-07 | 北京京东尚科信息技术有限公司 | Display rack shielding detection method and device and storage medium |
| CN110033487A (en) * | 2019-02-25 | 2019-07-19 | 上海交通大学 | Vegetables and fruits collecting method is blocked based on depth association perception algorithm |
| CN111079545A (en) * | 2019-11-21 | 2020-04-28 | 上海工程技术大学 | Three-dimensional target detection method and system based on image restoration |
| CN112530027A (en) * | 2020-12-11 | 2021-03-19 | 北京奇艺世纪科技有限公司 | Three-dimensional point cloud repairing method and device and electronic equipment |
Also Published As
| Publication number | Publication date |
|---|---|
| CN113099210B (en) | 2023-01-10 |
Similar Documents
| Publication | Publication Date | Title |
|---|---|---|
| CN112270249A (en) | Target pose estimation method fusing RGB-D visual features | |
| CN109658454B (en) | Pose information determination method, related device and storage medium | |
| US7764284B2 (en) | Method and system for detecting and evaluating 3D changes from images and a 3D reference model | |
| EP3654234B1 (en) | Moving object detection system and method | |
| US11080892B2 (en) | Computer-implemented methods and system for localizing an object | |
| CN111738032B (en) | Vehicle driving information determination method and device and vehicle-mounted terminal | |
| CN110415280B (en) | Remote sensing image and building vector registration method and system under multitask CNN model | |
| CN115527050A (en) | Image feature matching method, computer device and readable storage medium | |
| Ren et al. | Environment influences on uncertainty of object detection for automated driving systems | |
| WO2021230157A1 (en) | Information processing device, information processing method, and information processing program | |
| CN112270748B (en) | Three-dimensional reconstruction method and device based on image | |
| CN112967399A (en) | Three-dimensional time sequence image generation method and device, computer equipment and storage medium | |
| CN113099210B (en) | Three-dimensional image restoration method and device, computer equipment and storage medium | |
| CN110880003B (en) | Image matching method and device, storage medium and automobile | |
| CN112070035A (en) | Target tracking method and device based on video stream and storage medium | |
| CN109785312B (en) | Image blur detection method and system and electronic equipment | |
| CN114529543B (en) | Installation detection method and device for peripheral screw gasket of aero-engine | |
| CN113592947B (en) | Method for realizing visual odometer by semi-direct method | |
| CN112288817B (en) | Three-dimensional reconstruction processing method and device based on image | |
| CN112802112B (en) | Visual positioning method, device, server and storage medium | |
| CN111489439B (en) | Three-dimensional line graph reconstruction method and device and electronic equipment | |
| CN114241011A (en) | Target detection method, device, equipment and storage medium | |
| CN114966578A (en) | Radar external parameter calibration method and device based on shooting equipment and computer equipment | |
| CN112967388B (en) | Training method and device for three-dimensional time sequence image neural network model | |
| CN114387498A (en) | Target detection method and device, electronic equipment and storage medium |
Legal Events
| Date | Code | Title | Description |
|---|---|---|---|
| PB01 | Publication | ||
| PB01 | Publication | ||
| SE01 | Entry into force of request for substantive examination | ||
| SE01 | Entry into force of request for substantive examination | ||
| GR01 | Patent grant | ||
| GR01 | Patent grant | ||
| CF01 | Termination of patent right due to non-payment of annual fee | ||
| CF01 | Termination of patent right due to non-payment of annual fee |
Granted publication date: 20230110 |