CN107578389A - The method that the image color depth information collaboration of plane supervision is repaired - Google Patents
The method that the image color depth information collaboration of plane supervision is repaired Download PDFInfo
- Publication number
- CN107578389A CN107578389A CN201710823813.4A CN201710823813A CN107578389A CN 107578389 A CN107578389 A CN 107578389A CN 201710823813 A CN201710823813 A CN 201710823813A CN 107578389 A CN107578389 A CN 107578389A
- Authority
- CN
- China
- Prior art keywords
- image
- repaired
- color
- plane
- depth map
- Prior art date
- Legal status (The legal status is an assumption and is not a legal conclusion. Google has not performed a legal analysis and makes no representation as to the accuracy of the status listed.)
- Granted
Links
Landscapes
- Image Processing (AREA)
- Image Analysis (AREA)
Abstract
The present invention relates to the technical field of image, the method for cooperateing with reparation more particularly, to the image color depth information of plane supervision.During specific implementation, give a clue for depth reparation while the algorithm is mainly repaired by the image repair algorithm based on sample to color image, then split to tentatively repairing perfect color image using superpixel segmentation method.Tapered plane smoothing method is applied in depth map repair process simultaneously, the plane equation obtained using planar fit method estimates parallax value.The point not in the know obtained during using obstruction face or hinge face as the segmentation block and plane fitting of partitioning boundary will recalculate during next iteration.Repair perfect depth map and color image is repaired to give again and feed back.Last constantly iteration above step optimizes to obtain the reparation result of optimal color image and depth image, so as to reach the collaboration reparation to color depth information.
Description
Technical field
The present invention relates to the technical field of image, is cooperateed with more particularly, to the image color depth information of plane supervision
The method of reparation.
Background technology
In independent navigation and intelligent driving field, in order to successfully perceive and navigate in three-dimensional world, moving machine
Device people or vehicle need to extrapolate the information of the position of itself and surrounding three-dimensional environment.The instant positioning of view-based access control model is with building figure
It is estimated that the position of mobile robot and simultaneously progressively build surrounding environment three-dimensional map.When mobile robot is not
When being moved in the environment known, if the three-dimensional information that can obtain exterior ambient environment in time is being capable of avoiding barrier and rule
Draw the key precondition in path.
But in actual life, when appearance such as moving object is not desired to existing object, shot by binocular head
Image obtain three-dimensional reconstruction effect it is not always satisfactory.Therefore, it is necessary to will such as moving target these be not desired to exist
Object remove.However, these objects are removed from image can cause blank space occur on the map of three-dimensional reconstruction, seriously
The integrality of map is influenceed, the aesthetic property of map is reduced and is unfavorable for the further processing of computer.Asked to solve this
Topic to the blank space left after removing objects in terms of depth map and color image two, it is necessary to repair, so as to ensure three-dimensional
The integrality of map after reconstruction.
At present, existing most of image repairs or depth map reparation are all to complete alone, and both non-concurrent generals one
Block reparation, restriction relation between the two is not considered.And in the research work of minority, by color image and depth image simultaneously
Reparation is to complete to repair the part of color image by the coherence of image pair, then by repairing perfect depth map to chromaticity diagram
As remaining non-restoring area is repaired.
The content of the invention
The present invention is to overcome at least one defect described in above-mentioned prior art, there is provided the image color depth of plane supervision
The method of information synergism reparation, this method complete color depth information by the binding character between color information and depth information
Reparation.The integrality of impaired three-dimensional map is effectively ensured in this method, is advantageous to computer disposal.
The technical scheme is that：The method that the image color depth information collaboration of plane supervision is repaired, wherein, including
Color image repair module, image segmentation module, depth map repair module and color image are repaired again under depth map guidance
Module four is most of；
Color image repair module：Using based on the image repair method of the sample color that undamaged region obtains from color image
Multimedia message is filled into affected area, obtains visually reasonably repairing result；
Image splits block module：Calculated for the color image that color image repair module obtains using simple linear iteraction cluster
Method carries out super-pixel segmentation, obtains cut zone；
Depth map repair module：Each cut zone obtained for image segmentation block module, is unanimously calculated using random sampling
Method fits a plane equation；The parallax value of loss is estimated using the plane equation；Simultaneously by the border between cut zone
Carry out classification processing；Obtained during using obstruction face or hinge face as the cut zone and plane fitting of partitioning boundary not in the know
Point will recalculate during next iteration；
Color image repair module again under depth map guidance：For from depth map repair module feedback obtain need again
Restoring area is wanted, the similitude of never affected area acquisition color information institute foundation needs to consider to exist by the otherness of interplanar
It is interior.
Further, it is that same position, i.e. color image and depth map have in affected area color image and depth map
Uniformity, both mutually constrain.Repair perfect color image to give a clue to depth map, the reparation again of color image can obtain
The feedback provided to depth map.
Further, in described image repair module, affected area is repaired by block, multiblock referred to as to be repaired；It is damaged
Multiblock to be repaired at edges of regions is to be filled reparation according to the priority, is obtained before filling and first determines each multiblock to be repaired
Priority；Best matching blocks are found in undamaged region, the multiblock to be repaired of the match block and highest priority is in CIELab
Color space is the most similar.
Further, in described image repair module, the priority of multiblock to be repaired is multiplied by data item and confidence level item
Product determines；The number that multiblock to be repaired includes known pixels point is more, and its data item occurrence is bigger, comprising stronger linear structure, its
Confidence level entry value is bigger；After multiblock filling to be repaired is improved, damaged edge changes；Multiblock to be repaired positioned at affected area edge
Confidence level need to update.
Further, in described image segmentation block module, the color image obtained in image repair module is subjected to color
The conversion of color space, it is transformed into CIELab color spaces and five dimensional feature vectors of carrying space information；Foundation and five dimensional features
The distance of vector, the high pixel of similarity is flocked together.
Further, in described depth map repair module, for cut zone, constantly random selection is located at depth map
Three different pixel fit Plane equations in undamaged region, it is adapted to the plane side to the intra-office point arrived enough
Journey.
Further, in described depth map repair module, the pixel of the parallax value known to is calculated flat
Face parameter and the plane parameter by being obtained in color image repair process are as set element；Pass through random uniformity from the set
Algorithm obtains the plane equation of multiblock to be repaired in depth map, the depth information lost in being estimated by the plane equation.
Further, in described depth map repair module, coplanar, resistance can be divided into for the border classification between cut zone
Plug face and hinge face.
Further, iteration performs color image repair module, image segmentation module, depth map repair module and chromaticity diagram
As repair module four is most of again under depth map guidance；After stable naturally reparation result is obtained, the iteration in method
Process terminates.
Compared with prior art, beneficial effect is：The present invention is by between iterative manner and depth map and color image
Restriction relation, realize the reparation to color depth information.Tapered plane smoothing method is applied into depth map repair simultaneously.This
Invention can be used for repairing different degrees of breakage under different scenes, be a kind of more sane, more conform to human eye vision spy
The restorative procedure of sign.
Brief description of the drawings
The flow chart of the image color depth information restorative procedure of Fig. 1 planes supervision.
Fig. 2 cut zone border relations schematic diagrames.
Fig. 3 is that the color depth of application example of the present invention repairs result.
Embodiment
Accompanying drawing being given for example only property explanation, it is impossible to be interpreted as the limitation to this patent；It is attached in order to more preferably illustrate the present embodiment
Scheme some parts to have omission, zoom in or out, do not represent the size of actual product；To those skilled in the art,
Some known features and its explanation may be omitted and will be understood by accompanying drawing.Being given for example only property of position relationship described in accompanying drawing
Explanation, it is impossible to be interpreted as the limitation to this patent.
Embodiment：
As Fig. 1 be plane supervise image color depth information restorative procedure flow chart, including color image repair module, figure
As segmentation module, depth map repair module, color image, repair module four is most of again under depth map guidance.
Wherein, affected area is same position in color image and depth map.Color information and depth information are impaired
Region has uniformity, and both mutually constrain.
Wherein, in image repair module, the perfect color of preliminary reparation is obtained using the image repair algorithm based on sample
Coloured picture picture.Affected area is repaired by block, multiblock referred to as to be repaired.During image repair, it is thus necessary to determine that area to be repaired
In priority between each damaged block and find best matching blocks to repair damaged block.The height of priority determines multiblock to be repaired
Fill order.The priority of multiblock to be repaired is determined by data item and confidence level item product.Multiblock to be repaired includes known pixels point
Number it is more, its data item occurrence is bigger, bigger comprising stronger linear structure, its confidence level entry value.Multiblock to be repaired has been filled
Deal with problems arising from an accident, damaged edge changes.Confidence level positioned at the multiblock to be repaired at affected area edge needs to update.Best matching blocks
It is determined that it is according to the similitude between the pixel of equal size block in known pixels point in multiblock to be repaired and undamaged region.Should
The difference of two squares and determination of the similitude by pixel.
Wherein, in image segmentation module, the perfect color image of tentatively repairing obtained by image repair module is carried out
Segmentation.Color image is carried out to the conversion of color space, is transformed into CIELab color spaces and five Wei Te of carrying space information
Sign vector.Region number determination is determined, according to the distance with five dimensional feature vectors, the high pixel of similarity is gathered in one
Rise.In order to avoid isolated point be present and over-segmentation occurs, enhancing is connective, rejects zonule.By comparing it
Color space distance between adjacent segmentation block merges with the super-pixel block.Cross zonule and determine it is by heterochromia
Relatively determined compared with threshold value.
Wherein, in depth map repair module, the cut zone that is obtained for image segmentation module constantly randomly chooses position
Three different pixel fit Plane equations in the undamaged region of depth map, it is adapted to to the intra-office point arrived enough
The plane equation.By the pixel of the parallax value known to come Calculation Plane parameter and by obtaining in color image repair process
Plane parameter is as set element.The plane side of multiblock to be repaired in depth map is obtained from the set by random consistency algorithm
Journey, the depth information lost in being estimated by the plane equation.The border between cut zone is subjected to classification processing simultaneously, its side
Boundary's type is as shown in Figure 2.Obtained during using obstruction face or hinge face as the cut zone and plane fitting of partitioning boundary
Point not in the know will recalculate during next iteration.
Wherein, color image repair module again under depth map guidance, color image is repaired again can obtain depth map
Repair the feedback that result is given.Again the region for needing to repair includes the obtained point picture not in the know in plane parameter estimation procedure
Element and to block the combination of face and hinge face as the super-pixel segmentation block of border relations.It is optimal for being selected in restoring area again
During match block, need to take into account the otherness of interplanar during similitude between metrics match block and multiblock to be repaired.Obtaining
After stable naturally reparation result, the iterative process in method will terminate.
Fig. 3 is that the color depth of application example of the present invention repairs result, and experiment shows that this method can be repaired effectively not
With different degrees of image breakage under scene, the reparation to color depth information is realized.
Obviously, the above embodiment of the present invention is only intended to clearly illustrate example of the present invention, and is not pair
The restriction of embodiments of the present invention.For those of ordinary skill in the field, may be used also on the basis of the above description
To make other changes in different forms.There is no necessity and possibility to exhaust all the enbodiments.It is all this
All any modification, equivalent and improvement made within the spirit and principle of invention etc., should be included in the claims in the present invention
Protection domain within.
Claims (9)
1. the method that the image color depth information collaboration of plane supervision is repaired, it is characterised in that repair mould including color image
Repair module four is most of again under depth map guidance for block, image segmentation module, depth map repair module and color image；
Color image repair module：Using based on the image repair method of the sample color that undamaged region obtains from color image
Multimedia message is filled into affected area, obtains visually reasonably repairing result；
Image splits block module：Calculated for the color image that color image repair module obtains using simple linear iteraction cluster
Method carries out super-pixel segmentation, obtains cut zone；
Depth map repair module：Each cut zone obtained for image segmentation block module, is unanimously calculated using random sampling
Method fits a plane equation；The parallax value of loss is estimated using the plane equation；Simultaneously by the border between cut zone
Carry out classification processing；Obtained during using obstruction face or hinge face as the cut zone and plane fitting of partitioning boundary not in the know
Point will recalculate during next iteration；
Color image repair module again under depth map guidance：For from depth map repair module feedback obtain need again
Restoring area is wanted, the similitude of never affected area acquisition color information institute foundation needs to consider to exist by the otherness of interplanar
It is interior.
2. the method that the image color depth information collaboration of plane supervision according to claim 1 is repaired, it is characterised in that：
It is that same position, i.e. color image and depth map have uniformity in affected area color image and depth map, both are mutually about
Beam.
3. the method that the image color depth information collaboration of plane supervision according to claim 1 is repaired, it is characterised in that：
In described image repair module, affected area is repaired by block, multiblock referred to as to be repaired；Damaged area edge it is to be repaired
Block is to be filled reparation according to the priority, and the priority for first determining each multiblock to be repaired is obtained before filling；Undamaged
Best matching blocks are found in region, the match block is the most similar in CIELab color spaces to the multiblock to be repaired of highest priority.
4. the method that the image color depth information collaboration of plane supervision according to claim 2 is repaired, it is characterised in that：
In described image repair module, the priority of multiblock to be repaired is determined by data item and confidence level item product；Multiblock bag to be repaired
The number of the point containing known pixels is more, and its data item occurrence is bigger, bigger comprising stronger linear structure, its confidence level entry value；Treat
After reparation block filling is improved, damaged edge changes；Confidence level positioned at the multiblock to be repaired at affected area edge needs to update.
5. the method that the image color depth information collaboration of plane supervision according to claim 1 is repaired, it is characterised in that：
In described image segmentation block module, the color image obtained in image repair module is carried out to the conversion of color space, conversion
To CIELab color spaces and five dimensional feature vectors of carrying space information；, will be similar according to the distance with five dimensional feature vectors
High pixel is spent to flock together.
6. the method that the image color depth information collaboration of plane supervision according to claim 1 is repaired, it is characterised in that：
In described depth map repair module, for cut zone, constantly random selection is located at three in the undamaged region of depth map
Individual different pixel fit Plane equation, it is adapted to the plane equation to the intra-office point arrived enough.
7. the method that the image color depth information collaboration of plane supervision according to claim 1 is repaired, it is characterised in that：
In described depth map repair module, by the plane parameter that is calculated of pixel of parallax value known to and by color image
The plane parameter obtained in repair process is as set element；Obtain treating in depth map from the set by random consistency algorithm
The plane equation of block is repaired, the depth information lost in being estimated by the plane equation.
8. the method that the image color depth information collaboration of plane supervision according to claim 1 is repaired, it is characterised in that：
In described depth map repair module, coplanar, obstruction face and hinge face can be divided into for the border classification between cut zone.
9. the method that the image color depth information collaboration of plane supervision according to claim 1 is repaired, it is characterised in that：
Iteration performs color image repair module, image segmentation module, depth map repair module and color image under depth map guidance
Again repair module four is most of；After stable naturally reparation result is obtained, the iterative process in method terminates.
Priority Applications (1)
| Application Number | Priority Date | Filing Date | Title |
|---|---|---|---|
| CN201710823813.4A CN107578389B (en) | 2017-09-13 | 2017-09-13 | Plane-supervised image color depth information collaborative restoration system |
Applications Claiming Priority (1)
| Application Number | Priority Date | Filing Date | Title |
|---|---|---|---|
| CN201710823813.4A CN107578389B (en) | 2017-09-13 | 2017-09-13 | Plane-supervised image color depth information collaborative restoration system |
Publications (2)
| Publication Number | Publication Date |
|---|---|
| CN107578389A true CN107578389A (en) | 2018-01-12 |
| CN107578389B CN107578389B (en) | 2021-01-08 |
Family
ID=61036020
Family Applications (1)
| Application Number | Title | Priority Date | Filing Date |
|---|---|---|---|
| CN201710823813.4A Active CN107578389B (en) | 2017-09-13 | 2017-09-13 | Plane-supervised image color depth information collaborative restoration system |
Country Status (1)
| Country | Link |
|---|---|
| CN (1) | CN107578389B (en) |
Cited By (1)
| Publication number | Priority date | Publication date | Assignee | Title |
|---|---|---|---|---|
| CN114972129A (en) * | 2022-08-01 | 2022-08-30 | 电子科技大学 | Image restoration method based on depth information |
Citations (4)
| Publication number | Priority date | Publication date | Assignee | Title |
|---|---|---|---|---|
| US20050058366A1 (en) * | 2003-09-11 | 2005-03-17 | Yan-Ru Jiang | Method of reparing scratches in digital images |
| CN103489183A (en) * | 2012-10-17 | 2014-01-01 | 北京大学深圳研究生院 | Local stereo matching method based on edge segmentation and seed point |
| CN104616286A (en) * | 2014-12-17 | 2015-05-13 | 浙江大学 | Fast semi-automatic multi-view depth restoring method |
| CN105513064A (en) * | 2015-12-03 | 2016-04-20 | 浙江万里学院 | Image segmentation and adaptive weighting-based stereo matching method |
-
2017
- 2017-09-13 CN CN201710823813.4A patent/CN107578389B/en active Active
Patent Citations (4)
| Publication number | Priority date | Publication date | Assignee | Title |
|---|---|---|---|---|
| US20050058366A1 (en) * | 2003-09-11 | 2005-03-17 | Yan-Ru Jiang | Method of reparing scratches in digital images |
| CN103489183A (en) * | 2012-10-17 | 2014-01-01 | 北京大学深圳研究生院 | Local stereo matching method based on edge segmentation and seed point |
| CN104616286A (en) * | 2014-12-17 | 2015-05-13 | 浙江大学 | Fast semi-automatic multi-view depth restoring method |
| CN105513064A (en) * | 2015-12-03 | 2016-04-20 | 浙江万里学院 | Image segmentation and adaptive weighting-based stereo matching method |
Non-Patent Citations (1)
| Title |
|---|
| LEI FAN 等: "3D map repair after moving objects removing", 《2016 INTERNATIONAL CONFERENCE ON ADVANCED ROBOTICS AND MECHATRONICS (ICARM)》 * |
Cited By (1)
| Publication number | Priority date | Publication date | Assignee | Title |
|---|---|---|---|---|
| CN114972129A (en) * | 2022-08-01 | 2022-08-30 | 电子科技大学 | Image restoration method based on depth information |
Also Published As
| Publication number | Publication date |
|---|---|
| CN107578389B (en) | 2021-01-08 |
Similar Documents
| Publication | Publication Date | Title |
|---|---|---|
| US5734743A (en) | Image processing method and apparatus for block-based corresponding point extraction | |
| Belhumeur et al. | A Bayesian treatment of the stereo correspondence problem using half-occluded regions. | |
| Geiger et al. | Occlusions and binocular stereo | |
| EP2675175A2 (en) | Optimal depth mapping | |
| JP5005090B2 (en) | Cutting simulation display device, cutting simulation display method, and cutting simulation display program | |
| CN105809667A (en) | Shading effect optimization method based on depth camera in augmented reality | |
| Trivedi et al. | The role of disparity gradient in stereo vision | |
| EP1150254A3 (en) | Methods for creating an image for a three-dimensional display, for calculating depth information, and for image processing using the depth information | |
| JPH03218581A (en) | Picture segmentation method | |
| CN111753739B (en) | Object detection method, device, equipment and storage medium | |
| CN104639933A (en) | Real-time acquisition method and real-time acquisition system for depth maps of three-dimensional views | |
| Xu et al. | Depth-aided exemplar-based hole filling for DIBR view synthesis | |
| CN110660131A (en) | Virtual viewpoint hole filling method based on depth background modeling | |
| CN103384343A (en) | Image cavity filling method and device thereof | |
| CN110910452A (en) | Low-texture industrial part pose estimation method based on deep learning | |
| CN113223070A (en) | Depth image enhancement processing method and device | |
| KR20210001254A (en) | Method and apparatus for generating virtual view point image | |
| JP2023172882A (en) | Three-dimensional representation method and representation apparatus | |
| CN109448093B (en) | Method and device for generating style image | |
| CN107578389A (en) | The method that the image color depth information collaboration of plane supervision is repaired | |
| Gimeno et al. | Addressing the Occlusion Problem in Augmented Reality Environments with Phantom Hollow Objects. | |
| CN101464953A (en) | Outline extracting apparatus and method | |
| JP6901885B2 (en) | Foreground extractor and program | |
| CN114793457A (en) | Apparatus and method for improving the process of determining a depth map, relative pose or semantic segmentation | |
| CN108305269B (en) | Image segmentation method and system for binocular image |
Legal Events
| Date | Code | Title | Description |
|---|---|---|---|
| PB01 | Publication | ||
| PB01 | Publication | ||
| SE01 | Entry into force of request for substantive examination | ||
| SE01 | Entry into force of request for substantive examination | ||
| GR01 | Patent grant | ||
| GR01 | Patent grant |