CN114331919A - Depth recovery method, electronic device, and storage medium - Google Patents
Depth recovery method, electronic device, and storage medium Download PDFInfo
- Publication number
- CN114331919A CN114331919A CN202210221035.2A CN202210221035A CN114331919A CN 114331919 A CN114331919 A CN 114331919A CN 202210221035 A CN202210221035 A CN 202210221035A CN 114331919 A CN114331919 A CN 114331919A
- Authority
- CN
- China
- Prior art keywords
- value
- parallax
- seed point
- speckle pattern
- disparity
- Prior art date
- Legal status (The legal status is an assumption and is not a legal conclusion. Google has not performed a legal analysis and makes no representation as to the accuracy of the status listed.)
- Granted
Links
Images
Abstract
The embodiment of the invention relates to the field of image processing, and discloses a depth recovery method, electronic equipment and a storage medium, wherein the method comprises the following steps: determining the parallax values of the object speckle pattern and the reference speckle pattern by using the seed points and the parallax values thereof and adopting a region growing method to form a parallax pattern, and calculating the confidence coefficient of the matching cost value corresponding to the parallax values of various sub points in the region growing process; the method comprises the following steps that pixel points which grow successfully based on a current seed point are marked as new seed points and are divided into the same connected domain with the current seed point; determining a region corresponding to the connected domain in which the number of pixel points is less than a number threshold and the average value of the confidence degrees is less than a confidence degree threshold as a noise region in the disparity map; restoring to obtain a depth image based on the disparity map; and generating the depth value of the noise area in the depth image by adopting a specified denoising method. According to the scheme, the connected domain denoising can be realized while the depth is recovered, and the process of high-precision depth recovery is accelerated.
Description
Technical Field
The present invention relates to the field of image processing, and in particular, to a depth recovery method, an electronic device, and a storage medium.
Background
At present, the most active technical branch in the field of machine vision belongs to the depth perception technology, and the speckle structure light technology is an important part in the depth perception technology. The speckle structured light technology is used as the most common active stereoscopic vision technology and has wide application in the fields of face recognition, automatic driving, security monitoring and the like. The speckle structured light system projects pseudo-random speckles to a shot object, and then performs characteristic matching of the speckles according to a specific algorithm to obtain parallax information, so as to further obtain depth information of a scene. After the depth is obtained, filtering and other operations are generally used to remove noise of the depth map, such as gaussian filtering, median filtering, connected domain filtering, and the like. The speckle feature matching calculation amount is large, the time consumption is serious, and the real-time algorithm generally accelerates various technologies. The filtering denoising is generally also placed in the post-processing of the depth recovery, and is separated from the depth recovery, so that the algorithm with a better denoising effect consumes longer time.
In the existing depth recovery algorithm based on region growing, seed points are scattered according to a certain grid interval, after the best matching of the seed points is found and the matching cost meets a threshold value through large-range parallax search of the seed points, the growing of a region nearby the seed points is carried out, and the parallax search of the growing points is small-range parallax search. This approach speeds up the speckle image matching process. In the post-processing algorithm, the depth map is denoised by using a median filtering and connected domain denoising method, and a final depth map is output. Although the region growing algorithm accelerates the matching process of the speckle images, the denoising algorithm is still in a post-processing algorithm separately, and the time consumption cannot be underestimated.
Disclosure of Invention
An object of an embodiment of the present invention is to provide a depth recovery method, an electronic device, and a storage medium, which can implement connected domain denoising and accelerate a high-precision depth recovery process while performing depth recovery.
In order to solve the above technical problem, an embodiment of the present invention provides a depth recovery method, including:
determining the parallax values of the object speckle pattern and the reference speckle pattern by using the seed points and the parallax values thereof and adopting a region growing method to form a parallax pattern, and calculating the confidence coefficient of the matching cost value corresponding to the parallax values of various sub points in the region growing process; the method comprises the following steps that pixel points which grow successfully based on a current seed point are marked as new seed points and are divided into the same connected domain with the current seed point;
counting the number of pixel points in each connected domain in the depth image of the object speckle pattern and the average value of the confidence degrees corresponding to the pixel points, and determining the region, corresponding to the connected domain in which the number of the pixel points is less than a number threshold value and the average value of the confidence degrees is less than a confidence degree threshold value, in the parallax image as a noise region;
restoring to obtain a depth image based on the disparity map; and generating the depth value of the noise area in the depth image by adopting a specified denoising method.
An embodiment of the present invention also provides an electronic device, including:
at least one processor; and the number of the first and second groups,
a memory communicatively coupled to the at least one processor; wherein the content of the first and second substances,
the memory stores instructions executable by the at least one processor to enable the at least one processor to perform a depth recovery method as described above.
Embodiments of the present invention also provide a computer-readable storage medium storing a computer program which, when executed by a processor, implements the depth recovery method as described above.
Compared with the prior art, the embodiment of the invention determines the parallax values of the object speckle pattern and the reference speckle pattern to form the parallax pattern by using the seed points and the parallax values thereof and adopting a region growing method, and calculates the confidence coefficient of the matching cost value corresponding to the parallax values of various sub-points in the region growing process; the method comprises the following steps that pixel points which grow successfully based on a current seed point are marked as new seed points and are divided into the same connected domain with the current seed point; counting the number of pixel points in each connected domain in the depth image of the object speckle pattern and the average value of confidence degrees corresponding to the pixel points, and determining the region, corresponding to the connected domain in which the number of the pixel points is less than a number threshold value and the average value of the confidence degrees is less than a confidence degree threshold value, in the parallax image as a noise region; recovering to obtain a depth image based on the disparity map; and generating a depth value of the noise area in the depth image by adopting a specified denoising method. According to the scheme, the disparity map is determined by utilizing the seed points to perform region growing, and meanwhile, the connected domain is determined according to the growing information of the seed points; when the depth map is determined based on the disparity map, the depth value of the connected domain determined as the noise area in the depth image is generated by directly adopting a specified denoising method, so that the connected domain denoising is realized while the depth is recovered, and the process of high-precision depth recovery is accelerated.
Drawings
FIG. 1 is a first flowchart illustrating a depth recovery method according to an embodiment of the present invention;
FIG. 2 is a schematic diagram of a triangulation principle to calculate depth according to an embodiment of the invention;
FIG. 3 is a detailed flowchart II of a depth recovery method according to an embodiment of the present invention;
FIG. 4 is a diagram of a candidate seed point selection according to an embodiment of the present invention;
FIG. 5 is a flowchart illustrating a depth recovery method according to an embodiment of the present invention;
fig. 6 is a schematic structural diagram of an electronic device according to an embodiment of the present invention.
Detailed Description
In order to make the objects, technical solutions and advantages of the embodiments of the present invention more apparent, embodiments of the present invention will be described in detail below with reference to the accompanying drawings. However, it will be appreciated by those of ordinary skill in the art that numerous technical details are set forth in order to provide a better understanding of the present application in various embodiments of the present invention. However, the technical solution claimed in the present application can be implemented without these technical details and various changes and modifications based on the following embodiments.
An embodiment of the present invention relates to a depth recovery method, which is suitable for an image processing scene in which a depth image of a target object is recovered using a speckle pattern of the target object. As shown in fig. 1, the depth recovery method provided by this embodiment includes the following steps.
Step 103: determining the parallax values of the object speckle pattern and the reference speckle pattern by using the seed points and the parallax values thereof and adopting a region growing method to form a parallax pattern, and calculating the confidence coefficient of the matching cost value corresponding to the parallax values of various sub points in the region growing process; the method comprises the following steps that pixel points which grow successfully based on a current seed point are marked as new seed points and are divided into the same connected domain with the current seed point; and counting the number of pixel points in each connected domain in the depth image of the object speckle pattern and the average confidence of the pixel points.
The first seed point used in the process of growing the seed point by using the region growing method may be a pre-designated seed point or a seed point determined by preference judgment from a plurality of candidate seed points.
For example, if the candidate seed point is successfully determined as the seed point and the disparity value of the seed point is obtained, the growing link of the seed point is entered. Growth was performed around the seed point. For each neighborhood point around the seed point, carrying out [ -2,2] parallax search matching at the position of the difference value of the parallax value of the seed point, wherein the matching method and the cost value calculation method are the same as the correlation method used by the original seed point. And if the matching cost value is smaller than the growth threshold value, determining to find a matched parallax value. In this case, the disparity value corresponding to the matching cost value may be directly used as the disparity value of the current matching point, or the sub-pixel level disparity of the disparity value may be calculated as the final disparity value with reference to formula (5). And then, taking the current neighborhood point as a new seed point and carrying out region growth on the periphery of the new seed point so as to further obtain the parallax value of the new neighborhood point. And if the matching cost value is smaller than the growth threshold value, calculating the neighborhood point of the next seed point.
And (3) for each seed point, iterating and searching the parallax of the neighborhood point by adopting a region growing method, and finally determining the parallax value of the pixel point between the object speckle pattern and the reference speckle pattern to form a parallax map.
In the region growing process, the confidence of the matching cost value corresponding to the disparity value of each seed point can be calculated. The confidence of the matching cost value is defined as: and determining the credibility of the disparity value taking the corresponding disparity value as the seed point by using the matching cost value. Therefore, the higher the confidence of the matching cost value, the higher the confidence that the corresponding disparity value is the disparity value of the seed point.
In general, the matching cost value reflects the similarity between two image blocks to be matched determined under a certain disparity value, and the greater the similarity is, the greater the reliability of the disparity value as an actual disparity value is, the smaller the corresponding matching cost value is. Based on this, the calculation formula of the confidence coefficient of the matching cost value may be defined according to the trend that the smaller the matching cost value is, the larger the confidence coefficient of the matching cost value is, and the specific formula for calculating the confidence coefficient of the matching cost value is not limited in this embodiment.
Meanwhile, in the process of growing the region, a plurality of connected domains can be formed based on the growing condition of the seed points. Namely: and the pixel points which successfully grow based on the current seed point are marked as new seed points and are divided into the same connected domain with the current seed point, and the pixel points which do not successfully grow based on the current seed point are not divided into the connected domain to which the current seed point belongs.
For example, when the candidate seed point is determined to successfully perform seed growth, the disparity search may be performed on the neighboring pixel points of the current seed point within [ -2,2] of the disparity value of the seed point, which is performed according to the principle that the disparity change of the continuous object is not large, the neighboring pixel points with the disparity exceeding 2 and the current seed point are marked as belonging to different mark classes (or mark classes temporarily not marking the neighboring pixel points), and the neighboring pixel points with the disparity not exceeding 2, that is, the pixel points (new seed points) successfully grown based on the current seed point and the current seed point are marked as belonging to the same mark class. And after the region growing process is finished, dividing all the pixel points belonging to the same mark class into the same connected domain. In one example, calculating confidence of matching cost values corresponding to disparity values of various sub-points in the region growing process includes:
the confidence of the matching cost value corresponding to the disparity value of each seed point is calculated by the following formula (1).
Wherein the content of the first and second substances,cf d confidence of the matching cost value corresponding to the disparity value of the seed point, Z d 、Z d-1、Z d+1The parallax values of the seed points in turndThe corresponding matching cost value and the parallax valuedCorresponding adjacent disparity valued-a matching cost value and neighboring disparity value of 1dA matching cost value of + 1.
Specifically, when determining the disparity value of the current seed point, usually a disparity search is performed in a disparity search range of the seed point, and based on a matching cost value corresponding to each disparity value, a disparity value is determined from the disparity search range as a final disparity value of the seed pointdAnd the parallax valuedThe corresponding matching cost value is recorded as Z d And the parallax valuedTwo adjacent parallax values are respectivelyd-1、d+1, the corresponding matching cost value is noted as Z d-1、Z d+1. Based on Z d 、Z d-1、Z d+1The confidence coefficient of the matching cost value corresponding to the parallax value of the current seed point can be obtained by calculating the formula (1)cf d It can also be simply written as: the confidence of the current seed point correspondence.
Step 104: counting the number of pixel points in each connected domain in the depth image of the object speckle pattern and the average value of the confidence degrees corresponding to the pixel points, and determining the region corresponding to the connected domain in the parallax image, wherein the number of the pixel points is less than a number threshold value and the average value of the confidence degrees is less than a confidence degree threshold value, as a noise region.
Specifically, after the seed growth is finished, the average value of the number of the pixel points in the connected domain belonging to the same mark class and the confidence corresponding to the pixel points is counted. And presetting a quantity threshold and a confidence threshold, and determining the pixel points in the connected domain of which the quantity is lower than the quantity threshold and the average value of the confidence is lower than the confidence threshold as noise. Therefore, the judging factors influencing whether a connected domain is a noise point are the number of pixel points included in the connected domain and the reliability of the depth value in the connected domain, and the judging principle is that a region with too small number of pixel points is considered as a noise point, and a region with lower average confidence coefficient (higher matching cost value) is considered as a noise point.
Step 105: recovering to obtain a depth image based on the disparity map; and generating a depth value of the noise area in the depth image by adopting a specified denoising method.
Specifically, after the image growth is finished, the parallax values of all the pixel points are compareddThe depth Z is calculated according to the triangulation principle shown in fig. 2, and the calculation formula is as follows:
wherein z is0Is the reference plane distance in mm;f，lthe focal length and the baseline distance are respectively calibrated for the camera.
After the depth map is obtained, post-processing, such as median filtering, may be performed on the depth map to remove redundant noise and output a high-precision depth map.
In the process of recovering the depth image, a specified denoising method can be adopted for pixel points of the noise area to generate the depth value of the pixel points in the depth image. For example, the depth value of the pixel point in the noise area may be directly specified, or a calculation method may be specified to obtain the depth value.
In one example, the depth values of the pixel points in the noise region may be set to 0 by default.
Compared with the related art, the embodiment determines the parallax values of the object speckle pattern and the reference speckle pattern by using the seed points and the parallax values thereof and adopting a region growing method to form the parallax pattern, and calculates the confidence degrees of the matching cost values corresponding to the parallax values of various sub points in the region growing process; the method comprises the following steps that pixel points which grow successfully based on a current seed point are marked as new seed points and are divided into the same connected domain with the current seed point; counting the number of pixel points in each connected domain in the depth image of the object speckle pattern and the average value of confidence degrees corresponding to the pixel points, and determining the region, corresponding to the connected domain in which the number of the pixel points is less than a number threshold value and the average value of the confidence degrees is less than a confidence degree threshold value, in the parallax image as a noise region; recovering to obtain a depth image based on the disparity map; and generating a depth value of the noise area in the depth image by adopting a specified denoising method. According to the scheme, the disparity map is determined by utilizing the seed points to perform region growing, and meanwhile, the connected domain is determined according to the growing information of the seed points; when the depth map is determined based on the disparity map, the depth value of the connected domain determined as the noise area in the depth image is generated by directly adopting a specified denoising method, so that the connected domain denoising is realized while the depth is recovered, and the process of high-precision depth recovery is accelerated.
Another embodiment of the invention relates to a depth recovery method, as shown in fig. 3, which is an improvement over the method steps shown in fig. 1 in that the determination of the initial seed points and their disparity values utilized when using the region growing method is refined. As shown in FIG. 3, the method may further comprise the following steps (steps 101-102) before step 103.
Step 101: and aiming at the preprocessed object speckle pattern and the reference speckle pattern, selecting a plurality of candidate seed points from the object speckle pattern and a parallax search range corresponding to each candidate seed point.
Specifically, a speckle pattern of a target object can be photographed by a structured light camera (simply referred to as "camera") as an object speckle pattern; the reference speckle pattern is a planar speckle pattern of known distance. The object speckle pattern and the reference speckle pattern are preprocessed to improve the light-dark contrast and the brightness balance effect of the speckle.
In one example, pre-processing the object speckle pattern and the reference speckle pattern may include: and sequentially carrying out local gray scale normalization and shadow processing on the object speckle pattern and the reference speckle pattern.
Specifically, the local gray level normalization processing is performed on the object speckle pattern and the reference speckle pattern, and comprises the following steps:
the object speckle pattern and the reference speckle pattern are processed using the following formula:
wherein the content of the first and second substances,G(i,j) 、G’(i,j) The coordinates before and after the local gray level normalization treatment are sequentially (i,j) The gray value of the pixel point P of (a),avg、stdthe mean and standard deviation of the gray levels in the neighborhood window centered on point P are taken in turn.
Specifically, Local Contrast Normalization (LCN) is adopted, and coordinates on an object speckle pattern and a reference speckle pattern are respectively (i,j) Pixel point P of which the gray value isG(i,j) Taking a neighborhood window centered on point Pk*k(k is a constant value), calculating the average value of the gray levels in the windowavgStandard deviation ofstdThen, the gray value after local gray normalization is calculated according to the formula (3)G’(i,j)。
The LCN method can solve the problem of uneven brightness of the speckle images.
Then, the shading processing is carried out on the object speckle pattern and the reference speckle pattern after the local gray level normalization processing, and the shading processing comprises the following steps: and (4) processing the object speckle pattern after the local gray level normalization processing and the reference speckle pattern by adopting the following formula (4).
Wherein m is: (i,j) Is the shadow processed coordinate is (i,j) The gray value of the pixel point of (a),thresholdis a shaded border value.
Specifically, the gray value in the speckle image is expressed by the above formula (4)G’(i,j) Judging the too small pixel points as shadow, marking the pixel points as shadow (the gray value is 0), marking the gray values of the other pixel points as non-shadow (the gray value is 1), and establishing a shadow (mask) matrix m (i,j). The pixel points marked as shadows will not undergo subsequent depth recovery operations.
Matrix m (i,j) The element values in (a) correspond to the gray values in the speckle image.
For the preprocessed object speckle pattern and the reference speckle pattern, a plurality of candidate seed points can be selected from the object speckle pattern, and a parallax search range corresponding to each candidate seed point.
Specifically, the principle of depth recovery using the region growing algorithm is to consider the depth of the scene to have a certain continuity, which is equivalent to the cost aggregation part in the depth recovery process. Therefore, a plurality of pixel points are selected from the object speckle pattern as candidate seed points (such as solid points in fig. 4) according to a certain interval grid mode, and a queue of the candidate seed points is formed. And selecting candidate seed points in the queue in sequence, and determining a disparity search range for each candidate seed point so as to perform disparity search. The disparity search range here may be a full range or a partially continuous range of disparity search.
Step 102: and for each candidate seed point, performing parallax search in the preprocessed reference speckle pattern by using the corresponding parallax search range, determining whether the candidate seed point is a seed point or not based on the matching cost value corresponding to each parallax value obtained by the parallax search, and obtaining the parallax value of the seed point.
Specifically, for each candidate seed point, matching points corresponding to the disparity values are found in the preprocessed reference speckle pattern according to the disparity values in the corresponding disparity search range, and local image block matching is performed on the matching points and the corresponding candidate seed points, so that matching cost values corresponding to the disparity values are obtained. And finally, determining whether the candidate seed point is a seed point according to the matching cost value corresponding to each parallax value of each candidate seed point, and acquiring the parallax value of the seed point when the candidate seed point is determined to be one seed point.
When whether the candidate seed point is the seed point is determined according to the matching cost value corresponding to each disparity value of each candidate seed point, the smaller the matching cost value corresponding to each disparity value is, the higher the possibility that the candidate seed point is determined as the seed point is. After the disparity value corresponding to the seed point is determined as the seed point, the disparity value corresponding to the seed point may be determined from the disparity values based on the matching cost values corresponding to the disparity values, or may be further calculated and generated based on the disparity values.
Compared with the related art, the embodiment selects a plurality of candidate seed points from the object speckle pattern and a parallax search range corresponding to each candidate seed point by aiming at the preprocessed object speckle pattern and the reference speckle pattern; and for each candidate seed point, performing parallax search in the preprocessed reference speckle pattern by using the corresponding parallax search range, determining whether the candidate seed point is a seed point or not based on the matching cost value corresponding to each parallax value obtained by the parallax search, and obtaining the parallax value of the seed point, thereby obtaining the initial seed point and the parallax value thereof used for the subsequent seed growth link.
Another embodiment of the present invention relates to a depth recovery method, as shown in fig. 5, which is an improvement of the steps of the method shown in fig. 3 in that the process of determining a seed point and acquiring a disparity value of the seed point is refined. As shown in fig. 5, the step 102 may include the following sub-steps.
Substep 1021: and performing parallax search on the current candidate seed points in the preprocessed reference speckle pattern according to the corresponding parallax search range to obtain a plurality of matching cost values corresponding to the current candidate seed points.
Specifically, for each current candidate seed point, disparity search may be performed on each disparity value in the disparity search range corresponding to the current candidate seed point in the preprocessed reference speckle pattern, so as to determine a matching cost value corresponding to each disparity value. In this embodiment, the matching cost algorithm in the parallax search is not limited.
For example, a hamming distance between an object image block and a reference image block corresponding to each disparity value can be calculated by using a neighborhood window, and the hamming distance is used as a matching cost value corresponding to the corresponding disparity value; or calculating the Sum of Absolute Differences (SAD) between the object image block and the reference image block corresponding to each disparity value as the matching cost value corresponding to the corresponding disparity value by using the neighborhood window.
Substep 1022: and if the minimum value of the multiple matching cost values is smaller than the set threshold value, taking the current candidate seed point as a seed point, and taking the parallax value corresponding to the minimum value of the multiple matching cost values as the parallax value of the seed point.
Specifically, when the minimum matching cost value obtained by performing disparity search in the reference speckle pattern is smaller than a set threshold value in the disparity search range of the candidate seed point, it is considered that a better disparity value corresponding to the current candidate seed point is searched in the current disparity search range. At this time, the current candidate seed point may be directly determined as a seed point, and the disparity value of the seed point with the minimum matching cost value in the current disparity search range is used as the disparity value of the seed point.
Substep 1023: and if the minimum value of the plurality of matching cost values is not less than the set threshold value, discarding the current candidate seed point.
Specifically, when the minimum matching cost value obtained by performing disparity search in the reference speckle pattern is not less than a set threshold value within the disparity search range of the candidate seed point, it is determined that a better disparity value corresponding to the current candidate seed point is not searched within the disparity search range. At this time, the matching degree of the candidate seed point itself may not be high, and the current candidate seed point is determined to be failed, and then the determining process of other candidate seed points may be performed from sub-step 1021.
In one example, when the sub-step 1021 is satisfied and the minimum value of the matching cost values is smaller than the set threshold, the sub-pixel level disparity may be calculated to replace the original disparity, so as to increase the accuracy of the disparity. The treatment process comprises the following steps:
disparity values corresponding to minimum values of multiple matching cost valuesdDetermining the disparity valuedCorresponding adjacent disparity valued-a matching cost value Z of 1 d-1And adjacent disparity valued+1 matching cost value Z d+1(ii) a Calculating the parallax value corresponding to the minimum value of the matching cost values by adopting the following formuladSub-pixel level parallaxd’At sub-pixel level parallaxd’Replacing disparity valuesdAs the parallax value of the seed point:
wherein Z is d Disparity value corresponding to minimum value of multiple matching cost valuesdThe value of the matching cost of (c),L=Z d-1-Z d ，R=Z d+1-Z d 。
specifically, in the determining process of determining whether the candidate seed point is the seed point in this embodiment and the seed growing process in the foregoing embodiments, the matching cost value Z of the seed point is determined d 、Z d-1And Z d+1Are the same in the calculation method of (a). Therefore, the calculated matching cost values of the various sub-points when determining whether the candidate seed point is the seed point are suitable for calculating the matching cost values of the various seed points in the seed growth process, and certainly also suitable for calculating the confidence of the matching cost values, which is not described herein again.
For parallax at sub-pixel leveld’Replacing disparity valuesdThe method of the final disparity value of the seed point may be applied to the seed point determined by the candidate seed point, and also to the seed point determined during the growth of the seed.
Compared with the related art, the embodiment performs parallax search on the current candidate seed point in the preprocessed reference speckle pattern by using the corresponding parallax search range to obtain a plurality of matching cost values corresponding to the current candidate seed point; if the minimum value of the multiple matching cost values is smaller than a set threshold value, taking the current candidate seed point as a seed point, and taking the parallax value corresponding to the minimum value of the multiple matching cost values as the parallax value of the seed point; and if the minimum value in the plurality of matching cost values is not less than the set threshold value, discarding the current candidate seed point, thereby accurately determining the seed point and the parallax value of the seed point.
Another embodiment of the invention relates to an electronic device, as shown in FIG. 6, comprising at least one processor 202; and a memory 201 communicatively coupled to the at least one processor 202; wherein the memory 201 stores instructions executable by the at least one processor 202, the instructions being executable by the at least one processor 202 to enable the at least one processor 202 to perform any of the method embodiments described above.
Where the memory 201 and the processor 202 are coupled in a bus, the bus may comprise any number of interconnected buses and bridges that couple one or more of the various circuits of the processor 202 and the memory 201 together. The bus may also connect various other circuits such as peripherals, voltage regulators, power management circuits, and the like, which are well known in the art, and therefore, will not be described any further herein. A bus interface provides an interface between the bus and the transceiver. The transceiver may be one element or a plurality of elements, such as a plurality of receivers and transmitters, providing a means for communicating with various other apparatus over a transmission medium. The data processed by the processor 202 is transmitted over a wireless medium through an antenna, which further receives the data and transmits the data to the processor 202.
The processor 202 is responsible for managing the bus and general processing and may also provide various functions including timing, peripheral interfaces, voltage regulation, power management, and other control functions. And memory 201 may be used to store data used by processor 202 in performing operations.
Another embodiment of the present invention relates to a computer-readable storage medium storing a computer program. The computer program realizes any of the above-described method embodiments when executed by a processor.
That is, as can be understood by those skilled in the art, all or part of the steps in the method for implementing the embodiments described above may be implemented by a program instructing related hardware, where the program is stored in a storage medium and includes several instructions to enable a device (which may be a single chip, a chip, or the like) or a processor (processor) to execute all or part of the steps of the method described in the embodiments of the present application. And the aforementioned storage medium includes: various media capable of storing program codes, such as a usb disk, a removable hard disk, a Read-only memory (ROM), a Random Access Memory (RAM), a magnetic disk, or an optical disk.
It will be understood by those of ordinary skill in the art that the foregoing embodiments are specific examples for carrying out the invention, and that various changes in form and details may be made therein without departing from the spirit and scope of the invention in practice.
Claims (11)
1. A method of depth recovery, comprising:
determining the parallax values of the object speckle pattern and the reference speckle pattern by using the seed points and the parallax values thereof and adopting a region growing method to form a parallax pattern, and calculating the confidence coefficient of the matching cost value corresponding to the parallax values of various sub points in the region growing process; the method comprises the following steps that pixel points which grow successfully based on a current seed point are marked as new seed points and are divided into the same connected domain with the current seed point;
counting the number of pixel points in each connected domain in the depth image of the object speckle pattern and the average value of the confidence degrees corresponding to the pixel points, and determining the region, corresponding to the connected domain in which the number of the pixel points is less than a number threshold value and the average value of the confidence degrees is less than a confidence degree threshold value, in the parallax image as a noise region;
restoring to obtain a depth image based on the disparity map; and generating the depth value of the noise area in the depth image by adopting a specified denoising method.
2. The method of claim 1, wherein determining the seed point and its disparity value comprises:
aiming at the preprocessed object speckle pattern and the reference speckle pattern, selecting a plurality of candidate seed points from the object speckle pattern and a parallax search range corresponding to each candidate seed point;
and for each candidate seed point, performing parallax search in the preprocessed reference speckle pattern by using the corresponding parallax search range, determining whether the candidate seed point is a seed point or not based on the matching cost value corresponding to each parallax value obtained by the parallax search, and obtaining the parallax value of the seed point.
3. The method of claim 1, wherein pre-processing the object speckle pattern and the reference speckle pattern comprises:
and sequentially carrying out local gray scale normalization and shadow processing on the object speckle pattern and the reference speckle pattern.
4. The method of claim 3, wherein performing local gray scale normalization on the object speckle pattern and the reference speckle pattern comprises:
processing the object speckle pattern and the reference speckle pattern using the following formula:
wherein the content of the first and second substances,G(i,j) 、G’(i,j) Sequentially comprises the coordinates before and after the local gray level normalization treatment of (i,j) The gray value of the pixel point P of (a),avg、stdin turn, the mean and standard of the gray levels in the neighborhood window centered on point PAnd (4) poor.
5. The method of claim 4, wherein shading the object speckle pattern and the reference speckle pattern after the local gray scale normalization comprises:
processing the object speckle pattern and the reference speckle pattern after the local gray level normalization processing by adopting the following formula:
wherein m is: (i,j) Is the shadow processed coordinate is (i,j) The gray value of the pixel point of (a),thresholdis a shaded border value.
6. The method according to claim 2, wherein the performing, for each candidate seed point, a disparity search in the preprocessed reference speckle pattern with the corresponding disparity search range, determining whether the candidate seed point is a seed point based on a matching cost value corresponding to each disparity value obtained by the disparity search, and obtaining the disparity value of the seed point comprises:
performing parallax search on the current candidate seed points in the preprocessed reference speckle pattern according to the corresponding parallax search range to obtain a plurality of matching cost values corresponding to the current candidate seed points;
if the minimum value of the matching cost values is smaller than a set threshold value, taking the current candidate seed point as a seed point, and taking the parallax value corresponding to the minimum value of the matching cost values as the parallax value of the seed point;
and if the minimum value of the plurality of matching cost values is not less than a set threshold value, discarding the current candidate seed point.
7. The method of claim 6, wherein when a minimum value of the plurality of matching cost values is less than a set threshold, the method further comprises:
a disparity value corresponding to a minimum value of the plurality of matching cost valuesdDetermining the disparity valuedCorresponding adjacent disparity valued-a matching cost value Z of 1 d-1And adjacent disparity valued+1 matching cost value Z d+1；
Calculating a disparity value corresponding to the minimum value of the plurality of matching cost values by adopting the following formuladSub-pixel level parallaxd’At the sub-pixel level of parallaxd’Replacing the disparity valuedAs the disparity value of the seed point:
wherein Z is d A disparity value corresponding to the minimum value of the multiple matching cost valuesdThe value of the matching cost of (c),L=Z d-1-Z d ，R=Z d+1-Z d 。
8. the method according to claim 1, wherein the calculating the confidence of the matching cost value corresponding to the disparity value of each sub-point in the region growing process comprises:
calculating the confidence coefficient of the matching cost value corresponding to the disparity value of each seed point by the following formula:
wherein the content of the first and second substances,cf d confidence of the matching cost value corresponding to the disparity value of the seed point, Z d 、Z d-1、Z d+1The parallax values of the seed points are sequentiallydThe corresponding matching cost value and the parallax valuedCorresponding adjacent disparity valued-a matching cost value and neighboring disparity value of 1d+1The matching cost value of (c).
9. The method of claim 1, wherein the generating the depth value of the noise region in the depth image by using a specified denoising method comprises:
and setting the depth value of the pixel point in the noise area as 0 by default.
10. An electronic device, comprising:
at least one processor; and the number of the first and second groups,
a memory communicatively coupled to the at least one processor; wherein the content of the first and second substances,
the memory stores instructions executable by the at least one processor to enable the at least one processor to perform the depth recovery method of any one of claims 1 to 9.
11. A computer-readable storage medium storing a computer program, wherein the computer program, when executed by a processor, implements the depth recovery method of any one of claims 1 to 9.
Priority Applications (1)
| Application Number | Priority Date | Filing Date | Title |
|---|---|---|---|
| CN202210221035.2A CN114331919B (en) | 2022-03-09 | 2022-03-09 | Depth recovery method, electronic device, and storage medium |
Applications Claiming Priority (1)
| Application Number | Priority Date | Filing Date | Title |
|---|---|---|---|
| CN202210221035.2A CN114331919B (en) | 2022-03-09 | 2022-03-09 | Depth recovery method, electronic device, and storage medium |
Publications (2)
| Publication Number | Publication Date |
|---|---|
| CN114331919A true CN114331919A (en) | 2022-04-12 |
| CN114331919B CN114331919B (en) | 2022-05-31 |
Family
ID=81033408
Family Applications (1)
| Application Number | Title | Priority Date | Filing Date |
|---|---|---|---|
| CN202210221035.2A Active CN114331919B (en) | 2022-03-09 | 2022-03-09 | Depth recovery method, electronic device, and storage medium |
Country Status (1)
| Country | Link |
|---|---|
| CN (1) | CN114331919B (en) |
Cited By (1)
| Publication number | Priority date | Publication date | Assignee | Title |
|---|---|---|---|---|
| CN114820393A (en) * | 2022-06-28 | 2022-07-29 | 合肥的卢深视科技有限公司 | Depth recovery method for fusion hole repair, electronic device and storage medium |
Citations (6)
| Publication number | Priority date | Publication date | Assignee | Title |
|---|---|---|---|---|
| CN103268608A (en) * | 2013-05-17 | 2013-08-28 | 清华大学 | Depth estimation method and device based on near-infrared laser speckles |
| US20150326845A1 (en) * | 2014-05-09 | 2015-11-12 | Ricoh Company, Ltd. | Depth value restoration method and system |
| CN107133977A (en) * | 2017-05-18 | 2017-09-05 | 华中科技大学 | A kind of quick stereo matching process that model is produced based on probability |
| CN109146946A (en) * | 2018-08-29 | 2019-01-04 | 河海大学常州校区 | A kind of non local solid matching method of image |
| WO2020125637A1 (en) * | 2018-12-17 | 2020-06-25 | 深圳市道通智能航空技术有限公司 | Stereo matching method and apparatus, and electronic device |
| CN111402313A (en) * | 2020-03-13 | 2020-07-10 | 合肥的卢深视科技有限公司 | Image depth recovery method and device |
-
2022
- 2022-03-09 CN CN202210221035.2A patent/CN114331919B/en active Active
Patent Citations (6)
| Publication number | Priority date | Publication date | Assignee | Title |
|---|---|---|---|---|
| CN103268608A (en) * | 2013-05-17 | 2013-08-28 | 清华大学 | Depth estimation method and device based on near-infrared laser speckles |
| US20150326845A1 (en) * | 2014-05-09 | 2015-11-12 | Ricoh Company, Ltd. | Depth value restoration method and system |
| CN107133977A (en) * | 2017-05-18 | 2017-09-05 | 华中科技大学 | A kind of quick stereo matching process that model is produced based on probability |
| CN109146946A (en) * | 2018-08-29 | 2019-01-04 | 河海大学常州校区 | A kind of non local solid matching method of image |
| WO2020125637A1 (en) * | 2018-12-17 | 2020-06-25 | 深圳市道通智能航空技术有限公司 | Stereo matching method and apparatus, and electronic device |
| CN111402313A (en) * | 2020-03-13 | 2020-07-10 | 合肥的卢深视科技有限公司 | Image depth recovery method and device |
Non-Patent Citations (1)
| Title |
|---|
| 王伟等: "基于匹配扩散的多视稠密深度图估计", 《自动化学报》 * |
Cited By (1)
| Publication number | Priority date | Publication date | Assignee | Title |
|---|---|---|---|---|
| CN114820393A (en) * | 2022-06-28 | 2022-07-29 | 合肥的卢深视科技有限公司 | Depth recovery method for fusion hole repair, electronic device and storage medium |
Also Published As
| Publication number | Publication date |
|---|---|
| CN114331919B (en) | 2022-05-31 |
Similar Documents
| Publication | Publication Date | Title |
|---|---|---|
| CN110310320B (en) | Binocular vision matching cost aggregation optimization method | |
| CN107316326B (en) | Edge-based disparity map calculation method and device applied to binocular stereo vision | |
| US20130004079A1 (en) | Image processing apparatus, image processing method, and program thereof | |
| CN111582054B (en) | Point cloud data processing method and device and obstacle detection method and device | |
| CN106651897B (en) | Parallax correction method based on super-pixel segmentation | |
| CN114066950B (en) | Monocular speckle structure optical image matching method, electronic device and storage medium | |
| CN113744315B (en) | Semi-direct vision odometer based on binocular vision | |
| CN114283089B (en) | Jump acceleration based depth recovery method, electronic device, and storage medium | |
| CN116029996A (en) | Stereo matching method and device and electronic equipment | |
| CN114331919B (en) | Depth recovery method, electronic device, and storage medium | |
| Haq et al. | An edge-aware based adaptive multi-feature set extraction for stereo matching of binocular images | |
| CN111105452A (en) | High-low resolution fusion stereo matching method based on binocular vision | |
| CN111739071A (en) | Rapid iterative registration method, medium, terminal and device based on initial value | |
| CN115329111A (en) | Image feature library construction method and system based on point cloud and image matching | |
| CN113763449B (en) | Depth recovery method and device, electronic equipment and storage medium | |
| Chen et al. | Multi-stage matching approach for mobile platform visual imagery | |
| CN114283081B (en) | Depth recovery method based on pyramid acceleration, electronic device and storage medium | |
| CN115965961B (en) | Local-global multi-mode fusion method, system, equipment and storage medium | |
| CN110246169B (en) | Gradient-based window adaptive stereo matching method and system | |
| JP7312026B2 (en) | Image processing device, image processing method and program | |
| CN113808185B (en) | Image depth recovery method, electronic device and storage medium | |
| Luo et al. | Improved orb-slam2 algorithm based on information entropy and image sharpening adjustment | |
| CN107392936B (en) | Target tracking method based on meanshift | |
| CN113379816B (en) | Structure change detection method, electronic device, and storage medium | |
| CN110942102A (en) | Probability relaxation epipolar matching method and system |
Legal Events
| Date | Code | Title | Description |
|---|---|---|---|
| PB01 | Publication | ||
| PB01 | Publication | ||
| SE01 | Entry into force of request for substantive examination | ||
| SE01 | Entry into force of request for substantive examination | ||
| GR01 | Patent grant | ||
| GR01 | Patent grant |