CN111768434A - Disparity map acquisition method and device, electronic equipment and storage medium - Google Patents
Disparity map acquisition method and device, electronic equipment and storage medium Download PDFInfo
- Publication number
- CN111768434A CN111768434A CN202010603208.8A CN202010603208A CN111768434A CN 111768434 A CN111768434 A CN 111768434A CN 202010603208 A CN202010603208 A CN 202010603208A CN 111768434 A CN111768434 A CN 111768434A
- Authority
- CN
- China
- Prior art keywords
- image
- pyramid
- block
- shooting
- main
- Prior art date
- Legal status (The legal status is an assumption and is not a legal conclusion. Google has not performed a legal analysis and makes no representation as to the accuracy of the status listed.)
- Pending
Links
- 238000000034 method Methods 0.000 title claims abstract description 107
- 238000003860 storage Methods 0.000 title claims abstract description 14
- 238000004364 calculation method Methods 0.000 claims abstract description 53
- 230000000903 blocking effect Effects 0.000 claims description 34
- 230000002776 aggregation Effects 0.000 claims description 16
- 238000004220 aggregation Methods 0.000 claims description 16
- 238000012545 processing Methods 0.000 claims description 14
- 238000004590 computer program Methods 0.000 claims description 13
- 238000001914 filtration Methods 0.000 claims description 13
- 238000005070 sampling Methods 0.000 claims description 11
- 238000000638 solvent extraction Methods 0.000 claims description 11
- 238000003709 image segmentation Methods 0.000 claims description 7
- 230000006870 function Effects 0.000 claims description 6
- 238000010276 construction Methods 0.000 claims description 5
- 238000013135 deep learning Methods 0.000 claims description 5
- 238000004422 calculation algorithm Methods 0.000 description 12
- 238000010586 diagram Methods 0.000 description 9
- 230000008569 process Effects 0.000 description 8
- 238000012937 correction Methods 0.000 description 5
- 230000006835 compression Effects 0.000 description 3
- 238000007906 compression Methods 0.000 description 3
- 230000009467 reduction Effects 0.000 description 3
- 238000005516 engineering process Methods 0.000 description 2
- 230000009191 jumping Effects 0.000 description 2
- 230000001360 synchronised effect Effects 0.000 description 2
- 239000013598 vector Substances 0.000 description 2
- 230000000007 visual effect Effects 0.000 description 2
- 230000015572 biosynthetic process Effects 0.000 description 1
- 230000008859 change Effects 0.000 description 1
- 238000004891 communication Methods 0.000 description 1
- 238000013461 design Methods 0.000 description 1
- 238000011161 development Methods 0.000 description 1
- 230000018109 developmental process Effects 0.000 description 1
- 230000002349 favourable effect Effects 0.000 description 1
- 230000004927 fusion Effects 0.000 description 1
- 238000012804 iterative process Methods 0.000 description 1
- 238000004519 manufacturing process Methods 0.000 description 1
- 238000012986 modification Methods 0.000 description 1
- 230000004048 modification Effects 0.000 description 1
- 230000003287 optical effect Effects 0.000 description 1
- 238000004091 panning Methods 0.000 description 1
- 238000007670 refining Methods 0.000 description 1
- 230000011218 segmentation Effects 0.000 description 1
- 230000003068 static effect Effects 0.000 description 1
- 238000003786 synthesis reaction Methods 0.000 description 1
- 230000009466 transformation Effects 0.000 description 1
Images
Classifications
-
- G—PHYSICS
- G06—COMPUTING; CALCULATING OR COUNTING
- G06T—IMAGE DATA PROCESSING OR GENERATION, IN GENERAL
- G06T7/00—Image analysis
- G06T7/30—Determination of transform parameters for the alignment of images, i.e. image registration
-
- G—PHYSICS
- G06—COMPUTING; CALCULATING OR COUNTING
- G06F—ELECTRIC DIGITAL DATA PROCESSING
- G06F18/00—Pattern recognition
- G06F18/20—Analysing
- G06F18/24—Classification techniques
- G06F18/241—Classification techniques relating to the classification model, e.g. parametric or non-parametric approaches
- G06F18/2413—Classification techniques relating to the classification model, e.g. parametric or non-parametric approaches based on distances to training or reference patterns
-
- G—PHYSICS
- G06—COMPUTING; CALCULATING OR COUNTING
- G06T—IMAGE DATA PROCESSING OR GENERATION, IN GENERAL
- G06T7/00—Image analysis
- G06T7/10—Segmentation; Edge detection
- G06T7/11—Region-based segmentation
-
- G—PHYSICS
- G06—COMPUTING; CALCULATING OR COUNTING
- G06V—IMAGE OR VIDEO RECOGNITION OR UNDERSTANDING
- G06V10/00—Arrangements for image or video recognition or understanding
- G06V10/40—Extraction of image or video features
- G06V10/46—Descriptors for shape, contour or point-related descriptors, e.g. scale invariant feature transform [SIFT] or bags of words [BoW]; Salient regional features
- G06V10/462—Salient features, e.g. scale invariant feature transforms [SIFT]
-
- G—PHYSICS
- G06—COMPUTING; CALCULATING OR COUNTING
- G06T—IMAGE DATA PROCESSING OR GENERATION, IN GENERAL
- G06T2207/00—Indexing scheme for image analysis or image enhancement
- G06T2207/10—Image acquisition modality
- G06T2207/10028—Range image; Depth image; 3D point clouds
Abstract
The disparity map obtaining method, the disparity map obtaining device, the electronic device and the computer readable storage medium respectively construct pyramid images for the main shooting image and the auxiliary shooting image to obtain the main shooting pyramid image and the auxiliary shooting pyramid image, each layer of image of the main shooting pyramid image and each layer of image of the auxiliary shooting pyramid image are blocked, and the main shooting pyramid blocked image and the auxiliary shooting pyramid blocked image are adopted to carry out stereo matching to obtain the disparity map. Moreover, for each block image, the parallax range is small, and then the size of the DSI is small, so that the calculation amount can be effectively reduced.
Description
Technical Field
The present application relates to the field of images, and in particular, to a method and an apparatus for obtaining a disparity map, an electronic device, and a storage medium.
Background
With the appearance of dual-camera devices and the continuous development of other terminal devices, the method for obtaining the depth image by calculating the depth information through dual-camera has wider application.
At present, a common method for calculating depth information through double shooting involves a matching cost technology, wherein the matching cost is that data are converted from a two-dimensional image space to a three-dimensional image parallax space, and a parallax dimension is increased. For example, in the process of calculating depth information for double shooting, a disparity range input for matching cost calculation may only include an X direction (Dx >1, Dy ═ 1) or two XY directions (Dx >1, Dy >1), and this range defines a rectangular region ([ X, y, Dx, Dy ]) that may appear for a corresponding point of a pixel P (X, y) on a main shooting image on a sub-shooting image, and through a matching cost calculation step, a point P on the main shooting image may obtain D (D ═ Dx Dy) matching costs, and traverse all pixels on the main shooting image, and each pixel on the main shooting image obtains D matching costs, and a three-dimensional image disparity space is constructed through the D matching costs of each pixel on the main shooting image.
However, the above scheme has problems that the matching precision of the matching cost is not high, and the calculation cost is high.
Disclosure of Invention
The embodiment of the application provides a disparity map acquisition method and device, electronic equipment and a computer-readable storage medium, which can improve matching precision and effectively reduce calculated amount.
In a first aspect, an embodiment of the present application provides a disparity map obtaining method, including:
constructing pyramid images for the main shooting image and the auxiliary shooting image respectively to obtain a main shooting pyramid image and an auxiliary shooting pyramid image;
partitioning each layer of image of the main shooting pyramid image and the auxiliary shooting pyramid image to obtain a main shooting pyramid partitioned image and an auxiliary shooting pyramid partitioned image;
and carrying out stereo matching on the main shooting pyramid block image and the auxiliary shooting pyramid block image to obtain a disparity map.
In one embodiment, the performing stereo matching on the main shooting pyramid block image and the sub shooting pyramid block image to obtain a disparity map includes:
aiming at the main shooting pyramid block images, updating the parallax range of the corresponding block images of the current layer image according to the initial parallax image of each block image of the previous layer image until the parallax range of each first block image is obtained; the first block image is a block image of the image at the bottom layer in the main shooting pyramid block images;
according to the parallax range of each first block image, performing stereo matching on each first block image and the corresponding second block image to obtain an initial parallax image of each first block image; the second block image is a block image corresponding to the first block image in the bottommost layer image of the secondary shooting pyramid block images;
and merging the initial disparity maps of the first block images to obtain the disparity maps.
In one embodiment, the performing stereo matching on each first block image and a corresponding second block image according to a parallax range of each first block image to obtain a parallax map of each first block image includes:
according to the parallax range of each first block image, performing matching cost calculation on each first block image and the corresponding second block image to obtain an image parallax space of each first block image;
performing cost aggregation on the image parallax space of each first block image to obtain an updated image parallax space of each first block image;
and performing parallax calculation on the updated image parallax space of each first block image to obtain an initial parallax image of each first block image.
In one embodiment, the merging the initial disparity maps of the first block images to obtain the disparity map includes:
and merging the initial disparity maps of the first block images into an image aligned with the size of the main shot image to obtain the disparity map.
In one embodiment, the method further comprises:
and carrying out stereo matching on each block image of each layer of image in the main shooting pyramid block images and the corresponding block image in the auxiliary shooting pyramid block images to obtain an initial parallax image of each block image of each layer of image in the main shooting pyramid block images.
In one embodiment, the method further comprises:
and if the current layer image in the main shooting pyramid block image is the topmost layer image, the parallax range of the current layer image in the main shooting pyramid block image is a preset default parallax range.
In one embodiment, the constructing pyramid images for the main shot image and the sub shot image respectively to obtain the main shot pyramid image and the sub shot pyramid image includes:
and respectively taking the main shooting image and the auxiliary shooting image as bottom layers, and performing downsampling processing on the main shooting image and the auxiliary shooting image through a preset sampling frequency to obtain the main shooting pyramid image and the auxiliary shooting pyramid image.
In one embodiment, the constructing pyramid images for the main shot image and the sub shot image respectively further includes, before obtaining the main shot pyramid image and the sub shot pyramid image:
respectively filtering the main shot image and the auxiliary shot image by adopting a Gaussian kernel function with a preset size to obtain a filtered main shot image and a filtered auxiliary shot image;
the method for respectively constructing the pyramid images of the main shooting image and the auxiliary shooting image to obtain the pyramid images of the main shooting image and the auxiliary shooting image comprises the following steps:
and respectively constructing a pyramid image for the filtered main shooting image and the filtered auxiliary shooting image to obtain a main shooting pyramid image and an auxiliary shooting pyramid image.
In one embodiment, the blocking each layer of image of the main shooting pyramid image and the sub shooting pyramid image to obtain a main shooting pyramid blocked image and a sub shooting pyramid blocked image includes:
partitioning each layer of image of the main shooting pyramid image and the auxiliary shooting pyramid image by adopting a preset image partitioning method to obtain a main shooting pyramid partitioned image and an auxiliary shooting pyramid partitioned image; the image blocking method comprises an image segmentation method or a characteristic point clustering method based on deep learning.
In one embodiment, the method further comprises:
and processing the disparity map by a preset hole filling method or a preset filtering method to obtain a processed disparity map.
In one embodiment, the method further comprises:
and obtaining a depth map according to the calibration data and the processed disparity map by adopting a triangulation principle.
In one embodiment, the method further comprises:
performing epipolar line rectification on the parallel double-shot collected images under frame synchronization according to calibration data to obtain a main shot image and a secondary shot image; and the distance between the homonymous points of the main shot image and the auxiliary shot image in the preset direction is smaller than a preset threshold value.
In one embodiment, an overlapping area exists between each block image of the same layer of images of the main shooting pyramid block images; and an overlapping area exists between each block image of the same layer of image of the secondary shooting pyramid block image.
In a second aspect, an embodiment of the present application provides a disparity map obtaining apparatus, including:
the construction module is used for respectively constructing pyramid images for the main shooting image and the auxiliary shooting image to obtain the main shooting pyramid image and the auxiliary shooting pyramid image;
the blocking module is used for blocking each layer of image of the main shooting pyramid image and the auxiliary shooting pyramid image to obtain a main shooting pyramid blocking image and an auxiliary shooting pyramid blocking image;
and the matching module is used for carrying out three-dimensional matching on the main shooting pyramid block image and the auxiliary shooting pyramid block image to obtain a parallax map.
In a third aspect, an embodiment of the present application provides an electronic device, including a memory and a processor, where the memory stores a computer program, and when the computer program is executed by the processor, the processor executes the steps of the disparity map acquiring method according to any one of the first aspect.
In a fourth aspect, embodiments of the present application provide a computer-readable storage medium, on which a computer program is stored, which, when executed by a processor, implements the steps of the method of any one of the first aspect.
The disparity map obtaining method, the disparity map obtaining device, the electronic device and the computer readable storage medium respectively construct pyramid images for the main shooting image and the auxiliary shooting image to obtain the main shooting pyramid image and the auxiliary shooting pyramid image, each layer of image of the main shooting pyramid image and each layer of image of the auxiliary shooting pyramid image are blocked, and the main shooting pyramid blocked image and the auxiliary shooting pyramid blocked image are adopted to carry out stereo matching to obtain the disparity map. Moreover, for each block image, the parallax range is small, and then the size of the DSI is small, so that the calculation amount can be effectively reduced.
Drawings
In order to more clearly illustrate the embodiments of the present application or the technical solutions in the prior art, the drawings used in the description of the embodiments or the prior art will be briefly described below, it is obvious that the drawings in the following description are only some embodiments of the present application, and for those skilled in the art, other drawings can be obtained according to the drawings without creative efforts.
Fig. 1 is a flowchart of calculating depth information for panning according to an embodiment;
FIG. 2 is a schematic diagram of an exemplary embodiment of an application environment of a disparity map acquisition method;
fig. 3 is a flowchart of a disparity map obtaining method according to an embodiment;
fig. 4 is a flowchart of a disparity map obtaining method according to an embodiment;
fig. 5 is a flowchart of a disparity map obtaining method according to an embodiment;
FIG. 6 is a block image diagram provided in accordance with an embodiment;
FIG. 7 is a flow diagram for bi-shot computed depth information provided by an embodiment;
fig. 8 is a block diagram of a disparity map obtaining apparatus according to an embodiment;
fig. 9 is a block diagram of a disparity map obtaining apparatus according to an embodiment;
fig. 10 is a schematic diagram of an internal structure of an electronic device in one embodiment.
Detailed Description
In order to make the objects, technical solutions and advantages of the present application more apparent, the present application is described in further detail below with reference to the accompanying drawings and embodiments. It should be understood that the specific embodiments described herein are merely illustrative of the present application and are not intended to limit the present application.
It will be understood that, as used herein, the terms "first," "second," and the like may be used herein to describe various elements, but these elements are not limited by these terms. These terms are only used to distinguish one element from another. For example, a first client may be referred to as a second client, and similarly, a second client may be referred to as a first client, without departing from the scope of the present application. Both the first client and the second client are clients, but they are not the same client.
Fig. 1 is a flowchart of calculating depth information by bi-shooting according to an embodiment, and as shown in fig. 1, the method includes: STEP 1): performing polar line correction on the main and auxiliary shooting images, and outputting a main and auxiliary shooting image with the polar line corrected to be horizontal (master is the main shooting and slave is the auxiliary shooting); STEP 2): performing matching cost calculation on the main shot image and the auxiliary shot image to construct an image parallax space; STEP 3): performing cost aggregation on the image parallax space, and updating the image parallax space; STEP 4) calculating according to the updated image parallax space memorability parallax and outputting an initial parallax map; STEP 5): performing parallax refinement on the initial parallax image, and outputting a high-precision parallax image; STEP 6): and combining the high-precision disparity map and the calibration data, and calculating the depth map by using the triangulation principle.
STEP 2) the matching cost calculation converts data from a two-dimensional Image Space to a three-dimensional Image Disparity Space (DSI), adding one Disparity dimension. STEP3) cost aggregation is carried out on the DSI for noise reduction, and STEP 4) parallax calculation is carried out on the DSI for dimension reduction to obtain a parallax map aligned with an image space. The three steps are core steps of double-shot depth calculation.
Step 2), the parallax range input by the matching cost calculation may only include an X direction (Dx >1, Dy ═ 1) or two XY directions (Dx >1, Dy >1), and this range defines a rectangular region ([ X, y, Dx, Dy ]) that may appear for the corresponding point of the pixel P (X, y) on the main image on the sub image, and through the matching cost calculation step, the point P on the main image obtains D (D ═ Dx × Dy) matching costs, and traverses all the pixels (resolution W × H) on the main image, thereby forming a DSI of the three-dimensional cost cube with a size W × H. Step 2, outputting DSI with a large amount of noise, and step3, performing noise reduction operation on the DSI. And 4, calculating the minimum cost on the DSI to obtain an initial disparity map.
According to the scheme, a pair of double-shot images corresponds to a parallax range (Dx, Dy), and in order to successfully match all the areas, the parallax range must cover the maximum value and the minimum value of the parallax in a visual field, however, the extreme parallax value possibly occupies a small ratio in pixels, so that the scheme has a large number of areas to be matched, and the matching range of the areas is far larger than the real parallax range. In other words, the confidence area of the matching is not accurate enough, which is harmful to the matching accuracy, especially in the difficult scenes of repeated texture, weak texture and the like, the larger the confidence area is, the higher the probability of error is.
In addition, as can be seen from the above description, the steps 2, 3 and 4 operate on the DSI of the three-dimensional cost cube, and the calculation amount is large, and is proportional to the size of the DSI. In the case of image size determination, the parallax range determines the amount of calculation of the algorithm. Also, to cover the full-view disparity extremum, the fixed disparity range must be set large, which greatly increases the computational cost and reduces the real-time performance.
Therefore, the embodiment of the application provides a disparity map acquisition method, which is used for solving the problems of low matching precision and high calculation cost.
Fig. 2 is a schematic diagram of an application environment of the disparity map obtaining method in one embodiment. As shown in fig. 2, the application environment includes an image pickup apparatus 1, an image pickup apparatus 2, and an electronic apparatus 3, and the image pickup apparatus 1 and the image pickup apparatus 2 are respectively connected to the electronic apparatus 3 in communication. The camera device 1 and the camera device 2 may be cameras or other video capturing devices, and the electronic device 3 may be, but is not limited to, various personal computers, notebook computers, smart phones, tablet computers, portable wearable devices, and the like. It should be noted that the image pickup apparatus 1 and the image pickup apparatus 2 may be integrated with the electronic apparatus 3 or may be provided separately from the electronic apparatus. The image pickup apparatus 1 and the image pickup apparatus 2 photograph the same object of the same scene from different angles of view, and transmit the photographed double-shot image to the electronic apparatus 3, and the electronic apparatus 3 performs double-shot depth calculation from the double-shot image.
It should be noted that the application range of the disparity map obtaining method provided in the embodiment of the present application is not limited to bi-shot depth calculation, and the method can be used in an application scenario in which depth sensors such as Time of flight (TOF) and structured light calculate depth.
Fig. 3 is a flowchart of a disparity map obtaining method according to an embodiment. The disparity map obtaining method in this embodiment is described by taking the example of the disparity map obtaining method executed on the electronic device in fig. 1. As shown in fig. 3, the disparity map acquiring method includes the following steps:
s301, constructing pyramid images for the main shooting image and the auxiliary shooting image respectively to obtain the main shooting pyramid image and the auxiliary shooting pyramid image.
The main shooting image and the auxiliary shooting image are two images collected by the double-shooting equipment, and the main shooting image and the auxiliary shooting image can be images obtained by shooting different visual angles of the same object in the same scene by the double-shooting equipment. As shown in fig. 2, if the image capturing apparatus 1 is a main shooting and the image capturing apparatus 2 is a sub shooting, the image captured by the image capturing apparatus 1 is a main shooting image, and the image captured by the image capturing apparatus 2 is a sub shooting image; alternatively, if the image capturing apparatus 2 is a main shooting and the image capturing apparatus 1 is a sub-shooting, the image captured by the image capturing apparatus 2 is a main shooting image and the image captured by the image capturing apparatus 1 is a sub-shooting image.
In this embodiment, the main shooting pyramid image and the sub shooting pyramid image can be obtained by respectively constructing the main shooting image and the sub shooting image. For example, the main shot image and the sub shot image are sampled by a down sampling scheme to obtain a main shot pyramid image and a sub shot pyramid image, and for example, the main shot image and the sub shot image are sampled at resolutions of 256 × 256, 128 × 128, 64 × 64, and 32 × 32 to obtain a main shot pyramid image of 4 layers and a sub shot pyramid image of 4 layers. The number of layers of the main shooting pyramid image and the auxiliary shooting pyramid image is equal, and the image sizes of the corresponding layers of the main shooting pyramid image and the auxiliary shooting pyramid image are the same. For example, N-layer gaussian pyramid images may be constructed for the main image and the sub-image, respectively, or N-layer laplacian pyramid images may be constructed for the main image and the sub-image, respectively. The embodiments of the present application are not limited.
And S302, partitioning each layer of image of the main shooting pyramid image and the auxiliary shooting pyramid image to obtain a main shooting pyramid partitioned image and an auxiliary shooting pyramid partitioned image.
In this embodiment, each layer of images of the main shooting pyramid image and the sub shooting pyramid image may be partitioned into a plurality of partitioned images, and the number of the partitioned images of each layer of the main shooting pyramid image and the sub shooting pyramid image is the same. For example, each layer image of the main and sub-photographic pyramid images may be divided into M block images of equal size; alternatively, each layer of images of the main shooting pyramid image and the sub shooting pyramid image may be divided into M block images according to an image segmentation method, or each layer of images of the main shooting pyramid image and the sub shooting pyramid image may be divided into M block images by using a feature point clustering method, which is not limited in the embodiment of the present application.
And S303, carrying out stereo matching on the main shooting pyramid block image and the auxiliary shooting pyramid block image to obtain a disparity map.
In an embodiment, stereo matching is performed on each block image of the main shooting pyramid block image and a corresponding block image in the sub shooting pyramid block to obtain a disparity map of each block image, the disparity maps of the block images of each layer of images can be merged to obtain a disparity map of each layer of images, and the disparity map of the bottom layer of images can be used as the disparity map of the main shooting image. Or, the disparity range of the corresponding block image of the current layer can be further iterated layer by layer according to the disparity map of the block image of the previous layer to obtain the disparity map of each block image of the bottom layer image, and the disparity maps of each block image of the bottom layer image are combined to obtain the disparity map of the main shooting image. Or merging the disparity maps of the block images of the previous layer image to obtain the disparity map of the block image of the previous layer image, updating the disparity range of the current layer image according to the disparity map of the block image of the previous layer image, and iterating layer by layer to obtain the disparity map of the bottommost layer image as the disparity map of the main shooting image. The embodiments of the present application are not limited.
The disparity map obtaining method provided by the embodiment of the application includes the steps of respectively constructing pyramid images on a main shooting image and an auxiliary shooting image to obtain the main shooting pyramid image and the auxiliary shooting pyramid image, partitioning each layer of image of the main shooting pyramid image and the auxiliary shooting pyramid image, and performing stereo matching by adopting the main shooting pyramid partitioned images and the auxiliary shooting pyramid partitioned images to obtain the disparity map. Moreover, for each block image, the parallax range is small, and then the size of the DSI is small, so that the calculation amount can be effectively reduced.
Fig. 4 is a flowchart of a disparity map obtaining method according to an embodiment, where the embodiment relates to a specific implementation process of the step S303, and as shown in fig. 4, the method includes the following steps:
s401, aiming at the main shooting pyramid block images, updating the parallax range of the corresponding block images of the current layer image according to the initial parallax image of each block image of the previous layer image until the parallax range of each first block image is obtained; the first block image is a block image of the image at the bottom layer in the main shooting pyramid block image.
The bottom layer image in the shooting pyramid blocking image is a main shooting image, and the bottom layer image in the auxiliary shooting pyramid blocking image is an auxiliary shooting image.
In this embodiment, for a main shooting pyramid block image, according to an initial disparity map of a block image of an image on a layer above the pyramid obtained through calculation, a disparity range of a corresponding block image of a current layer image is updated, and iteration is performed layer by layer until a disparity range of a block image of a bottom layer image is obtained.
For example, a main shooting pyramid image is constructed on the main shooting image, and sequentially marked as PY _1, PY _2, and PY _ n from top to bottom, wherein a subscript n represents an nth layer of the pyramid; each layer of image of the main shooting pyramid image is partitioned, and a PY _ n layer is taken as an example, a partitioning operation is performed on the PY _ n layer and is partitioned into M blocks, and each partitioned image is marked as Tile _ n _1, Tile _ n _2, and. And updating the parallax range of the current image block Tile _ n _ m by using the initial parallax map Disp _ n-1_ m of the block image obtained by the pyramid calculation of the previous layer, wherein Disp _ n-1_ m represents the mth block image of the (n-1) th layer in the pyramid block image.
Optionally, if the current layer image in the main shooting pyramid block image is the topmost layer image, the parallax range of the current layer image in the main shooting pyramid block image is a preset default parallax range.
In this embodiment, for the top-most image in the main shot pyramid block image, a preset default parallax range may be adopted, and the default parallax range may be a parallax range determined according to image local features and a matching technique, for example, a Scale-invariant feature transform (SIFT) operator is adopted to extract feature points of the main shot image and the sub-shot image, the feature points are described as high-dimensional feature vectors, the high-dimensional feature vectors are matched by a proximity algorithm (k-nearest neighbor, kNN), outliers are removed by RANSAC, and then the default parallax range is determined.
Further, the disparity range of the corresponding block image of the current layer image is updated according to the initial disparity map of each block image of the previous layer image, and then before updating the disparity range, stereo matching needs to be performed on each block image of each layer image in the main shooting pyramid block image and the corresponding block image in the sub shooting pyramid block image, so as to obtain the initial disparity map of each block image of each layer image in the main shooting pyramid block image.
In this embodiment, the block image of the nth layer image of the main shooting pyramid block image is master _ n _ m, and the block image of the nth layer image of the sub shooting pyramid block image is slave _ n _ m, and then the master _ n _ m and the slave _ n _ m are subjected to stereo matching according to the disparity range of the master _ n _ m, so as to obtain an initial disparity map Disp _ n _ m of each master _ n _ m. The initial parallax map of each block image of each layer of image is calculated respectively, the calculation amount can be greatly reduced due to the small size of each block image and the small corresponding parallax range, and moreover, the calculation efficiency is improved due to the parallel calculation of a plurality of block images on the same layer. In addition, in the operation process, the parallax range of part of the block images can be adjusted, namely different parallax matching ranges can be adopted for different image parts in a differentiated mode, the error probability in cost matching can be effectively reduced, and the initial parallax precision is improved.
S402, according to the parallax range of each first block image, performing stereo matching on each first block image and the corresponding second block image to obtain an initial parallax image of each first block image; the second block image is a block image corresponding to the first block image in the bottommost layer image of the sub-shooting pyramid block image.
In this embodiment, stereo matching may be performed on a first block image in the main shooting pyramid block image and a corresponding second block image in the sub shooting pyramid block image according to a parallax range of the first block image, so as to obtain an initial parallax map of each first block image.
For example, if the block image in the bottom layer image of the main shooting pyramid block image is master _ n _ m, and the block image in the bottom layer image of the sub shooting pyramid block image is slave _ n _ m, the first block image master _ n _ m and the second block image slave _ n _ m are subjected to stereo matching according to the disparity range of the first block image master _ n _ m, so as to obtain the initial disparity map Disp _ n _ m of the first block image master _ n _ m.
And S403, merging the initial disparity maps of the first block images to obtain a disparity map.
In this embodiment, the initial disparity maps of the respective first block images may be combined to obtain a disparity map of the main-shot image. For example, if the block images in the bottom layer image of the main shooting pyramid block image are master _ n _1, master _ n _2, … …, and master _ n _ m, and the corresponding initial disparity maps are Disp _ n _1, Disp _ n _2, … …, and Disp _ n _ m, respectively, Disp _ n _1, Disp _ n _2, … …, and Disp _ n _ m are merged to obtain the disparity map of the main shooting image.
Optionally, merging the initial disparity maps of the first block images to obtain a disparity map, where the merging includes: and merging the initial disparity maps of the first block images into an image aligned with the size of the main shot image to obtain a disparity map. In this embodiment, all the partitioned images on the nth-layer pyramid are traversed, and if the traversal is completed, the partitioned initial disparities Disp _ n _ 1.. once, Disp _ n _ m are combined into a disparity map Disp _ n aligned with the size of the main shot image; otherwise, return to step S401.
The disparity map obtaining method provided in the embodiment of the application updates, for a main shooting pyramid block image, a disparity range of a corresponding block image of a current layer image according to an initial disparity map of each block image of a previous layer image until a disparity range of each first block image is obtained, and obtains, according to the disparity range of each first block image, carrying out stereo matching on each first block image and the corresponding second block image to obtain an initial disparity map of each first block image, and combining the initial parallax maps of the first block images to obtain a parallax map, iteratively updating the parallax range of the block images in each layer of image from the topmost layer of the pyramid image to the bottommost layer of image in sequence, iterating until the bottommost layer of image is obtained, obtaining the parallax map of the main shot image by adopting the parallax map of each block image of the bottommost layer of image, and iterating the parallax range of the block images to improve the precision of the parallax map of the main shot image. In the iterative process, the parallax range of the local image (block image) can be adaptively adjusted, and different parallax matching ranges can be adopted for different local images (block images) in a differentiated manner, so that the error probability in cost matching can be effectively reduced, and the initial parallax precision is improved.
In the embodiment shown in fig. 4, stereo matching generally includes matching cost calculation, cost aggregation and disparity calculation, and the specific implementation process of step S402 is explained in detail below by taking fig. 5 as an example. As shown in fig. 5, the stereo matching of each first block image and the corresponding second block image according to the parallax range of each first block image to obtain the parallax map of each first block image includes:
s501, according to the parallax range of each first block image, matching cost calculation is carried out on each first block image and the corresponding second block image, and the image parallax space of each first block image is obtained.
In this embodiment, the parallax range of each block image of the bottom layer image in the main shot image is determined, and according to the parallax range, matching cost calculation is performed on each block image of the bottom layer image in the main shot image and the corresponding block image in the bottom layer image in the sub shot image, so as to obtain the image parallax space of each block image of the bottom layer image in the main shot image. For example, according to the disparity range of the first segmented image master _ n _ m, the matching cost calculation is performed on the first segmented image master _ n _ m and the second segmented image slave _ n _ m, so as to obtain the image disparity space DSI _ n _ m of the first segmented image master _ n _ m.
Optionally, the matching cost calculation method may be a matching algorithm based on the sum of absolute gray differences, a matching algorithm based on the sum of squared gray differences, a matching algorithm based on statistical census transformation, and the like, and the embodiments of the present application are not limited thereto.
And S502, performing cost aggregation on the image parallax space of each first block image to obtain an updated image parallax space of each first block image.
In the present embodiment, the image parallax spaces of the respective block images of the bottommost layer image in the main shot image are subjected to cost aggregation to update the image parallax spaces. For example, cost aggregation is performed on the image parallax space DSI _ n _ m of the first block image master _ n _ m to obtain an updated image parallax space XDSI _ n _ m.
Optionally, the cost aggregation method may be a Cross-based cost aggregation algorithm (CBCA), a Minimum Spanning Tree (MST) based cost aggregation algorithm, a pyramid change Cross-scale JBF based cost aggregation, and the like, which is not limited in the embodiment of the present application.
And S503, performing parallax calculation on the updated image parallax space of each first block image to obtain an initial parallax image of each first block image.
In this embodiment, parallax calculation is performed on the updated image parallax space of each block image of the bottom layer image in the main shot image, so as to obtain an initial parallax map of each first block image. For example, the XDSI _ n _ m of the first block image is subjected to disparity calculation, so as to obtain an initial disparity map Disp _ n _ m of each first block image.
Alternatively, the parallax calculation method may use an absolute difference method, a graph segmentation method, a fixed window-based region matching method, a left-right consistency constraint principle, and the like, and the embodiment of the present application is not limited.
It should be noted that, when performing stereo matching on each first block image and the corresponding second block image, multiple block images of the same layer image may be calculated in parallel, so as to improve the matching speed and facilitate engineering.
According to the disparity map obtaining method provided by the embodiment of the application, according to the disparity range of each first block image, matching cost calculation is carried out on each first block image and the corresponding second block image to obtain the image disparity space of each first block image, cost aggregation is carried out on the image disparity space of each first block image to obtain the updated image disparity space of each first block image, disparity calculation is carried out on the updated image disparity space of each first block image to obtain the initial disparity map of each first block image, the DSI size is effectively reduced based on the operation of the block images and the compression of the block images on the disparity range, and the purposes of reducing the calculated amount and improving the algorithm instantaneity are achieved.
In one embodiment, constructing pyramid images for the main shot image and the sub shot image, and optionally constructing pyramid images for the main shot image and the sub shot image respectively, to obtain the main shot pyramid image and the sub shot pyramid image, may be performed by a downsampling method, including: and respectively taking the main shot image and the auxiliary shot image as bottom layers, and performing downsampling processing on the main shot image and the auxiliary shot image through a preset sampling frequency to obtain a main shot pyramid image and an auxiliary shot pyramid image.
In this embodiment, the main shot image and the sub-shot image are respectively used as the bottom layer image (nth layer) of the pyramid, and then the main shot image and the sub-shot image are subjected to down-sampling processing by a preset sampling frequency, so as to obtain a main shot pyramid image and a sub-shot pyramid image. For example, the main-shot image and the sub-shot image are respectively sampled at a sampling frequency of 1/2 to obtain an n-1 layer image of the main-shot pyramid image and the sub-shot pyramid image, then the n-1 layer of the main-shot pyramid image and the sub-shot pyramid image are sampled at a sampling frequency of 1/2 to obtain an n-2 layer of the main-shot pyramid image and the sub-shot pyramid image, and so on to obtain the main-shot pyramid image and the sub-shot pyramid image.
According to the parallax image acquisition method provided by the embodiment of the application, the main shooting image and the auxiliary shooting image are respectively used as bottom layers, and the main shooting image and the auxiliary shooting image are subjected to down-sampling processing through the preset sampling frequency to obtain the main shooting pyramid image and the auxiliary shooting pyramid image.
Further, in an embodiment, before constructing the pyramid images for the main shot image and the sub shot image respectively to obtain the main shot pyramid image and the sub shot pyramid image, the method further includes: adopting a Gaussian kernel function with a preset size to respectively carry out filtering processing on the main shot image and the auxiliary shot image to obtain a filtered main shot image and a filtered auxiliary shot image;
the method for respectively constructing the main shooting image and the auxiliary shooting image into the pyramid images to obtain the main shooting pyramid image and the auxiliary shooting pyramid image comprises the following steps: and respectively constructing pyramid images for the filtered main shooting image and the filtered auxiliary shooting image to obtain the main shooting pyramid image and the auxiliary shooting pyramid image.
In this embodiment, before constructing the main shooting pyramid image and the sub-shooting pyramid image, filtering processing may be performed on the main shooting image and the sub-shooting image respectively by using a gaussian kernel function with a preset size, so as to remove interference of image noise and improve the quality of the main shooting pyramid image and the sub-shooting pyramid image.
In the above-described embodiment, after the pyramid images are constructed for the main photographic image and the sub-photographic image, it is also necessary to block each layer of the images of the main photographic pyramid image and the sub-photographic pyramid image. In one embodiment, the blocking each layer of image of the main shooting pyramid image and the sub shooting pyramid image to obtain a main shooting pyramid blocked image and a sub shooting pyramid blocked image includes: partitioning each layer of image of the main shooting pyramid image and the auxiliary shooting pyramid image by adopting a preset image partitioning method to obtain a main shooting pyramid partitioned image and an auxiliary shooting pyramid partitioned image; the image blocking method comprises an image segmentation method or a characteristic point clustering method based on deep learning.
In this embodiment, the strategy of image blocking does not necessarily adopt uniform blocking, and a more appropriate blocking image can be calculated by combining the traditional image segmentation technology, the image segmentation/matting based on deep learning, the feature point clustering and other methods, so that the blocked image can more accurately reflect some features of the shot object, and the effectiveness of the blocking image is improved.
Further, an overlapping area exists between each block image of the same layer of image of the main shooting pyramid block image; and an overlapping area exists between each block image of the same layer of image of the secondary shooting pyramid block image.
In this embodiment, a certain overlap area may exist between the block images of the same layer of image, that is, an edge portion of the block images of the same layer of image may have a certain overlap, an overlap area may exist between each adjacent block image of the same layer of image, or an overlap area exists between part of adjacent block images, which is not limited in this embodiment of the application. The overlapped area may belong to two block images or four block images, and when the parallax is calculated for the block images, parallax values are calculated respectively, and the average value is calculated during fusion.
As shown in fig. 6, there is an overlapping area between the plurality of patch images, and the overlapping area is an overlap between the plurality of patch images. Generally, the number of X-direction overlapping pixels needs to be greater than the maximum value of the X-direction parallax range, and the number of Y-direction overlapping pixels needs to be greater than the maximum value of the Y-direction parallax range.
In the embodiment, an overlapping area exists between each block image of the same layer of images of the main shooting pyramid block image; the method has the advantages that the overlapping regions exist among the block images of the same layer of image of the sub-shooting pyramid block image, so that the overlapping portions exist among the adjacent block images of the same layer of image, the edge of each block image has the overlapping portions, the edge portions of the disparity map are strengthened when the disparity map of each block image is calculated, and the edge integrity of the disparity map can be ensured.
In the above embodiment, after the disparity map is obtained by calculation, the method may further include: and processing the disparity map by a preset hole filling method or a preset filtering method to obtain a processed disparity map. In this embodiment, the initial disparity map is refined by using algorithms such as hole filling and filtering, so as to obtain a high-precision disparity map. Or after the initial disparity map of each block image is calculated, the disparity map of each block image is processed by a preset hole filling method or a preset filtering method, so that the precision of the disparity map of each block image is improved.
Further, after obtaining the disparity map of the main shot image, the method further comprises the following steps: and obtaining a depth map according to the calibration data and the processed disparity map by adopting a triangulation principle. Because the processed disparity map is obtained according to the main shooting pyramid block image and the auxiliary shooting pyramid block image, the pyramid images are blocked in the operation process, the local matching disparity range of the images can be adjusted in a self-adaptive mode, different disparity matching ranges can be adopted for different image parts in a differentiated mode, the error probability in cost matching can be effectively reduced, and the initial disparity precision is improved. And the DSI size is effectively reduced based on the operation of the block images and the compression of the parallax range, so that the purposes of reducing the calculated amount and improving the algorithm instantaneity are achieved. In addition, multi-block image parallel operation can be realized for each layer of image of the pyramid, and engineering is facilitated.
As can be seen from the above description of the embodiments, in the calculation process of the embodiments of the present application, the main shot image and the sub shot image are required, and then the image quality of the main shot image and the image quality of the sub shot image must meet the calculation requirement. In one embodiment, epipolar rectification is carried out on parallel double-shot collected images under frame synchronization according to calibration data to obtain a main shot image and a secondary shot image; the distance between the homonymous points of the main shot image and the auxiliary shot image in the preset direction is smaller than a preset threshold value. And performing polar line rectification on the parallel double-shot collected images under frame synchronization according to the calibration data to ensure that the optical axes of the two cameras are completely parallel, so that the subsequent depth calculation and three-dimensional reconstruction are continued according to the main shot image and the auxiliary shot image, and the accuracy and reliability of the depth calculation are improved.
Fig. 7 is a flowchart of bi-shooting depth information calculation according to an embodiment, and as shown in fig. 7, the method includes the following steps:
s701, correcting polar lines: polar line correction is carried out on the parallel double-shot collected images under frame synchronization by using production line calibration data, and the polar line errors of the images after the correction of the main shot images and the auxiliary shot images meet the conditions; conditions are as follows: the distance of the homonymous point on the image in the y direction is less than a preset threshold value
S702, constructing an image pyramid: constructing a double-shot N (N > ═ 2) layer Gaussian pyramid, which is marked as PY _1, PY _2, and PY _ N from top to bottom, wherein a subscript N represents the nth layer of the pyramid;
s703, image blocking: performing a blocking operation on a pyramid PY _ n layer (n is 1 when the pyramid is entered for the first time), blocking M blocks, and leaving proper overlaps between blocked images, wherein the blocked image blocks are marked as Tile _ n _1, Tile _ n _2,. and Tile _ n _ M, where n denotes a pyramid layer number and M denotes a block number;
s704, updating the block parallax range: and updating the parallax matching range of the current image block Tile _ n _ m by using the initial parallax map Disp _ n-1 calculated by the previous pyramid. If the current pyramid is the uppermost layer, adopting a default parallax range;
s705, matching cost calculation: performing matching cost calculation on the partitioned image Tile _ n _ m to obtain an image parallax space DSI _ n _ m;
s706, cost aggregation: cost aggregation is executed on the image parallax space DSI _ n _ m, and the DSI _ n _ m is updated;
s707, parallax calculation: performing parallax calculation on the image parallax space DSI _ n _ m to obtain a partitioned initial parallax map Disp _ n _ m;
s708, whether all image blocks on the nth layer pyramid are traversed or not is judged, and if the traversal is finished, S709 is executed; if not, if m is m +1, jumping to the step S704;
s709, block parallax synthesis: merging the partitioned initial disparities Disp _ n _1,.. and Disp _ n _ m into a disparity map Disp _ n aligned with the full-size image;
s7010, traversing all layers of the top-down pyramid, and if traversing is finished, executing S7011; otherwise, if n is n +1, jumping to the step S703;
s7011, parallax refinement: and (5) refining the initial disparity map by using algorithms such as hole filling and filtering to obtain a high-precision disparity map.
S7012, calculating the triangulation depth: and calculating a depth map according to a triangulation principle by using the calibration data and the disparity map.
In the embodiment of the application, the local matching parallax range of the image can be adaptively adjusted, different parallax matching ranges can be adopted for different local images in a differentiated mode, the error probability in cost matching can be effectively reduced, and the initial parallax precision is improved. Based on block operation and parallax range compression, DSI size is effectively reduced, and the purposes of reducing calculated amount and improving algorithm instantaneity are achieved. In addition, the calculation flow outlined by the dotted line is a complete and independent stereo matching process, and the design is favorable for realizing multi-block parallel operation and engineering.
It should be understood that although the various steps in the flowcharts of fig. 3-5 are shown in order as indicated by the arrows, the steps are not necessarily performed in order as indicated by the arrows. The steps are not performed in the exact order shown and described, and may be performed in other orders, unless explicitly stated otherwise. Moreover, at least some of the steps in fig. 3-5 may include multiple sub-steps or multiple stages that are not necessarily performed at the same time, but may be performed at different times, and the order of performance of the sub-steps or stages is not necessarily sequential, but may be performed in turn or alternating with other steps or at least some of the sub-steps or stages of other steps.
Fig. 8 is a block diagram of a disparity map obtaining apparatus according to an embodiment. As shown in fig. 8, the apparatus includes:
the construction module 11 is configured to respectively construct pyramid images for the main shooting image and the sub shooting image, so as to obtain a main shooting pyramid image and a sub shooting pyramid image;
a blocking module 12, configured to block each layer of image of the main shooting pyramid image and the sub shooting pyramid image to obtain a main shooting pyramid blocking image and a sub shooting pyramid blocking image;
and the matching module 13 is configured to perform stereo matching on the main shooting pyramid block image and the auxiliary shooting pyramid block image to obtain a disparity map.
In the disparity map acquisition apparatus provided in the embodiment of the present application, the construction module 11 constructs pyramid images for the main shot image and the sub shot image, respectively, to obtain the main shot pyramid image and the sub shot pyramid image; the blocking module 12 blocks each layer of images of the main shooting pyramid image and the auxiliary shooting pyramid image to obtain a main shooting pyramid blocking image and an auxiliary shooting pyramid blocking image; the matching module 13 performs stereo matching on the main shooting pyramid block image and the auxiliary shooting pyramid block image to obtain a disparity map. Because each block image has a small size and a small corresponding parallax range, the probability of occurrence of extreme parallax values is also reduced, and the corresponding matching range is relatively close to the real parallax range, the parallax range of the block images can be adjusted in a self-adaptive manner, so that the aim of improving the matching precision is fulfilled. Moreover, for each block image, the parallax range is small, and then the size of the DSI is small, so that the calculation amount can be effectively reduced.
In one embodiment, the matching module 13 includes:
the updating unit is used for updating the parallax range of the corresponding block image of the current layer image according to the initial parallax image of each block image of the previous layer image aiming at the main shooting pyramid block image until the parallax range of each first block image is obtained; the first block image is a block image of the image at the bottom layer in the main shooting pyramid block images;
the stereo matching unit is used for carrying out stereo matching on each first block image and the corresponding second block image according to the parallax range of each first block image to obtain an initial parallax image of each first block image; the second block image is a block image corresponding to the first block image in the bottommost layer image of the secondary shooting pyramid block images;
and the merging unit is used for merging the initial disparity maps of the first block images to obtain the disparity maps.
In one embodiment, the stereo matching unit is configured to perform matching cost calculation on each first block image and a corresponding second block image according to a parallax range of each first block image, so as to obtain an image parallax space of each first block image; performing cost aggregation on the image parallax space of each first block image to obtain an updated image parallax space of each first block image; and performing parallax calculation on the updated image parallax space of each first block image to obtain an initial parallax image of each first block image.
In one embodiment, the merging unit is configured to merge the initial disparity maps of the first block images into an image aligned with the size of the main shot image, so as to obtain the disparity map.
In one embodiment, the stereo matching unit is further configured to perform stereo matching on each block image of each layer of image in the main shooting pyramid block image and a corresponding block image in the sub shooting pyramid block image to obtain an initial disparity map of each block image of each layer of image in the main shooting pyramid block image.
In one embodiment, if the current layer image in the main shooting pyramid block image is the topmost image, the parallax range of the current layer image in the main shooting pyramid block image is a preset default parallax range.
In one embodiment, the construction module 11 is configured to take the main shot image and the sub shot image as bottom layers, and perform downsampling processing on the main shot image and the sub shot image through a preset sampling frequency to obtain the main shot pyramid image and the sub shot pyramid image.
In one embodiment, as shown in fig. 9, the apparatus further comprises:
a filtering module 14, configured to perform filtering processing on the main shot image and the sub shot image respectively by using a gaussian kernel function with a preset size, so as to obtain a filtered main shot image and a filtered sub shot image;
and the constructing module 11 is configured to respectively construct a pyramid image for the filtered main shot image and the filtered sub shot image, so as to obtain a main shot pyramid image and a sub shot pyramid image.
In one embodiment, the blocking module 12 is configured to block each layer of images of the main shooting pyramid image and the sub shooting pyramid image by using a preset image blocking method to obtain a main shooting pyramid blocking image and a sub shooting pyramid blocking image; the image blocking method comprises an image segmentation method or a characteristic point clustering method based on deep learning.
In one embodiment, as shown in fig. 9, the apparatus further comprises:
and the processing module 15 is configured to process the disparity map by using a preset hole filling method or a preset filtering method to obtain a processed disparity map.
In one embodiment, the processing module 15 is further configured to obtain a depth map according to the calibration data and the processed disparity map by using a triangulation principle.
In one embodiment, as shown in fig. 9, the apparatus further comprises:
the correction module 16 is configured to perform epipolar line correction on the parallel dual-shot acquired images under frame synchronization according to the calibration data to obtain the main shot image and the auxiliary shot image; and the distance between the homonymous points of the main shot image and the auxiliary shot image in the preset direction is smaller than a preset threshold value.
In one embodiment, an overlapping area exists between each block image of the same layer of images of the main shooting pyramid block images; and an overlapping area exists between each block image of the same layer of image of the secondary shooting pyramid block image.
The division of each module in the disparity map acquisition apparatus is only used for illustration, and in other embodiments, the disparity map acquisition apparatus may be divided into different modules as needed to complete all or part of the functions of the disparity map acquisition apparatus.
For specific limitations of the disparity map acquisition device, reference may be made to the above limitations on the disparity map acquisition method, which are not described herein again. The modules in the disparity map acquisition device can be wholly or partially realized by software, hardware and a combination thereof. The modules can be embedded in a hardware form or independent from a processor in the computer device, and can also be stored in a memory in the computer device in a software form, so that the processor can call and execute operations corresponding to the modules.
Fig. 10 is a schematic diagram of an internal structure of an electronic device in one embodiment. As shown in fig. 10, the electronic device includes a processor and a memory connected by a system bus. Wherein, the processor is used for providing calculation and control capability and supporting the operation of the whole electronic equipment. The memory may include a non-volatile storage medium and an internal memory. The non-volatile storage medium stores an operating system and a computer program. The computer program can be executed by a processor to implement a disparity map acquisition X method provided in the following embodiments. The internal memory provides a cached execution environment for the operating system computer programs in the non-volatile storage medium. The electronic device may be any terminal device such as a mobile phone, a tablet computer, a PDA (Personal digital assistant), a Point of sale (POS), a vehicle-mounted computer, and a wearable device.
The implementation of each module in the disparity map obtaining apparatus provided in the embodiments of the present application may be in the form of a computer program. The computer program may be run on a terminal or a server. Program modules constituted by such computer programs may be stored on the memory of the electronic device. Which when executed by a processor, performs the steps of the method described in the embodiments of the present application.
The embodiment of the application also provides a computer readable storage medium. One or more non-transitory computer-readable storage media containing computer-executable instructions that, when executed by one or more processors, cause the processors to perform the steps of the disparity map acquisition method.
A computer program product comprising instructions which, when run on a computer, cause the computer to perform a disparity map acquisition method.
Any reference to memory, storage, database, or other medium used herein may include non-volatile and/or volatile memory. Non-volatile memory can include read-only memory (ROM), Programmable ROM (PROM), Electrically Programmable ROM (EPROM), Electrically Erasable Programmable ROM (EEPROM), or flash memory. Volatile memory can include Random Access Memory (RAM), which acts as external cache memory. By way of illustration and not limitation, RAM is available in a variety of forms, such as Static RAM (SRAM), Dynamic RAM (DRAM), Synchronous DRAM (SDRAM), double data rate SDRAM (DDR SDRAM), Enhanced SDRAM (ESDRAM), synchronous Link (Synchlink) DRAM (SLDRAM), Rambus Direct RAM (RDRAM), direct bus dynamic RAM (DRDRAM), and bus dynamic RAM (RDRAM).
The above-mentioned embodiments only express several embodiments of the present application, and the description thereof is more specific and detailed, but not construed as limiting the scope of the present application. It should be noted that, for a person skilled in the art, several variations and modifications can be made without departing from the concept of the present application, which falls within the scope of protection of the present application. Therefore, the protection scope of the present patent shall be subject to the appended claims.
Claims (16)
1. A disparity map acquisition method is characterized by comprising the following steps:
constructing pyramid images for the main shooting image and the auxiliary shooting image respectively to obtain a main shooting pyramid image and an auxiliary shooting pyramid image;
partitioning each layer of image of the main shooting pyramid image and the auxiliary shooting pyramid image to obtain a main shooting pyramid partitioned image and an auxiliary shooting pyramid partitioned image;
and carrying out stereo matching on the main shooting pyramid block image and the auxiliary shooting pyramid block image to obtain a disparity map.
2. The method according to claim 1, wherein the obtaining of the disparity map by stereo matching the main shooting pyramid block image and the sub shooting pyramid block image comprises:
aiming at the main shooting pyramid block images, updating the parallax range of the corresponding block images of the current layer image according to the initial parallax image of each block image of the previous layer image until the parallax range of each first block image is obtained; the first block image is a block image of the image at the bottom layer in the main shooting pyramid block images;
according to the parallax range of each first block image, performing stereo matching on each first block image and the corresponding second block image to obtain an initial parallax image of each first block image; the second block image is a block image corresponding to the first block image in the bottommost layer image of the secondary shooting pyramid block images;
and merging the initial disparity maps of the first block images to obtain the disparity maps.
3. The method according to claim 2, wherein the obtaining the disparity map of each first block image by performing stereo matching on each first block image and a corresponding second block image according to the disparity range of each first block image includes:
according to the parallax range of each first block image, performing matching cost calculation on each first block image and the corresponding second block image to obtain an image parallax space of each first block image;
performing cost aggregation on the image parallax space of each first block image to obtain an updated image parallax space of each first block image;
and performing parallax calculation on the updated image parallax space of each first block image to obtain an initial parallax image of each first block image.
4. The method according to claim 2 or 3, wherein the merging the initial disparity maps of the first block images to obtain the disparity map comprises:
and merging the initial disparity maps of the first block images into an image aligned with the size of the main shot image to obtain the disparity map.
5. The disparity map acquisition method according to claim 2 or 3, further comprising:
and carrying out stereo matching on each block image of each layer of image in the main shooting pyramid block images and the corresponding block image in the auxiliary shooting pyramid block images to obtain an initial parallax image of each block image of each layer of image in the main shooting pyramid block images.
6. The disparity map acquisition method according to claim 2 or 3, further comprising:
and if the current layer image in the main shooting pyramid block image is the topmost layer image, the parallax range of the current layer image in the main shooting pyramid block image is a preset default parallax range.
7. The method for acquiring the disparity map according to claim 1 or 2, wherein the step of constructing the pyramid images for the main shot image and the sub shot image respectively to obtain the main shot pyramid image and the sub shot pyramid image comprises:
and respectively taking the main shooting image and the auxiliary shooting image as bottom layers, and performing downsampling processing on the main shooting image and the auxiliary shooting image through a preset sampling frequency to obtain the main shooting pyramid image and the auxiliary shooting pyramid image.
8. The method according to claim 1 or 2, wherein before the step of constructing the pyramid images for the main-shot image and the sub-shot image respectively to obtain the main-shot pyramid image and the sub-shot pyramid image, the method further comprises:
respectively filtering the main shot image and the auxiliary shot image by adopting a Gaussian kernel function with a preset size to obtain a filtered main shot image and a filtered auxiliary shot image;
the method for respectively constructing the pyramid images of the main shooting image and the auxiliary shooting image to obtain the pyramid images of the main shooting image and the auxiliary shooting image comprises the following steps:
and respectively constructing a pyramid image for the filtered main shooting image and the filtered auxiliary shooting image to obtain a main shooting pyramid image and an auxiliary shooting pyramid image.
9. The method according to claim 1 or 2, wherein the step of blocking each layer of images of the main shooting pyramid image and the sub shooting pyramid image to obtain a main shooting pyramid block image and a sub shooting pyramid block image comprises:
partitioning each layer of image of the main shooting pyramid image and the auxiliary shooting pyramid image by adopting a preset image partitioning method to obtain a main shooting pyramid partitioned image and an auxiliary shooting pyramid partitioned image; the image blocking method comprises an image segmentation method or a characteristic point clustering method based on deep learning.
10. The disparity map acquisition method according to claim 1 or 2, further comprising:
and processing the disparity map by a preset hole filling method or a preset filtering method to obtain a processed disparity map.
11. The disparity map acquisition method according to claim 10, further comprising:
and obtaining a depth map according to the calibration data and the processed disparity map by adopting a triangulation principle.
12. The disparity map acquisition method according to claim 1 or 2, further comprising:
performing epipolar line rectification on the parallel double-shot collected images under frame synchronization according to calibration data to obtain a main shot image and a secondary shot image; and the distance between the homonymous points of the main shot image and the auxiliary shot image in the preset direction is smaller than a preset threshold value.
13. The disparity map acquisition method according to claim 1 or 2, wherein an overlapping region exists between the block images of the same layer of image of the main shooting pyramid block image; and an overlapping area exists between each block image of the same layer of image of the secondary shooting pyramid block image.
14. A disparity map acquisition apparatus, comprising:
the construction module is used for respectively constructing pyramid images for the main shooting image and the auxiliary shooting image to obtain the main shooting pyramid image and the auxiliary shooting pyramid image;
the blocking module is used for blocking each layer of image of the main shooting pyramid image and the auxiliary shooting pyramid image to obtain a main shooting pyramid blocking image and an auxiliary shooting pyramid blocking image;
and the matching module is used for carrying out three-dimensional matching on the main shooting pyramid block image and the auxiliary shooting pyramid block image to obtain a parallax map.
15. An electronic device comprising a memory and a processor, wherein the memory stores a computer program, and wherein the computer program, when executed by the processor, causes the processor to perform the steps of the disparity map acquisition method according to any one of claims 1 to 13.
16. A computer-readable storage medium, on which a computer program is stored, which, when being executed by a processor, carries out the steps of the method according to any one of claims 1 to 13.
Priority Applications (1)
| Application Number | Priority Date | Filing Date | Title |
|---|---|---|---|
| CN202010603208.8A CN111768434A (en) | 2020-06-29 | 2020-06-29 | Disparity map acquisition method and device, electronic equipment and storage medium |
Applications Claiming Priority (1)
| Application Number | Priority Date | Filing Date | Title |
|---|---|---|---|
| CN202010603208.8A CN111768434A (en) | 2020-06-29 | 2020-06-29 | Disparity map acquisition method and device, electronic equipment and storage medium |
Publications (1)
| Publication Number | Publication Date |
|---|---|
| CN111768434A true CN111768434A (en) | 2020-10-13 |
Family
ID=72722926
Family Applications (1)
| Application Number | Title | Priority Date | Filing Date |
|---|---|---|---|
| CN202010603208.8A Pending CN111768434A (en) | 2020-06-29 | 2020-06-29 | Disparity map acquisition method and device, electronic equipment and storage medium |
Country Status (1)
| Country | Link |
|---|---|
| CN (1) | CN111768434A (en) |
Citations (6)
| Publication number | Priority date | Publication date | Assignee | Title |
|---|---|---|---|---|
| CN105528785A (en) * | 2015-12-03 | 2016-04-27 | 河北工业大学 | Binocular visual image stereo matching method |
| CN107846556A (en) * | 2017-11-30 | 2018-03-27 | 广东欧珀移动通信有限公司 | imaging method, device, mobile terminal and storage medium |
| CN109559353A (en) * | 2018-11-30 | 2019-04-02 | Oppo广东移动通信有限公司 | Camera module scaling method, device, electronic equipment and computer readable storage medium |
| CN110473247A (en) * | 2019-07-30 | 2019-11-19 | 中国科学院空间应用工程与技术中心 | Solid matching method, device and storage medium |
| WO2020020160A1 (en) * | 2018-07-25 | 2020-01-30 | 北京市商汤科技开发有限公司 | Image parallax estimation |
| KR102090666B1 (en) * | 2019-02-11 | 2020-03-18 | 주식회사 넥스트칩 | Method and apparatus for generating a disparity map for stereo images |
-
2020
- 2020-06-29 CN CN202010603208.8A patent/CN111768434A/en active Pending
Patent Citations (6)
| Publication number | Priority date | Publication date | Assignee | Title |
|---|---|---|---|---|
| CN105528785A (en) * | 2015-12-03 | 2016-04-27 | 河北工业大学 | Binocular visual image stereo matching method |
| CN107846556A (en) * | 2017-11-30 | 2018-03-27 | 广东欧珀移动通信有限公司 | imaging method, device, mobile terminal and storage medium |
| WO2020020160A1 (en) * | 2018-07-25 | 2020-01-30 | 北京市商汤科技开发有限公司 | Image parallax estimation |
| CN109559353A (en) * | 2018-11-30 | 2019-04-02 | Oppo广东移动通信有限公司 | Camera module scaling method, device, electronic equipment and computer readable storage medium |
| KR102090666B1 (en) * | 2019-02-11 | 2020-03-18 | 주식회사 넥스트칩 | Method and apparatus for generating a disparity map for stereo images |
| CN110473247A (en) * | 2019-07-30 | 2019-11-19 | 中国科学院空间应用工程与技术中心 | Solid matching method, device and storage medium |
Similar Documents
| Publication | Publication Date | Title |
|---|---|---|
| WO2021233029A1 (en) | Simultaneous localization and mapping method, device, system and storage medium | |
| CN109683699B (en) | Method and device for realizing augmented reality based on deep learning and mobile terminal | |
| US9483703B2 (en) | Online coupled camera pose estimation and dense reconstruction from video | |
| CN108076338B (en) | Image visual processing method, device and equipment | |
| US8199977B2 (en) | System and method for extraction of features from a 3-D point cloud | |
| CN101086788B (en) | Method and device for generating a disparity map from stereo images and stereo matching method and device therefor | |
| US20150097827A1 (en) | Target Region Fill Utilizing Transformations | |
| CN106952247B (en) | Double-camera terminal and image processing method and system thereof | |
| CN113140036A (en) | Three-dimensional modeling method, device, equipment and storage medium | |
| CN110211169B (en) | Reconstruction method of narrow baseline parallax based on multi-scale super-pixel and phase correlation | |
| WO2023159757A1 (en) | Disparity map generation method and apparatus, electronic device, and storage medium | |
| US20230025058A1 (en) | Image rectification method and device, and electronic system | |
| TWI738196B (en) | Method and electronic device for image depth estimation and storage medium thereof | |
| CN112184768B (en) | SFM reconstruction method and device based on laser radar and computer equipment | |
| CN111295667A (en) | Image stereo matching method and driving assisting device | |
| WO2020173194A1 (en) | Image feature point tracking method and apparatus, image feature point matching method and apparatus, and coordinate obtaining method and apparatus | |
| CN113034666B (en) | Stereo matching method based on pyramid parallax optimization cost calculation | |
| CN111768434A (en) | Disparity map acquisition method and device, electronic equipment and storage medium | |
| CN112991207B (en) | Panoramic depth estimation method and device, terminal equipment and storage medium | |
| CN113570649B (en) | Gravity direction determination method and device based on three-dimensional model and computer equipment | |
| CN110866535B (en) | Disparity map acquisition method and device, computer equipment and storage medium | |
| CN113808033A (en) | Image document correction method, system, terminal and medium | |
| CN112615993A (en) | Depth information acquisition method, binocular camera module, storage medium and electronic equipment | |
| CN112150532A (en) | Image processing method and device, electronic equipment and computer readable medium | |
| CN114494612A (en) | Method, device and equipment for constructing point cloud map |
Legal Events
| Date | Code | Title | Description |
|---|---|---|---|
| PB01 | Publication | ||
| PB01 | Publication | ||
| SE01 | Entry into force of request for substantive examination | ||
| SE01 | Entry into force of request for substantive examination |