WO2022205934A1

WO2022205934A1 - Disparity map optimization method and apparatus, and electronic device and computer-readable storage medium

Info

Publication number: WO2022205934A1
Application number: PCT/CN2021/130994
Authority: WO
Inventors: 郑子华
Original assignee: 北京迈格威科技有限公司
Priority date: 2021-03-31
Filing date: 2021-11-16
Publication date: 2022-10-06
Also published as: CN113395504B; CN113395504A

Abstract

The present application relates to the field of image processing and relates to a disparity map optimization method and apparatus, and an electronic device and a computer-readable storage medium. The method comprises: obtaining a first image, a second image, and an initial disparity map corresponding to the first image and the second image; according to the initial disparity map, the first image, and the second image, determining an energy equation used for guiding filtering, the energy equation comprising a data item and a smooth item; and performing guiding filtering on the initial disparity map according to the energy equation, and obtaining an optimized disparity map. By means of the method, errors can be prevented from diffusing to a normal area, such that the accuracy of a disparity value and the accuracy of a value domain in the optimized disparity map are ensured.

Description

Disparity map optimization method, apparatus, electronic device, and computer-readable storage medium

CROSS-REFERENCE TO RELATED APPLICATIONS

This application claims the priority of the Chinese patent application with the application number 202110353871.1 and the title "Disparity Map Optimization Method, Apparatus, Electronic Device and Computer-readable Storage Medium" filed with the State Intellectual Property Office of China on March 31, 2021, which The entire contents of this application are incorporated by reference.

technical field

The present application belongs to the field of image processing, and in particular relates to a disparity map optimization method, apparatus, electronic device, and computer-readable storage medium.

Background technique

In the stereo matching algorithm, the disparity map between the first image and the second image can be calculated, so that a series of back-end applications can be performed based on the obtained disparity, such as dual-camera refocusing and dense reconstruction of multiple images. (MVS, Multi View System) etc.

Due to the influence of many factors such as the efficiency of stereo matching and the performance of the device performing stereo matching, in some technical solutions, there are many error areas (such as unclear boundaries, rough boundaries, etc.) in the disparity map obtained directly through the stereo matching algorithm. , there is noise on the image, etc.), so that the disparity map cannot be directly applied to subsequent application scenarios. Therefore, there is a need to optimize the disparity map.

In the existing disparity map optimization methods, conventional guided filtering techniques are generally used to optimize the disparity map, such as WLS (Weighted Least-Squares), FGS (Fast Global Smooth), and the like. However, for conventional guided filtering techniques, basically only the smoothness of the input image before and after filtering is considered. If the disparity map is used as the input image, and the disparity map itself has certain errors, after guided filtering, in addition to smoothing the pixel values of the errors, the pixel values of other normal areas of the disparity map will be destroyed. , resulting in error diffusion.

SUMMARY OF THE INVENTION

In view of this, the present application provides a disparity map optimization method, apparatus, electronic device, and computer-readable storage medium. When optimizing the disparity map, the smoothing range is constrained, so as to ensure the disparity value of the optimized disparity map and The accuracy of the range.

The embodiments of the present application are implemented as follows:

Some embodiments of the present application provide a disparity map optimization method, the method may include: acquiring a first image, a second image and an initial disparity map corresponding to the first image and the second image, the The first image and the second image are two images obtained from different shooting angles for the same shooting scene; the energy equation for guiding filtering is determined according to the initial disparity map, the first image and the second image , the energy equation includes a data item and a smoothing term; the initial disparity map is guided and filtered according to the energy equation to obtain an optimized disparity map.

In the above process, when conducting guided filtering on the initial disparity map, the data item and smoothing term included in the energy equation used for conducting the guided filtering may be based on the initial disparity map, the first image, and the second image used in stereo matching. so that the smooth term is affected by the gradient of the first image in the abscissa direction and the ordinate direction, and also by the gradient of the second image in the abscissa direction and the ordinate direction. Therefore, the first image and the The degree of matching between the second images constrains the smoothing term, so that the smoothing term can be controlled to smooth the areas that should be smoothed in the initial disparity map, and ignore the areas that should not be smoothed to prevent errors from spreading to normal areas to ensure optimization The accuracy of the disparity values and the accuracy of the range in the disparity map.

With reference to some embodiments of the present application, in a possible implementation manner, the determining an energy equation for conducting guided filtering according to the initial disparity map, the first image, and the second image may include: according to The data item E _data is determined from the first image and the second image; the smoothing item E _data is determined according to the initial disparity map; based on the formula E=min(E _data +αE _smooth ), the The energy equation E; wherein, α is the weight of the smooth item and the data item, and the data item constitutes a constraint effect on the smooth item.

With reference to some embodiments of the present application, in a possible implementation manner, the data item E data is determined according to the first image and the second image, and the data item E _data is determined according to the initial disparity map. The smoothing item E _data may include: determining the data item E _data based on the formula E _data =∫(I ₂ (x+d)-I ₁ (x)) ² dx; based on the formula E _smooth =∫w _x ( d _x ) ² +w _y ( _dy ) ² dx, determine the smoothing term E _smooth ; wherein, I ₁ (x) is the first image, I ₂ (x+d) is the second image, d is the initial disparity map, _x is the abscissa of any point in the first image, dx and _dy are the gradients of the initial disparity map in the abscissa direction and the ordinate direction respectively , w _x and w _y are smoothing weights.

With reference to some embodiments of the present application, in a possible implementation manner, performing guided filtering on the initial disparity map according to the energy equation to obtain an optimized disparity map may include: according to the energy equation, calculating the gradient d _x of the initial disparity map in the abscissa direction; the optimized disparity map is obtained by superimposing the initial disparity map d and the gradient d _x of the initial disparity map in the abscissa direction.

With reference to some embodiments of the present application, in a possible implementation manner, the method may further include: judging whether the gradient d _x of the current initial disparity map in the abscissa direction satisfies a preset rule; , and the optimized disparity map corresponding to the gradient d _x of the current initial disparity map in the abscissa direction is determined as the final disparity map.

With reference to some embodiments of the present application, in a possible implementation manner, the method may further include: when the judgment is no, performing the following steps until the gradient d _x of the current initial disparity map in the abscissa direction satisfies After the preset rule, the optimized disparity map corresponding to the gradient d _x of the current initial disparity map in the abscissa direction is determined as the final disparity map:

Update the initial disparity map according to the optimized disparity map to obtain the current disparity map; calculate the d _x of the current disparity map in the abscissa direction according to the energy equation; d is superimposed with the gradient d _x of the current initial disparity map in the abscissa direction to obtain an optimized disparity map corresponding to the gradient d _x of the current initial disparity map in the abscissa direction; return to judge the current initial disparity map The step of whether the gradient d _x in the abscissa direction satisfies the preset rule.

In this way, the accuracy of the finally obtained disparity map can be further guaranteed.

With reference to some embodiments of the present application, in a possible implementation, the preset rule may be: the gradient d _x of the current initial disparity map in the abscissa direction is smaller than the first threshold, and/or the current iteration The number of times satisfies the second threshold.

With reference to some embodiments of the present application, in a possible implementation manner, the guide map of the guide filter may be the first image.

Other embodiments of the present application provide a disparity map optimization apparatus, and the apparatus may include: an acquisition module, a determination module, and a guided filtering module. The acquisition module may be configured to acquire a first image, a second image, and an initial disparity map corresponding to the first image and the second image, and the first image and the second image may be for the same shooting scene, two images obtained through different shooting angles; the determining module may be configured to determine an equation for conducting guided filtering according to the initial disparity map, the first image and the second image, the energy equation A data item and a smoothing item may be included; the guided filtering module may be configured to perform guided filtering on the initial disparity map according to the energy equation to obtain an optimized disparity map.

With reference to other embodiments of the present application, in a possible implementation manner, the determining module may be configured to: determine the data item E _data according to the first image and the second image; The smooth term E _data is determined from the initial disparity map; the energy equation E is determined based on the formula E=min(E _data +αE _smooth );

Wherein, α is the weight of the smooth item and the data item, and the data item may constitute a constraint effect on the smooth item.

With reference to other embodiments of the present application, in a possible implementation manner, the determining module may be configured to: based on the formula E _data =∫(I ₂ (x+d)-I ₁ (x)) ² dx, the data item E _data is determined; based on the formula E _smooth =∫w _x (d _x ) ² +w _y (d _y ) ² dx, the smoothing item E _smooth is determined;

Wherein, I ₁ (x) is the first image, I ₂ (x+d) is the second image, d is the initial disparity map, and x is the abscissa of any point in the first image , d _x and _dy are the gradient of the initial disparity map in the abscissa direction and the gradient in the ordinate direction, respectively, and w _x and w _y are smoothing weights.

With reference to other embodiments of the present application, in a possible implementation, the guided filtering module may be configured to: according to the energy equation, calculate the gradient d _x of the initial disparity map in the abscissa direction ; By superimposing the initial disparity map d and the gradient d _x of the initial disparity map in the abscissa direction, the optimized disparity map is obtained.

With reference to other embodiments of the present application, in a possible implementation manner, the apparatus may further include a judgment module and an execution module, and the judgment module may be configured to judge that the current initial disparity map is in the abscissa direction Whether the gradient d _x of the current initial disparity map satisfies the preset rule; the execution module may be configured to determine the optimized disparity corresponding to the gradient d _x of the current initial disparity map in the abscissa direction when the judgment module judges that it is yes The picture shows the final disparity map.

With reference to other embodiments of the present application, in a possible implementation manner, the apparatus may further include a judgment module and an execution module, and the judgment module may be used to judge the gradient of the current initial disparity map in the abscissa direction Whether d _x satisfies the preset rule; the execution module may be configured to perform the following process when the judgment module judges no, until the gradient d _x of the current initial disparity map in the abscissa direction satisfies the preset rule , and then determine the optimized disparity map corresponding to the gradient d _x of the current initial disparity map in the abscissa direction as the final disparity map:

Update the initial disparity map according to the optimized disparity map to obtain the current disparity map; calculate the gradient d _x of the current disparity map in the abscissa direction according to the energy equation; Figure d is superimposed with the gradient d _x of the current initial disparity map in the abscissa direction to obtain an optimized disparity map corresponding to the gradient d _x of the current initial disparity map in the abscissa direction; return to judge the current initial disparity The step of whether the gradient d _x of the graph in the abscissa direction satisfies the preset rules.

With reference to other embodiments of the present application, in a possible implementation manner, the preset rule may be: the gradient d _x of the current initial disparity map in the abscissa direction is smaller than the first threshold, and/or the current initial disparity map The number of iterations satisfies the second threshold.

With reference to other embodiments of the present application, in a possible implementation manner, the guide map of the guide filter may be the first image.

Still other embodiments of the present application provide an electronic device, the electronic device may include: a memory and a processor, the memory is connected to the processor; the memory is used for storing a program; the processor invokes the storage The program stored in the memory is used to execute some of the above-mentioned embodiments of the present application and/or the methods provided in combination with any possible implementation manner of some of the above-mentioned embodiments.

Still other embodiments of the present application provide a non-volatile computer-readable storage medium (hereinafter referred to as a computer-readable storage medium), where a computer program can be stored on the computer-readable storage medium, and the computer program is executed by a computer The above-mentioned embodiments of the present application and/or the methods provided in combination with any possible implementation manner of the above-mentioned embodiments can be executed.

Other features and advantages of the present application will be set forth in the description which follows, and, in part, will be apparent from the description, or may be learned by practice of the embodiments of the present application. The objectives and other advantages of the application may be realized and attained by the structure particularly pointed out in the written description and drawings.

Description of drawings

In order to more clearly illustrate the technical solutions in the embodiments of the present application or related technologies, the accompanying drawings required in the embodiments will be briefly introduced below. Obviously, the drawings in the following description are only some implementations of the present application. For example, for those of ordinary skill in the art, other drawings can also be obtained based on these drawings without any creative effort. The above and other objects, features and advantages of the present application will be more apparent from the accompanying drawings. The same reference numerals refer to the same parts throughout the drawings. The drawings are not intentionally scaled to actual size, and the emphasis is on illustrating the subject matter of the present application.

FIG. 1 shows a schematic diagram of the effect of the existing guided filtering.

FIG. 2 shows a flowchart of a disparity map optimization method provided by an embodiment of the present application.

FIG. 3 shows a structural block diagram of a disparity map optimization apparatus provided by an embodiment of the present application.

FIG. 4 shows a schematic structural diagram of an electronic device provided by an embodiment of the present application.

Icons: 100-electronic device; 110-processor; 120-memory; 400-disparity map optimization device, 410-acquisition module; 420-determination module; 430-guide filtering module.

Detailed ways

The technical solutions in the embodiments of the present application will be described below with reference to the accompanying drawings in the embodiments of the present application.

It should be noted that like numerals and letters refer to like items in the following figures, so once an item is defined in one figure, it does not require further definition and explanation in subsequent figures. Meanwhile, in the description of this application, relational terms such as "first", "second", etc. are only used to distinguish one entity or operation from another entity or operation, and do not necessarily require or imply these entities or that there is any such actual relationship or sequence between operations. Moreover, the term "comprising" or any other variation thereof is intended to cover non-exclusive inclusion, whereby a process, method, article or device comprising a series of elements includes not only those elements, but also other elements not expressly listed, Or also include elements inherent to such a process, method, article or apparatus. Without further limitation, an element qualified by the phrase "comprising a..." does not preclude the presence of additional identical elements in a process, method, article or apparatus that includes the element.

Furthermore, the term "and/or" in this application is only an association relationship to describe related objects, indicating that there can be three kinds of relationships, for example, A and/or B, it can mean that A exists alone, and A and B exist at the same time. B, there are three cases of B alone.

In addition, the defect existing in optimizing the disparity map for the existing guided filtering technology (during the optimization process, the pixel values of other normal regions of the disparity map are destroyed, thereby causing error diffusion) is the applicant's experience and practice and The results obtained after careful study, therefore, the discovery process of the above-mentioned defects and the solutions proposed for the above-mentioned defects in the following embodiments of the present application should be regarded as contributions made by the applicant to the present application.

In order to solve the above problems, the embodiments of the present application provide a disparity map optimization method, apparatus, electronic device, and computer-readable storage medium. When optimizing the disparity map, the smoothing range is constrained, thereby ensuring the optimized disparity map. Disparity values and the accuracy of the range.

The technology can be implemented by corresponding software, hardware and combination of software and hardware. The embodiments of the present application are described in detail below.

The following will introduce the disparity map optimization method provided by this application.

In the embodiment of the present application, two camera modules arranged at different positions, such as dual-camera modules of a mobile phone or other point-and-shoot equipment, are used for the same shooting scene ( The two images obtained by photographing the object) may be referred to as the first image and the second image, respectively.

It can be understood that since the installation positions of the two camera modules are different, the shooting angles corresponding to the first image and the second image are also different.

Wherein, when the above-mentioned two different camera modules are arranged in a left-right relationship, the first image and the second image may be images obtained by the two camera modules arranged on the left and right for the same object, wherein the camera module on the left The image captured by the group is the left image, and the image captured by the camera module on the right is the right image; when the above two different camera modules are arranged in an up-down relationship, the first image and the second image can be arranged up and down respectively. The images captured by the two camera modules of the above are for the same object, wherein, the image captured by the upper camera module is the upper image, and the image captured by the lower camera module is the lower image.

In the embodiment of the present application, there is a certain mapping relationship between the first image and the second image, and a disparity map between the first image and the second image can be obtained according to the coordinates between the pixel points having the mapping relationship.

It is worth noting that, on the premise that the first image and the disparity map are known, the second image and the occluded area in the second image can be mapped out through the first image and the disparity map.

Of course, in some embodiments, the above-mentioned disparity map can also be used in other application scenarios, for example, the above-mentioned disparity map can be used for stereo matching.

When performing stereo matching, if too much attention is paid to matching efficiency, it may lead to an error area in the disparity map obtained through stereo matching. Therefore, it is necessary to optimize the disparity map through guided filtering technology.

The essence of guided filtering is to smooth the input image. On this basis, if there is an error in the disparity map used as the input image (the error may be caused by the existence of a certain occlusion area in the first image and/or the second image, or it may be due to the process of stereo matching, There is an error in the matching process between the first image and the second image), then the region with the error will also be weakened.

One of the weakening processes is: after setting the pixel value of the error region to 0, then taking the average pixel value of the pixel value of the error region and the local normal region adjacent to the error region, and then taking the error The pixel values of the area and the local normal area are modified to the above average pixel values.

After doing so, although the error area is weakened, the pixel values of the normal area adjacent to the error area are also forced to change, which will cause the error to spread to the normal area. That is, the existing parallax map processing technology cannot correctly process the pixel values of the regions where the error exists in the parallax map.

As shown in FIG. 1 , the left area in FIG. 1 is the disparity map before guided filtering, and the right area in FIG. 1 is the disparity map after guided filtering.

Area A in Figure 1 is a local area where there is an error in the disparity map before guided filtering. In this local area, the borders of leaves are blurred and need to be guided and filtered to make them sharper and clearer. Area B in Figure 1 is the local area corresponding to area A in the disparity map after guided filtering. In this local area, although the boundary of leaves becomes sharp and clear, the boundary of leaves can be clearly seen from area B. The pixel values of the black areas outside are corrupted by the pixel values of the gray borders.

In order to avoid the above problem, please refer to FIG. 2 . An embodiment of the present application provides a disparity map optimization method. When optimizing the disparity map, the method constrains the smoothing range, thereby ensuring the disparity value of the optimized disparity map. and the accuracy of the range. The steps involved will be described below with reference to FIG. 2 .

Step S110: Acquire a first image, a second image, and an initial disparity map corresponding to the first image and the second image.

Wherein, the first image and the second image may be two images captured at the same time by camera modules arranged in different positions for the same shooting scene.

For the embodiment of the present application, after the first image and the second image are acquired, calculation may be performed according to the first image and the second image, so as to obtain the initial disparity map.

When the disparity map is applied to a stereo matching scene, an initial disparity map may be calculated based on the first image and the second image through a stereo matching algorithm. As for which stereo matching algorithm is used to calculate the initial disparity map, it is not within the scope of consideration of the embodiments of the present application, and correspondingly, the embodiments of the present application do not limit it.

Of course, in some embodiments, a confidence map corresponding to the initial disparity map can also be obtained, and the confidence map is used to represent where there is an error in the corresponding disparity map, so as to facilitate subsequent evaluation of the initial disparity map after optimization. The optimized effect of the optimized disparity map.

Step S120: Determine an energy equation for guiding filtering according to the initial disparity map, the first image and the second image, where the energy equation may include a data item and a smoothing item.

The guided filtering technology adopted in the embodiments of the present application may implement guided filtering according to an energy equation; the energy equation includes a data item and a smoothing item.

Different from the existing guided filtering solutions, the data items and smoothing terms of the guided filtering solutions provided by the embodiments of the present application are no longer simply associated with the input disparity map, but also exist with the first image and the second image. and the data item constitutes a constraint on the smoothing item, so that when the smoothing item acts on the initial disparity map, it is no longer smoothing any area of the initial disparity map, but smoothing the area that should be smoothed in the initial disparity map.

Optionally, in some embodiments, the data item E _data may be determined according to the first image and the second image, the smoothing item E _data may be determined according to the initial disparity map, and then according to the data item E _data and the smoothing item E _smooth , Determine the energy equation E.

For example, the data item E _data may be determined based on the formula E _data =∫(I ₂ (x+d)-I ₁ (x)) ² dx. The data item may be composed in a matching manner, and the erroneous parallax can be corrected under the premise of maintaining the accuracy of the parallax. For example, the smoothing term E _smooth may be determined based on the formula E _smooth =∫w _x (d _x ) ² +w _y (d _y ) ² dx, and the smoothing term plays a role of guiding filtering.

For another example, the energy equation E may be determined based on the formula E=min(E _data +αE _smooth ).

Wherein, I ₁ (x) is the expression of the first image, and I ₂ (x+d) is the expression of the second image.

d is the initial disparity map.

α is the weight of the smooth item and the data item, which is preset by the staff, and the size of α can be adjusted according to the actual situation. It is worth pointing out that the size of α can be positively correlated with the degree of smoothness.

d _x is the gradient of the initial disparity map in the abscissa direction, _dy is the gradient of the initial disparity map in the ordinate direction, and the initial values of d _x and _dy are preset by the staff, and d _x and _dy can be Adjust according to the actual situation. In general, the smaller the gradient, the smoother it is.

The direction of the coordinate axis of the abscissa is the direction of the center line of the two camera modules. For example, when the two camera modules are arranged up and down, the direction of the coordinate axis of the abscissa can be the vertical direction; When the groups are arranged left and right, the coordinate axis direction of the abscissa can be the horizontal direction.

The coordinate axis direction of the ordinate may be perpendicular to the coordinate axis direction of the abscissa.

w _x is the smoothing weight in the abscissa direction, and w _y is the smoothing weight in the ordinate direction.

When conducting guided filtering, a guide map is required to guide smoothing. The size of the smoothing weights w _x , w _y is controlled by the guide map. In some embodiments, the first image may be directly determined as a guide image. In this embodiment, the sizes of w _x and w _y are determined according to the first image. Of course, in other embodiments, other images, such as the second image, may also be used as the guide image, which is not limited here.

x is the abscissa of any point in the first image; min represents the minimization of the energy equation.

Step S130: Perform guided filtering on the initial disparity map according to the energy equation to obtain an optimized disparity map.

In the embodiment of the present application, when the two camera modules are arranged up and down, the optimization is mainly performed for the column direction of the image, and when the two camera modules are arranged left and right, the optimization is mainly performed for the row direction of the image. Optimization, therefore, the process of conducting guided filtering on the initial disparity map is the process of solving _dx in the energy equation.

In some embodiments, step S130 may include:

According to the energy equation, calculate the gradient d _x of the initial disparity map d in the abscissa direction;

The optimized disparity map d' is obtained by superimposing the initial disparity map d and the gradient _dx of the initial disparity map in the abscissa direction, that is, d'=d+ _dx .

The following will introduce the process of solving _dx .

(1) Suppose the expressions of the smoothing weights w _x and w _y are as follows:

w _x =exp(-delta*I _1x )+EPSLON;

w _y =exp(-delta*I _1y )+EPSLON.

Wherein, I _1x is the gradient of the first image in the abscissa direction, and I _1y is the gradient of the first image in the ordinate direction.

delta is the first smoothing constraint parameter (exponential type), and EPSLON is the second smoothing constraint parameter (constant type). The first smoothing constraint parameter and the second smoothing constraint parameter are used to constrain the above-mentioned smoothing term E _smooth , and control the influence of the gradient of the guide image on the smoothing term E _smooth . So far, w _x and w _y can be considered as known quantities.

(2) Rewrite the energy equation:

(d)=min∫((I ₂ (x+d,y)−I ₁ (x,y)) ² +α(w _x (d _x ) ² +w _y (d _y ) ² ))dx.

(3) Taylor expansion of the rewritten energy equation:

I ₂ (x+d,y)−I ₁ (x,y)=I ₂ (x,y)−I ₁ (x,y)+I _2xdx +I _2y dy.

Wherein, I _2x is the gradient derivative of the second image in the abscissa direction, and I _2y is the gradient derivative of the second image in the ordinate direction.

Further, the above formula can be adjusted to the following form:

min∫((I ₂ (x,y)-I ₁ (x,y)+I _2x dx+I _2y dy) ² +α(w _x (d _x ) ² +w _y (d _y ) ² ))dx , to facilitate the derivation of d _x and _dy .

(4) For min∫((I ₂ (x,y)-I ₁ (x,y)+I _2x dx+I _2y dy) ² +α(w _x (d _x ) ² +w _y (d _y ) ² )) Calculate the first derivative of d _x and d _y in dx, and set the derivation result to 0 to obtain the following equation system:

The above equation system is a typical linear equation about d _x and d _y , which can be rewritten as the following equation system:

(5) Adjust the rewritten equation system to the matrix form of Ax=b, and obtain the following linear matrix equation system:

(6) Solve the above-mentioned linear matrix equation system by using the existing method, such as Gauss-Seidel method, Jacobi method, and successive over-relaxation iterative method (SOR method), so as to Obtain the gradient d _x of the initial disparity map in the abscissa direction.

After d _x is obtained, the disparity map d ′ optimized for the initial disparity map d can be obtained based on the formula d′=d+d _x .

In the traditional guided filtering scheme, the gradient d _x in the abscissa direction for the disparity map is only affected by the gradient of the first image. In the above-mentioned solution process for d _x of the present application, d _x is affected by the gradient of the first image in the abscissa and ordinate directions, and is also affected by the gradient of the second image in the abscissa and ordinate directions, Therefore, the degree of matching between the first image and the second image imposes constraints on _dx , so that the degree of matching between the first image and the second image also imposes constraints on the smooth term E _smooth including _dx , so that it is possible to control The smoothing term (the smoothing term is used to weaken the pixels in the image) smoothes the areas that should be smoothed in the initial disparity map, and ignores the areas that should not be smoothed; at the same time, the data item E _data contains the first image and the first image. The matching relationship between the two images (I ₂ (x+d)-I ₁ (x) in the data item E _data ), therefore, the data item can also correct the error area in the initial disparity map to ensure The accuracy of the disparity value and the accuracy of the range in the optimized disparity map d'.

In some optional embodiments, in order to ensure the accuracy of the disparity map as much as possible, the solution process of d _x may be limited, so as to further ensure the accuracy of the disparity map finally obtained.

Specifically, in some embodiments, the accuracy of the final disparity map can be ensured by the following methods:

A1: The solution process can be restricted by judging whether the current solution to d _x satisfies the preset rules;

A2: When the current solution to d _x satisfies the preset rule, the optimized disparity map d′ corresponding to the currently obtained d _x can be directly determined as the final disparity map;

A3: Otherwise, perform the following steps 1 to 4 until the gradient d _{x of the current initial disparity map in the abscissa direction satisfies the preset rule, and then compare the gradient d x} _of the current initial disparity map in the abscissa direction with the current initial disparity map. The corresponding optimized disparity map is determined as the final disparity map:

Step 1: Update the initial disparity map according to the optimized disparity map to obtain the current disparity map;

Step 2: Calculate the gradient d _x of the current disparity map in the abscissa direction according to the energy equation;

Step 3: By superimposing the initial disparity map d and the gradient _dx of the current initial disparity map in the abscissa direction, an optimized disparity map corresponding to the gradient _dx of the current initial disparity map in the abscissa direction is obtained. ;

Step 4: Return to the step of judging whether the gradient d _x of the current initial disparity map in the abscissa direction satisfies the preset rule.

Wherein, in some embodiments, the above-mentioned preset rule may be: the gradient d _x of the current initial disparity map in the abscissa direction is smaller than the first threshold, that is, the gradient d _x of the initial disparity map in the abscissa direction is sufficient After it is small, it can be considered that the preset rules are satisfied.

When the gradient d _x of the initial disparity map in the abscissa direction is sufficiently small, it indicates that there is a difference between the corresponding optimized disparity map obtained by solving d _x in the previous time and the corresponding optimized disparity map obtained by solving d _x this time. The difference is small enough that the disparity map is getting closer and closer to the true value of the guide map. Therefore, the above smoothing process is smoothing according to the true value of the guide map, not according to the ideal state, so that the smoothing term can be controlled. The areas that should be smoothed in the initial disparity map are smoothed, and the areas that should not be smoothed are ignored to ensure the accuracy of the disparity values and the accuracy of the value range in the final disparity map.

In some embodiments, the above-mentioned preset rule may also be: the number of iterations currently used for the calculation of d _x reaches the second threshold.

As shown in FIG. 3 , an embodiment of the present application further provides a disparity map optimization apparatus 400. The disparity map optimization apparatus 400 may include: an acquisition module 410, a determination module 420, and a guided filtering module 430.

The acquisition module 410 may be configured to acquire a first image, a second image, and an initial disparity map corresponding to the first image and the second image, and the first image and the second image may be for the same Shooting scene, two images obtained from different shooting angles;

The determining module 420 may be configured to determine an energy equation for guided filtering according to the initial disparity map, the first image, and the second image, the energy equation may include a data term and a smoothing term;

The guided filtering module 430 may be configured to perform guided filtering on the initial disparity map according to the energy equation to obtain an optimized disparity map.

In a possible implementation manner, the determining module 420 may be configured to: determine the data item E data according to the first image and the second image; determine the data item E _data according to the initial disparity map the smoothing term E _data ; based on the formula E=min(E _data +αE _smooth ), determine the energy equation E;

In a possible implementation manner, the determining module 420 may be configured to: determine the data item E based on the formula E _data =∫(I ₂ (x+d)-I ₁ (x)) ² dx _data ; based on the formula E _smooth =∫w _x (d _x ) ² +w _y (d _y ) ² dx, determine the smoothing term E _smooth ;

In a possible implementation manner, the guided filtering module 430 may be configured to: calculate the gradient d _x of the initial disparity map in the abscissa direction according to the energy equation; d is superimposed with the gradient d _x of the initial disparity map in the abscissa direction to obtain the optimized disparity map.

In a possible implementation manner, the apparatus may further include a judgment module and an execution module, and the judgment module may be configured to judge whether the gradient d _x of the current initial disparity map in the abscissa direction satisfies a preset rule ; the execution module may be configured to determine the optimized disparity map corresponding to the gradient d _x of the current initial disparity map in the abscissa direction as the final disparity map when the judgment module judges yes.

In a possible implementation manner, the apparatus may further include a judgment module and an execution module, and the judgment module may be configured to judge whether the gradient d _x of the current initial disparity map in the abscissa direction satisfies a preset rule The execution module can be configured to perform the following process when the judgment module judges that it is no, until the gradient d _x of the current initial disparity map in the abscissa direction satisfies the preset rule, and then compares it with the current The optimized disparity map corresponding to the gradient d _x of the initial disparity map in the abscissa direction is determined as the final disparity map: update the initial disparity map according to the optimized disparity map to obtain the current disparity map; According to the energy equation, calculate the gradient d _x of the current disparity map in the abscissa direction; by superimposing the initial disparity map d and the gradient d _x of the current initial disparity map in the abscissa direction, to obtain The optimized disparity map corresponding to the gradient d _x of the current initial disparity map in the abscissa direction; return to the step of judging whether the gradient d _x of the current initial disparity map in the abscissa direction satisfies the preset rule.

In a possible implementation manner, the preset rule may be: the gradient d _x of the current initial disparity map in the abscissa direction is smaller than the first threshold, and/or the current number of iterations satisfies the second threshold.

In a possible implementation manner, the guided image of the guided filtering may be the first image.

The realization principle and the technical effect of the parallax map optimization device 400 provided by the embodiment of the present application are the same as those of the foregoing method embodiments. For brief description, for the parts not mentioned in the device embodiments, reference may be made to the corresponding content in the foregoing method embodiments. .

In addition, an embodiment of the present application further provides a computer-readable storage medium, where a computer program may be stored thereon, and when the computer program is run by a computer, the steps included in the above disparity map optimization method are executed.

In addition, referring to FIG. 4 , an embodiment of the present application further provides an electronic device 100 for implementing the disparity map optimization method and apparatus of the embodiment of the present application.

The electronic device 100 may be provided with computing capability, and may perform guided filtering on the acquired image.

Optionally, the electronic device 100 may be, but is not limited to, a personal computer (Personal computer, PC), a smart phone, a tablet computer, a Mobile Internet Device (Mobile Internet Device, MID), a personal digital assistant, a server, and other devices. The server may be, but not limited to, a network server, a database server, a cloud server, and the like.

The electronic device 100 may include: a processor 110 and a memory 120 .

It should be noted that the components and structures of the electronic device 100 shown in FIG. 4 are only exemplary and not restrictive, and the electronic device 100 may also have other components and structures as required. For example, in some cases, the electronic device 100 may further include a display screen for presenting the guided filtered results to the user.

The processor 110 , the memory 120 and other components that may appear in the electronic device 100 are directly or indirectly electrically connected to each other to realize data transmission or interaction. For example, the processor 110, the memory 120, and other possible components may be electrically connected to each other through one or more communication buses or signal lines.

The memory 120 may be configured to store a program, for example, a program corresponding to a disparity map optimization method or a disparity map optimization apparatus that will appear later. Optionally, when the disparity map optimization apparatus is stored in the memory 120, the disparity map optimization apparatus may include at least one software function module that may be stored in the memory 120 in the form of software or firmware.

Optionally, the software function modules included in the disparity map optimization apparatus may also be solidified in an operating system (operating system, OS) of the electronic device 100 .

The processor 110 may be configured to execute executable modules stored in the memory 120, such as software function modules or computer programs included in the disparity map optimization apparatus. After receiving the execution instruction, the processor 110 may execute a computer program, for example, execute: acquiring a first image, a second image, and an initial disparity map corresponding to the first image and the second image, the first image and the second image. The image and the second image are two images captured by camera modules located at different positions at the same time for the same shooting scene; determined according to the initial disparity map, the first image and the second image An energy equation used for guiding filtering, the energy equation includes a data item and a smoothing term; guiding filtering is performed on the initial disparity map according to the energy equation to obtain an optimized disparity map.

Certainly, the method disclosed in any embodiment of the present application may be applied to the processor 110 or implemented by the processor 110 .

To sum up, the disparity map optimization method, device, electronic device, and computer-readable storage medium proposed in the embodiments of the present application, when the initial disparity map is guided filtering, the data items included in the energy equation used to perform the guided filtering And the smooth item is determined according to the initial disparity map, the first image and the second image used in stereo matching, and the data item constitutes a constraint on the smooth item, so that the smooth item is affected by the abscissa and ordinate of the first image. The influence of the gradient in the direction is also affected by the gradient of the second image in the abscissa direction and the ordinate direction. Therefore, the degree of matching between the first image and the second image imposes constraints on the smoothing term, so that the smoothing can be controlled. The item smoothes the areas that should be smoothed in the initial disparity map, and ignores the areas that should not be smoothed, so as to prevent errors from spreading to normal areas, so as to ensure the accuracy of the disparity values and the accuracy of the value range in the optimized disparity map.

It should be noted that the various embodiments in this specification are described in a progressive manner, and each embodiment focuses on the differences from other embodiments. For the same and similar parts among the various embodiments, refer to each other Can.

In the several embodiments provided in this application, it should be understood that the disclosed apparatus and method may also be implemented in other manners. The apparatus embodiments described above are merely illustrative, for example, the flowcharts and block diagrams in the accompanying drawings illustrate the architectures, functions and possible implementations of apparatuses, methods and computer program products according to various embodiments of the present application. operate. In this regard, each block in the flowchart or block diagrams may represent a module, segment, or portion of code, which may contain one or more logical functions for implementing the specified functions executable instructions. It should also be noted that, in some alternative implementations, the functions noted in the block may occur out of the order noted in the figures. For example, two blocks in succession may, in fact, be executed substantially concurrently, or the blocks may sometimes be executed in the reverse order, depending upon the functionality involved. It is also noted that each block of the block diagrams and/or flowchart illustrations, and combinations of blocks in the block diagrams and/or flowchart illustrations, can be implemented by dedicated hardware-based systems that perform the specified functions or actions, Alternatively, it may be implemented in a combination of dedicated hardware and computer instructions.

In addition, each functional module in each embodiment of the present application may be integrated together to form an independent part, or each module may exist independently, or two or more modules may be integrated to form an independent part.

If the functions are implemented in the form of software function modules and sold or used as independent products, they may be stored in a computer-readable storage medium. Based on this understanding, the technical solution of the present application can be embodied in the form of a software product in essence, or the part that contributes to the related technology or the part of the technical solution. The computer software product is stored in a storage medium, including several The instructions are used to cause a computer device (which may be a personal computer, a notebook computer, a server, or a network device, etc.) to execute all or part of the steps of the methods described in the various embodiments of the present application. The aforementioned storage medium includes: U disk, mobile hard disk, Read-Only Memory (ROM, Read-Only Memory), Random Access Memory (RAM, Random Access Memory), magnetic disk or optical disk and other media that can store program codes .

The above are only specific embodiments of the present application, but the protection scope of the present application is not limited to this. should be covered within the scope of protection of this application.

Industrial Applicability

The present application discloses a disparity map optimization method, apparatus, electronic device, and computer-readable storage medium. The method includes: acquiring a first image, a second image and an initial disparity map corresponding to the first image and the second image; An energy equation for conducting guided filtering, the energy equation including a data item and a smoothing term; performing guided filtering on the initial disparity map according to the energy equation to obtain an optimized disparity map. Through this method, the error can be prevented from spreading to the normal area, so as to ensure the accuracy of the disparity value and the accuracy of the value range in the optimized disparity map.

In addition, it can be understood that the disparity map optimization method, apparatus, electronic device and computer-readable storage medium of the present application are reproducible and can be applied in various industrial applications. For example, the disparity map optimization method of the present application can be applied to the field of image processing.

Claims

A disparity map optimization method, characterized in that the method comprises:

Obtain a first image, a second image, and an initial disparity map corresponding to the first image and the second image, where the first image and the second image are obtained from different shooting angles for the same shooting scene two images;

determining an energy equation for guided filtering from the initial disparity map, the first image, and the second image, the energy equation including a data term and a smoothing term;

Guide filtering is performed on the initial disparity map according to the energy equation to obtain an optimized disparity map.
The method according to claim 1, wherein the determining an energy equation for conducting guided filtering according to the initial disparity map, the first image and the second image comprises:

Determine the data item E data according to the first image and the second image;

determining the smoothing term E data according to the initial disparity map;

Based on the formula E=min(E data +αE smooth ), determine the energy equation E;

Wherein, α is the weight of the smooth item and the data item, and the data item constitutes a constraint effect on the smooth item.
The method according to claim 2, wherein the data item E data is determined according to the first image and the second image and the smoothing item is determined according to the initial disparity map E data includes:

The data item E data is determined based on the formula E data =∫(I 2 (x+d)-I 1 (x)) 2 dx;

The smoothing term E smooth is determined based on the formula E smooth =∫w x (d x ) 2 +w y (d y ) 2 dx;

Wherein, I 1 (x) is the first image, I 2 (x+d) is the second image, d is the initial disparity map, and x is the abscissa of any point in the first image , d x and dy are the gradient of the initial disparity map in the abscissa direction and the gradient in the ordinate direction, respectively, and w x and w y are smoothing weights.
The method according to any one of claims 1 to 3, wherein the obtaining an optimized disparity map by performing guided filtering on the initial disparity map according to the energy equation comprises:

According to the energy equation, calculate the gradient d x of the initial disparity map in the abscissa direction;

The optimized disparity map is obtained by superimposing the initial disparity map d and the gradient dx of the initial disparity map in the abscissa direction.
The method according to claim 4, wherein the method further comprises:

Determine whether the gradient d x of the current initial disparity map in the abscissa direction satisfies the preset rule;

When the judgment is yes, the optimized disparity map corresponding to the gradient d x of the current initial disparity map in the abscissa direction is determined as the final disparity map.
The method according to claim 5, wherein the method further comprises:

When it is judged to be no, perform the following steps until the gradient d x of the current initial disparity map in the abscissa direction satisfies the preset rule, and then compare the gradient d x of the current initial disparity map in the abscissa direction with the current initial disparity map d x The corresponding optimized disparity map is determined as the final disparity map:

Update the initial disparity map according to the optimized disparity map to obtain the current disparity map;

According to the energy equation, calculate the d x of the current disparity map in the abscissa direction;

By superimposing the initial disparity map d and the gradient d x of the current initial disparity map in the abscissa direction, an optimized disparity map corresponding to the gradient d x of the current initial disparity map in the abscissa direction is obtained;

Return to the step of judging whether the gradient d x of the current initial disparity map in the abscissa direction satisfies the preset rule.
The method according to claim 5 or 6, wherein the preset rule is: the gradient d x of the current initial disparity map in the abscissa direction is smaller than the first threshold, and/or the current number of iterations satisfies the first threshold Two thresholds.
The method according to any one of claims 1 to 7, wherein the guide map of the guide filter is the first image.
A disparity map optimization device, characterized in that the device comprises:

an acquisition module, the acquisition module is configured to acquire a first image, a second image and an initial disparity map corresponding to the first image and the second image, the first image and the second image are Two images obtained from different shooting angles for the same shooting scene;

a determination module configured to determine an energy equation for guided filtering based on the initial disparity map, the first image, and the second image, the energy equation including a data term and a smoothing term;

A guided filtering module configured to perform guided filtering on the initial disparity map according to the energy equation to obtain an optimized disparity map.
The device according to claim 9, wherein the determining module is configured to: determine the data item E data according to the first image and the second image; determine according to the initial disparity map The smoothing term E data is obtained; based on the formula E=min(E data +αE smooth ), the energy equation E is determined;

Wherein, α is the weight of the smooth item and the data item, and the data item constitutes a constraint effect on the smooth item.
The apparatus of claim 10, wherein the determining module is configured to: determine the data based on the formula E data =∫(I 2 (x+d)-I 1 (x)) 2 dx term E data ; based on the formula E smooth =∫w x (d x ) 2 +w y (d y ) 2 dx, determine the smoothing term E smooth ;

Wherein, I 1 (x) is the first image, I 2 (x+d) is the second image, d is the initial disparity map, and x is the abscissa of any point in the first image , d x and dy are the gradient of the initial disparity map in the abscissa direction and the gradient in the ordinate direction, respectively, and w x and w y are smoothing weights.
The apparatus according to any one of claims 9 to 11, wherein the guided filtering module is configured to: calculate the gradient d of the initial disparity map in the abscissa direction according to the energy equation x ; the optimized disparity map is obtained by superimposing the initial disparity map d and the gradient dx of the initial disparity map in the abscissa direction.
The apparatus of claim 12, wherein the apparatus further comprises:

a judgment module configured to judge whether the gradient d x of the current initial disparity map in the abscissa direction satisfies a preset rule; and

An execution module, the execution module is configured to determine the optimized disparity map corresponding to the gradient d x of the current initial disparity map in the abscissa direction as the final disparity map when the judgment module judges yes.
The apparatus of claim 12, wherein the apparatus further comprises:

a judgment module, the judgment module is configured to judge whether the gradient d x of the current initial disparity map in the abscissa direction satisfies a preset rule;

An execution module, the execution module is configured to perform the following process when the judgment module judges no, until the gradient d x of the current initial disparity map in the abscissa direction satisfies the preset rule, and then compares the The optimized disparity map corresponding to the gradient d x of the current initial disparity map in the abscissa direction is determined as the final disparity map:

Update the initial disparity map according to the optimized disparity map to obtain the current disparity map; calculate the gradient d x of the current disparity map in the abscissa direction according to the energy equation; Figure d is superimposed with the gradient d x of the current initial disparity map in the abscissa direction to obtain an optimized disparity map corresponding to the gradient d x of the current initial disparity map in the abscissa direction; return to judge the current initial disparity The step of whether the gradient d x of the graph in the abscissa direction satisfies the preset rule.
The device according to claim 13 or 14, wherein the preset rule is: the gradient d x of the current initial disparity map in the abscissa direction is smaller than the first threshold, and/or the current number of iterations satisfies the first threshold Two thresholds.
The apparatus according to any one of claims 9 to 15, wherein the guide map of the guide filter is the first image.
An electronic device, characterized in that the electronic device comprises: a memory and a processor, and the memory is connected to the processor;

the memory is used to store programs;

The processor invokes a program stored in the memory to perform the method according to any one of claims 1 to 8.
A computer-readable storage medium, characterized in that a computer program is stored on the computer-readable storage medium, and the computer program executes the method according to any one of claims 1 to 8 when the computer program is run by a computer.