WO2022012034A1

WO2022012034A1 - Image processing method and apparatus, electronic device, and storage medium

Info

Publication number: WO2022012034A1
Application number: PCT/CN2021/075111
Authority: WO
Inventors: 王光甫; 王珏; 刘帅成
Original assignee: 北京迈格威科技有限公司; 成都旷视金智科技有限公司
Priority date: 2020-07-14
Filing date: 2021-02-03
Publication date: 2022-01-20
Also published as: CN112053394A

Abstract

Embodiments of the present application provide an image processing method and apparatus, an electronic device, and a storage medium. The method comprises: determining all regions used for determining a region to be optimized, the all regions used for determining a region to be optimized comprising at least one of the following: at least one low-confidence region, at least one region corresponding to a weak texture region, and at least one region corresponding to a repeating texture region; determining, on the basis of all regions used for determining a region to be optimized, a region to be optimized in a disparity map; and optimizing disparity values of pixels in the region to be optimized to obtain an optimized disparity map. A region to be optimized in a disparity map is accurately determined, and disparity values of pixels in the region to be optimized in the disparity map are optimized to accurately optimize the disparity map, thereby obtaining an optimized disparity map having a better optimization effect.

Description

Image processing method, device, electronic device and storage medium

This application claims the priority of the Chinese patent application filed on July 14, 2020 with the application number 202010676768.6 and the title of the invention is "image processing method, device, electronic device and storage medium", the entire contents of which are incorporated by reference in this application.

technical field

computer field

Background technique

Binocular stereo vision has been widely used in industrial inspection, aerospace, robot navigation and other fields. Binocular stereo vision recovers the 3D depth information of the scene by calculating the disparity of spatial points on two images under the same scene.

Stereo matching is a key link in binocular stereo vision. The result of stereo matching directly affects the 3D reconstruction effect, and stereo matching depends on the disparity map.

In the disparity map obtained by stereo matching the two images used to generate the disparity map by the stereo matching algorithm, the disparity map often appears in the disparity map corresponding to some areas in the image used for generating the disparity map, such as weak texture areas A situation where the disparity value in the area is inaccurate. Therefore, after the disparity map is calculated, how to accurately determine the area to be optimized in the disparity map and how to optimize the disparity value of the pixels in the area to be optimized in the disparity map becomes a problem to be solved.

SUMMARY OF THE INVENTION

In order to overcome the problems existing in the related art, the present application provides an image processing method, an apparatus, an electronic device and a storage medium.

According to a first aspect of the embodiments of the present application, an image processing method is provided, including:

Determine all areas in the disparity map for determining the area to be optimized, and all areas used for determining the area to be optimized include at least one of the following items: at least one low-confidence area, at least one area corresponding to a weak texture area , at least one area corresponding to the repeated texture area, wherein, the low confidence area is determined by the confidence matrix of the disparity map output by the stereo matching algorithm used to generate the disparity map, and the weak texture area and the repeated texture area are determined by The images used to generate the disparity map are determined by detecting weak texture regions and repeating texture regions respectively;

Determine the area to be optimized in the disparity map based on all the areas used to determine the area to be optimized;

The disparity values of the pixels in the to-be-optimized area are optimized to obtain an optimized disparity map.

In some embodiments, based on all the regions used to determine the region to be optimized, determining the region to be optimized in the disparity map includes:

Determine the union of all regions used to determine the region to be optimized;

The determined union is determined as the region to be optimized.

In some embodiments, optimizing the disparity values of the pixels in the area to be optimized to obtain an optimized disparity map includes:

Determine all connected regions in the to-be-optimized region by using a connected region extraction algorithm;

Determine the target connected region among all connected regions;

For each target connected area, morphological gradient extraction is performed on the target connected area to determine the edge of the target connected area; based on the original disparity value of the pixels in the area occupied by the edge of the target connected area, calculate The target disparity value of each pixel in the target connected area; modify the disparity value of each pixel in the target connected area to the target disparity value of each pixel.

In some embodiments, determining a target connected region among all connected regions includes:

A connected region whose area is greater than the area threshold in all connected regions is determined as the target connected region.

In some embodiments, calculating the target disparity value of each pixel in the target connected region based on the original disparity value of the pixels in the region occupied by the edge of the target connected region includes:

Based on the original disparity values of all target pixels in the area occupied by the edge of the target connected area, calculate the target disparity value of each pixel in the target connected area, wherein the target pixel has the original disparity value of Pixels within a preset reasonable range.

In some embodiments, based on the original disparity values of all target pixels in the area occupied by the edge of the target connected area, calculating the target disparity value of each pixel in the target connected area includes:

Calculate the average value of the original disparity values of all target pixels in the area occupied by the edge of the target connected area; take the average value as the target disparity value of each pixel in the target connected area.

In some embodiments, the method further includes:

obtaining feature information of the region corresponding to the region to be smoothed in the optimized disparity map in the image used to generate the disparity map;

Based on the feature information, a guided filtering algorithm and a fast bilateral filtering algorithm are used to smooth the region to be smoothed in the optimized disparity map.

According to a second aspect of the embodiments of the present application, an image processing apparatus is provided, including:

an associated area determination unit, configured to determine all areas used for determining the area to be optimized, all areas used for determining the area to be optimized include at least one of the following items: at least one low-confidence area, at least one area with a weak texture a corresponding area, at least one area corresponding to a repetitive texture area, wherein the low confidence area is determined based on the confidence matrix of the disparity map output by the stereo matching algorithm used to generate the disparity map, the weak texture area , the repeated texture area is determined by performing weak texture area detection and repeated texture area detection on the image used to generate the disparity map respectively;

an area to be optimized determining unit, configured to determine the area to be optimized in the disparity map based on all the areas used to determine the area to be optimized;

The disparity map optimization unit is configured to optimize the disparity values of the pixels in the to-be-optimized area to obtain an optimized disparity map.

In some embodiments, the region-to-be-optimized determination unit is further configured to determine a union of all regions used to determine the region to be optimized; and the determined union is determined as the region to be optimized.

In some embodiments, the disparity map optimization unit includes:

The connected area optimization subunit is configured to use a connected area extraction algorithm to determine all connected areas in the area to be optimized; to determine the target connected area in all connected areas; for each target connected area, perform an operation on the target connected area. Morphological gradient extraction to determine the edge of the target connected area; based on the original disparity value of the pixels in the area occupied by the edge of the target connected area, calculate the target disparity of each pixel in the target connected area value; modify the disparity value of each pixel in the target connected region to the target disparity value of each pixel.

In some embodiments, the connected region optimization subunit is further configured to determine a connected region whose area is greater than an area threshold among all connected regions as a target connected region.

In some embodiments, the connected region optimization subunit is further configured to calculate the target for each pixel in the target connected region based on the original disparity values of all target pixels in the region occupied by the edge of the target connected region disparity value, where the target pixel is a pixel whose original disparity value is within a preset reasonable interval.

In some embodiments, the connected area optimization sub-unit is further configured to calculate the average value of the original disparity values of all target pixels in the area occupied by the edge of the target connected area; take the average value as the target connected area The target disparity value for each pixel in the region.

In some embodiments, the image processing apparatus further includes:

a smoothing unit, configured to acquire feature information of an area corresponding to the area to be smoothed in the optimized disparity map in the image used to generate the disparity map;

The pedestrian re-identification method and device provided by the embodiments of the present application realize the consideration of the low-confidence area in the disparity map, the area corresponding to the weak texture area in the disparity map, and the repeated texture area in the disparity map. The association of the area with the area to be processed, based on all areas used to determine the area to be optimized including at least one of the following: at least one low confidence area, at least one area corresponding to a weak texture area, at least one area with Repeat the area corresponding to the texture area, accurately determine the area to be optimized in the disparity map, optimize the disparity value of the pixels in the area to be optimized, so as to accurately optimize the disparity map, and obtain a process with better optimization effect. Optimized disparity map.

The above description is only an overview of the technical solutions of the present invention, in order to be able to understand the technical means of the present invention more clearly, it can be implemented according to the content of the description, and in order to make the above and other objects, features and advantages of the present invention more obvious and easy to understand , the following specific embodiments of the present invention are given.

Description of drawings

In order to more clearly illustrate the embodiments of the present invention or the technical solutions in the prior art, the following briefly introduces the accompanying drawings that need to be used in the description of the embodiments or the prior art. Obviously, the accompanying drawings in the following description These are some embodiments of the present invention. For those of ordinary skill in the art, other drawings can also be obtained according to these drawings without creative efforts.

The accompanying drawings, which are incorporated in and constitute a part of this specification, illustrate embodiments consistent with the application and together with the description serve to explain the principles of the application.

FIG. 1 shows a flowchart of an image processing method provided by an embodiment of the present application;

FIG. 2 shows a schematic flowchart of determining the area to be optimized in the disparity map;

3 shows a schematic flowchart of optimizing the disparity values of pixels in the area to be optimized in the disparity map;

FIG. 4 shows a structural block diagram of an image processing apparatus provided by an embodiment of the present application;

FIG. 5 shows a structural block diagram of an electronic device provided by an embodiment of the present application.

specific embodiment

In order to make the purposes, technical solutions and advantages of the embodiments of the present invention clearer, the technical solutions in the embodiments of the present invention will be clearly and completely described below with reference to the accompanying drawings in the embodiments of the present invention. Obviously, the described embodiments These are some embodiments of the present invention, but not all embodiments. Based on the embodiments of the present invention, all other embodiments obtained by those of ordinary skill in the art without creative efforts shall fall within the protection scope of the present invention.

The present application will be further described in detail below with reference to the accompanying drawings and embodiments. It should be understood that the specific embodiments described herein are only used to explain the related invention, but not to limit the invention. In addition, it should be noted that, for the convenience of description, only the parts related to the related invention are shown in the drawings.

It should be noted that the embodiments in the present application and the features of the embodiments may be combined with each other in the case of no conflict. The present application will be described in detail below with reference to the accompanying drawings and in conjunction with the embodiments.

FIG. 1 shows a flowchart of an image processing method provided by an embodiment of the present application, and the method includes:

Step 101: Determine all areas in the disparity map for determining the area to be optimized.

In this application, all regions used to determine the region to be optimized include at least one of the following items: at least one low confidence region, at least one region corresponding to a weak texture region, at least one region corresponding to a repeated texture region area.

In the present application, if the disparity map includes a low-confidence region, the low-confidence region in the disparity map can be used as the region for determining the region to be optimized. If the image used to generate the disparity map includes a weak texture area, correspondingly, the disparity map includes an area corresponding to the weak texture area, then the area corresponding to the weak texture area in the disparity map can be used as the area for determining the area to be optimized. area. If the image used to generate the disparity map includes a repeated texture area, correspondingly, the disparity map includes an area corresponding to the repeated texture area, and the area corresponding to the repeated texture area in the disparity map can be used as the area for determining the area to be optimized. area.

The disparity map is generated by stereo matching the left RGB image used to generate the disparity map and the right RGB image used to generate the disparity map by a stereo matching algorithm. The pixel value of each pixel in the disparity map is a disparity value. Therefore, the pixel value of the pixel in the disparity map can be referred to as the disparity value of the pixel.

The stereo matching algorithm may be a traditional stereo matching algorithm such as SGBM and BM, or a stereo matching algorithm based on a convolutional neural network.

The output of the stereo matching algorithm includes a disparity map and a confidence matrix for the disparity map.

The confidence matrix elements in the confidence matrix correspond one-to-one with the disparity values of the pixels in the disparity map. The value of the confidence matrix element in the confidence matrix is the confidence of the disparity value of the pixel in the disparity map corresponding to the confidence matrix element.

For each pixel in the disparity map, the position of the disparity value of the pixel in the disparity map is the same as the position of the confidence matrix element corresponding to the pixel in the confidence matrix, and the value of the confidence matrix element corresponding to the pixel is the confidence of the disparity value of this pixel.

The low-confidence area in the disparity map is an area in which the confidences of the pixels included in the disparity map are all smaller than the confidence threshold. The number of low confidence regions in the disparity map may be one or more.

All pixels in the disparity map whose confidences of disparity values are smaller than the confidence threshold may be determined according to the confidence of each pixel in the disparity map described by the confidence matrix and the confidence threshold. At least one low-confidence region in the disparity map is determined according to the positions of all pixels in the disparity map having a confidence level of a disparity value less than a confidence level threshold.

In this application, the weak texture area may refer to the weak texture area in the left RGB image used to generate the disparity map. The repeated texture area may refer to a repeated texture area in the left RGB image used to generate the disparity map.

In the present application, weak texture area detection may be performed on the left RGB image used for generating the disparity map to determine the weak texture area in the left RGB image used for generating the disparity map. Repeat texture region detection may be performed on the left RGB image used to generate the disparity map to determine repeated texture regions in the left RGB image used to generate the disparity map.

When the left RGB image used for generating the disparity map includes weak texture areas, the determined number of weak texture areas may be one or more, in other words, the left RGB image used for generating the disparity map may include one or more Weak textured areas. When the left RGB image used for generating the disparity map includes repeated texture regions, the determined number of repeated texture regions may be one or more, in other words, the left RGB image used for generating the disparity map may include one or more Repeat texture area.

When there are multiple weak texture regions, each weak texture region corresponds to one region in the disparity map. When the number of repeated texture regions is multiple, each repeated texture region corresponds to one region in the disparity map.

Each pixel in the disparity map corresponds to a pixel in the left RGB image used to generate the disparity map.

For each weak texture area, all pixels in the disparity map in the area corresponding to the weak texture area are composed of pixels in the disparity map corresponding to each pixel in the weak texture area, and the disparity map is the same as the pixel in the disparity map. The area corresponding to the weak texture area is composed of the area occupied by the corresponding pixel of each pixel in all the weak texture areas. Therefore, for each weak texture area, after the weak texture area is determined through weak texture area detection, the area corresponding to the weak texture area in the disparity map can be determined.

For each repeated texture region, all pixels in the disparity map in the region corresponding to the repeated texture region are composed of pixels in the disparity map corresponding to each pixel in the repeated texture region. Therefore, for each repeated texture region, after determining the repeated texture region through repeated texture region detection, a region corresponding to the repeated texture region in the disparity map can be determined.

In the present application, a first pixel-level detection convolutional neural network may be used to perform weak texture region detection on the image used for generating the disparity map to determine the weak texture region in the image used for generating the disparity map. A second pixel-level detection convolutional neural network may be used to perform repetitive texture region detection on the image used to generate the disparity map to determine repeated texture regions in the image used to generate the disparity map.

The first pixel-level detection convolutional neural network and the second pixel-level detection convolutional neural network are both neural networks used for semantic segmentation. The structure of the first pixel-level detection convolutional neural network and the second pixel-level detection convolutional neural network same.

In the present application, the first pixel-level detection convolutional neural network may be used to detect weak texture regions on the left RGB image used to generate the disparity map, so as to determine weak texture areas in the left RGB image used to generate the disparity map texture area.

The first pixel-level detection convolutional neural network is pre-trained with the training images. Each weakly textured region in the training image utilized to train the first pixel-level detection convolutional neural network is annotated.

The left RGB image used to generate the disparity map is input to the first pixel-level detection convolutional neural network, and the detection result output by the first pixel-level detection convolutional neural network may be an indication of the left RGB used to generate the disparity map. Whether each pixel in the image belongs to a weakly textured region of the image. Thus, according to the detection result, at least one weak texture area can be determined.

In the present application, a second pixel-level detection convolutional neural network can be used to perform repetitive texture region detection on the left RGB image used to generate the disparity map to determine the repetition in the left RGB image used to generate the disparity map texture area.

The second pixel-level detection convolutional neural network is pre-trained with training. Repeated texture regions in the training image used to train the second pixel-level detection convolutional neural network are annotated.

The left RGB image used to generate the disparity map is input to the second pixel-level detection convolutional neural network, and the detection result output by the second pixel-level detection convolutional neural network may be an indication of the left RGB used to generate the disparity map Whether each pixel in the image belongs to an image with a repeating texture area. Thus, based on the detection result, at least one repeating texture region can be determined.

Step 102: Determine the area to be optimized in the disparity map based on all the areas used to determine the area to be optimized.

In this application, when determining the area to be optimized in the disparity map based on all the areas used to determine the area to be optimized, for each independent area in all the areas used to determine the area to be optimized, the independent area can be used as A subregion of the region to be optimized in the disparity map.

When the number of weak texture areas is multiple, each area corresponding to the weak texture area corresponds to one weak texture area.

When the number of repeated texture regions is multiple, each region corresponding to the repeated texture region corresponds to one repeated texture region.

The independent area may refer to all the areas used for determining the area to be optimized that do not have overlapping parts with any other area used for determining the area to be optimized.

The non-independent regions may refer to all regions used for determining the regions to be optimized that have overlapping portions with at least one other region used for determining the regions to be optimized.

In the case where any two areas in all the areas used to determine the area to be optimized do not overlap, each area in all the areas used to determine the area to be optimized is an independent area, and the area to be optimized in the disparity map is determined by All areas used for determining the area to be optimized are composed, that is, the area to be optimized in the disparity map includes all areas used for determining the area to be optimized.

For example, all the regions used to determine the region to be optimized include: at least one region of low confidence, at least one region corresponding to a weak texture region, and at least one region corresponding to a repeated texture region. Under the condition that any two regions in all the regions used to determine the region to be optimized do not overlap, each low-confidence region is an independent region, and each region in the disparity map corresponding to the weak texture region is The independent area, each area corresponding to the repeated texture area in the disparity map is an independent area, then the area to be optimized in the disparity map is composed of each low-confidence area in the disparity map, each in the disparity map and weak. The area corresponding to the texture area and each area in the disparity map corresponding to the repeated texture area are composed.

In the case that all the regions used to determine the region to be optimized include at least two dependent regions, each overlapping portion belonging to the at least two dependent regions at the same time can be regarded as a sub-region of the region to be optimized in the disparity map.

In the case that all the regions used to determine the region to be optimized include at least two dependent regions, the region to be optimized in the disparity map may be composed of each independent region, each of which simultaneously belongs to the overlapping portion of the at least two dependent regions. The to-be-optimized area in the disparity map may include: each independent area, each overlapping part belonging to at least two dependent areas at the same time.

In some embodiments, based on all the regions in the disparity map for determining the region to be optimized, determining the region to be optimized in the disparity map includes: determining a union of all regions used for determining the region to be optimized; combining the determined union The set is determined as the region to be optimized in the disparity map.

In the case where any two regions in all the regions used to determine the region to be optimized do not overlap, the union of all the regions used to determine the region to be optimized is all the regions used to determine the region to be optimized. The union of the regions used to determine the region to be optimized is taken as the region to be optimized, in other words, the region to be optimized consists of all the regions used to determine the region to be optimized.

For example, all the regions used to determine the region to be optimized include: at least one region of low confidence, at least one region corresponding to a weak texture region, and at least one region corresponding to a repeated texture region. In the case that any two regions in all the regions used to determine the region to be optimized have no overlap, the union of all the regions used to determine the region to be optimized is determined by each low-confidence region in the disparity map, the disparity map Each of the regions corresponding to the weak texture region and each region corresponding to the repeated texture region in the disparity map is composed of, then the region to be optimized in the disparity map includes: each low-confidence region in the disparity map, the parallax Each region in the map corresponds to a weak texture region, and each region in the disparity map corresponds to a repeated texture region.

In the case that all the regions used for determining the region to be optimized include at least two dependent regions, the union of all the regions used for determining the region to be optimized is determined by each independent region, each belonging to at least two dependent regions at the same time. Overlapping parts, each of which belongs to only one dependent region and which are not overlapping parts. Then, the areas to be optimized in the disparity map include: each independent area, each overlapping part belonging to at least two dependent areas at the same time, and each part belonging to only one dependent area and not the overlapping part.

Please refer to FIG. 2 , which shows a schematic flowchart of determining an area to be optimized.

The area to be optimized may also be referred to as a parallax inaccuracy area. The low confidence region in the disparity map can be determined according to the confidence matrix and the confidence threshold.

A low-confidence mask can be generated first. Each pixel in the low-confidence mask corresponds to a pixel in the disparity map.

In the low-confidence mask, for each pixel corresponding to a pixel belonging to the low-confidence region, the pixel value of the pixel is a value of 1 indicating that the corresponding pixel belongs to the low-confidence region. For each pixel except all pixels corresponding to pixels belonging to the low-confidence region, the pixel value of the pixel is a value of 0 indicating that its corresponding pixel does not belong to the low-confidence region.

In the low-confidence mask, for each pixel whose value is 0 and the pixel in the corresponding disparity map belongs to the region corresponding to the weak texture region or the region corresponding to the repeated texture region, the pixel value of the pixel is Modify it to 1 to get the mask image of the region to be optimized.

The to-be-optimized area mask image may be referred to as the to-be-optimized area mask.

Each pixel in the area mask to be optimized corresponds to one pixel in the disparity map.

In the mask of the area to be optimized, the pixel value of the pixel corresponding to the pixel in the area to be optimized is 1 indicating that the corresponding pixel belongs to the area to be optimized, and the pixel value of the pixel corresponding to the pixel in the area not to be optimized It is a value of 0 indicating that its corresponding pixel does not belong to the area to be optimized.

All the pixels in the to-be-optimized area in the disparity map are composed of pixels corresponding to each pixel with a pixel value of 1 in each of the to-be-optimized area masks. Therefore, after obtaining the area mask to be optimized, the area to be optimized in the disparity map can be determined.

Step 103: Optimize the disparity values of the pixels in the area to be optimized to obtain an optimized disparity map.

In the present application, after the area to be optimized in the disparity map is determined, the disparity values of the pixels in the area to be optimized are optimized.

Each pixel in the disparity map has an original disparity value. When optimizing the disparity value of the pixel in the area to be optimized, for each pixel in the area to be optimized, the target disparity value of the pixel can be determined, and the disparity value of the pixel can be modified to the target disparity value. Therefore, the disparity value of the pixel is changed from the original disparity value of the pixel to the target disparity value of the pixel.

After the pixel value of each pixel in the to-be-optimized area of the disparity map is modified to the corresponding target disparity value, an optimized disparity map is obtained.

For each pixel in the area to be optimized, when determining the target disparity value of the pixel, for each pixel in the area to be optimized, you can search for an effective pixel in the disparity map that is closest to the pixel and does not belong to the area to be optimized. For disparity pixels, the disparity value of the found effective disparity pixel can be used as the target pixel value of the pixel.

In some embodiments, the method further includes: acquiring feature information of a region corresponding to the region to be smoothed in the optimized disparity map in the image used to generate the disparity map; based on the feature information, using a guided filtering algorithm and a fast bilateral The filtering algorithm smoothes the area to be smoothed in the optimized disparity map.

The area corresponding to the area to be smoothed in the optimized disparity map may be the area occupied by the subject in the left RGB image used to generate the disparity map.

For example, in the portrait mode, the area corresponding to the to-be-smoothed area in the optimized disparity map may occupy the portrait area for the portrait in the left RGB image used to generate the disparity map. In other words, the region to be smoothed in the optimized disparity map corresponds to the portrait region in the left RGB image for generating the disparity map.

The feature information of the area corresponding to the area to be smoothed in the optimized disparity map in the image used for generating the disparity map is information based on smoothing the area to be smoothed in the optimized disparity map.

For example, the area corresponding to the area to be smoothed in the optimized disparity map is the portrait area in the left RGB image used to generate the disparity map, and the feature information may be the left RGB image used to generate the disparity map. The gradient of each pixel in the portrait region.

Based on the feature information of the region corresponding to the region to be smoothed in the optimized disparity map in the image used to generate the disparity map, the guided filtering algorithm and the fast bilateral filtering algorithm can be used to determine the region to be smoothed in the optimized disparity map. for smoothing. Therefore, the optimized disparity map is smoother, especially in the edge region, the change is not abrupt and the consistency is maintained.

In some embodiments, optimizing the disparity values of pixels in the region to be optimized to obtain an optimized disparity map includes: determining all connected regions in the region to be optimized by using a connected region extraction algorithm; determining target connectivity in all connected regions area; for each target connected area, perform morphological gradient extraction on the target connected area to determine the edge of the target connected area; based on the original disparity value of the pixels in the area occupied by the edge of the target connected area, calculate the The target disparity value of each pixel in the target connected area; modify the disparity value of each pixel in the target connected area to the target disparity value of each pixel.

When using the connected domain extraction algorithm to determine each connected area in the area to be optimized, the connected area extraction algorithm can be used to determine each connected area in the mask area to be optimized based on the mask image of the area to be optimized corresponding to the disparity map, that is, the area mask to be optimized. a connected region. The pixel value of each pixel in the area to be optimized mask image corresponding to each pixel in the area to be optimized is 1, and the area to be optimized in the disparity map can be quickly located through the mask image of the area to be optimized. Then, a connected region extraction algorithm may be used to extract a connected region in the region to be optimized in the disparity map, and each connected region in the region to be optimized in the disparity map is determined.

In the present application, after each connected region in the region to be optimized is determined, a target connected region in the region to be optimized can be determined.

In the present application, each connected area can be individually used as a target connected area.

In some embodiments, determining a target connected area in all connected areas includes: determining a connected area whose area is greater than an area threshold in all connected areas as a target connected area.

By determining the connected regions whose area is larger than the area threshold in all connected regions as target connected regions, the connected regions with smaller areas can be eliminated, and the eliminated connected regions can be called ineffective regions.

In this application, after determining the target connected regions in all connected regions, for each target connected region, morphological gradient extraction can be performed on the target connected region to determine the edge of the target connected region; based on the target connected region The original disparity value of the pixels in the area occupied by the edge of , calculate the target disparity value of each pixel in the target connected area; modify the disparity value of each pixel in the target connected area to Target disparity value.

For each target connected region, morphological gradient extraction is performed on the target connected region to calculate the difference between the expansion map of the target connected region and the erosion map of the connected region to determine the edge of the target connected region.

After the edge of each target connected region is determined, for each target connected region, the target of each pixel in the target connected region can be calculated based on the original disparity value of the pixels in the region occupied by the edge of the target connected region Parallax value.

For example, for each target connected region, the median of the original disparity values of all pixels in the region occupied by the edge of the target connected region or the average of all pixels in the region occupied by the edge of the target connected region can be determined value, the median or the average value is used as the target disparity value of each pixel in the target connected area, and the target disparity value of each pixel in the target connected area is the median or the average value value.

For each target connected area, after calculating the target disparity value of each pixel in the target connected area, you can, for each target connected area, modify the disparity value of each pixel in the target connected area to each The target disparity value of one pixel, that is, the disparity value filling is performed for each target connected area respectively, and the target disparity value of each pixel in the target connected area is filled to the position of the pixel. For each pixel in any target connected region, the disparity value of the pixel changes from the original disparity value to the target disparity value.

Therefore, the disparity value of the pixels in each target connected region is optimized to obtain an optimized disparity map.

In some embodiments, for each target connected region, based on the original disparity value of the pixels in the region occupied by the edge of the target connected region, calculating the target disparity value of each pixel in the target connected region includes: based on The original disparity values of all target pixels in the area occupied by the edge of the target connected area are calculated, and the target disparity value of each pixel in the target connected area is calculated, wherein the original disparity value of the target pixel is in the preset value Pixels within a reasonable range.

In this application, for each target connected region, the target disparity value of each pixel in the connected region can be calculated based on the original disparity values of all target pixels in the region occupied by the edge of the target connected region.

For each target connected region, the median of the original disparity values of all target pixels in the region occupied by the edge of the target connected region can be determined, and the median is taken as the value of each pixel in the target connected region. The target disparity value, the target disparity value of each pixel in the target connected region is the median.

In some embodiments, for each target connected region, calculating the target disparity value of each pixel in the target connected region based on the original disparity values of all target pixels in the region occupied by the edge of the target connected region includes: : Calculate the average value of the original disparity values of all target pixels in the area occupied by the edge of the target connected area; use the average value as the target disparity value of each pixel in the target connected area.

In this application, for each target connected area, the average value of the original disparity values of all target pixels in the area occupied by the edge of the target connected area can be used as the target disparity of each pixel in the target connected area The target disparity value of each pixel in the target connected region is the average value of the original disparity values of all target pixels in the region occupied by the edge of the target connected region.

Therefore, for each target connected region, after modifying the disparity value of each pixel in the target connected region to the target disparity value of each pixel, the disparity value of each pixel in the target connected region is determined by the original The disparity value becomes the average value of the original disparity values of all target pixels in the area occupied by the edge of the target connected area.

Please refer to FIG. 3 , which shows a schematic flowchart of optimizing the disparity values of pixels in the area to be optimized in the disparity map.

Optimizing the disparity values of pixels in the area to be optimized in the disparity map may also be referred to as a local compensation algorithm.

First, a connected region extraction algorithm is used to determine each connected region in the region to be optimized based on the mask image of the region to be optimized corresponding to the disparity map, that is, the region mask to be optimized. The pixel value of each pixel in the to-be-optimized area mask corresponding to each pixel in the to-be-optimized area mask is 1, and the to-be-optimized area mask can be used to quickly locate the to-be-optimized area. Then, the connected region extraction algorithm can be used to extract the connected region of the region to be optimized, and each connected region in the region to be optimized can be determined.

In all connected regions, the connected regions whose area is larger than the area threshold are determined as the target connected regions, so that the connected regions with smaller areas are eliminated, that is, the ineffective regions.

For each target connected region, morphological gradient extraction is performed on the target connected region to determine the edge of the target connected region.

For each target connected region, the average value of the original disparity values of all target pixels in the region occupied by the edge of the target connected region can be used as the target disparity value of each pixel in the target connected region.

Disparity filling is performed on each target connected area, that is, for each target connected area, the target disparity value of each pixel in the target connected area is filled to the position of the pixel.

For each target connected area, the disparity value of each pixel in the target connected area is changed from the original disparity value to the average value of the original disparity values of all target pixels in the area occupied by the edge of the target connected area.

Please refer to FIG. 4 , which shows a structural block diagram of an image processing apparatus provided by an embodiment of the present application. The image processing apparatus includes: an associated area determination unit 401 , a to-be-optimized area determination unit 402 , and a disparity map optimization unit 403 .

an associated area determination unit, configured to determine all areas in the disparity map for determining the area to be optimized, all areas used for determining the area to be optimized include at least one of the following items: at least one low-confidence area, at least one regions corresponding to regions of weak texture, at least one region corresponding to regions of repeated texture, wherein regions of low confidence are determined based on a confidence matrix of the disparity map output by a stereo matching algorithm used to generate the disparity map , the weak texture area and the repeated texture area are determined by respectively performing weak texture area detection and repeated texture area detection on the image used to generate the disparity map;

In some embodiments, the region-to-be-optimized determination unit is further configured to determine a union of all regions used to determine the region to be optimized; and determine the determined union as the region to be optimized in the disparity map.

In some embodiments, the disparity map optimization unit includes:

In some embodiments, the connected region optimization subunit is further configured to calculate a target for each pixel in the target connected region based on the original disparity values of all target pixels in the region occupied by the edge of the target connected region disparity value, where the target pixel is a pixel whose original disparity value is within a preset reasonable interval.

In some embodiments, the image processing apparatus further includes:

FIG. 5 is a structural block diagram of an electronic device provided in this embodiment. The electronic device includes a processing component 522, which further includes one or more processors, and a memory resource, represented by memory 532, for storing instructions, such as application programs, executable by the processing component 522. An application program stored in memory 532 may include one or more modules, each corresponding to a set of instructions. Additionally, the processing component 522 is configured to execute instructions to perform the above-described methods.

The electronic device may also include a power supply assembly 526 configured to perform power management of the electronic device, a wired or wireless network interface 550 configured to connect the electronic device to a network, and an input output (I/O) interface 558. The electronic device may operate based on an operating system stored in memory 532, such as Windows Server™, MacOS X™, Unix™, Linux™, FreeBSD™ or the like.

In an exemplary embodiment, a storage medium including instructions, such as a memory including instructions, is also provided, and the instructions are executable by an electronic device to perform the above method. Alternatively, the storage medium may be a non-transitory computer-readable storage medium, for example, the non-transitory computer-readable storage medium may be ROM, random access memory (RAM), CD-ROM, magnetic tape, floppy disk, and optical data storage equipment, etc.

Other embodiments of the present application will readily occur to those skilled in the art upon consideration of the specification and practice of the invention disclosed herein. This application is intended to cover any variations, uses or adaptations of this application that follow the general principles of this application and include common knowledge or conventional techniques in the technical field not disclosed in this application . The specification and examples are to be regarded as exemplary only, with the true scope and spirit of the application being indicated by the following claims.

It is to be understood that the present application is not limited to the precise structures described above and illustrated in the accompanying drawings and that various modifications and changes may be made without departing from the scope thereof. The scope of the application is limited only by the appended claims.

Reference herein to "one embodiment," "an embodiment," or "one or more embodiments" means that a particular feature, structure, or characteristic described in connection with an embodiment is included in at least one embodiment of the present invention. Also, please note that instances of the phrase "in one embodiment" herein are not necessarily all referring to the same embodiment.

In the description provided herein, numerous specific details are set forth. It will be understood, however, that embodiments of the invention may be practiced without these specific details. In some instances, well-known methods, structures and techniques have not been shown in detail in order not to obscure an understanding of this description.

In the claims, any reference signs placed between parentheses shall not be construed as limiting the claim. The word "comprising" does not exclude the presence of elements or steps not listed in a claim. The word "a" or "an" preceding an element does not exclude the presence of a plurality of such elements. The invention can be implemented by means of hardware comprising several different elements and by means of a suitably programmed computer. In a unit claim enumerating several means, several of these means may be embodied by one and the same item of hardware. The use of the words first, second, and third, etc. do not denote any order. These words can be interpreted as names.

Claims

An image processing method, characterized in that the method comprises:

Determine all areas in the disparity map for determining the area to be optimized, and all areas used for determining the area to be optimized include at least one of the following items: at least one low-confidence area, at least one area corresponding to a weak texture area , at least one area corresponding to the repeated texture area, wherein, the low confidence area is determined based on the confidence matrix of the disparity map output by the stereo matching algorithm used to generate the disparity map, the weak texture area, the repeated texture area Determined by performing weak texture region detection and repeated texture region detection on the image used to generate the disparity map;

determining the region to be optimized in the disparity map based on all the regions used to determine the region to be optimized;

The disparity values of the pixels in the to-be-optimized area are optimized to obtain an optimized disparity map.
The method according to claim 1, wherein, based on all the regions used to determine the region to be optimized, determining the region to be optimized in the disparity map comprises:

Determine the union of all regions used to determine the region to be optimized;

The determined union is determined as the region to be optimized in the disparity map.
The method according to claim 1, wherein, optimizing the disparity value of the pixels in the area to be optimized, and obtaining an optimized disparity map comprises:

Determine all connected regions in the to-be-optimized region by using a connected region extraction algorithm;

Determine the target connected region among all connected regions;

For each target connected area, morphological gradient extraction is performed on the target connected area to determine the edge of the target connected area; based on the original disparity value of the pixels in the area occupied by the edge of the target connected area, calculate The target disparity value of each pixel in the target connected area; modify the disparity value of each pixel in the target connected area to the target disparity value of each pixel.
The method according to claim 3, wherein determining the target connected region in all connected regions comprises:

A connected region whose area is greater than the area threshold in all connected regions is determined as the target connected region.
The method according to claim 4, wherein calculating the target disparity value of each pixel in the target connected area based on the original disparity value of the pixels in the area occupied by the edge of the target connected area comprises: :

Based on the original disparity values of all target pixels in the area occupied by the edge of the target connected area, calculate the target disparity value of each pixel in the target connected area, wherein the target pixel has the original disparity value of Pixels within a preset reasonable range.
The method according to claim 5, wherein the target disparity of each pixel in the target connected area is calculated based on the original disparity values of all target pixels in the area occupied by the edge of the target connected area Values include:

Calculate the average value of the original disparity values of all target pixels in the area occupied by the edge of the target connected area;

The average value is taken as the target disparity value of each pixel in the target connected region.
The method according to any one of claims 1-6, wherein the method further comprises:

acquiring feature information of the region corresponding to the region to be smoothed in the optimized disparity map in the image used to generate the disparity map;

Based on the feature information, a guided filtering algorithm and a fast bilateral filtering algorithm are used to smooth the region to be smoothed in the optimized disparity map.
An image processing device, characterized in that the device comprises:

an associated area determination unit, configured to determine all areas in the disparity map for determining the area to be optimized, all areas used for determining the area to be optimized include at least one of the following items: at least one low-confidence area, at least one regions corresponding to regions of weak texture, at least one region corresponding to regions of repeated texture, wherein regions of low confidence are determined based on a confidence matrix of the disparity map output by a stereo matching algorithm used to generate the disparity map , the weak texture area and the repeated texture area are determined by respectively performing weak texture area detection and repeated texture area detection on the image used to generate the disparity map;

an area to be optimized determining unit, configured to determine the area to be optimized in the disparity map based on all the areas used to determine the area to be optimized;

The disparity map optimization unit is configured to optimize the disparity values of the pixels in the to-be-optimized area to obtain an optimized disparity map.
An electronic device comprising:

processor;

a memory for storing the processor-executable instructions;

wherein the processor is configured to execute the instructions to implement the method of any of claims 1-7.
A storage medium, when instructions in the storage medium are executed by a processor of an electronic device, enabling the electronic device to perform the method according to any one of claims 1 to 7.
A computer program product comprising computer readable code which, when run on an electronic device, causes the electronic device to perform the method of any of claims 1-7.