WO2022012034A1 - Procédé et appareil de traitement d'image, dispositif électronique et support de stockage - Google Patents

Procédé et appareil de traitement d'image, dispositif électronique et support de stockage Download PDF

Info

Publication number
WO2022012034A1
WO2022012034A1 PCT/CN2021/075111 CN2021075111W WO2022012034A1 WO 2022012034 A1 WO2022012034 A1 WO 2022012034A1 CN 2021075111 W CN2021075111 W CN 2021075111W WO 2022012034 A1 WO2022012034 A1 WO 2022012034A1
Authority
WO
WIPO (PCT)
Prior art keywords
area
optimized
region
disparity
target
Prior art date
Application number
PCT/CN2021/075111
Other languages
English (en)
Chinese (zh)
Inventor
王光甫
王珏
刘帅成
Original Assignee
北京迈格威科技有限公司
成都旷视金智科技有限公司
Priority date (The priority date is an assumption and is not a legal conclusion. Google has not performed a legal analysis and makes no representation as to the accuracy of the date listed.)
Filing date
Publication date
Application filed by 北京迈格威科技有限公司, 成都旷视金智科技有限公司 filed Critical 北京迈格威科技有限公司
Publication of WO2022012034A1 publication Critical patent/WO2022012034A1/fr

Links

Images

Classifications

    • G—PHYSICS
    • G06—COMPUTING; CALCULATING OR COUNTING
    • G06T—IMAGE DATA PROCESSING OR GENERATION, IN GENERAL
    • G06T7/00—Image analysis
    • G06T7/50—Depth or shape recovery
    • G06T7/55—Depth or shape recovery from multiple images
    • G06T7/593—Depth or shape recovery from multiple images from stereo images
    • G—PHYSICS
    • G06—COMPUTING; CALCULATING OR COUNTING
    • G06N—COMPUTING ARRANGEMENTS BASED ON SPECIFIC COMPUTATIONAL MODELS
    • G06N3/00—Computing arrangements based on biological models
    • G06N3/02—Neural networks
    • G06N3/04—Architecture, e.g. interconnection topology
    • G06N3/045—Combinations of networks
    • G—PHYSICS
    • G06—COMPUTING; CALCULATING OR COUNTING
    • G06N—COMPUTING ARRANGEMENTS BASED ON SPECIFIC COMPUTATIONAL MODELS
    • G06N3/00—Computing arrangements based on biological models
    • G06N3/02—Neural networks
    • G06N3/08—Learning methods
    • G—PHYSICS
    • G06—COMPUTING; CALCULATING OR COUNTING
    • G06T—IMAGE DATA PROCESSING OR GENERATION, IN GENERAL
    • G06T5/00—Image enhancement or restoration
    • G06T5/70—Denoising; Smoothing
    • G—PHYSICS
    • G06—COMPUTING; CALCULATING OR COUNTING
    • G06T—IMAGE DATA PROCESSING OR GENERATION, IN GENERAL
    • G06T7/00—Image analysis
    • G06T7/10—Segmentation; Edge detection
    • G06T7/136—Segmentation; Edge detection involving thresholding
    • G—PHYSICS
    • G06—COMPUTING; CALCULATING OR COUNTING
    • G06T—IMAGE DATA PROCESSING OR GENERATION, IN GENERAL
    • G06T7/00—Image analysis
    • G06T7/10—Segmentation; Edge detection
    • G06T7/155—Segmentation; Edge detection involving morphological operators
    • G—PHYSICS
    • G06—COMPUTING; CALCULATING OR COUNTING
    • G06T—IMAGE DATA PROCESSING OR GENERATION, IN GENERAL
    • G06T7/00—Image analysis
    • G06T7/10—Segmentation; Edge detection
    • G06T7/187—Segmentation; Edge detection involving region growing; involving region merging; involving connected component labelling
    • G—PHYSICS
    • G06—COMPUTING; CALCULATING OR COUNTING
    • G06T—IMAGE DATA PROCESSING OR GENERATION, IN GENERAL
    • G06T2207/00—Indexing scheme for image analysis or image enhancement
    • G06T2207/20—Special algorithmic details
    • G06T2207/20024—Filtering details
    • G06T2207/20028—Bilateral filtering
    • G—PHYSICS
    • G06—COMPUTING; CALCULATING OR COUNTING
    • G06T—IMAGE DATA PROCESSING OR GENERATION, IN GENERAL
    • G06T2207/00—Indexing scheme for image analysis or image enhancement
    • G06T2207/20—Special algorithmic details
    • G06T2207/20228—Disparity calculation for image-based rendering

Definitions

  • Binocular stereo vision has been widely used in industrial inspection, aerospace, robot navigation and other fields. Binocular stereo vision recovers the 3D depth information of the scene by calculating the disparity of spatial points on two images under the same scene.
  • Stereo matching is a key link in binocular stereo vision.
  • the result of stereo matching directly affects the 3D reconstruction effect, and stereo matching depends on the disparity map.
  • the disparity map obtained by stereo matching the two images used to generate the disparity map by the stereo matching algorithm
  • the disparity map often appears in the disparity map corresponding to some areas in the image used for generating the disparity map, such as weak texture areas A situation where the disparity value in the area is inaccurate. Therefore, after the disparity map is calculated, how to accurately determine the area to be optimized in the disparity map and how to optimize the disparity value of the pixels in the area to be optimized in the disparity map becomes a problem to be solved.
  • the present application provides an image processing method, an apparatus, an electronic device and a storage medium.
  • an image processing method including:
  • Determine all areas in the disparity map for determining the area to be optimized, and all areas used for determining the area to be optimized include at least one of the following items: at least one low-confidence area, at least one area corresponding to a weak texture area , at least one area corresponding to the repeated texture area, wherein, the low confidence area is determined by the confidence matrix of the disparity map output by the stereo matching algorithm used to generate the disparity map, and the weak texture area and the repeated texture area are determined by The images used to generate the disparity map are determined by detecting weak texture regions and repeating texture regions respectively;
  • the disparity values of the pixels in the to-be-optimized area are optimized to obtain an optimized disparity map.
  • determining the region to be optimized in the disparity map includes:
  • the determined union is determined as the region to be optimized.
  • optimizing the disparity values of the pixels in the area to be optimized to obtain an optimized disparity map includes:
  • morphological gradient extraction is performed on the target connected area to determine the edge of the target connected area; based on the original disparity value of the pixels in the area occupied by the edge of the target connected area, calculate The target disparity value of each pixel in the target connected area; modify the disparity value of each pixel in the target connected area to the target disparity value of each pixel.
  • determining a target connected region among all connected regions includes:
  • a connected region whose area is greater than the area threshold in all connected regions is determined as the target connected region.
  • calculating the target disparity value of each pixel in the target connected region based on the original disparity value of the pixels in the region occupied by the edge of the target connected region includes:
  • calculating the target disparity value of each pixel in the target connected area includes:
  • the method further includes:
  • a guided filtering algorithm and a fast bilateral filtering algorithm are used to smooth the region to be smoothed in the optimized disparity map.
  • an image processing apparatus including:
  • an associated area determination unit configured to determine all areas used for determining the area to be optimized, all areas used for determining the area to be optimized include at least one of the following items: at least one low-confidence area, at least one area with a weak texture a corresponding area, at least one area corresponding to a repetitive texture area, wherein the low confidence area is determined based on the confidence matrix of the disparity map output by the stereo matching algorithm used to generate the disparity map, the weak texture area , the repeated texture area is determined by performing weak texture area detection and repeated texture area detection on the image used to generate the disparity map respectively;
  • an area to be optimized determining unit configured to determine the area to be optimized in the disparity map based on all the areas used to determine the area to be optimized
  • the disparity map optimization unit is configured to optimize the disparity values of the pixels in the to-be-optimized area to obtain an optimized disparity map.
  • the region-to-be-optimized determination unit is further configured to determine a union of all regions used to determine the region to be optimized; and the determined union is determined as the region to be optimized.
  • the disparity map optimization unit includes:
  • the connected area optimization subunit is configured to use a connected area extraction algorithm to determine all connected areas in the area to be optimized; to determine the target connected area in all connected areas; for each target connected area, perform an operation on the target connected area. Morphological gradient extraction to determine the edge of the target connected area; based on the original disparity value of the pixels in the area occupied by the edge of the target connected area, calculate the target disparity of each pixel in the target connected area value; modify the disparity value of each pixel in the target connected region to the target disparity value of each pixel.
  • the connected region optimization subunit is further configured to determine a connected region whose area is greater than an area threshold among all connected regions as a target connected region.
  • the connected region optimization subunit is further configured to calculate the target for each pixel in the target connected region based on the original disparity values of all target pixels in the region occupied by the edge of the target connected region disparity value, where the target pixel is a pixel whose original disparity value is within a preset reasonable interval.
  • the connected area optimization sub-unit is further configured to calculate the average value of the original disparity values of all target pixels in the area occupied by the edge of the target connected area; take the average value as the target connected area The target disparity value for each pixel in the region.
  • the image processing apparatus further includes:
  • a smoothing unit configured to acquire feature information of an area corresponding to the area to be smoothed in the optimized disparity map in the image used to generate the disparity map
  • a guided filtering algorithm and a fast bilateral filtering algorithm are used to smooth the region to be smoothed in the optimized disparity map.
  • the pedestrian re-identification method and device realize the consideration of the low-confidence area in the disparity map, the area corresponding to the weak texture area in the disparity map, and the repeated texture area in the disparity map.
  • the association of the area with the area to be processed, based on all areas used to determine the area to be optimized including at least one of the following: at least one low confidence area, at least one area corresponding to a weak texture area, at least one area with Repeat the area corresponding to the texture area, accurately determine the area to be optimized in the disparity map, optimize the disparity value of the pixels in the area to be optimized, so as to accurately optimize the disparity map, and obtain a process with better optimization effect. Optimized disparity map.
  • FIG. 1 shows a flowchart of an image processing method provided by an embodiment of the present application
  • FIG. 2 shows a schematic flowchart of determining the area to be optimized in the disparity map
  • 3 shows a schematic flowchart of optimizing the disparity values of pixels in the area to be optimized in the disparity map
  • FIG. 4 shows a structural block diagram of an image processing apparatus provided by an embodiment of the present application
  • FIG. 5 shows a structural block diagram of an electronic device provided by an embodiment of the present application.
  • FIG. 1 shows a flowchart of an image processing method provided by an embodiment of the present application, and the method includes:
  • Step 101 Determine all areas in the disparity map for determining the area to be optimized.
  • all regions used to determine the region to be optimized include at least one of the following items: at least one low confidence region, at least one region corresponding to a weak texture region, at least one region corresponding to a repeated texture region area.
  • the low-confidence region in the disparity map can be used as the region for determining the region to be optimized. If the image used to generate the disparity map includes a weak texture area, correspondingly, the disparity map includes an area corresponding to the weak texture area, then the area corresponding to the weak texture area in the disparity map can be used as the area for determining the area to be optimized. area. If the image used to generate the disparity map includes a repeated texture area, correspondingly, the disparity map includes an area corresponding to the repeated texture area, and the area corresponding to the repeated texture area in the disparity map can be used as the area for determining the area to be optimized. area.
  • the disparity map is generated by stereo matching the left RGB image used to generate the disparity map and the right RGB image used to generate the disparity map by a stereo matching algorithm.
  • the pixel value of each pixel in the disparity map is a disparity value. Therefore, the pixel value of the pixel in the disparity map can be referred to as the disparity value of the pixel.
  • the stereo matching algorithm may be a traditional stereo matching algorithm such as SGBM and BM, or a stereo matching algorithm based on a convolutional neural network.
  • the output of the stereo matching algorithm includes a disparity map and a confidence matrix for the disparity map.
  • the confidence matrix elements in the confidence matrix correspond one-to-one with the disparity values of the pixels in the disparity map.
  • the value of the confidence matrix element in the confidence matrix is the confidence of the disparity value of the pixel in the disparity map corresponding to the confidence matrix element.
  • the position of the disparity value of the pixel in the disparity map is the same as the position of the confidence matrix element corresponding to the pixel in the confidence matrix, and the value of the confidence matrix element corresponding to the pixel is the confidence of the disparity value of this pixel.
  • the low-confidence area in the disparity map is an area in which the confidences of the pixels included in the disparity map are all smaller than the confidence threshold.
  • the number of low confidence regions in the disparity map may be one or more.
  • All pixels in the disparity map whose confidences of disparity values are smaller than the confidence threshold may be determined according to the confidence of each pixel in the disparity map described by the confidence matrix and the confidence threshold. At least one low-confidence region in the disparity map is determined according to the positions of all pixels in the disparity map having a confidence level of a disparity value less than a confidence level threshold.
  • the weak texture area may refer to the weak texture area in the left RGB image used to generate the disparity map.
  • the repeated texture area may refer to a repeated texture area in the left RGB image used to generate the disparity map.
  • weak texture area detection may be performed on the left RGB image used for generating the disparity map to determine the weak texture area in the left RGB image used for generating the disparity map.
  • Repeat texture region detection may be performed on the left RGB image used to generate the disparity map to determine repeated texture regions in the left RGB image used to generate the disparity map.
  • the determined number of weak texture areas may be one or more, in other words, the left RGB image used for generating the disparity map may include one or more Weak textured areas.
  • the determined number of repeated texture regions may be one or more, in other words, the left RGB image used for generating the disparity map may include one or more Repeat texture area.
  • each weak texture region corresponds to one region in the disparity map.
  • each repeated texture region corresponds to one region in the disparity map.
  • Each pixel in the disparity map corresponds to a pixel in the left RGB image used to generate the disparity map.
  • all pixels in the disparity map in the area corresponding to the weak texture area are composed of pixels in the disparity map corresponding to each pixel in the weak texture area, and the disparity map is the same as the pixel in the disparity map.
  • the area corresponding to the weak texture area is composed of the area occupied by the corresponding pixel of each pixel in all the weak texture areas. Therefore, for each weak texture area, after the weak texture area is determined through weak texture area detection, the area corresponding to the weak texture area in the disparity map can be determined.
  • all pixels in the disparity map in the region corresponding to the repeated texture region are composed of pixels in the disparity map corresponding to each pixel in the repeated texture region. Therefore, for each repeated texture region, after determining the repeated texture region through repeated texture region detection, a region corresponding to the repeated texture region in the disparity map can be determined.
  • a first pixel-level detection convolutional neural network may be used to perform weak texture region detection on the image used for generating the disparity map to determine the weak texture region in the image used for generating the disparity map.
  • a second pixel-level detection convolutional neural network may be used to perform repetitive texture region detection on the image used to generate the disparity map to determine repeated texture regions in the image used to generate the disparity map.
  • the first pixel-level detection convolutional neural network and the second pixel-level detection convolutional neural network are both neural networks used for semantic segmentation.
  • the structure of the first pixel-level detection convolutional neural network and the second pixel-level detection convolutional neural network same.
  • the first pixel-level detection convolutional neural network may be used to detect weak texture regions on the left RGB image used to generate the disparity map, so as to determine weak texture areas in the left RGB image used to generate the disparity map texture area.
  • the first pixel-level detection convolutional neural network is pre-trained with the training images. Each weakly textured region in the training image utilized to train the first pixel-level detection convolutional neural network is annotated.
  • the left RGB image used to generate the disparity map is input to the first pixel-level detection convolutional neural network, and the detection result output by the first pixel-level detection convolutional neural network may be an indication of the left RGB used to generate the disparity map. Whether each pixel in the image belongs to a weakly textured region of the image. Thus, according to the detection result, at least one weak texture area can be determined.
  • a second pixel-level detection convolutional neural network can be used to perform repetitive texture region detection on the left RGB image used to generate the disparity map to determine the repetition in the left RGB image used to generate the disparity map texture area.
  • the second pixel-level detection convolutional neural network is pre-trained with training. Repeated texture regions in the training image used to train the second pixel-level detection convolutional neural network are annotated.
  • the left RGB image used to generate the disparity map is input to the second pixel-level detection convolutional neural network, and the detection result output by the second pixel-level detection convolutional neural network may be an indication of the left RGB used to generate the disparity map Whether each pixel in the image belongs to an image with a repeating texture area. Thus, based on the detection result, at least one repeating texture region can be determined.
  • Step 102 Determine the area to be optimized in the disparity map based on all the areas used to determine the area to be optimized.
  • the independent area when determining the area to be optimized in the disparity map based on all the areas used to determine the area to be optimized, for each independent area in all the areas used to determine the area to be optimized, the independent area can be used as A subregion of the region to be optimized in the disparity map.
  • each area corresponding to the weak texture area corresponds to one weak texture area.
  • each region corresponding to the repeated texture region corresponds to one repeated texture region.
  • the independent area may refer to all the areas used for determining the area to be optimized that do not have overlapping parts with any other area used for determining the area to be optimized.
  • the non-independent regions may refer to all regions used for determining the regions to be optimized that have overlapping portions with at least one other region used for determining the regions to be optimized.
  • each area in all the areas used to determine the area to be optimized is an independent area, and the area to be optimized in the disparity map is determined by All areas used for determining the area to be optimized are composed, that is, the area to be optimized in the disparity map includes all areas used for determining the area to be optimized.
  • all the regions used to determine the region to be optimized include: at least one region of low confidence, at least one region corresponding to a weak texture region, and at least one region corresponding to a repeated texture region.
  • each low-confidence region is an independent region
  • each region in the disparity map corresponding to the weak texture region is The independent area
  • each area corresponding to the repeated texture area in the disparity map is an independent area
  • the area to be optimized in the disparity map is composed of each low-confidence area in the disparity map, each in the disparity map and weak.
  • the area corresponding to the texture area and each area in the disparity map corresponding to the repeated texture area are composed.
  • each overlapping portion belonging to the at least two dependent regions at the same time can be regarded as a sub-region of the region to be optimized in the disparity map.
  • the region to be optimized in the disparity map may be composed of each independent region, each of which simultaneously belongs to the overlapping portion of the at least two dependent regions.
  • the to-be-optimized area in the disparity map may include: each independent area, each overlapping part belonging to at least two dependent areas at the same time.
  • determining the region to be optimized in the disparity map includes: determining a union of all regions used for determining the region to be optimized; combining the determined union The set is determined as the region to be optimized in the disparity map.
  • the union of all the regions used to determine the region to be optimized is all the regions used to determine the region to be optimized.
  • the union of the regions used to determine the region to be optimized is taken as the region to be optimized, in other words, the region to be optimized consists of all the regions used to determine the region to be optimized.
  • all the regions used to determine the region to be optimized include: at least one region of low confidence, at least one region corresponding to a weak texture region, and at least one region corresponding to a repeated texture region.
  • the union of all the regions used to determine the region to be optimized is determined by each low-confidence region in the disparity map, the disparity map
  • Each of the regions corresponding to the weak texture region and each region corresponding to the repeated texture region in the disparity map is composed of, then the region to be optimized in the disparity map includes: each low-confidence region in the disparity map, the parallax Each region in the map corresponds to a weak texture region, and each region in the disparity map corresponds to a repeated texture region.
  • the union of all the regions used for determining the region to be optimized is determined by each independent region, each belonging to at least two dependent regions at the same time.
  • the areas to be optimized in the disparity map include: each independent area, each overlapping part belonging to at least two dependent areas at the same time, and each part belonging to only one dependent area and not the overlapping part.
  • FIG. 2 shows a schematic flowchart of determining an area to be optimized.
  • the area to be optimized may also be referred to as a parallax inaccuracy area.
  • the low confidence region in the disparity map can be determined according to the confidence matrix and the confidence threshold.
  • a low-confidence mask can be generated first.
  • Each pixel in the low-confidence mask corresponds to a pixel in the disparity map.
  • the pixel value of the pixel is a value of 1 indicating that the corresponding pixel belongs to the low-confidence region.
  • the pixel value of the pixel is a value of 0 indicating that its corresponding pixel does not belong to the low-confidence region.
  • the pixel value of the pixel is Modify it to 1 to get the mask image of the region to be optimized.
  • the to-be-optimized area mask image may be referred to as the to-be-optimized area mask.
  • Each pixel in the area mask to be optimized corresponds to one pixel in the disparity map.
  • the pixel value of the pixel corresponding to the pixel in the area to be optimized is 1 indicating that the corresponding pixel belongs to the area to be optimized, and the pixel value of the pixel corresponding to the pixel in the area not to be optimized It is a value of 0 indicating that its corresponding pixel does not belong to the area to be optimized.
  • All the pixels in the to-be-optimized area in the disparity map are composed of pixels corresponding to each pixel with a pixel value of 1 in each of the to-be-optimized area masks. Therefore, after obtaining the area mask to be optimized, the area to be optimized in the disparity map can be determined.
  • Step 103 Optimize the disparity values of the pixels in the area to be optimized to obtain an optimized disparity map.
  • the disparity values of the pixels in the area to be optimized are optimized.
  • Each pixel in the disparity map has an original disparity value.
  • the target disparity value of the pixel can be determined, and the disparity value of the pixel can be modified to the target disparity value. Therefore, the disparity value of the pixel is changed from the original disparity value of the pixel to the target disparity value of the pixel.
  • the target disparity value of the pixel For each pixel in the area to be optimized, when determining the target disparity value of the pixel, for each pixel in the area to be optimized, you can search for an effective pixel in the disparity map that is closest to the pixel and does not belong to the area to be optimized. For disparity pixels, the disparity value of the found effective disparity pixel can be used as the target pixel value of the pixel.
  • the method further includes: acquiring feature information of a region corresponding to the region to be smoothed in the optimized disparity map in the image used to generate the disparity map; based on the feature information, using a guided filtering algorithm and a fast bilateral The filtering algorithm smoothes the area to be smoothed in the optimized disparity map.
  • the area corresponding to the area to be smoothed in the optimized disparity map may be the area occupied by the subject in the left RGB image used to generate the disparity map.
  • the area corresponding to the to-be-smoothed area in the optimized disparity map may occupy the portrait area for the portrait in the left RGB image used to generate the disparity map.
  • the region to be smoothed in the optimized disparity map corresponds to the portrait region in the left RGB image for generating the disparity map.
  • the feature information of the area corresponding to the area to be smoothed in the optimized disparity map in the image used for generating the disparity map is information based on smoothing the area to be smoothed in the optimized disparity map.
  • the area corresponding to the area to be smoothed in the optimized disparity map is the portrait area in the left RGB image used to generate the disparity map, and the feature information may be the left RGB image used to generate the disparity map.
  • the gradient of each pixel in the portrait region is the portrait area in the left RGB image used to generate the disparity map.
  • the guided filtering algorithm and the fast bilateral filtering algorithm can be used to determine the region to be smoothed in the optimized disparity map. for smoothing. Therefore, the optimized disparity map is smoother, especially in the edge region, the change is not abrupt and the consistency is maintained.
  • optimizing the disparity values of pixels in the region to be optimized to obtain an optimized disparity map includes: determining all connected regions in the region to be optimized by using a connected region extraction algorithm; determining target connectivity in all connected regions area; for each target connected area, perform morphological gradient extraction on the target connected area to determine the edge of the target connected area; based on the original disparity value of the pixels in the area occupied by the edge of the target connected area, calculate the The target disparity value of each pixel in the target connected area; modify the disparity value of each pixel in the target connected area to the target disparity value of each pixel.
  • the connected area extraction algorithm can be used to determine each connected area in the mask area to be optimized based on the mask image of the area to be optimized corresponding to the disparity map, that is, the area mask to be optimized. a connected region.
  • the pixel value of each pixel in the area to be optimized mask image corresponding to each pixel in the area to be optimized is 1, and the area to be optimized in the disparity map can be quickly located through the mask image of the area to be optimized.
  • a connected region extraction algorithm may be used to extract a connected region in the region to be optimized in the disparity map, and each connected region in the region to be optimized in the disparity map is determined.
  • a target connected region in the region to be optimized can be determined.
  • each connected area can be individually used as a target connected area.
  • determining a target connected area in all connected areas includes: determining a connected area whose area is greater than an area threshold in all connected areas as a target connected area.
  • the connected regions with smaller areas can be eliminated, and the eliminated connected regions can be called ineffective regions.
  • morphological gradient extraction can be performed on the target connected region to determine the edge of the target connected region; based on the target connected region The original disparity value of the pixels in the area occupied by the edge of , calculate the target disparity value of each pixel in the target connected area; modify the disparity value of each pixel in the target connected area to Target disparity value.
  • morphological gradient extraction is performed on the target connected region to calculate the difference between the expansion map of the target connected region and the erosion map of the connected region to determine the edge of the target connected region.
  • the target of each pixel in the target connected region can be calculated based on the original disparity value of the pixels in the region occupied by the edge of the target connected region Parallax value.
  • the median of the original disparity values of all pixels in the region occupied by the edge of the target connected region or the average of all pixels in the region occupied by the edge of the target connected region can be determined value, the median or the average value is used as the target disparity value of each pixel in the target connected area, and the target disparity value of each pixel in the target connected area is the median or the average value value.
  • the disparity value of each pixel in the target connected area For each target connected area, after calculating the target disparity value of each pixel in the target connected area, you can, for each target connected area, modify the disparity value of each pixel in the target connected area to each The target disparity value of one pixel, that is, the disparity value filling is performed for each target connected area respectively, and the target disparity value of each pixel in the target connected area is filled to the position of the pixel. For each pixel in any target connected region, the disparity value of the pixel changes from the original disparity value to the target disparity value.
  • the disparity value of the pixels in each target connected region is optimized to obtain an optimized disparity map.
  • calculating the target disparity value of each pixel in the target connected region includes: based on The original disparity values of all target pixels in the area occupied by the edge of the target connected area are calculated, and the target disparity value of each pixel in the target connected area is calculated, wherein the original disparity value of the target pixel is in the preset value Pixels within a reasonable range.
  • the target disparity value of each pixel in the connected region can be calculated based on the original disparity values of all target pixels in the region occupied by the edge of the target connected region.
  • the median of the original disparity values of all target pixels in the region occupied by the edge of the target connected region can be determined, and the median is taken as the value of each pixel in the target connected region.
  • the target disparity value, the target disparity value of each pixel in the target connected region is the median.
  • calculating the target disparity value of each pixel in the target connected region based on the original disparity values of all target pixels in the region occupied by the edge of the target connected region includes: : Calculate the average value of the original disparity values of all target pixels in the area occupied by the edge of the target connected area; use the average value as the target disparity value of each pixel in the target connected area.
  • the average value of the original disparity values of all target pixels in the area occupied by the edge of the target connected area can be used as the target disparity of each pixel in the target connected area
  • the target disparity value of each pixel in the target connected region is the average value of the original disparity values of all target pixels in the region occupied by the edge of the target connected region.
  • the disparity value of each pixel in the target connected region is determined by the original
  • the disparity value becomes the average value of the original disparity values of all target pixels in the area occupied by the edge of the target connected area.
  • FIG. 3 shows a schematic flowchart of optimizing the disparity values of pixels in the area to be optimized in the disparity map.
  • Optimizing the disparity values of pixels in the area to be optimized in the disparity map may also be referred to as a local compensation algorithm.
  • a connected region extraction algorithm is used to determine each connected region in the region to be optimized based on the mask image of the region to be optimized corresponding to the disparity map, that is, the region mask to be optimized.
  • the pixel value of each pixel in the to-be-optimized area mask corresponding to each pixel in the to-be-optimized area mask is 1, and the to-be-optimized area mask can be used to quickly locate the to-be-optimized area.
  • the connected region extraction algorithm can be used to extract the connected region of the region to be optimized, and each connected region in the region to be optimized can be determined.
  • the connected regions whose area is larger than the area threshold are determined as the target connected regions, so that the connected regions with smaller areas are eliminated, that is, the ineffective regions.
  • morphological gradient extraction is performed on the target connected region to determine the edge of the target connected region.
  • the average value of the original disparity values of all target pixels in the region occupied by the edge of the target connected region can be used as the target disparity value of each pixel in the target connected region.
  • Disparity filling is performed on each target connected area, that is, for each target connected area, the target disparity value of each pixel in the target connected area is filled to the position of the pixel.
  • the disparity value of each pixel in the target connected area is changed from the original disparity value to the average value of the original disparity values of all target pixels in the area occupied by the edge of the target connected area.
  • FIG. 4 shows a structural block diagram of an image processing apparatus provided by an embodiment of the present application.
  • the image processing apparatus includes: an associated area determination unit 401 , a to-be-optimized area determination unit 402 , and a disparity map optimization unit 403 .
  • an associated area determination unit configured to determine all areas in the disparity map for determining the area to be optimized, all areas used for determining the area to be optimized include at least one of the following items: at least one low-confidence area, at least one regions corresponding to regions of weak texture, at least one region corresponding to regions of repeated texture, wherein regions of low confidence are determined based on a confidence matrix of the disparity map output by a stereo matching algorithm used to generate the disparity map , the weak texture area and the repeated texture area are determined by respectively performing weak texture area detection and repeated texture area detection on the image used to generate the disparity map;
  • an area to be optimized determining unit configured to determine the area to be optimized in the disparity map based on all the areas used to determine the area to be optimized
  • the disparity map optimization unit is configured to optimize the disparity values of the pixels in the to-be-optimized area to obtain an optimized disparity map.
  • the region-to-be-optimized determination unit is further configured to determine a union of all regions used to determine the region to be optimized; and determine the determined union as the region to be optimized in the disparity map.
  • the disparity map optimization unit includes:
  • the connected area optimization subunit is configured to use a connected area extraction algorithm to determine all connected areas in the area to be optimized; to determine the target connected area in all connected areas; for each target connected area, perform an operation on the target connected area. Morphological gradient extraction to determine the edge of the target connected area; based on the original disparity value of the pixels in the area occupied by the edge of the target connected area, calculate the target disparity of each pixel in the target connected area value; modify the disparity value of each pixel in the target connected region to the target disparity value of each pixel.
  • the connected region optimization subunit is further configured to determine a connected region whose area is greater than an area threshold among all connected regions as a target connected region.
  • the connected region optimization subunit is further configured to calculate a target for each pixel in the target connected region based on the original disparity values of all target pixels in the region occupied by the edge of the target connected region disparity value, where the target pixel is a pixel whose original disparity value is within a preset reasonable interval.
  • the connected area optimization sub-unit is further configured to calculate the average value of the original disparity values of all target pixels in the area occupied by the edge of the target connected area; take the average value as the target connected area The target disparity value for each pixel in the region.
  • the image processing apparatus further includes:
  • a smoothing unit configured to acquire feature information of an area corresponding to the area to be smoothed in the optimized disparity map in the image used to generate the disparity map
  • a guided filtering algorithm and a fast bilateral filtering algorithm are used to smooth the region to be smoothed in the optimized disparity map.
  • FIG. 5 is a structural block diagram of an electronic device provided in this embodiment.
  • the electronic device includes a processing component 522, which further includes one or more processors, and a memory resource, represented by memory 532, for storing instructions, such as application programs, executable by the processing component 522.
  • An application program stored in memory 532 may include one or more modules, each corresponding to a set of instructions.
  • the processing component 522 is configured to execute instructions to perform the above-described methods.
  • the electronic device may also include a power supply assembly 526 configured to perform power management of the electronic device, a wired or wireless network interface 550 configured to connect the electronic device to a network, and an input output (I/O) interface 558.
  • the electronic device may operate based on an operating system stored in memory 532, such as Windows ServerTM, MacOS XTM, UnixTM, LinuxTM, FreeBSDTM or the like.
  • a storage medium including instructions such as a memory including instructions, is also provided, and the instructions are executable by an electronic device to perform the above method.
  • the storage medium may be a non-transitory computer-readable storage medium, for example, the non-transitory computer-readable storage medium may be ROM, random access memory (RAM), CD-ROM, magnetic tape, floppy disk, and optical data storage equipment, etc.
  • references herein to "one embodiment,” “an embodiment,” or “one or more embodiments” means that a particular feature, structure, or characteristic described in connection with an embodiment is included in at least one embodiment of the present invention. Also, please note that instances of the phrase “in one embodiment” herein are not necessarily all referring to the same embodiment.
  • any reference signs placed between parentheses shall not be construed as limiting the claim.
  • the word “comprising” does not exclude the presence of elements or steps not listed in a claim.
  • the word “a” or “an” preceding an element does not exclude the presence of a plurality of such elements.
  • the invention can be implemented by means of hardware comprising several different elements and by means of a suitably programmed computer. In a unit claim enumerating several means, several of these means may be embodied by one and the same item of hardware.
  • the use of the words first, second, and third, etc. do not denote any order. These words can be interpreted as names.

Landscapes

  • Engineering & Computer Science (AREA)
  • Physics & Mathematics (AREA)
  • Theoretical Computer Science (AREA)
  • General Physics & Mathematics (AREA)
  • Computer Vision & Pattern Recognition (AREA)
  • General Health & Medical Sciences (AREA)
  • Computing Systems (AREA)
  • Computational Linguistics (AREA)
  • Data Mining & Analysis (AREA)
  • Evolutionary Computation (AREA)
  • Biomedical Technology (AREA)
  • Molecular Biology (AREA)
  • Biophysics (AREA)
  • General Engineering & Computer Science (AREA)
  • Artificial Intelligence (AREA)
  • Mathematical Physics (AREA)
  • Software Systems (AREA)
  • Life Sciences & Earth Sciences (AREA)
  • Health & Medical Sciences (AREA)
  • Image Processing (AREA)

Abstract

Selon des modes de réalisation, la présente invention concerne un procédé et un appareil de traitement d'image, un dispositif électronique et un support de stockage. Le procédé comprend : la détermination de l'ensemble des régions utilisées pour déterminer une région à optimiser, l'ensemble des régions utilisées pour déterminer une région à optimiser comprenant au moins une des catégories suivantes : au moins une région à faible confiance, au moins une région correspondant à une région peu texturée, et au moins une région correspondant à une région à texture répétitive ; la détermination, sur la base de l'ensemble des régions utilisées pour déterminer une région à optimiser, d'une région à optimiser dans une carte de disparité ; et l'optimisation de valeurs de disparité de pixels présents dans la région à optimiser pour obtenir une carte de disparité optimisée. Une région à optimiser dans une carte de disparité est déterminée avec précision, et des valeurs de disparité de pixels présents dans la région à optimiser dans la carte de disparité sont optimisées pour optimiser précisément la carte de disparité, ce qui permet d'obtenir une carte de disparité optimisée ayant un meilleur effet d'optimisation.
PCT/CN2021/075111 2020-07-14 2021-02-03 Procédé et appareil de traitement d'image, dispositif électronique et support de stockage WO2022012034A1 (fr)

Applications Claiming Priority (2)

Application Number Priority Date Filing Date Title
CN202010676768.6A CN112053394A (zh) 2020-07-14 2020-07-14 图像处理方法、装置、电子设备及存储介质
CN202010676768.6 2020-07-14

Publications (1)

Publication Number Publication Date
WO2022012034A1 true WO2022012034A1 (fr) 2022-01-20

Family

ID=73602728

Family Applications (1)

Application Number Title Priority Date Filing Date
PCT/CN2021/075111 WO2022012034A1 (fr) 2020-07-14 2021-02-03 Procédé et appareil de traitement d'image, dispositif électronique et support de stockage

Country Status (2)

Country Link
CN (1) CN112053394A (fr)
WO (1) WO2022012034A1 (fr)

Families Citing this family (2)

* Cited by examiner, † Cited by third party
Publication number Priority date Publication date Assignee Title
CN112053394A (zh) * 2020-07-14 2020-12-08 北京迈格威科技有限公司 图像处理方法、装置、电子设备及存储介质
CN116563186A (zh) * 2023-05-12 2023-08-08 中山大学 一种基于专用ai感知芯片的实时全景感知系统及方法

Citations (5)

* Cited by examiner, † Cited by third party
Publication number Priority date Publication date Assignee Title
US7561732B1 (en) * 2005-02-04 2009-07-14 Hrl Laboratories, Llc Method and apparatus for three-dimensional shape estimation using constrained disparity propagation
CN103226821A (zh) * 2013-04-27 2013-07-31 山西大学 基于视差图像素分类校正优化的立体匹配方法
CN110443186A (zh) * 2019-07-31 2019-11-12 深圳市道通智能航空技术有限公司 立体匹配方法、图像处理芯片及移动载具
CN111402152A (zh) * 2020-03-10 2020-07-10 北京迈格威科技有限公司 视差图的处理方法、装置、计算机设备和存储介质
CN112053394A (zh) * 2020-07-14 2020-12-08 北京迈格威科技有限公司 图像处理方法、装置、电子设备及存储介质

Family Cites Families (4)

* Cited by examiner, † Cited by third party
Publication number Priority date Publication date Assignee Title
US8666144B2 (en) * 2010-09-02 2014-03-04 Edge 3 Technologies, Inc. Method and apparatus for determining disparity of texture
CN104915927B (zh) * 2014-03-11 2018-08-07 株式会社理光 视差图像优化方法及装置
US10424075B2 (en) * 2015-05-06 2019-09-24 Peking University Shenzhen Graduate School Depth/disparity map post-processing method and device
CN108596975B (zh) * 2018-04-25 2022-03-29 华南理工大学 一种针对弱纹理区域的立体匹配算法

Patent Citations (5)

* Cited by examiner, † Cited by third party
Publication number Priority date Publication date Assignee Title
US7561732B1 (en) * 2005-02-04 2009-07-14 Hrl Laboratories, Llc Method and apparatus for three-dimensional shape estimation using constrained disparity propagation
CN103226821A (zh) * 2013-04-27 2013-07-31 山西大学 基于视差图像素分类校正优化的立体匹配方法
CN110443186A (zh) * 2019-07-31 2019-11-12 深圳市道通智能航空技术有限公司 立体匹配方法、图像处理芯片及移动载具
CN111402152A (zh) * 2020-03-10 2020-07-10 北京迈格威科技有限公司 视差图的处理方法、装置、计算机设备和存储介质
CN112053394A (zh) * 2020-07-14 2020-12-08 北京迈格威科技有限公司 图像处理方法、装置、电子设备及存储介质

Also Published As

Publication number Publication date
CN112053394A (zh) 2020-12-08

Similar Documents

Publication Publication Date Title
Yao et al. Curvature aided Hough transform for circle detection
US9767567B2 (en) Method and apparatus for separating foreground image, and non-transitory computer-readable recording medium
Feng et al. Local background enclosure for RGB-D salient object detection
CN108446694B (zh) 一种目标检测方法及装置
US11748894B2 (en) Video stabilization method and apparatus and non-transitory computer-readable medium
WO2022012034A1 (fr) Procédé et appareil de traitement d'image, dispositif électronique et support de stockage
CN110349086B (zh) 一种非同心成像条件的图像拼接方法
CN111340749B (zh) 图像质量的检测方法、装置、设备及存储介质
JP2011138500A (ja) ステレオ映像内の視差探索範囲を求めるための方法及びシステム
CN111160291B (zh) 基于深度信息与cnn的人眼检测方法
CN114463603B (zh) 图像检测模型的训练方法、装置、电子设备及存储介质
KR20140048449A (ko) 의료영상에서의 간 영역 검출방법
EP2536123A1 (fr) Appareil et procédé de traitement dýimages
US11461597B2 (en) Object likelihood estimation device, method, and program
Srikakulapu et al. Depth estimation from single image using defocus and texture cues
CN116758280A (zh) 目标检测方法、装置、设备以及存储介质
Cai et al. Hole-filling approach based on convolutional neural network for depth image-based rendering view synthesis
CN112967305B (zh) 一种复杂天空场景下的图像云背景检测方法
CN110599456A (zh) 一种医学图像的特定区域的提取方法
WO2018223370A1 (fr) Procédé et système de test de relief vidéo basé sur des contraintes temporelles et spatiales
JP2020144686A (ja) モデル作成システム、モデル作成方法
CN104751431A (zh) 一种基于图像处理的方法和装置
JPH05256613A (ja) ステレオ画像からの視差演算方法と装置並びに奥行き測定装置
KR102389295B1 (ko) 인스턴스 분할 기반 단일 실내 영상으로부터의 물체 크기 추정 방법 및 장치
US11475614B2 (en) Image processing apparatus and non-transitory computer readable medium for calculating a position at which to superimpose a design object onto an image

Legal Events

Date Code Title Description
121 Ep: the epo has been informed by wipo that ep was designated in this application

Ref document number: 21842022

Country of ref document: EP

Kind code of ref document: A1

NENP Non-entry into the national phase

Ref country code: DE

32PN Ep: public notification in the ep bulletin as address of the adressee cannot be established

Free format text: NOTING OF LOSS OF RIGHTS PURSUANT TO RULE 112(1) EPC (EPO FORM 1205 DATED 16.05.2023)

122 Ep: pct application non-entry in european phase

Ref document number: 21842022

Country of ref document: EP

Kind code of ref document: A1