CN114359123A - Image processing method and device - Google Patents
Image processing method and device Download PDFInfo
- Publication number
- CN114359123A CN114359123A CN202210033560.1A CN202210033560A CN114359123A CN 114359123 A CN114359123 A CN 114359123A CN 202210033560 A CN202210033560 A CN 202210033560A CN 114359123 A CN114359123 A CN 114359123A
- Authority
- CN
- China
- Prior art keywords
- image
- foreground
- target
- depth map
- area
- Prior art date
- Legal status (The legal status is an assumption and is not a legal conclusion. Google has not performed a legal analysis and makes no representation as to the accuracy of the status listed.)
- Granted
Links
- 238000003672 processing method Methods 0.000 title claims abstract description 18
- 238000005457 optimization Methods 0.000 claims abstract description 31
- 238000000034 method Methods 0.000 claims abstract description 24
- 230000004927 fusion Effects 0.000 claims abstract description 14
- 238000012545 processing Methods 0.000 claims description 33
- 238000003708 edge detection Methods 0.000 claims description 24
- 238000004590 computer program Methods 0.000 claims description 18
- 238000001914 filtration Methods 0.000 claims description 12
- 238000003860 storage Methods 0.000 claims description 8
- 238000013507 mapping Methods 0.000 claims description 6
- 238000004891 communication Methods 0.000 claims description 4
- 238000010586 diagram Methods 0.000 description 17
- 238000004422 calculation algorithm Methods 0.000 description 11
- 238000004364 calculation method Methods 0.000 description 8
- 230000008569 process Effects 0.000 description 7
- 239000011159 matrix material Substances 0.000 description 6
- 230000006870 function Effects 0.000 description 5
- 230000009471 action Effects 0.000 description 3
- 230000000694 effects Effects 0.000 description 3
- 239000013598 vector Substances 0.000 description 3
- 238000012986 modification Methods 0.000 description 2
- 230000004048 modification Effects 0.000 description 2
- 230000003287 optical effect Effects 0.000 description 2
- 230000004075 alteration Effects 0.000 description 1
- 238000013459 approach Methods 0.000 description 1
- 230000008859 change Effects 0.000 description 1
- 238000005520 cutting process Methods 0.000 description 1
- 238000001514 detection method Methods 0.000 description 1
- 238000005516 engineering process Methods 0.000 description 1
- 238000000605 extraction Methods 0.000 description 1
- 238000004519 manufacturing process Methods 0.000 description 1
- 230000000750 progressive effect Effects 0.000 description 1
- 230000011218 segmentation Effects 0.000 description 1
- 238000000926 separation method Methods 0.000 description 1
- 230000009466 transformation Effects 0.000 description 1
- 230000000007 visual effect Effects 0.000 description 1
Images
Landscapes
- Image Processing (AREA)
- Image Analysis (AREA)
Abstract
The embodiment of the invention provides an image processing method and device, wherein the method comprises the following steps: acquiring an original depth map and a gray scale map; generating a fitting plane according to the original depth map; generating a fusion image according to the original depth map and the gray level map; determining a foreground region in the original depth map based on the fitted plane and the fused image; performing resolution optimization on the foreground area to obtain a target foreground; and generating a target depth map based on the target foreground and the fitting plane. The embodiment of the invention can accurately detect the foreground region in the depth map, optimize the resolution of the foreground region and improve the definition of the foreground region.
Description
Technical Field
The present invention relates to the field of image technologies, and in particular, to an image processing method and an image processing apparatus.
Background
Stereoscopic vision has important application in a plurality of fields of foresight, such as mapping, three-dimensional dense reconstruction, automatic driving, visual navigation and the like, and the generation of a depth image is always a very key factor in the field of stereoscopic vision. The accuracy of the depth map output directly affects the overall stability and usability of the stereoscopic vision system.
Usually, the depth map is generated by a stereo matching and disparity algorithm, and the current stereo matching algorithm is relatively mature. Because the stereo matching algorithm is more computationally expensive, in a real-time processing system, the stereo matching algorithm is usually integrated inside the camera device (i.e., the device directly outputs the depth map), and the stereo matching algorithm is not required to be implemented at the back end of the user. Because the device manufacturer burns the stereo matching algorithm on the chip of the device, the user cannot optimize the algorithm and adjust the algorithm parameters according to the specific use scene unless the manufacturer supports the user customization. However, the customization has the problems of high customization cost and long time period.
Disclosure of Invention
In view of the above, embodiments of the present invention are proposed to provide an image processing method and a corresponding image processing apparatus that overcome or at least partially solve the above-mentioned problems.
In order to solve the above problem, an embodiment of the present invention discloses an image processing method, including:
acquiring an original depth map and a gray scale map;
generating a fitting plane according to the original depth map;
generating a fusion image according to the original depth map and the gray level map;
determining a foreground region in the original depth map based on the fitted plane and the fused image;
performing resolution optimization on the foreground area to obtain a target foreground;
and generating a target depth map based on the target foreground and the fitting plane.
Optionally, the step of generating a fused image according to the original depth map and the grayscale map includes:
carrying out edge detection on the gray-scale image to obtain an edge detection result; the edge detection result comprises a first pixel value;
determining a second pixel value in the depth map corresponding to the position of the first pixel value;
fusing the first pixel value and the second pixel value according to a preset linear relation to obtain a third pixel value;
generating a fused image for the third pixel value.
Optionally, the step of determining a foreground region in the original depth map based on the fitting plane and the fused image includes:
generating a mask image for the fused image; the mask image is composed of fourth pixel values;
determining a boundary length corresponding to the fourth pixel value;
determining at least one region to be distinguished in the original depth map except for the position of the fitting plane;
determining the depth value of a pixel point in the area to be distinguished;
obtaining a foreground reference value of the area to be distinguished according to the depth value and the boundary length;
obtaining the number of edges for the original depth map;
determining a foreground region based on the foreground reference value and the number of edges.
Optionally, the step of determining a foreground region based on the foreground reference value and the number of edges includes:
if the foreground reference value of the area to be distinguished is not larger than a first preset threshold value and the number of edges corresponding to the area to be distinguished is not smaller than a first preset number, determining the area to be distinguished as a foreground area;
and if the foreground reference value of the area to be distinguished is greater than a first preset threshold value and the number of edges corresponding to the area to be distinguished is less than a first preset number, determining that the area to be distinguished is a background area.
Optionally, the step of performing resolution optimization on the foreground region to obtain a target foreground includes:
adopting preset active contour information to identify a target virtual object in the foreground area;
and carrying out resolution optimization on the target virtual object to obtain a target foreground.
Optionally, the step of performing resolution optimization on the target virtual object to obtain a target foreground includes:
acquiring an external sub-image matched with the target virtual object in the original depth map;
acquiring an edge mask matched with the external sub-image in the gray-scale image;
carrying out interpolation processing on the external sub-image to obtain an interpolated image;
overlapping the interpolated image and the edge mask to obtain an overlapped image;
performing edge filtering on the superposed image to obtain a filtered image containing a connected region;
performing depth value restoration processing on the filtered image to obtain a target sub-image;
and generating a target foreground based on the target sub-image.
Optionally, the performing depth value restoration processing on the filtered image to obtain a target sub-image includes:
carrying out mean value filtering on two sides of the edge of the communication area to obtain a mean value image;
generating three-dimensional point cloud data corresponding to the mean value image;
fitting the three-dimensional point cloud data to obtain a virtual plane;
performing inverse mapping on the virtual plane to obtain an image with gradually changed depth values;
and performing downsampling processing on the depth value gradient image to generate a target sub-image.
The embodiment of the invention also discloses an image processing device, which comprises:
the image acquisition module is used for acquiring an original depth map and a gray scale map;
a fitting plane generating module for generating a fitting plane according to the original depth map;
the fusion module is used for generating a fusion image according to the original depth map and the gray level map;
a foreground region determination module, configured to determine a foreground region in the original depth map based on the fitting plane and the fused image;
the target foreground generation module is used for carrying out resolution optimization on the foreground area to obtain a target foreground;
and the target depth map generating module is used for generating a target depth map based on the target foreground and the fitting plane.
The embodiment of the invention also discloses an electronic device, which comprises: a processor, a memory and a computer program stored on the memory and executable on the processor, which computer program, when executed by the processor, carries out the steps of the image processing method as described above.
The embodiment of the invention also discloses a computer readable storage medium, wherein a computer program is stored on the computer readable storage medium, and when the computer program is executed by a processor, the steps of the image processing method are realized.
The embodiment of the invention has the following advantages:
after a gray scale image and an original depth image output by a binocular camera are obtained, a fitting plane is extracted according to the original depth image, the gray scale image and the original depth image which are subjected to edge detection processing are fused to obtain a fused image, a foreground region in the original depth image is detected based on the fitting plane and the fused image, resolution optimization is carried out on the foreground region to obtain a target foreground of which the definition is improved, and the target foreground and the fitting plane are overlapped to obtain a target depth image which is used for carrying out local resolution optimization on the target foreground of interest.
Drawings
FIG. 1 is a flow chart of the steps of an embodiment of an image processing method of the present invention;
FIG. 2 is a gray scale diagram provided by an embodiment of the present invention;
FIG. 3 is a schematic diagram of the structure of FIG. 2;
FIG. 4 is a graph illustrating edge results of a gray scale map provided by the present invention;
FIG. 5 is a circumscribed rectangular word diagram provided by the present invention;
FIG. 6 is a flowchart of an exemplary image processing of the present invention;
fig. 7 is a block diagram of an image processing apparatus according to an embodiment of the present invention.
Detailed Description
In order to make the aforementioned objects, features and advantages of the present invention comprehensible, embodiments accompanied with figures are described in further detail below.
One of the core ideas of the embodiment of the invention is that after a gray level image and an original depth image output by a binocular camera are obtained, a fitting plane is extracted according to the original depth image, the gray level image and the original depth image which are subjected to edge detection processing are fused to obtain a fused image, a foreground region in the original depth image is detected based on the fitting plane and the fused image, resolution optimization is carried out on the foreground region to obtain a target foreground with improved definition, and the target foreground and the fitting plane are overlapped to obtain a target depth image which is used for carrying out local resolution optimization on the target foreground of interest, so that global-range global optimization is avoided, and the optimization is more accurate and obvious than global optimization.
Referring to fig. 1, a flowchart illustrating steps of an embodiment of an image processing method according to the present invention is shown, which may specifically include the following steps:
the embodiment of the invention can be applied to a movable body capable of flying, the movable body is provided with a binocular camera, when the movable body is positioned in the air, images can be collected aiming at the ground, and an original depth map and a gray scale map aiming at the ground are output.
The original depth map is an image processed by a preset binocular stereo matching algorithm. The invention does not limit the specific binocular stereo matching algorithm method.
102, generating a fitting plane according to the original depth map;
firstly, converting the depth map into a 3D point cloud through formulas (1) to (2):
wherein f is the focal length of the lens, dx and dy are the internal parameters of the camera and represent the physical length corresponding to a single pixel on the x/y axis of the image plane, R and T represent the external parameters of the camera, and the default is an identity matrix (u)0，v0) Zc is the preset optical axis of the camera for the position of the camera in the pixel coordinates. By combining the calculation formula (1) and the calculation formula (2), the image point [ u, v ] can be calculated]TTo world coordinate point [ x ]w,yw,zw]T。
Firstly, selecting a subset from a three-dimensional point set corresponding to a depth map, and then performing plane fitting on the 3D point cloud subset, wherein a plane equation is as the following calculation formula (3):
ax+by+cz＝d(d≥0),2+2+2＝1
(i,yii) is a set of points, i ═ 1,2,3 … … n
The fitting plane is a least squares problem, as shown in the following equation (4), diDistance to plane is represented:
converting the least square calculation formula (4) into an extreme value calculation to obtain a calculation formula (5):
calculating the partial derivatives of d, a, b and c in the formula (5) to obtain a formula (6):
in the calculation formula (6), the left brace is a covariance matrix of n points, and (a, b, c) T is an eigenvector of the covariance matrix. And selecting the minimum feature vector from the obtained feature vectors as a normal vector of the best fitting plane. Because the smaller the eigenvector is, the smaller the difference between the points in the point set covered by the covariance matrix is, the better the consistency is.
Through the above calculation, a plurality of fitting planes are obtained, including a ground surface fitting plane for the ground.
In another alternative embodiment, image clustering or region segmentation based on a gray scale map may be adopted, and then a segmented region with a larger area is selected (a threshold value σ is set) to segment a point cloud subset for plane fitting, and a fitting plane is generated based on the point cloud subset.
Referring to fig. 2, a gray scale diagram provided by an embodiment of the invention is shown. Referring to fig. 3, a block diagram for fig. 2 is shown. As can be seen from fig. 2 and 3, in some scenarios, the binocular camera may acquire image data for a building facade (e.g., a building surface), and a corresponding building facade fitting plane may be obtained for the image data for the building exterior surface.
The fitting plane can be determined to be a ground surface fitting plane or a building appearance fitting plane through the sizes and the angles of different fitting planes.
103, generating a fusion image according to the original depth map and the gray level map;
the pixels in the original depth map and the pixels in the gray level map can be fused to obtain a fused image, and the fused image has better edge detection characteristics.
104, determining a foreground area in the original depth map based on the fitting plane and the fused image;
the original depth map is divided into a foreground area, a background area and a fitting plane area, wherein the background area is an area where image data collected by an unstructured object (such as a plant) is located, the fitting plane area can be excluded from the original depth map, and the foreground area and the background area are divided from the rest areas in the original depth map based on a fused image.
105, performing resolution optimization on the foreground area to obtain a target foreground;
and (4) carrying out resolution optimization on the foreground area obtained in the step to obtain a target foreground with higher definition.
And 106, generating a target depth map based on the target foreground and the fitting plane.
And superposing the target foreground and the fitting plane to reconstruct the target depth map, wherein the target foreground is optimized, so that the target foreground in the target depth map has higher definition relative to other regions, and the local definition optimization of the foreground region of the original depth map is realized.
In specific implementation, a target depth map can be generated by superposing a target foreground, a fitting plane and a background area.
In the embodiment of the invention, after a gray scale image and an original depth image output by a binocular camera are obtained, a fitting plane is extracted according to the original depth image, the gray scale image and the original depth image which are subjected to edge detection processing are fused to obtain a fused image, a foreground region in the original depth image is detected based on the fitting plane and the fused image, resolution optimization is carried out on the foreground region to obtain a target foreground with improved definition, and the target foreground and the fitting plane are overlapped to obtain a target depth image which is used for carrying out local resolution optimization on the target foreground of interest, so that the global range optimization is avoided, and the optimization is more accurate and obvious than the global optimization.
In an alternative embodiment of the present invention, the step 103 comprises:
a substep S11, performing edge detection on the gray level image to obtain an edge detection result; the edge detection result comprises a first pixel value;
a sub-step S12 of determining a second pixel value in the depth map corresponding to the position of the first pixel value;
a substep S13, fusing the first pixel value and the second pixel value according to a preset linear relation to obtain a third pixel value;
and a substep S14 of generating a fused image for the third pixel value.
Referring to fig. 4, a schematic diagram of an edge result of a gray scale provided by the present invention is shown. In order to enhance the robustness of edge detection, the gray scale is subjected to edge detection to obtain an edge detection result, and the edge detection result containing the first pixel value is fused to the depth map to obtain a fused image.
Specifically, the third pixel value included in the fused image may be calculated by the following formula (7):
wherein, Pi' is the fused pixel value, i.e. the third pixel value, where α, β, γ are preset coefficients,
the pixel value after the edge detection of the original gray image is the first pixel value,
the value of the corresponding position of the depth map, i.e. the second pixel value, is the pixel index for distinguishing the pixel points at different positions.
In an alternative embodiment of the present invention, the step 104 comprises:
a substep S21 of generating a mask image for the fused image; the mask image is composed of fourth pixel values;
a sub-step S22 of determining a boundary length corresponding to the fourth pixel value;
a substep S23 of determining at least one region to be distinguished in the original depth map except for the position of the fitting plane;
substep S24, determining depth values of pixel points in the region to be distinguished;
a substep S25, obtaining a foreground reference value of the to-be-distinguished region according to the depth value and the boundary length;
a substep S26 of obtaining the number of edges for the original depth map;
and a substep S27 of determining a foreground region based on the foreground reference value and the number of edges.
In particular implementations, a matched mask image may be generated for the fused image, and a fourth pixel value P included in the mask image may be determinedjAnd determining the boundary length K of the mask image corresponding to the fourth pixel value.
Clustering the original depth map, performing region cutting on the original depth map based on a clustering result, and dividing at least one region except the region where the fitting plane is located in the original depth map into regions to be distinguished. After the depth value in the area to be distinguished is obtained, calculating a foreground reference value according to a formula (8) for the area to be distinguished which meets the size of a preset area:
And carrying out edge detection on the original depth map to obtain the number of edges in the original depth map.
And determining the foreground area by combining the foreground reference value and the edge number.
In an alternative embodiment of the present invention, the sub-step S27 includes:
in the substep S271, if the foreground reference value of the to-be-distinguished region is not greater than a first preset threshold value and the number of edges corresponding to the to-be-distinguished region is not less than a first preset number, determining that the to-be-distinguished region is a foreground region;
in the substep S272, if the foreground reference value of the to-be-distinguished region is greater than a first preset threshold and the number of edges corresponding to the to-be-distinguished region is less than a first preset number, determining that the to-be-distinguished region is a background region.
If the foreground reference value in the area to be distinguished is larger than the first threshold value, and the position corresponding to the area to be distinguished has a first preset number of edges, namely the area to be distinguished is a foreground area, otherwise the area is a background area.
In an alternative embodiment of the present invention, said step 105 comprises:
a substep S31 of recognizing a target virtual object in the foreground region using preset active contour information;
an active contour line can be constructed, and a target virtual object of interest is determined in a target foreground region by the active contour line self-adapting to the 4-space curve, wherein the target virtual object is an object with a structural characteristic. For example: vehicle a, vehicle B, and vehicle C in fig. 3.
And a substep S32, performing resolution optimization on the target virtual object to obtain a target foreground.
And the resolution of the target virtual object is improved in a targeted manner, so that a target foreground is obtained.
In a specific implementation, the active contour line may be constructed as in expression (9):
v(s)＝(x(s),y())∈[0,1]
and determining the energy function as expression (10):
wherein E isextThe external energy is represented by the external energy,
and alpha and beta are constant coefficients and are used for expressing the influence of each energy term on the optimization result. And corresponds to the squares of the modes of the first and second derivatives of the activity curve, respectively. When the contour C is close to the edge of the target image, the gradient of C will increase, EextIt is reduced. During the energy minimization, the contour lines will evolve toward the approach of the target edges. After the moving contour, a single object block (target virtual object) can be obtained from the foreground, and the object depth data is refined by performing super-resolution transformation on the object block to obtain the target foreground.
In an alternative embodiment of the present invention, the sub-step S32 includes:
the substep S321 is to obtain an external sub-image matched with the target virtual object in the original depth map;
substep S322, obtaining the edge mask matched with the circumscribed image in the gray-scale image;
a substep S323, carrying out interpolation processing on the external sub-image to obtain an interpolated image;
substep S324, superimposing the interpolated image and the edge mask to obtain a superimposed image;
substep S325, performing edge filtering on the superposed image to obtain a filtered image containing a connected region;
substep S326, performing depth value restoration processing on the filtered image to obtain a target sub-image;
and a substep S327 of generating a target foreground based on the target sub-image.
Referring to fig. 5, a circumscribed rectangular word diagram provided by the present invention is shown. In a specific implementation, a circumscribed rectangle subgraph of the active contour can be cut from the original depth map. And then carrying out trilinear interpolation on the rectangular subgraph to improve the resolution, superposing the edge flooded mode of the original gray graph in the area, and carrying out filtering on the edge in the original gray graph corresponding to the subgraph with the improved resolution (the edge can be deleted according to the length of the edge, and the edge with the shorter edge is removed). The object edge processed by the second step divides the sub-depth map into several connected regions, so that the depth value restoration can be finally performed on the connected region.
In an optional embodiment of the present invention, the substep S326 comprises:
a substep S3261 of performing mean filtering on two sides of the edge of the communication area to obtain a mean image;
a substep S3262 of generating three-dimensional point cloud data corresponding to the mean image;
a substep S3263 of fitting the three-dimensional point cloud data to obtain a virtual plane;
step S3264, performing inverse mapping on the virtual plane to obtain a depth value gradient image;
and a substep S3265 of performing downsampling processing on the depth value gradient image to generate a target sub-image.
In a specific implementation, mean filtering may be performed on at least some pixels in the two side regions of the edge, then the connected region is mapped to the three-dimensional point cloud and a corresponding virtual plane is fitted, and then the virtual plane is back-mapped to the depth map (which is changed to continuous gradual change of values) to obtain a depth value gradient image. And splicing the continuous depth value gradient images with gradually changed values to obtain the final optimized target sub-image of the object. And finally, filling the target sub-image into the original depth map through downsampling. By generating the virtual plane and reflecting the shot-back depth map, the continuity and consistency of the values of the non-edge cultural relic area of the depth map object can be obviously improved.
In the following, embodiments of the invention are further illustrated by way of an example:
referring to fig. 6, there is shown a flowchart of an exemplary image processing method provided by the present invention, which includes the following steps:
601, original grey scale map. The binocular camera outputs an original gray scale map according to the acquired image data.
And 602, performing binocular stereo matching and outputting a depth map. And the binocular camera reads and processes the acquired image data according to a preset binocular stereo matching algorithm aiming at the acquired image data, and outputs a depth map.
603, detecting the fusion edge. And carrying out edge detection on the gray-scale image, and fusing pixels in the gray-scale image and the depth image based on the detection result to obtain a fused image.
604, separating foreground from background. And distinguishing a foreground region and a background region in the depth map based on the fused image so as to achieve foreground and background separation of the depth map.
605, foreground optimization. And determining an interested area for the foreground area, and optimizing the resolution of the interested area, thereby optimizing the foreground area and obtaining the target foreground.
And 606, plane fitting based on the depth map. And carrying out plane fitting processing on the depth map to obtain at least one fitting plane.
607, ground extraction. And performing plane fitting processing on the depth map to generate a ground surface fitting plane corresponding to the part of the depth map corresponding to the ground surface.
And 608, extracting the outer facade of the building. And performing plane fitting processing on the depth map to generate a building appearance fitting plane corresponding to the part of the depth map corresponding to the building facade.
And 609, reconstructing the depth map. And combining the target foreground, the background area, the ground surface fitting plane and the building appearance fitting plane to reconstruct the depth map and generate a new depth map.
It should be noted that, for simplicity of description, the method embodiments are described as a series of acts or combination of acts, but those skilled in the art will recognize that the present invention is not limited by the illustrated order of acts, as some steps may occur in other orders or concurrently in accordance with the embodiments of the present invention. Further, those skilled in the art will appreciate that the embodiments described in the specification are presently preferred and that no particular act is required to implement the invention.
Referring to fig. 7, a block diagram of an embodiment of an image processing apparatus according to the present invention is shown, and may specifically include the following modules:
an image obtaining module 701, configured to obtain an original depth map and a grayscale map;
a fitting plane generating module 702, configured to generate a fitting plane according to the original depth map;
a fusion module 703, configured to generate a fusion image according to the original depth map and the grayscale map;
a foreground region determining module 704, configured to determine a foreground region in the original depth map based on the fitting plane and the fused image;
a target foreground generating module 705, configured to perform resolution optimization on the foreground region to obtain a target foreground;
and a target depth map generating module 706, configured to generate a target depth map based on the target foreground and the fitting plane.
In an optional embodiment of the invention, the fusion module comprises:
the edge detection submodule is used for carrying out edge detection on the gray level image to obtain an edge detection result; the edge detection result comprises a first pixel value;
a second pixel value determination submodule for determining a second pixel value in the depth map corresponding to the position of the first pixel value;
the third pixel value determining submodule is used for fusing the first pixel value and the second pixel value according to a preset linear relation to obtain a third pixel value;
and the image fusion submodule is used for generating a fusion image aiming at the third pixel value.
In an optional embodiment of the present invention, the foreground region determining module includes:
a mask image generation submodule for generating a mask image for the fused image; the mask image is composed of fourth pixel values;
a boundary length determination submodule for determining a boundary length corresponding to the fourth pixel value;
a region to be distinguished determining submodule, configured to determine at least one region to be distinguished in the original depth map, except for a position of the fitting plane;
the depth value determining submodule is used for determining the depth value of the pixel point in the area to be distinguished;
the foreground reference value submodule is used for obtaining a foreground reference value of the area to be distinguished according to the depth value and the boundary length;
an edge number determination submodule for obtaining the number of edges for the original depth map;
and the foreground area determining submodule is used for determining a foreground area based on the foreground reference value and the edge number.
In an optional embodiment of the present invention, the foreground region determining sub-module includes:
the first judging unit is used for determining the area to be distinguished as a foreground area if the foreground reference value of the area to be distinguished is not larger than a first preset threshold value and the number of edges corresponding to the area to be distinguished is not smaller than a first preset number;
a second determining unit, configured to determine that the region to be distinguished is a background region if the foreground reference value of the region to be distinguished is greater than a first preset threshold and the number of edges corresponding to the region to be distinguished is less than a first preset number.
In an optional embodiment of the present invention, the target foreground generating module includes:
the target virtual object identification submodule is used for identifying a target virtual object in the foreground area by adopting preset active contour information;
and the target foreground determining submodule is used for carrying out resolution optimization on the target virtual object to obtain a target foreground.
In an optional embodiment of the present invention, the target foreground determining sub-module includes:
the external subimage acquisition unit is used for acquiring an external subimage matched with the target virtual object in the original depth map;
the edge mask unit is used for acquiring an edge mask matched with the external subimage in the gray-scale image;
the interpolation unit is used for carrying out interpolation processing on the external sub-image to obtain an interpolated image;
the superposition unit is used for superposing the interpolated image and the edge mask to obtain a superposed image;
the filtering image unit is used for carrying out edge filtering on the superposed image to obtain a filtering image containing a connected region;
the target sub-image unit is used for carrying out depth value restoration processing on the filtered image to obtain a target sub-image;
and the target foreground generating unit is used for generating a target foreground based on the target sub-image.
In an optional embodiment of the invention, the target sub-image cell comprises:
the mean value subunit is used for carrying out mean value filtering on two sides of the edge of the communication area to obtain a mean value image;
the point cloud data subunit is used for generating three-dimensional point cloud data corresponding to the average value image;
the virtual plane subunit is used for fitting the three-dimensional point cloud data to obtain a virtual plane;
the inverse mapping subunit is configured to perform inverse mapping on the virtual plane to obtain an image with a gradually-changed depth value;
and the target sub-image sub-unit is used for carrying out downsampling processing on the depth value gradient image to generate a target sub-image.
For the device embodiment, since it is basically similar to the method embodiment, the description is simple, and for the relevant points, refer to the partial description of the method embodiment.
An embodiment of the present invention further provides an electronic device, including: the image processing method comprises a processor, a memory and a computer program which is stored on the memory and can run on the processor, wherein when the computer program is executed by the processor, each process of the image processing method embodiment is realized, the same technical effect can be achieved, and the details are not repeated here to avoid repetition.
The embodiment of the present invention further provides a computer-readable storage medium, where a computer program is stored on the computer-readable storage medium, and when being executed by a processor, the computer program implements each process of the embodiment of the image processing method, and can achieve the same technical effect, and is not described herein again to avoid repetition.
The embodiments in the present specification are described in a progressive manner, each embodiment focuses on differences from other embodiments, and the same and similar parts among the embodiments are referred to each other.
As will be appreciated by one skilled in the art, embodiments of the present invention may be provided as a method, apparatus, or computer program product. Accordingly, embodiments of the present invention may take the form of an entirely hardware embodiment, an entirely software embodiment or an embodiment combining software and hardware aspects. Furthermore, embodiments of the present invention may take the form of a computer program product embodied on one or more computer-usable storage media (including, but not limited to, disk storage, CD-ROM, optical storage, and the like) having computer-usable program code embodied therein.
Embodiments of the present invention are described with reference to flowchart illustrations and/or block diagrams of methods, terminal devices (systems), and computer program products according to embodiments of the invention. It will be understood that each flow and/or block of the flow diagrams and/or block diagrams, and combinations of flows and/or blocks in the flow diagrams and/or block diagrams, can be implemented by computer program instructions. These computer program instructions may be provided to a processor of a general purpose computer, special purpose computer, embedded processor, or other programmable data processing terminal to produce a machine, such that the instructions, which execute via the processor of the computer or other programmable data processing terminal, create means for implementing the functions specified in the flowchart flow or flows and/or block diagram block or blocks.
These computer program instructions may also be stored in a computer-readable memory that can direct a computer or other programmable data processing terminal to function in a particular manner, such that the instructions stored in the computer-readable memory produce an article of manufacture including instruction means which implement the function specified in the flowchart flow or flows and/or block diagram block or blocks.
These computer program instructions may also be loaded onto a computer or other programmable data processing terminal to cause a series of operational steps to be performed on the computer or other programmable terminal to produce a computer implemented process such that the instructions which execute on the computer or other programmable terminal provide steps for implementing the functions specified in the flowchart flow or flows and/or block diagram block or blocks.
While preferred embodiments of the present invention have been described, additional variations and modifications of these embodiments may occur to those skilled in the art once they learn of the basic inventive concepts. Therefore, it is intended that the appended claims be interpreted as including preferred embodiments and all such alterations and modifications as fall within the scope of the embodiments of the invention.
Finally, it should also be noted that, herein, relational terms such as first and second, and the like may be used solely to distinguish one entity or action from another entity or action without necessarily requiring or implying any actual such relationship or order between such entities or actions. Also, the terms "comprises," "comprising," or any other variation thereof, are intended to cover a non-exclusive inclusion, such that a process, method, article, or terminal that comprises a list of elements does not include only those elements but may include other elements not expressly listed or inherent to such process, method, article, or terminal. Without further limitation, an element defined by the phrase "comprising an … …" does not exclude the presence of other like elements in a process, method, article, or terminal that comprises the element.
The foregoing detailed description of an image processing method and an image processing apparatus according to the present invention has been presented, and the principles and embodiments of the present invention are explained herein by using specific examples, which are only used to help understand the method and the core idea of the present invention; meanwhile, for a person skilled in the art, according to the idea of the present invention, there may be variations in the specific embodiments and the application scope, and in summary, the content of the present specification should not be construed as a limitation to the present invention.
Claims (10)
1. An image processing method, comprising:
acquiring an original depth map and a gray scale map;
generating a fitting plane according to the original depth map;
generating a fusion image according to the original depth map and the gray level map;
determining a foreground region in the original depth map based on the fitted plane and the fused image;
performing resolution optimization on the foreground area to obtain a target foreground;
and generating a target depth map based on the target foreground and the fitting plane.
2. The method of claim 1, wherein the step of generating a fused image from the original depth map and the grayscale map comprises:
carrying out edge detection on the gray-scale image to obtain an edge detection result; the edge detection result comprises a first pixel value;
determining a second pixel value in the depth map corresponding to the position of the first pixel value;
fusing the first pixel value and the second pixel value according to a preset linear relation to obtain a third pixel value;
generating a fused image for the third pixel value.
3. The method of claim 2, wherein the step of determining the foreground region in the original depth map based on the fitted plane and the fused image comprises:
generating a mask image for the fused image; the mask image is composed of fourth pixel values;
determining a boundary length corresponding to the fourth pixel value;
determining at least one region to be distinguished in the original depth map except for the position of the fitting plane;
determining the depth value of a pixel point in the area to be distinguished;
obtaining a foreground reference value of the area to be distinguished according to the depth value and the boundary length;
obtaining the number of edges for the original depth map;
determining a foreground region based on the foreground reference value and the number of edges.
4. The method of claim 3, wherein the step of determining a foreground region based on the foreground reference value and the number of edges comprises:
if the foreground reference value of the area to be distinguished is not larger than a first preset threshold value and the number of edges corresponding to the area to be distinguished is not smaller than a first preset number, determining the area to be distinguished as a foreground area;
and if the foreground reference value of the area to be distinguished is greater than a first preset threshold value and the number of edges corresponding to the area to be distinguished is less than a first preset number, determining that the area to be distinguished is a background area.
5. The method according to any one of claims 1-4, wherein the step of performing resolution optimization on the foreground region to obtain the target foreground comprises:
adopting preset active contour information to identify a target virtual object in the foreground area;
and carrying out resolution optimization on the target virtual object to obtain a target foreground.
6. The method of claim 5, wherein the step of performing resolution optimization on the target virtual object to obtain a target foreground comprises:
acquiring an external sub-image matched with the target virtual object in the original depth map;
acquiring an edge mask matched with the external sub-image in the gray-scale image;
carrying out interpolation processing on the external sub-image to obtain an interpolated image;
overlapping the interpolated image and the edge mask to obtain an overlapped image;
performing edge filtering on the superposed image to obtain a filtered image containing a connected region;
performing depth value restoration processing on the filtered image to obtain a target sub-image;
and generating a target foreground based on the target sub-image.
7. The method according to claim 6, wherein the step of performing depth value restoration processing on the filtered image to obtain a target sub-image comprises:
carrying out mean value filtering on two sides of the edge of the communication area to obtain a mean value image;
generating three-dimensional point cloud data corresponding to the mean value image;
fitting the three-dimensional point cloud data to obtain a virtual plane;
performing inverse mapping on the virtual plane to obtain an image with gradually changed depth values;
and performing downsampling processing on the depth value gradient image to generate a target sub-image.
8. An image processing apparatus characterized by comprising:
the image acquisition module is used for acquiring an original depth map and a gray scale map;
a fitting plane generating module for generating a fitting plane according to the original depth map;
the fusion module is used for generating a fusion image according to the original depth map and the gray level map;
a foreground region determination module, configured to determine a foreground region in the original depth map based on the fitting plane and the fused image;
the target foreground generation module is used for carrying out resolution optimization on the foreground area to obtain a target foreground;
and the target depth map generating module is used for generating a target depth map based on the target foreground and the fitting plane.
9. An electronic device, comprising: processor, memory and a computer program stored on the memory and executable on the processor, which computer program, when executed by the processor, carries out the steps of the image processing method according to any one of claims 1 to 7.
10. A computer-readable storage medium, on which a computer program is stored which, when being executed by a processor, carries out the steps of the image processing method according to any one of claims 1 to 7.
Priority Applications (1)
| Application Number | Priority Date | Filing Date | Title |
|---|---|---|---|
| CN202210033560.1A CN114359123B (en) | 2022-01-12 | 2022-01-12 | Image processing method and device |
Applications Claiming Priority (1)
| Application Number | Priority Date | Filing Date | Title |
|---|---|---|---|
| CN202210033560.1A CN114359123B (en) | 2022-01-12 | 2022-01-12 | Image processing method and device |
Publications (2)
| Publication Number | Publication Date |
|---|---|
| CN114359123A true CN114359123A (en) | 2022-04-15 |
| CN114359123B CN114359123B (en) | 2024-09-20 |
Family
ID=81110095
Family Applications (1)
| Application Number | Title | Priority Date | Filing Date |
|---|---|---|---|
| CN202210033560.1A Active CN114359123B (en) | 2022-01-12 | 2022-01-12 | Image processing method and device |
Country Status (1)
| Country | Link |
|---|---|
| CN (1) | CN114359123B (en) |
Citations (15)
| Publication number | Priority date | Publication date | Assignee | Title |
|---|---|---|---|---|
| US20130106848A1 (en) * | 2011-05-12 | 2013-05-02 | Khang Nguyen | Image generation apparatus and image generation method |
| US20130243313A1 (en) * | 2010-10-01 | 2013-09-19 | Telefonica, S.A. | Method and system for images foreground segmentation in real-time |
| CN105069808A (en) * | 2015-08-31 | 2015-11-18 | 四川虹微技术有限公司 | Video image depth estimation method based on image segmentation |
| WO2017067390A1 (en) * | 2015-10-20 | 2017-04-27 | 努比亚技术有限公司 | Method and terminal for obtaining depth information of low-texture regions in image |
| CN106898002A (en) * | 2017-01-04 | 2017-06-27 | 努比亚技术有限公司 | A kind of method for realizing interactive image segmentation, device and terminal |
| CN110211061A (en) * | 2019-05-20 | 2019-09-06 | 清华大学 | List depth camera depth map real time enhancing method and device neural network based |
| US20200184673A1 (en) * | 2018-12-11 | 2020-06-11 | Wistron Corporation | Method for identifying foreground object in image and electronic device |
| CN111553940A (en) * | 2020-05-19 | 2020-08-18 | 上海海栎创微电子有限公司 | Depth image portrait edge optimization method and processing device |
| CN111899266A (en) * | 2020-07-17 | 2020-11-06 | 深圳奥比中光科技有限公司 | Matting method and system based on RGBD camera |
| CN112330709A (en) * | 2020-10-29 | 2021-02-05 | 奥比中光科技集团股份有限公司 | Foreground image extraction method and device, readable storage medium and terminal equipment |
| CN112927279A (en) * | 2021-02-24 | 2021-06-08 | 中国科学院微电子研究所 | Image depth information generation method, device and storage medium |
| WO2021136078A1 (en) * | 2019-12-31 | 2021-07-08 | RealMe重庆移动通信有限公司 | Image processing method, image processing system, computer readable medium, and electronic apparatus |
| CN113139997A (en) * | 2020-01-19 | 2021-07-20 | 武汉Tcl集团工业研究院有限公司 | Depth map processing method, storage medium and terminal device |
| CN113223070A (en) * | 2021-05-13 | 2021-08-06 | 深圳地理人和科技有限公司 | Depth image enhancement processing method and device |
| CN113240700A (en) * | 2021-05-27 | 2021-08-10 | 广州博冠信息科技有限公司 | Image processing method and device, computer readable storage medium and electronic device |
-
2022
- 2022-01-12 CN CN202210033560.1A patent/CN114359123B/en active Active
Patent Citations (15)
| Publication number | Priority date | Publication date | Assignee | Title |
|---|---|---|---|---|
| US20130243313A1 (en) * | 2010-10-01 | 2013-09-19 | Telefonica, S.A. | Method and system for images foreground segmentation in real-time |
| US20130106848A1 (en) * | 2011-05-12 | 2013-05-02 | Khang Nguyen | Image generation apparatus and image generation method |
| CN105069808A (en) * | 2015-08-31 | 2015-11-18 | 四川虹微技术有限公司 | Video image depth estimation method based on image segmentation |
| WO2017067390A1 (en) * | 2015-10-20 | 2017-04-27 | 努比亚技术有限公司 | Method and terminal for obtaining depth information of low-texture regions in image |
| CN106898002A (en) * | 2017-01-04 | 2017-06-27 | 努比亚技术有限公司 | A kind of method for realizing interactive image segmentation, device and terminal |
| US20200184673A1 (en) * | 2018-12-11 | 2020-06-11 | Wistron Corporation | Method for identifying foreground object in image and electronic device |
| CN110211061A (en) * | 2019-05-20 | 2019-09-06 | 清华大学 | List depth camera depth map real time enhancing method and device neural network based |
| WO2021136078A1 (en) * | 2019-12-31 | 2021-07-08 | RealMe重庆移动通信有限公司 | Image processing method, image processing system, computer readable medium, and electronic apparatus |
| CN113139997A (en) * | 2020-01-19 | 2021-07-20 | 武汉Tcl集团工业研究院有限公司 | Depth map processing method, storage medium and terminal device |
| CN111553940A (en) * | 2020-05-19 | 2020-08-18 | 上海海栎创微电子有限公司 | Depth image portrait edge optimization method and processing device |
| CN111899266A (en) * | 2020-07-17 | 2020-11-06 | 深圳奥比中光科技有限公司 | Matting method and system based on RGBD camera |
| CN112330709A (en) * | 2020-10-29 | 2021-02-05 | 奥比中光科技集团股份有限公司 | Foreground image extraction method and device, readable storage medium and terminal equipment |
| CN112927279A (en) * | 2021-02-24 | 2021-06-08 | 中国科学院微电子研究所 | Image depth information generation method, device and storage medium |
| CN113223070A (en) * | 2021-05-13 | 2021-08-06 | 深圳地理人和科技有限公司 | Depth image enhancement processing method and device |
| CN113240700A (en) * | 2021-05-27 | 2021-08-10 | 广州博冠信息科技有限公司 | Image processing method and device, computer readable storage medium and electronic device |
Non-Patent Citations (4)
| Title |
|---|
| YOUNG-JOO SEO; JONGMIN KIM; HOYONG JANG; TAE-HO KIM; CHANG D. YOO: "《Joint learning of foreground region labeling and depth ordering》", 《 2014 IEEE INTERNATIONAL CONFERENCE ON ACOUSTICS, SPEECH AND SIGNAL PROCESSING (ICASSP)》, 14 July 2014 (2014-07-14), pages 5100 - 5104 * |
| 包加桐;宋爱国;洪泽;沈天鹤;唐鸿儒;: "基于物体逻辑状态推理的未知物体视觉分割", 机器人, no. 04, 15 July 2017 (2017-07-15), pages 49 - 56 * |
| 吴剑锋: "基于点云融合算法的Kinect三维重建技术及其应用研究", 计算机应用与软件, no. 08, 12 August 2018 (2018-08-12), pages 266 - 270 * |
| 来伊丽;唐向宏;楼幸欣;: "利用梯度融合与聚类的三维视频图像修复", 计算机辅助设计与图形学学报, no. 03, 15 March 2018 (2018-03-15), pages 113 - 120 * |
Also Published As
| Publication number | Publication date |
|---|---|
| CN114359123B (en) | 2024-09-20 |
Similar Documents
| Publication | Publication Date | Title |
|---|---|---|
| CN106651938B (en) | A kind of depth map Enhancement Method merging high-resolution colour picture | |
| EP3489898B1 (en) | Method and apparatus for estimating disparity | |
| US20200234397A1 (en) | Automatic view mapping for single-image and multi-view captures | |
| AU2017324923A1 (en) | Predicting depth from image data using a statistical model | |
| CN112991413A (en) | Self-supervision depth estimation method and system | |
| US20230252664A1 (en) | Image Registration Method and Apparatus, Electronic Apparatus, and Storage Medium | |
| US11783443B2 (en) | Extraction of standardized images from a single view or multi-view capture | |
| CN107924571A (en) | Three-dimensional reconstruction is carried out to human ear from a cloud | |
| CN107369131B (en) | Conspicuousness detection method, device, storage medium and the processor of image | |
| CN111209770A (en) | Lane line identification method and device | |
| CN113724379B (en) | Three-dimensional reconstruction method and device for fusing image and laser point cloud | |
| CN112258568B (en) | High-precision map element extraction method and device | |
| CN110443228B (en) | Pedestrian matching method and device, electronic equipment and storage medium | |
| CN115222889A (en) | 3D reconstruction method and device based on multi-view image and related equipment | |
| CN111862321B (en) | Parallax map acquisition method, device, system and storage medium | |
| CN117315210B (en) | Image blurring method based on stereoscopic imaging and related device | |
| CN117274605B (en) | Method and device for extracting water area outline from photo shot by unmanned aerial vehicle | |
| CN111260712B (en) | Depth estimation method and device based on refocusing polar line graph neighborhood distribution | |
| WO2021167910A1 (en) | A method for generating a dataset, a method for generating a neural network, and a method for constructing a model of a scene | |
| CN110148086B (en) | Depth filling method and device for sparse depth map and three-dimensional reconstruction method and device | |
| CN114359123A (en) | Image processing method and device | |
| CN118696347A (en) | Method for head image registration and head model generation and corresponding device | |
| KR102633159B1 (en) | Apparatus and method for restoring 3d-model using the image-processing | |
| CN111383185A (en) | Hole filling method based on dense disparity map and vehicle-mounted equipment | |
| CN117710931A (en) | Environment information sensing method, device, system, computer equipment and storage medium |
Legal Events
| Date | Code | Title | Description |
|---|---|---|---|
| PB01 | Publication | ||
| PB01 | Publication | ||
| SE01 | Entry into force of request for substantive examination | ||
| SE01 | Entry into force of request for substantive examination | ||
| GR01 | Patent grant | ||
| GR01 | Patent grant |