WO2018209969A1

WO2018209969A1 - Depth map creation method and system and image blurring method and system

Info

Publication number: WO2018209969A1
Application number: PCT/CN2017/120331
Authority: WO
Inventors: 龙学军; 刘勇; 周剑
Original assignee: 成都通甲优博科技有限责任公司
Priority date: 2017-05-19
Filing date: 2017-12-29
Publication date: 2018-11-22
Also published as: CN107170008B; CN107170008A

Abstract

The present application discloses a depth map creation method and system and an image blurring method and system. The depth map creation method comprises: obtaining a first image and a second image by a binocular shooting system photographing a target scene; determining respective feature points corresponding to the images to obtain a first set of feature points and a second set of feature points; using the first set of feature points and the second set of feature points to correspondingly determine support points corresponding to the images, to obtain a first set of support points and a second set of support points; performing, based on the first set of support points and the second set of support points, dense stereo matching on the first image and the second image, so as to determine parallax between the first image and the second image; and using the parallax to determine a depth map corresponding to the target scene. The present application can greatly improve accuracy of stereo matching between images, obtaining more accurate parallax between the images. A depth map obtained based on the parallax has higher quality, improving the effect of subsequent image blurring.

Description

Depth map creation method and system and image blurring method and system

This application claims priority to Chinese Patent Application No. 201710361218.3, entitled "A Method, System for Creating a Depth Map, and Image Blur Method, System", submitted on May 19, 2017, and the entire contents of which are hereby incorporated by reference. The manner of citation is incorporated in the present application.

Technical field

The present invention relates to the field of image processing technologies, and in particular, to a depth map creation method and system, and an image blurring method and system.

Background technique

At present, with the rapid development of image processing technology, more and more devices such as smart phones and tablet computers have gradually increased the ability to use the depth information of the scene to blur the image, which brings a lot of interesting to the user. Photo experience.

In the existing image blurring process, the quality of the depth map directly affects the subsequent image blurring effect. How to create a high-quality depth map is still a problem to be solved.

Summary of the invention

In view of this, the object of the present invention is to provide a depth map creation method and system and an image blurring method and system, which can obtain a high-quality depth map, thereby facilitating an image blurring effect. The specific plan is as follows:

A depth map creation method, including:

Obtaining a first image and a second image obtained by the binocular shooting system after shooting the target scene;

Determining feature points corresponding to the first image and the second image, respectively, to obtain a first set of feature points and a second set of feature points;

Using the first set of feature points and the second set of feature points respectively, respectively determining support points corresponding to the first image and the second image to obtain a first set of support points and a second set of support points ;

Performing dense stereo matching on the first image and the second image based on the first set of support points and the second set of support points to determine between the first image and the second image Parallax

A depth map corresponding to the target scene is determined using the parallax.

Optionally, the process of determining feature points of any image includes:

Determining candidate points from the image;

Counting the total number of pixel points satisfying the preset condition around the candidate point;

Determining whether the total number is greater than a preset number threshold, and if so, determining that the candidate point is a feature point of the image, and if not, determining that the candidate point is not a feature point of the image.

Optionally, the process of counting the total number of pixel points satisfying the preset condition around the candidate point includes:

The total number of pixels satisfying the preset condition around the candidate points is counted by using a preset pixel number statistical formula; wherein the predetermined pixel number statistics formula is:

Where N represents the total number, p represents the candidate point, circle(p) represents a circle centered on the candidate point p, with a predetermined value as a radius, and x represents the circumference circle(p) any of a pixel, I (x) denotes the gray value of pixel x, I (p) denotes the gray value of the candidate point p, ε _d represents the difference between a predetermined gradation threshold.

Optionally, the process of performing dense stereo matching on the first image and the second image based on the first set of support points and the second set of support points, including

Constructing a corresponding Delaunay triangular mesh on the first image according to the first set of support points;

Calculating a disparity corresponding to a pixel point located in the Delaunay triangular mesh to obtain corresponding disparity data;

Performing dense stereo matching on the first image and the second image by using the disparity data and a disparity probability model to find out from the second set of support points that the first set of support points are matched Support point.

Optionally, the process of calculating a disparity corresponding to a pixel point in the Delaunay triangular mesh includes:

Calculating a parallax corresponding to a pixel point located in the Delaunay triangular mesh by using a preset parallax calculation formula; wherein the preset disparity calculation formula is:

Where d _p represents the parallax corresponding to the pixel point p in the Delaunay triangular mesh, (u _p , v _p ) represents the coordinates of the pixel point p, and a, b, c are the Delaunay triangular regions where the pixel point p is located The coefficient obtained by fitting the support point plane, h represents the minimum support distance of the pixel point p and the adjacent three support points,

Expressed in the interval

Take a random number.

The invention further discloses an image blurring method, comprising:

Obtaining a depth map obtained by the foregoing method;

The image is blurred by the depth map to obtain a blurred image.

Optionally, the process of performing image blurring processing by using the depth map includes:

Determining a focus area on the depth map;

Using the depth information on the depth map, and combining the preset function, obtaining a blurring coefficient of each pixel in the depth map; wherein the preset function is:

Wherein, C _i represents a blurring coefficient of the i-th pixel point, z _i represents a depth value of the i-th pixel point, and f represents a focal length.

Representing an average depth value on the focus area, Z _far represents a maximum depth value on the focus area, Z _near represents a minimum depth value on the focus area, and w represents an adjustment factor;

Using the preset blurring formula, performing image blurring processing on the target pixel point set to obtain the blurred image; wherein the target pixel point set is a pixel point whose depth value is in the range of [Z _near , Z _far ] The set, the preset blur formula is:

Wherein m × n represents the number of pixel points in a circle having a coordinate (x, y) of the target pixel as a center and a blur factor of the target pixel as a radius, the target pixel being the Any pixel of the target pixel set,

The pixel value of the target pixel after the blurring process is indicated, and I _i,j represents the pixel value corresponding to the pixel point (i, j) in the circle before the blurring process.

The invention also correspondingly discloses a depth map creation system, comprising:

An image acquisition module, configured to acquire a first image and a second image obtained by the binocular shooting system after capturing the target scene;

a feature point determining module, configured to respectively determine feature points corresponding to the first image and the second image, to obtain a first set of feature points and a second set of feature points;

a support point determining module, configured to respectively determine a support point corresponding to the first image and the second image by using the first set of feature points and the second set of feature points, to obtain a first set of supports Point and second set of support points;

a matching module, configured to perform dense stereo matching on the first image and the second image based on the first set of support points and the second set of support points to determine the first image and the first Parallax between two images;

a depth map determining module configured to determine a depth map corresponding to the target scene by using the parallax.

The invention also correspondingly discloses an image blurring system, comprising:

a depth map obtaining module, configured to acquire a depth map created by the foregoing depth map creation system;

An image blurring module is configured to perform image blurring processing by using the depth map to obtain a blurred image.

Optionally, the image blurring module includes:

a focus area determining unit, configured to determine a focus area on the depth map;

a blurring coefficient calculation unit, configured to use the depth information on the depth map, and combine with a preset function to obtain a blurring coefficient of each pixel in the depth map; wherein the preset function is:

a blurring processing unit, configured to perform image blurring processing on the target pixel point set by using a preset blurring formula to obtain the blurred image; wherein the target pixel point set is a depth value in [Z _near , Z _far a collection of pixels within the range, the preset blur formula is:

In the present invention, the depth map creation method includes: acquiring a first image and a second image obtained by capturing a target scene by the binocular shooting system; respectively determining feature points corresponding to the first image and the second image, and obtaining the first group a feature point and a second set of feature points; respectively, using the first set of feature points and the second set of feature points, correspondingly determining support points corresponding to the first image and the second image, and obtaining the first set of support points and the second set of support Pointing; performing dense stereo matching on the first image and the second image based on the first set of support points and the second set of support points to determine a disparity between the first image and the second image; determining the corresponding to the target scene by using the disparity Depth map.

It can be seen that after acquiring the first image and the second image, the present invention first determines the feature points corresponding to the image, and then uses the feature points of the image to determine the support points corresponding to the image, which may be subsequently based on the first image and the second image. Dense stereo matching is performed on the corresponding support points of the image, so that the stereo matching precision between the first image and the second image can be greatly improved, thereby obtaining a more accurate parallax between the first image and the second image, which is obtained based on the parallax The depth map will have a higher quality, which will help to improve the subsequent image blurring effect.

DRAWINGS

In order to more clearly illustrate the embodiments of the present invention or the technical solutions in the prior art, the drawings used in the embodiments or the description of the prior art will be briefly described below. Obviously, the drawings in the following description are only It is an embodiment of the present invention, and those skilled in the art can obtain other drawings according to the provided drawings without any creative work.

FIG. 1 is a flowchart of a method for creating a depth map according to an embodiment of the present invention;

FIG. 2 is a schematic structural diagram of a depth map creation system according to an embodiment of the present invention.

detailed description

The technical solutions in the embodiments of the present invention are clearly and completely described in the following with reference to the accompanying drawings in the embodiments of the present invention. It is obvious that the described embodiments are only a part of the embodiments of the present invention, but not all embodiments. All other embodiments obtained by those skilled in the art based on the embodiments of the present invention without creative efforts are within the scope of the present invention.

The embodiment of the invention discloses a depth map creation method. Referring to FIG. 1 , the method includes:

Step S11: Acquire a first image and a second image obtained after the binocular shooting system captures the target scene.

Step S12: determining feature points corresponding to the first image and the second image, respectively, to obtain a first set of feature points and a second set of feature points.

Step S13: Using the first set of feature points and the second set of feature points respectively, respectively determining the support points corresponding to the first image and the second image to obtain the first set of support points and the second set of support points.

In the embodiment of the present invention, the polar line constraint and the feature point descriptor can be used to quickly perform matching processing between feature points. Specifically, a WTA strategy (WTA, ie, Winner Takes All) can be used to select a matching cost in the parallax space. The smallest point is used as the feature point for matching success, and then the feature points that match the success are determined as the support points, and the unmatched feature points are eliminated, thereby obtaining the first set of support points and the second set of support points respectively.

Step S14: Perform dense stereo matching on the first image and the second image based on the first set of support points and the second set of support points to determine a disparity between the first image and the second image.

Step S15: Determine a depth map corresponding to the target scene by using the parallax.

It can be seen that, after acquiring the first image and the second image, the embodiment of the present invention first determines a feature point corresponding to the image, and then uses the feature point of the image to determine a support point corresponding to the image, which may be subsequently based on the first image and The corresponding support points of the second image are densely stereo-matched, which can greatly improve the stereo matching precision between the first image and the second image, thereby obtaining a more accurate parallax between the first image and the second image, based on the parallax The obtained depth map will have higher quality, which will help to improve the subsequent image blurring effect.

The embodiment of the present invention discloses a specific depth map creation method. Compared with the previous embodiment, the technical solution is further illustrated and optimized in this embodiment. specific:

In step S12 of the previous embodiment, it is necessary to separately determine feature points corresponding to the first image and the second image. In this embodiment, the determining process of the feature points of any image may specifically include the following steps S121 to S123:

Step S121: determining a candidate point from the image;

Step S122: Count the total number of pixel points satisfying the preset condition around the candidate point;

Step S123: determining whether the total number is greater than a preset number threshold, and if so, determining that the candidate point is a feature point of the image, and if not, determining that the candidate point is not a feature point of the image.

In the above step S122, the process of counting the total number of the pixel points that meet the preset condition around the candidate point may specifically include:

The total number of pixels satisfying the preset condition around the candidate points is counted by using a preset pixel number statistical formula; wherein the predetermined number of preset pixel points is:

Where N represents the total number, p represents a candidate point, circle(p) represents a circle centered on the candidate point p, with a predetermined value as a radius, and x represents any pixel on the circumference circle(p), I (x) denotes the gray value of pixel x, I (p) denotes the gray value candidate point p, ε _d represents the difference between a predetermined gradation threshold. It should be noted that the preset value may be specifically set according to actual needs, and is not specifically limited herein.

Further, in the step S14 of the previous embodiment, the process of performing dense stereo matching on the first image and the second image based on the first group of support points and the second group of support points may specifically include the following steps S141 to S143:

Step S141: Construct a corresponding Delaunay triangular mesh on the first image according to the first set of support points;

Step S142: Calculating the disparity corresponding to the pixel points located in the Delaunay triangular mesh, and obtaining corresponding disparity data;

Step S143: Perform dense stereo matching on the first image and the second image by using the disparity data and the disparity probability model to find support points matching the first set of support points from the second set of support points.

The process of calculating the disparity corresponding to the pixel points in the Delaunay triangular mesh in the above step S142 may specifically include:

Using the preset parallax calculation formula, the parallax corresponding to the pixel points in the Delaunay triangular mesh is calculated; wherein the preset parallax calculation formula is:

Where d _p represents the parallax corresponding to the pixel point p in the Delaunay triangular mesh, (u _p , v _p ) represents the coordinates of the pixel point p, and a, b, c are supported by the Delaunay triangular region where the pixel point p is located. The coefficient obtained by fitting the point plane, h represents the minimum support distance of the pixel point p and the adjacent three support points,

Expressed in the interval

Take a random number.

Further, an embodiment of the present invention further discloses an image blurring method, including the following steps S21 and S22:

Step S21: acquiring a depth map obtained by the depth map creation method disclosed in the foregoing embodiment;

Step S22: performing image blurring processing using the depth map to obtain a blurred image.

Specifically, in the above step S22, the process of performing image blurring processing by using the depth map may specifically include the following steps S221 to S223:

Step S221: determining a focus area on the depth map.

Step S222: using the depth information on the depth map and combining the preset function to obtain the ambiguity coefficient of each pixel in the depth map; wherein, the preset function is:

Indicates the average depth value on the focus area, Z _far represents the maximum depth value on the focus area, Z _near represents the minimum depth value on the focus area, and w represents the adjustment factor.

Step S223: performing image blurring processing on the target pixel point set by using a preset blurring formula to obtain a blurred image; wherein the target pixel point set is a set of pixel points whose depth values are in the range of [Z _near , Z _far ] The default blur formula is:

In the formula, m × n represents the number of pixels in a circle whose coordinates (x, y) are the center of the target pixel and whose radius is the radius of the target pixel, and the target pixel is the target pixel set. One pixel,

Indicates the pixel value of the target pixel after the blurring process, and I _i,j represents the pixel value corresponding to the pixel point (i, j) in the circle before the blurring process.

Based on the foregoing technical solutions, the embodiment of the present invention can better improve the overall blurring effect of the blurred image by suppressing the leakage of the focusing layer to the defocused layer and determining the imaginary coefficient by pixel.

Correspondingly, an embodiment of the present invention further discloses a depth map creation system. Referring to FIG. 2, the system includes:

The image obtaining module 11 is configured to obtain a first image and a second image obtained by the binocular shooting system after capturing the target scene;

The feature point determining module 12 is configured to respectively determine feature points corresponding to the first image and the second image, to obtain a first set of feature points and a second set of feature points;

The support point determining module 13 is configured to respectively determine a support point corresponding to the first image and the second image by using the first set of feature points and the second set of feature points, to obtain the first set of support points and the second set of support points ;

The matching module 14 is configured to perform dense stereo matching on the first image and the second image based on the first set of support points and the second set of support points to determine a disparity between the first image and the second image;

The depth map determining module 15 is configured to determine a depth map corresponding to the target scene by using the parallax.

For a more specific working process of each module described above, reference may be made to the corresponding content disclosed in the foregoing embodiments, and details are not described herein again.

Further, the present invention also discloses an image blurring system, including a depth map acquiring module and an image blurring module; wherein

a depth map obtaining module, configured to acquire a depth map created by the depth map creation system disclosed in the foregoing embodiment;

The image blurring module is configured to perform image blurring processing by using a depth map to obtain a blurred image.

Specifically, the image blurring module may include a focus area determining unit, a blurring coefficient calculating unit, and a blurring processing unit;

a focus area determining unit for determining a focus area on the depth map;

The imaginary coefficient calculation unit is configured to use the depth information on the depth map and combine the preset function to obtain the ambiguity coefficient of each pixel in the depth map; wherein, the preset function is:

Indicates the average depth value on the focus area, Z _far represents the maximum depth value on the focus area, Z _near represents the minimum depth value on the focus area, and w represents the adjustment factor;

The blurring processing unit is configured to perform image blurring processing on the target pixel point set by using a preset blurring formula to obtain a blurred image; wherein the target pixel point set is in the range of [Z _near , Z _far ] The set of pixels, the default blur formula is:

Finally, it should also be noted that in this context, relational terms such as first and second are used merely to distinguish one entity or operation from another entity or operation, and do not necessarily require or imply these entities. There is any such actual relationship or order between operations. Furthermore, the term "comprises" or "comprises" or "comprises" or any other variations thereof is intended to encompass a non-exclusive inclusion, such that a process, method, article, or device that comprises a plurality of elements includes not only those elements but also Other elements, or elements that are inherent to such a process, method, item, or device. An element that is defined by the phrase "comprising a ..." does not exclude the presence of additional equivalent elements in the process, method, item, or device that comprises the element.

The method and system for creating a depth map provided by the present invention and the image blurring method and system are described in detail. The principle and implementation manner of the present invention are described in the following. The description of the above embodiment is only used for the description. To help understand the method of the present invention and its core idea; at the same time, for those skilled in the art, according to the idea of the present invention, there will be changes in specific embodiments and application scopes. It should not be construed as limiting the invention.

Claims

A depth map creation method, comprising:

Obtaining a first image and a second image obtained by the binocular shooting system after shooting the target scene;

Determining feature points corresponding to the first image and the second image, respectively, to obtain a first set of feature points and a second set of feature points;

Using the first set of feature points and the second set of feature points respectively, respectively determining support points corresponding to the first image and the second image to obtain a first set of support points and a second set of support points ;

Performing dense stereo matching on the first image and the second image based on the first set of support points and the second set of support points to determine between the first image and the second image Parallax

A depth map corresponding to the target scene is determined using the parallax.
The depth map creation method according to claim 1, wherein the determining process of the feature points of any image comprises:

Determining candidate points from the image;

Counting the total number of pixel points satisfying the preset condition around the candidate point;

Determining whether the total number is greater than a preset number threshold, and if so, determining that the candidate point is a feature point of the image, and if not, determining that the candidate point is not a feature point of the image.
The depth map creation method according to claim 2, wherein the process of counting the total number of pixel points satisfying the preset condition around the candidate point comprises:

The total number of pixels satisfying the preset condition around the candidate points is counted by using a preset pixel number statistical formula; wherein the predetermined pixel number statistics formula is:

Where N represents the total number, p represents the candidate point, circle(p) represents a circle centered on the candidate point p, with a predetermined value as a radius, and x represents the circumference circle(p) any of a pixel, I (x) denotes the gray value of pixel x, I (p) denotes the gray value of the candidate point p, ε d represents the difference between a predetermined gradation threshold.
The depth map creation method according to any one of claims 1 to 3, wherein said first image and said first image are based on said first set of support points and said second set of support points The process of performing dense stereo matching on two images, including

Constructing a corresponding Delaunay triangular mesh on the first image according to the first set of support points;

Calculating a disparity corresponding to a pixel point located in the Delaunay triangular mesh to obtain corresponding disparity data;

Performing dense stereo matching on the first image and the second image by using the disparity data and a disparity probability model to find out from the second set of support points that the first set of support points are matched Support point.
The depth map creation method according to claim 4, wherein the calculating the parallax corresponding to the pixel points in the Delaunay triangular mesh comprises:

Calculating a parallax corresponding to a pixel point located in the Delaunay triangular mesh by using a preset parallax calculation formula; wherein the preset disparity calculation formula is:

Where d p represents the parallax corresponding to the pixel point p in the Delaunay triangular mesh, (u p , v p ) represents the coordinates of the pixel point p, and a, b, c are the Delaunay triangular regions where the pixel point p is located The coefficient obtained by fitting the support point plane, h represents the minimum support distance of the pixel point p and the adjacent three support points,
Expressed in the interval
Take a random number.
The image blurring method is characterized in that it comprises:

Obtaining a depth map obtained by the method according to any one of claims 1 to 5;

The image is blurred by the depth map to obtain a blurred image.
The image blurring method according to claim 6, wherein the process of performing image blurring processing by using the depth map comprises:

Determining a focus area on the depth map;

Using the depth information on the depth map, and combining the preset function, obtaining a blurring coefficient of each pixel in the depth map; wherein the preset function is:

Wherein, C i represents a blurring coefficient of the i-th pixel point, z i represents a depth value of the i-th pixel point, and f represents a focal length.
Representing an average depth value on the focus area, Z far represents a maximum depth value on the focus area, Z near represents a minimum depth value on the focus area, and w represents an adjustment factor;

Using the preset blurring formula, performing image blurring processing on the target pixel point set to obtain the blurred image; wherein the target pixel point set is a pixel point whose depth value is in the range of [Z near , Z far ] The set, the preset blur formula is:

Wherein m × n represents the number of pixel points in a circle having a coordinate (x, y) of the target pixel as a center and a blur factor of the target pixel as a radius, the target pixel being the Any pixel of the target pixel set,
The pixel value of the target pixel after the blurring process is indicated, and I i,j represents the pixel value corresponding to the pixel point (i, j) in the circle before the blurring process.
A depth map creation system, comprising:

An image acquisition module, configured to acquire a first image and a second image obtained by the binocular shooting system after capturing the target scene;

a feature point determining module, configured to respectively determine feature points corresponding to the first image and the second image, to obtain a first set of feature points and a second set of feature points;

a support point determining module, configured to respectively determine a support point corresponding to the first image and the second image by using the first set of feature points and the second set of feature points, to obtain a first set of supports Point and second set of support points;

a matching module, configured to perform dense stereo matching on the first image and the second image based on the first set of support points and the second set of support points to determine the first image and the first Parallax between two images;

a depth map determining module configured to determine a depth map corresponding to the target scene by using the parallax.
An image blurring system, comprising:

a depth map obtaining module, configured to acquire a depth map created by the depth map creation system according to claim 8;

An image blurring module is configured to perform image blurring processing by using the depth map to obtain a blurred image.
The image blurring system according to claim 9, wherein the image blurring module comprises:

a focus area determining unit, configured to determine a focus area on the depth map;

a blurring coefficient calculation unit, configured to use the depth information on the depth map, and combine with a preset function to obtain a blurring coefficient of each pixel in the depth map; wherein the preset function is:

Wherein, C i represents a blurring coefficient of the i-th pixel point, z i represents a depth value of the i-th pixel point, and f represents a focal length.
Representing an average depth value on the focus area, Z far represents a maximum depth value on the focus area, Z near represents a minimum depth value on the focus area, and w represents an adjustment factor;

a blurring processing unit, configured to perform image blurring processing on the target pixel point set by using a preset blurring formula to obtain the blurred image; wherein the target pixel point set is a depth value in [Z near , Z far a collection of pixels within the range, the preset blur formula is:

Wherein m × n represents the number of pixel points in a circle having a coordinate (x, y) of the target pixel as a center and a blur factor of the target pixel as a radius, the target pixel being the Any pixel of the target pixel set,
The pixel value of the target pixel after the blurring process is indicated, and I i,j represents the pixel value corresponding to the pixel point (i, j) in the circle before the blurring process.