CN114119701A

CN114119701A - Image processing method and device

Info

Publication number: CN114119701A
Application number: CN202111438638.XA
Authority: CN
Inventors: 李佐广
Original assignee: Vivo Mobile Communication Co Ltd
Current assignee: Vivo Mobile Communication Co Ltd
Priority date: 2021-11-29
Filing date: 2021-11-29
Publication date: 2022-03-01

Abstract

The application discloses an image processing method and an image processing device, and belongs to the technical field of image processing. The image processing method comprises the following steps: acquiring a first image and a second image, wherein the first image and the second image are two images of the same shooting scene captured by a first camera and a second camera of the electronic equipment; determining sparse depth maps of the first camera and the second camera according to the first image and the second image; acquiring a dense depth map of an image of a shooting scene; according to the dense depth map, adjusting the depth of pixel points in the sparse depth map to obtain a target depth map; and performing blurring processing on a target image according to the target depth map, wherein the target image is one of the first image and the second image.

Description

Image processing method and device

Technical Field

The present application belongs to the field of image processing technology, and in particular, relates to an image processing method and apparatus.

Background

With the rapid development of communication technology, intelligent electronic devices, such as mobile phones, have become indispensable tools in various aspects of user life. Among them, taking pictures is becoming one of the common functions of mobile phones. In addition, at present, a photo with a background blurring effect can be shot through a mobile phone.

At present, it is a common technique to implement virtual photography based on two cameras. However, for the regions of sparse texture, repetitive texture, dim light, and occlusion, the parallax processing of the two cameras is weak, so that the depth calculation is not accurate, and therefore blurring omission or blurring error is caused.

Therefore, the blurring method based on the two cameras in the prior art has the problems of missing blurring or false blurring.

Disclosure of Invention

An object of the embodiments of the present application is to provide an image processing method and an image processing apparatus, which can solve the problem of missing blurring or false blurring in a blurring method based on two cameras in the prior art.

In a first aspect, an embodiment of the present application provides an image processing method, including:

acquiring a first image and a second image, wherein the first image and the second image are two images of the same shooting scene captured by a first camera and a second camera of the electronic equipment;

determining sparse depth maps of the first camera and the second camera according to the first image and the second image;

obtaining a dense depth map of the image of the shooting scene;

according to the dense depth map, adjusting the depth of pixel points in the sparse depth map to obtain a target depth map;

and performing blurring processing on a target image according to the target depth map, wherein the target image is one of the first image and the second image.

In a second aspect, an embodiment of the present application provides an image processing apparatus, including:

the device comprises an image acquisition module, a storage module and a display module, wherein the image acquisition module is used for acquiring a first image and a second image, and the first image and the second image are two images of the same shooting scene captured by a first camera and a second camera of the electronic equipment;

the sparse depth map acquisition module is used for determining sparse depth maps of the first camera and the second camera according to the first image and the second image;

the dense depth map acquisition module is used for acquiring a dense depth map of the image of the shooting scene;

the depth adjusting module is used for adjusting the depth of pixel points in the sparse depth map according to the dense depth map to obtain a target depth map;

and the blurring processing module is used for blurring a target image according to the target depth map, wherein the target image is one of the first image and the second image.

In a third aspect, embodiments of the present application provide an electronic device, which includes a processor and a memory, where the memory stores a program or instructions executable on the processor, and the program or instructions, when executed by the processor, implement the steps of the method according to the first aspect.

In a fourth aspect, embodiments of the present application provide a readable storage medium, on which a program or instructions are stored, which when executed by a processor implement the steps of the method according to the first aspect.

In a fifth aspect, an embodiment of the present application provides a chip, where the chip includes a processor and a communication interface, where the communication interface is coupled to the processor, and the processor is configured to execute a program or instructions to implement the method according to the first aspect.

In a sixth aspect, embodiments of the present application provide a computer program product, stored on a storage medium, for execution by at least one processor to implement the method according to the first aspect.

In the embodiment of the application, a first image and a second image of the same shooting scene captured by a first camera and a second camera of an electronic device can be obtained, sparse depth maps of the first camera and the second camera are determined according to the first image and the second image, a dense depth map of the image of the shooting scene can also be obtained, so that the depth of pixel points in the sparse depth map is adjusted according to the dense depth map, a target depth map is obtained, and then one of the first image and the second image is subjected to blurring processing according to the target depth map.

Therefore, in the embodiment of the application, the sparse depth map of the two cameras (namely the first camera and the second camera) can be adjusted according to the dense depth map of the image of the shooting scene, so that more accurate depth calculation can be obtained, the problem that the two cameras are weak in processing in sparse texture, repeated texture, dim light and sheltered areas can be solved, the problem of blurring leakage or blurring error can be solved, and the blurring effect is improved.

Drawings

Fig. 1 is a flowchart of an image processing method provided in an embodiment of the present application;

FIG. 2 is a schematic diagram of imaging planes of a first camera and a second camera before stereo rectification in the embodiment of the present application;

fig. 3 is a schematic view of imaging planes of a first camera and a second camera after performing stereo rectification in the embodiment of the present application;

FIG. 4 is a flowchart of an embodiment of an image processing method according to an embodiment of the present disclosure;

fig. 5 is a block diagram of an image processing apparatus according to an embodiment of the present application;

fig. 6 is a block diagram of an electronic device according to an embodiment of the present disclosure;

fig. 7 is a block diagram of another electronic device according to an embodiment of the present disclosure.

Detailed Description

The technical solutions in the embodiments of the present application will be described clearly below with reference to the drawings in the embodiments of the present application, and it is obvious that the described embodiments are some, but not all, embodiments of the present application. All other embodiments that can be derived by one of ordinary skill in the art from the embodiments given herein are intended to be within the scope of the present disclosure.

The terms first, second and the like in the description and in the claims of the present application are used for distinguishing between similar elements and not necessarily for describing a particular sequential or chronological order. It will be appreciated that the data so used may be interchanged under appropriate circumstances such that embodiments of the application may be practiced in sequences other than those illustrated or described herein, and that the terms "first," "second," and the like are generally used herein in a generic sense and do not limit the number of terms, e.g., the first term can be one or more than one. In addition, "and/or" in the specification and claims means at least one of connected objects, a character "/" generally means that a preceding and succeeding related objects are in an "or" relationship.

The image processing method provided by the embodiment of the present application is described in detail below with reference to the accompanying drawings through specific embodiments and application scenarios thereof.

An embodiment of the present application provides an image processing method, as shown in fig. 1, which may include the following steps 101 to 104:

step 101: a first image and a second image are acquired.

The first image and the second image are two images of the same shooting scene captured by a first camera and a second camera of the electronic equipment. The first camera is used for capturing the image of the shooting scene, and the second camera is used for capturing the image of the shooting scene.

In addition, in order to reduce the difference in image content between the first image and the second image, the first image and the second image may be two images of the same shooting scene of the first camera and the second camera.

In addition, an operation control may be set in the photographing setting interface, and when the operation control is in an open state, the image processing method provided in the embodiment of the present application is used to perform blurring processing, that is, steps 101 to 105 are executed; when the operation control is in a closed state, a blurring processing method based on two cameras in the prior art is adopted.

It should be further noted that the image processing method of the embodiment of the present application can be applied not only to the photographing process, but also to the video photographing process.

Step 102: determining sparse depth maps of the first camera and the second camera according to the first image and the second image.

The parallax matching of the first image and the second image is difficult due to sparse texture, repeated texture, dim light, occlusion and the like, namely in the areas of sparse texture, repeated texture, dim light and occlusion, the first image and the second image have poor quality during parallax matching, so that the confidence of the parallax of the areas is relatively low, the parallax values with low confidence are filtered, a parallax image is obtained, and a depth image can be obtained from the parallax image. The depth map obtained by lifting the disparity value is generally referred to as a sparse depth map.

Step 103: obtaining a dense depth map of an image of the capture scene.

Wherein, a dense depth map of the image of the above-mentioned shooting scene may be acquired by a time-of-flight (tof) sensor, which may be disposed on the same plane as the first and second cameras, so that the tof sensor and the first and second cameras capture the image of the same shooting scene.

In addition, in order to reduce the difference in image content between the first image and the second image in the image of the shooting scene captured by the tof sensor, the first camera, the second camera and the tof sensor can be controlled to capture the image of the shooting scene simultaneously.

Further, alternatively, a spot tof (lattice time of flight) sensor may be used in the embodiment of the present application. Wherein, the spot tof sensor has longer measuring distance and stronger anti-illumination capability.

Step 104: and according to the dense depth map, adjusting the depth of a pixel point in the sparse depth map to obtain a target depth map.

As described above, the sparse depth map is obtained by filtering out the disparity map with low confidence parallax, and therefore, the depth of some regions may be lacking in the sparse depth map, and in the embodiment of the present application, a dense depth map of an image of the same shooting scene captured by the tof sensor may also be obtained, so that the depth of a pixel point in the sparse depth map may be adjusted according to the dense depth map to compensate for the depth of the region lacking in the sparse depth map, and thus a more accurate depth map may be obtained, and further, the blurring processing effect may be improved.

Step 105: and performing blurring processing on the target image according to the target depth map.

Wherein the target image is one of the first image and the second image.

In addition, after the target image is subjected to the blurring processing, the blurred image can be displayed on a photographing preview interface, so that a user can view the blurring processing effect. After the blurring process is performed, if a photographing instruction is received, a photograph can be generated of the blurred image.

As can be seen from the foregoing steps 101 to 105, in the embodiment of the present application, a first image and a second image of the same shooting scene captured by a first camera and a second camera of an electronic device can be obtained, sparse depth maps of the first camera and the second camera can be determined according to the first image and the second image, and a dense depth map of an image of the shooting scene can also be obtained, so that depths of pixel points in the sparse depth map are adjusted according to the dense depth map, a target depth map is obtained, and then one of the first image and the second image is blurred according to the target depth map.

Optionally, before the acquiring the first image and the second image, the method further includes:

and adjusting the imaging planes of the first camera and the second camera to be in the same plane.

In the three-dimensional imaging principle, the depth information of an object point needs to be estimated through two images, and the same object point needs to be accurately matched in the two images, so that the object depth can be calculated according to the position relation of the object point in the two images. In order to reduce the amount of matching calculation, the embodiments of the present application may further adjust the imaging planes of the first camera and the second camera to be in the same plane, that is, perform stereo rectification on the first camera and the second camera. For example, the imaging planes of the first camera and the second camera before the stereo correction are performed are shown in fig. 2, and the imaging planes of the first camera and the second camera after the stereo correction are shown in fig. 3, where Pl denotes the imaging plane of the first camera and Pr denotes the imaging plane of the second camera in fig. 2 and 3.

Optionally, the obtaining a dense depth map of the image of the shooting scene includes:

and adopting a Fast Bilateral Solver algorithm (Fast Bilateral Solver) to perform densification on the image of the shooting scene captured by the flight time sensor to obtain the dense depth map.

The fast Bilateral Solver algorithm is accelerated on the basis of the binary Solver, namely, a pixel point is projected to a Bilateral space (binary space) to reduce the calculated amount, then the binary space is subjected to filtering processing, and then data is projected to the pixel space.

Specifically, the specific process of performing densification on the image of the shooting scene captured by the time-of-flight sensor by using the fast bilateral solver algorithm may be as follows:

first, pixel values of an image of a photographic scene captured by a tof sensor are mapped onto small-scale Grid (Grid) or Grid (lattice) vertices, thereby compressing the data. For example, a YUV image with width W and height H is projected onto Grid to form a five-dimensional (YUVXY) point, where YUV represents color and XY represents position coordinates.

Next, positional compression is performed, for example, for an image with a width W and a height H, a Block (for example, a Block of 8 × 8) can be represented by one position. The YUV colors are compressed in each Block, and the YUV values can be divided by 8, so that the number of repeated points in the Block is increased, the number of the blocks is increased after the repeated points are removed, and the data volume (namely, the calculated data points) is greatly reduced.

And thirdly: the vertices (vertices) are filtered (blu).

And finally: and mapping each filtered vertex back to the pixel space to obtain a dense depth map.

Optionally, the determining the sparse depth maps of the first camera and the second camera according to the first image and the second image includes:

acquiring a disparity map of the first camera and the second camera and a confidence map of the disparity map according to the first image and the second image;

and acquiring the sparse depth map according to the disparity map and the confidence map of the disparity map.

The disparity maps of the first camera and the second camera can be obtained by adopting a block matching method. Namely, the following steps H1 to H4 are respectively performed for each pixel point in the first image:

step H1: taking a target pixel point in the first image as a center, and acquiring a first window (for example, a 9 × 9 window expanded from the target pixel point in four directions, namely, up, down, left and right directions) where the target pixel point is located;

step H2: determining a second window corresponding to the first window position in the second image (namely, a 9 × 9 window which is expanded by taking a pixel point corresponding to the target pixel point position in the second image as a center);

step H3: moving the second window one pixel point to the left and right respectively, and calculating the absolute value of the difference between the gray value of the v-th pixel point in the first window and the gray value of the v-th pixel point in the second window when v is each integer from 1 to W when the second window moves once, so as to be used as a second parameter of the v-th pixel point in the first window, and further calculating the sum of the second parameters of the 1-W pixel points, so as to be used as a third parameter of the moved second window; wherein W represents the total number of pixel points in the first window;

step H4: and finding the central pixel point of the second window with the minimum third parameter, wherein the difference between the abscissa of the pixel point and the abscissa of the target pixel point is the parallax of the target pixel point.

The target pixel point is one of the pixel points in the first image.

In addition, after the parallax of each pixel point is obtained, the sparse depth map can be obtained according to the confidence map and the parallax map.

In addition, any method in the prior art may be adopted to obtain the confidence map of the disparity map, which is not described herein again.

Optionally, the obtaining the sparse depth map according to the disparity map and the confidence map of the disparity map includes:

according to the confidence map, acquiring a first pixel point with the confidence degree smaller than a first preset threshold value in the disparity map, and setting the depth of the first pixel point as a first preset value;

determining the depth of a second pixel point according to a binocular triangulation principle and the parallax of the second pixel point, wherein the second pixel point is a pixel point except the first pixel point in the parallax map;

and obtaining the sparse depth map according to the depth of the first pixel point and the depth of the second pixel point.

Therefore, in the embodiment of the application, according to the confidence map, a first pixel point with a lower confidence (that is, the confidence is smaller than a first preset threshold) may be obtained, and then the parallax of the first pixel point is set to a first preset value (for example, -1), that is, the pixel point with the low confidence is marked by using the first preset value; and calculating the depth of the second pixel point according to the parallax of the second pixel point (namely the pixel point with high confidence level, namely the pixel point with the confidence level larger than or equal to the first preset threshold) by adopting a binocular triangulation principle, thereby obtaining a sparse depth map.

Optionally, the adjusting the depth of the pixel point in the sparse depth map according to the dense depth map to obtain a target depth map includes:

acquiring a first region and a second region in the sparse depth map, wherein the depth of a pixel point in the first region is a first preset value, the absolute value of the difference between the depth of the pixel point in the second region and the depth of a third pixel point is greater than or equal to a second preset threshold value, and the third pixel point is a pixel point corresponding to the pixel point position in the second region in the dense depth map;

setting the depth of a pixel point of the first region in the sparse depth map as the depth of a fourth pixel point, wherein the fourth pixel point is a pixel point corresponding to the position of the pixel point of the first region in the dense depth map;

when k is each integer from 1 to M, calculating the depth of a kth pixel point in the second area, and obtaining the depth of a kth fifth pixel point by linear weighted sum of the depth of the kth pixel point and the depth of the kth third pixel point, wherein the fifth pixel point is a pixel point corresponding to the position of the pixel point in the second area in the target depth map, and M represents the total number of the pixel points in the second area.

In the sparse depth map, the depth of the pixel point is a first preset value, which indicates that the depth of the pixel point is lacked, namely, in the process of acquiring the sparse depth map, the confidence of the parallax of the pixel point is low, so that the depth is not calculated according to the parallax of the pixel point, but the depth of the pixel point is directly set to be the first preset value. That is, in the embodiment of the present application, the pixel lacking the depth is marked by using the first preset value.

Therefore, in the sparse depth map, the depth of the pixel points in the first region is a first preset value, and the pixel points in the first region lack depth, so that the depth of the pixel points in the region corresponding to the position of the first region in the dense depth map can be assigned to the corresponding pixel points in the first region.

In addition, the absolute value of the difference between the depth of the pixel point in the second region and the depth of the pixel point corresponding to the position of the pixel point in the dense depth map is greater than a second preset threshold, which indicates that the difference between the depth of the pixel point in the second region and the depth of the pixel point in the region corresponding to the position of the second region in the dense depth map is greater, that is, there is a region with a greater depth difference in the sparse depth map and the dense depth map.

Specifically, in the embodiment of the present application, when the depth of the sparse depth map is adjusted according to the dense depth map, the sparse depth map may be counted, the depth is a value of-1, and if an area with a depth of-1 is greater than a certain range, the area may be considered to be a sparse texture or a dim light area. The scene is considered sparse depth missing. The pixel of the area position is marked as 1, that is, the pixel point marked as 1 constitutes the first area.

And the depth of the pixel point at the corresponding position in the sparse depth map and the dense depth map can be subtracted to obtain an absolute value, and when the absolute value is greater than a second preset threshold, the pixel point is marked as 2, that is, the pixel point marked as 2 forms the second region.

Further, for the pixel point labeled 1: and replacing the depth of the pixel marked as 1 in the sparse depth map by the depth of the pixel point at the corresponding position in the depth of the dense depth map, namely filling the corresponding depth of the dense depth map to the region marked as 1 in the sparse depth map.

For the pixel labeled 2: fusing the depth of the pixel point marked as 2 in the sparse depth map with the depth of the pixel point at the corresponding position in the dense depth map by linear weighting, such as dfusion_k＝d1_k*a+d2_k(a-1), wherein d1_kRepresenting the depth of the k-th pixel labeled 2 in the sparse depth map, d2_kRepresenting the depth of the k-th pixel point marked as 2 in the dense depth map, a being a predetermined constant between 0 and 1, dfusion_kAnd the fused depth of the k-th pixel point marked as 2 in the sparse depth map is represented.

Optionally, before the obtaining the first region and the second region in the sparse depth map, the adjusting the depth of the pixel point in the sparse depth map according to the dense depth map further includes:

determining a first connected region of the sparse depth map and a second connected region of the dense depth map;

when i is each integer from 1 to N, acquiring a region with the largest area in an overlapping region of the ith first connected region and each first target region, and determining the region as the ith candidate region, wherein N is the number of the first connected regions, and the first target region is a region corresponding to the position of the second connected region in the dense depth map;

obtaining a first average value of depths of pixel points in an ith candidate region in the sparse depth map and a second average value of depths of pixel points in an ith second target region in the dense depth map, wherein the ith second target region is a region corresponding to the ith candidate region in the dense depth map;

calculating the ratio of the first average value to the second average value, and determining the ratio as an ith target parameter;

dividing the depth of the pixel points in the ith candidate region in the sparse depth map by the ith target parameter to obtain the adjusted depth of the pixel points in the ith candidate region in the sparse depth map;

and adjusting the depth of the pixel points outside the candidate region in the sparse depth map by adopting an interpolation method according to the adjusted depth of the pixel points in each candidate region in the sparse depth map.

For example, three connected regions L1, L2, and L3 exist in the sparse depth map, and three connected regions L4, L5, and L6 exist in the dense depth map, in the embodiment of the present application, it is necessary to determine a region corresponding to the position of L4 in the sparse depth map, and overlap regions with L1, L2, and L3, respectively, and if two overlapping regions C1 and C2 exist, a region with the largest area (i.e., the largest number of pixels) is selected as a first candidate region; similarly, a region corresponding to the position of L5 in the sparse depth map needs to be determined, and the region is respectively overlapped with the overlapping regions of L1, L2 and L3, and if there is one overlapping region of C3, the region is a second candidate region; similarly, regions corresponding to the positions of L6 in the dense depth map, i.e., overlapping regions with L1, L2, and L3, respectively, need to be determined, and if there are two overlapping regions of C4 and C5, a region with the largest area (i.e., the largest number of pixels) is selected as the third candidate region.

After the first to three candidate regions are obtained, for each candidate region, a first average value of depths of pixel points in the candidate region in the sparse depth map and a second average value of depths of pixel points in a region of a corresponding position of the candidate region in the dense depth map need to be calculated, and a ratio of the first average value to the second average value is calculated to serve as a target parameter.

For example, the obtained target parameters are S1, S2, S3, where S1 corresponds to the first candidate region, S2 corresponds to the second candidate region, and S3 corresponds to the third candidate region. Dividing the depth of each pixel point belonging to the first candidate region in the sparse depth map by S1; dividing the depth of each pixel point belonging to the second candidate region in the sparse depth map by S2; and for each pixel point in the third candidate region in the sparse depth map, dividing the depth of each pixel point by S3, and completing the process of adjusting the depth of the pixel point in the candidate region in the sparse depth map according to the target parameter. Thus, a depth adjustment relationship table, i.e., a correspondence relationship table of the depth before adjustment and the depth after adjustment, can be obtained.

Then, the depth of the pixel point outside the candidate region in the sparse depth map can be adjusted by adopting an interpolation method according to the corresponding relation table. For example, the depth of a certain pixel point outside the candidate area before adjustment is 1.2 meters, and the relationship table has a correspondence between 1 meter (before adjustment) and 0.8 meter (after adjustment) and a correspondence between 2 meters (before adjustment) and 1.5 meters (after adjustment), and 1.2 meters is between 1 meter and 2 meters, a value between 0.8 meter and 1.5 meters can be selected as the depth of the pixel point after adjustment.

Optionally, after the depth of the pixel point in the sparse depth map is adjusted according to the dense depth map to obtain a target depth map, the method further includes:

obtaining a second connected region of the dense depth map;

when j takes each integer from 1 to L, calculating the average value of pixel points of a jth second connected region and the average value of the depth of the pixel points of a jth third target region, wherein the jth third target region is a region corresponding to the jth second connected region in the sparse depth map;

calculating the average value of the pixel points of the jth second connected region and the absolute value of the difference value of the average value of the pixel points of the jth third target region to serve as a first parameter of the jth third target region;

acquiring a third target area with the first parameter being greater than or equal to a third preset threshold value to serve as a fourth target area;

and filtering pixel points in the target depth map corresponding to the pixel point positions in the fourth target area.

For example, if three connected regions L4, L5, and L6 exist in the dense depth map, it is necessary to determine regions corresponding to positions of L4, L5, and L6 in the sparse depth map, for example, L7, L8, and L9, and then calculate an absolute value of a difference between an average value of depths of pixels in L4 and an average value of depths of pixels in L6 as a first parameter of L7; similarly, the first parameters of L8 and L9 may be obtained; then, a region with the first parameter greater than or equal to a third preset threshold value can be selected, and pixel points in a region corresponding to the position of the region in the target depth map are filtered out.

The obtained first parameter of the fourth target region is greater than or equal to the third preset threshold, and the depth of the pixel points in the regions in the sparse depth map is incorrect, so that the pixel points need to be filtered, the accuracy of the finally obtained target depth map can be higher, and the blurring effect can be further improved.

Optionally, before performing blurring processing on the target image according to the target depth map, the method further includes:

and carrying out depth filtering processing on the target depth map.

Wherein the target depth map may be depth filtered in conjunction with RGB values of images captured by the dual cameras (i.e., the first camera and the second camera). For example, if a human image is taken, the segmentation map of the human image can be used as a constraint of dense growth in depth filtering.

In addition, the depth filtering may be, for example, a guided edge preserving filtering. The guided edge-preserving filtering refers to a special filter which can effectively preserve edge information in an image in a filtering process.

Optionally, the blurring processing on the target image according to the target depth map includes:

performing depth segmentation on the target depth map to obtain a depth range of a clear region;

determining a foreground area and a background area according to the depth range of the clear area;

respectively calculating the virtual radius of each pixel point of the foreground region and the virtual radius of each pixel point of the background region according to the target depth map to obtain a virtual radius map;

and performing blurring processing on the target image according to the blurring radius map.

The position of the focus point on the image may be determined first, then the depth value of the focus point position in the target depth map is obtained, and then the depth range of the clear region is obtained by searching the focus depth value table, for example, may be represented by near, far, near representing the lower limit value of the depth range of the clear region, and far representing the upper limit value of the depth range of the clear region. The focus depth value table is a depth of field range correspondence table at different distances obtained by simulating a single-lens reflex aperture.

In addition, in the target depth map, a region formed by pixel points with the depth smaller than near belongs to a foreground region; and the region formed by the pixel points with the depth greater than far belongs to the background region.

In addition, the blurring radius of the pixel points in the foreground region is (near-d) max _ key _ r/background _ length; blurring radius of a pixel point in the background area is (d-far) max _ bokeh _ r/formgo _ length; wherein d represents the depth of a pixel point in the target depth map, max _ boot _ r represents the maximum blurring radius corresponding to the selected aperture (e.g., F2.0), background _ length represents the background blurring distance, background _ length is near-minutia, Foreground _ length represents the foreground blurring distance for foreground _ length is maxdist-far, minidisk represents the minimum depth in the target depth map, and maxdist is the maximum depth in the target depth map.

Therefore, in the embodiment of the present application, the blurring radius is determined by the depth d of the pixel, the depth range [ near, far ] of the clear region, the aperture size, the foreground blurring distance, and the background blurring distance. The blurring radius is the radius of a circle of the blurring or blurred image, and the smaller the blurring radius is, the lighter the blurring degree is, and the larger the blurring radius is, the heavier the blurring degree is.

In addition, a circular filter may be used to blur the target image according to the aforementioned blurring radius map.

Optionally, before obtaining the dense depth map of the image of the shooting scene, the method further includes:

and converting the image coordinate system of the time sensor into the image coordinate system of the first camera or the second camera, wherein the coordinate systems of the first image and the second image are the same.

The tof sensor is registered to an image coordinate system of the first camera or the second camera so as to obtain a dense depth map, and the depth of pixel points of the sparse depth map is adjusted according to the dense depth map.

As described above, a specific implementation of the image processing method according to the embodiment of the present application can be shown in fig. 4. The image processing method is applied to electronic equipment, and the electronic equipment is provided with a first camera, a second camera and a tof sensor which are located on the same plane. This includes steps 401 to 410 as described below.

Step 401: adjusting the imaging planes of the first camera and the second camera to be in the same plane;

step 402: respectively acquiring images of the same shooting scene captured by a first camera, a second camera and a tof sensor at the same time;

step 403: determining sparse depth maps of the first camera and the second camera according to a first image captured by the first camera and a second image captured by the second camera, wherein the depth of a pixel point with the reliability lower than a first preset threshold value in the sparse depth maps is a first preset value, and in addition, please refer to the foregoing description for the process of specifically obtaining the sparse depth maps, which is not repeated herein;

step 404: converting the image coordinate of the tof sensor into an image coordinate system of the first camera or the second camera, wherein the coordinate systems of the first image and the second image are the same;

step 405: obtaining a dense depth map of an image of a photographic scene captured by a tof sensor;

step 406: performing overlapping calculation on the sparse depth map and the dense depth map, namely determining a first connected region of the sparse depth map and a second connected region of the dense depth map, so that when i is each integer from 1 to N, a region with the largest area in the overlapping region of the ith first connected region and each first target region is obtained and determined as an ith candidate region, N is the number of the first connected regions, and the first target region is a region corresponding to the position of the second connected region in the dense depth map;

step 407: performing lifting proportion calculation, namely when i is each integer from 1 to N, acquiring a first average value of depths of pixel points in an ith candidate region in the sparse depth map and a second average value of depths of pixel points in an ith second target region in the dense depth map, so as to calculate a ratio of the first average value to the second average value, and determining the ratio as an ith target parameter, wherein the ith second target region is a region corresponding to the ith candidate region in the dense depth map;

step 408: pulling up the depth of the sparse depth map, namely when i is each integer from 1 to N, respectively dividing the depth of the pixel points in the ith candidate region in the sparse depth map by the ith target parameter to obtain the adjusted depth of the pixel points in the ith candidate region in the sparse depth map, and adjusting the depth of the pixel points out of the candidate region in the sparse depth map according to the adjusted depth of the pixel points in each candidate region in the sparse depth map by adopting an interpolation method;

step 409: determining a region to be fused in a sparse depth map, namely acquiring a first region and a second region in the sparse depth map, wherein the depth of a pixel point in the first region is a first preset value, the absolute value of the difference between the depth of the pixel point in the second region and the depth of a third pixel point is greater than or equal to a second preset threshold value, and the third pixel point is a pixel point corresponding to the position of the pixel point in the second region in the dense depth map;

step 410: the method comprises the steps of fusing a sparse depth map and a dense depth map aiming at a region to be fused to obtain a target depth map, namely setting the depth of a pixel point of a first region in the sparse depth map as the depth of a fourth pixel point, wherein the fourth pixel point is the pixel point corresponding to the pixel point position of the first region in the dense depth map; when k is an integer from 1 to M, calculating the depth of a kth pixel point in the second area, and obtaining the depth of a kth fifth pixel point by linear weighted sum of the depth of the kth pixel point and the depth of a kth third pixel point, wherein the fifth pixel point is a pixel point in the target depth map corresponding to the position of the pixel point in the second area, and M represents the total number of the pixel points in the second area;

step 411: filtering the error depth in the target depth map, namely when j is each integer from 1 to L, calculating the average value of pixel points of a jth second connected region and the average value of the depth of the pixel points of a jth third target region, so as to calculate the average value of the pixel points of the jth second connected region and the absolute value of the difference value of the average value of the pixel points of the jth third target region, wherein the absolute value is used as a first parameter of the jth third target region, and further the third target region of which the first parameter is greater than or equal to a third preset threshold is obtained and is used as a fourth target region, and filtering the pixel points corresponding to the pixel point positions in the fourth target region in the target depth map, wherein the jth third target region is a region corresponding to the jth second connected region position in the sparse depth map;

step 412: carrying out depth filtering processing on the target depth map;

step 413: blurring the first image or the second image according to the filtered target depth map, namely performing depth segmentation on the target depth map to obtain a depth range of a clear region, determining a foreground region and a background region according to the depth range of the clear region, respectively calculating a blurring radius of each pixel point of the foreground region and a blurring radius of each pixel point of the background region according to the target depth map to obtain a blurring radius map, and further blurring the first image or the second image according to the blurring radius map. For a specific method for calculating the virtual radius, please refer to the above description, and further details are omitted here.

Therefore, according to the embodiment of the application, the sparse depth map of the two cameras (namely the first camera and the second camera) can be adjusted according to the dense depth map of the image of the shooting scene captured by the time flight sensor, so that more accurate depth calculation can be obtained, the problem that the two cameras are weak in processing of sparse texture, repeated texture, dim light and shielding areas can be solved, the problem of leakage blurring or false blurring can be solved, and the blurring effect is improved.

The embodiment of the present application may also be used to implement different blurring strategies, for example, when a person is far away from a camera, the person and the background are not distinguished obviously, and the blurring method of the embodiment of the present application may be adopted to perform blurring on the background region with weak blurring strength.

In addition, it should be noted that, in this document, a certain region in one image corresponds to a position in another image, that is, a certain region in one image is mapped to a region in another image.

In the blurring method provided by the embodiment of the application, the execution subject may be an image processing apparatus. The image processing apparatus provided in the embodiment of the present application is described with an example in which an image processing apparatus executes an image processing method.

An embodiment of the present application further provides an image processing apparatus, and as shown in fig. 5, the image processing apparatus may include the following modules:

an image obtaining module 501, configured to obtain a first image and a second image, where the first image and the second image are two images of a same shooting scene captured by a first camera and a second camera of an electronic device;

a sparse depth map obtaining module 502, configured to determine sparse depth maps of the first camera and the second camera according to the first image and the second image;

a dense depth map acquisition module 503, configured to acquire a dense depth map of the image of the shooting scene;

a depth adjusting module 504, configured to adjust depths of pixel points in the sparse depth map according to the dense depth map to obtain a target depth map;

a blurring module 505, configured to perform blurring processing on a target image according to the target depth map, where the target image is one of the first image and the second image.

Optionally, the apparatus further comprises:

and the imaging plane adjusting module is used for adjusting the imaging planes of the first camera and the second camera to be in the same plane.

Optionally, the dense depth map obtaining module 503 is specifically configured to:

and adopting a rapid bilateral solver algorithm to perform densification on the image of the shooting scene captured by the flight time sensor to obtain the dense depth map.

Optionally, the sparse depth map obtaining module 502 includes:

a reference image obtaining submodule, configured to obtain a disparity map of the first camera and the second camera and a confidence map of the disparity map according to the first image and the second image;

and the sparse depth map acquisition sub-module is used for acquiring the sparse depth map according to the disparity map and the confidence map of the disparity map.

Optionally, the sparse depth map obtaining sub-module is specifically configured to:

Optionally, the depth adjusting module 504 includes:

a first region obtaining submodule, configured to obtain a first region and a second region in the sparse depth map, where a depth of a pixel in the first region is a first preset value, an absolute value of a difference between the depth of the pixel in the second region and a depth of a third pixel is greater than or equal to a second preset threshold, and the third pixel is a pixel in the dense depth map corresponding to a pixel in the second region;

a first adjusting submodule, configured to set a depth of a pixel point in the first region in the sparse depth map to a depth of a fourth pixel point, where the fourth pixel point is a pixel point in the dense depth map corresponding to a pixel point position in the first region;

and the second adjusting submodule is used for calculating the depth of a kth pixel point in the second area and obtaining the depth of a kth fifth pixel point by linear weighted sum of the depth of the kth pixel point and the depth of the kth third pixel point when k is each integer from 1 to M, wherein the fifth pixel point is a pixel point corresponding to the position of the pixel point in the second area in the target depth map, and M represents the total number of the pixel points in the second area.

Optionally, the depth adjusting module 504 further includes:

a second region acquisition submodule for determining a first connected region of the sparse depth map and a second connected region of the dense depth map;

a third region obtaining sub-module, configured to, when i is each integer from 1 to N, obtain a region with a largest area in an overlapping region between an ith first connected region and each first target region, and determine the region as an ith candidate region, where N is the number of the first connected regions, and the first target region is a region in the dense depth map corresponding to the second connected region;

the first calculation submodule is used for obtaining a first average value of depths of pixel points in an ith candidate region in the sparse depth map and a second average value of depths of pixel points in an ith second target region in the dense depth map, wherein the ith second target region is a region corresponding to the ith candidate region in the dense depth map;

the second calculation submodule is used for calculating the ratio of the first average value to the second average value and determining the ratio as the ith target parameter;

the third adjusting submodule is used for dividing the depth of the pixel points in the ith candidate region in the sparse depth map by the ith target parameter to obtain the adjusted depth of the pixel points in the ith candidate region in the sparse depth map;

and the fourth adjusting submodule is used for adjusting the depth of the pixel points outside the candidate region in the sparse depth map by adopting an interpolation method according to the adjusted depth of the pixel points in each candidate region in the sparse depth map.

Optionally, the apparatus further comprises a first filtering module configured to:

obtaining a second connected region of the dense depth map;

Optionally, the apparatus further comprises:

and the second filtering module is used for carrying out depth filtering processing on the target depth map.

Optionally, the blurring processing module 505 is specifically configured to:

Optionally, the apparatus further comprises:

and the coordinate conversion module is used for converting the image coordinate system of the time sensor into the image coordinate system of the first camera or the second camera, wherein the coordinate systems of the first image and the second image are the same.

As can be seen from the above description, in the embodiment of the present application, a first image and a second image of the same shooting scene captured by a first camera and a second camera of an electronic device can be obtained, sparse depth maps of the first camera and the second camera can be determined according to the first image and the second image, and a dense depth map of an image of the shooting scene can also be obtained, so that depths of pixel points in the sparse depth map are adjusted according to the dense depth map, a target depth map is obtained, and then one of the first image and the second image is subjected to blurring processing according to the target depth map.

The image processing apparatus in the embodiment of the present application may be an electronic device, or may be a component in an electronic device, such as an integrated circuit or a chip. The electronic device may be a terminal, or may be a device other than a terminal. The electronic Device may be, for example, a Mobile phone, a tablet computer, a notebook computer, a palm top computer, a vehicle-mounted electronic Device, a Mobile Internet Device (MID), an Augmented Reality (AR)/Virtual Reality (VR) Device, a robot, a wearable Device, an ultra-Mobile personal computer (UMPC), a netbook or a Personal Digital Assistant (PDA), and the like, and may also be a server, a Network Attached Storage (NAS), a Personal Computer (PC), a Television (TV), a teller machine, a self-service machine, and the like, and the embodiments of the present application are not particularly limited.

The image processing apparatus in the embodiment of the present application may be an apparatus having an operating system. The operating system may be an Android (Android) operating system, an ios operating system, or other possible operating systems, and embodiments of the present application are not limited specifically.

The image processing apparatus provided in the embodiment of the present application can implement each process implemented in the method embodiment of fig. 1, and is not described here again to avoid repetition.

Optionally, as shown in fig. 6, an electronic device 600 is further provided in an embodiment of the present application, and includes a processor 601, a memory 602, and a program or an instruction stored in the memory 602 and executable on the processor 601, where the program or the instruction is executed by the processor 601 to implement each process of the above-mentioned embodiment of the image processing method, and can achieve the same technical effect, and in order to avoid repetition, details are not repeated here.

It should be noted that the electronic device in the embodiment of the present application includes the mobile electronic device and the non-mobile electronic device described above.

Fig. 7 is a schematic diagram of a hardware structure of an electronic device implementing an embodiment of the present application.

The electronic device 700 includes, but is not limited to: a radio frequency unit 701, a network module 702, an audio output unit 703, an input unit 704, a sensor 705, a display unit 706, a user input unit 707, an interface unit 708, a memory 709, and a processor 710.

Those skilled in the art will appreciate that the electronic device 700 may also include a power supply (e.g., a battery) for powering the various components, and the power supply may be logically coupled to the processor 710 via a power management system, such that the functions of managing charging, discharging, and power consumption may be performed via the power management system. The electronic device structure shown in fig. 7 does not constitute a limitation of the electronic device, and the electronic device may include more or less components than those shown, or combine some components, or arrange different components, and thus, the description is omitted here.

In the embodiment of the present application, the sensor 705 includes a first camera, a second camera, and a time-of-flight sensor.

Additionally, processor 710 is configured to perform the following process:

obtaining a dense depth map of the image of the shooting scene;

Optionally, before acquiring the first image and the second image, the processor 710 is further configured to:

Optionally, when the processor 710 obtains the dense depth map of the image of the shooting scene, it is specifically configured to:

Optionally, when determining the sparse depth maps of the first camera and the second camera according to the first image and the second image, the processor 710 is specifically configured to:

Optionally, when the processor 710 acquires the sparse depth map according to the disparity map and the confidence map of the disparity map, the processor is specifically configured to:

Optionally, when the processor 710 adjusts the depth of the pixel point in the sparse depth map according to the dense depth map to obtain the target depth map, the processor is specifically configured to:

when k is an integer from 7 to M, calculating the depth of a kth pixel point in the second area, and obtaining the depth of a kth fifth pixel point by linear weighted sum of the depth of the kth pixel point and the depth of the kth third pixel point, wherein the fifth pixel point is a pixel point corresponding to the position of the pixel point in the second area in the target depth map, and M represents the total number of the pixel points in the second area.

Optionally, when the processor 710 adjusts the depth of the pixel point in the sparse depth map according to the dense depth map, the processor is further configured to:

when i is each integer from 7 to N, acquiring a region with the largest area in the overlapping region of the ith first connected region and each first target region, and determining the region as the ith candidate region, wherein N is the number of the first connected regions, and the first target region is a region corresponding to the position of the second connected region in the dense depth map;

Optionally, after the processor 710 adjusts the depth of the pixel point in the sparse depth map according to the dense depth map to obtain a target depth map, the processor is further configured to:

obtaining a second connected region of the dense depth map;

when j takes each integer from 7 to L, calculating the average value of pixel points of a jth second connected region and the average value of the depth of the pixel points of a jth third target region, wherein the jth third target region is a region corresponding to the jth second connected region in the sparse depth map;

Optionally, before performing blurring processing on the target image according to the target depth map, the processor is further configured to:

and carrying out depth filtering processing on the target depth map.

Optionally, when performing blurring processing on the target image according to the target depth map, the processor 710 is specifically configured to:

Optionally, before acquiring the dense depth map of the image of the shooting scene, the processor 710 is further configured to:

It should be understood that in the embodiment of the present application, the input Unit 704 may include a Graphics Processing Unit (GPU) 7041 and a microphone 7042, and the Graphics Processing Unit 7041 processes image data of still pictures or videos obtained by an image capturing device (e.g., a camera) in a video capturing mode or an image capturing mode. The display unit 706 may include a display panel 7061, and the display panel 7061 may be configured in the form of a liquid crystal display, an organic light emitting diode, or the like. The user input unit 707 includes at least one of a touch panel 7071 and other input devices 7072. The touch panel 7071 is also referred to as a touch screen. The touch panel 7071 may include two parts of a touch detection device and a touch controller. Other input devices 7072 may include, but are not limited to, a physical keyboard, function keys (e.g., volume control keys, switch keys, etc.), a trackball, a mouse, and a joystick, which are not described in detail herein.

The memory 709 may be used to store software programs as well as various data. The memory 709 may mainly include a first storage area for storing a program or an instruction and a second storage area for storing data, wherein the first storage area may store an operating system, an application program or an instruction (such as a sound playing function, an image playing function, and the like) required by at least one function, and the like. Further, the memory 709 may include volatile memory or nonvolatile memory, or the memory 709 may include both volatile and nonvolatile memory. The non-volatile memory may be a Read-only memory (ROM), a programmable Read-only memory (PROM), an erasable programmable Read-only memory (erasabprom, EPROM), an electrically erasable programmable Read-only memory (EEPROM), or a flash memory. The volatile Memory may be a Random Access Memory (RAM), a Static Random Access Memory (Static RAM, SRAM), a Dynamic Random Access Memory (Dynamic RAM, DRAM), a Synchronous Dynamic Random Access Memory (Synchronous DRAM, SDRAM), a Double Data Rate Synchronous Dynamic Random Access Memory (Double Data Rate SDRAM, ddr SDRAM), an Enhanced Synchronous SDRAM (ESDRAM), a Synchronous DRAM (SLDRAM), and a Direct Memory bus RAM (DRRAM). The memory 709 in the embodiments of the present application includes, but is not limited to, these and any other suitable types of memory.

Processor 710 may include one or more processing units; optionally, the processor 710 integrates an application processor, which primarily handles operations related to the operating system, user interface, and applications, and a modem processor, which primarily handles wireless communication signals, such as a baseband processor. It will be appreciated that the modem processor described above may not be integrated into processor 710.

The embodiment of the present application further provides a readable storage medium, where a program or an instruction is stored on the readable storage medium, and when the program or the instruction is executed by a processor, the program or the instruction implements each process of the embodiment of the image processing method, and can achieve the same technical effect, and in order to avoid repetition, details are not repeated here.

The processor is the processor in the electronic device described in the above embodiment. The readable storage medium includes a computer readable storage medium, such as a computer read only memory ROM, a random access memory RAM, a magnetic or optical disk, and the like.

The embodiment of the present application further provides a chip, where the chip includes a processor and a communication interface, the communication interface is coupled to the processor, and the processor is configured to execute a program or an instruction to implement each process of the embodiment of the image processing method, and can achieve the same technical effect, and the details are not repeated here to avoid repetition.

It should be understood that the chips mentioned in the embodiments of the present application may also be referred to as system-on-chip, system-on-chip or system-on-chip, etc.

Embodiments of the present application provide a computer program product, where the program product is stored in a storage medium, and the program product is executed by at least one processor to implement the processes of the foregoing embodiments of the image processing method, and achieve the same technical effects, and in order to avoid repetition, details are not repeated here.

It should be noted that, in this document, the terms "comprises," "comprising," or any other variation thereof, are intended to cover a non-exclusive inclusion, such that a process, method, article, or apparatus that comprises a list of elements does not include only those elements but may include other elements not expressly listed or inherent to such process, method, article, or apparatus. Without further limitation, an element defined by the phrase "comprising an … …" does not exclude the presence of other like elements in a process, method, article, or apparatus that comprises the element. Further, it should be noted that the scope of the methods and apparatus of the embodiments of the present application is not limited to performing the functions in the order illustrated or discussed, but may include performing the functions in a substantially simultaneous manner or in a reverse order based on the functions involved, e.g., the methods described may be performed in an order different than that described, and various steps may be added, omitted, or combined. In addition, features described with reference to certain examples may be combined in other examples.

Through the above description of the embodiments, those skilled in the art will clearly understand that the method of the above embodiments can be implemented by software plus a necessary general hardware platform, and certainly can also be implemented by hardware, but in many cases, the former is a better implementation manner. Based on such understanding, the technical solutions of the present application may be embodied in the form of a computer software product, which is stored in a storage medium (such as ROM/RAM, magnetic disk, optical disk) and includes instructions for enabling a terminal (such as a mobile phone, a computer, a server, or a network device) to execute the method according to the embodiments of the present application.

While the present embodiments have been described with reference to the accompanying drawings, it is to be understood that the invention is not limited to the precise embodiments described above, which are meant to be illustrative and not restrictive, and that various changes may be made therein by those skilled in the art without departing from the spirit and scope of the invention as defined by the appended claims.

Claims

1. An image processing method, characterized in that the method comprises:

obtaining a dense depth map of the image of the shooting scene;

2. The method of claim 1, wherein determining sparse depth maps for the first camera and the second camera from the first image and the second image comprises:

3. The method according to claim 2, wherein the obtaining the sparse depth map according to the disparity map and the confidence map of the disparity map comprises:

4. The method according to any one of claims 1 to 3, wherein the adjusting the depth of the pixel points in the sparse depth map according to the dense depth map to obtain a target depth map comprises:

5. The method of claim 4, wherein before the obtaining the first region and the second region in the sparse depth map, the adjusting the depth of the pixel points in the sparse depth map according to the dense depth map further comprises:

6. The method according to any one of claims 1 to 3, wherein after the adjusting the depth of the pixel points in the sparse depth map according to the dense depth map to obtain a target depth map, the method further comprises:

obtaining a second connected region of the dense depth map;

7. An image processing apparatus, characterized in that the apparatus comprises:

8. The apparatus of claim 7, wherein the sparse depth map acquisition module comprises:

9. The apparatus of claim 8, wherein the sparse depth map acquisition submodule is specifically configured to:

10. The apparatus of any of claims 7 to 9, wherein the depth adjustment module comprises:

11. The apparatus of claim 10, wherein the depth adjustment module further comprises:

12. The apparatus of any one of claims 7 to 9, further comprising a first filtration module to:

obtaining a second connected region of the dense depth map;