WO2022188102A1

WO2022188102A1 - Depth image inpainting method and apparatus, camera assembly, and electronic device

Info

Publication number: WO2022188102A1
Application number: PCT/CN2021/080255
Authority: WO
Inventors: 苏雨曦; 罗俊
Original assignee: Oppo广东移动通信有限公司
Priority date: 2021-03-11
Filing date: 2021-03-11
Publication date: 2022-09-15
Also published as: CN116897532A

Abstract

A depth image inpainting method and apparatus (10), a camera assembly (100), and an electronic device (1000). The inpainting method comprises: obtaining a current scene image of a depth image, the current scene image comprising a plurality of different object areas, and mapping the object areas of the current scene image to different pixel value ranges to obtain a guide image; and constructing a target function according to the depth image and the guide image to perform global optimization calculation to inpaint the depth image.

Description

Depth image restoration method and device, camera assembly and electronic device

technical field

The present application relates to the field of imaging technologies, and in particular, to a depth image restoration method, restoration device, camera assembly, and electronic equipment.

Background technique

In order to enhance the functions of electronic devices and enable them to be applied to various application scenarios, electronic devices are equipped with depth image devices to obtain depth information. However, due to factors such as occlusion and measurement range limitations, depth images may have holes and other abnormalities.

SUMMARY OF THE INVENTION

Embodiments of the present application provide a depth image restoration method, a restoration device, a camera assembly, and an electronic device.

The restoration method of the embodiment of the present application includes: acquiring a current scene image of the depth image, where the current scene image includes a plurality of different object regions, and mapping each of the object regions of the current scene image to different pixel value ranges to obtain a guide image; construct an objective function according to the depth image and the guide image, and perform a global optimization calculation to repair the depth image.

According to an embodiment of the present application, a depth image restoration device includes a first acquisition module and a first processing module. The first acquisition module is used to acquire a current scene image of the depth image, the current scene image includes a plurality of different object regions, and each of the object regions of the current scene image is mapped to different pixel value ranges to obtaining a guide image; the first processing module is configured to construct an objective function according to the depth image and the guide image and perform a global optimization calculation to repair the depth image.

A camera assembly according to an embodiment of the present application, the camera assembly includes an image sensor, a depth sensor, and a processor, and the processor is configured to acquire a current scene image, where the current scene image includes a plurality of different object regions, and the current scene image is Each of the object regions of the scene image is mapped to different pixel value ranges to obtain a guide image; an objective function is constructed according to the depth image and the guide image, and a global optimization calculation is performed to restore the depth image.

The electronic device of the embodiment of the present application includes the camera assembly and the casing of the above-mentioned embodiment, and the camera assembly is disposed on the casing.

The above depth image restoration method, restoration device, camera assembly and electronic device obtain a guide image by acquiring the current scene image of the depth image and mapping each object area of the current scene image to different pixel value ranges. The guide image can reflect the difference in depth changes of different object areas in the scene image, and can effectively enhance the edge effect of each different object area. Further, constructing an objective function according to the depth image and the guide image to perform a global optimization calculation to repair the depth image can effectively fill and repair holes of various areas in the depth image. At the same time, because each object area in the guide image has a different pixel value range, the edge information in the image is enhanced to a certain extent, and the holes at the edge can be effectively filled and repaired when repairing the depth image. The edge information is preserved to a certain extent.

Additional aspects and advantages of embodiments of the present application will be set forth, in part, in the following description, and in part will be apparent from the following description, or learned by practice of the present application.

Description of drawings

The above and/or additional aspects and advantages of the present application will become apparent and readily understood from the following description of embodiments in conjunction with the accompanying drawings, wherein:

1 is a schematic flowchart of a repair method according to an embodiment of the present application;

FIG. 2 is an exemplary diagram of a repair method according to an embodiment of the present application;

3 is a schematic flowchart of a repair method according to an embodiment of the present application;

4 is a schematic flowchart of a repair method according to an embodiment of the present application;

FIG. 5 is an exemplary diagram of a repair method according to an embodiment of the present application;

6 is a schematic flowchart of a repair method according to an embodiment of the present application;

7 is a schematic flowchart of a repair method according to an embodiment of the present application;

8 is a schematic flowchart of a repair method according to an embodiment of the present application;

FIG. 9 is an exemplary diagram of a repair method according to an embodiment of the present application;

FIG. 10 is a module diagram of a repair device according to an embodiment of the present application;

FIG. 11 is a block diagram of a repair device according to an embodiment of the present application;

FIG. 12 is a module diagram of a repair device according to an embodiment of the present application;

13 is a schematic diagram of a camera assembly according to an embodiment of the present application;

FIG. 14 is a schematic diagram of an electronic device according to an embodiment of the present application.

Detailed ways

Embodiments of the present application are described in detail below, examples of which are illustrated in the accompanying drawings, wherein the same or similar reference numerals refer to the same or similar elements or elements having the same or similar functions throughout. The embodiments described below with reference to the accompanying drawings are exemplary, only used to explain the present application, and should not be construed as a limitation on the present application.

Referring to FIG. 1 , the present application provides a method for repairing a depth image, and the repair method includes:

S10: obtaining a current scene image of the depth image, where the current scene image includes a plurality of different object regions, and mapping each object region of the current scene image to different pixel value ranges to obtain a guide image;

S20: Construct an objective function according to the depth image and the guide image to perform a global optimization calculation to repair the depth image.

Specifically, in step S10, the depth image includes depth information of objects within the current shooting range. The same sensor can be obtained simultaneously by active ranging sensing methods such as TOF camera components or structured light components with depth sensors, or passive ranging sensing methods such as two camera components with RGB filter arrays separated by a certain distance. Two images of the scene, and then data processing and depth calculation are performed to obtain a depth image. Further, multiple frames of depth images can be stored in the depth map buffer space. In some embodiments, two images of the same scene are simultaneously acquired by two camera assemblies having image sensors with RGB filter arrays, please refer to FIG. 2, FIG. (b) is the image captured by the sub-camera assembly, through data processing and depth calculation, the depth image shown in Figure 2(c) is obtained.

Meanwhile, the current scene image corresponding to the depth image may include scene information of objects within the current shooting range. The current scene image may also be a pre-stored image that needs to be displayed currently, that is, the current scene image may include scene information of objects within the original shooting range. The current scene image can be captured by the camera assembly of the image sensor with the RGB filter array. In some embodiments, the depth image is acquired by two camera assemblies of image sensors with RGB filter arrays to simultaneously acquire two images of the same scene. Please refer to FIG. 2 again, FIG. 2(a) The image captured by the main camera assembly , FIG. 2(b) is an image captured by the sub-camera assembly, and the current scene image may include FIG. 2(a) and/or FIG. 2(b).

Further, after the current scene image is acquired, each object area of the current scene image is mapped to different pixel value ranges, thereby enhancing the edges of each object area in the current scene image, and obtaining a guide image. Specifically, each object area in the current scene image may include single or multiple persons, and/or single or multiple non-personal objects. In some embodiments, each object region of the current scene image can be segmented by means of a machine learning algorithm, and each object region can be mapped to different pixel value ranges. In some embodiments, the current scene image can also be segmented by means of semantic segmentation, instance segmentation, etc., and then through superposition calculation, so that each object region has different pixel value ranges.

In step S20, the global optimization calculation of the objective function is performed according to the obtained guide image and the depth image to restore the depth image. It is understandable that in the depth image, due to certain factors such as when the illuminated object is a transparent object, the surface of the object is a light-absorbing material, and the surface of the object is very smooth, or the object is in the blind area of the depth camera, For example, in the area that is too close or too far, the data will be lost due to the inability to capture the reflected infrared light, resulting in errors and holes in the depth image. Example expansion.

Specifically, the depth image is used as the input of the objective function, and the guide image is used as the weighting coefficient of edge enhancement, and each pixel in the image is maximized to be close to the pixel value of the surrounding neighborhood pixels, so as to construct the objective function and pass the global maximum The optimized solution process obtains the output in the objective function, that is, the repaired depth image, which can effectively fill and repair the holes in the depth image.

In this way, the above-mentioned depth image restoration method obtains the guide image by acquiring the current scene image of the depth image, and mapping each object region of the current scene image to different pixel value ranges. The guide image can reflect the difference in depth changes of different object areas in the scene image, and can effectively enhance the edge effect of each different object area. Further, in the related art, depth image restoration methods are mainly based on joint bilateral filtering methods or local spatial filtering methods such as median filtering or Gaussian filtering, but are often used to deal with small-area holes. When there is a hole in the edge, it will cause the problem of blurring or disappearing of the edge. In this application, the depth image is used as the input of the objective function, and the guide image is used as the weighting coefficient for edge enhancement, so that each pixel in the image is at most close to the pixel value of the surrounding neighborhood pixels, so as to construct the objective function through the global optimization solution. The process obtains the output of the objective function, that is, the repaired depth image, which can effectively fill and repair the holes of various areas in the depth image. At the same time, since the guide image is used as a weighting coefficient for edge enhancement, the holes at the edge can be effectively filled and repaired, and the edge information can be preserved to a certain extent.

Referring to FIG. 3, in some embodiments, step S10 includes:

S11: Perform scene detection on the current scene image to determine the scene type;

S12: perform portrait segmentation when the scene type is a human image;

S13: perform object segmentation when the scene type is a non-person image;

S14: Determine each object area according to the segmentation result;

S15: Map each object area to different pixel value ranges to obtain a guide image.

Specifically, in step S11, based on machine learning, data such as pictures of different scene types can be used for pre-training to improve the scene detection ability. In this way, after the current scene image is acquired, the corresponding image of the current scene image can be more accurately determined. Scene type. Scene types may include human images, non-human images, and human-non-human images, where the human image may include a portrait subject and a background, a non-human image may include an object subject and background, and a human-non-human image may include portrait subject, object subject And background, further, the character subject includes one character or more than one character, and the non-character subject includes one non-character or more than one non-character. In some embodiments, when performing scene detection on the current scene image, first detect whether there is a portrait subject, then detect whether there is an object subject, and then combine the results of pre-machine learning to determine the scene type corresponding to the current scene image.

In step S12, portrait segmentation is performed on the current scene image, that is, the portrait subject and the background are segmented. In step S13, object segmentation is performed on the current scene image, that is, the object subject and the background are segmented. In some embodiments, when the current scene image includes both a portrait subject and an object subject, the portrait subject is segmented first, and then the object subject is segmented.

In step S14, each object region can be divided into binary results and multi-valued results according to the actual application. Among them, the binary result includes dividing the required single or multiple object regions into main regions, and the rest are background regions. Further, the subject area is mapped to one pixel value range, and the background area is mapped to another pixel value range. The multi-valued result may include multiple characters and/or multiple non-characters to form multiple regions, that is, the current scene image is divided into at least three different object regions, and the at least three different object regions include at least two subject regions and one Background area, multiple areas are mapped to their respective different pixel value ranges. In an example, the current scene image is a single person and a background non-person area, the single person is divided into the main area according to the binary result, the mapped pixel value range is 155-255, the background non-person area is the background area, and the mapped pixel value range is 0 to 100.

In step S15, after each object region is mapped to different pixel value ranges, the brightness displayed by each object region is different, and the boundary between each object region is clearer, thereby obtaining a guide image.

In this way, a more accurate guide image can be obtained, thereby enhancing the edge of each object area in the current scene image.

Referring to FIG. 4, in some embodiments, step S15 includes:

S151: Determine each object region according to the segmentation result and form a segmented image, and each object region is represented by the same pixel value in the segmented image;

S152: Perform weighting processing on the segmented image and the current scene image to obtain a guide image.

Specifically, the pixel value range of each object region in the segmented image is preset, the pixel value range of each object region in the segmented image is different, and the pixel value of the same object region in different current scene images is the same, for example, the characters in different current scene images In the image, the pixel value of the portrait subject is uniformly set to (155,255), and the pixel value of the background is uniformly set to (0,100), so that the portrait subject and the background can be distinguished to form a segmented image. Further, the guide image can be obtained by performing weighting processing on the pixel value of the segmented image and the corresponding pixel value of the current scene image. In the segmented image, the pixel value ranges of different object regions are different, and the edges of different object regions in the segmented image are enhanced compared to the current scene image. Wherein, the weighted weight coefficient can be set according to the actual need to distinguish the degree of each object area.

Please refer to Figure 5. In an example, by detecting the scene of the current scene image, it is determined that the scene type of the current scene image is a portrait image, and the portrait is segmented to obtain two object regions, one of which is the portrait subject, and the other object. The area is the background, the pixel value of the main body of the portrait is set to (155, 255), and the pixel value of the background is set to (0, 100), so as to obtain the segmented image as shown in Figure 5(d). The pixel values of the current scene image shown in Fig. 5(e) are weighted and summed to obtain the guide image shown in Fig. 5(f).

It should be noted that the purpose of obtaining the guide image by weighting the segmented image and the current scene image is to enable each object region in the current scene image to be displayed with different pixel value ranges, or to make the distinction of each object region more obvious. It can be understood that the weighting processing is only a mathematical processing method, and there may be other methods such as linear functions. Therefore, the transformation of simple mathematical form for this purpose can be regarded as a simple replacement of this embodiment.

In this way, by weighting the segmented image and the current scene image to obtain the guide image, the implementation method is simpler and more effective, and the weight coefficient can be adjusted according to actual business requirements, thereby enhancing the edge of each object area in the current scene image.

Referring to FIG. 6, in some embodiments, step S15 further includes:

S153: Determine the pixel value range mapped by each object area according to the number of object areas;

S154: Map each object area to a corresponding pixel value range to obtain a guide image.

Specifically, the corresponding relationship between the number of object regions, the type of object regions and the range of pixel values mapped by each type of object region in the number is preset, so that when determining the number and type of object regions (for example, the first subject region, the third After two main areas and background areas), according to the corresponding relationship, map each object area to the corresponding pixel value range, and then the edge-enhanced guide image can be obtained.

In this way, by mapping each object area to a corresponding pixel value range, a more accurate guide image is obtained, thereby enhancing the edge of each object area in the current scene image.

In some embodiments, a preset range is spaced between two adjacent pixel value ranges, and the difference between the maximum value of the preset range and the minimum value of the preset range is greater than 1.

It can be understood that the range of pixel values includes multiple ranges, and the multiple ranges of pixel values include adjacent first pixel value ranges and second pixel value ranges, and the maximum value of the first pixel value range is smaller than the minimum value of the second pixel value range. , the difference between the minimum value of the second pixel value range and the maximum value of the first pixel value range is greater than 1.

In an example, if there are two object regions, the pixel value range of one object region may be [0, 100], the pixel value range of the other object region may be [155, 255], and the preset range may be (100, 155). In another example, if there are 5 object regions, the pixel value ranges of the 5 object regions may be [0,41], [51,92], [102,143], [153,194] and [204,245] respectively. The range can be (41,51), (92,102), (143,153), (194,204), (245,255).

In this way, different object regions correspond to different pixel value ranges, and two adjacent pixel value ranges are separated by a preset range, so that the brightness of different object regions is different, and the boundaries of different object regions are clearer.

Referring to FIG. 7, in some embodiments, the method for repairing the depth image further includes:

S30: Acquire a historical frame depth image, where the shooting time of the historical frame depth image is before the shooting time of the depth image;

S40: obtain the hole pixel set of the depth image;

S50: Obtain the repaired pixel set according to the depth image of the historical frame;

S60: replace the hole pixel set with the repair pixel set to obtain an enhanced depth image;

Step S20 includes:

S21: Construct an objective function according to the enhanced depth image and the guide image to perform a global optimization calculation to repair the depth image.

In the above-mentioned embodiments, the depth image can be obtained by an active ranging sensing method, such as a TOF camera assembly or a structured light assembly with a depth sensor, or a passive ranging sensing method, such as two sensors with RGB filter arrays separated by a certain distance. The camera component of the image sensor acquires two images of the same scene at the same time, and then performs data processing and depth calculation. The depth image obtained in this way may be an original depth image, which contains holes, and a set of hole pixels is obtained, wherein the set of hole pixels is a set of all the holes in the original depth image.

Further, the historical weighted depth value of the hole point can be calculated by using the depth map buffer to perform preliminary filling and repairing on the hole point set in the original depth image.

Specifically, the historical frame depth image in the depth map buffer is acquired, and the shooting time of the historical frame depth image is before the shooting time of the depth image, including a single frame or multiple frames of depth images. If it is a single-frame historical depth image and has a non-zero pixel value at the corresponding hole position of the original depth image, the pixel value of the corresponding hole pixel in the single-frame historical depth image is selected as the repair pixel set. If it is a multi-frame historical depth image, the historical depth image of the required number of frames can be extracted in chronological order, and the hole pixels in the multi-frame historical depth image can be weighted and summed to obtain a repaired pixel set. For example, the current time is t, and the original depth image at time t, t-1, and t-2 is stored in the buffer. Then the historical weighted value of the hole pixel can be calculated by the following formula:

D _t ′=w ₁ *D _t +w ₂ *D _t-1 +w ₃ *D _t-2

The sum of the weights w is 1, that is, w ₁ +w ₂ +w ₃ =1, and the closer the time is to the current frame, the greater the weight.

Further, the hole pixel set of the original depth image is replaced with the repair pixel set to obtain an enhanced depth image. Then the objective function is constructed according to the enhanced depth image and the guide image for global optimization calculation to repair the depth image. The specific implementation is the same as that of the above-mentioned embodiment, and is not further expanded here.

In this way, the depth map buffer is used to calculate the historical weighted depth value of the hole point to initially fill and repair the hole point in the original depth image, so that the pixel value of the hole pixel point entering the objective function is more optimized, and then the objective function can be optimized. Get better output functions, or better inpainting results for depth images.

Referring to FIG. 8, in some embodiments, step S20 further includes:

S22: Optimize the objective function so that the objective function achieves the minimum value, and outputs the pixel value of the current pixel of the repaired depth image corresponding to the minimum value. The objective function is:

Among them, i is the position of the current pixel, _ui is the pixel value of the current pixel, λ is the total weight coefficient in the frame, j is the pixel position of the neighborhood N(i) of i, g is the guide image, w _{i ,j} (g) is the edge weight coefficient corresponding to the guide image, u _j is the pixel value of the pixel point in the neighborhood of the current pixel point, and f _i is the pixel value corresponding to the current pixel point in the depth image.

Specifically, the guide image g uses the function wi _,j (g) as a guide item to control the edge weight coefficient of each object region, the coefficient is small when the edge is strong, and the coefficient is large when the edge is weak. It can be understood that the minimum value of the objective function J(u) is solved by mathematical methods, so that the error between the output function and the input function in the function is minimized, and the current pixel point is close to the neighborhood pixel point at the most, and the total smoothing weight in the frame is passed. The coefficient λ and the edge enhancement coefficient w _i,j (g) corresponding to the guide image perform edge control.

In some embodiments, f _i may be an enhanced depth image, that is, in the above embodiment, the enhanced depth obtained by initially filling and repairing the hollow points in the original depth image by using the depth map buffer to calculate the historical weighted depth value of the hollow points image. Further, an objective function is constructed according to the enhanced depth image and the guide image to perform a global optimization calculation to repair the depth image.

Please refer to Fig. 9, in an example, 9(g) is the input depth image of the objective function, in which, the oval box in 9(g) is a hollow example, and 9(h) is the guide image. Minimize the solution, and finally get the repaired depth map 9(i). It can be seen from the figure that the holes are effectively filled and repaired to a certain extent.

In this way, by optimizing the objective function J(u), the holes of various areas in the depth image can be effectively filled and repaired. Compared with other objective functions, J(u) minimizes the input and output errors from the global optimization, and the solution process is a linear weighted solution, which is more simple and effective to fill and repair the holes in the depth image. At the same time, since the guide image is used as a weighting coefficient for edge enhancement, the holes at the edge can be effectively filled and repaired, and the edge information can be preserved to a certain extent. Further, compared with the restoration methods in the prior art, such as using Gaussian filtering, the objective function of machine learning can be solved faster, and to a certain extent, high-speed filling and restoration of depth images can be achieved.

In certain embodiments, the neighborhood N(i) is a 4-neighborhood or an 8-neighborhood.

It can be understood that when the current pixel i is in the center of the nine-square grid, the 4 neighborhoods of i, that is, a pixel above adjacent to i, a pixel below adjacent to i, and a pixel adjacent to i. One pixel on the left side of the neighbor, one pixel on the right side adjacent to i; 8 neighbors of i, that is, on the basis of the 4 neighbors of i, add four pixels that are diagonally adjacent to i .

In this way, the pixel points in the 4-neighborhood or the 8-neighborhood of the current pixel point i of the current frame can be filtered, so as to obtain a repaired depth image corresponding to the depth image of the current frame.

In some embodiments, the value range of λ is [100, 10000].

Specifically, the value of λ may be 100, 500, 700, 1000, 3000, 5000, 7000, 10000 or other values between 100-10000.

In this way, the total weight coefficient in the frame can be set as required, so as to obtain a better objective function.

In certain embodiments,

g _i is the pixel value of the guide image corresponding to the current pixel point, g _j is the pixel value of the guide image corresponding to the j point in the neighborhood N(i), and the value range of σ is [1, 10].

It can be understood that the farther the j point is from the current pixel point i, the smaller the influence on the pixel value of the current pixel point i, that is, the farther the j point is from the current pixel point i, the smaller the edge enhancement coefficient is. Specifically, the value of σ may be 1, 2, 3, 4, 5, 6, 7, 8, 9, 10, or other values between 1-10.

Referring to FIG. 10 , the present application provides a depth image restoration apparatus 10 . The restoration apparatus 10 includes a first acquisition module 11 and a first processing module 12 . The first acquisition module 11 is configured to acquire a current scene image of the depth image, the current scene image includes a plurality of different object regions, and each object region of the current scene image is mapped to different pixel value ranges to obtain a guide image. The first processing module 12 is configured to construct an objective function according to the depth image and the guide image to perform global optimization calculation to repair the depth image.

Specifically, the depth image includes depth information of objects within the current shooting range. The same sensor can be obtained simultaneously by active ranging sensing methods such as TOF camera components or structured light components with depth sensors, or passive ranging sensing methods such as two camera components with RGB filter arrays separated by a certain distance. Two images of the scene, and then data processing and depth calculation are performed to obtain a depth image. Further, multiple frames of depth images can be stored in the depth map buffer space.

At the same time, the current scene image corresponding to the depth image can be acquired by the first acquisition module 11, including scene information of objects within the current shooting range. The current scene image may also be a pre-stored image acquired by the first acquiring module 11 and currently required to be displayed, that is, the current scene image may include scene information of objects within the original shooting range. The current scene image can be captured by the camera assembly of the image sensor with the RGB filter array.

Further, after acquiring the current scene image, the first acquisition module 11 maps each object area of the current scene image to different pixel value ranges, thereby enhancing the edges of each object area in the current scene image to obtain a guide image. Specifically, each object area in the current scene image may include single or multiple persons, and/or single or multiple non-personal objects. In some embodiments, each object region of the current scene image can be segmented by means of a machine learning algorithm, and each object region can be mapped to different pixel value ranges. In some embodiments, the current scene image can also be segmented by means of semantic segmentation, instance segmentation, etc., and then through superposition calculation, so that each object region has different pixel value ranges.

After the guide image is determined, the first processing module 12 performs a global optimization calculation of the objective function according to the obtained guide image and the depth image to restore the depth image. It is understandable that in the depth image, due to certain factors such as when the illuminated object is a transparent object, the surface of the object is a light-absorbing material, and the surface of the object is very smooth, or the object is in the blind area of the depth camera, For example, in the area that is too close or too far, the data will be lost due to the inability to capture the reflected infrared light, resulting in the problem of holes in the depth image.

In this way, the above-mentioned depth image restoration apparatus 10 obtains the current scene image of the depth image through the first obtaining module 11, and maps each object area of the current scene image to different pixel value ranges to obtain the guide image. The guide image can reflect the difference in depth changes of different object areas in the scene image, and can effectively enhance the edge effect of each different object area. Further, the first processing module 12 uses the depth image as the input of the objective function, and the guide image is used as the weighting coefficient of edge enhancement, so that each pixel in the image is at most close to the pixel value of the surrounding neighborhood pixels, so as to construct the objective function. The output of the objective function, that is, the repaired depth image, can be effectively filled and repaired for holes of various areas in the depth image through the global optimization solution process. At the same time, since the guide image is used as a weighting coefficient for edge enhancement, the holes at the edge can be effectively filled and repaired, and the edge information can be preserved to a certain extent.

It should be pointed out that the above-mentioned explanations of the embodiments and beneficial effects of the depth image restoration method are also applicable to the depth image restoration apparatus 10 of this embodiment and the camera assembly and electronic equipment described in the following embodiments, in order to avoid redundant I will not expand it in detail here.

Referring to FIG. 10 again, in some embodiments, the first acquisition module 11 includes a detection unit 111 , a first segmentation unit 112 , a second segmentation unit 113 , a determination unit 114 and a mapping unit 115 . The detection unit 111 is configured to perform scene detection on the current scene image to determine the scene type. The first segmentation unit 112 is configured to perform portrait segmentation when the scene type is a human image. The second segmentation unit 113 is configured to perform object segmentation when the scene type is a non-human image. The determining unit 114 is used for determining each object region according to the segmentation result. The mapping unit 115 is used for mapping each object region to different pixel value ranges to obtain a guide image.

In this way, the first obtaining module 11 can obtain a relatively accurate guide image, thereby enhancing the edges of each object region in the current scene image.

Referring to FIG. 10 again, in some embodiments, the mapping unit 115 includes a first determination subunit 1151 and a weighting processing subunit 1152 . The first determination subunit 1151 is configured to determine each object region according to the segmentation result and form a segmented image, and each object region is represented by the same pixel value in the segmented image. The weighting processing subunit 1152 is configured to perform weighting processing on the segmented image and the current scene image to obtain the guide image.

In this way, the mapping unit 115 obtains the guide image by weighting the segmented image and the current scene image, which is simpler and more effective, and can adjust the weight coefficients according to actual business requirements, thereby enhancing the edge of each object area in the current scene image.

Referring to FIG. 11 , in some embodiments, the mapping unit 115 includes a second determining subunit 1153 and a mapping subunit 1154 . The second determination subunit 1153 is configured to determine the pixel value range mapped by each object region according to the number of object regions. The mapping subunit 1154 is used to map each object region to a corresponding pixel value range to obtain a guide image.

In this way, the mapping unit 115 obtains a more accurate guide image by mapping each object region to a corresponding pixel value range, thereby enhancing the edge of each object region in the current scene image.

Referring to FIG. 12 , in some embodiments, the repairing apparatus 10 further includes a second acquiring module 13 and a second processing module 14 . The second acquisition module 13 is used to acquire the depth image of the historical frame, the shooting time of the depth image of the historical frame is before the shooting time of the depth image, and to acquire the hole pixel set of the depth image. The second processing module 14 is configured to obtain the repaired pixel set according to the depth image of the historical frame, and replace the hole pixel set of the depth image with the repaired pixel set to obtain the enhanced depth image. At the same time, the first processing module 12 is further configured to construct an objective function according to the enhanced depth image and the guide image to perform global optimization calculation to repair the depth image.

In this way, the second processing module 14 uses the depth map buffer to calculate the historical weighted depth value of the hole point to perform preliminary filling and repairing on the hole point in the original depth image, so that the pixel value of the hole pixel point entering the objective function is more optimized, which can make The first processing module 12 performs the objective function optimization solution to obtain a better output function, or in other words, obtains a better restoration effect for the depth image.

Referring again to FIG. 10 or FIG. 11 , in some embodiments, the first processing module 12 includes an optimization unit 210 . The optimization unit 210 is configured to optimize the objective function so that the objective function obtains a minimum value, and outputs the pixel value of the current pixel point of the repaired depth image corresponding to the minimum value, and the objective function is:

Among them, _i is the position of the current pixel, ui is the pixel value of the current pixel, λ is the total smoothing weight coefficient in the frame, j is the pixel position of the neighborhood N(i) of i, and g is For the guide image, w _i,j (g) is the edge enhancement coefficient corresponding to the guide image, u _j is the pixel value of the pixel point in the neighborhood of the current pixel point, and f _i is the difference between the depth image and the depth image. The pixel value corresponding to the current pixel point.

In this way, the first processing module 12 optimizes the objective function J(u) through the optimization unit 210, and can effectively fill and repair holes of various areas in the depth image. Compared with other objective functions, J(u) minimizes the input and output errors from the global optimization, and the solution process is a linear weighted solution, which is more simple and effective to fill and repair the holes in the depth image. At the same time, since the guide image is used as a weighting coefficient for edge enhancement, the holes at the edge can be effectively filled and repaired, and the edge information can be preserved to a certain extent. Further, compared with the restoration methods in the prior art, such as using Gaussian filtering, the objective function of machine learning can be solved faster, and to a certain extent, high-speed filling and restoration of depth images can be achieved.

In some embodiments, the value range of λ is [100, 10000].

In certain embodiments,

Referring to FIG. 13 , the present application provides a camera assembly 100 . The camera assembly 100 includes an image sensor 101 , a depth sensor 102 and a processor 103 . The image sensor 101 is used to capture the current scene image, the processor 103 is used to obtain the current scene image, the current scene image includes a plurality of different object regions, and each object region of the current scene image is mapped to different pixel value ranges to obtain the guide image, And construct the objective function according to the depth image and guide image for global optimization calculation to repair the depth image.

The above-mentioned camera assembly 100 obtains the current scene image of the depth image through the image sensor 101, and maps each object area of the current scene image to different pixel value ranges to obtain the guide image. The guide image can reflect the difference in depth changes of different object areas in the scene image, and can effectively enhance the edge effect of each different object area. Further, the depth sensor 102 is used to obtain the depth image, the processor 103 uses the depth image as the input of the objective function, and the guide image is used as the weighting coefficient for edge enhancement, so that each pixel in the image is at most close to the pixels of the surrounding neighborhood pixels. The value of the objective function is constructed to obtain the output of the objective function through the global optimization solution process, that is, the repaired depth image, which can effectively fill and repair the holes of various areas in the depth image. At the same time, since the guide image is used as a weighting coefficient for edge enhancement, the holes at the edge can be effectively filled and repaired, and the edge information can be preserved to a certain extent. Further, compared with the restoration methods in the prior art, such as using Gaussian filtering, the objective function of machine learning can be solved faster, and to a certain extent, high-speed filling and restoration of depth images can be achieved.

The processor 103 may be configured to implement the depth image restoration method described in any one of the foregoing embodiments, and details are not described herein again.

Referring to FIG. 14 , the present application provides an electronic device 1000 . The electronic device 1000 includes the camera assembly 100 and the casing 200 of the above-mentioned embodiments, and the camera assembly 100 is disposed on the casing 200 .

The above electronic device 1000 obtains the current scene image of the depth image through the camera assembly 100, and maps each object area of the current scene image to different pixel value ranges to obtain the guide image. The guide image can reflect the difference in depth changes of different object areas in the scene image, and can effectively enhance the edge effect of each different object area. Further, the depth image is used as the input of the objective function, and the guide image is used as the weighting coefficient of edge enhancement, so that each pixel in the image is at most close to the pixel value of the surrounding neighborhood pixels, so as to construct the objective function through the global optimization. The solution process obtains the output of the objective function, that is, the repaired depth image, which can effectively fill and repair the holes of various areas in the depth image. At the same time, since the guide image is used as a weighting coefficient for edge enhancement, the holes at the edge can be effectively filled and repaired, and the edge information can be preserved to a certain extent. Further, compared with the restoration methods in the prior art, such as using Gaussian filtering, the objective function of machine learning can be solved faster, and to a certain extent, high-speed filling and restoration of depth images can be achieved.

Specifically, in the embodiment shown in FIG. 14 , the electronic device 1000 is a smart phone, and in other embodiments, the electronic device can be a camera, a tablet computer, a notebook computer, a smart home appliance, a game console, a head-mounted display device, a wearable device Other devices with camera functions, such as devices.

In the description of this specification, reference to the terms "one embodiment," "some embodiments," "exemplary embodiment," "example," "specific example," or "some examples" or the like is meant to be used in conjunction with the described embodiments. A particular feature, structure, material, or characteristic described in a manner or example is included in at least one embodiment or example of the present application. In this specification, schematic representations of the above terms do not necessarily refer to the same embodiment or example. Furthermore, the particular features, structures, materials or characteristics described may be combined in any suitable manner in any one or more embodiments or examples. Furthermore, those skilled in the art may combine and combine the different embodiments or examples described in this specification, as well as the features of the different embodiments or examples, without conflicting each other.

In addition, the terms "first" and "second" are only used for descriptive purposes, and should not be construed as indicating or implying relative importance or implying the number of indicated technical features. Thus, features delimited with "first", "second" may expressly or implicitly include at least one of said features. In the description of the present application, "plurality" means at least two, such as two, three, unless expressly and specifically defined otherwise.

Any description of a process or method in the flowcharts or otherwise described herein may be understood to represent a module, segment or portion of code comprising one or more executable instructions for implementing a specified logical function or step of the process , and the scope of the preferred embodiments of the present application includes alternative implementations in which the functions may be performed out of the order shown or discussed, including performing the functions substantially concurrently or in the reverse order depending upon the functions involved, which should It is understood by those skilled in the art to which the embodiments of the present application belong.

Although the embodiments of the present application have been shown and described above, it should be understood that the above embodiments are exemplary and should not be construed as limitations to the present application. Embodiments are subject to variations, modifications, substitutions and alterations.

Claims

A depth image repair method, characterized in that the repair method includes:

Acquiring a current scene image of the depth image, where the current scene image includes a plurality of different object regions, and mapping each of the object regions of the current scene image to different pixel value ranges to obtain a guide image;

An objective function is constructed according to the depth image and the guide image to perform a global optimization calculation to repair the depth image.
The repair method according to claim 1, wherein the acquisition of a current scene image of the depth image, the current scene image includes a plurality of different object regions, and each of the object regions of the current scene image is Mapping to different ranges of pixel values to obtain wizard images include:

Perform scene detection on the current scene image to determine the scene type;

Perform portrait segmentation when the scene type is a human image;

Perform object segmentation when the scene type is a non-human image;

Determine each of the object regions according to the segmentation result;

Each of the object regions is mapped to different ranges of the pixel values to obtain the guide image.
The repair method according to claim 2, wherein the mapping each of the object regions to different ranges of the pixel values to obtain the guide image comprises:

Each of the object regions is determined according to the segmentation result and a segmented image is formed, and each of the object regions is represented by the same pixel value in the segmented image;

The divided image and the current scene image are weighted to obtain the guide image.
The repair method according to claim 2, wherein the mapping each of the object regions to different ranges of the pixel values to obtain the guide image further comprises:

Determine the pixel value range mapped by each of the object regions according to the number of the object regions;

Each of the object regions is mapped to the corresponding pixel value range to obtain the guide image.
The repair method according to any one of claims 1-4, wherein a preset range is spaced between two adjacent pixel value ranges, and the maximum value of the preset range is the same as the preset range. The difference between the minimum values is greater than 1.
The repair method according to claim 1, wherein the repair method further comprises:

Obtaining a depth image of a historical frame, the shooting time of the depth image of the historical frame is before the shooting time of the depth image;

obtaining the hole pixel set of the depth image;

obtaining a repaired pixel set according to the depth image of the historical frame;

replacing the set of hole pixels with the set of repaired pixels to obtain an enhanced depth image;

The constructing an objective function according to the depth image and the guide image and performing a global optimization calculation to repair the depth image includes:

According to the enhanced depth image and the guide image, an objective function is constructed to perform a global optimization calculation to repair the depth image.
The repair method according to claim 1, wherein the constructing an objective function according to the depth image and the guide image and performing a global optimization calculation to repair the depth image further comprises:

The objective function is optimized so that the objective function obtains the minimum value, and the pixel value of the current pixel of the repaired depth image corresponding to the minimum value is output, and the objective function is:

Among them, i is the position of the current pixel, ui is the pixel value of the current pixel, λ is the total weight coefficient in the frame, j is the pixel position of the neighborhood N(i) of i, and g is the For the guide image, w i,j (g) is the edge enhancement coefficient corresponding to the guide image, u j is the pixel value of the pixel point in the neighborhood of the current pixel point, and f i is the depth image and the The pixel value corresponding to the current pixel point.
The repair method according to claim 7, wherein the neighborhood N(i) is a 4-neighborhood or an 8-neighborhood.
The repair method according to claim 7, wherein the value range of λ is [100, 10000].
The repair method according to claim 7, wherein,
g i is the pixel value of the guide image corresponding to the current pixel point, g j is the pixel value of the guide image corresponding to the j point of the neighborhood N(i), and the value range of σ is [1 , 10].
A depth image repairing device, characterized in that the repairing device comprises:

The first acquisition module is used to acquire the current scene image of the depth image, the current scene image includes a plurality of different object regions, and each of the object regions of the current scene image is mapped to different pixel value ranges to obtain wizard image;

The first processing module is configured to construct an objective function according to the depth image and the guide image and perform a global optimization calculation to repair the depth image.
The repair device according to claim 11, wherein the first acquisition module comprises:

a detection unit, configured to perform scene detection on the current scene image to determine a scene type;

a first segmentation unit, for performing portrait segmentation when the scene type is a human image;

a second segmentation unit, configured to perform object segmentation when the scene type is a non-personal image;

a determining unit, configured to determine each of the object regions according to the segmentation result;

a mapping unit, configured to map each of the object regions to different ranges of the pixel values to obtain the guide image.
The repair device according to claim 12, wherein the mapping unit comprises:

a first determination subunit, configured to determine each of the object regions according to the segmentation result and form a segmented image, and each of the object regions is represented by the same pixel value in the segmented image;

A weighting processing subunit, configured to perform weighting processing on the segmented image and the current scene image to obtain the guide image.
The repair device according to claim 12, wherein the mapping unit further comprises:

a second determination subunit, configured to determine the pixel value range mapped by each of the object regions according to the number of the object regions;

A mapping subunit, configured to map each of the object regions to the corresponding pixel value ranges to obtain the guide image.
The repair device according to any one of claims 11-14, wherein a preset range is spaced between two adjacent pixel value ranges, and the maximum value of the preset range is the same as the preset range. The difference between the minimum values is greater than 1.
The repair device according to claim 11, wherein the repair device further comprises:

a second acquisition module, configured to acquire a depth image of a historical frame, the shooting time of the depth image of the historical frame is before the shooting time of the depth image; and

obtaining the hole pixel set of the depth image;

a second processing module for obtaining a repaired pixel set according to the historical frame depth image; and

replacing the set of hole pixels of the depth image with the set of repaired pixels to obtain an enhanced depth image;

The first processing module is configured to construct an objective function according to the enhanced depth image and the guide image and perform a global optimization calculation to repair the depth image.
The repair device according to claim 11, wherein the first processing module comprises:

An optimization unit, configured to optimize the objective function so that the objective function obtains a minimum value, and output the pixel value of the current pixel of the repaired depth image corresponding to the minimum value, and the objective function is:

Among them, i is the position of the current pixel, ui is the pixel value of the current pixel, λ is the total smoothing weight coefficient in the frame, j is the pixel position of the neighborhood N(i) of i, and g is For the guide image, w i,j (g) is the edge enhancement coefficient corresponding to the guide image, u j is the pixel value of the pixel point in the neighborhood of the current pixel point, and f i is the difference between the depth image and the depth image. The pixel value corresponding to the current pixel point.
The repair device according to claim 17, wherein the neighborhood N(i) is a 4-neighborhood or an 8-neighborhood.
The repair device according to claim 17, wherein the value range of λ is [100, 10000].
The repair device of claim 17, wherein
g i is the pixel value of the guide image corresponding to the current pixel point, g j is the pixel value of the guide image corresponding to the j point of the neighborhood N(i), and the value range of σ is [1 , 10].
A camera assembly, characterized in that the camera assembly includes an image sensor, a depth sensor, and a processor, and the processor is used to obtain a current scene image, the current scene image includes a plurality of different object regions, and the current scene Each of the object regions of the image is mapped to different pixel value ranges to obtain a guide image; an objective function is constructed according to the depth image and the guide image, and a global optimization calculation is performed to restore the depth image.
An electronic device, characterized in that the electronic device comprises:

The camera assembly of claim 21; and

A casing, the camera assembly is arranged on the casing.