CN115456998A

CN115456998A - Depth map processing method, device and storage medium

Info

Publication number: CN115456998A
Application number: CN202211128056.6A
Authority: CN
Inventors: 赵明喜; 钱哲弘
Original assignee: Yinniu Microelectronics Wuxi Co ltd
Current assignee: Yinniu Microelectronics Wuxi Co ltd
Priority date: 2022-09-16
Filing date: 2022-09-16
Publication date: 2022-12-09

Abstract

The embodiment of the disclosure provides a depth map processing method, a depth map processing device and a storage medium, wherein a depth map sequence of continuous frames and a color map sequence which is registered with the depth map sequence one by one are obtained, the depth map sequence comprises a target depth map of the current time and each frame depth map of the previous time, the color map sequence is used for estimating the displacement change of each pixel in the target depth map, under the condition that the displacement change of at least one target pixel is displayed to be static, the target pixel is positioned in the depth map sequence based on the registration relation between the depth map sequence and the color map sequence, time filtering is adopted, and the depth information of the target pixel in each frame depth map of the previous time is used for replacing the depth information of the target pixel in the target depth map. The method and the device can avoid adopting time filtering on the moving object in the depth map, thereby improving the accuracy of hole filling in the depth map.

Description

Depth map processing method, device and storage medium

Technical Field

The present disclosure relates to the field of image processing technologies, and in particular, to a depth map processing method and apparatus, and a storage medium.

Background

In 3D computer graphics and computer vision, a depth map is an image or image channel that contains information about the distance of the surface of scene objects from a viewpoint for simulated 3D shapes or reconstructing them.

However, when the depth map is acquired, there may be a region or an algorithm problem that is not captured by the camera, and thus a hole at an edge portion appears in the depth map, as shown in fig. 1, a black region 10 is a hole compared to an adjacent region 11. The holes contain both foreground and background, but the depth values of the foreground and background are very different.

Therefore, the related art effectively fills the hole through the temporal filtering.

In the related art, temporal filtering refers to refining a depth map of a current time based on depth information of a depth map of a previous time. The term "depth map for the current time" refers to a depth image that is currently being processed by a depth image refinement device to obtain a refined depth image. The term "depth map of a previous time" refers to a depth image that was previously processed by a depth image refinement device to obtain a refined depth image.

Wherein the temporal filtering includes replacing the depth information of the depth image of the current time with the depth information of the depth image of the previous time, or replacing the depth information of the depth image of the current time with a weighted sum of the depth information of the depth image of the current time and the depth information of the depth image of the previous time. However, these are merely examples, and other types of temporal filtering may be used.

To summarize, temporal filtering generally works as follows: and for a certain pixel point, obtaining the depth value of the pixel point in the previous n frames of depth images, and replacing the depth value of the current pixel point with the depth information of the n pixel points.

Since holes are unlikely to appear in the depth maps of frames before the current time depth map at the same time, the current time depth map can be filled by adopting the temporal filtering to solve the problem of the holes in the depth map.

Disclosure of Invention

In view of the above drawbacks of the related art, an object of the present disclosure is to provide a depth map processing method, device and storage medium, so as to solve the technical problem that the related art cannot accurately fill a depth map hole.

A first aspect of the present disclosure provides a depth map processing method, including:

acquiring a depth map sequence of continuous frames and a color map sequence which is registered with the depth map sequence one by one, wherein the depth map sequence comprises a target depth map of the current time and each frame depth map of the previous time;

estimating a displacement change of each pixel in the target depth map by using the color map sequence;

locating a target pixel in the depth map sequence based on a registration relationship between the depth map sequence and the color map sequence in a case where a displacement change of at least one target pixel appears to be stationary;

and adopting the time filtering to replace the depth information of the target pixel in the target depth map by the depth information of the target pixel in each frame depth map at the previous time.

In some embodiments, estimating a change in displacement for each pixel in the target depth map using the sequence of color maps, including;

and estimating a motion vector of each pixel in the target depth map by using the color map sequence by adopting an optical flow method, and obtaining displacement change based on the motion vector.

In some embodiments, the depth processing method further comprises:

for pixels whose displacement variation is estimated to be moving, their depth values are retained in the target depth map.

In some embodiments, estimating a change in displacement for each pixel in the target depth map using the sequence of color maps comprises:

in the color image sequence, the target color image registered with the target depth image is taken, and for each pixel in the target color image, the displacement change between every two adjacent frame color images in the previous time is detected.

In some embodiments, in a case where the change in displacement of the at least one target pixel appears to be stationary, the depth map processing method further includes, before locating the target pixel in the depth map sequence based on the registration relationship between the depth map sequence and the color map sequence:

for any pixel, if the displacement between each pair of adjacent two-frame color images is estimated to be static, the pixel is determined to be a static target pixel.

for any pixel, if the displacement between at least one pair of adjacent two-frame color images is estimated to be non-static, it is determined as a moving pixel, and the depth value is kept in the target depth image.

In some embodiments, in a case that the displacement change of the at least one target pixel appears to be static, based on the registration relationship between the depth map sequence and the color map sequence, before locating the target pixel in the depth map sequence, the depth map processing method further includes:

judging whether the displacement change of each pixel is not greater than a threshold value;

and taking the pixel of which the displacement change is not more than the threshold value as a target pixel, wherein the displacement change of the target pixel is static.

A second aspect of the present disclosure provides a depth map processing apparatus, including:

the acquisition module is used for acquiring a depth map sequence of continuous frames and a color map sequence which is registered with the depth map sequence one by one, wherein the depth map sequence comprises a target depth map at the current time and each frame depth map at the previous time;

the displacement estimation module is used for estimating the displacement change of each pixel in the target depth map by utilizing the color map sequence;

the pixel positioning module is used for positioning the target pixel in the depth map sequence on the basis of the registration relation between the depth map sequence and the color map sequence under the condition that the displacement change of at least one target pixel is displayed to be static;

and the time filtering module adopts time filtering to replace the depth information of the target pixel in the target depth map by the depth information of the target pixel in each frame depth map at the previous time.

A fifth aspect of the present disclosure provides a computer apparatus comprising: a communicator, a memory, and a processor; the communicator is used for communicating with the outside; the memory stores program instructions; the processor is configured to execute program instructions to perform a depth map processing method as claimed in any one of the first aspect.

A fourth aspect of the present disclosure provides a computer-readable storage medium storing program instructions that are executed to perform the depth map processing method according to any one of the first aspects.

As described above, in the embodiments of the present disclosure, a depth map processing method, an apparatus, and a storage medium are provided, where a depth map sequence of consecutive frames and a color map sequence in one-to-one registration with the depth map sequence are obtained, the depth map sequence includes a target depth map at a current time and frame depth maps at previous times, a displacement change of each pixel in the target depth map is estimated by using the color map sequence, and in a case where a displacement change of at least one target pixel is displayed to be stationary, the target pixel is located in the depth map sequence based on a registration relationship between the depth map sequence and the color map sequence, and temporal filtering is used to replace depth information in the target depth map with depth information in the frame depth maps at previous times for the target pixel. In the embodiment of the present disclosure, pixel displacement detection is performed on the target depth map at the current time in advance to determine stationary pixel points in the target depth map, and time filtering is performed on the stationary pixel points. Therefore, the method and the device can avoid adopting time filtering on the moving object in the depth map, so that the accuracy rate of filling holes in the depth map is improved.

Drawings

Fig. 1 shows a depth map of the related art.

Fig. 2 shows one of the flowcharts of the depth map processing method according to the embodiment of the present disclosure.

Fig. 3 shows a second flowchart of a depth map processing method according to an embodiment of the disclosure.

Fig. 4 shows a depth map obtained using a related art temporal filtering process.

Fig. 5 shows a depth map obtained using the temporal filtering process of an embodiment of the present disclosure.

Fig. 6 shows a module schematic diagram of a depth map processing apparatus according to an embodiment of the present disclosure.

Fig. 7 shows a schematic structural diagram of a computer device according to an embodiment of the disclosure.

Detailed Description

Embodiments of the present disclosure are described below with reference to specific examples, and other advantages and effects of the present disclosure will be readily apparent to those skilled in the art from the disclosure. The disclosure may be embodied or carried out in various other specific embodiments and with various modifications or alterations from various aspects and applications of the disclosure without departing from the spirit of the disclosure. It is to be noted that the embodiments and features of the embodiments in the present disclosure may be combined with each other without conflict.

Embodiments of the present disclosure are described in detail below with reference to the accompanying drawings so that those skilled in the art to which the present disclosure pertains can easily carry out the embodiments. The present disclosure may be embodied in many different forms and is not limited to the embodiments described herein.

Reference in the representation of the present disclosure to the terms "one embodiment," "some embodiments," "an example," "a specific example," or "some examples," etc., means that a particular feature, structure, material, or characteristic described in connection with the embodiment or example is included in at least one embodiment or example of the present disclosure. Furthermore, the particular features, structures, materials, or characteristics illustrated may be combined in any suitable manner in any one or more embodiments or examples. Moreover, various embodiments or examples and features of different embodiments or examples presented in this disclosure can be combined and combined by one skilled in the art without contradiction.

Furthermore, the terms "first", "second" are used for descriptive purposes only and are not to be construed as indicating or implying relative importance or implicitly indicating the number of technical features indicated. Thus, a feature defined as "first" or "second" may explicitly or implicitly include at least one such feature. In the expressions of the present disclosure, "plurality" means two or more unless specifically defined otherwise.

In order to clearly explain the present disclosure, components that are not related to the description are omitted, and the same reference numerals are given to the same or similar components throughout the specification.

Throughout the specification, when a device is referred to as being "connected" to another device, this includes not only the case of being "directly connected" but also the case of being "indirectly connected" with another element interposed therebetween. In addition, when a device "includes" a certain constituent element, unless otherwise specified, it means that the other constituent element is not excluded, but may be included.

Although the terms first, second, etc. may be used herein to describe various elements in some instances, these elements should not be limited by these terms. These terms are only used to distinguish one element from another. For example, a first interface, a second interface, etc. Also, as used herein, the singular forms "a", "an" and "the" are intended to include the plural forms as well, unless the context indicates otherwise. It will be further understood that the terms "comprises," "comprising," "includes" and/or "including," when used in this specification, specify the presence of stated features, steps, operations, elements, modules, items, species, and/or groups, but do not preclude the presence, or addition of one or more other features, steps, operations, elements, modules, items, species, and/or groups thereof. The terms "or" and/or "as used herein are to be construed as inclusive or meaning any one or any combination. Thus, "a, B or C" or "a, B and/or C" means "any of the following: a; b; c; a and B; a and C; b and C; A. b and C ". An exception to this definition will occur only when a combination of elements, functions, steps or operations performed in some manner are inherently mutually exclusive.

The terminology used herein is for the purpose of describing particular embodiments only and is not intended to be limiting of the disclosure. As used herein, the singular forms "a", "an" and "the" include plural forms as long as the words do not expressly indicate a contrary meaning. The use of "including" in the specification is meant to specify the presence of stated features, regions, integers, steps, elements, and/or components, but does not preclude the presence or addition of other features, regions, integers, steps, elements, components, and/or groups thereof.

Terms representing relative spatial terms such as "lower", "upper", and the like may be used to more readily describe one element's relationship to another element as illustrated in the figures. Such terms are intended to have not only the meaning indicated in the drawings, but also other meanings or executions of the device in use. For example, if the device in the figures is turned over, elements described as "below" other elements would then be oriented "above" the other elements. Thus, the exemplary terms "under" and "beneath" all include above and below. The device may be rotated 90 or other angles and the terminology representing relative space is also to be interpreted accordingly.

Although not defined differently, including technical and scientific terms used herein, all have the same meaning as commonly understood by one of ordinary skill in the art to which this disclosure belongs. Terms defined in commonly used dictionaries are to be additionally interpreted as having meanings consistent with those of related art documents and currently prompted messages, and should not be excessively interpreted as having ideal or very formulaic meanings unless defined.

The inventor finds in practice that temporal filtering can only be applied to static scenes, and if a scene has moving objects, the values after temporal filtering will be incorrect, so that temporal filtering will not be suitable for scenes with moving objects.

Therefore, the embodiment of the present disclosure provides that, by performing displacement change detection on a pixel in a current depth map, when it is detected that the pixel is still, temporal filtering is applied to the pixel, which can avoid applying temporal filtering processing to a moving object in the depth map, thereby improving accuracy of hole filling.

Fig. 2 is a flowchart of a depth map processing method provided in an embodiment of the present disclosure, an execution subject of the method is a depth map processing system, and as shown in fig. 2, the method includes the following steps:

step 210: acquiring a depth map sequence of continuous frames and a color map sequence which is registered with the depth map sequence one by one, wherein the depth map sequence comprises a target depth map of the current time and each frame depth map of the previous time;

step 220: estimating a displacement change of each pixel in the target depth map by using the color map sequence;

step 230: under the condition that the displacement change of at least one target pixel is displayed to be static, positioning the target pixel in the depth map sequence based on the registration relation between the depth map sequence and the color map sequence;

step 240: and adopting the time filtering to replace the depth information of the target pixel in the target depth map by the depth information of the target pixel in each frame depth map at the previous time.

In the embodiment of the present disclosure, pixel displacement detection is performed on the target depth map at the current time in advance to determine stationary pixel points in the target depth map, and time filtering is performed on the stationary pixel points. Therefore, the method and the device can avoid adopting time filtering on the moving object in the depth map, so that the accuracy rate of filling holes in the depth map is improved.

In the embodiment of the disclosure, the one-to-one registration of the depth map sequence and the color map sequence means that the shooting time, the image size and the shooting position of each pair of registered depth map and color map are consistent, which can improve the accuracy and feasibility of the scheme.

The depth map sequence is a multi-frame depth map which is continuously shot, and the color map sequence is a multi-frame color map which is continuously shot. Wherein the depth map of each frame at the previous time is used for temporally filtering the target depth map at the current time.

In the embodiment of the present disclosure, because the depth map sequence and the color map sequence are registered one by one, for each pixel in the target depth map, a corresponding pixel can be found in the color map at the current time, so that each pixel in the target depth map can be estimated to be a displacement change by using the color map sequence.

In an alternative embodiment, estimating a change in displacement for each pixel in the target depth map using the sequence of color maps, includes;

Optical flow (optical flow) is the instantaneous velocity of pixel motion of a spatially moving object on the viewing imaging plane. The optical flow method is a method for calculating motion information of an object between adjacent frames by finding out a corresponding relationship between a previous frame and a current frame by using a change of pixels in an image sequence in a time domain and a correlation between the adjacent frames.

In summary: the so-called optical flow is the instantaneous velocity, and is equivalent to the displacement of the target point when the time interval is small (such as between two consecutive frames of the video).

Specifically, the basic principle of the optical flow method is:

1. basic assumption condition

(1) The brightness is constant. I.e. the brightness of the same pixel does not change when it moves between different frames. This is an assumption of the basic optical flow method (all optical flow variants have to be satisfied) for obtaining the basic equations of the optical flow method;

(2) The temporal succession or motion is "small motion". That is, the time variation does not cause a drastic change in the target position, and the displacement between adjacent frames is relatively small. It is also an indispensable assumption of the optical flow method.

2. Basic constraint equation

Consider the light intensity of a pixel I (xy, t) in the first frame (where t represents the time dimension in which it is located). It moves the distance (dx, dy) to the next frame, taking dt times. Because it is the same pixel point, according to the first assumption mentioned above, the light intensity of the pixel before and after the motion is unchanged, that is:

I(x,y,t)＝I(x+dx,y+dy,t+dt) (1)

performing Taylor expansion on the right end of the formula (1) to obtain:

where ε represents the second order infinite small term and is negligible. Then, dt is removed after the generation (2) of people (1) to obtain:

setting mu and ν as velocity vectors of optical flows along an X axis and a Y axis respectively, and obtaining:

order to

Respectively representing the partial derivatives of the gray levels of pixel points in the image along the X, Y and T directions.

In summary, equation (3) can be written as:

I _x μ+I _u ν+I _t ＝0 (5)

wherein, I _x ，I _y ，I _t All can be obtained from the image data, and (mu, v) is the obtained optical flow vector.

The constraint equation has only one and the unknowns of the equation have two, in which case the exact values of μ and ν cannot be found. At this time, additional constraint conditions need to be introduced, and the constraint conditions are introduced from different angles, so that different optical flow field calculation methods are caused. They are divided into four categories according to the difference between the theoretical basis and the mathematical method: gradient (differential) based methods, matching based methods, energy (frequency) based methods, phase based methods, and neurodynamic methods, without limitation.

In this embodiment, optical flow detection is performed on pixels in every two adjacent frames of color images in the color image sequence to obtain a motion vector of each pixel, where the motion vector is a two-dimensional vector, so that a displacement change can be obtained according to the two-dimensional vector. In this case, for each pixel, its displacement variation in the multi-frame color map can be obtained.

Specifically, estimating a displacement change of each pixel in the target depth map by using the color map sequence may specifically include:

and taking a target color image registered with the target depth image in the color image sequence, and detecting the displacement change between every two adjacent frames of color images in the previous time for each pixel in the target color image.

In this embodiment, for each pixel, each two adjacent frames of color images can obtain a displacement change correspondingly, and a plurality of displacement changes can be obtained correspondingly to the color image sequence.

In one embodiment, for any pixel, if it is estimated to be stationary between each pair of adjacent two frame color maps, it is determined to be a stationary target pixel.

In another embodiment, for any pixel, in case that the displacement between at least one pair of adjacent two frame color maps is estimated to be non-static, it is determined as a moving pixel, and its depth value is retained in the target depth map.

Therefore, for pixels for which the displacement change is estimated to be moving, the depth value is retained in the target depth map and is not temporally filtered.

In an embodiment of the disclosure, in a case where the displacement change of the at least one target pixel is displayed as being stationary, based on the registration relationship between the depth map sequence and the color map sequence, before locating the target pixel in the depth map sequence, the depth map processing method further includes:

and taking the pixel of which the displacement change is not more than the threshold value as a target pixel, wherein the displacement change of the target pixel is static, and otherwise, determining the target pixel as a moving pixel.

Next, fig. 3 provides a specific depth map processing method versus a flowchart, and specifically, the method specifically includes the following steps:

step 310: acquiring a current frame depth map Dm and a previous m-1 frame depth map D1 \8230 \ 8230;, dm and a color map sequence R1 \8230;, rm after registration, wherein the previous m1 frame depth map is used as time filtering;

step 320: carrying out dense optical flow detection on each color image Ri and R (i-1) before the color image Ri and R (i-1), detecting the displacement of each pixel point to obtain a motion vector of each pixel, wherein the motion vector is a two-dimensional vector, and displacement change is obtained

And setting a threshold Th, wherein if the threshold Th is larger than the Th, the pixel is considered to be a moving pixel, and otherwise, the pixel is a static target pixel.

Step 330: for each pixel in the current frame depth map Dm, checking whether m-1 pixel points in the color map sequence R2 \8230areall static points or not. If the depth values are all static points, the median (maximum value, minimum value or mean value) of m-1 depths of the corresponding pixels in the D2 \8230Dmis taken as the depth value of the current point to replace the depth value of the corresponding pixel in the current frame depth map Dm; if not all the points are the static points, the depth value in the Dm is reserved, and filtering is not performed.

Fig. 4 shows a depth map obtained by using a temporal filtering process of the related art, and fig. 5 shows a depth map obtained by using a temporal filtering process of an embodiment of the present disclosure, and fig. 5 has fewer hole regions compared to fig. 4, indicating that the embodiment of the present disclosure can fill the holes of the depth map more accurately.

Fig. 6 is a block diagram of a depth map processing apparatus according to an embodiment of the disclosure. It should be noted that, the principle of the depth map processing apparatus may refer to the depth map processing method in the previous embodiment, and therefore, repeated description of the same technical content is omitted here.

The depth map processing apparatus 600 may include:

an obtaining module 610, configured to obtain a depth map sequence of consecutive frames and a color map sequence in one-to-one registration with the depth map sequence, where the depth map sequence includes a target depth map at a current time and depth maps of frames at previous times;

a displacement estimation module 620, which estimates the displacement change of each pixel in the target depth map by using the color map sequence;

a pixel locating module 630, for locating the target pixel in the depth map sequence based on the registration relationship between the depth map sequence and the color map sequence under the condition that the displacement change of the at least one target pixel is displayed as static;

the temporal filtering module 640, using temporal filtering, replaces the depth information of the target pixel in the target depth map with the depth information of the target pixel in the depth maps of the frames at the previous time.

In some embodiments, the displacement estimation module 620 is specifically configured to:

In some embodiments, the temporal filtering module 640 is specifically further configured to:

In some embodiments, the pixel location module 630 is further specifically configured to:

in the case that the displacement change of at least one target pixel is displayed to be static, based on the registration relation between the depth map sequence and the color map sequence, before the target pixel is positioned in the depth map sequence, for any pixel, if the displacement between each pair of adjacent two frames of color maps is estimated to be static, the pixel is determined to be the static target pixel.

in the case that the displacement change of at least one target pixel is displayed as static, based on the registration relationship between the depth map sequence and the color map sequence, before locating the target pixel in the depth map sequence, for any pixel, if the displacement between at least one pair of adjacent two-frame color maps is estimated to be non-static, determining the pixel as a moving pixel, and keeping the depth value of the pixel in the target depth map.

In some embodiments, the displacement estimation module 620 is further specifically configured to:

under the condition that the displacement change of at least one target pixel is displayed to be static, judging whether the displacement change of each pixel is not greater than a threshold value or not before positioning the target pixel in the depth map sequence based on the registration relation between the depth map sequence and the color map sequence;

It should be noted that, in particular, each functional module in the embodiment of fig. 6 may be implemented in whole or in part by software, hardware, firmware, or any combination thereof. When implemented in software, may be implemented in whole or in part in the form of program instruction products. The program instruction product includes one or more program instructions. The processes or functions according to the present disclosure are produced in whole or in part when program instruction instructions are loaded and executed on a computer. The computer may be a general purpose computer, a special purpose computer, a network of computers, or other programmable device. The program instructions may be stored in a computer readable storage medium or transmitted from one computer readable storage medium to another computer readable storage medium.

Moreover, the apparatus disclosed in the embodiment of fig. 6 can be implemented by other module division methods. The above-described embodiments of the apparatus are merely illustrative, and for example, the division of modules is merely a logical division, and other divisions may be implemented in practice, for example, a plurality of modules or modules may be combined or may be dynamic to another system, or some features may be omitted, or not implemented. In addition, the shown or discussed mutual coupling or direct coupling or communication connection may be through some interfaces, indirect coupling or communication connection of devices or modules, and may be in an electrical or other form.

In addition, each functional module and sub-module in the embodiment of fig. 6 may be dynamically in one processing unit, or each module may exist alone physically, or two or more modules may be dynamically in one unit. The dynamic component can be implemented in the form of hardware, or in the form of a software functional module. The dynamic components described above, if implemented in the form of software functional modules and executed as separate products for sale or use, may also be stored in a computer readable storage medium. The storage medium may be a read-only memory, a magnetic or optical disk, or the like.

A computer readable storage medium may also be provided in the disclosed embodiments, which stores program instructions that, when executed, perform the method steps in the previous fig. 2 embodiment.

The method steps in the above-described embodiments are implemented as software or computer code storable in a recording medium such as a CD ROM, a RAM, a floppy disk, a hard disk, or a magneto-optical disk, or computer code originally stored in a remote recording medium or a non-transitory machine-readable medium and to be stored in a local recording medium downloaded through a network, so that the method represented herein can be stored in such software processes on a recording medium using a general-purpose computer, a dedicated processor, or programmable or dedicated hardware such as an ASIC or FPGA.

It should be particularly noted that the flowcharts or method representations of the above-described embodiments of the present disclosure may be understood as representing modules, segments or portions of code which include one or more executable instructions for implementing specific logical functions or steps of the process. And the scope of the preferred embodiments of the present disclosure includes additional implementations in which functions may be executed out of order from that shown or discussed, including substantially concurrently or in reverse order, depending on the functionality involved.

For example, the order of the steps in the embodiment shown in fig. 2 may be changed in a specific scenario, and is not limited to the above representation.

Fig. 7 is a schematic structural diagram of a computer device according to an embodiment of the disclosure.

In some embodiments, a computer device is used to load program instructions implementing a depth map processing method. The computer device may be embodied as, for example, a server, desktop, laptop, mobile terminal, etc., as may be used by an implementer who stores and/or executes such program instructions for commercial purposes such as development, testing, etc.

The computer device 700 illustrated in fig. 7 is only an example and should not impose any limitations on the functionality or scope of use of embodiments of the present disclosure.

As shown in fig. 7, the computer apparatus 700 is in the form of a general purpose computing device. The components of the computer device 700 may include, but are not limited to: the at least one processing unit 710, the at least one memory unit 720, and a bus 730 that couples various system components including the memory unit 720 and the processing unit 710.

The storage unit stores program codes, which can be executed by the processing unit 710, so that the computer apparatus is configured to implement the method steps described in the above embodiment (for example, the embodiment in fig. 2) of the present disclosure.

In some embodiments, the memory unit 720 may include volatile memory units such as a random access memory unit (RAM) 721 and/or a cache memory unit 722, and may further include a read only memory unit (ROM) 723.

In some embodiments, the memory unit 720 may also include a program/utility 724 having a set (at least one) of program modules 725, such program modules 725 including, but not limited to: an operating system, one or more application programs, other program modules, and program data, each of which or some combination thereof may comprise an implementation of a network environment.

In some embodiments, bus 730 may include a data bus, an address bus, and a control bus.

In some embodiments, the computer apparatus 700 may also communicate with one or more external devices 70 (e.g., keyboard, pointing device, bluetooth device, etc.), which may be through an input/output (I/O) interface 750. Optionally, the computer device 700 further comprises a display unit 740 connected to the input/output (I/O) interface 750 for display. Moreover, computer device 700 may also communicate with one or more networks (e.g., a Local Area Network (LAN), a Wide Area Network (WAN), and/or a public network such as the Internet) via network adapter 760. As shown, network adapter 760 communicates with the other modules of computer device 700 via bus 730. It should be appreciated that although not shown in the figures, other hardware and/or software modules may be used in conjunction with computer device 700, including but not limited to: microcode, device drivers, redundant processing units, external disk drive arrays, RAID systems, tape drives, and data backup storage systems, among others.

In summary, embodiments of the present disclosure provide a depth map processing method, an apparatus, and a storage medium, in which a depth map sequence of consecutive frames and a color map sequence in one-to-one registration with the depth map sequence are obtained, the depth map sequence includes a target depth map at a current time and frame depth maps at previous times, a displacement change of each pixel in the target depth map is estimated by using the color map sequence, and when a displacement change of at least one target pixel is displayed to be stationary, the target pixel is located in the depth map sequence based on a registration relationship between the depth map sequence and the color map sequence, and temporal filtering is used to replace depth information of the target pixel in the target depth map by using depth information in the frame depth maps at previous times.

In the embodiment of the present disclosure, pixel displacement detection is performed on the target depth map at the current time in advance to determine stationary pixel points in the target depth map, and time filtering is performed on the stationary pixel points. Therefore, the moving object in the depth map which can be avoided by the embodiment of the disclosure adopts time filtering, so that the accuracy of filling the hole in the depth map is improved.

The above-described embodiments are merely illustrative of the principles of the present disclosure and their efficacy, and are not intended to limit the disclosure. Any person skilled in the art can modify or change the above-described embodiments without departing from the spirit and scope of the present disclosure. Accordingly, it is intended that all equivalent modifications or changes which may be made by those skilled in the art without departing from the spirit and technical spirit of the present disclosure be covered by the claims of the present disclosure.

Claims

1. A depth map processing method, comprising:

estimating a displacement variation of each pixel in the target depth map using the color map sequence;

in the case that the displacement change of at least one target pixel is displayed to be static, positioning the target pixel in the depth map sequence based on the registration relation between the depth map sequence and the color map sequence;

and replacing the depth information of the target pixel in the target depth map by the depth information of the target pixel in each frame depth map at the previous time by adopting the time filtering.

2. The depth map processing method of claim 1, wherein said estimating a change in displacement for each pixel in the target depth map using the color map sequence comprises;

3. The depth map processing method according to claim 1, further comprising:

4. The depth map processing method of claim 1, wherein said estimating a change in displacement for each pixel in the target depth map using the color map sequence comprises:

and taking a target color image registered with the target depth image in the color image sequence, and detecting the displacement change of each pixel in the target color image between every two adjacent frames of color images at the previous time.

5. The depth map processing method of claim 4, wherein in a case where a displacement change of at least one target pixel appears stationary, based on a registration relationship between a depth map sequence and a color map sequence, before locating the target pixel in the depth map sequence, the depth map processing method further comprises:

6. The depth map processing method of claim 4, wherein in a case that the displacement change of at least one target pixel appears to be static, based on a registration relationship between a depth map sequence and a color map sequence, before the target pixel is located in the depth map sequence, the depth map processing method further comprises:

for any pixel, if the displacement between at least one pair of adjacent two-frame color images is estimated to be non-static, determining the pixel as a moving pixel, and keeping the depth value of the pixel in the target depth image.

7. The depth map processing method of claim 1, wherein in a case that a displacement change of at least one target pixel appears to be stationary, based on a registration relationship between a depth map sequence and a color map sequence, before the target pixel is located in the depth map sequence, the depth map processing method further comprises:

and taking the pixel with the displacement change not larger than the threshold value as the target pixel, wherein the displacement change of the target pixel is static.

8. A depth map processing apparatus, comprising:

the acquisition module is used for acquiring a depth map sequence of continuous frames and a color map sequence which is in one-to-one registration with the depth map sequence, wherein the depth map sequence comprises a target depth map at the current time and each frame depth map at the previous time;

a displacement estimation module, which estimates the displacement change of each pixel in the target depth map by using the color map sequence;

9. A computer device, comprising: a communicator, a memory, and a processor; the communicator is used for communicating with the outside; the memory stores program instructions; the processor is configured to execute the program instructions to perform the depth map processing method of claims 1 to 7.

10. A computer-readable storage medium storing program instructions that are executed to perform the depth map processing method according to claims 1 to 7.