CN114842063A - Depth map optimization method, device, equipment and storage medium - Google Patents

Abstract

The present application provides a depth map optimization method, device, device and storage medium, and relates to the field of image processing. The method includes: acquiring a depth map and an RGB map corresponding to a real scene; segmenting the RGB map to obtain a segment map corresponding to the RGB map; determining an effective depth in the depth map according to the segment map; The optimized depth map corresponding to the real scene is obtained by fusion. This method can solve the problems of holes, noise, and environmental influences in the depth map collected by the depth camera, and output an optimized depth map with better quality and higher resolution with low power consumption and low delay.

Description

Depth map optimization method, device, device and storage medium

technical field

The embodiments of the present application relate to the field of image processing, and in particular, to a depth map optimization method, apparatus, device, and storage medium.

Background technique

Augmented reality (AR) technology is a technology that skillfully integrates virtual information with real scenes. The main principle of AR technology is to integrate virtual information with real scenes. In the process of fusing the virtual information with the real scene, the virtual and real occlusion processing between the virtual information and the real scene will be involved. When performing virtual and real occlusion processing, it is necessary to obtain the depth information of the real scene and the depth information of the virtual information, and then realize the occlusion effect between the real scene and the virtual information according to the depth information of the two. Among them, the virtual information is generated by calculation and simulation, so the depth information of the virtual information is known. Therefore, one of the keys to the virtual and real occlusion processing is how to obtain the depth information of the real scene.

At present, the method of acquiring the depth information of the real scene is generally: collecting a depth map corresponding to the real scene through a depth camera (or called a depth sensor), and the depth map contains the depth information of the real scene.

However, the depth map corresponding to the real scene obtained by the depth camera is of poor quality and low resolution.

SUMMARY OF THE INVENTION

The embodiments of the present application provide a depth map optimization method, device, device, and storage medium, which can solve the problems of holes, noise, and environmental influences in the depth map collected by the depth camera, and have better output quality with low power consumption and low delay. Optimized depth map with higher resolution.

In a first aspect, an embodiment of the present application provides a depth map optimization method, the method includes: acquiring a depth map and an RGB map corresponding to a real scene; segmenting the RGB map to obtain a segment map corresponding to the RGB map; according to the segment map, Determine the effective depth in the depth map; according to the effective depth in the depth map and the segmentation map, the optimized depth map corresponding to the real scene is obtained by fusion.

The depth optimization method preprocesses the RGB image collected by the color camera and the depth image collected by the depth camera, segmentes the preprocessed RGB image, and obtains the segmented image corresponding to the RGB image, and then associates the depth image with the RGB image. Fusion of the segmentation maps can obtain an optimized depth map with better quality and higher resolution. At the same time, by obtaining the optimized depth map through the depth map optimization method, lower latency and power consumption can be achieved.

For example, the depth map collected by the depth camera has many problems such as holes, noise, and environmental influence. After the depth map collected by the depth camera is optimized by this method, the quality of the optimized depth map will be better.

For another example, in general, the resolution of the TOF depth camera is 240*180. When the application scene usually requires a depth of 5 meters (or even further), the TOF depth camera requires strong exposure, and high exposure and high resolution must be lead to high power consumption. Through this depth map optimization method, the resolution of the depth map can be improved. Therefore, the resolution (such as 48*30) and frame rate (such as 10fps) of the depth map collected by the TOF depth camera can be reduced to achieve low power consumption. Purpose. In addition, since the depth map optimization method does not need to use a special algorithm to perform superdivision processing on the depth map, the processing process is more lightweight and simple, so that a lower delay can be achieved.

Optionally, determining the effective depth in the depth map according to the segmentation map includes: traversing the depth map pixel by pixel, acquiring information of the first object instance corresponding to each pixel in the depth map, and obtaining each first object instance in the depth map. The sum of the effective depths corresponding to an object instance, and the number of pixels in the effective depths. According to the effective depth in the depth map and the segmentation map, the optimized depth map corresponding to the real scene is obtained by fusion, including: according to the sum of the effective depths corresponding to each first object instance in the depth map and the number of pixels of the effective depth , determine the average depth of each first object instance; upsample the segmentation map to the first resolution, and fill in the average depth of the first object instance for each first object instance to obtain an optimized depth map corresponding to the real scene .

Optionally, the segmentation map is a portrait segmentation map, and the first object instance is a portrait instance.

The embodiments of the present application are also applicable to optimizing the depth of other objects (such as animals, trees, etc.) in the depth map. In other words, the embodiments of the present application can be extended and applied to optimize depth maps in more scenes, not limited to portrait depth.

Optionally, before determining the effective depth in the depth map according to the segmentation map, the method further includes: scaling the segmentation map to the same size as the depth map.

Optionally, before the RGB image is segmented, the method further includes: aligning and normalizing the RGB image and the depth map.

Optionally, the step of segmenting the RGB image to obtain a segmentation image corresponding to the RGB image includes: using a trained deep neural network model to infer the RGB image to obtain a mask image corresponding to the RGB image; The connected regions are segmented to obtain a segmentation map.

Optionally, the depth map is a depth map collected by a depth camera, or a monocular estimated depth map; wherein the depth camera includes any one of a structured light depth camera, a time-of-flight depth camera, and a binocular depth camera.

That is, through the depth map optimization method, the depth map acquired by the depth camera can be optimized, and the depth map estimated by the monocular can also be optimized.

In a second aspect, an embodiment of the present application provides an apparatus for optimizing a depth map, and the apparatus can be used to implement the method described in the first aspect above. The functions of the apparatus may be implemented by hardware, or by executing corresponding software by hardware. The hardware or software includes one or more modules or units corresponding to the above functions, for example, an acquisition module, a segmentation module, a multi-information fusion module, and the like.

Among them, the acquisition module is used to obtain the depth map and RGB map corresponding to the real scene; the segmentation module is used to segment the RGB image to obtain the segmentation map corresponding to the RGB image; the multi-information fusion module is used to determine the depth according to the segmentation map. The effective depth in the map is obtained, and the optimized depth map corresponding to the real scene is obtained by fusion according to the effective depth in the depth map and the segmentation map.

Optionally, the multi-information fusion module is specifically used to traverse the depth map pixel by pixel, obtain information of the first object instance corresponding to each pixel in the depth map, and obtain the effective depth sum corresponding to each first object instance in the depth map. , and the number of pixels of the effective depth; determine the average depth of each first object instance according to the sum of the effective depths corresponding to each first object instance in the depth map and the number of pixels of the effective depth; upsample the segmentation map to the first resolution, and fill in the average depth of the first object instance for each first object instance to obtain an optimized depth map corresponding to the real scene.

Optionally, the segmentation map is a portrait segmentation map, and the first object instance is a portrait instance.

Optionally, before determining the effective depth in the depth map according to the segmentation map, the multi-information fusion module is further configured to scale the segmentation map to the same size as the depth map.

Optionally, the apparatus further includes: a preprocessing module for aligning and normalizing the RGB map and the depth map.

Optionally, the segmentation module is specifically used for inferring the RGB image by using the trained deep neural network model to obtain a mask image corresponding to the RGB image; and segmenting the connected area of the mask image to obtain a segmentation image.

Optionally, the depth map is a depth map collected by a depth camera, or a monocular estimated depth map; wherein the depth camera includes any one of a structured light depth camera, a time-of-flight depth camera, and a binocular depth camera.

In a third aspect, an embodiment of the present application provides an electronic device, including: a processor, a memory for storing instructions executable by the processor; when the processor is configured to execute the instructions, the electronic device achieves the implementation of the first aspect. The depth map optimization method described above.

The electronic device may be a mobile terminal such as a mobile phone, a tablet computer, a wearable device, a vehicle-mounted device, an AR/VR device, a notebook computer, a super mobile personal computer, a netbook, a personal digital assistant, and the like.

In a fourth aspect, an embodiment of the present application provides a computer-readable storage medium on which computer program instructions are stored; when the computer program instructions are executed by an electronic device, the electronic device is enabled to implement the depth map optimization method described in the first aspect .

In a fifth aspect, the embodiments of the present application further provide a computer program product, including computer-readable codes, which, when the computer-readable codes are executed in an electronic device, enable the electronic device to implement the depth map optimization described in the foregoing first aspect method.

For the beneficial effects of the second aspect to the fifth aspect, reference may be made to the description in the first aspect, which will not be repeated here.

It should be understood that the description of technical features, technical solutions, beneficial effects or similar language in this application does not imply that all features and advantages may be realized in any single embodiment. On the contrary, it can be understood that the description of features or beneficial effects means that a specific technical feature, technical solution or beneficial effect is included in at least one embodiment. Therefore, descriptions of technical features, technical solutions or beneficial effects in this specification do not necessarily refer to the same embodiments. Furthermore, the technical features, technical solutions and beneficial effects described in this embodiment can also be combined in any appropriate manner. Those skilled in the art will understand that an embodiment can be implemented without one or more specific technical features, technical solutions or beneficial effects of a specific embodiment. In other embodiments, additional technical features and benefits may also be identified in specific embodiments that do not embody all embodiments.

Description of drawings

FIG. 1 shows a schematic interface diagram of an AR application;

FIG. 2 shows a schematic structural diagram of a terminal device provided by an embodiment of the present application;

FIG. 3 shows a schematic flowchart of a depth map optimization method provided by an embodiment of the present application;

FIG. 4 shows a schematic logical diagram of a depth map optimization method provided by an embodiment of the present application;

FIG. 5 shows a schematic diagram of the effect of the depth map optimization method provided by the embodiment of the present application;

FIG. 6 shows a schematic structural diagram of a depth map optimization apparatus provided by an embodiment of the present application;

FIG. 7 shows another schematic structural diagram of the depth map optimization apparatus provided by the embodiment of the present application.

Detailed ways

Augmented reality (AR) technology is a technology that skillfully integrates virtual information with real scenes. The computer-generated text, images, 3D models, music, videos and other virtual information are simulated and applied to the real scene.

Taking an AR application installed on a mobile phone as an example, FIG. 1 shows a schematic interface diagram of an AR application. As shown in Figure 1, in the AR application running on the mobile phone, the image of the real scene captured by the camera can be fused with the virtual information synthesized virtually, and displayed on the screen of the mobile phone. The virtual information may include virtual text, pictures, videos, and the like.

It can be seen that the main principle of AR technology is to integrate virtual information with real scenes. In the process of fusing the virtual information with the real scene, the virtual and real occlusion processing between the virtual information and the real scene will be involved. For example, there will be a certain spatial positional relationship between virtual information and the real scene. From the user's perspective or from the display effect, this spatial positional relationship is also the occlusion relationship between the virtual information and the real scene. For example: when the virtual information When it is a certain virtual object, the occlusion relationship between the virtual object and the object in the real scene may be that the virtual object occludes the object in the real scene, or the object in the real scene occludes the virtual object. Therefore, in the process of fusing the virtual information with the real scene, it is necessary to combine the occlusion relationship between the virtual information and the real scene to perform corresponding virtual and real occlusion processing.

When performing virtual and real occlusion processing, it is necessary to obtain the depth information of the real scene and the depth information of the virtual information, and then realize the occlusion effect between the real scene and the virtual information according to the depth information of the two. Among them, the virtual information is generated by calculation and simulation, so the depth information of the virtual information is known. Therefore, one of the keys to the virtual and real occlusion processing is how to obtain the depth information of the real scene.

At present, the method of acquiring the depth information of the real scene is generally: collecting a depth map corresponding to the real scene through a depth camera (or called a depth sensor), and the depth map contains the depth information of the real scene. Depth cameras can include two types of active depth cameras and passive depth cameras.

Among them, the active depth cameras mainly include: structured light (structured light) depth cameras and time of flight (time of flight, TOF) depth cameras. The basic principle of the structured light depth camera is to project light with certain structural characteristics onto the object to be photographed (that is, the object in the real scene) through the near-infrared laser, and then collect it by a special infrared camera. This kind of light with a certain structure will collect different image phase information due to different depth regions of the subject, and then convert the change of this structure into depth information through the computing unit, so as to obtain the depth map corresponding to the real scene. The basic principle of the TOF depth camera is to transmit modulated light pulses through an infrared transmitter. After encountering the reflection of objects in the real scene, the light pulses reflected by the receiver are used, and the TOF depth camera is calculated according to the round-trip time of the light pulses. The distance between objects in the scene, so as to obtain the depth map corresponding to the real scene.

Passive cameras mainly include binocular depth cameras. The basic principle of the binocular depth camera is based on the principle of parallax and using imaging equipment to obtain two images of objects in the real environment from different positions, and then perform image feature matching on the two images, by calculating the corresponding feature points of the two images. The position deviation between them can be obtained to obtain the depth map corresponding to the real scene.

However, the quality of the depth map corresponding to the real scene obtained by the above-mentioned depth camera is poor. For example, when a depth map corresponding to a real scene is obtained by a structured light depth camera, it is easily disturbed by ambient light (such as sunlight), resulting in poor quality of the depth map. The depth map acquired by the TOF depth camera has a lot of noise, such as multi-path interference (MPI) noise, crosstalk, scattering, etc., and because it emits active light, it is difficult to calculate for objects with low reflectivity in real scenes Depth, like black hair. When the depth map corresponding to the real scene is acquired by the binocular depth camera, depending on the quality of the acquired image, it is difficult to calculate the depth in the brighter, darker or textureless areas of the image.

In addition, the resolution of the depth map corresponding to the real scene obtained by the depth camera is also low. For example, the resolution of the depth map obtained by a common depth camera is 240*180, but when performing virtual and real occlusion processing, the resolution of the color map (ie RGB map) corresponding to the real scene is generally higher, such as: The ratio is 1920×1080, so the depth map also needs to be super-divided.

Based on the problems of poor quality and low resolution of a depth map corresponding to a real scene obtained by a depth camera at present, an embodiment of the present application provides a depth map optimization method. The method includes: acquiring an RGB image collected by a color camera and a depth image collected by a depth camera. Preprocess the RGB map and depth map. The preprocessed RGB image is segmented to obtain the segmented image corresponding to the RGB image. The depth map and the corresponding segmentation map of the RGB map are fused to obtain the optimized depth map. Through this method, the depth map collected by the depth camera can be optimized, and an optimized depth map with better quality and higher resolution can be obtained.

It can be understood that the RGB image collected by the color camera and the depth image collected by the depth camera are the RGB image and the depth image corresponding to the real scene.

Exemplarily, the method can be applied to a terminal device. The terminal device can be a mobile phone, a tablet computer, a wearable device, a vehicle-mounted device, an augmented reality (AR)/virtual reality (VR) device, a laptop, an ultra-mobile personal computer (UMPC) ), a netbook, a personal digital assistant (personal digital assistant, PDA) and other mobile terminals, the embodiments of the present application do not limit the specific type of the terminal device.

The embodiments of the present application will be specifically described below with reference to the accompanying drawings, taking the terminal device as a mobile phone as an example.

It should be noted that, in the description of this application, "at least one" refers to one or more, and "a plurality" refers to two or more. The words "first", "second" and the like are only used for distinguishing descriptions, and are not used for special limitation on a certain feature. "And/or" is used to describe the association relationship of associated objects, indicating that there are three kinds of relationships. For example, A and/or B can mean that A exists alone, A and B exist at the same time, and B exists alone. The character "/" generally indicates that the associated objects are an "or" relationship.

Taking the terminal device as a mobile phone as an example, FIG. 2 shows a schematic structural diagram of a terminal device provided by an embodiment of the present application. As shown in FIG. 2, the mobile phone may include a processor 210, an external memory interface 220, an internal memory 221, a universal serial bus (USB) interface 230, a charging management module 240, a power management module 241, a battery 242, an antenna 1. Antenna 2, Mobile Communication Module 250, Wireless Communication Module 260, Audio Module 270, Speaker 270A, Receiver 270B, Microphone 270C, Headphone Interface 270D, Sensor Module 280, Key 290, Motor 291, Indicator 292, Camera 293, Display screen 294, and a subscriber identification module (SIM) card interface 295 and the like.

The processor 210 may include one or more processing units, for example, the processor 210 may include an application processor (application processor, AP), a modem processor, a graphics processor (graphics processing unit, GPU), an image signal processor ( image signal processor, ISP), controller, memory, video codec, digital signal processor (DSP), baseband processor, and/or neural-network processing unit (NPU), etc. . Wherein, different processing units may be independent devices, or may be integrated in one or more processors.

Among them, the controller can be the nerve center and command center of the mobile phone. The controller can generate an operation control signal according to the instruction operation code and timing signal, and complete the control of fetching and executing instructions.

A memory may also be provided in the processor 210 for storing instructions and data. In some embodiments, the memory in processor 210 is cache memory. This memory may hold instructions or data that have just been used or recycled by the processor 210 . If the processor 210 needs to use the instruction or data again, it can be called directly from the memory. Repeated accesses are avoided, and the waiting time of the processor 210 is reduced, thereby improving the efficiency of the system.

In some embodiments, the processor 210 may include one or more interfaces. The interface may include an integrated circuit (inter-integrated circuit, I2C) interface, an integrated circuit built-in audio (inter-integrated circuitsound, I2S) interface, a pulse code modulation (pulse code modulation, PCM) interface, a universal asynchronous receiver (universal asynchronous receiver) interface /transmitter, UART) interface, mobile industry processor interface (mobile industry processor interface, MIPI), general-purpose input/output (general-purpose input/output, GPIO) interface, SIM interface, and/or USB interface, etc.

The external memory interface 220 can be used to connect an external memory card, such as a Micro SD card, to expand the storage capacity of the mobile phone. The external memory card communicates with the processor 210 through the external memory interface 220 to realize the data storage function. For example to save files like music, video etc in external memory card.

Internal memory 221 may be used to store computer executable program code, which includes instructions. The processor 210 executes various functional applications and data processing of the mobile phone by executing the instructions stored in the internal memory 221 . The internal memory 221 may include a storage program area and a storage data area. The storage program area can store an operating system, an application program required for at least one function (such as a sound playback function, an image playback function, etc.), and the like. The storage data area can store data (such as image data, phone book, etc.) created during the use of the mobile phone. In addition, the internal memory 221 may include high-speed random access memory, and may also include non-volatile memory, such as at least one magnetic disk storage device, flash memory device, universal flash storage (UFS), and the like.

The charging management module 240 is used to receive charging input from the charger. While the charging management module 240 charges the battery 242 , it can also supply power to the mobile phone through the power management module 241 . The power management module 241 is used to connect the battery 242 , the charging management module 240 , and the processor 210 . The power management module 241 can also receive the input of the battery 242 to supply power to the mobile phone.

The wireless communication function of the mobile phone can be realized by the antenna 1, the antenna 2, the mobile communication module 250, the wireless communication module 260, the modulation and demodulation processor, the baseband processor, and the like. Antenna 1 and Antenna 2 are used to transmit and receive electromagnetic wave signals. Each antenna in a cell phone can be used to cover a single or multiple communication frequency bands. Different antennas can also be reused to improve antenna utilization. For example, the antenna 1 can be multiplexed as a diversity antenna of the wireless local area network. In other embodiments, the antenna may be used in conjunction with a tuning switch.

The mobile phone can implement audio functions through an audio module 270, a speaker 270A, a receiver 270B, a microphone 270C, an earphone interface 270D, and an application processor. Such as music playback, recording, etc.

The sensor module 280 may include a pressure sensor 280A, a gyro sensor 280B, an air pressure sensor 280C, a magnetic sensor 280D, an acceleration sensor 280E, a distance sensor 280F, a proximity light sensor 280G, a fingerprint sensor 280H, a temperature sensor 280J, a touch sensor 280K, and an ambient light sensor 280L, bone conduction sensor 280M, etc.

Display screen 294 is used to display images, videos, and the like. Display screen 294 includes a display panel. The display panel can be a liquid crystal display (LCD), an organic light-emitting diode (OLED), an active-matrix organic light-emitting diode or an active-matrix organic light-emitting diode (active-matrix organic light-emitting diode). , AMOLED), flexible light-emitting diodes (flex light-emitting diodes, FLED), Miniled, MicroLed, Micro-oLed, quantum dot light-emitting diodes (quantum dot light emitting diodes, QLED) and so on. In some embodiments, the cell phone may include 1 or N display screens 294, where N is a positive integer greater than 1. For example, the display screen 294 may be used to display a photo-taking interface, a photo-playing interface, and the like.

The mobile phone realizes the display function through the GPU, the display screen 294, and the application processor. The GPU is a microprocessor for image processing, and is connected to the display screen 294 and the application processor. The GPU is used to perform mathematical and geometric calculations for graphics rendering. Processor 210 may include one or more GPUs that execute program instructions to generate or alter display information.

It can be understood that the structure shown in FIG. 2 does not constitute a specific limitation on the mobile phone. In some embodiments, the mobile phone may also include more or less components than those shown in FIG. 2 , or combine some components, or separate some components, or different component arrangements, and the like. Alternatively, some of the components shown in FIG. 2 may be implemented in hardware, software, or a combination of software and hardware.

In addition, when the terminal device is other tablet computer, wearable device, vehicle-mounted device, AR/VR device, notebook computer, UMPC, netbook, PDA and other mobile terminals, the specific structure of these other terminal devices can also be referred to as shown in FIG. 2 . Exemplarily, other terminal devices may have components added or reduced on the basis of the structure given in FIG. 2 , which will not be repeated here.

FIG. 3 shows a schematic flowchart of a depth map optimization method provided by an embodiment of the present application. As shown in FIG. 3 , the depth map optimization method provided in this embodiment of the present application may include S301-S304.

S301. Acquire an RGB image collected by a color camera and a depth image collected by a depth camera.

The depth camera may be any one of a structured light (structured light) depth camera, a time-of-flight (TOF) depth camera, and a binocular depth camera, or may also be other types of depth map cameras. This is not limited.

S302. Preprocess the RGB map and the depth map.

For example, the field of view (FOV) of the RGB map and the depth map can be aligned, and preprocessing such as normalization can be performed to obtain the preprocessed RGB map and depth map.

S303 , segment the preprocessed RGB image to obtain a segmented image.

Exemplarily, the preprocessed RGB image can be input into a deep convolutional neural network (DCNN) model for portrait segmentation inference, and the DCNN model can perform a portrait mask on the preprocessed RGB image. (mask) inference to get the mask map output by the DCNN model. Then, the mask image output by the DCNN model can be divided into connected regions to obtain the portrait segmentation map corresponding to the RGB image. That is, the segmentation map obtained in S303 may be a portrait segmentation map.

Optionally, the above DCNN model is a neural network model trained offline. In some other implementation manners, other network models for segmentation inference can also be used when segmenting the preprocessed RGB, which is not limited to the DCNN model for portrait segmentation inference, which is not limited here.

S304 , fuse the segmentation map and the preprocessed depth map to obtain an optimized depth map.

Exemplarily, taking the segmentation map obtained in S303 as an example of a portrait segmentation map, the step of fusing the segmentation map and the preprocessed depth map may include: traversing the preprocessed depth map pixel by pixel, and obtaining the preprocessed depth map. Portrait instance information corresponding to each pixel in the depth map (in the embodiment of this application, a portrait instance or other object instance may be referred to as the first object instance), if it is a background (not a portrait), ignore it; if it is a portrait, record effective depth. Count the sum of the effective depths corresponding to each portrait instance in the preprocessed depth map and the number of pixels in the effective depth. number to calculate the average depth of each portrait instance. Then, the portrait segmentation map is up-sampled to the first resolution and the aforementioned average depth is filled for each portrait instance, so that a portrait depth map with better quality and higher resolution can be obtained, that is, the portrait depth map is optimized depth map.

The depth map optimization method is illustrated in more detail below by taking the segmentation map as a portrait segmentation map as an example.

Referring to FIG. 4 and FIG. 5 , FIG. 4 shows a schematic diagram of the logic of the depth map optimization method provided by the embodiment of the present application, and FIG. 5 is a schematic diagram of the effect of the depth map optimization method provided by the embodiment of the present application.

As shown in FIG. 4 , in the embodiment of the present application, the process of preprocessing the RGB image collected by the color camera and the depth image collected by the depth camera may be completed by a preprocessing module. (Mobile phone) After obtaining the RGB image collected by the color camera and the depth image collected by the depth camera, the RGB image and the depth image can be input into the preprocessing module, and the preprocessing module can preprocess the RGB image and the depth map.

Assuming that the RGB image is Rgb@1440*1080 (ie, the resolution is 1440*1080), and the depth map is DepthLR@48*30 (ie, the resolution is 48*30), the preprocessing module preprocesses the RGB image. The steps can be as follows. Wherein, Rgb@1440*1080 may refer to (a) in FIG. 5 , and DepthLR@48*30 may refer to (b) in FIG. 5 .

1) Scale Rgb@1440*1080 to 288*288 to get RgbSmall@288*288.

2) From the rotation angle R, rotate RgbSamll@288*288 to keep the character level, and get RgbSmallR@288*288.

3) According to the normalization coefficients (r1, r2, g1, g2, b1, b2), the three channels of RgbSmallR@288*288 are respectively normalized, for example, the R channel is subtracted by r1, and divided by r2, G The channel is subtracted by g1 and divided by g2, and the B channel is subtracted by b1 and divided by b2 to get RGBSmallRN@288*288.

4) Perform channel conversion on RGBSmallRN@288*288 to obtain RgbSmallRNC@288*288.

Specifically, since the data arrangement of RGBSmallRN@288*288 is HWC (H represents height, W represents width, and C represents channel), and the input data arrangement of DCNN model is CHW, therefore, it is necessary to perform RGBSmallRN@288*288 Channel conversion to obtain RgbSmallRNC@288*288 whose data arrangement is CHW.

The above preprocessing is performed on Rgb@1440*1080, and the obtained RgbSmallRNC@288*288 can be referred to as shown in (c) in FIG. 5 .

The steps of preprocessing the depth map by the preprocessing module may be as follows.

1) Crop DepthLR@48*30 to get DepthLRCrop@40*30.

Specifically, due to the data format problem of DepthLR, although the resolution of DepthLR is 48*30, only 40*30 is actually available, so it is necessary to crop the rightmost 8*30 first to obtain DepthLRCrop@40*30.

2) Perform data analysis on DepthLRCrop@40*30 to obtain DepthLRCropT@40*30.

Specifically, the DepthLR data obtained in step 1) is generally stored in the ushort type, and each value is stored in 16 bits, while the real depth value is only the last 13 bits, so it is necessary to parse the data to obtain the actual depth of the last 13 bits. DepthLRCropT@40*30.

3) According to the Rgb-D alignment coefficient (scale), align DepthLRCropT@40*30 with the preprocessed RGB image to obtain DepthAlign@54*40. scale can also be called a scaling factor, which can be 1.348 here.

Specifically, the resolution of the aligned depth map calculated according to the Rgb-D alignment coefficient should be 54*40. Therefore, DepthLRCropT@40*30 needs to be filled with DepthAlign@54*40.

The above preprocessing is performed on DepthLR@48*30, and the obtained DepthAlign@54*40 can be referred to as shown in (d) in FIG. 5 .

It can be understood that in the aforementioned process of preprocessing the depth map and RGB, the involved parameters such as the rotation angle R, the Rgb-D alignment coefficient, and the normalization coefficient can be determined according to the acquired depth map and RGB image. For example, it can be determined according to the camera parameters when the color camera and the depth camera collect the RGB image and the depth image, and will not be described in detail here.

After preprocessing the RGB image and the depth map respectively, the preprocessed RGB image can be segmented to obtain a portrait segmentation map corresponding to the RGB image. Among them, the process of segmenting the preprocessed RGB image can be completed by the DCNN segmentation module.

The DCNN segmentation module can include DCNN_PeopleSeg (DCNN_PeopleSeg is a deep convolutional neural network model for portrait segmentation inference, which has been trained offline), and a connected region segmentation sub-module. Taking the preprocessed RGB image: RgbSmallRNC@288*288 as an example, you can input RgbSmallRNC@288*288 into DCNN_PeopleSeg for inference, and get the MaskOut@288*288 output by DCNN_PeopleSeg. Then, the connected area segmentation sub-module can segment the connected area of MaskOut@288*288 to obtain Mask@288*28 in ushort format, where the label of the background in Mask@288*28 is 0, and the label of the portrait instance starts from 1 . For example, Mask@288*28 can refer to (e) in Figure 5, including two portrait instances, then the label of the first portrait instance (such as the left) is 1, and the second portrait instance (right) The label is 2.

After the portrait segmentation map corresponding to the RGB image is obtained, the portrait segmentation map and the preprocessed depth map can be fused to obtain an optimized depth map. The process of fusing the portrait segmentation map and the aforementioned preprocessed depth map can be realized by a multi-information fusion module.

Taking the aforementioned portrait segmentation map: Mask@288*28 and the preprocessed depth map DepthAlign@54*40 as examples, the multi-information fusion module fuses the portrait segmentation map and the aforementioned preprocessed depth map The steps can be as follows.

1) Scale Mask@288*288 to the size of DepthAlign@54*40 to get MaskLR@54*40.

2) Traverse MaskLR@54*40 and DepthAlign@54*40 at the same time to obtain the number of valid depth values num_i and the sum of depth values value_i corresponding to each label i (i is equal to 0, 1, 2, etc.). Taking label 2 as an example (that is, the second portrait instance), traverse each pixel (x, y) in MaskLR@54*40 and DepthAlign@54*40 at the same time, x and y represent pixel coordinates, when MaskLR(x , y) is equal to 2, and when DepthALign(x, y) is greater than 0, add 1 to num_2, and add DepthALign(x, y) to value_2. According to this step, traverse the whole image to get num_i and value_i of all labels.

3) Calculate the average depth value of each label, that is, divide value_i by num_i to get avg_i. At this time, the avg_i of each label i is the average depth value corresponding to the label i.

4) Initialize DepthT@288*288, traverse all pixels (x, y) in Mask@288*288, when Mask(x, y) is equal to i, the value of DepthT(x, y) is avg_i. After all pixels are traversed, DepthT@288*288 with the true depth value is obtained.

5) Scale DepthT@288*288 to the first resolution and output, for example, when the first resolution is 640*480, you can output DepthHR@640*480. DepthHR@640*480 can refer to (f) in FIG. 5 .

Compared with the depth map DepthLR@48*30 collected by the depth camera, in the embodiment of the present application, in the depth map DepthHR@640*480 obtained through the aforementioned processing process, the human body area is filled with the complete depth value, and the human body area is filled with the complete depth value. The depth of the instance is more accurate and the quality will be better, in addition, the resolution of DepthHR@640*480 is also higher. Thus, optimization of the depth map acquired by the depth camera is achieved.

Optionally, although optimizing the depth of a portrait in a depth map is described as an example in the foregoing embodiments of the present application, the embodiments of the present application are also applicable to optimizing the depths of other objects (such as animals, trees, etc.) in the depth map. . In other words, the embodiments of the present application can be extended and applied to optimize depth maps in more scenes, not limited to portrait depth. For example, the aforementioned DCNN model can be trained as a network model for other object segmentation inference to optimize depth maps in more scenarios.

In addition, it should also be noted that the depth map optimization method provided in this embodiment of the present application can not only be applied to the virtual-real fusion scene in the AR technology mentioned in the foregoing embodiment, but also can be applied to large-aperture blurring and collision detection In other application scenarios based on depth map, there is no limit here.

Optionally, in some other embodiments of the present application, the depth map acquired by the depth camera described in S301 can also be replaced with a depth map obtained in other ways, for example, it can be a monocular estimated depth map. The application is also not limited here.

Further, at present, there are also some depth map processing methods. For some mobile application scenarios (such as mobile phones, tablet computers, wearable devices, etc.), real-time occlusion processing requires real-time processing. For the RGB image corresponding to a frame of real scene, the corresponding depth image needs to be collected by the depth camera for virtual and real occlusion processing. Therefore, these shooting hardware devices will occupy a high power consumption of the mobile phone. With the depth map optimization method provided by the embodiments of the present application, the power consumption of the mobile phone can also be reduced to a certain extent, and the virtual and real occlusion processing with low power consumption and low delay can be realized.

Take the mobile phone collecting the depth map corresponding to the real scene through the TOF depth camera (or TOF depth map sensor) as an example. Generally speaking, the resolution of the TOF depth camera is 240*180, and the application scene usually requires 5 meters (or even further) When the depth is high, the TOF depth camera requires strong exposure, and high exposure and high resolution will inevitably lead to high power consumption. The depth map optimization method provided by the embodiments of the present application can improve the resolution of the depth map. Therefore, the resolution (such as 48*30) and frame rate (such as 10fps) of the depth map collected by the TOF depth camera can be achieve the purpose of low power consumption. In addition, since the depth map optimization method does not need to use a special algorithm to perform superdivision processing on the depth map, the processing process is more lightweight and simple, so that a lower delay can be achieved.

To sum up, the depth optimization method provided by the embodiments of the present application performs preprocessing on the RGB image collected by the color camera and the depth image collected by the depth camera, and then divides the preprocessed RGB image to obtain the corresponding RGB image. , and then fuse the depth map with the corresponding segmentation map of the RGB image to obtain an optimized depth map with better quality and higher resolution. At the same time, by obtaining the optimized depth map through the depth map optimization method, lower latency and power consumption can be achieved.

That is, the depth optimization method provided by the embodiments of the present application can solve the problems such as holes, noise, and environmental influences in the depth map collected by the depth camera, and the output quality and resolution are better with low power consumption and low delay. The optimized depth map of .

It should be understood that the depth map optimization method provided in the embodiments of the present application can also be applied to other non-mobile terminals, such as servers, computers and other devices, and can also achieve the same beneficial effects as those described in the foregoing embodiments, which will not be repeated here. .

Corresponding to the depth map optimization method described in the foregoing embodiments, an embodiment of the present application further provides a depth map optimization apparatus, which can be applied to a terminal device. The functions of the apparatus may be implemented by hardware, or by executing corresponding software by hardware. The hardware or software includes one or more modules or units corresponding to the above functions. For example, FIG. 6 shows a schematic structural diagram of an apparatus for optimizing a depth map provided by an embodiment of the present application. As shown in FIG. 6, the depth map optimization apparatus may include: an acquisition module 601, a segmentation module 602, a multi-information fusion module 603, and the like.

Among them, the obtaining module 601 is used to obtain the depth map and RGB map corresponding to the real scene. The segmentation module 602 is configured to segment the RGB image to obtain a segmentation image corresponding to the RGB image. The multi-information fusion module 603 is configured to determine the effective depth in the depth map according to the segmentation map, and fuse to obtain an optimized depth map corresponding to the real scene according to the effective depth in the depth map and the segmentation map.

Optionally, the multi-information fusion module 603 is specifically used to traverse the depth map pixel by pixel, obtain the information of the first object instance corresponding to each pixel in the depth map, and obtain the effective depth corresponding to each first object instance in the depth map. The sum and the number of pixels of the effective depth; the average depth of each first object instance is determined according to the sum of the effective depths corresponding to each first object instance in the depth map and the number of pixels of the effective depth; the segmentation map is upsampled to the first resolution, and fill in the average depth of the first object instance for each first object instance to obtain an optimized depth map corresponding to the real scene.

Optionally, the segmentation map is a portrait segmentation map, and the first object instance is a portrait instance.

Optionally, before determining the effective depth in the depth map according to the segmentation map, the multi-information fusion module 603 is further configured to scale the segmentation map to the same size as the depth map.

Optionally, FIG. 7 shows another schematic structural diagram of the depth map optimization apparatus provided by the embodiment of the present application. As shown in FIG. 7 , the depth map optimization apparatus further includes: a preprocessing module 604 for aligning and normalizing the RGB map and the depth map.

Optionally, the segmentation module 602 is specifically configured to use the trained deep neural network model to infer the RGB image to obtain a mask image corresponding to the RGB image; and to segment the connected region of the mask image to obtain a segmentation image.

For example, the segmentation module 602 may be the aforementioned DCNN segmentation module.

Optionally, the depth map is a depth map collected by a depth camera, or a monocular estimated depth map; wherein the depth camera includes any one of a structured light depth camera, a time-of-flight depth camera, and a binocular depth camera.

It should be understood that the division of units or modules (hereinafter referred to as units) in the above apparatus is only a division of logical functions, and in actual implementation, all or part of them may be integrated into a physical entity, or may be physically separated. And all the units in the device can be realized in the form of software calling through the processing element; also can all be realized in the form of hardware; some units can also be realized in the form of software calling through the processing element, and some units can be realized in the form of hardware.

For example, each unit can be a separately established processing element, or can be integrated in a certain chip of the device to be implemented, and can also be stored in the memory in the form of a program, which can be called by a certain processing element of the device and execute the unit's processing. Function. In addition, all or part of these units can be integrated together, and can also be implemented independently. The processing element described here may also be called a processor, which may be an integrated circuit with signal processing capability. In the implementation process, each step of the above method or each of the above units may be implemented by an integrated logic circuit of hardware in the processor element or implemented in the form of software being invoked by the processing element.

In one example, a unit in the above apparatus may be one or more integrated circuits configured to implement the above method, eg, one or more application specific integrated circuits (ASICs), or, one or more A digital signal processor (DSP), or, one or more field programmable gate arrays (FPGA), or a combination of at least two of these integrated circuit forms.

For another example, when a unit in the apparatus can be implemented in the form of a processing element scheduler, the processing element can be a general-purpose processor, such as a CPU or other processors that can invoke programs. For another example, these units can be integrated together and implemented in the form of a system-on-a-chip (SOC).

In one implementation, the unit of the above apparatus for implementing each corresponding step in the above method may be implemented in the form of a processing element scheduler. For example, the apparatus may include a processing element and a storage element, and the processing element invokes a program stored in the storage element to execute the method described in the above method embodiments. The storage element may be a storage element on the same chip as the processing element, ie, an on-chip storage element.

In another implementation, the program for performing the above method may be in a storage element on a different chip from the processing element, ie, an off-chip storage element. At this time, the processing element calls or loads the program from the off-chip storage element to the on-chip storage element, so as to call and execute the methods described in the above method embodiments.

For example, an embodiment of the present application may further provide an apparatus, such as an electronic device, which may include a processor, a memory for storing instructions executable by the processor. When the processor is configured to execute the above instructions, the electronic device implements the depth map optimization method described in the foregoing embodiments. The memory may be located within the electronic device or external to the electronic device. And the processor includes one or more.

In yet another implementation, the unit of the apparatus implementing each step in the above method may be configured as one or more processing elements, where the processing elements may be integrated circuits, such as: one or more ASICs, or, one or more Multiple DSPs, or, one or more FPGAs, or a combination of these types of integrated circuits. These integrated circuits can be integrated together to form chips.

For example, an embodiment of the present application further provides a chip, which can be applied to the above-mentioned electronic device. The chip includes one or more interface circuits and one or more processors; the interface circuit and the processor are interconnected by lines; the processor receives and executes computer instructions from the memory of the electronic device through the interface circuit, so as to realize the above-mentioned embodiments. Depth map optimization.

Embodiments of the present application also provide a computer program product, including computer-readable codes, which, when the computer-readable codes are executed in an electronic device, enable the electronic device to implement the depth map optimization described in the foregoing embodiments.

From the description of the above embodiments, those skilled in the art can clearly understand that for the convenience and brevity of the description, only the division of the above functional modules is used as an example for illustration. In practical applications, the above functions can be allocated as required. It is completed by different functional modules, that is, the internal structure of the device is divided into different functional modules to complete all or part of the functions described above.

In the several embodiments provided in this application, it should be understood that the disclosed apparatus and method may be implemented in other manners. For example, the device embodiments described above are only illustrative. For example, the division of the modules or units is only a logical function division. In actual implementation, there may be other division methods. For example, multiple units or components may be Incorporation may either be integrated into another device, or some features may be omitted, or not implemented. On the other hand, the shown or discussed mutual coupling or direct coupling or communication connection may be through some interfaces, indirect coupling or communication connection of devices or units, and may be in electrical, mechanical or other forms.

The units described as separate components may or may not be physically separated, and the components shown as units may be one physical unit or multiple physical units, that is, they may be located in one place, or may be distributed to multiple different places . Some or all of the units may be selected according to actual needs to achieve the purpose of the solution in this embodiment.

In addition, each functional unit in each embodiment of the present application may be integrated into one processing unit, or each unit may exist physically alone, or two or more units may be integrated into one unit. The above-mentioned integrated units may be implemented in the form of hardware, or may be implemented in the form of software functional units.

If the integrated unit is implemented in the form of a software functional unit and sold or used as an independent product, it may be stored in a readable storage medium. Based on this understanding, the technical solutions of the embodiments of the present application essentially or contribute to the prior art, or all or part of the technical solutions may be embodied in the form of software products, such as programs. The software product is stored in a program product, such as a computer-readable storage medium, and includes several instructions to cause a device (which may be a single-chip microcomputer, a chip, etc.) or a processor (processor) to execute all of the methods described in the various embodiments of the present application. or partial steps. The aforementioned storage medium includes: a U disk, a removable hard disk, a ROM, a RAM, a magnetic disk, or an optical disk and other mediums that can store program codes.

For example, the embodiments of the present application may further provide a computer-readable storage medium on which computer program instructions are stored. The computer program instructions, when executed by an electronic device, cause the electronic device to implement the depth map optimization as described in the previous embodiments.

The above are only specific embodiments of the present application, but the protection scope of the present application is not limited to this, and any changes or substitutions within the technical scope disclosed in the present application should be covered within the protection scope of the present application. . Therefore, the protection scope of the present application should be subject to the protection scope of the claims.

Claims (10)

1. A method for depth map optimization, the method comprising:

acquiring a depth map and an RGB map corresponding to a real scene;

dividing the RGB map to obtain a division map corresponding to the RGB map;

determining effective depth in the depth map according to the segmentation map;

and according to the effective depth in the depth map and the segmentation map, fusing to obtain an optimized depth map corresponding to the real scene.

2. The method of claim 1, wherein determining the effective depth in the depth map from the segmentation map comprises:

traversing the depth map pixel by pixel, acquiring information of a first object example corresponding to each pixel in the depth map, and obtaining the effective depth sum corresponding to each first object example in the depth map and the number of pixels of the effective depth;

the obtaining of the optimized depth map corresponding to the real scene through fusion according to the effective depth in the depth map and the segmentation map includes:

determining the average depth of each first object instance according to the effective depth sum corresponding to each first object instance in the depth map and the number of pixels of the effective depth;

and upsampling the segmentation map to a first resolution, and filling the average depth of the first object instance for each first object instance to obtain an optimized depth map corresponding to the real scene.

3. The method of claim 2, wherein the segmentation map is a portrait segmentation map and the first object instance is a portrait instance.

4. The method of any of claims 1-3, wherein prior to determining the effective depth in the depth map from the segmentation map, the method further comprises:

scaling the segmentation map to the same size as the depth map.

5. The method according to any of claims 1-4, wherein prior to said segmenting said RGB map, said method further comprises:

and aligning and normalizing the RGB map and the depth map.

6. The method according to any one of claims 1 to 5, wherein the segmenting the RGB map to obtain the segmentation map corresponding to the RGB map comprises:

reasoning the RGB map by adopting a trained deep neural network model to obtain a mask map corresponding to the RGB map;

and carrying out connected region segmentation on the mask image to obtain the segmentation image.

7. The method according to any of claims 1-6, wherein the depth map is a depth map acquired by a depth camera or a monocular estimated depth map;

wherein the depth camera comprises: any one of a structured light depth camera, a time-of-flight depth camera, and a binocular depth camera.

8. A depth map optimization apparatus, characterized in that the apparatus comprises:

the acquisition module is used for acquiring a depth map and an RGB map corresponding to a real scene;

the segmentation module is used for segmenting the RGB image to obtain a segmentation image corresponding to the RGB image;

and the multi-information fusion module is used for determining the effective depth in the depth map according to the segmentation map, and fusing to obtain the optimized depth map corresponding to the real scene according to the effective depth in the depth map and the segmentation map.

9. An electronic device, comprising: a processor, a memory for storing the processor-executable instructions;

the processor is configured to, when executing the instructions, cause the electronic device to implement the method of any of claims 1-7.

10. A computer readable storage medium having stored thereon computer program instructions; it is characterized in that the preparation method is characterized in that,

the computer program instructions, when executed by an electronic device, cause the electronic device to implement the method of any of claims 1-7.

Images