WO2021087812A1

WO2021087812A1 - Method for determining depth value of image, image processor and module

Info

Publication number: WO2021087812A1
Application number: PCT/CN2019/116021
Authority: WO
Inventors: 贾玉虎
Original assignee: Oppo广东移动通信有限公司
Priority date: 2019-11-06
Filing date: 2019-11-06
Publication date: 2021-05-14
Also published as: CN114365191A

Abstract

A method for determining a depth value of an image, an image processor (1000) and a module. The method for determining a depth value of an image comprises: for an object of a target image, acquiring a first depth image from a TOF module (11), and acquiring a second depth image from a binocular vision module (12) (S401); and using, according to a depth value of the first depth image, an accuracy value corresponding to the depth value of the first depth image, a depth value of the second depth image, and an accuracy value corresponding to the depth value of the second depth image, a preset weighting algorithm to perform weighted calculation to obtain a depth value of a target image (S402). The accuracy of a depth value of an image can thus be improved.

Description

Method for determining depth value of image, image processor and module

Technical field

The embodiments of the present application relate to depth image synthesis technology, and in particular, to a method for determining the depth value of an image, an image processor, and a module.

Background technique

The Time of Flight (TOF) module includes a modulated light transmitter, a modulated light receiver, and a depth image synthesis processor. The modulated light transmitter continuously sends modulated light signals to the target, and the receiver receives the return from the object In the depth image synthesis processor, the distance of the target object is obtained by detecting the round-trip time of the modulated light signal in the depth image synthesis processor, thereby obtaining the depth signal of the object. The TOF module has good depth perception characteristics for short-to-medium distance objects, but for long-distance and ultra-long-distance objects, accurate depth values cannot be obtained, and when the TOF module is affected by external noise and ambient light , Its depth image imaging accuracy is greatly affected.

The binocular vision module imitates the characteristics of the human eye, by matching the feature points of the object in the left and right images, using the parallax caused by the dual viewpoint imaging of the same object, and obtaining the depth signal of the object through geometric calculation. In practical applications, the feature points of the object are not accurately matched, which becomes the biggest obstacle for the binocular vision module to obtain high-precision depth values.

It can be seen that the TOF module is affected by ambient light or noise, and the accuracy of the depth image obtained is not high, and the TOF module is only suitable for short and medium distances, while the binocular vision module is caused by inaccurate matching of feature points, resulting in depth images In other words, the accuracy of the depth image obtained by the existing TOF module or binocular vision module is low.

Summary of the invention

The embodiments of the present application provide a method, an image processor, and a module for determining the depth value of an image, which can improve the accuracy of the depth value of the image.

The technical solutions of the embodiments of the present application can be implemented as follows:

In the first aspect, an embodiment of the present application provides a method for determining the depth value of an image, including:

For the object in the target image, the first depth image is obtained from the TOF module and the second depth image is obtained from the binocular vision module; wherein, the first depth image and the second depth image respectively include An accuracy value corresponding to the depth value of, where the accuracy value is used to characterize the accuracy of the corresponding depth value;

According to the depth value of the first depth image, the accuracy value corresponding to the depth value of the first depth image, the depth value of the second depth image, and the accuracy value corresponding to the depth value of the second depth image , Performing weighting calculation using a preset weighting algorithm to obtain the depth value of the target image.

In a second aspect, an embodiment of the present application provides a method for determining the depth value of an image, including:

Obtain the depth value of the target image;

According to the depth value of the target image, find the accuracy value corresponding to the depth value of the target image from the preset corresponding relationship between the depth value and the accuracy value;

Forming a first depth image by using the depth value of the target image and the accuracy value corresponding to the depth value of the target image;

The first depth image is sent to an image processor, so that the image processor determines the depth value of the target image.

In a third aspect, an embodiment of the present application provides a method for determining the depth value of an image, including:

After acquiring the depth value of the target image and the binocular vision image of the target image, the feature points of the first vision image in the binocular vision image and the features of the second vision image in the binocular vision image are respectively determined point;

Matching the feature points of the first visual image with the feature points of the second visual image to obtain a matching result of the target image;

According to the matching result of the target image, the accuracy value corresponding to the matching result of the target image is found from the preset corresponding relationship between the matching result and the accuracy value;

Using the depth value of the target image and the accuracy value corresponding to the matching result of the target image to form a second depth image;

The second depth image is sent to an image processor, so that the image processor determines the depth value of the target image.

In a fourth aspect, an embodiment of the present application provides an image processor, including:

The first acquisition module is configured to acquire a first depth image from the TOF module and a second depth image from the binocular vision module for the object in the target image; wherein, the first depth image and the second depth The images also respectively include accuracy values corresponding to the respective depth values, and the accuracy values are used to characterize the accuracy of the corresponding depth values;

A weighting module, configured to be based on the depth value of the first depth image, the accuracy value corresponding to the depth value of the first depth image, the depth value of the second depth image, and the depth value of the second depth image The corresponding accuracy value is weighted and calculated using a preset weighting algorithm to obtain the depth value of the target image.

In a fifth aspect, an embodiment of the present application provides a TOF module, including:

The second acquiring module is configured to acquire the depth value of the target image;

The first searching module is configured to find the accuracy value corresponding to the depth value of the target image from the preset corresponding relationship between the depth value and the accuracy value according to the depth value of the target image;

A first forming module configured to use the depth value of the target image and the accuracy value corresponding to the depth value of the target image to form a first depth image;

The first sending module is configured to send the first depth image to an image processor, so that the image processor determines the depth value of the target image.

In a sixth aspect, an embodiment of the present application provides a binocular vision module, including:

The determining module is configured to, after acquiring the depth value of the target image and the binocular vision image of the target image, respectively determine the feature points of the first vision image in the binocular vision image and the second vision image in the binocular image Feature points of the visual image;

A matching module, configured to match the feature points of the first visual image with the feature points of the second visual image to obtain a matching result of the target image;

The second searching module is configured to find the accuracy value corresponding to the matching result of the target image from the preset corresponding relationship between the matching result and the accuracy value according to the matching result of the target image;

A second forming module configured to use the depth value of the target image and the accuracy value corresponding to the depth value of the target image to form a second depth image;

The second sending module is configured to send the second depth image to an image processor, so that the image processor determines the depth value of the target image.

In a seventh aspect, an embodiment of the present application provides an image processor, and the image processor includes:

A processor and a storage medium storing instructions executable by the processor, the storage medium relies on the processor to perform operations through a communication bus, and when the instructions are executed by the processor, perform one or more of the foregoing implementations An example is the method for determining the depth value of the image executed by the image processor.

In an eighth aspect, an embodiment of the present application provides a TOF module, and the TOF module includes:

A processor and a storage medium storing instructions executable by the processor, the storage medium relies on the processor to perform operations through a communication bus, and when the instructions are executed by the processor, perform one or more of the foregoing implementations An example is the method for determining the depth value of the image executed by the TOF module.

In a ninth aspect, an embodiment of the present application provides a binocular vision module, and the binocular vision module includes:

A processor and a storage medium storing instructions executable by the processor, the storage medium relies on the processor to perform operations through a communication bus, and when the instructions are executed by the processor, perform one or more of the foregoing implementations An example is the method for determining the depth value of the image executed by the binocular vision module.

In a tenth aspect, an embodiment of the present application provides a computer-readable storage medium, in which executable instructions are stored, and when the executable instructions are executed by one or more processors, the processors execute one or more of the above The method for determining the depth value of the image executed by the image processor in various embodiments, or the method for determining the depth value of the image executed by the TOF module, or the binocular vision module The method for determining the depth value of the image is performed.

The embodiment of the application provides a method for determining the depth value of an image, an image processor, and a module. For the object in the target image, a first depth image is obtained from the TOF module, and a second depth image is obtained from the binocular vision module. , Wherein the first depth image and the second depth image also include accuracy values corresponding to the respective depth values, and the accuracy values are used to characterize the accuracy of the corresponding depth values, according to the preset resolution of the target image, According to the depth value of the first depth image, the accuracy value corresponding to the depth value of the first depth image, the depth value of the second depth image and the accuracy value corresponding to the depth value of the second depth image, the preset weighting algorithm is used to perform Weighted calculation to obtain the depth value of the target image; that is to say, in the embodiment of the present application, the first depth image and the second depth image are obtained from the TOF module and the binocular vision module, and then according to the depth of the first depth image Value and the corresponding accuracy value, the depth value of the second depth image and the corresponding accuracy value are weighted, so that in the process of determining the depth value of the target image, the accuracy of the depth value of the first depth image is taken into account And the accuracy of the depth value of the second depth image makes the determined depth value of the target image more accurate, which is conducive to capturing a more realistic image, thereby improving the user experience.

Description of the drawings

FIG. 1 is a schematic structural diagram of an optional image processing system provided by an embodiment of the application;

2 is a schematic diagram of flow interaction of an optional method for determining a depth value of an image provided by an embodiment of the application;

3 is a schematic flowchart of an example of an optional method for determining a depth value of an image provided by an embodiment of the application;

4 is a schematic flowchart of an optional method for determining the depth value of an image provided by an embodiment of the application;

FIG. 5 is a schematic flowchart of another optional method for determining the depth value of an image according to an embodiment of the application;

6 is a schematic flowchart of yet another optional method for determining the depth value of an image provided by an embodiment of the application;

FIG. 7 is a schematic structural diagram of an optional image processor provided by an embodiment of the application;

FIG. 8 is a schematic structural diagram of an optional TOF module provided by an embodiment of the application;

FIG. 9 is a schematic structural diagram of an optional binocular vision module provided by an embodiment of the application;

FIG. 10 is a schematic structural diagram of another optional image processor provided by an embodiment of this application;

FIG. 11 is a schematic structural diagram of another optional TOF module provided by an embodiment of the application;

FIG. 12 is a schematic structural diagram of another optional binocular vision module provided by an embodiment of the application.

Detailed ways

The technical solutions in the embodiments of the present application will be clearly and completely described below in conjunction with the drawings in the embodiments of the present application. It can be understood that the specific embodiments described here are only used to explain the related application, but not to limit the application. In addition, it should be noted that, for ease of description, only the parts related to the relevant application are shown in the drawings.

An embodiment of the application provides a method for determining the depth value of an image, which is applied to an image processing system. FIG. 1 is a schematic structural diagram of an optional image processing system provided by an embodiment of the application, as shown in FIG. 1. As shown, the image processing system includes a TOF module 11, a binocular vision module 12, and an image processor 13. A communication connection is established between the TOF module 11 and the image processor 13, and the binocular vision module 12 is connected to the A communication connection is established between the image processors 13.

Among them, in the image processing system, the TOF module 11 can obtain the distance between itself and the object in the target image by detecting the round-trip time of the modulated light signal, that is, the depth value, but the depth value determined by this method is specific to the short-to-medium-distance objects. Good depth perception characteristics, but for long-distance or ultra-long-distance objects, an accurate depth value cannot be obtained, and the TOF module 11 is easily affected by external noise and ambient light, thereby affecting the accuracy of the depth value.

In the image processing system, the binocular vision module 12 can match the feature points of the objects in the left and right images, use the parallax caused by the dual viewpoint imaging of the same object, and obtain the distance between itself and the object in the target image through geometric calculation. That is the depth value, but in practical applications, if the feature point matching is not accurate, the accuracy of the depth value will be affected.

Here, the above-mentioned image processor may be used to collect depth values from the TOF module 11 and the binocular vision module 12 respectively, and process the multiple collected depth values to determine a more accurate depth value of the target image.

In order to overcome the error in determining the depth value of the TOF module 11 or the binocular vision module 12, an embodiment of the present application provides a method for determining the depth value of an image. The method is applied to the above-mentioned image processing system. The application embodiment provides a schematic diagram of the flow interaction of an optional method for determining the depth value of an image. As shown in FIG. 2, the method may include:

S201: The TOF module 11 determines the first depth image of the target image;

In order to overcome the inability of the TOF module 11 to obtain accurate depth values for long-distance or ultra-long-distance objects, and the impact of inaccurate matching of feature points of the binocular vision module 12 on the accuracy of the depth value, here, the TOF module 11 is used to determine The first depth image of the object in the target image, the binocular vision module 12 is used to determine the second depth image of the target image, and then the image processor 13 can combine the first depth image and the second depth image to determine the depth value of the target image .

In a specific implementation process, in order to determine the first depth image of the target image by the TOF module 11, S201 may include:

S2011: The TOF module 11 obtains the depth value of the target image;

Specifically, the TOF module 11 obtains the distance between itself and the object in the target image by detecting the round-trip time of the modulated light signal, that is, the depth value of the target image can be obtained. Wherein, the depth value of the target image is the depth value of each pixel position of the target image.

S2012: The TOF module 11 finds the accuracy value corresponding to the depth value of the target image from the preset corresponding relationship between the depth value and the accuracy value according to the depth value of the target image;

After the TOF module 11 obtains the depth value of the target image in S2011, since the corresponding relationship between the preset depth value and the accuracy value is pre-stored in the TOF module 11, then, after the depth value is determined, The accuracy value corresponding to the depth value of the target image can be found from the corresponding relationship.

In practical applications, the effective distance of the TOF module 11 is used for judgment, and the transmission time of the signal is calculated according to the time point of the laser modulation signal transmission and reception to obtain the distance between the TOF module 11 and the object in the target image, that is, the depth value. , The accuracy can be set to the highest within the effective distance, or it can be set from near to far, with the accuracy gradually becoming lower or other situations; outside the effective distance, the accuracy is the lowest. For example, the range of the accuracy value can be between 0 and 1. In this way, the corresponding relationship between the depth value and the accuracy value is established to determine the accuracy value.

S2013: The TOF module 11 uses the depth value of the target image and the accuracy value corresponding to the depth value of the target image to form a first depth image.

Among them, the accuracy value is used to characterize the accuracy of the depth value.

That is to say, the first depth image determined by the TOF module 11 not only includes the depth value of the target image, but also includes the accuracy value corresponding to the depth value of the target image. The meaning of the accuracy value is that the region corresponds to The credibility of the depth value.

S202: The TOF module 11 sends the first depth image to the image processor;

In S202, the TOF module 11 sends the first depth image including the depth value and the accuracy value to the image processor, so that the image processor 13 can obtain the depth value and the accuracy of the depth value collected from the TOF module 11. The degree information helps the image processor 13 to more accurately determine the depth value of the target image.

S203: The binocular vision module 12 determines the second depth image of the target image;

In a specific implementation process, in order for the binocular vision module 12 to determine the second depth image of the target image, S203 may include:

S2031: After the binocular vision module 12 obtains the depth value of the target image and the binocular vision image of the target image, it respectively determines the feature points of the first vision image in the binocular vision image and the second vision image in the binocular image Feature points;

Specifically, the binocular vision module 12 matches the feature points of the objects in the left and right images, and uses the parallax caused by the dual viewpoint imaging of the same object to obtain the distance between itself and the object image of the target image through geometric calculations, that is, Get the depth value of the target image.

After the binocular vision module 12 obtains the depth value of the target image, it needs to determine the feature point of each visual image in the binocular image and the feature point of the second visual image, which can reflect the essence of the image Features can identify the target object in the image, and the matching of the image can be completed through the matching of feature points.

S2032: The binocular vision module 12 matches the feature points of the first vision image with the feature points of the second vision image to obtain a matching result of the target image;

Here, the binocular vision module 12 matches the feature points of the first vision image with the feature points of the second vision image, and the obtained matching results include matching success and matching failure. The matching success indicates that the first vision image and the second vision image are matched successfully. The feature points of the binocular vision image are the same or similar, which can be represented by the degree of matching. In practical applications, the degree of matching may be the sum of the absolute values of differences of the same feature point. Here, the embodiment of the present application does not specifically limit it. Matching failure means that the feature points of the first visual image and the second visual image are completely different, that is, there is no feature point that can be matched.

S2033: According to the matching result, the binocular vision module 12 finds the accuracy value corresponding to the matching result of the target image from the preset corresponding relationship between the matching result and the accuracy value;

After the binocular vision module 11 obtains the depth value of the target image in S2032, since the preset correspondence between the matching result and the accuracy value is pre-stored in the binocular vision module 12, then, when it is determined After the matching result, the accuracy value corresponding to the determined matching result can be found from the corresponding relationship.

In practical applications, based on feature point matching, in the left and right images, the pixel-level traversal is performed with a specified step length, and the feature points of the two images are searched for matching. If there is no feature point that can be matched in a certain area, the The position accuracy is the lowest. If there are feature points that can be matched, the accuracy is proportional to the matching degree. The higher the matching degree, the higher the accuracy. Among them, the matching degree can be represented by the sum of the absolute values of the differences of the same feature points in the two images, or in other forms. For example, the range of the accuracy value can be between 0 and 1. In this way, the corresponding relationship between the matching degree and the accuracy value is established to determine the accuracy value.

S2034: The binocular vision module 12 uses the depth value of the target image and the accuracy value corresponding to the matching result of the target image to form a second depth image.

That is, the second depth image determined by the binocular vision module 12 not only includes the depth value of the target image, but also includes the accuracy value corresponding to the matching result of the target image.

Wherein, the accuracy value of the first depth image and the accuracy value of the second depth image may be different in size and in different units, and may be pixel level, block level, row level or frame level.

S204: The binocular vision module 12 sends the second depth image to the image processor 13;

In S203, the binocular vision module 12 sends the second depth image including the depth value and the accuracy value to the image processor 13, so that the image processor 13 can obtain the depth value collected from the binocular vision module 12 The accuracy information of the sum depth value is beneficial to the image processor 13 to more accurately determine the depth value of the target image.

Here, it should be noted that the process of determining the first depth image by the TOF module 11 and the process of determining the second depth image by the binocular vision module 12 may be executed in parallel or sequentially. At this time, the process of the TOF module 11 determining the first depth image may precede the process of the binocular vision module 12 determining the second depth image, or the process of the binocular vision module 12 determining the second depth image before the TOF The process of the module 11 determining the first depth image, which is not specifically limited in the embodiment of the present application.

S205: The image processor 13 determines the depth value of the target image.

Here, it should be noted that, when the resolution of the target image is the same as the resolution of the first depth image and the resolution of the second depth image, S205 may include:

The image processor 13 uses a preset according to the depth value of the first depth image, the accuracy value corresponding to the depth value of the first depth image, the depth value of the second depth image, and the accuracy value corresponding to the depth value of the second depth image. The weighting algorithm performs weighting calculation to obtain the depth value of the target image.

However, when the resolution of the target image is different from the resolution of the first depth image and the resolution of the second depth image, S205 may include:

S2051: The image processor 13 respectively performs scaling processing on the first depth image and the second depth image according to the preset resolution of the target image, to obtain the processed first depth image and the processed second depth image;

S2052: The image processor 13 according to the depth value of the processed first depth image, the accuracy value corresponding to the depth value of the processed first depth image, the depth value of the processed second depth image, and the processed second depth image. The accuracy value corresponding to the depth value of the depth image is weighted and calculated using a preset weighting algorithm to obtain the depth value of the target image.

That is to say, when the resolution of the acquired first depth image is different from the resolution of the preset target image, and the resolution of the second depth image is different from the resolution of the preset target image, the first depth image needs to be checked. The image and the second depth image are scaled so that the resolution of the first depth image and the resolution of the second image depth are the same as the resolution of the preset target image, thereby obtaining the processed first depth image and the second depth image .

Here, in order for the image processor 13 to determine a more accurate depth value for the target image, here, the depth value of the processed first depth image, the accuracy value corresponding to the depth value of the processed first depth image, and the The depth value of the second depth image and the accuracy value corresponding to the depth value of the processed second depth image are input into a preset weighting algorithm for calculation, and then the depth value of the target image can be obtained.

Specifically, assuming that the row and column coordinates are i and j, the depth value of the first depth image generated by the TOF module 11 is represented by P_TOF, the accuracy value of the first depth image is represented by A_TOF, and the first depth image generated by the binocular vision module 12 is represented by P_TOF. The depth value of the second depth image is represented by P_BV, and the accuracy value of the second depth image is represented by A_BV. The depth value TMP _i,j of the target image generated by the image processor 13 in the i-th row and j-th column is calculated as follows:

Among them, K is the pixel compensation value of the accuracy value of the TOF module 11, T is the pixel compensation value of the accuracy value of the binocular vision module 12, K and T can be set according to actual debugging, and F is the overall compensation value , Can be configured according to the device and the scene, a and b are the trustworthiness values, and different values can be set according to the depth values of different distances.

In practical applications, the depth value of the processed first depth image, the accuracy value of the processed first depth image, the depth value of the second depth image, and the accuracy value of the processed second depth image are substituted into The above formula (1) is calculated to obtain the depth value of the target image.

In addition, it should be noted that in the process of determining the depth value of the target image, the image processor 13 may perform filtering processing on the depth image after the scaling process, or may perform filtering processing on the depth image after determining the depth value of the target image. The depth value of the target image can be filtered, or the depth image can be filtered after the scaling process, and the depth value of the target image can also be filtered after the depth value of the target image is determined. Here, the embodiment of the application does not do this Specific restrictions.

In an optional embodiment, S2051 may include:

The image processor 13 respectively performs scaling processing on the first depth image and the second depth image according to the preset resolution of the target image, to obtain the scaled first depth image and the scaled second depth image;

The image processor 13 respectively performs filtering processing on the scaled first depth image and the scaled second depth image to obtain the processed first depth image and the processed second depth image.

Here, the depth image is filtered after the scaling process. Specifically, the image processor 13 performs the scaling process on the first depth image and the second depth image, and then respectively performs the scaling process on the first depth image and the scaled depth image. The subsequent second depth image is subjected to filtering processing, which may be Gaussian filtering, smoothing filtering, or filtering with configurable coefficients, which is not specifically limited in the embodiment of the present application.

In an optional embodiment, after S205, the method may further include:

The image processor 13 performs filtering processing on the depth value of the target image to obtain the depth value after the filtering processing of the target image.

Here, after the depth value of the target image is determined, the depth value of the target image is filtered. Specifically, after the image processor 13 determines the depth value of the target image, the image processor 13 performs filtering processing on the depth value of the target image. The filtering process obtains the filtered depth value, which can improve the accuracy of the depth value, which may be Gaussian filtering, smoothing filtering, or filtering with configurable coefficients, which is not specifically limited in the embodiment of the present application.

The following examples are given to illustrate the method for determining the depth value of the image described in one or more embodiments above.

Example 1: First, the image processor will scale the first depth image and the second depth image obtained from the TOF module and the binocular vision module to generate a new first depth image with the same resolution as the final output target image And the new second depth image.

Secondly, let the row and column coordinates be i and j, the depth value of the new first depth image generated is P_TOF, the depth value of the new first depth image corresponds to the accuracy value A_TOF, and the depth value of the new second depth image Is P_BV, the accuracy value corresponding to the depth value of the new second depth image is A_BV, and the depth value of the target image is calculated using the above formula (1).

Finally, the image processor performs Gaussian, smoothing or configurable coefficient filtering on the depth value of the target image to obtain the final filtered depth value.

Example 2: The image processor scales the first depth image and the second depth image obtained from the TOF module and the binocular vision module, and the resolution is the same as the final output target image, and generates a new first depth image and a new The second depth image.

The image processor performs smoothing or configurable coefficient filtering on the new first depth image and the new second depth image, respectively, and obtains the depth value P_TOF of the new first depth image corresponding to the depth value of the new first depth image The accuracy value A_TOF, the depth value P_BV of the new second depth image and the accuracy value A_BV corresponding to the depth value of the new second depth image; set the row and column coordinates as i and j, and use the above formula (1) to calculate the target The depth value of the image.

Example 3: First, the image processor will scale the first depth image and the second depth image obtained from the TOF module and the binocular vision module, and generate a new first depth image at the same resolution as the final output target image And the new second depth image.

Secondly, the image processor performs smoothing or configurable coefficient filtering on the new first depth image and the new second depth image, respectively, to obtain the depth value P_TOF of the new first depth image and the depth of the new first depth image. The accuracy value corresponding to the value A_TOF, the depth value P_BV of the new second depth image and the accuracy value A_BV corresponding to the depth value of the new second depth image; set the row and column coordinates as i and j, and use the above formula (1) to calculate Get the depth value of the target image.

Example 4: FIG. 3 is a schematic flowchart of an example of an optional method for determining the depth value of an image provided by an embodiment of the application. As shown in FIG. 3, object A is a close-range object, and object B is a long-range object. According to TOF The depth judgment of the module and the binocular vision module for the A object is based on the TOF module, and the depth of the B object in the binocular vision module is corrected from b1 to b2; in the final synthesis, the near-field object adopts the depth of the TOF module Value, the depth value of the binocular vision module is used for the distant objects.

It can be seen from Fig. 3 that in the virtual depth of field photography, the effective distance of the TOF module is limited, and the depth information of objects at a long distance or an ultra-long distance cannot be obtained. The embodiment of the application uses a binocular vision module to make up for this feature. According to the binocular vision module to determine the depth value of objects at long distances and ultra-long distances, the depth value of the final synthesized image can be obtained with higher accuracy and more depth details. Abundant depth value for taking pictures that blur the depth of field.

Through the above examples, the first depth image and the second depth image obtained by the TOF module and the binocular vision module are synthesized, not only can effectively avoid the respective shortcomings of the two technologies, but also improve the depth image of close objects Its robustness and accuracy, and can enhance each other, no matter the distance of the object, more accurate depth images can be obtained.

For example, to shoot portraits in rainy and foggy weather, you can use the TOF module and binocular vision module to cooperate. For close-up portraits, it can effectively avoid the low accuracy of the depth image acquisition caused by noise. More in-depth information.

For example, to shoot objects within the effective distance of the TOF module, first obtain an accurate low-resolution depth image through the TOF module, and then use binocular vision to expand the depth details based on the depth image to achieve a higher resolution Depth image.

The method for determining the depth value of the image provided by the embodiment of the application adopts TOF technology and binocular vision technology for joint processing. The depth value and corresponding accuracy value of the two depth images are used to obtain the obtained value by means of weighting and filtering. More accurate depth values, not only can use the respective advantages of TOF technology and binocular vision technology to obtain more high-precision depth information at medium and short distances, but also obtain depth values at long distances and ultra-long distances, and finally achieve any The depth value of the viewing distance is perceptually acquired.

The embodiment of the present application provides a method for determining the depth value of an image. For an object in a target image, a first depth image is obtained from a TOF module, and a second depth image is obtained from a binocular vision module, where the first depth image And the second depth image also includes the accuracy value corresponding to the respective depth value, the accuracy value is used to represent the accuracy of the corresponding depth value, according to the depth value of the first depth image, the depth value of the first depth image corresponds The accuracy value of the second depth image, the depth value of the second depth image, and the accuracy value corresponding to the depth value of the second depth image are weighted and calculated using a preset weighting algorithm to obtain the depth value of the target image; that is, in this application In the embodiment, the first depth image and the second depth image are obtained from the TOF module and the binocular vision module, and then according to the depth value of the first depth image and the corresponding accuracy value, the depth value of the second depth image and the corresponding Weighted calculation is performed on the accuracy value of the target image. In this way, in the process of determining the depth value of the target image, the accuracy of the depth value of the first depth image and the accuracy of the depth value of the second depth image are taken into account, so that the determined target The depth value of the image is more accurate, which is conducive to capturing a more realistic image, thereby improving the user experience.

The method for determining the depth value of the above-mentioned image will be described below with respect to each device side deployed in the image processing system.

First, the method for determining the depth value of the image is described on the image processor side.

FIG. 4 is a schematic flowchart of an optional method for determining the depth value of an image provided by an embodiment of the application. As shown in FIG. 4, the method for determining the depth value of the image may include:

S401: For the object in the target image, obtain a first depth image from the TOF module, and obtain a second depth image from the binocular vision module;

Wherein, the first depth image and the second depth image respectively further include an accuracy value corresponding to the respective depth value, and the accuracy value is used to represent the accuracy of the corresponding depth value;

S402: According to the depth value of the first depth image, the accuracy value corresponding to the depth value of the first depth image, the depth value of the second depth image and the accuracy value corresponding to the depth value of the second depth image, use preset weighting The algorithm performs weighted calculation to obtain the depth value of the target image.

In an optional embodiment, S402 may include:

Respectively performing scaling processing on the first depth image and the second depth image according to the preset resolution of the target image, to obtain the processed first depth image and the processed second depth image;

According to the depth value of the processed first depth image, the accuracy value corresponding to the depth value of the processed first depth image, the depth value of the processed second depth image and the depth value of the processed second depth image correspond The accuracy value of is calculated using a preset weighting algorithm to obtain the depth value of the target image.

In an optional embodiment, according to the depth value of the processed first depth image, the accuracy value corresponding to the depth value of the processed first depth image, the depth value of the processed second depth image, and After the accuracy value corresponding to the depth value of the processed second depth image is weighted and calculated using a preset weighting algorithm to obtain the depth value of the target image, the method may include:

The depth value of the target image is filtered to obtain the depth value of the target image after filtering.

In an optional embodiment, the first depth image and the second depth image are respectively scaled according to the preset resolution of the target image to obtain the processed first depth image and the processed second depth image Images, including:

Respectively performing scaling processing on the first depth image and the second depth image according to the preset resolution of the target image, to obtain a scaled first depth image and a scaled second depth image;

Filtering is performed on the scaled first depth image and the scaled second depth image respectively to obtain the processed first depth image and the processed second depth image.

Secondly, the method for determining the depth value of the above image is described on the TOF module side.

FIG. 5 is a schematic flowchart of another optional method for determining the depth value of an image provided by an embodiment of the application. As shown in FIG. 5, the method may further include:

S501: Obtain the depth value of the target image;

S502: According to the depth value of the target image, find the accuracy value corresponding to the depth value of the target image from the preset corresponding relationship between the depth value and the accuracy value;

S503: Use the depth value of the target image and the accuracy value corresponding to the depth value of the target image to form a first depth image;

S504: Send the first depth image to the image processor, so that the image processor determines the depth value of the target image.

Once again, the method for determining the depth value of the above-mentioned image is described on the side of the binocular vision module.

FIG. 6 is a schematic flowchart of another optional method for determining the depth value of an image provided by an embodiment of the application. As shown in FIG. 6, the method may further include:

S601: After acquiring the depth value of the target image and the binocular vision image of the target image, respectively determine the feature points of the first vision image in the binocular vision image and the feature points of the second vision image in the binocular vision image;

S602: Match the feature points of the first visual image with the feature points of the second visual image to obtain a matching result of the target image;

S603: According to the matching result of the target image, find the accuracy value corresponding to the matching result of the target image from the preset corresponding relationship between the matching result and the accuracy value;

S604: Use the depth value of the target image and the accuracy value corresponding to the matching result of the target image to form a second depth image;

S605: Send the second depth image to the image processor, so that the image processor determines the depth value of the target image.

Based on the same inventive concept, an embodiment of the application provides an image processor. FIG. 7 is a schematic structural diagram of an optional image processor provided by an embodiment of the application. As shown in FIG. 7, the image processor may include: The first acquisition module 71 and the weighting module 72; among them,

The first acquisition module 71 is configured to acquire a first depth image from the TOF module and a second depth image from the binocular vision module for the object in the target image; wherein, the first depth image and the second depth image are respectively It also includes the accuracy value corresponding to the respective depth value, and the accuracy value is used to characterize the accuracy of the corresponding depth value;

The weighting module 72 is configured to use the depth value of the first depth image, the accuracy value corresponding to the depth value of the first depth image, the depth value of the second depth image, and the accuracy value corresponding to the depth value of the second depth image, using The preset weighting algorithm performs weighting calculation to obtain the depth value of the target image.

In an optional embodiment, the weighting module 72 includes:

The scaling sub-module is configured to perform scaling processing on the first depth image and the second depth image respectively according to the preset resolution of the target image, to obtain the processed first depth image and the processed second depth image;

The weighting sub-module is configured to be based on the depth value of the processed first depth image, the accuracy value corresponding to the depth value of the processed first depth image, the depth value of the processed second depth image, and the processed second depth value The accuracy value corresponding to the depth value of the depth image is weighted and calculated using a preset weighting algorithm to obtain the depth value of the target image.

In an optional embodiment, the image processor is further configured to:

According to the depth value of the processed first depth image, the accuracy value corresponding to the depth value of the processed first depth image, the depth value of the processed second depth image, and the depth value of the processed second depth image The corresponding accuracy value is weighted and calculated using a preset weighting algorithm, and after the depth value of the target image is obtained, the depth value of the target image is filtered to obtain the depth value after the filtering process of the target image.

In an optional embodiment, the zoom sub-module is specifically configured as follows:

In practical applications, the aforementioned first acquisition module 71, weighting module 72, scaling sub-module, and weighting sub-module can be implemented by a processor located on an image processor, specifically a central processing unit (CPU, Central Processing Unit), a microprocessor (MPU, Microprocessor Unit), Digital Signal Processor (DSP, Digital Signal Processing), or Field Programmable Gate Array (FPGA, Field Programmable Gate Array) and other implementations.

Based on the same inventive concept, an embodiment of the application provides a TOF module. FIG. 8 is a schematic structural diagram of an optional TOF module provided by an embodiment of the application. As shown in FIG. 8, the TOF module may include: The second acquiring module 81, the first searching module 82, the first forming module 83 and the first sending module 84; among them,

The second obtaining module 81 is configured to obtain the depth value of the target image;

The first searching module 82 is configured to find the accuracy value corresponding to the depth value of the target image from the preset corresponding relationship between the depth value and the accuracy value according to the depth value of the target image;

The first forming module 83 is configured to use the depth value of the target image and the accuracy value corresponding to the depth value of the target image to form a first depth image;

The first sending module 84 is configured to send the first depth image to the image processor, so that the image processor determines the depth value of the target image.

In practical applications, the above-mentioned second acquisition module 81, first search module 82, first formation module 83, and first sending module 84 can be implemented by a processor located on the TOF module, specifically a CPU, MPU, DSP, or FPGA, etc. achieve.

Based on the same inventive concept, an embodiment of the present application provides a binocular vision module. FIG. 9 is a schematic structural diagram of an optional binocular vision module provided by an embodiment of the application. As shown in FIG. 9, the binocular vision module is The vision module may include: a determining module 91, a matching module 92, a second searching module 93, a second forming module 94, and a second sending module 95; among them,

The determining module 91 is configured to, after acquiring the depth value of the target image and the binocular vision image of the target image, respectively determine the feature points of the first vision image in the binocular vision image and the feature points of the second vision image in the binocular image ；

The matching module 92 is configured to match the feature points of the first visual image with the feature points of the second visual image to obtain a matching result of the target image;

The second searching module 93 is configured to find the accuracy value corresponding to the matching result of the target image from the preset correspondence between the matching result and the accuracy value according to the matching result of the target image;

The second forming module 94 is configured to use the depth value of the target image and the accuracy value corresponding to the matching result of the target image to form a second depth image;

The second sending module 95 is configured to send the second depth image to the image processor, so that the image processor determines the depth value of the target image.

In practical applications, the determination module 91, the matching module 92, the second search module 93, the second formation module 94, and the second sending module 95 can be implemented by a processor located on the binocular vision module, specifically CPU, MPU, Realized by DSP or FPGA.

FIG. 10 is a schematic structural diagram of another optional image processor provided by an embodiment of the application. As shown in FIG. 10, an embodiment of the application provides an image processor 1000, including:

A processor 101 and a storage medium 102 storing executable instructions of the processor 101. The storage medium 102 relies on the processor 101 to perform operations through the communication bus 103. When the instructions are executed by the processor 101, The method for determining the depth value of the image executed by the above-mentioned image processor is executed.

It should be noted that in actual applications, various components in the terminal are coupled together through the communication bus 103. It can be understood that the communication bus 103 is used to implement connection and communication between these components. In addition to the data bus, the communication bus 103 also includes a power bus, a control bus, and a status signal bus. However, for the sake of clear description, various buses are marked as the communication bus 103 in FIG. 10.

FIG. 11 is a schematic structural diagram of another optional TOF module provided by an embodiment of the application. As shown in FIG. 11, an embodiment of the present application provides a TOF module 1100, including:

A processor 111 and a storage medium 112 storing executable instructions of the processor 111. The storage medium 112 relies on the processor 111 to perform operations through the communication bus 113. When the instructions are executed by the processor 111, The method for determining the depth value of the image executed by the TOF module is executed.

It should be noted that in actual applications, various components in the terminal are coupled together through the communication bus 113. It can be understood that the communication bus 113 is used to implement connection and communication between these components. In addition to the data bus, the communication bus 113 also includes a power bus, a control bus, and a status signal bus. However, for clarity of description, various buses are marked as the communication bus 113 in FIG. 11.

FIG. 12 is a schematic structural diagram of another optional binocular vision module provided by an embodiment of the application. As shown in FIG. 12, an embodiment of the present application provides a binocular vision module 1200, including:

A processor 121 and a storage medium 122 storing instructions executable by the processor 121. The storage medium 122 relies on the processor 121 to perform operations through the communication bus 123. When the instructions are executed by the processor 121, The method for determining the depth value of the image executed by the binocular vision module is executed.

It should be noted that in actual applications, various components in the terminal are coupled together through the communication bus 123. It can be understood that the communication bus 123 is used to implement connection and communication between these components. In addition to the data bus, the communication bus 123 also includes a power bus, a control bus, and a status signal bus. However, for clarity of description, various buses are marked as the communication bus 123 in FIG. 12.

An embodiment of the present application provides a computer storage medium that stores executable instructions. When the executable instructions are executed by one or more processors, the processors perform the image processing in one or more embodiments above. The method for determining the depth value of the image executed by the device, or the method for determining the depth value of the image executed by the TOF module in one or more embodiments, or the method executed by the binocular vision module in one or more embodiments How to determine the depth value of the image.

It can be understood that the memory in the embodiments of the present application may be a volatile memory or a non-volatile memory, or may include both volatile and non-volatile memory. Among them, the non-volatile memory can be read-only memory (Read-Only Memory, ROM), programmable read-only memory (Programmable ROM, PROM), erasable programmable read-only memory (Erasable PROM, EPROM), and electrically available Erase programmable read-only memory (Electrically EPROM, EEPROM) or flash memory. The volatile memory may be random access memory (Random Access Memory, RAM), which is used as an external cache. By way of exemplary but not restrictive description, many forms of RAM are available, such as static random access memory (Static RAM, SRAM), dynamic random access memory (Dynamic RAM, DRAM), synchronous dynamic random access memory (Synchronous DRAM, SDRAM), Double Data Rate Synchronous Dynamic Random Access Memory (Double Data Rate SDRAM, DDRSDRAM), Enhanced Synchronous Dynamic Random Access Memory (Enhanced SDRAM, ESDRAM), Synchronous Link Dynamic Random Access Memory (Synchlink DRAM, SLDRAM) And Direct Rambus RAM (DRRAM). The memories of the systems and methods described herein are intended to include, but are not limited to, these and any other suitable types of memories.

The processor may be an integrated circuit chip with signal processing capabilities. In the implementation process, each step of the above method can be completed by an integrated logic circuit of hardware in the processor or instructions in the form of software. The above-mentioned processor may be a general-purpose processor, a digital signal processor (Digital Signal Processor, DSP), an application specific integrated circuit (ASIC), a ready-made programmable gate array (Field Programmable Gate Array, FPGA) or other Programming logic devices, discrete gates or transistor logic devices, discrete hardware components. The methods, steps, and logical block diagrams disclosed in the embodiments of the present application can be implemented or executed. The general-purpose processor may be a microprocessor or the processor may also be any conventional processor or the like. The steps of the method disclosed in the embodiments of the present application can be directly embodied as being executed and completed by a hardware decoding processor, or executed and completed by a combination of hardware and software modules in the decoding processor. The software module can be located in a mature storage medium in the field, such as random access memory, flash memory, read-only memory, programmable read-only memory, or electrically erasable programmable memory, registers. The storage medium is located in the memory, and the processor reads the information in the memory and completes the steps of the above method in combination with its hardware.

It can be understood that the embodiments described herein can be implemented by hardware, software, firmware, middleware, microcode, or a combination thereof. For hardware implementation, the processing unit can be implemented in one or more application specific integrated circuits (ASIC), digital signal processor (Digital Signal Processing, DSP), digital signal processing equipment (DSP Device, DSPD), programmable Logic device (Programmable Logic Device, PLD), Field-Programmable Gate Array (Field-Programmable Gate Array, FPGA), general-purpose processors, controllers, microcontrollers, microprocessors, and others for performing the functions described in this application Electronic unit or its combination.

For software implementation, the technology described herein can be implemented by modules (such as procedures, functions, etc.) that perform the functions described herein. The software codes can be stored in the memory and executed by the processor. The memory can be implemented in the processor or external to the processor.

It should be noted that in this article, the terms "include", "include" or any other variants thereof are intended to cover non-exclusive inclusion, so that a process, method, article or device including a series of elements not only includes those elements, It also includes other elements not explicitly listed, or elements inherent to the process, method, article, or device. If there are no more restrictions, the element defined by the sentence "including a..." does not exclude the existence of other identical elements in the process, method, article, or device that includes the element.

The serial numbers of the foregoing embodiments of the present application are only for description, and do not represent the superiority of the embodiments.

Through the description of the above implementation manners, those skilled in the art can clearly understand that the above-mentioned embodiment method can be implemented by means of software plus the necessary general hardware platform, of course, it can also be implemented by hardware, but in many cases the former is better.的实施方式。 Based on this understanding, the technical solution of this application essentially or the part that contributes to the existing technology can be embodied in the form of a software product, and the computer software product is stored in a storage medium (such as ROM/RAM, magnetic disk, The optical disc) includes several instructions to enable a terminal (which may be a mobile phone, a computer, a server, or a network device, etc.) to execute the method described in each embodiment of the present application.

The embodiments of the application are described above with reference to the accompanying drawings, but the application is not limited to the above-mentioned specific embodiments. The above-mentioned specific embodiments are only illustrative and not restrictive. Those of ordinary skill in the art are Under the enlightenment of this application, many forms can be made without departing from the purpose of this application and the scope of protection of the claims, and these are all within the protection of this application.

Industrial applicability

The embodiment of the application provides a method for determining the depth value of an image, an image processor, and a module. For the object in the target image, a first depth image is obtained from a TOF module and a second depth is obtained from a binocular vision module. Image; wherein the first depth image and the second depth image respectively include accuracy values corresponding to their respective depth values, and the accuracy values are used to characterize the accuracy of the corresponding depth values, according to the depth value of the first depth image , The accuracy value corresponding to the depth value of the first depth image, the depth value of the second depth image and the accuracy value corresponding to the depth value of the second depth image are weighted and calculated using a preset weighting algorithm to obtain the depth of the target image Value, which can improve the accuracy of the depth value of the image.

Claims

A method for determining the depth value of an image includes:

For the object in the target image, the first depth image is obtained from the TOF module and the second depth image is obtained from the binocular vision module; wherein, the first depth image and the second depth image respectively include An accuracy value corresponding to the depth value of, where the accuracy value is used to characterize the accuracy of the corresponding depth value;

According to the depth value of the first depth image, the accuracy value corresponding to the depth value of the first depth image, the depth value of the second depth image, and the accuracy value corresponding to the depth value of the second depth image , Performing weighting calculation using a preset weighting algorithm to obtain the depth value of the target image.
The method according to claim 1, wherein the accuracy value corresponding to the depth value of the first depth image, the depth value of the second depth image, and the accuracy value corresponding to the depth value of the first depth image are The accuracy value corresponding to the depth value of the second depth image is weighted and calculated using a preset weighting algorithm to obtain the depth value of the target image, including:

Respectively performing scaling processing on the first depth image and the second depth image according to the preset resolution of the target image to obtain a processed first depth image and a processed second depth image;

According to the depth value of the processed first depth image, the accuracy value corresponding to the depth value of the processed first depth image, the depth value of the processed second depth image, and the processed depth value The accuracy value corresponding to the depth value of the second depth image is weighted and calculated using a preset weighting algorithm to obtain the depth value of the target image.
3. The method according to claim 2, wherein, according to the depth value of the processed first depth image, the accuracy value corresponding to the depth value of the processed first depth image, and the processed first depth image After the depth value of the second depth image and the accuracy value corresponding to the depth value of the processed second depth image are weighted and calculated using a preset weighting algorithm, after the depth value of the target image is obtained, the method further includes :

Filter processing is performed on the depth value of the target image to obtain the depth value after the filter processing of the target image.
The method according to claim 2 or 3, wherein the first depth image and the second depth image are respectively scaled according to the preset resolution of the target image to obtain the processed The first depth image and the processed second depth image include:

Respectively performing scaling processing on the first depth image and the second depth image according to the preset resolution of the target image to obtain a scaled first depth image and a scaled second depth image;

Filtering processing is performed on the scaled first depth image and the scaled second depth image respectively to obtain the processed first depth image and the processed second depth image.
A method for determining the depth value of an image includes:

Obtain the depth value of the target image;

According to the depth value of the target image, find the accuracy value corresponding to the depth value of the target image from the preset corresponding relationship between the depth value and the accuracy value;

Forming a first depth image by using the depth value of the target image and the accuracy value corresponding to the depth value of the target image;

The first depth image is sent to an image processor, so that the image processor determines the depth value of the target image.
A method for determining the depth value of an image includes:

After acquiring the depth value of the target image and the binocular vision image of the target image, the feature points of the first vision image in the binocular vision image and the features of the second vision image in the binocular vision image are respectively determined point;

Matching the feature points of the first visual image with the feature points of the second visual image to obtain a matching result of the target image;

According to the matching result of the target image, the accuracy value corresponding to the matching result of the target image is found from the preset corresponding relationship between the matching result and the accuracy value;

Using the depth value of the target image and the accuracy value corresponding to the matching result of the target image to form a second depth image;

The second depth image is sent to an image processor, so that the image processor determines the depth value of the target image.
An image processor, including:

The first acquisition module is configured to acquire a first depth image from the TOF module and a second depth image from the binocular vision module for the object in the target image; wherein, the first depth image and the second depth The images also respectively include accuracy values corresponding to the respective depth values, and the accuracy values are used to characterize the accuracy of the corresponding depth values;

A weighting module, configured to be based on the depth value of the first depth image, the accuracy value corresponding to the depth value of the first depth image, the depth value of the second depth image, and the depth value of the second depth image The corresponding accuracy value is weighted and calculated using a preset weighting algorithm to obtain the depth value of the target image.
A TOF module, including:

The second acquiring module is configured to acquire the depth value of the target image;

The first searching module is configured to find the accuracy value corresponding to the depth value of the target image from the preset corresponding relationship between the depth value and the accuracy value according to the depth value of the target image;

A first forming module configured to use the depth value of the target image and the accuracy value corresponding to the depth value of the target image to form a first depth image;

The first sending module is configured to send the first depth image to an image processor, so that the image processor determines the depth value of the target image.
A binocular vision module, including:

The determining module is configured to, after acquiring the depth value of the target image and the binocular vision image of the target image, respectively determine the feature points of the first vision image in the binocular vision image and the second vision image in the binocular image Feature points of the visual image;

A matching module, configured to match the feature points of the first visual image with the feature points of the second visual image to obtain a matching result of the target image;

The second searching module is configured to find the accuracy value corresponding to the matching result of the target image from the preset corresponding relationship between the matching result and the accuracy value according to the matching result of the target image;

A second forming module, configured to use the depth value of the target image and the accuracy value corresponding to the matching result of the target image to form a second depth image;

The second sending module is configured to send the second depth image to an image processor, so that the image processor determines the depth value of the target image.
An image processor, wherein the image processor includes:

A processor and a storage medium storing instructions executable by the processor. The storage medium relies on the processor to perform operations through a communication bus. When the instructions are executed by the processor, the foregoing claims 1 to are executed. 4. The method for determining the depth value of an image according to any one of the items.
A TOF module, wherein the TOF module includes:

A processor and a storage medium storing instructions executable by the processor, the storage medium relies on the processor to perform operations through a communication bus, and when the instructions are executed by the processor, execute the above-mentioned claim 5 The method for determining the depth value of the image described above.
A binocular vision module, wherein the binocular vision module includes:

A processor and a storage medium storing instructions executable by the processor, the storage medium relies on the processor to perform operations through a communication bus, and when the instructions are executed by the processor, execute the above-mentioned claim 6 The method for determining the depth value of the image described above.
A computer-readable storage medium, wherein executable instructions are stored, and when the executable instructions are executed by one or more processors, the processor executes any one of claims 1 to 4 The method for determining the depth value of the image, or the method for determining the depth value of the image according to claim 5, or the method for determining the depth value of the image according to claim 6.