CN112950721A

CN112950721A - Depth information determination method and device and binocular vision system

Info

Publication number: CN112950721A
Application number: CN202110176677.0A
Authority: CN
Inventors: 吴勇辉; 刘川熙
Original assignee: Shenzhen Goodix Technology Co Ltd
Current assignee: Shenzhen Goodix Technology Co Ltd
Priority date: 2021-02-09
Filing date: 2021-02-09
Publication date: 2021-06-11

Abstract

The application provides a depth information determination method, a depth information determination device and a binocular vision system, wherein the method comprises the following steps: acquiring a first image containing a target object through a first camera, and determining a first focal length of the first camera during image acquisition according to a first parameter of a focal length adjusting element when the first camera acquires the first image; acquiring a second image containing the target object through a second camera; and determining the depth information of the target object according to the camera parameters of the first camera, the camera parameters of the second camera, the first image and the second image, wherein the camera parameters of the first camera comprise a first focal length. And in the process of determining the depth information, obtaining the depth information of the target object according to the changed focal length and the acquired image. Because the changed focal length value is the focal length value when the image is collected, the accuracy of the depth information can be ensured when the depth information of the target object is determined according to the changed focal length value.

Description

Depth information determination method and device and binocular vision system

Technical Field

The application relates to the technical field of binocular vision, in particular to a depth information determining method and device and a binocular vision system.

Background

Binocular stereo vision is taken as a branch of machine vision, and has wide application in industries such as industrial robots, navigation, automatic inspection, monitoring and the like with continuous maturation of technology in recent years. The binocular vision system usually uses two cameras with the same model and fixed focal length to acquire depth information, and because the focal length of the cameras is fixed, the obtained field of view is also fixed, and therefore if the distance of the target object to be photographed is long, the image photographed by the cameras is also blurred, and thus the depth information may not be obtained.

In the prior art, the zoom camera is introduced into a binocular vision system to solve the problem, and as the focal length of the camera is variable, target objects at different distances can be shot clearly.

However, when the zoom camera shoots target objects at different distances, the focal length of the zoom camera may change, so that the depth information obtained by the prior art may be inaccurate when the image is stereoscopically corrected according to the internal reference and the external reference calibrated by the camera.

Disclosure of Invention

The application provides a depth information determination method and device and a binocular vision system, which are used for solving the problems in the prior art.

In a first aspect, the present application provides a depth information determining method, applied to a binocular vision system, where the binocular vision system includes a first camera and a second camera, where the first camera is a zoom camera, the first camera includes a focal length adjusting element for adjusting a focal length, and the method includes:

acquiring a first image containing a target object through the first camera, and determining a first focal length of the first camera during image acquisition according to a first parameter of a focal length adjusting element when the first camera acquires the first image;

acquiring a second image containing the target object through the second camera;

and determining the depth information of the target object according to the camera parameters of the first camera, the camera parameters of the second camera, the first image and the second image, wherein the camera parameters of the first camera comprise the first focal length.

In some embodiments, the determining a first focal length of the first camera at the time of image acquisition according to the first parameter of the focal length adjustment element at the time of image acquisition of the first image by the first camera includes:

determining a focal length corresponding to the first parameter according to a pre-stored corresponding relationship between the parameter of the focal length adjusting element and the focal length of the first camera;

determining a focal length corresponding to the first parameter as the first focal length.

determining parameter variation of a focal length adjusting element according to a first parameter of the focal length adjusting element corresponding to the first camera when the first camera is used for image acquisition;

determining a focal length variable quantity corresponding to the parameter variable quantity according to a pre-stored corresponding relation between the parameter of the focal length adjusting element and the focal length of the first camera;

determining a second focal length of the first camera before image acquisition;

and obtaining the first focal length according to the second focal length and the focal length variable quantity.

In some embodiments, the determining, according to a first parameter of a focal length adjustment element corresponding to the first camera when the first camera performs image acquisition, a parameter variation of the focal length adjustment element includes:

determining a second parameter of a focal length adjusting element corresponding to the first camera before image acquisition of the first camera;

and determining the parameter variation of the focal length adjusting element according to the first parameter and the second parameter.

In some embodiments, the second focal length of the first camera before image acquisition is obtained by performing stereo calibration on the binocular vision system;

the second parameter of the focal length adjusting element is a parameter for performing stereoscopic calibration on the binocular vision system.

In some embodiments, when the second camera is a fixed-focus camera, the determining the depth information of the target object according to the camera parameters of the first camera, the camera parameters of the second camera, the first image, and the second image includes:

acquiring a first internal reference value and a first external reference value of the first camera, and a second internal reference value and a second external reference value of the second camera;

and determining the depth information of the target object according to the first focal length, other parameters except the focal length in the first internal reference value, the first external reference value, the second internal reference value, the second external reference value, the first image and the second image.

In some embodiments, the determining the depth information of the target object according to the first focal length, the other parameter except for the focal length in the first internal reference value, the first external reference value, the second internal reference value, the second external reference value, the first image and the second image includes:

performing image transformation on the first image and the second image according to the first focal length, other parameters except the focal length in the first internal reference value, the first external reference value, the second internal reference value and the second external reference value to obtain a line alignment image;

determining disparity information for matching points in the line aligned image;

and obtaining the depth information of the target object according to the visual information of the matching points.

In some embodiments, when the second camera is a zoom camera, the method further comprises:

determining a third focal length of the second camera during image acquisition according to a third parameter of a focal length adjusting element corresponding to the second camera when the second camera acquires the second image;

correspondingly, the determining the depth information of the target object according to the camera parameter of the first camera, the camera parameter of the second camera, the first image and the second image includes:

and determining the depth information of the target object according to the first focal length, the other parameters except the focal length in the first internal reference value, the first external reference value, the third focal length, the other parameters except the focal length in the second internal reference value, the second external reference value, the first image and the second image.

In some embodiments, the determining the depth information of the target object according to the first focal length, the other parameter except for the focal length in the first internal reference value, the first external reference value, the third focal length, the other parameter except for the focal length in the second internal reference value, the second external reference value, the first image, and the second image includes:

performing image transformation on the first image and the second image according to the first focal length, the other parameters except the focal length in the first internal reference value, the first external reference value, the third focal length, the other parameters except the focal length in the second internal reference value and the second external reference value to obtain a line alignment image;

In some embodiments, the obtaining a first internal reference value and a first external reference value of the first camera, and a second internal reference value and a second external reference value of the second camera includes:

performing three-dimensional calibration on the first camera and the second camera through a combined calibration plate to obtain a first internal reference value and a first external reference value of the first camera, and a second internal reference value and a second external reference value of the second camera;

wherein the combined calibration plate comprises calibration patterns of at least three different viewing angles.

In a second aspect, the present application provides a depth information determination apparatus for a binocular vision system, the binocular vision system includes a first camera and a second camera, wherein the first camera is a zoom camera, the first camera includes a focal length adjustment element for adjusting a focal length, the apparatus includes:

the image acquisition module is used for acquiring a first image containing a target object through the first camera and acquiring a second image containing the target object through the second camera;

the focal length determining module is used for determining a first focal length of the first camera during image acquisition according to a first parameter of a focal length adjusting element when the first camera acquires the first image;

and the depth determining module is used for determining the depth information of the target object according to the camera parameters of the first camera, the camera parameters of the second camera, the first image and the second image, wherein the camera parameters of the first camera comprise the first focal length.

In a third aspect, the present application provides a binocular vision system comprising: the camera comprises a first camera, a second camera and a processor, wherein the first camera is a zoom camera and comprises a focal length adjusting element for adjusting a focal length;

the first camera is used for acquiring a first image containing a target object;

the second camera is used for acquiring a second image containing a target object;

the processor is used for determining a first focal length of the first camera during image acquisition according to a first parameter of the focal length adjusting element when the first camera acquires the first image; and determining the depth information of the target object according to the camera parameters of the first camera, the camera parameters of the second camera, the first image and the second image, wherein the camera parameters of the first camera comprise the first focal length.

In some embodiments, the second camera is a zoom camera, the second camera including a focus adjustment element for adjusting a focus;

the processor is further configured to determine a third focal length of the second camera during image acquisition according to a third parameter of a focal length adjustment element corresponding to the second camera when the second camera acquires the second image; and determining the depth information of the target object according to the camera parameters of the first camera, the camera parameters of the second camera, the first image and the second image, wherein the camera parameters of the first camera comprise the first focal length, and the camera parameters of the second camera comprise the third focal length.

In a fourth aspect, the present application provides a computer device comprising: a memory, a processor and a computer program stored on the memory and executable on the processor, the processor implementing the depth information determination method described above when executing the program.

In a fifth aspect, the present application provides a computer-readable storage medium having stored therein computer-executable instructions for implementing the depth information determining method described above when executed by a processor.

In a sixth aspect, the present application provides a computer program product comprising a computer program which, when executed by a processor, implements the depth information determination method described above.

The depth information determining method, the depth information determining device and the binocular vision system provided by the application comprise the following steps: acquiring a first image containing a target object through a first camera, and determining a first focal length of the first camera during image acquisition according to a first parameter of a focal length adjusting element when the first camera acquires the first image; acquiring a second image containing the target object through a second camera; and determining the depth information of the target object according to the camera parameters of the first camera, the camera parameters of the second camera, the first image and the second image, wherein the camera parameters of the first camera comprise a first focal length. In the process of determining the depth information, firstly, the changed focal length of the zoom camera during image acquisition is determined according to the parameters of the focal length adjusting element, and then the depth information of the target object is obtained according to the changed focal length and the acquired image. Because the changed focal length value is the focal length value when the image is collected, the accuracy of the depth information can be ensured when the depth information of the target object is determined according to the changed focal length value.

Drawings

The accompanying drawings, which are incorporated in and constitute a part of this specification, illustrate embodiments consistent with the present disclosure and together with the description, serve to explain the principles of the disclosure.

Fig. 1 is a schematic diagram of an application scenario of the solution of the present application;

fig. 2 is a schematic diagram of a depth information determining method according to an embodiment of the present application;

FIG. 3 is a diagram illustrating a relationship curve between a parameter of a focal length adjustment element and a focal length of a zoom camera according to an embodiment of the present disclosure;

FIG. 4 is a diagram illustrating a relationship curve between a parameter of a focal length adjustment element and a focal length of a zoom camera according to an embodiment of the present disclosure;

FIG. 5 is an exemplary diagram of a combination calibration plate used in embodiments of the present application;

FIG. 6 is a schematic diagram illustrating a principle of determining depth information of a target object in an embodiment of the present application;

fig. 7 is a schematic diagram of a depth information determining apparatus in an embodiment of the present application;

fig. 8 is a schematic structural diagram of a computer device according to an embodiment of the present application.

With the foregoing drawings in mind, certain embodiments of the disclosure have been shown and described in more detail below. These drawings and written description are not intended to limit the scope of the disclosed concepts in any way, but rather to illustrate the concepts of the disclosure to those skilled in the art by reference to specific embodiments.

Detailed Description

In order to make the objects, technical solutions and advantages of the embodiments of the present invention clearer, the technical solutions in the embodiments of the present invention will be clearly and completely described below with reference to the drawings in the embodiments of the present invention, and it is obvious that the described embodiments are some, but not all, embodiments of the present invention. All other embodiments, which can be derived by a person skilled in the art from the embodiments given herein without making any creative effort, shall fall within the protection scope of the present invention.

The terminology used in the embodiments of the present application is for the purpose of describing particular embodiments only and is not intended to be limiting of the invention. As used in the embodiments of the present application, the singular forms "a", "an" and "the" are intended to include the plural forms as well, unless the context clearly indicates otherwise.

It is also noted that the terms "comprises," "comprising," or any other variation thereof, are intended to cover a non-exclusive inclusion, such that a good or system that comprises a list of elements does not include only those elements but may include other elements not expressly listed or inherent to such good or system. Without further limitation, an element defined by the phrase "comprising an … …" does not exclude the presence of additional like elements in the article or system in which the element is included.

In the prior art, the zoom camera is introduced into a binocular vision system to solve the problem of image blurring caused by the fact that the distance of a target object is long, and clear images can be shot for the target objects located at different distances due to the fact that the focal length of the camera is variable.

However, when the zoom camera shoots target objects at different distances, the focal length of the zoom camera may change, and if the internal parameters (including parameters such as focal length, distortion coefficient, image center, and the like) and the external parameters (including translation matrix, rotation matrix, and the like) calibrated by the zoom camera are directly used to perform stereo correction on the image, the obtained depth information may be inaccurate because the actual focal length of the zoom camera is inconsistent with the calibrated focal length.

To solve the above problem, another approach of the prior art is to calculate the parameter adjustment caused by zooming by using an algorithm in determining the depth information. For example, feature points are detected on two images shot by two cameras, the feature points are matched to obtain feature point pairs, a basic matrix is calculated according to the feature point pairs, and a stereo correction parameter is updated; or, calculating a homography matrix according to the characteristic point pairs, and then carrying out three-dimensional correction through affine transformation approximation.

However, the above algorithm based on feature point matching has problems of slow processing speed, unstable effect, and the like, so that the accuracy of depth information is low, and even the depth information acquisition fails.

The application provides a depth information determining method and device and a binocular vision system, and aims to solve the technical problems in the prior art.

The main conception of the scheme of the application is as follows: the main reason why the accuracy of the depth information is low in the prior art is that the focal lengths of the zoom cameras are different when the zoom cameras shoot target objects at different distances, and other parameters of the zoom cameras do not change, so that in the process of determining the depth information, the changed focal length of the zoom cameras during image acquisition can be determined according to the focal length adjusting elements corresponding to the zoom cameras, and then the three-dimensional correction of the images is performed according to the changed focal length and other parameters which do not change, so as to obtain the depth information of the target objects. Because the changed focal length value of the zoom camera is adopted during the stereo correction, and the changed focal length value is the focal length value of the image obtained by shooting, the precision of the stereo correction result can be ensured when the changed focal length value is adopted to carry out the stereo correction of the image, thereby ensuring the accuracy of the acquired depth information.

The following describes the technical solutions of the present application and how to solve the above technical problems with specific embodiments. The following several specific embodiments may be combined with each other, and details of the same or similar concepts or processes may not be repeated in some embodiments. Embodiments of the present application will be described below with reference to the accompanying drawings.

First, an application scenario related to the present application is explained:

fig. 1 is a schematic view of an application scenario of the solution of the present application, and as shown in fig. 1, the binocular vision system includes: the camera comprises at least one zoom camera which comprises a focal length adjusting element.

Specifically, the camera includes a first camera and a second camera, and the camera is configured to acquire an image including a target object, where the image includes at least a first image captured by the first camera and a second image captured by the second camera. It will be appreciated that the first camera and the second camera take images simultaneously, and thus the processor may determine depth information of the target object from the first image and the second image taken simultaneously.

The camera comprises at least one zoom camera, and specifically, the first camera can be a zoom camera, and the second camera is a fixed-focus camera; or the first camera can be a fixed-focus camera, and the second camera can be a zoom camera; the first camera and the second camera can be both zoom cameras.

And the focal length adjusting element is used for adjusting parameters of the focal length adjusting element to change the focal length of the zoom camera in the process of carrying out automatic focusing on the zoom camera to acquire an image containing a target object.

For example, in the binocular vision system shown in fig. 1, when the first camera is a zoom camera and the second camera is a fixed-focus camera, only the inside of the first camera is correspondingly provided with a focal length adjusting element, and the focal length adjusting element can change the focal length of the first camera by adjusting its own parameters.

Alternatively, the focus adjusting element may be, for example, a Voice Coil Motor (VCM), which is a common focusing Motor and may be used to help focus the camera for an auto-focus function.

The processor is used for determining a first focal length of the zoom camera during image acquisition according to a first parameter of a focal length adjusting element corresponding to the zoom camera during image acquisition; and determining the depth information of the target object according to the first focal length and two images respectively shot by a camera in the binocular vision system.

Specifically, for a scene in which the first camera is a zoom camera and the second camera is a fixed-focus camera, the processor is specifically configured to determine a first focal length of the first camera during image acquisition according to a first parameter of the focal length adjustment element when the first camera acquires a first image; and determining the depth information of the target object according to the camera parameters of the first camera, the camera parameters of the second camera, the first image and the second image, wherein the camera parameters of the first camera comprise a first focal length.

In addition, for a scene in which the first camera and the second camera are both zoom cameras, the processor is specifically configured to determine a first focal length of the first camera during image acquisition according to a first parameter of the focal length adjustment element when the first camera acquires a first image; determining a third focal length of the second camera during image acquisition according to a third parameter of a focal length adjusting element corresponding to the second camera when the second camera acquires a second image; and determining the depth information of the target object according to the camera parameters of the first camera, the camera parameters of the second camera, the first image and the second image, wherein the camera parameters of the first camera comprise a first focal length, and the camera parameters of the second camera comprise a third focal length.

After the camera finishes image shooting, the shot image is sent to the processor for depth calculation processing, at the moment, the processor can determine a first focal length of the first camera during image acquisition according to parameters of the focal length adjusting element, and then determine depth information of the target object according to the first focal length, other fixed parameters of the first camera, fixed parameters of the second camera and the image containing the target object.

Fig. 2 is a schematic diagram of a depth information determining method provided in the embodiment of the present application, and as shown in fig. 2, the method of the present embodiment is explained by taking the application of the method to the binocular vision system shown in fig. 1 as an example, and the method mainly includes the following steps:

s100, acquiring a first image containing a target object through a first camera, and determining a first focal length of the first camera during image acquisition according to a first parameter of a focal length adjusting element when the first camera acquires the first image;

s200, acquiring a second image containing a target object through a second camera;

s300, determining depth information of the target object according to the camera parameters of the first camera, the camera parameters of the second camera, the first image and the second image, wherein the camera parameters of the first camera comprise a first focal length.

Specifically, a scenario in which the first camera is a zoom camera and the second camera is a fixed-focus camera is taken as an example to explain the scheme of the present application.

When a target object is shot through the first camera and the second camera, the focal length of the first camera changes when a first image is shot due to the fact that the first camera is a zoom camera; and because the second camera is the fixed-focus camera, the focal length of the second camera is kept unchanged when the second camera shoots the second image. Therefore, in the process of capturing an image, the present embodiment further obtains a first parameter of the focal length adjustment element when the zoom camera (i.e., the first camera) performs image capturing, so as to determine a first focal length of the first camera during image capturing according to the first parameter, where the first focal length is a focal length after the first camera changes.

It can be understood that, for a zoom camera, during image capturing, the change of the camera parameters is mainly the change of the focal length, and other parameters can be regarded as fixed.

After the first focal length of the first camera during image acquisition is obtained, the camera parameters of the first camera including the first focal length and the camera parameters of the second camera are combined, so that the first image and the second image including the target object can be subjected to stereo correction, and the depth information of the target object is determined.

In the process of determining the depth information, firstly, a changed focal length of the zoom camera during image acquisition is determined according to parameters of a focal length adjusting element, and then the depth information of a target object is obtained according to the changed focal length and an acquired image. Because the changed focal length value is the focal length value when the image is collected, the accuracy of the depth information can be ensured when the depth information of the target object is determined according to the changed focal length value.

In some embodiments, determining a first focal length of the zoom camera during image capturing according to a first parameter of a focal length adjustment element corresponding to the zoom camera during image capturing includes:

s211, determining a focal length corresponding to the first parameter according to a pre-stored corresponding relation between the parameter of the focal length adjusting element and the focal length of the zoom camera;

and S212, determining the focal length corresponding to the first parameter as a first focal length.

The first parameter of the focal length adjusting element corresponding to the zoom camera during image acquisition can be directly obtained when the first focal length of the zoom camera during image acquisition is determined, and the first parameter is a parameter corresponding to the changed focal length of the zoom camera, so that the first focal length of the zoom camera during image acquisition can be directly determined according to the first parameter.

Specifically, the corresponding relationship between the parameters of the focal length adjustment element and the focal length of the zoom camera can be embodied in the form of a relationship curve in a coordinate system.

Fig. 3 is a schematic diagram of a relationship curve between parameters of a focal length adjusting element and a focal length of a zoom camera in an embodiment of the present application, where an abscissa is a parameter P of the focal length adjusting element, and an ordinate is a focal length f of the zoom camera, and after a first parameter of the focal length adjusting element during image acquisition is obtained, a first focal length of the zoom camera during image acquisition can be directly determined according to the relationship curve shown in fig. 3.

For example, if the value of the first parameter of the focal length adjustment element corresponding to the zoom camera at the time of image capture is P1, and the focal length corresponding to P1 is f1 according to the relationship curve shown in fig. 3, it can be determined that the first focal length of the zoom camera at the time of image capture is f 1. Similarly, when the value of the first parameter is P2, the corresponding first focal length is f 2.

It can be understood that the corresponding relationship between the parameter of the focal length adjustment element and the focal length of the zoom camera may also be embodied in other manners, such as a table, and the application does not limit the specific form of the corresponding relationship between the parameter and the focal length.

For example, the correspondence of the parameters of the focus adjustment element to the focal length of the zoom camera can be represented by the following table 1:

parameter(s)	Focal length
		P1	f1
P2	f2
		P3	f3
…	…
		Pn	fn

TABLE 1

Therefore, according to the first parameter and the corresponding relation between the parameter of the focal length adjusting element and the focal length of the zoom camera, the focal length corresponding to the first parameter can be quickly determined to be the first focal length of the zoom camera during image acquisition.

s221, determining parameter variation of a focal length adjusting element according to a first parameter of the focal length adjusting element corresponding to the zoom camera when the zoom camera performs image acquisition;

s222, determining a focal length variable quantity corresponding to the parameter variable quantity according to a pre-stored corresponding relation between the parameter of the focal length adjusting element and the focal length of the zoom camera;

s223, determining a second focal length of the zoom camera before image acquisition;

and S224, obtaining the first focal length according to the second focal length and the focal length variable quantity.

Optionally, when the zoom camera performs image acquisition, determining a parameter variation of the focal length adjusting element according to a first parameter of the focal length adjusting element corresponding to the zoom camera, including:

S221A, determining a second parameter of a focal length adjusting element corresponding to the zoom camera before image acquisition of the zoom camera;

and S221B, determining the parameter variation of the focal length adjusting element according to the first parameter and the second parameter.

Optionally, the second focal length of the zoom camera before image acquisition is obtained by performing stereo calibration on the binocular vision system; the second parameter of the focal length adjusting element is a parameter when the binocular vision system is subjected to stereo calibration.

Specifically, the first focal length of the zoom camera may be indirectly determined by combining a parameter variation of the focal length adjustment element during image acquisition, that is, before image acquisition by the zoom camera, the second parameter of the focal length adjustment element is obtained; after image acquisition is carried out through the zoom camera, a first parameter of the focal length adjusting element is obtained, then a parameter variation of the focal length adjusting element during image acquisition is obtained through calculation of the first parameter and the second parameter, and finally a first focal length of the zoom camera during image acquisition is determined according to the parameter variation.

In practical applications, due to uncertain factors such as an error of a camera manufacturing process or improper use of the camera, an actual corresponding relationship between a parameter of the focal length adjusting element and a focal length of the zoom camera may deviate from a preset corresponding relationship.

Specifically, the following explanation is given by taking an example of a correspondence relationship between a parameter of the focal length adjustment element and the focal length of the zoom camera, which is represented by a relationship curve in a coordinate system:

fig. 4 is a schematic diagram of a relationship curve between a parameter of a focal length adjusting element and a focal length of a zoom camera in an embodiment of the present application, where an abscissa is a parameter P of the focal length adjusting element, an ordinate is a focal length f of the zoom camera, a solid line indicates an actual correspondence (the actual correspondence is unknown) between the parameter of the focal length adjusting element and the focal length of the zoom camera, and a dashed solid line indicates a preset correspondence (the preset correspondence is unknown) between the parameter of the focal length adjusting element and the focal length of the zoom camera, and the actual correspondence and the preset correspondence are deviated due to an influence of an uncertain factor, and at this time, if a first focal length is directly determined according to a first parameter, a problem that a result of the first focal length is inaccurate may occur.

For example, if the first parameter is P3, if the first focal length is determined according to the preset corresponding relationship between the parameter and the focal length, the first focal length corresponding to P3 is f3, however, since the actual corresponding relationship deviates from the preset corresponding relationship, the true first focal length corresponding to P3 is f 3', which results in inaccurate first focal length result.

Based on this, the present embodiment provides another method for determining a first focal length, so as to solve the problem that the determination result of the first focal length is inaccurate when the actual corresponding relationship between the parameter and the focal length is different from the preset corresponding relationship.

Specifically, referring to fig. 4, in fig. 4, f4 indicates a second focal length of the zoom camera before image capture is performed, the second focal length is obtained by performing stereo calibration on the binocular vision system, P4 indicates a second parameter of the focal length adjusting element before image capture, the second parameter is a parameter of the focal length adjusting element when the binocular vision system is subjected to stereo calibration, and during the process of performing stereo calibration, a relationship between the parameter of the focal length adjusting element and the focal length of the zoom camera is consistent with an actual corresponding relationship, so that P4 and f4 satisfy a relationship curve indicated by a solid line in fig. 4.

After obtaining the first parameter P3, the parameter change dP of the focus adjustment element during image acquisition may be first calculated, i.e., dP — P3-P4.

After the parameter variation dP is obtained, the focal length variation df corresponding to the parameter variation dP can be obtained according to the preset corresponding relationship between the parameter of the focal length adjustment element and the focal length of the zoom camera, i.e., the relationship curve in the form of a dotted line in fig. 4. Specifically, according to the preset correspondence relationship, the focal length of the point B corresponding to P3 and the focal length of the point a corresponding to P4 can be obtained, so that the focal length variation df can be obtained according to the focal lengths of the point a and the point B.

After the focal length variation df is obtained, the second focal length f4 is combined to obtain the first focal length f3 of the zoom camera during image capturing, that is, f3 is f4+ df. Thus, the first focal length obtained is the same as the real first focal length.

In this embodiment, when the first focal length is determined, the corresponding focal length variation is determined according to the parameter variation, and then the first focal length of the zoom camera during image acquisition is obtained by combining the second focal length of the zoom camera before image acquisition, so that even when the actual corresponding relationship between the parameter and the focal length has a deviation from the preset corresponding relationship, the accuracy of the determination result of the first focal length can be ensured.

In some embodiments, before acquiring the image including the target object by the camera of the binocular vision system, the method further includes: performing three-dimensional calibration on a camera of the binocular vision system through a combined calibration plate to obtain an internal reference calibration value and an external reference calibration value of the camera of the binocular vision system; the combined calibration plate comprises at least three calibration patterns with different viewing angles.

Specifically, in the process of calculating the depth information of the target object through the binocular vision system, because the internal reference and the external reference of the camera of the binocular vision system need to be used, before image acquisition is performed through the camera, the method further comprises the step of performing three-dimensional calibration on the camera of the binocular vision system to obtain the internal reference calibration value and the external reference calibration value of the camera.

In the prior art, when a fixed-focus camera is subjected to three-dimensional calibration, at least 3 calibration patterns with different viewing angles need to be collected respectively so as to accurately determine internal and external parameters of the camera. And because in this application, binocular vision system includes at least one camera that zooms, if use this camera that zooms to gather the calibration pattern of 3 different visual angles at least respectively, because the calibration pattern of gathering different visual angles can lead to the focus of camera that zooms to change, at the in-process of the calibration pattern of shooting 3 different visual angles, the focus of camera that zooms can't keep the same, consequently, can lead to the three-dimensional calibration failure of camera that zooms, can't accurately obtain camera's internal and external reference promptly.

Based on this, this application is through making up the calibration pattern to different visual angles to obtain and be applicable to carrying out the combination calibration board that three-dimensional was markd to zooming camera.

Fig. 5 is an exemplary diagram of a combined calibration board used in the embodiment of the present application, and as shown in fig. 5, the combined calibration board includes 4 calibration patterns with different viewing angles, and an image including the 4 calibration patterns with different viewing angles is obtained by one-time shooting, so that it can be ensured that the focal length of each calibration pattern shot by the zoom camera is the same, and thus, the internal reference value and the external reference value of the zoom camera can be accurately obtained according to the shot image.

It can be understood that the combination calibration board including the calibration patterns with 4 different viewing angles is only an exemplary illustration that the combination calibration board includes the calibration patterns with at least three different viewing angles, and the combination calibration board may also include other numbers of calibration patterns with different viewing angles as long as the other numbers are greater than or equal to 3, so as to ensure that the internal and external parameters of the camera can be accurately determined.

In this embodiment, when carrying out the three-dimensional calibration to binocular vision system's camera, the combination calibration board that uses includes the calibration pattern of at least three different visual angles, thereby, the camera of zooming can obtain the image including the calibration pattern of at least three different visual angles through once taking, to the calibration pattern of every different visual angles, the focus homogeneous phase of the camera of zooming is the same, thereby, can accurately obtain the interior reference calibration value and the outer reference calibration value of camera of zooming according to this image, and then can accurately obtain the interior reference calibration value and the outer reference calibration value of binocular vision system's camera.

In some embodiments, for a scene in which the first camera is a zoom camera and the second camera is a fixed-focus camera, determining depth information of the target object according to the camera parameters of the first camera, the camera parameters of the second camera, the first image, and the second image includes:

s310, acquiring a first internal reference value and a first external reference value of a first camera, and a second internal reference value and a second external reference value of a second camera;

s320, determining the depth information of the target object according to the first focal length, other parameters except the focal length in the first internal reference calibration value, the first external reference calibration value, the second internal reference calibration value, the second external reference calibration value, the first image and the second image.

Specifically, after a first focal length corresponding to the zoom camera is determined, according to the first focal length, by combining other parameters except for the focal length and an external reference calibration value in an internal reference calibration value of the zoom camera, and the internal reference calibration value and the external reference calibration value of the fixed-focus camera, the three-dimensional correction of a first image and a second image containing a target object can be achieved, and therefore the depth information of the target object is determined.

Therefore, when the stereo correction is carried out, the focal length value of the zoom camera after being changed is adopted, and the focal length value after being changed is the focal length value of the image obtained by shooting, so that the precision of the stereo correction result can be ensured, and the accuracy of the acquired depth information is ensured.

In some embodiments, determining the depth information of the target object according to the first focal length, the other parameter except the focal length in the first internal reference value, the first external reference value, the second internal reference value, the second external reference value, the first image and the second image includes:

s321, performing image transformation on the first image and the second image according to the first focal length, other parameters except the focal length in the first internal reference calibration value, the first external reference calibration value, the second internal reference calibration value and the second external reference calibration value to obtain a line alignment image;

s322, determining parallax information of matching points in the line alignment image;

and S323, obtaining the depth information of the target object according to the visual information of the matching point.

Specifically, when the depth information of the target object is determined, the image is first subjected to stereo correction, that is, the first image and the second image including the target object are subjected to image transformation according to the first focal length, other parameters except the focal length in the first internal reference value, the first external reference value, the second internal reference value, and the second external reference value, so that the first image and the second image are transformed into the line-aligned image. The processing procedure can be implemented by the existing image transformation method, and is not described herein again.

The first image and the second image can be converted into line alignment images by performing stereo correction, which is equivalent to using one virtual camera for imaging, the focal distance of the virtual camera is fv, that is, the focal distance corresponding to the line alignment images is fv, and the values of fv are different, the line alignment images obtained by stereo correction are also different (but can both generate the line alignment images), and the focal distance fv is used for calculating depth information.

The value of fv can be set manually, and specifically can be an average value of focal lengths of cameras of a binocular vision system or other preset values. For example, when the camera of the binocular vision system includes a zoom camera and a fixed-focus camera, the value of fv is an average value of a first focal length corresponding to the zoom camera and a fixed focal length of the fixed-focus camera; when the camera of the binocular vision system comprises two zoom cameras, the value of fv is the mean value of the changed focal length corresponding to the first zoom camera and the changed focal length corresponding to the second zoom camera.

After the line alignment image is obtained, for each pixel on the first image, a pixel point which is most matched with the pixel is searched in a certain parallax range of the same line on the second image, a matching point pair is obtained, and parallax information of each matching point pair is calculated.

After obtaining the disparity information of each matching point pair, the depth information of the target object can be determined according to the disparity information.

Fig. 6 is a schematic diagram illustrating the principle of determining depth information of a target object in the embodiment of the present application, as shown in fig. 6, M is a point representing the target object, Z is a depth of the point M, ML is a left imaging point on an image captured by a left camera, MR is a right imaging point on an image captured by a right camera, OL is an aperture center of the left camera, OR is an aperture center of the right camera, b is a baseline, i.e., a distance between aperture centers of the two cameras, UL is a distance between a left optical center and the left imaging point ML, UR is a distance between a right optical center and the right imaging point MR, and fv is a focal length corresponding to a line-aligned image obtained by stereo correction.

Referring to FIG. 6, the triangle M-ML-MR and the triangle M-OL-OR are similar triangle relationships, and according to the characteristics of the similar triangles, it can be found that:

when d represents parallax, there are: combining the above formula, one can obtain:

accordingly, the depth information of the target object can be obtained through calculation according to the formula.

In some embodiments, for a scene in which both the first camera and the second camera are zoom cameras, the method further comprises: and determining a third focal length of the second camera during image acquisition according to a third parameter of the focal length adjusting element corresponding to the second camera when the second camera acquires the second image.

Specifically, when the second camera is also a zoom camera, the third focal length of the second camera during image capturing is also determined according to the third parameter of the focal length adjustment element of the second camera during image capturing, that is, the focal length after the change of the second camera is determined, so as to facilitate the calculation of the depth information.

It can be understood that the principle of determining the third focal length of the second camera in this embodiment is the same as the principle of determining the first focal length of the first camera in the foregoing embodiments, and details are not repeated here.

Correspondingly, according to the camera parameter of the first camera, the camera parameter of the second camera, the first image and the second image, determining the depth information of the target object, including:

s350, acquiring a first internal reference value and a first external reference value of the first camera, and a second internal reference value and a second external reference value of the second camera;

s360, determining depth information of the target object according to the first focal length, other parameters except focal length in the first internal reference calibration value, the first external reference calibration value, the third focal length, other parameters except focal length in the second internal reference calibration value, the second external reference calibration value, the first image and the second image.

Optionally, determining the depth information of the target object according to the first focal length, the other parameters except the focal length in the first internal reference calibration value, the first external reference calibration value, the third focal length, the other parameters except the focal length in the second internal reference calibration value, the second external reference calibration value, the first image, and the second image, includes:

s361, performing image transformation on the first image and the second image according to the first focal length, other parameters except the focal length in the first internal reference calibration value, the first external reference calibration value, the third focal length, other parameters except the focal length in the second internal reference calibration value and the second external reference calibration value to obtain a line alignment image;

s362, determining parallax information of matching points in the line alignment image;

and S363, obtaining the depth information of the target object according to the observation information of the matching point.

The specific processing flow of each step can refer to the content in the foregoing embodiments, and is not described herein again.

Therefore, for the scene that the first camera and the second camera are zooming cameras, when the stereo correction is carried out, the focal length values of the two zooming cameras after being changed are adopted, and the focal length values after being changed are the focal length values of images obtained through shooting, so that the precision of the stereo correction result can be ensured, and the accuracy of the obtained depth information is ensured.

It should be understood that, although the respective steps in the flowcharts in the above-described embodiments are sequentially shown as indicated by arrows, the steps are not necessarily performed sequentially as indicated by the arrows. The steps are not performed in the exact order shown and may be performed in other orders unless explicitly stated herein. Moreover, at least some of the steps in the figures may include multiple sub-steps or multiple stages that are not necessarily performed at the same time, but may be performed at different times, in different orders, and may be performed alternately or at least partially with respect to other steps or sub-steps of other steps.

In some embodiments, there is provided a depth information determining apparatus, applied to a binocular vision system, the binocular vision system including a first camera and a second camera, wherein the first camera is a zoom camera, the first camera includes a focal length adjusting element for adjusting a focal length, fig. 7 is a schematic diagram of the depth information determining apparatus in the embodiment of the present application, and as shown in fig. 7, the apparatus includes:

an image acquisition module 100, configured to acquire a first image including a target object through a first camera, and acquire a second image including the target object through a second camera;

the focal length determining module 200 is configured to determine a first focal length of the first camera during image acquisition according to a first parameter of the focal length adjusting element when the first camera acquires a first image;

the depth determining module 300 is configured to determine depth information of the target object according to the camera parameter of the first camera, the camera parameter of the second camera, the first image, and the second image, where the camera parameter of the first camera includes a first focal length.

For specific limitations of the depth information determining apparatus, reference may be made to the above limitations of the depth information determining method, which are not described herein again. The various modules in the depth information determining apparatus described above may be implemented in whole or in part by software, hardware, and combinations thereof. The modules can be embedded in a hardware form or independent from a processor in the computer device, and can also be stored in a memory in the computer device in a software form, so that the processor can call and execute operations corresponding to the modules.

The application provides a depth information determining device, and in the process of determining depth information, firstly, the changed focal length of a zoom camera during image acquisition is determined according to parameters of a focal length adjusting element, and then the depth information of a target object is obtained according to the changed focal length and an acquired image. Because the changed focal length value is the focal length value when the image is collected, the accuracy of the depth information can be ensured when the depth information of the target object is determined according to the changed focal length value.

In some embodiments, the focal length determination module 200 is further configured to: determining a focal length corresponding to the first parameter according to a pre-stored corresponding relation between the parameter of the focal length adjusting element and the focal length of the first camera; and determining the focal length corresponding to the first parameter as the first focal length.

In some embodiments, the focal length determination module 200 is further configured to: determining parameter variation of a focal length adjusting element according to a first parameter of the focal length adjusting element corresponding to a first camera when the first camera is used for image acquisition; determining a focal length variable quantity corresponding to the parameter variable quantity according to a pre-stored corresponding relation between the parameter of the focal length adjusting element and the focal length of the first camera; determining a second focal length of the first camera before image acquisition; and obtaining the first focal length according to the second focal length and the focal length variable quantity.

In some embodiments, the focal length determination module 200 is further configured to: determining a second parameter of a focal length adjusting element corresponding to the first camera before image acquisition of the first camera; and determining the parameter variation of the focal length adjusting element according to the first parameter and the second parameter.

In some embodiments, the second focal length of the first camera before image acquisition is obtained by performing stereo calibration on the binocular vision system; the second parameter of the focal length adjusting element is a parameter when the binocular vision system is subjected to stereo calibration.

In some embodiments, the depth information determining apparatus further comprises: the three-dimensional calibration module is used for carrying out three-dimensional calibration on the first camera and the second camera through the combined calibration plate to obtain a first internal reference calibration value and a first external reference calibration value of the first camera, and a second internal reference calibration value and a second external reference calibration value of the second camera; the combined calibration plate comprises at least three calibration patterns with different viewing angles.

In some embodiments, the depth determination module 300 is further configured to: acquiring a first internal reference value and a first external reference value of a first camera, and a second internal reference value and a second external reference value of a second camera; and determining the depth information of the target object according to the first focal length, other parameters except the focal length in the first internal reference calibration value, the first external reference calibration value, the second internal reference calibration value, the second external reference calibration value, the first image and the second image.

In some embodiments, the depth determination module 300 is further configured to: performing image transformation on the first image and the second image according to the first focal length, other parameters except the focal length in the first internal reference calibration value, the first external reference calibration value, the second internal reference calibration value and the second external reference calibration value to obtain a line alignment image; determining parallax information of matching points in the line alignment image; and obtaining the depth information of the target object according to the visual information of the matching points.

In some embodiments, the focal length determination module 200 is further configured to: and determining a third focal length of the second camera during image acquisition according to a third parameter of the focal length adjusting element corresponding to the second camera when the second camera acquires the second image.

In some embodiments, the depth determination module 300 is further configured to: acquiring a first internal reference value and a first external reference value of a first camera, and a second internal reference value and a second external reference value of a second camera; and determining the depth information of the target object according to the first focal length, other parameters except the focal length in the first internal reference calibration value, the first external reference calibration value, the third focal length, other parameters except the focal length in the second internal reference calibration value, the second external reference calibration value, the first image and the second image.

In some embodiments, the depth determination module 300 is further configured to: performing image transformation on the first image and the second image according to the first focal length, other parameters except the focal length in the first internal reference calibration value, the first external reference calibration value, the third focal length, other parameters except the focal length in the second internal reference calibration value and the second external reference calibration value to obtain a line alignment image; determining parallax information of matching points in the line alignment image; and obtaining the depth information of the target object according to the visual information of the matching points.

In some embodiments, there is provided a computer device comprising: the depth information determining method of the embodiments is implemented by a memory, a processor and a computer program stored in the memory and capable of running on the processor.

Fig. 8 is a schematic structural diagram of a computer device according to an embodiment of the present application, and as shown in fig. 8, the computer device includes: a processor 10 and a memory 20.

The memory 20 is used for storing programs and data, and the processor 10 calls the programs stored in the memory 10 to execute the technical solution of any one of the foregoing method embodiments.

In the above computer devices, the memory and the processor are electrically connected directly or indirectly to enable data transmission or interaction. For example, the components may be electrically connected to each other via one or more communication buses or signal lines, such as a bus. The memory stores computer-executable instructions for implementing the data access control method, and includes at least one software functional module which can be stored in the memory in the form of software or firmware, and the processor executes various functional applications and data processing by running the software programs and modules stored in the memory.

The Memory may be, but is not limited to, a Random Access Memory (RAM), a Read Only Memory (ROM), a Programmable Read Only Memory (PROM), an Erasable Read Only Memory (EPROM), an electrically Erasable Read Only Memory (EEPROM), and the like. The memory is used for storing programs, and the processor executes the programs after receiving the execution instructions. Further, the software programs and modules within the aforementioned memories may also include an operating system, which may include various software components and/or drivers for managing system tasks (e.g., memory management, storage device control, power management, etc.), and may communicate with various hardware or software components to provide an operating environment for other software components.

The processor may be an integrated circuit chip having signal processing capabilities. The Processor may be a general-purpose Processor including a Central Processing Unit (CPU), a Network Processor (NP), and the like. The various methods, steps, and logic blocks disclosed in the embodiments of the present application may be implemented or performed. A general purpose processor may be a microprocessor or the processor may be any conventional processor or the like.

In some embodiments, a computer-readable storage medium is provided, in which computer-executable instructions are stored, and when executed by a processor, the computer-executable instructions are used for implementing the depth information determination method of the above embodiments.

In some embodiments, a computer program product is provided, comprising a computer program which, when executed by a processor, implements the depth information determination method of the above embodiments.

It will be understood by those skilled in the art that all or part of the processes of the methods of the embodiments described above can be implemented by hardware related to instructions of a computer program, which can be stored in a non-volatile computer-readable storage medium, and when executed, the computer program can include the processes of the embodiments of the methods described above. Any reference to memory, storage, database, or other medium used in the embodiments provided herein may include non-volatile and/or volatile memory, among others. Non-volatile memory can include read-only memory (ROM), Programmable ROM (PROM), Electrically Programmable ROM (EPROM), Electrically Erasable Programmable ROM (EEPROM), or flash memory. Volatile memory can include Random Access Memory (RAM) or external cache memory. By way of illustration and not limitation, RAM is available in a variety of forms such as Static RAM (SRAM), Dynamic RAM (DRAM), Synchronous DRAM (SDRAM), Double Data Rate SDRAM (DDRSDRAM), Enhanced SDRAM (ESDRAM), Synchronous Link DRAM (SLDRAM), Rambus Direct RAM (RDRAM), direct bus dynamic RAM (DRDRAM), and memory bus dynamic RAM (RDRAM).

Other embodiments of the present disclosure will be apparent to those skilled in the art from consideration of the specification and practice of the application disclosed herein. This application is intended to cover any variations, uses, or adaptations of the disclosure following, in general, the principles of the disclosure and including such departures from the present disclosure as come within known or customary practice within the art to which the disclosure pertains. It is intended that the specification and examples be considered as exemplary only, with a true scope and spirit of the disclosure being indicated by the following claims.

It will be understood that the present disclosure is not limited to the precise arrangements described above and shown in the drawings and that various modifications and changes may be made without departing from the scope thereof. The scope of the present disclosure is limited only by the appended claims.

Claims

1. The depth information determination method is applied to a binocular vision system, the binocular vision system comprises a first camera and a second camera, the first camera is a zoom camera, the first camera comprises a focal length adjusting element for adjusting a focal length, and the method comprises the following steps:

2. The method of claim 1, wherein determining the first focal length of the first camera during image acquisition according to the first parameter of the focal length adjustment element during image acquisition of the first image by the first camera comprises:

3. The method of claim 1, wherein determining the first focal length of the first camera during image acquisition according to the first parameter of the focal length adjustment element during image acquisition of the first image by the first camera comprises:

determining a second focal length of the first camera before image acquisition;

4. The method according to claim 3, wherein the determining a parameter variation of the focal length adjustment element according to a first parameter of the focal length adjustment element corresponding to the first camera when the first camera performs image acquisition comprises:

5. The method of claim 4, wherein the second focal length of the first camera prior to image acquisition is obtained by stereo calibrating the binocular vision system;

6. The method according to claim 1, wherein when the second camera is a fixed-focus camera, the determining the depth information of the target object according to the camera parameters of the first camera, the camera parameters of the second camera, the first image and the second image comprises:

7. The method of claim 6, wherein determining the depth information of the target object according to the first focal length, the other parameter of the first internal reference value except for the focal length, the first external reference value, the second internal reference value, the second external reference value, the first image, and the second image comprises:

8. The method of claim 1, wherein when the second camera is a zoom camera, the method further comprises:

9. The method of claim 8, wherein determining the depth information of the target object from the first focal length, the other of the first internal reference values except focal length, the first external reference value, the third focal length, the other of the second internal reference values except focal length, the second external reference value, the first image, and the second image comprises:

10. The method of claim 6 or 8, wherein the obtaining a first internal reference value and a first external reference value of the first camera and a second internal reference value and a second external reference value of the second camera comprises:

11. The utility model provides a degree of depth information confirming device, is applied to binocular vision system, binocular vision system includes first camera and second camera, wherein, first camera is the camera that zooms, first camera is including the focus adjusting element who is used for focus adjusting, its characterized in that, the device includes:

12. A binocular vision system, comprising: the camera comprises a first camera, a second camera and a processor, wherein the first camera is a zoom camera and comprises a focal length adjusting element for adjusting a focal length;

13. The binocular vision system of claim 12, wherein the second camera is a zoom camera, the second camera including a focus adjustment element for adjusting a focus;