CN115937293A - Binocular depth perception method and device, electronic equipment and storage medium - Google Patents
Binocular depth perception method and device, electronic equipment and storage medium Download PDFInfo
- Publication number
- CN115937293A CN115937293A CN202211700282.7A CN202211700282A CN115937293A CN 115937293 A CN115937293 A CN 115937293A CN 202211700282 A CN202211700282 A CN 202211700282A CN 115937293 A CN115937293 A CN 115937293A
- Authority
- CN
- China
- Prior art keywords
- image
- longitude
- map
- camera
- target
- Prior art date
- Legal status (The legal status is an assumption and is not a legal conclusion. Google has not performed a legal analysis and makes no representation as to the accuracy of the status listed.)
- Pending
Links
Images
Landscapes
- Image Processing (AREA)
Abstract
The application discloses a binocular depth perception method and device, electronic equipment and a storage medium, and relates to the technical field of image processing. The method comprises the following steps: the method comprises the steps of obtaining a first image collected by a first camera, obtaining a second image collected by a second camera, conducting spherical projection and expansion on the first image to obtain a first longitude and latitude map, conducting spherical projection and expansion on the second image to obtain a second longitude and latitude map, obtaining a longitude parallax map according to the first longitude and latitude map and the second longitude and latitude map, converting the longitude parallax map into a depth map, and obtaining the depth of a common view area based on the depth map. The image that this application was gathered through the camera that will have the common vision region projects to the sphere and expands, and then the degree of depth of perception this common vision region has improved the precision of perception environmental depth.
Description
Technical Field
The present application relates to the field of image processing technologies, and in particular, to a binocular depth perception method and apparatus, an electronic device, and a storage medium.
Background
With the development of science and technology, the use of aircraft is more and more extensive, and the function is more and more. For example, the existence of aircraft with autopilot capability has emerged, during which the aircraft's precise perception of the depth of the environment is of critical importance. Therefore, in the related art, in developing an aircraft, there is a challenge to accurately perceive the depth of the environment.
Disclosure of Invention
In view of the above problems, the present application provides a binocular depth perception method, an apparatus, an electronic device, and a storage medium, which may be used to perceive the depth of a common viewing area by projecting an image collected by a camera having the common viewing area onto a spherical surface and then expanding the image, thereby improving the accuracy of perceiving the depth of an environment.
In a first aspect, an embodiment of the present application provides a binocular depth perception method, which is applied to an aircraft, where the aircraft includes a first camera and a second camera, where there is a common view area between the first camera and the second camera, and the method includes: acquiring an image of the common-view area acquired by the first camera as a first image, and acquiring an image of the common-view area acquired by the second camera as a second image; performing spherical projection and expansion on the first image to obtain a first longitude and latitude map, and performing spherical projection and expansion on the second image to obtain a second longitude and latitude map; obtaining a longitude disparity map according to the first longitude and latitude map and the second longitude and latitude map; and converting the longitude disparity map into a depth map, and obtaining the depth of the common view area based on the depth map.
In a second aspect, an embodiment of the present application provides a binocular depth perception device, which is applied to an aircraft, the aircraft includes a first camera and a second camera, wherein the first camera and the second camera have a common viewing area, and the device includes: the device comprises a camera image acquisition module, a longitude and latitude map acquisition module, a longitude and latitude disparity map acquisition module and a depth perception module. The camera image acquisition module is used for acquiring the image of the common-view area acquired by the first camera as a first image and acquiring the image of the common-view area acquired by the second camera as a second image; the longitude and latitude map obtaining module is used for performing spherical projection and expansion on the first image to obtain a first longitude and latitude map and performing spherical projection and expansion on the second image to obtain a second longitude and latitude map; a longitude disparity map obtaining module for obtaining a longitude disparity map according to the first longitude disparity map and the second longitude disparity map; and the depth perception module is used for converting the longitude disparity map into a depth map and obtaining the depth of the common view area based on the depth map.
In a third aspect, an embodiment of the present application provides an electronic device, which includes a memory and a processor, where the memory is coupled to the processor, and the memory stores instructions, and when the instructions are executed by the processor, the processor executes the method described above.
In a fourth aspect, the present application provides a computer-readable storage medium, in which a program code is stored, and the program code can be called by a processor to execute the above method.
According to the binocular depth perception method, the binocular depth perception device, the electronic equipment and the storage medium, the image of the common-view area acquired by the first camera is acquired as the first image, and the image of the common-view area acquired by the second camera is acquired as the second image; performing spherical projection and expansion on the first image to obtain a first longitude and latitude map, and performing spherical projection and expansion on the second image to obtain a second longitude and latitude map; obtaining a longitude disparity map according to the first longitude and latitude map and the second longitude and latitude map; the longitude disparity map is converted into a depth map, the depth of the common-view area is obtained based on the depth map, and then the image collected by the camera with the common-view area is projected to a spherical surface and then unfolded, so that the depth of the common-view area is sensed, and the accuracy of sensing the depth of the environment is improved.
Drawings
In order to more clearly illustrate the technical solutions in the embodiments of the present application, the drawings needed to be used in the description of the embodiments are briefly introduced below, and it is obvious that the drawings in the following description are only some embodiments of the present application, and it is obvious for those skilled in the art to obtain other drawings based on these drawings without creative efforts.
Fig. 1 is a schematic flowchart illustrating a binocular depth perception method according to an embodiment of the present application;
fig. 2 is a schematic flowchart illustrating a binocular depth perception method according to an embodiment of the present application;
fig. 3 is a schematic flowchart illustrating a binocular depth perception method according to an embodiment of the present application;
FIG. 4 illustrates a schematic view of a spherical map provided by an embodiment of the present application;
FIG. 5 illustrates a schematic diagram of a longitude and latitude map provided by an embodiment of the present application;
fig. 6 is a schematic flowchart illustrating a binocular depth perception method according to an embodiment of the present application;
FIG. 7 is a schematic diagram illustrating a first warp and weft map and a second warp and weft map provided in an example of the present application;
fig. 8 is a schematic flowchart illustrating a binocular depth perception method according to an embodiment of the present application;
fig. 9 is a schematic diagram illustrating a longitude disparity map provided in an embodiment of the present application;
fig. 10 is a schematic diagram illustrating a conversion of a longitude disparity map provided by an embodiment of the present application into a depth map;
fig. 11 is a schematic flowchart of a binocular depth perception method according to an embodiment of the present application;
fig. 12 is a schematic flowchart illustrating a binocular depth perception method according to an embodiment of the present application;
fig. 13 is a block diagram illustrating a binocular depth perception device according to an embodiment of the present application;
fig. 14 is a block diagram of an electronic device for executing a binocular depth perception method according to an embodiment of the present application;
fig. 15 illustrates a storage unit for storing or carrying program code implementing a binocular depth perception method according to an embodiment of the present application.
Detailed Description
In order to make the technical solutions better understood by those skilled in the art, the technical solutions in the embodiments of the present application will be clearly and completely described below with reference to the drawings in the embodiments of the present application.
With the development of big data and the improvement of computing power, artificial intelligence is developed rapidly. Wherein, also there is artificial intelligence's application in the development of aircraft, for example, hovercar, unmanned aerial vehicle etc.. In the related art, the perception of the environmental depth in the automatic driving of the flying automobile is a key part, and the general flying automobile does not have omnidirectional dual-purpose hardware configuration for the reasons of beauty, weight reduction, cost saving and the like. At present, depth perception is mostly realized by utilizing equipment such as laser radar, millimeter wave radar, homogeneous binocular and monocular; the laser radar and the millimeter radar cannot be installed and arranged in an omnidirectional manner due to the cost and the volume; and homogeneous binocular requires that the binocular camera is connected on a rigid structure, and the length of the base line influences the distance precision of binocular perception, so that the binocular equipment is difficult to be arranged omnidirectionally on the flying automobile. In addition, the monocular depth estimation is performed in a deep learning mode, but the perceived depth scale is poor in stability and accuracy and is limited greatly.
Therefore, in the related art, in developing an aircraft, there is a challenge to accurately perceive the depth of the environment.
In order to solve the above problems, the inventor finds and provides a binocular depth perception method, a binocular depth perception device, an electronic device, and a storage medium provided in the embodiments of the present application through long-term research, and the method, the device, the electronic device, and the storage medium perceive the depth of a common-view region by projecting an image acquired by a camera having the common-view region onto a spherical surface and then expanding the image, thereby improving the accuracy of perceiving the depth of an environment. The specific binocular depth perception method is described in detail in the following embodiments.
Referring to fig. 1, fig. 1 is a schematic flowchart illustrating a binocular depth perception method according to an embodiment of the present application. According to the binocular depth perception method, the image collected by the camera with the common-view area is projected to the spherical surface and then expanded, so that the depth of the common-view area is perceived, and the accuracy of perceiving the depth of the environment is improved. In a specific embodiment, the binocular depth perception method may be applied to the binocular depth perception device 200 as shown in fig. 13 and the electronic apparatus 100 (fig. 14) equipped with the binocular depth perception device 200. The specific flow of the present embodiment will be described below by taking an electronic device as an example, and it is understood that the electronic device applied in the present embodiment may include an aircraft, and the aircraft may include a first camera and a second camera, where the first camera and the second camera have a common viewing area. It is understood that the electronic device may also include a vehicle, a robot, a ship, and other movable devices, which are not limited herein. As will be explained in detail with respect to the flow shown in fig. 1, the binocular depth perception method may specifically include the following steps:
step S110: and acquiring the image of the common-view area acquired by the first camera as a first image, and acquiring the image of the common-view area acquired by the second camera as a second image.
In some embodiments, the aircraft may be a drone, a flying automobile, or an airship. Wherein the aerial vehicle may comprise a first camera and a second camera; the first camera can be installed at the front side position, the rear side position and the like of the aircraft body, and the second camera can be installed at the front side position, the rear side position and the like of the aircraft body.
The type of the first camera can be a pinhole camera, a wide-angle camera, a fisheye camera and the like; the type of the second camera may be a pinhole camera, a wide-angle camera, a fisheye camera, or the like. The first camera of any type and the second camera of any type have a common viewing area; wherein, the first camera of different grade type can constitute heterogeneous two meshes with the second camera under the prerequisite that has the region of looking altogether. In the embodiment of the present application, on the premise of ensuring that the first camera and the second camera have the common viewing area, the aircraft includes the installation positions of the first camera and the second camera, which is not limited herein.
Illustratively, the aircraft is a flying car, wherein the side of the flying car may employ a front-side or rear-side camera for view coverage of the flying car for target detection. The side fisheye camera in the flying automobile can be used as a first camera, and the pinhole camera in front of or behind the flying automobile can be used as a second camera; wherein, the fish-eye camera of hovercar perception side and the pinhole camera before or behind the side can be combined into heterogeneous two eyes.
In some embodiments, the processor in the aircraft may calibrate internal and external parameters of the first camera and the second camera based on a preset calibration algorithm, and store the calibrated parameters obtained after calibration. The preset calibration algorithm may be a Zhangyingyou calibration method, a traditional camera calibration method, an active vision camera calibration method, a camera self-calibration method, and the like. The calibration parameters may include one or more combinations of internal and external parameters of the first camera and the second camera, such as a first focal length of the first camera, a second focal length of the second camera, a first optical center of the first camera, a second optical center of the second camera, and a rotation matrix of the first camera and the second camera.
Illustratively, the aircraft is a flying car, wherein a flying car side fisheye camera is used as the first camera, and a flying car side forward camera or a side backward camera is used as the second camera; the side fisheye camera and the side forward camera or the side backward camera of the hovercar are combined into a heterogeneous binocular, and binocular depth perception in the embodiment of the application is achieved. The processor in the hovercar can perform internal reference calibration on the side fisheye, the front-side camera and the rear-side camera by adopting a Zhang Zhengyou calibration method, and can also calibrate the external reference (such as rotation and translation) relation between the side fisheye and the front-side (or rear-side) camera by adopting the Zhang Zhengyou calibration method to obtain external parameters, such as a rotation matrix of the side fisheye and the front-side (or rear-side) camera.
In some embodiments, the aircraft may also obtain calibration parameters of the first camera and the second camera from the associated cloud or electronic device directly through a wireless communication technology (e.g., bluetooth, wiFi, zigbee technology, etc.), and store the calibration parameters; the aircraft can also obtain calibration parameters of the first camera and the second camera from the associated electronic equipment directly through a serial communication interface (such as a serial peripheral interface).
Further, the aircraft may acquire an image of the common-view area acquired by the first camera as a first image and acquire an image of the common-view area acquired by the second camera as a second image.
In some embodiments, the first image acquired by the aircraft has a frame-triggered relationship with the second image. Illustratively, the first image acquired by the hovercar fish-eye camera and the second image acquired by the side-view camera are triggered images of the same frame. The method for acquiring the images triggered by the two cameras in the same frame can be that trigger lines are added into the two cameras, and then the trigger lines are driven by hardware to trigger the cameras to expose at the same time, so that image data triggered by the two cameras in the same frame is obtained.
Step S120: and performing spherical projection and expansion on the first image to obtain a first longitude and latitude map, and performing spherical projection and expansion on the second image to obtain a second longitude and latitude map.
In some embodiments, after the aircraft obtains the first image and the second image, the first image may be subjected to perspective projection, and a projection image obtained after the perspective projection may be used as the first warp-weft map; the second image may be perspective-projected, and a projection image obtained by the perspective projection may be used as the second longitude and latitude map.
In the embodiment of the application, in view of reserving the field of view of the wide-angle camera (the first camera and/or the second camera), the first image may be spherically projected and expanded to obtain the first longitude and latitude map, and the second image may be spherically projected and expanded to obtain the second longitude and latitude map, so that the field of view information of the camera is reserved to the maximum extent.
In some embodiments, step S120 may include steps S1210-S1240.
Step 1210: and projecting the first image to a first unit spherical surface to obtain a first spherical surface image, and projecting the second image to a second unit spherical surface to obtain a second spherical surface image.
In some embodiments, after the aircraft obtains the first image and the second image, the aircraft may project the first image onto the first unit sphere to obtain a first spherical image and project the second image onto the second unit sphere to obtain a second spherical image.
Illustratively, a first image obtained by a fish-eye camera in the hovercar is projected onto the first unit spherical surface, and a second image obtained by a side view camera of the hovercar is projected onto the second unit spherical surface, respectively. The first unit spherical surface and the second unit spherical surface may be the same unit spherical surface or different unit spherical surfaces. The fisheye camera can project the first image to the first unit spherical surface based on the equidistant projection model to obtain a first spherical surface image; the side view camera may project a second image to a second unit sphere based on the pinhole projection model, obtaining a second spherical image.
In some embodiments, referring to fig. 2, step S1210 may include steps S1211-S1213.
Step S1211: and acquiring a first focal length of the first camera and a second focal length of the second camera.
In some embodiments, the processor in the aircraft may calibrate the calibration parameters of the first camera and the second camera according to a preset calibration algorithm (e.g., a zhangying calibration method, etc.), and store the calibrated parameters; the calibration parameters may include a first focal length of the first camera and a second focal length of the second camera.
In the process that the aircraft projects the first image to the first unit spherical surface, a processor in the aircraft can directly acquire the first focal length of the first camera and the second focal length of the second camera from the memory.
Step S1212: and projecting the first image to the first unit spherical surface based on the first focal length and an equidistant projection model to obtain the first spherical image.
In some embodiments, an equidistant projection model may be preset in the aircraft, and after the aircraft obtains the first focal length of the first camera, an incident angle (θ) at which light corresponding to the first image is incident on the first camera may be further obtained. Further, the aircraft may project the first image to the first unit sphere based on the first focal length and the equidistant projection model, to obtain a first spherical image.
Wherein the first focal length (f) can be adjusted 1 ) And substituting the incidence angle (theta) of the light ray corresponding to the first image entering the first camera into the equidistant projection model formula (r) 1 ＝f 1 θ) of r 1 Is the distance from a certain pixel point in the imaged first spherical image to the main light point of the first spherical image. Wherein r is 1 Can be calculated by the distance formula of the pixel and the central point
And (4) obtaining. Wherein u is 1 Indicating the abscissa position, v, of a certain pixel in the first image 1 Indicating the ordinate position, cx, of a pixel in the first image 1 The abscissa position, cy, representing the optical center (center point) of the first image 1 Indicating the ordinate position of the first image optical center (center point). Further, the aircraft can obtain a first spherical image coordinate (x) obtained by projecting to a first unit spherical surface according to an incident angle theta of the first image corresponding to the light incident to the first camera, a coordinate of a pixel point in the first image and a first spherical coordinate calculation formula 1 ，y 1 ，z 1 ). Wherein, the first spherical calculation formula is as follows:
z 1 ＝cos(θ)；
x 1 ＝sin(θ)cos(α 1 )；
y 1 ＝sin(θ)cos(α 1 )。
wherein alpha is 1 And the included angle between the abscissa and the ordinate of the pixel in the first spherical image is expressed.
Step S1213: and projecting the second image to the second unit spherical surface based on the second focal length and the pinhole projection model to obtain the second spherical image.
In some embodiments, a pinhole projection model may be preset in the aircraft, and after the aircraft obtains the second focal length of the second camera, an incident angle (θ) at which light corresponding to the second image is incident on the second camera may be further obtained. The incident angle of the light corresponding to the first image incident to the first camera and the incident angle of the light corresponding to the second image incident to the second camera may be the same. Further, the aircraft may project the second image to a second unit sphere based on the second focal length and the equidistant pinhole projection model, to obtain a second spherical image.
Wherein the second focal length (f) can be adjusted 2 ) And substituting the incidence angle (theta) of the light ray corresponding to the second image entering the second camera into the pinhole projection model formula (r) 2 ＝f 2 tan (θ)), where r 2 Is the distance from a certain pixel point in the imaged second spherical image to the main light spot of the second spherical image. Wherein r is 2 Can be calculated by the distance formula of the pixel and the central point
And (4) obtaining. Wherein u is 2 Indicating the abscissa position, v, of a certain pixel in the second image 2 Indicating the position of the ordinate, cx, of a certain pixel in the second image 2 The abscissa position, cy, representing the optical center (center point) of the second image 2 Indicating the ordinate position of the optical center (center point) of the second image.
Furthermore, the aircraft can obtain a second unit sphere projected to the second unit sphere according to the incident angle theta of the light corresponding to the second image and incident on the second camera, the coordinates of the pixel points in the second image and a second sphere calculation formulaTwo spherical surface image coordinate (x) 2 ，y 2 ，z 2 ). Wherein, the second sphere computational formula is as follows:
z 2 ＝cos(θ)；
x 2 ＝sin(θ)cos(α 2 )；
y 2 ＝sin(θ)cos(α 2 )。
wherein alpha is 2 And the included angle between the abscissa and the ordinate of the pixel in the second spherical image is represented. Although the projection models of the first camera and the second camera which are correspondingly projected to the unit spherical surface are different, the incident angles of the first image and the second image which correspond to the light rays incident to the corresponding cameras can be converted into the same angle, for example, the incident angles are all theta.
Step S1220: and carrying out polar line alignment on the first spherical image and the second spherical image to obtain a target first spherical image and a target second spherical image.
In some embodiments, after the aircraft obtains the first spherical image and the second spherical image, the aircraft may perform epipolar alignment on the first spherical image and the second spherical image to obtain the target first spherical image and the target second spherical image.
The aircraft can acquire calibration parameters of the first camera and the second camera, and further, the aircraft can perform polar line alignment on the first spherical image and the second spherical image based on the acquired calibration parameters of the first camera and the second camera to acquire a first spherical image of a target and a second spherical image of the target.
For example, the aircraft may obtain the rotation matrices of the fisheye camera and the side view camera by calling a stereoRectify function of OpenCV; further, the aircraft may perform epipolar alignment on the first spherical image of the corresponding fisheye camera and the second spherical image of the corresponding side view camera based on the rotation matrices of the fisheye camera and the side view camera to obtain the target first spherical image and the target second spherical image. And obtaining an effect image after polar lines of the first spherical image and the second spherical image are aligned, namely the target first spherical image and the target second spherical image.
In some embodiments, referring to fig. 3, step S1220 may include steps S1221 to S1222.
Step S1221: and acquiring a rotation matrix of the first camera and the second camera.
In some embodiments, the processor in the aircraft may calibrate the calibration parameters of the first camera and the second camera according to a preset calibration algorithm (e.g., a zhangying calibration method, etc.), and store the calibrated parameters; the calibration parameters may include a rotation matrix of the first camera and the second camera.
In the process that the aircraft carries out polar line alignment on the first spherical image and the second spherical image to obtain the first spherical image and the second spherical image of the target, a processor in the aircraft can directly obtain the rotation matrix of the first camera and the second camera from a memory.
Step S1222: and performing epipolar alignment on the first spherical image and the second spherical image based on the rotation matrix to obtain the target first spherical image and the target second spherical image.
In some embodiments, after the aircraft obtains the rotation matrices of the first camera and the second camera, the aircraft may perform polar alignment on the first spherical image and the second spherical image based on the rotation matrices to obtain a target first spherical image and a target second spherical image.
For example, the aircraft may obtain the rotation matrix R of the fisheye camera (first camera) and the side view camera (second camera) by calling the stereorectification function of OpenCV. Further, the aircraft can acquire coordinates (x) of each pixel point in the first spherical image 1 ，y 1 ，z 1 ) And the coordinates (x) of each pixel point in the second spherical image 2 ，y 2 ，z 2 ) And the rotation matrix R and the coordinates (x) of each pixel point in the first spherical image are compared 1 ，y 1 ，z 1 ) And the coordinates (x) of each pixel point in the second spherical image 2 ，y 2 ，z 2 ) Substituting the alignment formula:
Step S1230: and unfolding the first spherical image of the target to obtain a first longitude and latitude map.
In some embodiments, after the aircraft obtains the target first spherical image, the aircraft may expand the target first spherical image to obtain a first longitude and latitude map.
For example, referring to fig. 4 and 5, fig. 4 shows a first spherical image of a target, and fig. 5 shows a first longitude and latitude map. Wherein, each pixel point in the target first spherical image can utilize (x) 1 '，y 1 '，z 1 ') coordinate point representation; the first longitude and latitude map obtained by expanding the first spherical image of the target can be obtained by converting spherical coordinates into longitude and latitude according to the diagram 4, and finally taking the longitude lambda as a horizontal coordinate and taking the latitude lambda as a horizontal coordinate
And unfolding a longitude and latitude map for the ordinate, namely a first longitude and latitude map.
In some embodiments, referring to fig. 6, step S1230 may include steps S1231-S1234.
Step S1231: and determining a second target pixel point from the target first spherical image, and acquiring a first abscissa, a first ordinate and a first ordinate of the second target pixel point in the target first spherical image.
In some embodiments, after the aircraft obtains the target first spherical image, a second target pixel point may be determined from the target first spherical image, and a first abscissa (x) corresponding to the second target pixel point in the target first spherical image is obtained 1 '), first ordinate (y) 1 ') and a first vertical coordinate (z) 1 ')。
Step S1232: and obtaining the abscissa of the second target pixel point corresponding to the first warp-weft image according to the fact that the first abscissa is equal to the negative number of cosine values of the abscissa in the first warp-weft image.
In some embodiments, the aircraft may be preset with a conversion formula of the spherical image expanded into a longitude map:
x'＝-cos(λ)；
wherein x ' represents the abscissa of the pixel point in the spherical image, y ' represents the ordinate of the pixel point in the spherical image, z ' represents the ordinate of the pixel point in the spherical image, λ represents the abscissa of the pixel point in the graticule,
the abscissa of a pixel in the longitude and latitude map is represented.
Further, the aircraft obtains a first abscissa (x) of the second target pixel point corresponding to the first spherical surface of the target 1 ') thereafter, may be based on a first abscissa (x) 1 ') is equal to the abscissa (λ) of the first longitude and latitude plot 1 ) The negative of the cosine value of (1), i.e. x 1 '＝-cos(λ 1 ) Obtaining the abscissa (lambda) of the second target pixel point corresponding to the first warp-weft map 1 )。
Step S1233: and obtaining the ordinate, corresponding to the first warp-weft map, of the second target pixel point according to the fact that the first ordinate is equal to the negative number of the product of the sine value of the abscissa and the cosine value of the ordinate in the first warp-weft map.
In some embodiments, the aircraft obtains the second target pixel point corresponding to a first ordinate (y) in the target first spherical image 1 ') and the second target pixel point corresponds to the abscissa (λ) in the first warp map 1 ) Then, it can be determined from the first ordinate (y) 1 ') is equal to the abscissa (λ) of the first longitude and latitude plot 1 ) Sine value and ordinate of
Is negative of the product of the cosine value of (i), i.e. < >>
Obtaining a ordinate ÷ greater than or equal to a first warp image corresponding to a second target pixel point>
In some embodiments, step S1233 may be step S12331.
Step S12331: and obtaining the ordinate of the second target pixel point corresponding to the first warp-weft image according to the fact that the first vertical coordinate is equal to the product of the sine value of the abscissa and the sine value of the ordinate in the first warp-weft image.
In some embodiments, the aircraft obtains the second target pixel point corresponding to a first ordinate (y) in the target first spherical image 1 ') and the second target pixel point corresponds to the abscissa (λ) in the first warp map 1 ) Then, the first vertical coordinate (z) can be used 1 ') is equal to the abscissa (λ) of the first longitude and latitude plot 1 ) Sine value and ordinate (y) 1 ') the product of the sine values, i.e.
Obtaining a longitudinal coordinate ≧ in the first warp map that the second target pixel point corresponds to>
Step S1234: obtaining the first warp-weft map based on the abscissa of the second target pixel point corresponding to the first warp-weft map and the ordinate of the second target pixel point corresponding to the first warp-weft map.
In some embodiments, after the aircraft obtains the second target pixel point corresponding to the abscissa in the first warp map and the second target pixel point corresponding to the ordinate in the first warp map, the first warp map may be obtained based on the second target pixel point corresponding to the abscissa in the first warp map and the second target pixel point corresponding to the ordinate in the first warp map.
Step S1240: and unfolding the second spherical image of the target to obtain a second longitude and latitude map.
In some embodiments, after the aircraft obtains the second spherical image of the target, the second spherical image of the target may be expanded to obtain the second theodolite.
Illustratively, referring again to fig. 4 and 5, fig. 4 shows a second spherical image of the target, and fig. 5 shows a second longitude and latitude map. Wherein, each pixel point in the target second spherical image can utilize (x) 2 '，y 2 '，z 2 ') coordinate point representation; the second longitude and latitude map obtained by expanding the second spherical image of the target can be obtained by converting spherical coordinates into longitude and latitude according to the diagram 4, and finally taking the longitude lambda as a horizontal coordinate and taking the latitude lambda as a horizontal coordinate
And unfolding a longitude and latitude map for the ordinate, namely a second longitude and latitude map.
For example, please refer to fig. 7, which shows a first longitude and latitude map and a second longitude and latitude map. Wherein the horizontal lines in fig. 7 represent the epipolar lines in alignment; the aerocar carries out polar line alignment on a first spherical image corresponding to the fisheye camera and a second spherical image corresponding to the side-looking camera based on the rotation matrix of the fisheye camera and the side-looking camera to obtain a first spherical image of a target and a second spherical image of the target; further, the hovercar expands the first spherical image of the target to obtain a first longitude and latitude map, as shown in the left side of fig. 7, corresponding to the fisheye camera, and expands the second spherical image of the target to obtain a second longitude and latitude map, as shown in the right side of fig. 7, corresponding to the rear side camera.
In some embodiments, referring to fig. 8, step S1240 may include steps S1241 to S1244.
Step S1241: and determining a third target pixel point from the target second spherical image, and acquiring a second abscissa, a second ordinate and a second ordinate, which correspond to the third target pixel point, in the target second spherical image.
In some embodiments, after the aircraft obtains the target second spherical image, a third target pixel point may be determined from the target second spherical image, and a second abscissa (x) corresponding to the third target pixel point in the target second spherical image is obtained 2 '), second ordinate (y) 2 ') and a second vertical coordinate (z) 2 ')。
Step S1242: and obtaining the abscissa of the third target pixel point corresponding to the second longitude and latitude map according to the fact that the second abscissa is equal to the negative number of the cosine value of the abscissa in the second longitude and latitude map.
In some embodiments, the aircraft may be preset with a conversion formula of the spherical image expanded into the longitude map:
x'＝-cos(λ)；
wherein x ' represents the abscissa of the pixel point in the spherical image, y ' represents the ordinate of the pixel point in the spherical image, z ' represents the ordinate of the pixel point in the spherical image, λ represents the abscissa of the pixel point in the graticule,
the abscissa of the pixel point in the longitude and latitude map is represented.
Further, the aircraft obtains a second abscissa (x) of the third target pixel point corresponding to the target second sphere 2 ') thereafter, can be based on two abscissas (x) 2 ') is equal to the abscissa (λ) of the second longitude and latitude plot 2 ) The negative of the cosine value of (1), i.e. x 2 '＝-cos(λ 2 ) Obtaining the abscissa (lambda) of the third target pixel point corresponding to the second longitude and latitude map 2 )。
Step S1243: and obtaining the ordinate of the third target pixel point corresponding to the second longitude and latitude map according to the second ordinate being equal to the negative number of the product of the sine value of the abscissa and the cosine value of the ordinate in the second longitude and latitude map.
In some embodiments, the aircraft obtains a second ordinate (y) where the third target pixel point corresponds to in the target second sphere 2 ') and the third target pixel point corresponds to the abscissa (λ) in the second longitude and latitude map 2 ) After that, it can be determined from the second ordinate (y) 2 ') is equal to the abscissa (λ) of the second longitude and latitude plot 2 ) Sine value and ordinate of
The negative of the product of cosine values of (i.e. of)
Obtaining the longitudinal coordinate ^ in the second longitude and latitude map corresponding to the third target pixel point>
In some embodiments, step S1243 may be step S12431.
Step S12431: and obtaining the longitudinal coordinate of the third target pixel point corresponding to the second longitude and latitude map according to the second vertical coordinate equal to the product of the sine value of the abscissa and the sine value of the ordinate in the second longitude and latitude map.
In some embodiments, the aircraft obtains a third target pixel point corresponding to the target second pixel pointSecond ordinate (y) in sphere 2 ') and the third target pixel point corresponds to the abscissa (λ) in the second longitude and latitude map 2 ) Thereafter, the second vertical coordinate (z) can be used 2 ') is equal to the abscissa (λ) of the second longitude and latitude plot 2 ) Sine value of (a) and ordinate (y) 2 ') the product of the sine values, i.e.
Obtaining the longitudinal coordinate ^ in the second longitude and latitude map corresponding to the third target pixel point>
Step S1244: and obtaining the second longitude and latitude map based on the corresponding abscissa of the third target pixel point in the second longitude and latitude map and the corresponding ordinate of the third target pixel point in the second longitude and latitude map.
In some embodiments, after the aircraft obtains the third target pixel point corresponding to the abscissa in the second longitude and latitude map and the ordinate corresponding to the third target pixel point in the second longitude and latitude map, the second longitude and latitude map may be obtained based on the third target pixel point corresponding to the abscissa in the second longitude and latitude map and the ordinate corresponding to the third target pixel point in the second longitude and latitude map.
It can be understood that the projection model of the camera is converted into the spherical projection from the traditional perspective projection, and then the projected spherical image is expanded, so that the visual field of the camera is retained to the maximum extent, and the accuracy of depth perception is improved.
Step S130: and obtaining a longitude disparity map according to the first longitude and latitude map and the second longitude and latitude map.
In some embodiments, after the aircraft obtains the first longitude map and the second longitude map, a longitude disparity map can be obtained from the first longitude map and the second longitude map. The aircraft can be preset with a stereoscopic vision matching algorithm, such as BM and SGBM algorithms; further, the aircraft can perform stereo matching on the first longitude and latitude map and the second longitude and latitude map based on a stereo matching algorithm to obtain a longitude and disparity map.
For example, please refer to fig. 7 and fig. 9, wherein the first longitude and latitude map is shown on the left side of fig. 7, and the second longitude and latitude map is shown on the right side of fig. 7; the hovercar can call a StereoBM or stereogbm algorithm of OpenCV to perform stereo matching on the first longitude and latitude map and the second longitude and latitude map to obtain a longitude disparity map (as shown in fig. 9).
Step S140: and converting the longitude disparity map into a depth map, and obtaining the depth of the common view area based on the depth map.
In some embodiments, after the aircraft obtains the longitudinal disparity map, the longitudinal disparity map may be converted into a depth map, and the depth of the common view region may be obtained based on the depth map.
For example, please refer to fig. 10, which shows a schematic diagram of converting a longitude disparity map into a depth map. Wherein, the aircraft can acquire the optical center O of the first camera l I.e. the optical center of the left camera, the optical center O of the second camera can also be obtained r I.e. the optical center of the right camera; further, the aircraft can obtain the base length Bl according to the first optical center and the second optical center, that is, the line segment
I.e. the length of the base line between the two cameras. The aircraft can determine a first target pixel point (p) from the longitude disparity map and acquire a longitude coordinate (lambda) of the first target pixel point in the longitude disparity map 0 ) And latitude coordinate->
Wherein the arc p in FIG. 10 l p r Is the first target pixel point ^ in the longitude disparity map>
Lower longitude disparity value (d). Further, the aircraft may convert the longitude disparity map into a depth map, and obtain the depth of the common view region based on the depth map, that is, obtain the three-dimensional world coordinate (x, y, z) of the first target pixel point p, where the depth of the common view region may be obtained according to the depth value corresponding to z。
In some embodiments, referring to fig. 10 and 11, step S140 may include steps S141 to S148.
Step S141; and acquiring a first optical center of the first camera and acquiring a second optical center of the second camera.
In some embodiments, the processor in the aircraft may calibrate and store the calibration parameters of the first camera and the second camera according to a preset calibration algorithm (e.g., a Zhang friend calibration method, etc.); the calibration parameters may include a first optical center of the first camera and a second optical center of the second camera.
Wherein, the aircraft is based on the depth in-process of the heterogeneous binocular perception common vision zone that first camera and second camera constitute, and the first optical center of first camera and the second optical center of second camera can directly be obtained from the memory to the treater in the aircraft.
Step S142: and obtaining the length of a base line according to the first optical center and the second optical center.
In some embodiments, after the aircraft obtains the first optical center and the second optical center, the baseline length Bl may be obtained from the first optical center and the second optical center.
Step S143: and determining a first target pixel point from the longitude disparity map, and acquiring a longitude coordinate and a latitude coordinate of the first target pixel point in the longitude disparity map.
In some embodiments, the aircraft may determine the first target pixel point (p) from the longitude disparity map, and acquire a longitude coordinate (λ) of the first target pixel point in the longitude disparity map 0 ) And latitude coordinate
The first target pixel point may be any one pixel point in the longitude disparity map. For example, please refer to fig. 10, wherein the first target pixel point may be any one pixel point in the longitude disparity map shown in fig. 10.
Step S144: and acquiring a first target longitude of the first target pixel point in the target first spherical image, and acquiring a second target longitude of the first target pixel point in the target second spherical image.
In some embodiments, after the aircraft determines the first target pixel point from the longitude disparity map, a first target longitude (p) corresponding to the first target pixel point in the target first spherical image may be obtained l Radian) of the first target pixel point, and acquiring a second target longitude (p) of the first target pixel point corresponding to the second spherical image of the target r Arc of).
Step S145: and subtracting the second target longitude from the first target longitude to obtain a longitude disparity value of the first target pixel point in the longitude disparity map.
In some embodiments, the aircraft obtains a first target longitude p l And a second target longitude p r Then, the second target longitude may be subtracted from the first target longitude to obtain a longitude disparity value, i.e., p, corresponding to the first target pixel point in the longitude disparity map l -p r ＝d。
Step S146: multiplying the first baseline by a sine of the second target longitude to obtain a first distance.
In some embodiments, the aircraft obtains a first baseline Bl and a second target longitude p r Thereafter, the first baseline may be multiplied by a sine value of the second target longitude to obtain a first distance AO l I.e. AO l ＝Bl×sin(p r )。
Step S147: dividing the first distance by a sine of the longitude disparity value to obtain a second distance.
In some embodiments, the aircraft obtains the first distance AO l And a longitude disparity value d, the first distance may be divided by the sine of the longitude disparity value to obtain a second distance
I.e. <' > means>
Step S148: and multiplying the second distance by the cosine value of the longitude coordinate and the cosine value of the latitude coordinate to obtain the depth value of the first target pixel point in the depth map, and obtaining the depth of the common-view area based on the depth value.
In some embodiments, the aircraft obtains the second distance
And longitude coordinates (lambda) of the first target pixel point in the longitude disparity map 0 ) And latitude coordinate>
Then, the second distance may be multiplied by a cosine value of the longitude coordinate and a cosine value of the latitude coordinate to obtain a depth value z of the first target pixel point in the depth map, that is, the depth value z is obtained
Wherein the aircraft acquires a second distance>
And longitude coordinates (lambda) of the first target pixel point in the longitude disparity map 0 ) Then, the second distance may also be multiplied by a sine value of the longitude coordinate of the first target pixel point in the longitude disparity map to obtain an abscissa value x corresponding to the first target pixel point in the depth map, that is, the abscissa value x is ≥ r>
Wherein the aircraft acquires a second distance>
And longitude coordinates (lambda) of the first target pixel point in the longitude disparity map 0 ) And latitude coordinate>
Then, also can beThe second distance is multiplied by a cosine value of a longitude coordinate of the first target pixel point in the longitude disparity map and multiplied by a sine value of a latitude coordinate of the first target pixel point in the longitude disparity map, so that a longitudinal coordinate value y, namely ^ er, corresponding to the first target pixel point in the depth map is obtained>
Further, the aircraft obtains a three-dimensional world coordinate system corresponding to the first target pixel point P, namely coordinates (x, y, z) of the depth map; where z is the corresponding depth value, the aerial vehicle may obtain the depth of the first target pixel point corresponding to the common-view region based on the depth value.
For example, referring to fig. 12, the binocular depth perception method provided in the present application is applied to a flying car, where the flying car includes a fisheye camera, i.e., a first camera, and a front-side (or rear-side) camera, i.e., a second camera, and the fisheye camera and the front-side (or rear-side) camera have a common viewing area, and a specific hardware configuration is not limited herein. The hovercar can respectively project images of a common-view area acquired by a fisheye camera and a front-side (or rear-side) camera to a unit spherical surface, further, polar alignment is carried out on the spherical images obtained by projecting to the unit spherical surface, longitude and latitude expansion maps corresponding to the cameras are generated, further, a longitude parallax map can be generated by utilizing a stereo matching algorithm based on the longitude and latitude expansion maps, then the longitude parallax map is converted into a depth map, and the depth of the common-view area is sensed based on the depth map.
It can be understood that the cost of perception depth is reduced based on heterogeneous binocular perception environment depth, meanwhile, the projection model of the camera is converted from traditional perspective projection to spherical projection and then expanded, the visual field of the wide-angle camera is reserved to the maximum extent, and the accuracy of depth perception is improved.
In some embodiments, the binocular depth perception method provided in the embodiments of the present application may further include step S150 after step S140.
Step S150: controlling the aircraft to fly based on the depth.
In some embodiments, after the aircraft obtains the depth of the common viewing area, a processor in the aircraft may plan the attitude, speed, path, etc. of the flight of the aircraft based on the depth during left and right flight of the aircraft. It can be appreciated that the safety of the aircraft is improved in conjunction with the environmental depth control of the aircraft flight.
The binocular depth perception method provided by the embodiment of the application is applied to an aircraft, the aircraft can comprise a first camera and a second camera, wherein the first camera and the second camera have a common-view region, an image of the common-view region acquired by the first camera is acquired as a first image, and an image of the common-view region acquired by the second camera is acquired as a second image; performing spherical projection and expansion on the first image to obtain a first longitude and latitude map, and performing spherical projection and expansion on the second image to obtain a second longitude and latitude map; obtaining a longitude disparity map according to the first longitude and latitude map and the second longitude and latitude map; the longitude disparity map is converted into a depth map, the depth of the common-view area is obtained based on the depth map, and then the image collected by the camera with the common-view area is projected to a spherical surface and then unfolded, so that the depth of the common-view area is sensed, and the accuracy of sensing the depth of the environment is improved. In addition, the flight of the aircraft can be controlled based on the depth, and the flight safety of the aircraft is further improved.
Referring to fig. 13, fig. 13 is a block diagram illustrating a binocular depth perception device according to an embodiment of the present application. This binocular depth perception device 200 is applied to above-mentioned aircraft, the aircraft includes first camera and second camera, wherein, first camera with there is the region of looking altogether in the second camera. As will be explained in detail with respect to the flow shown in fig. 13, the binocular depth perception device 200 includes: a camera image obtaining module 210, a longitude and latitude map obtaining module 220, a longitude and latitude disparity map obtaining module 230, and a depth perception module 240, wherein:
the camera image obtaining module 210 is configured to obtain an image of the common-view area collected by the first camera as a first image, and obtain an image of the common-view area collected by the second camera as a second image.
The longitude and latitude map obtaining module 220 is configured to perform spherical projection on the first image and expand the first image to obtain a first longitude and latitude map, and perform spherical projection on the second image and expand the second image to obtain a second longitude and latitude map.
A longitude disparity map obtaining module 230 configured to obtain a longitude disparity map according to the first longitude disparity map and the second longitude disparity map.
A depth perception module 240, configured to convert the longitude disparity map into a depth map, and obtain a depth of the common view region based on the depth map.
Further, after the converting the longitude disparity map into a depth map and obtaining the depth of the common viewing area based on the depth map, the binocular depth perception device 200 may include: a flight control module, wherein:
and the flight control module is used for controlling the flight of the aircraft based on the depth.
Further, the longitude and latitude map obtaining module 220 may include: the spherical image acquisition module, the spherical image polar alignment module, the first longitude and latitude map acquisition module and the second longitude and latitude map acquisition module, wherein:
and the spherical image obtaining module is used for projecting the first image to a first unit spherical surface to obtain a first spherical image, and projecting the second image to a second unit spherical surface to obtain a second spherical image.
And the spherical image polar line alignment module is used for carrying out polar line alignment on the first spherical image and the second spherical image to obtain a target first spherical image and a target second spherical image.
And the first longitude and latitude map acquisition module is used for unfolding the first spherical image of the target to acquire a first longitude and latitude map.
And the second longitude and latitude map acquisition module is used for unfolding the second spherical image of the target to acquire a second longitude and latitude map.
Further, the spherical image obtaining module may include: focus obtains unit, first spherical image acquisition unit and second spherical image acquisition unit, wherein:
and the focal length acquisition unit is used for acquiring a first focal length of the first camera and a second focal length of the second camera.
A first spherical image obtaining unit, configured to project the first image to the first unit sphere based on the first focal length and the equidistant projection model, so as to obtain the first spherical image.
And the second spherical image obtaining unit is used for projecting the second image to the second unit spherical surface based on the second focal length and the pinhole projection model to obtain the second spherical image.
Further, the spherical image epipolar line alignment module may include: rotation matrix acquisition unit and spherical image polar line alignment subunit, wherein:
and the rotation matrix acquisition unit is used for acquiring the rotation matrix of the first camera and the second camera.
And the spherical image polar line alignment subunit is used for carrying out polar line alignment on the first spherical image and the second spherical image based on the rotation matrix to obtain the target first spherical image and the target second spherical image.
Further, the depth perception module 240 may include: the device comprises an optical center obtaining unit, a base length obtaining unit, a first target pixel point coordinate obtaining unit, a target longitude obtaining unit, a longitude parallax value obtaining unit, a first distance obtaining unit, a second distance obtaining unit and a depth value obtaining unit, wherein:
and the optical center acquisition unit is used for acquiring a first optical center of the first camera and acquiring a second optical center of the second camera.
And the base length obtaining unit is used for obtaining the base length according to the first optical center and the second optical center.
And the first target pixel point coordinate obtaining unit is used for determining a first target pixel point from the longitude disparity map and obtaining the longitude coordinate and the latitude coordinate of the first target pixel point in the longitude disparity map.
And the target longitude acquiring unit is used for acquiring a first target longitude of the first target pixel point in the target first spherical image and acquiring a second target longitude of the first target pixel point in the target second spherical image.
And a longitude disparity value obtaining unit, configured to subtract the second target longitude from the first target longitude, and obtain a longitude disparity value of the first target pixel point in the longitude disparity map.
A first distance obtaining unit configured to multiply the first baseline by a sine value of the second target longitude to obtain a first distance.
A second distance obtaining unit configured to divide the first distance by a sine value of the longitude disparity value to obtain a second distance.
And the depth value obtaining unit is used for multiplying the second distance by the cosine value of the longitude coordinate and the cosine value of the latitude coordinate to obtain the depth value of the first target pixel point in the depth map, and obtaining the depth of the common-view area based on the depth value.
Further, the first theodolite acquisition module may include: the second target pixel point coordinate acquisition unit, the first longitude and latitude map abscissa acquisition unit, the first longitude and latitude map ordinate acquisition unit and the first longitude and latitude map acquisition subunit, wherein:
and the second target pixel point coordinate obtaining unit is used for determining a second target pixel point from the target first spherical image and obtaining a first horizontal coordinate, a first vertical coordinate and a first vertical coordinate of the second target pixel point corresponding to the target first spherical image.
The first longitude and latitude map abscissa obtaining unit is used for obtaining the abscissa of the second target pixel point corresponding to the first warp-weft map according to the fact that the first abscissa is equal to the negative number of cosine values of the abscissa in the first warp-weft map.
A first longitude and latitude map longitudinal coordinate obtaining unit, configured to obtain a longitudinal coordinate of the second target pixel point in the first longitude and latitude map according to a negative number of a product of a sine value of an abscissa and a cosine value of an ordinate in the first longitude and latitude map, where the first longitudinal coordinate is equal to the product of the sine value of an abscissa in the first longitude and latitude map; or alternatively
Obtaining the ordinate of the second target pixel point corresponding to the first warp-weft image according to the fact that the first vertical coordinate is equal to the product of the sine value of the abscissa and the sine value of the ordinate in the first warp-weft image;
a first longitude and latitude map obtaining subunit, configured to obtain the first longitude and latitude map based on that the second target pixel point corresponds to an abscissa in the first longitude and latitude map and that the second target pixel point corresponds to an ordinate in the first longitude and latitude map.
Further, the second theodolite acquisition module may include: the third target pixel point coordinate acquisition unit, the second longitude and latitude map abscissa acquisition unit, the second longitude and latitude map ordinate acquisition unit and the second longitude and latitude map acquisition subunit, wherein:
and the third target pixel point coordinate acquisition unit is used for determining a third target pixel point from the target second spherical image and acquiring a second abscissa, a second ordinate and a second ordinate of the third target pixel point corresponding to the target second spherical image.
And the second longitude and latitude map abscissa obtaining unit is used for obtaining the third target pixel point corresponding to the abscissa in the second longitude and latitude map according to the second abscissa being equal to the negative number of the cosine value of the abscissa in the second longitude and latitude map.
A second longitude and latitude map longitudinal coordinate obtaining unit, configured to obtain a longitudinal coordinate, in which the third target pixel point corresponds to the second longitude and latitude map, according to a negative number of a product, in which the second longitudinal coordinate is equal to a sine value of an abscissa in the second longitude and latitude map and a cosine value of an ordinate; or
Obtaining the longitudinal coordinate of the third target pixel point corresponding to the second longitude and latitude map according to the second vertical coordinate being equal to the product of the sine value of the abscissa and the sine value of the ordinate in the second longitude and latitude map;
and the second longitude and latitude map obtaining subunit is used for obtaining the second longitude and latitude map based on that the third target pixel point corresponds to the abscissa in the second longitude and latitude map and the third target pixel point corresponds to the ordinate in the second longitude and latitude map.
It can be clearly understood by those skilled in the art that, for convenience and simplicity of description, the specific working processes of the above-described devices and modules may refer to the corresponding processes in the foregoing method embodiments, and are not described herein again.
In the several embodiments provided in the present application, the coupling between the modules may be electrical, mechanical or other type of coupling.
In addition, functional modules in the embodiments of the present application may be integrated into one processing module, or each of the modules may exist alone physically, or two or more modules are integrated into one module. The integrated module can be realized in a hardware mode, and can also be realized in a software functional module mode.
Referring to fig. 14, a block diagram of an electronic device according to an embodiment of the present disclosure is shown. The electronic device 100 may be a removable electronic device capable of running applications, such as a drone, a flying automobile, a boat, an automobile, and the like. The electronic device 100 in the present application may include one or more of the following components: a processor 110, a memory 120, and one or more applications, wherein the one or more applications may be stored in the memory 120 and configured to be executed by the one or more processors 110, the one or more programs configured to perform a method as described in the aforementioned method embodiments.
Processor 110 may include one or more processing cores, among other things. The processor 110 connects various parts within the overall electronic device 100 using various interfaces and lines, and performs various functions of the electronic device 100 and processes data by executing or executing instructions, programs, code sets, or instruction sets stored in the memory 120 and calling data stored in the memory 120. Alternatively, the processor 110 may be implemented in hardware using at least one of Digital Signal Processing (DSP), field-Programmable Gate Array (FPGA), and Programmable Logic Array (PLA). The processor 110 may integrate one or more of a Central Processing Unit (CPU), a Graphics Processing Unit (GPU), a modem, and the like. Wherein, the CPU mainly processes an operating system, a user interface, an application program and the like; the GPU is used for rendering and drawing the content to be displayed; the modem is used to handle wireless communications. It is understood that the modem may not be integrated into the processor 110, but may be implemented by a communication chip.
The memory 120 may include a random access memory or a read only memory. The memory 120 may be used to store instructions, programs, code sets, or instruction sets. The memory 120 may include a stored program area and a stored data area, wherein the stored program area may store instructions for implementing an operating system, instructions for implementing at least one function (such as a touch function, a sound playing function, an image playing function, etc.), instructions for implementing various method embodiments described below, and the like. The memory data area may also store data created by electronic device 100 during use (e.g., phone books, audiovisual data, chat log data), etc.
Referring to fig. 15, a block diagram of a computer-readable storage medium according to an embodiment of the present disclosure is shown. The computer readable medium 300 has stored therein program code that can be invoked by a processor to perform the methods described in the above method embodiments.
The computer-readable storage medium 300 may be an electronic memory such as a flash memory, an EEPROM (electrically erasable and programmable read only memory), an EPROM, a hard disk, or a ROM. Alternatively, the computer-readable storage medium 300 includes a non-volatile computer-readable medium. The computer readable storage medium 300 has storage space for program code 310 for performing any of the method steps described above. The program code can be read from or written to one or more computer program products. The program code 310 may be compressed, for example, in a suitable form.
To sum up, the binocular depth perception method, the binocular depth perception device, the electronic device and the storage medium provided by the embodiment of the application are applied to an aircraft, the aircraft may include a first camera and a second camera, wherein the first camera and the second camera have a common-view region, and an image of the common-view region acquired by the first camera is acquired as a first image and an image of the common-view region acquired by the second camera is acquired as a second image; performing spherical projection and expansion on the first image to obtain a first longitude and latitude map, and performing spherical projection and expansion on the second image to obtain a second longitude and latitude map; obtaining a longitude disparity map according to the first longitude and latitude map and the second longitude and latitude map; the longitude disparity map is converted into a depth map, the depth of the common-view area is obtained based on the depth map, and then the image collected by the camera with the common-view area is projected to a spherical surface and then unfolded, so that the depth of the common-view area is sensed, and the accuracy of sensing the depth of the environment is improved. In addition, the flight of the aircraft can be controlled based on the depth, and the flight safety of the aircraft is further improved.
Finally, it should be noted that: the above embodiments are only used to illustrate the technical solutions of the present application, and not to limit the same; although the present application has been described in detail with reference to the foregoing embodiments, those of ordinary skill in the art will understand that: the technical solutions described in the foregoing embodiments may still be modified, or some technical features may be equivalently replaced; such modifications and substitutions do not necessarily depart from the spirit and scope of the corresponding technical solutions in the embodiments of the present application.
Claims (10)
1. A binocular depth perception method is applied to an aircraft, the aircraft comprises a first camera and a second camera, wherein a common vision area exists between the first camera and the second camera, and the method comprises the following steps:
acquiring an image of the common-view area acquired by the first camera as a first image, and acquiring an image of the common-view area acquired by the second camera as a second image;
performing spherical projection and expansion on the first image to obtain a first longitude and latitude map, and performing spherical projection and expansion on the second image to obtain a second longitude and latitude map;
obtaining a longitude disparity map according to the first longitude and latitude map and the second longitude and latitude map;
and converting the longitude disparity map into a depth map, and obtaining the depth of the common view area based on the depth map.
2. The method according to claim 1, wherein after the converting the longitudinal disparity map into a depth map and obtaining the depth of the common viewing area based on the depth map, the method further comprises:
controlling the aircraft to fly based on the depth.
3. The method of claim 1, wherein the spherically projecting and unfolding the first image to obtain a first longitude map and the spherically projecting and unfolding the second image to obtain a second longitude map comprises:
projecting the first image to a first unit spherical surface to obtain a first spherical surface image, and projecting the second image to a second unit spherical surface to obtain a second spherical surface image;
polar line alignment is carried out on the first spherical image and the second spherical image, and a target first spherical image and a target second spherical image are obtained;
unfolding the target first spherical image to obtain a first warp-weft image;
and unfolding the second spherical image of the target to obtain a second longitude and latitude map.
4. The method of claim 3, wherein projecting the first image onto a first unit sphere to obtain a first spherical image and projecting the second image onto a second unit sphere to obtain a second spherical image comprises:
acquiring a first focal length of the first camera and a second focal length of the second camera;
projecting the first image to the first unit sphere based on the first focal length and an equidistant projection model to obtain a first spherical image;
and projecting the second image to the second unit spherical surface based on the second focal length and the pinhole projection model to obtain the second spherical image.
5. The method of claim 3, wherein epipolar aligning the first spherical image with the second spherical image to obtain a target first spherical image and a target second spherical image comprises:
acquiring a rotation matrix of the first camera and the second camera;
and performing epipolar alignment on the first spherical image and the second spherical image based on the rotation matrix to obtain the target first spherical image and the target second spherical image.
6. The method of claim 1, wherein converting the longitudinal disparity map into a depth map and obtaining the depth of the common view region based on the depth map comprises:
acquiring a first optical center of the first camera and a second optical center of the second camera;
obtaining a base length according to the first optical center and the second optical center;
determining a first target pixel point from the longitude disparity map, and acquiring a longitude coordinate and a latitude coordinate of the first target pixel point in the longitude disparity map;
acquiring a first target longitude of the first target pixel point in the target first spherical image and acquiring a second target longitude of the first target pixel point in the target second spherical image;
subtracting the second target longitude from the first target longitude to obtain a longitude disparity value of the first target pixel point in the longitude disparity map;
multiplying the first baseline by a sine of the second target longitude to obtain a first distance;
dividing the first distance by a sine of the longitude disparity value to obtain a second distance;
and multiplying the second distance by the cosine value of the longitude coordinate and the cosine value of the latitude coordinate to obtain the depth value of the first target pixel point in the depth map, and obtaining the depth of the common-view area based on the depth value.
7. The method of claim 3, wherein said unwrapping said first spherical image of said object to obtain a first warp map comprises:
determining a second target pixel point from the target first spherical image, and acquiring a first horizontal coordinate, a first vertical coordinate and a first vertical coordinate of the second target pixel point in the target first spherical image;
according to the fact that the first abscissa is equal to the negative number of cosine values of the abscissa in the first warp-weft image, the abscissa of the second target pixel point corresponding to the first warp-weft image is obtained;
according to the first ordinate being equal to the negative of the product of the sine value of the abscissa and the cosine value of the ordinate in the first warp-weft map, obtaining the ordinate of the second target pixel point corresponding to the first warp-weft map; or
Obtaining the ordinate of the second target pixel point corresponding to the first warp-weft image according to the fact that the first vertical coordinate is equal to the product of the sine value of the abscissa and the sine value of the ordinate in the first warp-weft image;
obtaining the first warp-weft map based on the abscissa of the second target pixel point corresponding to the first warp-weft map and the ordinate of the second target pixel point corresponding to the first warp-weft map.
8. The utility model provides a binocular depth perception device which characterized in that is applied to the aircraft, the aircraft includes first camera and second camera, wherein, first camera with there is the region of looking altogether for the second camera, the device includes:
the camera image acquisition module is used for acquiring the image of the common-view area acquired by the first camera as a first image and acquiring the image of the common-view area acquired by the second camera as a second image;
the longitude and latitude map obtaining module is used for performing spherical projection and expansion on the first image to obtain a first longitude and latitude map and performing spherical projection and expansion on the second image to obtain a second longitude and latitude map;
a longitude disparity map obtaining module for obtaining a longitude disparity map according to the first longitude and latitude map and the second longitude and latitude map;
and the depth perception module is used for converting the longitude disparity map into a depth map and obtaining the depth of the common view area based on the depth map.
9. An electronic device, comprising:
one or more processors;
a memory;
one or more applications, wherein the one or more applications are stored in the memory and configured to be executed by the one or more processors, the one or more programs configured to perform the method of any of claims 1-7.
10. A computer-readable storage medium, having stored thereon program code that can be invoked by a processor to perform the method according to any one of claims 1 to 7.
Priority Applications (1)
| Application Number | Priority Date | Filing Date | Title |
|---|---|---|---|
| CN202211700282.7A CN115937293A (en) | 2022-12-28 | 2022-12-28 | Binocular depth perception method and device, electronic equipment and storage medium |
Applications Claiming Priority (1)
| Application Number | Priority Date | Filing Date | Title |
|---|---|---|---|
| CN202211700282.7A CN115937293A (en) | 2022-12-28 | 2022-12-28 | Binocular depth perception method and device, electronic equipment and storage medium |
Publications (1)
| Publication Number | Publication Date |
|---|---|
| CN115937293A true CN115937293A (en) | 2023-04-07 |
Family
ID=86650776
Family Applications (1)
| Application Number | Title | Priority Date | Filing Date |
|---|---|---|---|
| CN202211700282.7A Pending CN115937293A (en) | 2022-12-28 | 2022-12-28 | Binocular depth perception method and device, electronic equipment and storage medium |
Country Status (1)
| Country | Link |
|---|---|
| CN (1) | CN115937293A (en) |
Cited By (1)
| Publication number | Priority date | Publication date | Assignee | Title |
|---|---|---|---|---|
| CN116563186A (en) * | 2023-05-12 | 2023-08-08 | 中山大学 | Real-time panoramic sensing system and method based on special AI sensing chip |
-
2022
- 2022-12-28 CN CN202211700282.7A patent/CN115937293A/en active Pending
Cited By (1)
| Publication number | Priority date | Publication date | Assignee | Title |
|---|---|---|---|---|
| CN116563186A (en) * | 2023-05-12 | 2023-08-08 | 中山大学 | Real-time panoramic sensing system and method based on special AI sensing chip |
Similar Documents
| Publication | Publication Date | Title |
|---|---|---|
| CN110582798B (en) | System and method for virtual enhanced vision simultaneous localization and mapping | |
| CN106529495B (en) | Obstacle detection method and device for aircraft | |
| CN107113376B (en) | A kind of image processing method, device and video camera | |
| US11748906B2 (en) | Gaze point calculation method, apparatus and device | |
| JP4825971B2 (en) | Distance calculation device, distance calculation method, structure analysis device, and structure analysis method. | |
| EP3167430A1 (en) | Camera calibration | |
| WO2020133172A1 (en) | Image processing method, apparatus, and computer readable storage medium | |
| WO2019155335A1 (en) | Unmanned aerial vehicle including an omnidirectional depth sensing and obstacle avoidance aerial system and method of operating same | |
| CN112052788B (en) | Binocular vision-based environment sensing method and device and unmanned aerial vehicle | |
| JP2010530086A (en) | Imaging model and image processing apparatus | |
| WO2019005644A1 (en) | A wearable eye tracking system with slippage detection and correction | |
| CN113366491A (en) | Eyeball tracking method, device and storage medium | |
| EP4057626A1 (en) | Stereo camera apparatus having wide field of view, and depth image processing method using same | |
| CN115937293A (en) | Binocular depth perception method and device, electronic equipment and storage medium | |
| CN111680758B (en) | Image training sample generation method and device | |
| JP2023505891A (en) | Methods for measuring environmental topography | |
| CN115933718A (en) | Unmanned aerial vehicle autonomous flight technical method integrating panoramic SLAM and target recognition | |
| KR102372298B1 (en) | Method for acquiring distance to at least one object located in omni-direction of vehicle and vision device using the same | |
| KR102361133B1 (en) | Method for acquiring distance to at least one object located in omni-direction of vehicle and vision device using the same | |
| WO2020019175A1 (en) | Image processing method and apparatus, and photographing device and unmanned aerial vehicle | |
| CN115131433A (en) | Non-cooperative target pose processing method and device and electronic equipment | |
| CN116443256A (en) | Methods, systems, and computer program products for airborne fueling | |
| CN116612459B (en) | Target detection method, target detection device, electronic equipment and storage medium | |
| WO2023130465A1 (en) | Aerial vehicle, image processing method and apparatus, and movable platform | |
| CN113888401B (en) | Image conversion method and device |
Legal Events
| Date | Code | Title | Description |
|---|---|---|---|
| PB01 | Publication | ||
| PB01 | Publication | ||
| SE01 | Entry into force of request for substantive examination | ||
| SE01 | Entry into force of request for substantive examination |