CN117516594A - Binocular camera ranging self-correction method, system and storage medium - Google Patents

Binocular camera ranging self-correction method, system and storage medium Download PDF

Info

Publication number
CN117516594A
CN117516594A CN202410020632.8A CN202410020632A CN117516594A CN 117516594 A CN117516594 A CN 117516594A CN 202410020632 A CN202410020632 A CN 202410020632A CN 117516594 A CN117516594 A CN 117516594A
Authority
CN
China
Prior art keywords
parallax
binocular camera
value
fitting
ground
Prior art date
Legal status (The legal status is an assumption and is not a legal conclusion. Google has not performed a legal analysis and makes no representation as to the accuracy of the status listed.)
Granted
Application number
CN202410020632.8A
Other languages
Chinese (zh)
Other versions
CN117516594B (en
Inventor
范欣
郑继川
季苏徽
Current Assignee (The listed assignees may be inaccurate. Google has not performed a legal analysis and makes no representation or warranty as to the accuracy of the list.)
Yuanxiang Technology Suzhou Co ltd
Metoak Technology Beijing Co ltd
Original Assignee
Yuanxiang Technology Suzhou Co ltd
Metoak Technology Beijing Co ltd
Priority date (The priority date is an assumption and is not a legal conclusion. Google has not performed a legal analysis and makes no representation as to the accuracy of the date listed.)
Filing date
Publication date
Application filed by Yuanxiang Technology Suzhou Co ltd, Metoak Technology Beijing Co ltd filed Critical Yuanxiang Technology Suzhou Co ltd
Priority to CN202410184528.2A priority Critical patent/CN118010067A/en
Priority to CN202410020632.8A priority patent/CN117516594B/en
Publication of CN117516594A publication Critical patent/CN117516594A/en
Application granted granted Critical
Publication of CN117516594B publication Critical patent/CN117516594B/en
Active legal-status Critical Current
Anticipated expiration legal-status Critical

Links

Classifications

    • G—PHYSICS
    • G01—MEASURING; TESTING
    • G01C—MEASURING DISTANCES, LEVELS OR BEARINGS; SURVEYING; NAVIGATION; GYROSCOPIC INSTRUMENTS; PHOTOGRAMMETRY OR VIDEOGRAMMETRY
    • G01C25/00—Manufacturing, calibrating, cleaning, or repairing instruments or devices referred to in the other groups of this subclass
    • G—PHYSICS
    • G06—COMPUTING; CALCULATING OR COUNTING
    • G06T—IMAGE DATA PROCESSING OR GENERATION, IN GENERAL
    • G06T7/00—Image analysis
    • G06T7/50—Depth or shape recovery
    • G06T7/55—Depth or shape recovery from multiple images
    • G—PHYSICS
    • G06—COMPUTING; CALCULATING OR COUNTING
    • G06T—IMAGE DATA PROCESSING OR GENERATION, IN GENERAL
    • G06T7/00—Image analysis
    • G06T7/60—Analysis of geometric attributes
    • G—PHYSICS
    • G06—COMPUTING; CALCULATING OR COUNTING
    • G06T—IMAGE DATA PROCESSING OR GENERATION, IN GENERAL
    • G06T2207/00—Indexing scheme for image analysis or image enhancement
    • G06T2207/20—Special algorithmic details
    • G06T2207/20228—Disparity calculation for image-based rendering

Landscapes

  • Engineering & Computer Science (AREA)
  • Physics & Mathematics (AREA)
  • General Physics & Mathematics (AREA)
  • Computer Vision & Pattern Recognition (AREA)
  • Theoretical Computer Science (AREA)
  • Manufacturing & Machinery (AREA)
  • Radar, Positioning & Navigation (AREA)
  • Remote Sensing (AREA)
  • Geometry (AREA)
  • Image Processing (AREA)

Abstract

The application discloses a binocular camera ranging self-correction method, a system and a storage medium, wherein the method comprises the following steps: step 1: acquiring a sampling image of the current sampling moment of the binocular camera based on a preset sampling period; step 2: calculating a state value of the binocular camera based on the sampled image, wherein the state value includes a height of the binocular camera from the ground; step 3: judging whether the state value meets the correction condition, if so, calculating a parallax compensation value of the binocular camera based on the corresponding fitting ground in the sampled image, and if not, re-executing the step 1; wherein the correction condition is that the relative difference between the state value and the initial state value is greater than the state threshold. Through the technical scheme in this application, select for use binocular camera self true measurable physical parameter as the correction basis, need not in specific occasion, utilize specific target to correct the binocular camera that installs, reduced the operation degree of difficulty of binocular camera self-correction.

Description

Binocular camera ranging self-correction method, system and storage medium
Technical Field
The present application relates to the technical field of binocular vision, and in particular, to a binocular camera ranging self-correction method, a binocular camera ranging self-correction system, and a computer readable storage medium.
Background
In an actual application scene of the binocular camera after delivery to a customer, the ranging accuracy of the binocular camera may be degraded due to the influence of vibration, environmental temperature change and the like. In order to ensure that the range accuracy of the binocular camera can be kept in an acceptable range continuously, the health status self-monitoring and self-correction of the binocular camera range measurement are required.
The binocular camera ranging correction method generally uses the difference between the measured value of a binocular camera on a specific target and the true value of the distance from the binocular camera to the specific target as the basis of ranging monitoring, and adjusts the measured value of the distance with larger error to an acceptable range by correcting the internal parameter or the external parameter of the binocular camera. However, since it is difficult to find a suitable specific target as a reference for the true value of the distance in an actual natural working scene, a target (such as a checkerboard) specially designed is often adopted as a specific target for the binocular camera correction.
In the prior art, because the binocular camera is already installed on equipment such as a robot or a vehicle, in the self-correction process of the binocular camera, a specific target is required to be placed at least one fixed position, and the operation in the calibration process is troublesome, so that a customer can even need to detach the binocular camera to return to a factory for calibration, and reinstallation is performed after the calibration is completed, and the operation is time-consuming and labor-consuming.
Disclosure of Invention
The purpose of the present application is: how to take physical quantity in actual natural working scene as correction reference basis, to realize the self-correction of range finding for the binocular camera, reduce the operation degree of difficulty of self-correction of the binocular camera.
The technical scheme of the first aspect of the application is that: there is provided a binocular camera ranging self-correcting method, the method comprising: step 1: acquiring a sampling image of the current sampling moment of the binocular camera based on a preset sampling period; step 2: calculating a state value of the binocular camera based on the sampled image, wherein the state value includes a height of the binocular camera from the ground; step 3: judging whether the state value meets the correction condition, if so, calculating a parallax compensation value of the binocular camera based on the corresponding fitting ground in the sampled image, and if not, re-executing the step 1; wherein the correction condition is that the relative difference between the state value and the initial state value is greater than the state threshold.
In some embodiments, in step 3, the parallax compensation value of the binocular camera is calculated based on the corresponding fitting ground in the sampled image, specifically including: determining optical center coordinates based on the resolution of the sampled image; calculating a fitting height based on the optical center coordinates and the fitting ground; when the fluctuation of the values of the fitting heights in the preset correction period is less than or equal to the fluctuation threshold value, calculating a height average value of the fitting heights, and calculating a parallax compensation value corresponding to the height average value based on a parallax calculation principle.
In some embodiments, the method further comprises: determining a fitting ground based on the sampled image, wherein the determining the fitting ground specifically comprises: based on the sampling image, obtaining a parallax image of the binocular camera; selecting effective parallax points in the parallax map row by row, and generating a first histogram based on the selected effective parallax points of each row; when the ratio of the effective parallax points in the preset range in the first histogram is larger than or equal to a first preset proportion, calculating the row optimal parallax values in a mean value solving mode based on the effective parallax points, wherein the preset range is determined by the median of the first histogram; and calculating the fitting ground by a fitting mode based on the calculated optimal parallax value of each row.
In some embodiments, before calculating the line best disparity value by means of averaging, the method further comprises: and deleting the effective parallax points outside the preset range, and calculating the row optimal parallax value by means of averaging based on the remaining effective parallax points.
In some embodiments, prior to computing the fitted ground, further comprising: and judging whether the ratio of the number of the optimal parallax values of the lines to the number of the lines of the parallax images is larger than or equal to a second preset ratio, if so, calculating a fitting ground, and if not, acquiring a sampling image again based on a preset sampling period.
In some embodiments, the method further comprises: determining a fitting ground based on the sampled image, wherein the determining the fitting ground specifically comprises: step 41: based on the sampling image, obtaining a parallax image of the binocular camera; step 42: sequentially acquiring first-class parallax and second-class parallax in the parallax map by taking the upper end and the lower end of the parallax map as starting positions, and sequentially alternately arranging the acquired first-class parallax and second-class parallax to form a target parallax map, wherein the first-class parallax is an odd-numbered line in the parallax map, and the second-class parallax is an even-numbered line in the parallax map; step 43: selecting effective parallax points in the target parallax map line by line, and generating a second histogram based on the selected effective parallax points of the current line; step 44: judging whether the duty ratio of the effective parallax point in the preset range in the second histogram is greater than or equal to a third preset proportion, if so, calculating a row optimal parallax value by means of averaging based on the effective parallax point in the preset range, adding one to the row effective value count, and if not, re-executing step 43, and selecting the effective parallax point in the next row of target parallax map, wherein the preset range is determined by the median of the second histogram; step 45: judging whether the value of the row effective value count is greater than or equal to a first row number threshold value, if so, calculating a fitting ground based on the row optimal parallax value in a fitting mode, and if not, re-executing step 43, and selecting effective parallax points in the next row of target parallax images.
In some embodiments, the method further comprises: when the line number of the effective parallax point line corresponding to the first type of line parallax and the line number of the effective parallax point line corresponding to the second type of line parallax are both larger than or equal to a second line number threshold, calculating a first fitting plane and a second fitting plane in a fitting mode based on the line best parallax value of the effective parallax point line corresponding to the first type of line parallax and the line best parallax value of the effective parallax point line corresponding to the second type of line parallax respectively; calculating a plane included angle between the first fitting plane and the second fitting plane; judging whether the plane included angle is larger than or equal to an included angle threshold value, if so, re-executing the step 1, and if not, executing the step 45.
In some embodiments, the binocular camera is mounted on a mobile platform, the method further comprising: when it is determined that the scene in the sample image is static, acquisition of the sample image is stopped.
The technical scheme of the second aspect of the application is that: there is provided a binocular camera ranging self-correcting system, the system comprising: the sampling unit is configured to acquire a sampling image of the current sampling moment of the binocular camera based on a preset sampling period; a calculation unit configured to calculate a state value of the binocular camera based on the sampled image, wherein the state value includes a height of the binocular camera from the ground; the parallax compensation unit is configured to judge whether the state value meets the correction condition, and if so, the parallax compensation value of the binocular camera is calculated based on the corresponding fitting ground in the sampled image; wherein the correction condition is that the relative difference between the state value and the initial state value is greater than the state threshold.
The technical scheme of the third aspect of the application is that: there is provided a readable storage medium having stored therein a computer program which, when executed by a processor, implements a method according to any of the above-mentioned aspects.
The beneficial effects of this application are: according to the technical scheme, the real measurable physical parameters of the binocular camera are selected as the correction basis, if the height of the binocular camera from the ground calculated through the image in the current state and/or the included angle between the optical axis of the binocular camera and the ground and the initial state value are larger in difference, the binocular camera is self-corrected, the mounted binocular camera is not required to be corrected in a specific occasion by using a specific target, and the operation difficulty of the self-correction of the binocular camera is reduced.
In the ground fitting process, in order to improve the operation rate and reduce the occupation of calculation force on the premise of ensuring the fitting accuracy, the rapid ground fitting operation is performed from the upper end and the lower end of the parallax map according to the parity of the number of lines.
Drawings
The advantages of the foregoing and/or additional aspects of the present application will become apparent and readily appreciated from the description of the embodiments, taken in conjunction with the accompanying drawings, wherein:
FIG. 1 is a schematic flow chart of a binocular camera ranging self-correction method according to one embodiment of the present application;
FIG. 2 is a schematic diagram of a binocular camera ranging self-correcting scene according to one embodiment of the present application;
FIG. 3 is a schematic diagram of a camera coordinate system according to one embodiment of the present application;
FIG. 4 is a schematic view of a ramp scene according to one embodiment of the present application;
fig. 5 is a schematic block diagram of a binocular camera ranging self-correction system according to one embodiment of the present application.
Detailed Description
In order that the above-recited objects, features and advantages of the present application will be more clearly understood, a more particular description of the application will be rendered by reference to the appended drawings and appended detailed description. It should be noted that, without conflict, the embodiments of the present application and features of the embodiments may be combined with each other.
In the following description, numerous specific details are set forth in order to provide a thorough understanding of the present application, however, the present application may be practiced otherwise than as described herein, and thus the scope of the present application is not limited to the specific embodiments disclosed below.
Examples
As shown in fig. 1, the present embodiment provides a binocular camera ranging self-correction method, which includes.
Step 1: acquiring a sampling image of the current sampling moment of the binocular camera based on a preset sampling period; step 2: calculating a state value of the binocular camera based on the sampled image, wherein the state value includes a height of the binocular camera from the ground; step 3: judging whether the state value meets the correction condition, if so, calculating a parallax compensation value of the binocular camera based on the corresponding fitting ground in the sampled image, and if not, re-executing the step 1; wherein the correction condition is that the relative difference (difference absolute value) between the state value and the initial state value is greater than the state threshold.
Specifically, taking an example that the binocular camera 100 is mounted on the robot 200, the position of the binocular camera 100 on the robot 200 is set to be relatively fixed, as shown in fig. 2, the field angle 101 of the binocular camera 100 can observe the ground 300, the center area of the field angle 101 of the binocular camera is selected as the ROI region of interest 102, the height of the binocular camera from the ground is H, the included angle between the optical axis of the binocular camera and the ground is α, and the distance measurement value of the binocular camera is Z.
The state that the binocular camera 100 is mounted on the robot 200 and calibrated is set as an initial state, at this time, the binocular camera 100 can accurately realize ranging, and based on the ground image acquired by the binocular camera, the height of the binocular camera from the ground in the initial state can be calculated and recorded as an initial height H 0 。
For more visual and easy understanding of the basis of binocular camera correction, the embodiment of the application selects the initial height H 0 As an initial state value of the range finding accuracy of the binocular camera, namely a correction index, if it is determined that the relative difference value (difference absolute value) between the state value of the binocular camera (the height of the binocular camera from the ground) and the initial state value is greater than the state threshold value in the current state, it indicates that the binocular camera is distorted, and it is necessary to correct the range finding of the binocular camera, and correct the binocular camera to the initial state or to the range allowed by the range finding error, so as to ensure the range finding accuracy of the binocular camera. The state value of the binocular camera may include the height of the binocular camera from the ground (fitting the ground), and the state threshold is a height threshold, and the size of the state threshold may be set manually according to the actual situation.
It should be noted that, the state value of the binocular camera may further include an included angle between the optical axis of the binocular camera and the ground (the fitting ground), and the state threshold may further include an angle threshold correspondingly.
In the embodiment of the present application, the right camera of the binocular camera is set as the reference camera, and the coordinate system of the camera is set as the world coordinate system, as shown in fig. 3, the plane in which the binocular camera 100 is located is the plane XOY, which is a known plane.
The height of the binocular camera from the ground will now be described as a state value.
When the binocular camera 100 obtains a sampled image including the ground and a corresponding parallax map, ground fitting can be performed to obtain a fitted ground, and an included angle (first included angle) between the fitted ground and the plane where the binocular camera is located can be calculated by combining the plane where the binocular camera 100 is located, namely a plane XOY, wherein the included angle (first included angle) is a complementary angle of an included angle (second included angle) between an optical axis of the binocular camera and the ground, and the height (state value) of the binocular camera in the current state can be calculated through a parallax principle and a trigonometric function relation.
Therefore, if the height H of the binocular camera from the ground in the current state 1 Height H of binocular camera in initial state 0 When the relative difference Δh between them is greater than the height threshold (state threshold), then the binocular camera correction is required.
In the correction process, the calculated height H of the binocular camera in the current state from the ground can be calculated by means of parallax compensation 1 Correcting to an initial state or correcting to an error allowable range, wherein a calculation formula involved in the correction process comprises the following steps:
α 1 =90°-β
wherein Z is 1 Is a distance measurement value, D is a parallax value, B is a lens base line length of the binocular camera, F is a focal length of the binocular camera, deltaD is a parallax compensation value, H 1 The height of the binocular camera from the ground in the current state is beta, the included angle (first included angle) between the plane of the binocular camera and the fitting ground in the current state is alpha 1 Is the complement angle (second included angle) of the included angle (first included angle) between the plane where the binocular camera is positioned and the fitting ground in the current state, alpha 0 Is the complement angle of the included angle between the plane where the binocular camera is positioned and the fitting ground in the initial state.
Therefore, the self-correction scheme in the embodiment of the application does not need to perform self-correction on the binocular camera 100 installed on the robot 200 on a specific occasion by using a specific target, and does not need to disassemble the binocular camera to return to a factory, so that the self-correction of the range measurement of the binocular camera by taking the physical quantity in an actual natural working scene as a correction reference basis is realized, and the operation difficulty of the self-correction of the binocular camera is reduced.
When the calculated relative difference is determined to be greater than the state threshold by a certain proportion, such as 20%, the data is abnormal and should be discarded.
On the basis of the above embodiment 1, in order to accurately fit the current ground plane and further accurately perform ranging correction on the binocular camera, the ranging self-correction method of the binocular camera in the embodiment of the present application further includes: determining a fitting ground based on the sampled image, wherein the determining the fitting ground specifically comprises: based on the sampling image, obtaining a parallax image of the binocular camera;
selecting effective parallax points in the parallax map row by row, and generating a first histogram, namely a histogram of intra-row parallax, based on parallax values of the selected effective parallax points of each row; when the ratio of the effective parallax points in the preset range in the first histogram is larger than or equal to a first preset proportion, calculating the row optimal parallax values in a mean value solving mode based on the effective parallax points, wherein the preset range is determined by the median of the first histogram; and calculating a fitting ground by a fitting mode based on the row optimal parallax value.
Specifically, the right camera in the binocular camera is set as a reference camera, the coordinate system of the camera is taken as a world coordinate system, as shown in fig. 3, a fitting plane equation of the ground can be determined, and the corresponding calculation formula is as follows:
A L x+B L y+C L z+D L =0
wherein A is L 、B L 、C L 、D L Fitting parameters of a plane to the ground, and (x, y, z) are three-dimensional coordinates of any point in the plane.
Since the position of the binocular camera 100 on the robot 200 is set to be relatively fixed in the embodiment of the present application, i.e., notThe range error due to both positional changes is taken into account. Therefore, the parallax value between any one line in the ground horizontal direction (parallel to the x-axis) and the binocular camera 100 is equal, and the fitting parameter a in the plane equation is fitted L Does not change, so that the fitting parameter A can be set in the process of calculating the fitting plane of the ground L Set to 0.
To sum up, in order to fit a more accurate plane where the ground is located, the effective parallax points in the parallax map can be counted row by row, the positions (screen coordinates) of the rows where the effective parallax points are located and the parallax values are respectively converted into y coordinates and z coordinates in the above-mentioned fit ground equation by means of parallax transformation, and the fitting parameters B of the fit plane can be determined by combining a least square method or a plane fitting mode L 、C L And D L And further determining a fitting plane of the ground.
In any disparity map, a certain amount of disparity holes, that is, disparity points with a disparity value of 0, are included, so in the embodiment of the present application, the disparity points with a disparity value in the disparity map are marked as effective disparity points, and the corresponding disparity values are marked as effective disparity values. In order to ensure that a more accurate plane on which the ground is located can be obtained, each row of the parallax map should be ensured to contain a certain number of effective parallax points with more concentrated value distribution, so that the concept of a histogram is introduced.
And (3) making a histogram based on the parallax values of the effective parallax points of each row in the parallax map, recording the histogram as a first histogram, and counting the occurrence times of each parallax value, so that if the first histogram shows an effect similar to Gaussian distribution, the pixels of the image row corresponding to the current parallax row are basically in the same horizontal plane.
In the embodiment of the application, the median of the first histogram is selected to be + -10% as a preset range, the duty ratio of the effective parallax point in the range in the first histogram is calculated, and when the duty ratio is greater than or equal to a preset ratio set by the user, for example, 50%, the pixels of the image row corresponding to the current parallax row are considered to be basically in the same horizontal plane and the value distribution is more concentrated, so that the method can be used for calculating the fitting ground; otherwise, it needs to be discarded.
Setting doubleThe resolution of the sampled image obtained by the eye camera is 1920×1080, and the corresponding parallax map is 1920×1080, that is, each line corresponds to 1920 parallax points, and the total lines are 1080. Therefore, in order to reduce the data volume, a concept of line best disparity value is introduced, that is, a line best disparity value (that is, the average value of the effective data after removing the discrete points) is calculated by combining filtering and denoising with averaging for each line, and is converted into corresponding (y, z) coordinates, and then based on the (maximum) 1080 (y, z) coordinates of the composition, a parameter B corresponding to the fitting ground is calculated by a fitting mode (such as linear fitting) L 、C L And D L 。
In some embodiments, to further reduce the data size while ensuring the accuracy of the calculated line best disparity value, the method further includes, before calculating the line best disparity value by averaging, retaining the most effective image depth information: deleting effective parallax points outside a preset range, and calculating a row optimal parallax value by means of averaging based on the remaining effective parallax points; or calculating the row optimal parallax value by means of averaging based on the effective parallax points in the preset range.
In some embodiments, further to improve the accuracy of the fitting ground, it is further necessary to determine the validity of the disparity map in the "row dimension", and therefore, before calculating the fitting ground, further includes: and judging whether the ratio of the number of the line optimal parallax values to the number of the parallax image lines (the number of the sampling image lines) is larger than or equal to a second preset ratio, if so, calculating a fitting ground, and if not, acquiring a sampling image again based on a preset sampling period.
In particular, in an ideal case, each line in the disparity map corresponds to a line optimal disparity value, but in a real environment, such as a situation that the ground is not horizontal, there is obstacle shielding, the indoor scene ground is limited in proportion, light, shadow and other factors, and there is no line optimal disparity value corresponding to the current line, so when the ratio of the number of the line optimal disparity values to the number of the lines of the disparity map is greater than or equal to a second preset ratio (such as 40%), the current disparity map is considered to correspond to a ground with a large enough area, and the line optimal disparity values can be used for fitting a ground plane; otherwise, the ground area is considered too small and needs to be discarded.
On the basis of the above embodiment 1, by analyzing the data of the plane where the fitted ground is located, when the parallax data at the upper and lower ends in the image can be fitted into a plane, the middle data is highly probable on the plane, and when the ground is fitted from the two ends, the obtained fitted ground is relatively stable relative to the ground fitted from the middle, so in the process of determining the plane where the relatively accurate ground is located based on the image, in order to improve the operation rate and reduce the occupation of calculation force, the binocular camera ranging self-correction method in the embodiment of the present application performs rapid ground fitting operation according to the parity of the number of rows from the upper and lower ends of the parallax image in the process of determining the fitted ground based on the sampling image, and specifically comprises:
step 41: based on the sampling image, obtaining a parallax image of the binocular camera;
step 42: sequentially acquiring first-class parallax and second-class parallax in the parallax map by taking the upper end and the lower end of the parallax map as starting positions, and sequentially alternately arranging the acquired first-class parallax and second-class parallax to form a target parallax map, wherein the first-class parallax is an odd-numbered line in the parallax map, and the second-class parallax is an even-numbered line in the parallax map;
step 43: selecting effective parallax points in the target parallax map line by line, and generating a second histogram based on the selected effective parallax points of the current line;
step 44: judging whether the duty ratio of the effective parallax point in the preset range in the second histogram is greater than or equal to a third preset proportion,
if so, calculating the row optimal parallax value by means of averaging based on the effective parallax points in the preset range, and adding one to the row effective value count,
if not, re-executing step 43, selecting the effective parallax point in the next row of target parallax map, wherein the preset range is determined by the median of the second histogram;
step 45: judging whether the value of the row effective value count is greater than or equal to a first row number threshold value,
if so, calculating a fitting ground based on the row optimal parallax value in a fitting mode,
if not, step 43 is re-executed, and the effective parallax point in the next line target parallax map is selected, where the value of the first line number threshold is determined by the percentage of the line number of the sampled image, and may be 5% -25%.
Specifically, in the process of fitting a ground plane, the embodiment of the present application uses a manner of fast calculating the parity of the number of rows from the upper end and the lower end of the disparity map to reconstruct the disparity map, and uses a resolution of 1920×1080 as an example, and the row number of the reconstructed target disparity map is shown in table 1.
TABLE 1
Original line number 1 1080 3 1078 5 1076 ...
Current line number 1 2 3 4 5 6 ...
I.e. sequentially selecting the 1 st row, 1080 th row, 3 rd row and 1078 th row … in the disparity map, and so on, and recombining the target disparity map according to the sequence of top-down, odd-first and even-first. By means of the mode that the target parallax images are formed through alternate arrangement and recombination, effective data in the parallax images can be fully utilized, repeated operation is avoided, the data at two ends are utilized to fit planes where the ground is located, and the fact that the ground is accurately fitted under the condition that the selected data are fewer can be guaranteed.
In any disparity map, a certain amount of disparity holes, that is, disparity points with a disparity value of 0, are included, so in the embodiment of the present application, the disparity points with a disparity value in the disparity map are marked as effective disparity points, and the corresponding disparity values are marked as effective disparity values. In order to ensure that a more accurate plane on which the ground is located can be obtained, it should be ensured that the disparity map is sufficiently clear, i.e. each row contains a certain number of effective disparity points which are distributed and sufficiently concentrated, so that the concept of a histogram is introduced.
And (3) making a histogram based on the parallax value of each line of effective parallax points in the target parallax map, recording the histogram as a second histogram, and counting the occurrence times of each parallax value, so that if the first histogram shows an effect similar to Gaussian distribution, the pixels of the image line corresponding to the current parallax line are basically in the same horizontal plane.
In the embodiment of the application, 10% of the median of the second histogram is selected as a preset range, the duty ratio of an effective parallax point in the range in the second histogram is calculated, when the duty ratio is greater than or equal to three preset ratios, such as 50%, the parallax map of the line is considered to be clear enough, and most parallax data are basically equal in value, namely pixels of the image line corresponding to the current parallax line are basically in the same horizontal plane, and the method can be used for calculating a fitting ground, and the line effective value count is increased by one; otherwise, it is considered unclear and needs to be discarded, and step 43 is performed again, and the valid parallax point in the next line of target parallax map is selected.
Similarly, in order to ensure that the disparity map used for fitting the ground plane is sufficiently clear, the accuracy of fitting the ground is improved, and the validity of the disparity map in the "row dimension" is also required to be judged, so that the row valid value is counted, and the number of sufficiently clear disparity rows selected in the target disparity map is counted.
It should be noted that, each sufficiently clear parallax line in the target parallax map corresponds to a line best parallax value calculated by means of averaging based on the effective parallax points in the line.
Therefore, when the value of the line effective value count is greater than or equal to the first line number threshold (for example, 200 lines), it is considered that enough clear parallax lines are acquired currently, the acquired parallax lines are distributed at the upper end and the lower end of the original parallax map, and then the fitting ground is calculated in a fitting mode based on the line optimal parallax value calculated previously.
If the row valid value count is smaller than the first row number threshold, step 43 is re-executed, and a valid parallax point in the next row of the target parallax map is selected. If all the line parallaxes in the target parallax image are calculated, the value of the line effective value count still does not reach the first line number threshold value, the target parallax image and the corresponding original parallax image are discarded, and the sampling image is acquired again.
Even if the disparity map is discarded, the occupied computing power is basically equivalent to that of performing ground fitting calculation on the whole original disparity map.
In some embodiments, considering that there may be a slope (up slope, down slope) in the real scene, when the robot is located near the slope, especially for a binocular camera with a larger field of view, if the ground fitting is directly performed, a slope will be forcibly fitted, and if the binocular camera performs self-correction, a phenomenon of miscorrection will occur, so on the basis of the above-mentioned quick ground fitting from the upper and lower ends of the disparity map according to the parity of the number of lines, the method further includes:
when the number of lines of the effective parallax point lines corresponding to the first type of line parallaxes and the number of lines of the effective parallax point lines corresponding to the second type of line parallaxes are both larger than or equal to a second line number threshold, wherein the second line number threshold is smaller than the first line number threshold, and the value of the second line number threshold can be 10% -40% of the first line number threshold if the second line number threshold is the first line number threshold;
calculating a first fitting plane and a second fitting plane in a fitting mode based on the line optimal parallax value of the effective parallax point line corresponding to the first type of line parallax and the line optimal parallax value of the effective parallax point line corresponding to the second type of line parallax respectively;
calculating a plane included angle between the first fitting plane and the second fitting plane;
judging whether the plane included angle is larger than or equal to an included angle threshold, wherein the size of the included angle threshold can be set manually according to actual requirements;
if yes, discarding the current data, and re-executing the step 1 to obtain a sampling image; if not, go to step 45.
As shown in fig. 4, ground fitting is performed separately through the upper and lower parts of the parallax map, if a certain included angle exists on the fitted ground and is greater than or equal to a set included angle threshold, the ground in front is considered to be non-planar (such as a slope), at this time, binocular camera ranging correction is not performed based on the current sampling image, the current data is discarded, step 1 is performed again, sampling images are obtained, and correction errors are avoided.
In some embodiments, in order to ensure accuracy of data in a calculation process and avoid oscillation in a correction process, an embodiment of the present application selects an optical center coordinate of a binocular camera as a reference for calculating a height and an angle in a current state, and introduces an idea of mean value calculation, and in the step 3, a parallax compensation value of the binocular camera is calculated based on a fitting ground corresponding to a sampled image, which specifically includes:
determining optical center coordinates based on the resolution of the sampled image;
calculating the fitting height of the binocular camera from the ground based on the optical center coordinates and the fitting ground;
when the fluctuation of the values of the fitting heights in the preset correction period is smaller than or equal to the fluctuation threshold value, calculating a height average value of the fitting heights, and calculating a parallax compensation value corresponding to the height average value based on a parallax calculation principle, wherein the preset correction period is multiple times of the preset sampling period.
Specifically, the center position of the sampling image, namely 1/2 of the resolution, is selected as the optical center coordinate of the binocular camera, the height of the binocular camera from the ground in the current state is calculated by taking the optical center coordinate as a reference, and the calculated height is recorded as the fitting height, and the specific calculation process is not repeated.
Thus, for each calculated fitting height, the fitting height relative to the initial height H can be calculated 0 Difference between the initial height H and the initial height H 0 As a judgment as to whether the robot (binocular camera) is in a bumpy state.
As will be appreciated by those skilled in the art, upon setting the position of the binocular camera 100 on the robot 200 to be relatively fixed, the ranging accuracy of the binocular camera does not change drastically even if the binocular camera is distorted. Therefore, when the binocular camera needs to be corrected, the ranging value of the binocular camera is necessarily changed within a certain range, and therefore, when the ranging value of the binocular camera is greatly changed, the mounting carrier of the binocular camera can be estimated to be in a bumpy state.
When the binocular camera is judged to be corrected, based on a preset correction period, a plurality of sampling images are acquired, each sampling image can calculate corresponding fitting heights, and when the fluctuation of the values of the fitting heights in the preset correction period is judged to be smaller than or equal to a fluctuation threshold value, the binocular camera is distorted and is required to be corrected.
At this time, the height average value of the plurality of fitting heights is used as a basis, a parallax compensation value corresponding to the height average value is calculated based on a parallax calculation principle, and the height average value is corrected to an initial state.
It should be noted that the preset sampling period may include a plurality of preset correction periods, and the time interval between the two may be manually set according to the actual requirement. For example, the preset sampling period is set to 1 hour, and the preset correction period is set to 5 minutes.
In some embodiments, the binocular camera is mounted on a mobile platform, the method further comprising: when the scene in the sampling image is judged to be static, the sampling image is stopped to be acquired, so that the influence of large data occupation in the static scene on the final parallax compensation result when the height average value of the fitting height is calculated is avoided.
As shown in fig. 5, the present embodiment provides a binocular camera ranging self-correcting system 500, the system 500 comprising:
the sampling unit 501, the sampling unit 501 is configured to obtain a sampling image of the current sampling moment of the binocular camera based on a preset sampling period; a calculation unit 502, the calculation unit 502 being configured to calculate a state value of the binocular camera based on the sampled image, wherein the state value comprises a height of the binocular camera from the ground; the parallax compensation unit 503, the parallax compensation unit 503 is configured to determine whether the state value satisfies the correction condition, if yes, calculate the parallax compensation value of the binocular camera based on the corresponding fitting ground in the sampled image, if no, the sampled image is re-acquired by the sampling unit 501; wherein the correction condition is that the relative difference (difference absolute value) between the state value and the initial state value is greater than the state threshold.
Specifically, taking an example in which the binocular camera 100 is mounted on the robot 200, the position of the binocular camera 100 on the robot 200 is set to be relatively fixed, as shown in fig. 2, the field angle 101 of the binocular camera 100 can observe the ground 300, and the center region of the field angle 101 of the binocular camera is selected as the ROI region of interest 102.
The state that the binocular camera 100 is mounted on the robot 200 and calibrated is set as an initial state, at this time, the binocular camera 100 can accurately realize ranging, and based on the ground image acquired by the binocular camera, the height of the binocular camera from the ground in the initial state can be calculated and recorded as an initial height H 0 。
For more visual and easy understanding of the basis of binocular camera correction, the embodiment of the application selects the initial height H 0 As a binocular camera ranging accuracyIf it is determined that the relative difference (absolute difference) between the state value of the binocular camera (the height of the binocular camera from the ground) and the initial state value is greater than the state threshold in the current state, it indicates that the binocular camera is distorted, and the binocular camera needs to be subjected to ranging correction, and the binocular camera is corrected to the initial state or to the range allowed by the ranging error, so as to ensure the ranging accuracy of the binocular camera. The state value of the binocular camera may include the height of the binocular camera from the ground (fitting the ground), and the state threshold is a height threshold, and the size of the state threshold may be set manually according to the actual situation.
It should be noted that, the state value of the binocular camera may further include an included angle between the optical axis of the binocular camera and the ground (the fitting ground), and the state threshold may further include an angle threshold correspondingly.
In the embodiment of the present application, the right camera of the binocular camera is set as the reference camera, and the coordinate system of the camera is set as the world coordinate system, as shown in fig. 3, the plane in which the binocular camera 100 is located is the plane XOY, which is a known plane.
The height of the binocular camera from the ground will now be described as a state value.
When the binocular camera 100 obtains a sampled image including the ground and a corresponding parallax map, ground fitting can be performed to obtain a fitted ground, and an included angle (first included angle) between the fitted ground and the plane where the binocular camera is located can be calculated by combining the plane where the binocular camera 100 is located, namely a plane XOY, wherein the included angle (first included angle) is a complementary angle of an included angle (second included angle) between an optical axis of the binocular camera and the ground, and the height (state value) of the binocular camera in the current state can be calculated through a parallax principle and a trigonometric function relation.
Therefore, if the height H of the binocular camera from the ground in the current state 1 Height H of binocular camera in initial state 0 When the relative difference Δh between them is greater than the height threshold (state threshold), then the binocular camera needs to be corrected.
In the correction process, the parallax compensation mode can be adopted to calculateHeight H of binocular camera from ground in current state 1 Correcting to an initial state or correcting to an error allowable range, wherein a calculation formula involved in the correction process comprises the following steps:
α 1 =90°-β
wherein Z is 1 Is a distance measurement value, D is a parallax value, B is a lens base line length of the binocular camera, F is a focal length of the binocular camera, deltaD is a parallax compensation value, H 1 The height of the binocular camera from the ground in the current state is beta, the included angle (first included angle) between the plane of the binocular camera and the fitting ground in the current state is alpha 1 Is the complement angle (second included angle) of the included angle (first included angle) between the plane where the binocular camera is positioned and the fitting ground in the current state, alpha 0 Is the complement angle of the included angle between the plane where the binocular camera is positioned and the fitting ground in the initial state.
Therefore, the self-correction scheme in the embodiment of the application does not need to perform self-correction on the binocular camera 100 installed on the robot 200 on a specific occasion by using a specific target, and does not need to disassemble the binocular camera to return to a factory, so that the self-correction of the range measurement of the binocular camera by taking the physical quantity in an actual natural working scene as a correction reference basis is realized, and the operation difficulty of the self-correction of the binocular camera is reduced.
When the calculated relative difference is determined to be greater than the state threshold by a certain proportion, the data is indicated to be abnormal and should be discarded.
The present application further provides a computer readable storage medium having a computer program stored therein, which when executed by a processor, implements the method steps of any of the embodiments described above.
In particular, the computer program includes computer instructions stored in a computer readable storage medium; the processor of the computer device reads the computer instructions from the computer-readable storage medium, and the processor executes the computer instructions, so that the computer device performs the method steps in any of the embodiments described above.
Thus, various embodiments of the present application have been described in detail. In order to avoid obscuring the concepts of the present application, some details known in the art are not described. How to implement the solutions disclosed herein will be fully apparent to those skilled in the art from the above description.
Although some specific embodiments of the present application have been described in detail by way of example, it should be understood by those skilled in the art that the above examples are for illustration only and are not intended to limit the scope of the present application.
The steps in the present application may be sequentially adjusted, combined, and pruned according to actual requirements.
Although the present application is disclosed in detail with reference to the accompanying drawings, it is to be understood that such descriptions are merely illustrative and are not intended to limit the application of the present application. The scope of the present application is defined by the appended claims and may include various modifications, alterations, and equivalents to the invention without departing from the scope and spirit of the application.

Claims (10)

1. A binocular camera ranging self-correction method, the method comprising:
step 1: acquiring a sampling image of the current sampling moment of the binocular camera based on a preset sampling period;
step 2: calculating a state value of the binocular camera based on the sampled image, wherein the state value includes a height of the binocular camera from the ground;
step 3: judging whether the state value meets a correction condition, if so, calculating a parallax compensation value of the binocular camera based on the corresponding fitting ground in the sampling image, and if not, re-executing the step 1;
wherein the correction condition is that a relative difference between the state value and an initial state value is greater than a state threshold.
2. The method according to claim 1, wherein in the step 3, the calculating the parallax offset value of the binocular camera based on the corresponding fitting ground in the sampled image specifically includes:
determining optical center coordinates based on the resolution of the sampled image;
calculating a fitting height based on the optical center coordinates and the fitting ground;
when the fluctuation of the values of the fitting heights in the preset correction period is smaller than or equal to a fluctuation threshold value, calculating a height average value of the fitting heights, and calculating the parallax compensation value corresponding to the height average value based on a parallax calculation principle.
3. The binocular camera ranging self-correcting method of claim 1, further comprising:
determining the fitting ground based on the sampled image, wherein the determining the fitting ground specifically comprises:
acquiring a parallax map of the binocular camera based on the sampling image;
selecting effective parallax points in the parallax map line by line, and generating a first histogram based on the selected effective parallax points of each line;
when the ratio of the effective parallax points in the preset range in the first histogram is larger than or equal to a first preset proportion, calculating a row optimal parallax value in a mean value mode based on the effective parallax points, wherein the preset range is determined by the median of the first histogram;
and calculating the fitting ground in a fitting mode based on the calculated row optimal parallax value of each row.
4. A binocular camera ranging self-correcting method according to claim 3, wherein before calculating the line best disparity value by means of averaging, further comprising:
and deleting the effective parallax points outside the preset range, and calculating the row optimal parallax value by means of averaging based on the remaining effective parallax points.
5. The binocular camera ranging self-correcting method of claim 3, further comprising, prior to the computing the fitted ground:
judging whether the ratio of the number of the optimal parallax values to the number of the parallax image lines is larger than or equal to a second preset ratio, if so, calculating the fitting ground,
and if not, acquiring the sampling image again based on the preset sampling period.
6. The binocular camera ranging self-correcting method of claim 1, further comprising:
determining the fitting ground based on the sampled image, wherein the determining the fitting ground specifically comprises:
step 41: acquiring a parallax map of the binocular camera based on the sampling image;
step 42: sequentially acquiring a first-class parallax and a second-class parallax in the parallax map by taking the upper end and the lower end of the parallax map as starting positions respectively, sequentially alternately arranging the acquired first-class parallax and the acquired second-class parallax to form a target parallax map,
the first-type row parallaxes are odd rows in the parallax map, and the second-type row parallaxes are even rows in the parallax map;
step 43: selecting effective parallax points in the target parallax map line by line, and generating a second histogram based on the selected effective parallax points of the current line;
step 44: judging whether the duty ratio of the effective parallax points in the preset range in the second histogram is larger than or equal to a third preset proportion,
if so, calculating the row optimal parallax value by means of averaging based on the effective parallax points in the preset range, and adding one to the row effective value count,
if not, the step 43 is re-executed, the valid disparity points in the target disparity map of the next line are selected,
wherein the preset range is determined by the median of the second histogram;
step 45: judging whether the value of the row effective value count is larger than or equal to a first row number threshold value,
if so, calculating the fitting ground by a fitting mode based on the row optimal parallax value,
if not, the step 43 is re-executed, and the effective parallax point in the target parallax map of the next row is selected.
7. The binocular camera ranging self-correcting method of claim 6, further comprising:
when the number of the lines of the effective parallax point lines corresponding to the first type of line parallaxes and the number of the lines of the effective parallax point lines corresponding to the second type of line parallaxes are both larger than or equal to a second line number threshold value,
calculating a first fitting plane and a second fitting plane in a fitting mode based on the line optimal parallax value of the effective parallax point line corresponding to the first type of line parallax and the line optimal parallax value of the effective parallax point line corresponding to the second type of line parallax respectively;
calculating a plane included angle between the first fitting plane and the second fitting plane;
judging whether the plane included angle is larger than or equal to an included angle threshold value,
if yes, the step 1 is executed again,
if not, the step 45 is performed.
8. The binocular camera ranging self-correcting method of claim 1, wherein the binocular camera is mounted on a mobile platform, the method further comprising:
and stopping acquiring the sampling image when the scene in the sampling image is judged to be static.
9. A binocular camera ranging self-correcting system, the system comprising:
the sampling unit is configured to acquire a sampling image of the current sampling moment of the binocular camera based on a preset sampling period;
a computing unit configured to compute a state value of the binocular camera based on the sampled image, wherein the state value includes a height of the binocular camera from a ground surface;
the parallax compensation unit is configured to judge whether the state value meets a correction condition, and if so, calculates a parallax compensation value of the binocular camera based on the corresponding fitting ground in the sampling image;
wherein the correction condition is that a relative difference between the state value and an initial state value is greater than a state threshold.
10. A computer readable storage medium, characterized in that the computer readable storage medium has stored therein a computer program which, when executed by a processor, implements the method according to any of claims 1 to 8.
CN202410020632.8A 2024-01-08 2024-01-08 Binocular camera ranging self-correction method, system and storage medium Active CN117516594B (en)

Priority Applications (2)

Application Number Priority Date Filing Date Title
CN202410184528.2A CN118010067A (en) 2024-01-08 2024-01-08 Binocular camera ranging self-correction method and system
CN202410020632.8A CN117516594B (en) 2024-01-08 2024-01-08 Binocular camera ranging self-correction method, system and storage medium

Applications Claiming Priority (1)

Application Number Priority Date Filing Date Title
CN202410020632.8A CN117516594B (en) 2024-01-08 2024-01-08 Binocular camera ranging self-correction method, system and storage medium

Related Child Applications (1)

Application Number Title Priority Date Filing Date
CN202410184528.2A Division CN118010067A (en) 2024-01-08 2024-01-08 Binocular camera ranging self-correction method and system

Publications (2)

Publication Number Publication Date
CN117516594A true CN117516594A (en) 2024-02-06
CN117516594B CN117516594B (en) 2024-03-22

Family

ID=89742443

Family Applications (2)

Application Number Title Priority Date Filing Date
CN202410020632.8A Active CN117516594B (en) 2024-01-08 2024-01-08 Binocular camera ranging self-correction method, system and storage medium
CN202410184528.2A Pending CN118010067A (en) 2024-01-08 2024-01-08 Binocular camera ranging self-correction method and system

Family Applications After (1)

Application Number Title Priority Date Filing Date
CN202410184528.2A Pending CN118010067A (en) 2024-01-08 2024-01-08 Binocular camera ranging self-correction method and system

Country Status (1)

Country Link
CN (2) CN117516594B (en)

Citations (7)

* Cited by examiner, † Cited by third party
Publication number Priority date Publication date Assignee Title
US20200128225A1 (en) * 2018-10-23 2020-04-23 Xi'an Jiaotong University Depth Information Acquisition Method and Device
CN111862234A (en) * 2020-07-22 2020-10-30 中国科学院上海微系统与信息技术研究所 Binocular camera self-calibration method and system
CN112330754A (en) * 2020-11-25 2021-02-05 上海欧菲智能车联科技有限公司 Binocular vision correction method and device, electronic equipment and storage medium
CN113554714A (en) * 2021-07-22 2021-10-26 元橡科技(苏州)有限公司 Vehicle-mounted binocular stereo camera self-correcting method and system thereof, and FPGA chip
WO2022037633A1 (en) * 2020-08-19 2022-02-24 展讯通信（上海）有限公司 Calibration method and apparatus for binocular camera, image correction method and apparatus for binocular camera, storage medium, terminal and intelligent device
CN116996657A (en) * 2023-08-11 2023-11-03 元橡科技(苏州)有限公司 Binocular camera parallax correction method suitable for mowing robot
CN117061719A (en) * 2023-08-11 2023-11-14 元橡科技（北京）有限公司 Parallax correction method for vehicle-mounted binocular camera

Patent Citations (7)

* Cited by examiner, † Cited by third party
Publication number Priority date Publication date Assignee Title
US20200128225A1 (en) * 2018-10-23 2020-04-23 Xi'an Jiaotong University Depth Information Acquisition Method and Device
CN111862234A (en) * 2020-07-22 2020-10-30 中国科学院上海微系统与信息技术研究所 Binocular camera self-calibration method and system
WO2022037633A1 (en) * 2020-08-19 2022-02-24 展讯通信（上海）有限公司 Calibration method and apparatus for binocular camera, image correction method and apparatus for binocular camera, storage medium, terminal and intelligent device
CN112330754A (en) * 2020-11-25 2021-02-05 上海欧菲智能车联科技有限公司 Binocular vision correction method and device, electronic equipment and storage medium
CN113554714A (en) * 2021-07-22 2021-10-26 元橡科技(苏州)有限公司 Vehicle-mounted binocular stereo camera self-correcting method and system thereof, and FPGA chip
CN116996657A (en) * 2023-08-11 2023-11-03 元橡科技(苏州)有限公司 Binocular camera parallax correction method suitable for mowing robot
CN117061719A (en) * 2023-08-11 2023-11-14 元橡科技（北京）有限公司 Parallax correction method for vehicle-mounted binocular camera

Non-Patent Citations (1)

* Cited by examiner, † Cited by third party
Title
周前飞;刘晶红;居波;李刚;: "面阵CCD航空相机斜视图像的几何校正", 液晶与显示, no. 03, 15 June 2015 (2015-06-15) *

Also Published As

Publication number Publication date
CN117516594B (en) 2024-03-22
CN118010067A (en) 2024-05-10

Similar Documents

Publication Publication Date Title
CN110969662B (en) Method and device for calibrating internal parameters of fish-eye camera, calibration device controller and system
CN110163918B (en) Line structure cursor positioning method based on projective geometry
US20100103259A1 (en) Object Distance Deriving Device
CN112070845A (en) Calibration method and device of binocular camera and terminal equipment
IL269984B (en) Ladar range estimate with range rate compensation
CN110763306B (en) Monocular vision-based liquid level measurement system and method
CN112381847B (en) Pipeline end space pose measurement method and system
CN110349257B (en) Phase pseudo mapping-based binocular measurement missing point cloud interpolation method
CN112422960B (en) Offset estimation method and device of camera module, storage medium and terminal
CN103124334B (en) A kind of method of lens distortion calibration
CN112070657B (en) Image processing method, device, system, equipment and computer storage medium
CN109272555A (en) A kind of external parameter of RGB-D camera obtains and scaling method
CN117516594B (en) Binocular camera ranging self-correction method, system and storage medium
CN111649669A (en) Non-calibration linear structured light vision sensor coordinate measuring method and system
US11445162B2 (en) Method and device for calibrating binocular camera
CN114792342A (en) Line structure light positioning method, device, equipment and storage medium
CN111839580B (en) Dental film image generation method, device, electronic equipment and storage medium
CN112529792B (en) Distortion correction method for distortion-free model camera
Wu et al. Nonmetric calibration of camera lens distortion using concentric circles pattern
CN114739321B (en) Structural light striation correction method for precise grating measurement
CN112881999B (en) Semi-automatic calibration method for multi-line laser radar and vision sensor
US20170124688A1 (en) Image processor, image processing method, and measuring apparatus
CN117576226A (en) Method and system for improving parameter-free sub-pixel calibration of low-cost image equipment
CN117011349B (en) Line laser image optimization method, system and medium based on two-dimensional motion platform
CN115704737B (en) Method and related device for measuring shaking amount of projection picture of projector

Legal Events

Date Code Title Description
PB01 Publication
PB01 Publication
SE01 Entry into force of request for substantive examination
SE01 Entry into force of request for substantive examination
GR01 Patent grant
GR01 Patent grant