CN114926654A - Polar line correction method and device - Google Patents

Abstract

The invention provides an epipolar line correction method and device, wherein the method comprises the following steps: acquiring a first image and a second image based on a binocular vision system; determining a plurality of first characteristic points in the first image and a plurality of second characteristic points in the second image based on the gray value of the pixel point of the image; matching the plurality of first characteristic points with the plurality of second characteristic points to obtain a plurality of initial matching point pairs; determining a target basis matrix based on a normalization eight-point algorithm, a random consistency algorithm and a plurality of initial matching point pairs; determining a first pole and a second pole based on the target basis matrix and epipolar constraint, and determining a first projection matrix and a second projection matrix based on the center point of the first image, the first pole and the second pole; and performing projection transformation on the first characteristic points and the second characteristic points based on the first projection matrix and the second projection matrix to obtain a corrected image. The invention improves the accuracy and efficiency of epipolar line correction.

Description

Polar line correction method and device

Technical Field

The invention relates to the technical field of image matching, in particular to an epipolar line correction method and device.

Background

The matching of binocular stereo vision is a great hot direction of research at present, the binocular stereo vision has wide practical application, and most matching algorithms of targets are based on the hypothesis that: the characteristic points corresponding to the two images obtained by the binocular stereo vision system are on the same scanning line. Therefore, the matching search range is reduced from two dimensions to one dimension, and the matching speed and the matching precision are improved. However, in reality, due to the influence of the camera structure and the manufacturing process, it is difficult to ensure that the main axes of the cameras are parallel, so that the binocular systems are not parallel any more. Therefore, a method of correcting the binocular system to be parallel, that is, epipolar line correction, is presented.

The epipolar line correction method in the prior art is mostly based on a projective transformation matrix, and the method is a method completely depending on a basic matrix, and has the following technical problems: the method depends excessively on the precision of the basic matrix, and when mismatching exists, the epipolar line correction result is inaccurate, the image has larger image distortion, and the matching precision is reduced.

Therefore, it is urgently needed to provide an epipolar line correction method and device for improving the accuracy of the basic matrix, thereby improving the epipolar line correction accuracy.

Disclosure of Invention

In view of the above, it is necessary to provide an epipolar line correction method and apparatus for solving the technical problem of low accuracy of epipolar line correction in the prior art when the matching accuracy of the basic matrix is not high.

In one aspect, the present invention provides an epipolar line correction method, including:

acquiring a first image and a second image based on a binocular vision system;

determining a plurality of first characteristic points in the first image and a plurality of second characteristic points in the second image based on gray values of image pixel points;

matching the plurality of first characteristic points with the plurality of second characteristic points to obtain a plurality of initial matching point pairs;

determining a target basis matrix based on a normalized eight-point algorithm, a random consistency algorithm and the initial matching point pairs;

determining a first pole and a second pole based on the target basis matrix and epipolar constraints, and determining a first projection matrix and a second projection matrix based on a center point of the first image, the first pole, and the second pole;

and performing projection transformation on the first characteristic points and the second characteristic points based on the first projection matrix and the second projection matrix to obtain a corrected image.

In some possible implementations, the determining a plurality of first feature points in the first image and a plurality of second feature points in the second image based on gray-scale values of pixels of the image includes:

performing smoothing processing on the first image and the second image to obtain a first smooth image and a second smooth image correspondingly;

performing multi-scale decomposition on the first smooth image and the second smooth image to obtain a first Gaussian pyramid and a second Gaussian pyramid correspondingly;

dividing the first Gaussian pyramid and the second Gaussian pyramid based on a preset division rule to correspondingly obtain a plurality of first sub image blocks and a plurality of second sub image blocks;

and constructing a local sliding window according to the first sub image block or the second sub image block, determining the plurality of first characteristic points according to the local sliding window and the plurality of first sub image blocks, and determining the plurality of second characteristic points according to the local sliding window and the plurality of second sub image blocks.

In some possible implementation manners, the dividing the first gaussian pyramid based on a preset dividing rule to obtain a plurality of first sub image blocks includes:

dividing each first image layer in the plurality of first image layers to obtain a plurality of first image areas;

and dividing each first image area in the plurality of first image areas to obtain a plurality of first sub image blocks.

In some possible implementations, the plurality of first image layers includes a current first image layer and a first adjacent image layer and a second adjacent image layer adjacent to the current first image layer, and the plurality of first sub image blocks includes a current first sub image block and a plurality of adjacent first sub image blocks adjacent to the current first sub image block; the determining the first feature points according to the local sliding window and the first sub image blocks includes:

acquiring the original image gray scale of the current first sub image block in the local sliding window;

moving the local sliding window to traverse the current first sub image block, and acquiring the current image gray scale of the current first sub image block in the moved local sliding window;

determining a gray difference sum based on the original image gray and the current image gray;

when the gray difference sum is larger than or equal to a difference threshold value, taking the pixel point with the largest gray value in the current first sub-image block as a key point;

determining a first adjacent point in the first adjacent image layer corresponding to the key point and a second adjacent point in the second adjacent image layer corresponding to the key point;

and when the gray value of the key point is greater than the gray value of the first adjacent point, the gray value of the key point is greater than the gray value of the second adjacent point, and the gray value of the key point is greater than the gray value of each pixel point in the plurality of adjacent first sub-image blocks, the key point is the first feature point.

In some possible implementations, the matching the plurality of first feature points and the plurality of second feature points to obtain a plurality of initial matching point pairs includes:

determining Euclidean distances between each first feature point in the plurality of first feature points and each second feature point in the plurality of second feature points;

and judging whether the Euclidean distance is greater than a preset distance, and if so, taking the first characteristic point and the second characteristic point as the initial matching point pair.

In some possible implementations, the determining a target basis matrix based on the normalized eight-point algorithm, the random consistency algorithm, and the plurality of initial matching point pairs includes:

step one, randomly determining eight first initial matching point pairs from the plurality of initial matching point pairs, and determining an initial basic matrix and an epipolar line according to the eight first initial matching point pairs;

determining the distance between a plurality of second initial matching point pairs except the first initial matching point pair and the polar line in the plurality of pairs of initial matching points;

step three, judging whether the distance is greater than a threshold distance, when the distance is less than the threshold distance, the second initial matching point pair is an inner point, and when the distance is greater than or equal to the threshold distance, the second initial matching point pair is an outer point;

and step four, judging whether the ratio of the number of the inner points to the number of the initial matching point pairs is greater than a threshold ratio and whether the iteration times are greater than the maximum iteration times, if the ratio of the number of the inner points to the number of the initial matching point pairs is greater than the threshold ratio or the iteration times are greater than the maximum iteration times, the initial basis matrix is a target basis matrix, and if the ratio of the number of the inner points to the number of the initial matching point pairs is less than or equal to the threshold ratio and the iteration times are less than or equal to the maximum iteration times, repeating the step one to the step four.

In some possible implementations, the determining a first projection matrix and a second projection matrix based on the center point, the first pole, and the second pole of the first image includes:

performing homogeneous transformation and simplification on the central point of the first image to obtain an initial translation transformation matrix;

performing rotation mapping on the first image and the second image to correspondingly obtain a first rotation matrix and a second rotation matrix;

projecting the first pole and the second pole to a horizontal infinite point, and correspondingly obtaining a first standard projective transformation matrix and a second standard projective transformation matrix;

performing translation transformation on the first image and the second image to obtain a target translation transformation matrix;

determining the first projection matrix based on the initial translation transformation matrix, the first rotation matrix, the first standard projection transformation matrix, and the target translation transformation matrix;

determining the second projection matrix based on the initial translation transformation matrix, the second rotation matrix, and the second standard projective transformation matrix.

In some possible implementations, the epipolar line correction method further includes:

optimizing the first projection matrix and the second projection matrix based on an optimization model to obtain a first optimized projection matrix and a second optimized projection matrix;

performing projection transformation on the first feature points and the second feature points based on the first projection matrix and the second projection matrix to obtain a corrected image, specifically:

and performing projection transformation on the first characteristic point and the second characteristic point based on the first optimized projection matrix and the second optimized projection matrix to obtain a corrected image.

In some possible implementations, the initial matching point pair includes a first matching point and a second matching point, and the optimization model is:

in the formula, S is an optimization target; h ₀ Is a first projection matrix; h ₁ Is a second projection matrix; u. of _0i Is a coordinate value of the first matching point; u. of _1i The coordinate value of the second matching point; and N is the number of the initial matching point pairs.

In another aspect, the present invention also provides an epipolar line correction device, including:

an image acquisition unit for acquiring a first image and a second image based on a binocular vision system;

the characteristic point determining unit is used for determining a plurality of first characteristic points in the first image and a plurality of second characteristic points in the second image based on gray values of image pixel points;

a matching point determining unit, configured to match the plurality of first feature points and the plurality of second feature points to obtain a plurality of initial matching point pairs;

a basic matrix determining unit, configured to determine a target basic matrix based on a normalized eight-point algorithm, a random consistency algorithm, and the plurality of initial matching point pairs;

a projection matrix determination unit for determining a first pole and a second pole based on the target basis matrix and epipolar constraints, and determining a first projection matrix and a second projection matrix based on a center point of the first image, the first pole, and the second pole;

and the image correction unit is used for carrying out projection transformation on the first characteristic points and the second characteristic points based on the first projection matrix and the second projection matrix to obtain a corrected image.

The beneficial effects of adopting the above embodiment are: according to the epipolar line correction method provided by the invention, the target base matrix is determined by setting the normalization eight-point algorithm, the random consistency algorithm and the plurality of initial matching point pairs, so that mismatching points in the plurality of initial matching point pairs can be eliminated, the precision of the obtained target base matrix is improved, and the accuracy of epipolar line correction is improved. And moreover, the random consistency algorithm is a random process and has robustness, a high-precision target base matrix can be estimated from a plurality of initial matching point pairs, and the accuracy of epipolar line correction can be further improved.

Furthermore, the method and the device can improve the acquisition speed of the first characteristic points and the second characteristic points by determining the plurality of first characteristic points in the first image and the plurality of second characteristic points in the second image based on the gray value of the image pixel points, thereby improving the epipolar line correction efficiency.

Furthermore, the target basic matrix is determined based on the normalization eight-point algorithm, and the normalization eight-point algorithm is linear solving, so that the operation speed is high, the efficiency of obtaining the target basic matrix can be improved, and the epipolar line correction efficiency is improved.

Drawings

In order to more clearly illustrate the technical solutions in the embodiments of the present invention, the drawings needed to be used in the description of the embodiments will be briefly introduced below, and it is obvious that the drawings in the following description are only some embodiments of the present invention, and it is obvious for those skilled in the art to obtain other drawings based on these drawings without creative efforts.

FIG. 1 is a schematic flow chart of an exemplary epipolar rectification method according to the present invention;

FIG. 2 is a schematic flow chart diagram illustrating one embodiment of S102 of FIG. 1 according to the present invention;

FIG. 3 is a schematic flow chart of one embodiment of S203 of FIG. 2;

FIG. 4 is a flowchart illustrating an embodiment of S204 in FIG. 2;

FIG. 5 is a flowchart illustrating one embodiment of S103 of FIG. 1 according to the present invention;

FIG. 6 is a flowchart illustrating an embodiment of S104 of FIG. 1 according to the present invention;

FIG. 7 is a flowchart illustrating an embodiment of S105 of FIG. 1 according to the present invention;

FIG. 8 is a schematic structural diagram of an exemplary epipolar line rectification device according to the present invention;

fig. 9 is a schematic structural diagram of an embodiment of an electronic device provided in the present invention.

Detailed Description

The technical solutions in the embodiments of the present invention will be clearly and completely described below with reference to the drawings in the embodiments of the present invention. It is to be understood that the described embodiments are merely exemplary of the invention, and not restrictive of the full scope of the invention. All other embodiments, which can be derived by a person skilled in the art from the embodiments given herein without making any creative effort, shall fall within the protection scope of the present invention.

It should be understood that the schematic drawings are not necessarily to scale. The flowcharts used in this disclosure illustrate operations implemented according to some embodiments of the present invention. It should be understood that the operations of the flow diagrams may be performed out of order, and that steps without logical context may be reversed in order or performed concurrently. One skilled in the art, under the direction of this summary, may add one or more other operations to, or remove one or more operations from, the flowchart.

Some of the block diagrams shown in the figures are functional entities and do not necessarily correspond to physically or logically separate entities. These functional entities may be implemented in the form of software, or in one or more hardware modules or integrated circuits, or in different networks and/or processor systems and/or microcontroller systems.

Reference herein to "an embodiment" means that a particular feature, structure, or characteristic described in connection with the embodiment can be included in at least one embodiment of the invention. The appearances of the phrase in various places in the specification are not necessarily all referring to the same embodiment, nor are separate or alternative embodiments mutually exclusive of other embodiments. It is explicitly and implicitly understood by one skilled in the art that the embodiments described herein may be combined with other embodiments.

The embodiments of the present invention provide an epipolar line calibration method and apparatus, which are described below.

Fig. 1 is a schematic flow chart of an embodiment of an epipolar line rectification method provided by the present invention, and as shown in fig. 1, the epipolar line rectification method includes:

s101, acquiring a first image and a second image based on a binocular vision system;

s102, determining a plurality of first characteristic points in a first image and a plurality of second characteristic points in a second image based on the gray value of the pixel points of the image;

s103, matching the plurality of first characteristic points with the plurality of second characteristic points to obtain a plurality of initial matching point pairs;

s104, determining a target base matrix based on a normalization eight-point algorithm, a random consistency algorithm and a plurality of initial matching point pairs;

s105, determining a first pole and a second pole based on the target basis matrix and epipolar constraint, and determining a first projection matrix and a second projection matrix based on the central point of the first image, the first pole and the second pole;

and S106, performing projection transformation on the first characteristic points and the second characteristic points based on the first projection matrix and the second projection matrix to obtain a corrected image.

Compared with the prior art, the epipolar line correction method provided by the embodiment of the invention determines the target base matrix by setting the target base matrix based on the normalization eight-point algorithm, the random consistency algorithm and the plurality of initial matching point pairs, can eliminate mismatching points in the plurality of initial matching point pairs, improves the precision of the obtained target base matrix and further improves the accuracy of epipolar line correction. And moreover, the random consistency algorithm is a random process and has robustness, a high-precision target base matrix can be estimated from a plurality of initial matching point pairs, and the accuracy of epipolar line correction can be further improved.

Furthermore, according to the embodiment of the invention, the plurality of first characteristic points in the first image and the plurality of second characteristic points in the second image are determined based on the gray value of the pixel point of the image, so that the acquisition speed of the first characteristic points and the second characteristic points can be increased, and the epipolar line correction efficiency is improved.

Furthermore, the target basic matrix is determined based on the normalization eight-point algorithm, and the normalization eight-point algorithm is linear solving, so that the operation speed is high, the efficiency of obtaining the target basic matrix can be improved, and the epipolar line correction efficiency is improved.

In some embodiments of the present invention, as shown in fig. 2, step S102 comprises:

s201, smoothing the first image and the second image to correspondingly obtain a first smooth image and a second smooth image;

s202, performing multi-scale decomposition on the first smooth image and the second smooth image to correspondingly obtain a first Gaussian pyramid and a second Gaussian pyramid;

s203, dividing the first Gaussian pyramid and the second Gaussian pyramid based on a preset division rule to correspondingly obtain a plurality of first sub image blocks and a plurality of second sub image blocks;

s204, a local sliding window is constructed according to the first sub image block or the second sub image block, a plurality of first feature points are determined according to the local sliding window and the plurality of first sub image blocks, and a plurality of second feature points are determined according to the local sliding window and the plurality of second sub image blocks.

According to the embodiment of the invention, the first smooth image and the second smooth image are more stable by performing smoothing processing on the first image and the second image, and the robustness of the first smooth image and the second smooth image is improved, so that the robustness of the epipolar line correction method is improved.

Wherein, step S201 specifically includes: firstly, corroding and then expanding the first image and the second image by using a morphological open operation method, preliminarily filtering noise existing in the first image and the second image, and further filtering noise points in the first image and the second image by adopting Gaussian filtering to obtain a first smooth image and a second smooth image which are relatively stable.

Specifically, the gaussian filter function is:

wherein, G (x, y) is the pixel value of each pixel point in the first smooth image or the second smooth image; x is the abscissa of the pixel point in the first image or the second image; y is the ordinate of the pixel point in the first image or the second image; and sigma is the pixel standard deviation of each pixel point in the first image or the second image.

It should be noted that: the first Gaussian pyramid comprises a plurality of first image layers, the second Gaussian pyramid comprises a plurality of second image layers, the first image layers and the second image layers are different scale spaces caused by imitating the distance of the human sight, and the scales between the first image layers and the second image layers are consistent. Since the number of pixels in the image layers decreases as the scale increases, the first gaussian pyramid includes 6 first image layers, and the second gaussian pyramid includes 6 second image layers.

It should also be noted that: the local sliding window in step S204 includes, but is not limited to, a rectangular window, a gaussian window, and the like.

In some embodiments of the present invention, as shown in fig. 3, step S203 comprises:

s301, dividing each first image layer in the plurality of first image layers to obtain a plurality of first image areas;

and S302, dividing each first image area in the plurality of first image areas to obtain a plurality of first sub image blocks.

In an embodiment of the present invention, step S301 specifically includes: each first image layer was divided into 3 × 3 squared regions, and 9 first image regions were obtained. Step S302 specifically includes: and dividing each first image area into 4-by-4 small areas to obtain a plurality of first image blocks.

It should be understood that: the obtaining manner of the second sub image block is the same as that of the first sub image block, and see steps S301 to S302 for details, which are not described herein again.

In some embodiments of the present invention, the plurality of first image layers include a current first image layer and a first adjacent image layer and a second adjacent image layer adjacent to the current first image layer, and the plurality of first sub image blocks include a current first sub image block and a plurality of adjacent first sub image blocks adjacent to the current first sub image block, as shown in fig. 4, step S204 includes:

s401, obtaining the original image gray of the current first sub image block in the local sliding window;

s402, moving a local sliding window to traverse the current first sub image block, and acquiring the current image gray scale of the current first sub image block in the moved local sliding window;

s403, determining a gray difference sum based on the gray of the original image and the gray of the current image;

s404, when the sum of the gray differences is larger than or equal to the difference threshold, taking the pixel point with the largest gray value in the current first sub-image block as a key point;

s405, determining a first adjacent point corresponding to the key point in the first adjacent image layer and a second adjacent point corresponding to the key point in the second adjacent image layer;

s406, when the gray value of the key point is greater than the gray value of the first adjacent point, the gray value of the key point is greater than the gray value of the second adjacent point, and the gray value of the key point is greater than the gray value of each pixel point in the plurality of adjacent first sub-image blocks, the key point is a first feature point.

Specifically, the sum of the grayscale differences in step S403 is:

E(u,v)＝∑ _x,y w(x,y)[I(x+u,y+v)-I(x,y)] ²

wherein E (u, v) is the sum of the gray-scale differences; w (x, y) is a window function; (u, v) is the local sliding window movement amount; i (x + u, y + v) is the gray level of the current image; and I (x, y) is the gray scale of the original image.

It should be noted that: the difference threshold in step S404 may be adjusted or set according to an actual application scenario, which is not specifically limited herein.

It should be understood that: the obtaining method of the second feature point is the same as the obtaining method of the first feature point, and is described in detail in steps S401 to S406, which are not described herein again.

In some embodiments of the present invention, as shown in fig. 5, step S103 comprises:

s501, determining Euclidean distances between each first feature point in the first feature points and each second feature point in the second feature points;

s502, judging whether the Euclidean distance is larger than a preset distance or not, and when the Euclidean distance is larger than the preset distance, taking the first characteristic point and the second characteristic point as an initial matching point pair.

It should be noted that: the preset distance may be set or adjusted according to an actual application scenario, and is not specifically limited herein.

In some embodiments of the present invention, as shown in fig. 6, step S104 includes:

s601, randomly determining eight first initial matching point pairs from the plurality of initial matching point pairs, and determining an initial basic matrix and an epipolar line according to the eight first initial matching point pairs;

s602, determining the distances between a plurality of second initial matching point pairs except the first initial matching point pair and an epipolar line in the plurality of pairs of initial matching points;

s603, judging whether the distance is greater than the threshold distance, wherein when the distance is less than the threshold distance, the second initial matching point pair is an inner point, and when the distance is greater than or equal to the threshold distance, the second initial matching point pair is an outer point;

s604, judging whether the ratio of the number of the inner points to the number of the initial matching point pairs is larger than a threshold ratio and the iteration frequency is larger than the maximum iteration frequency, if the ratio of the number of the inner points to the number of the initial matching point pairs is larger than the threshold ratio or the iteration frequency is larger than the maximum iteration frequency, taking the initial basis matrix as a target basis matrix, and if the ratio of the number of the inner points to the number of the initial matching point pairs is smaller than or equal to the threshold ratio and the iteration frequency is smaller than or equal to the maximum iteration frequency, repeating the steps S601-S604.

It should be understood that: the threshold distance, the threshold ratio and the maximum iteration number can be set or adjusted according to the actual application scene, and in the specific embodiment of the invention, the threshold ratio is 95%.

In some embodiments of the present invention, the determining the first pole and the second pole based on the target basis matrix and the epipolar constraint in step S105 specifically includes:

from the epipolar constraint:

Fe ₀ ＝0 F ^T e ₁ ＝0

in the formula, F is a target basis matrix; e.g. of a cylinder ₀ Is the first pole, e ₁ The second pole.

The first pole and the second pole can be obtained according to the epipolar constraint and the target basis matrix.

In some embodiments of the present invention, as shown in fig. 7, step S105 includes:

s701, performing homogeneous transformation and simplification on the central point of the first image to obtain an initial translation transformation matrix;

s702, carrying out rotation mapping on the first image and the second image to correspondingly obtain a first rotation matrix and a second rotation matrix;

s703, projecting the first pole and the second pole to a horizontal infinite point, and correspondingly obtaining a first standard projective transformation matrix and a second standard projective transformation matrix;

s704, performing translation transformation on the first image and the second image to obtain a target translation transformation matrix;

s705, determining a first projection matrix based on the initial translation transformation matrix, the first rotation matrix, the first standard projection transformation matrix and the target translation transformation matrix;

and S706, determining a second projection matrix based on the initial translation transformation matrix, the second rotation matrix and the second standard projection transformation matrix.

Wherein, step S701 specifically includes: transforming the coordinates of the center point of the first image to (a) ₀ ，b ₀ 1) to avoid unnecessary trouble, will (a) ₀ ，b ₀ 1) to (0, 0, 1), the initial translation transformation matrix T is:

step S702 specifically includes: projecting the first pole and the second pole to infinity, firstly carrying out rotation mapping on the first image and the second image, wherein the rotation intersection bases are alpha respectively ₀ And alpha ₁ Let the first pole and the second pole fall on the x-axis, then the first rotation matrix R ₀ And a second rotation matrix R ₁ Respectively as follows:

first standard projective transformation matrix M in step S703 ₀ And a second standard projective transformation matrix M ₁ Respectively as follows:

in the formula, f ₁ The image perspective deformation amount of the first image is obtained; f. of ₂ The amount of image perspective distortion for the second image.

After step S703, because the polar lines are already parallel, but not yet on the same scanning line, and there is a vertical time difference, it is also necessary to perform translation transformation on the first image and the second image, and assuming that the moving component of the ordinate is l, the target translation transformation matrix in step S704 is:

the first projection matrix H in step S705 ₀ Comprises the following steps:

H ₀ ＝NM ₀ R ₀ T

the second projection matrix H in step S706 ₁ Comprises the following steps:

H ₁ ＝M ₁ R ₁ T

and correcting the first characteristic point and the second characteristic point through the first projection matrix and the second projection matrix to obtain a corrected image.

To further improve the correction accuracy, in some embodiments of the present invention, after step S105, the epipolar correction method further comprises:

optimizing the first projection matrix and the second projection matrix based on the optimization model to obtain a first optimized projection matrix and a second optimized projection matrix;

step S106 specifically includes: and performing projection transformation on the first characteristic points and the second characteristic points based on the first optimized projection matrix and the second optimized projection matrix to obtain a corrected image.

According to the embodiment of the invention, the first projection matrix and the second projection matrix are optimized, so that the precision of the first projection matrix and the second projection matrix can be further improved, and the precision of the obtained corrected image can be further improved.

In a specific embodiment of the present invention, the initial matching point pair includes a first matching point and a second matching point, and the optimization model is:

in the formula, S is an optimization target; h ₀ Is a first projection matrix; h ₁ Is a second projection matrix; u. u _0i The coordinate value of the first matching point; u. of _1i Is the coordinate value of the second matching point; and N is the number of the initial matching point pairs.

The specific optimization process comprises the following steps: and (3) minimizing the optimization target, namely S is 0, and solving the optimization model by using the constraint condition that the corrected matching points have the same vertical coordinate to obtain a first optimization projection matrix and a second optimization projection matrix.

In order to better implement the epipolar rectification method in the embodiment of the present invention, on the basis of the epipolar rectification method, correspondingly, the embodiment of the present invention further provides an epipolar rectification device, as shown in fig. 8, the epipolar rectification device 800 includes:

an image acquisition unit 801 for acquiring a first image and a second image based on a binocular vision system;

a feature point determining unit 802, configured to determine, based on a gray-scale value of an image pixel point, a plurality of first feature points in a first image and a plurality of second feature points in a second image;

a matching point determining unit 803, configured to match the plurality of first feature points and the plurality of second feature points to obtain a plurality of initial matching point pairs;

a basis matrix determining unit 804, configured to determine a target basis matrix based on a normalized eight-point algorithm, a random consistency algorithm, and a plurality of initial matching point pairs;

a projection matrix determination unit 805 configured to determine a first pole and a second pole based on the target basis matrix and the epipolar constraint, and determine a first projection matrix and a second projection matrix based on the center point of the first image, the first pole, and the second pole;

and an image correction unit 806, configured to perform projection transformation on the first feature point and the second feature point based on the first projection matrix and the second projection matrix to obtain a corrected image.

The epipolar line correction device 800 provided in the above-mentioned embodiment can implement the technical solutions described in the above-mentioned epipolar line correction method embodiments, and the specific implementation principle of each module or unit can refer to the corresponding contents in the above-mentioned epipolar line correction method embodiments, and will not be described herein again.

As shown in fig. 9, the present invention further provides an electronic device 900 accordingly. The electronic device 900 includes a processor 901, a memory 902, and a display 903. Fig. 9 shows only some of the components of the electronic device 900, but it is to be understood that not all of the shown components are required to be implemented, and that more or fewer components can be implemented instead.

The processor 901 may be a Central Processing Unit (CPU), a microprocessor or other data Processing chip in some embodiments, and is used for executing program codes stored in the memory 902 or Processing data, such as the epipolar rectification method in the present invention.

In some embodiments, processor 901 may be a single server or a group of servers. The server groups may be centralized or distributed. In some embodiments, the processor 901 may be local or remote. In some embodiments, processor 901 may be implemented in a cloud platform. In an embodiment, the cloud platform may include a private cloud, a public cloud, a hybrid cloud, a community cloud, a distributed cloud, an intra-house, a multi-cloud, and the like, or any combination thereof.

The storage 902 may be an internal storage unit of the electronic device 900 in some embodiments, such as a hard disk or memory of the electronic device 900. The memory 902 may also be an external storage device of the electronic device 900 in other embodiments, such as a plug-in hard disk, a Smart Media Card (SMC), a Secure Digital (SD) Card, a Flash memory Card (Flash Card), etc., provided on the electronic device 900.

Further, the memory 902 may also include both internal storage units and external storage devices of the electronic device 900. The memory 902 is used for storing application software and various data installed in the electronic device 900.

The display 903 may be an LED display, a liquid crystal display, a touch-sensitive liquid crystal display, an OLED (Organic Light-Emitting Diode) touch panel, or the like in some embodiments. The display 903 is used to display information at the electronic device 900 and to display a visual user interface. The components 901 and 903 of the electronic device 900 communicate with each other over a system bus.

In some embodiments of the present invention, when the processor 901 executes the epipolar rectification program in the memory 902, the following steps may be implemented:

acquiring a first image and a second image based on a binocular vision system;

determining a plurality of first characteristic points in the first image and a plurality of second characteristic points in the second image based on the gray value of the pixel point of the image;

matching the plurality of first characteristic points with the plurality of second characteristic points to obtain a plurality of initial matching point pairs;

determining a target basis matrix based on a normalization eight-point algorithm, a random consistency algorithm and a plurality of initial matching point pairs;

determining a first pole and a second pole based on the target basis matrix and epipolar constraint, and determining a first projection matrix and a second projection matrix based on the center point of the first image, the first pole and the second pole;

and performing projection transformation on the first characteristic points and the second characteristic points based on the first projection matrix and the second projection matrix to obtain a corrected image.

It should be understood that: the processor 901, when executing the epipolar line rectification program in the memory 902, may also implement other functions in addition to the above functions, which may be referred to in particular in the description of the corresponding method embodiments above.

Further, the type of the electronic device 900 mentioned in the embodiment of the present invention is not particularly limited, and the electronic device 900 may be a portable electronic device such as a mobile phone, a tablet computer, a Personal Digital Assistant (PDA), a wearable device, and a laptop computer (laptop). Exemplary embodiments of portable electronic devices include, but are not limited to, portable electronic devices that carry an IOS, android, microsoft, or other operating system. The portable electronic device may also be other portable electronic devices such as laptop computers (laptop) with touch sensitive surfaces (e.g., touch panels), etc. It should also be understood that in other embodiments of the present invention, the electronic device 900 may not be a portable electronic device, but may be a desktop computer having a touch-sensitive surface (e.g., a touch panel).

Accordingly, the present application also provides a computer-readable storage medium, which is used for storing a computer-readable program or instruction, and when the program or instruction is executed by a processor, the steps or functions in the epipolar line correction method provided by the above-mentioned method embodiments can be implemented.

Those skilled in the art will appreciate that all or part of the flow of the method implementing the above embodiments may be implemented by instructing relevant hardware (such as a processor, a controller, etc.) by a computer program, and the computer program may be stored in a computer readable storage medium. The computer readable storage medium is a magnetic disk, an optical disk, a read-only memory or a random access memory.

The epipolar line correction method and device provided by the present invention are described in detail above, and the principle and the implementation mode of the present invention are explained in the present document by applying specific examples, and the description of the above examples is only used to help understanding the method and the core idea of the present invention; meanwhile, for those skilled in the art, according to the idea of the present invention, there may be variations in the specific embodiments and the application scope, and in summary, the content of the present specification should not be construed as a limitation to the present invention.

Claims (10)

1. An epipolar rectification method, comprising:

acquiring a first image and a second image based on a binocular vision system;

determining a plurality of first characteristic points in the first image and a plurality of second characteristic points in the second image based on gray values of image pixel points;

matching the plurality of first characteristic points with the plurality of second characteristic points to obtain a plurality of initial matching point pairs;

determining a target basis matrix based on a normalized eight-point algorithm, a random consistency algorithm and the initial matching point pairs;

determining a first pole and a second pole based on the target basis matrix and an epipolar constraint, and determining a first projection matrix and a second projection matrix based on a center point of the first image, the first pole, and the second pole;

and performing projection transformation on the first characteristic points and the second characteristic points based on the first projection matrix and the second projection matrix to obtain a corrected image.

2. The epipolar rectification method of claim 1, wherein determining a plurality of first feature points in the first image and a plurality of second feature points in the second image based on image pixel gray scale values comprises:

performing smoothing processing on the first image and the second image to correspondingly obtain a first smooth image and a second smooth image;

performing multi-scale decomposition on the first smooth image and the second smooth image to obtain a first Gaussian pyramid and a second Gaussian pyramid correspondingly;

dividing the first Gaussian pyramid and the second Gaussian pyramid based on a preset division rule to correspondingly obtain a plurality of first sub image blocks and a plurality of second sub image blocks;

and constructing a local sliding window according to the first sub image block or the second sub image block, determining the plurality of first characteristic points according to the local sliding window and the plurality of first sub image blocks, and determining the plurality of second characteristic points according to the local sliding window and the plurality of second sub image blocks.

3. The epipolar rectification method of claim 2, wherein the first gaussian pyramid comprises a plurality of first image layers, and wherein the dividing the first gaussian pyramid based on a preset dividing rule to obtain a plurality of first sub-image blocks comprises:

dividing each first image layer in the plurality of first image layers to obtain a plurality of first image areas;

and dividing each first image area in the plurality of first image areas to obtain a plurality of first sub image blocks.

4. The epipolar rectification method of claim 3, wherein the plurality of first image layers includes a current first image layer and first and second adjacent image layers adjacent to the current first image layer, and the plurality of first sub-image blocks includes a current first sub-image block and a plurality of adjacent first sub-image blocks adjacent to the current first sub-image block; the determining the first feature points according to the local sliding window and the first sub image blocks includes:

acquiring the original image gray scale of the current first sub image block in the local sliding window;

moving the local sliding window to traverse the current first sub image block, and acquiring the current image gray scale of the current first sub image block in the moved local sliding window;

determining a gray difference sum based on the original image gray and the current image gray;

when the gray difference sum is larger than or equal to a difference threshold, taking the pixel point with the maximum gray value in the current first sub-image block as a key point;

determining a first adjacent point in the first adjacent image layer corresponding to the key point and a second adjacent point in the second adjacent image layer corresponding to the key point;

and when the gray value of the key point is greater than the gray value of the first adjacent point, the gray value of the key point is greater than the gray value of the second adjacent point, and the gray value of the key point is greater than the gray value of each pixel point in the plurality of adjacent first sub-image blocks, the key point is the first feature point.

5. The epipolar rectification method of claim 1, wherein said matching the plurality of first feature points and the plurality of second feature points to obtain a plurality of initial matched point pairs comprises:

determining Euclidean distances between each first feature point in the plurality of first feature points and each second feature point in the plurality of second feature points;

and judging whether the Euclidean distance is greater than a preset distance, and if so, taking the first characteristic point and the second characteristic point as the initial matching point pair.

6. The epipolar rectification method of claim 1, wherein determining a target basis matrix based on the normalized eight-point algorithm, the random consensus algorithm, and the plurality of initial matching point pairs comprises:

step one, randomly determining eight first initial matching point pairs from the plurality of initial matching point pairs, and determining an initial base matrix and an epipolar line according to the eight first initial matching point pairs;

determining the distance between a plurality of second initial matching point pairs out of the first initial matching point pairs in the plurality of pairs of initial matching points and the epipolar line;

step three, judging whether the distance is greater than a threshold distance, when the distance is less than the threshold distance, the second initial matching point pair is an inner point, and when the distance is greater than or equal to the threshold distance, the second initial matching point pair is an outer point;

step four, judging whether the ratio of the number of the inner points to the number of the plurality of initial matching point pairs is larger than a threshold ratio and whether the iteration times is larger than the maximum iteration times, if the ratio of the number of the inner points to the number of the plurality of initial matching point pairs is larger than the threshold ratio or the iteration times is larger than the maximum iteration times, the initial basis matrix is a target basis matrix, and if the ratio of the number of the inner points to the number of the plurality of initial matching point pairs is smaller than or equal to the threshold ratio and the iteration times is smaller than or equal to the maximum iteration times, repeating the step one to the step four.

7. The epipolar correction method of claim 1, wherein the determining a first projection matrix and a second projection matrix based on the center point, the first pole, and the second pole of the first image comprises:

performing homogeneous transformation and simplification on the central point of the first image to obtain an initial translation transformation matrix;

performing rotation mapping on the first image and the second image to correspondingly obtain a first rotation matrix and a second rotation matrix;

projecting the first pole and the second pole to a horizontal infinite point to correspondingly obtain a first standard projective transformation matrix and a second standard projective transformation matrix;

performing translation transformation on the first image and the second image to obtain a target translation transformation matrix;

determining the first projection matrix based on the initial translation transformation matrix, the first rotation matrix, the first standard projection transformation matrix, and the target translation transformation matrix;

determining the second projection matrix based on the initial translation transformation matrix, the second rotation matrix, and the second standard projective transformation matrix.

8. The epipolar rectification method according to claim 1, further comprising:

optimizing the first projection matrix and the second projection matrix based on an optimization model to obtain a first optimized projection matrix and a second optimized projection matrix;

performing projection transformation on the first feature points and the second feature points based on the first projection matrix and the second projection matrix to obtain a corrected image, specifically:

and performing projection transformation on the first characteristic point and the second characteristic point based on the first optimized projection matrix and the second optimized projection matrix to obtain a corrected image.

9. The epipolar rectification method of claim 8, wherein the initial pair of matched points comprises a first matched point and a second matched point, and the optimization model is:

in the formula, S is an optimization target; h ₀ Is a first projection matrix; h ₁ Is a second projection matrix; u. of _0i The coordinate value of the first matching point; u. u _1i The coordinate value of the second matching point; and N is the number of the initial matching point pairs.

10. An epipolar line rectification device, comprising:

an image acquisition unit for acquiring a first image and a second image based on a binocular vision system;

the characteristic point determining unit is used for determining a plurality of first characteristic points in the first image and a plurality of second characteristic points in the second image based on gray values of image pixel points;

a matching point determining unit, configured to match the plurality of first feature points and the plurality of second feature points to obtain a plurality of initial matching point pairs;

a basic matrix determining unit, configured to determine a target basic matrix based on a normalized eight-point algorithm, a random consistency algorithm, and the plurality of initial matching point pairs;

a projection matrix determination unit for determining a first pole and a second pole based on the target basis matrix and epipolar constraints, and determining a first projection matrix and a second projection matrix based on a center point of the first image, the first pole, and the second pole;

and the image correction unit is used for performing projection transformation on the first characteristic point and the second characteristic point based on the first projection matrix and the second projection matrix to obtain a corrected image.

Images