CN113223094A

CN113223094A - Binocular imaging system, control method and device thereof, and storage medium

Info

Publication number: CN113223094A
Application number: CN202110567005.2A
Authority: CN
Inventors: 常治国
Original assignee: Shenzhen Geesunn Technology Co ltd
Current assignee: Shenzhen Geesunn Technology Co ltd
Priority date: 2021-05-24
Filing date: 2021-05-24
Publication date: 2021-08-06

Abstract

The invention relates to the technical field of image processing, and provides a binocular imaging system, a control method and a control device thereof, and a storage medium. The method comprises the following steps: establishing a mapping table; controlling a wide-angle camera to perform imaging to obtain a calibration image; determining a pixel point from the calibration image as a calibration point, retrieving n control parameters from a mapping table by taking the wide-angle image coordinate of the calibration point as an index, wherein n is a positive integer greater than 1; performing interpolation calculation based on the n control parameters obtained by retrieval and the wide-angle image coordinates corresponding to the n control parameters in the mapping table to obtain calibration control parameters; and controlling the tele camera to perform imaging according to the calibration control parameters. According to the embodiment of the invention, through interpolation calculation, firstly, the entries of the mapping table can be reduced, the load of building the mapping table is reduced, the retrieval speed is improved, and secondly, the control parameters in the mapping table can be corrected firstly, so that the problem of inaccurate positioning caused by optical path distortion is avoided, and the telephoto camera is driven to aim at the calibration point quickly and accurately.

Description

Binocular imaging system, control method and device thereof, and storage medium

Technical Field

The invention relates to the technical field of image processing, in particular to a binocular imaging system, a control method and a control device thereof, and a storage medium.

Background

In order to realize tracking and high-resolution imaging of an interested target in a large scene range, a binocular imaging system can be formed by two cameras, wherein the first camera is a wide-angle camera which images the whole scene and is used for finding and positioning the interested target, the second camera is a long-focus high-resolution camera which aims at the interested target to take a picture or record a video under the drive of a control system, and a high-resolution picture or video of the target is obtained. Such a design has a cost advantage over a single-eye, ultra-high resolution camera that can cover a large, full-scale scene, because the cost of the imaging system is not linearly related to the resolution, and the cost of the system increases exponentially when the resolution is high to some extent.

When the relative positions of a holder (a control system of a long-focus camera), the long-focus camera and a wide-angle camera in the binocular imaging system are fixed, the long-focus camera can be driven to aim at a target by calculating the angle position parameter of a holder motor corresponding to the wide-angle image coordinate of the wide-angle camera.

However, because the optical path systems of the wide-angle camera and the telephoto camera have unavoidable distortion, and there are also unavoidable errors in the machining and assembly of mechanical components, under the influence of the superposition of various errors, it is difficult to actually align the imaging center point of the telephoto camera to the position corresponding to the wide-angle image coordinate through the angular position parameter of the pan-tilt motor calculated by the wide-angle image coordinate of the wide-angle camera.

Therefore, a great challenge faced by the binocular imaging systems in the prior art is how to quickly and accurately drive the telephoto camera to aim at the target of interest after the target of interest is identified and positioned by the wide-angle camera.

Disclosure of Invention

In view of this, embodiments of the present invention provide a binocular imaging system, a control method and apparatus thereof, and a storage medium, so as to solve the technical problem that it is difficult to quickly and accurately drive a telephoto camera to aim at an interested target in the prior art.

In a first aspect, an embodiment of the present invention provides a method for controlling a binocular imaging system, where the binocular imaging system includes a wide-angle camera and a telephoto camera, and the method includes:

s10: establishing a mapping table; the index of the mapping table is a wide-angle image coordinate of the wide-angle camera, and the return value of the mapping table is a control parameter of the telephoto camera;

s20: controlling the wide-angle camera to perform imaging to obtain a calibration image;

s30: determining a pixel point from the calibration image as a calibration point, retrieving n control parameters from the mapping table by taking the wide-angle image coordinate of the calibration point as an index, wherein n is a positive integer greater than 1;

s40: performing interpolation calculation based on the n control parameters obtained by retrieval and the wide-angle image coordinates corresponding to the n control parameters in the mapping table to obtain calibration control parameters;

s50: and controlling the tele camera to perform imaging according to the calibration control parameters.

According to the embodiment of the invention, the wide-angle image coordinate of the wide-angle camera is used as an index, the control parameter of the telephoto camera is used as a return value to establish the mapping table, and after the wide-angle camera is controlled to image to obtain the calibration image, the n control parameters can be retrieved from the mapping table according to the calibration point in the calibration image. Then, interpolation calculation is carried out based on the n control parameters, namely the n control parameters jointly determine a calibration control parameter, so that the telephoto camera is driven to carry out imaging. On one hand, the entries of the mapping table can be reduced, and when the mapping table is established, all reachable points of the telephoto camera do not need to be established, so that the burden of establishing the mapping table is greatly reduced, and the working efficiency is improved. Meanwhile, due to the reduction of mapping table entries, the retrieval speed can be effectively improved. On the other hand, because each calibration control parameter is obtained from the preset and stored mapping table, the operation problem in the system operation process is avoided, the working efficiency is improved, and because the control parameters in the mapping table are preset, the problem of inaccurate positioning caused by light path distortion can be avoided after the control parameters are corrected in advance. Therefore, the long-focus camera can be driven to aim at the calibration point quickly and accurately by adopting the embodiment of the invention.

Preferably, in the S40, the method includes:

s41: determining the weight coefficients corresponding to the n control parameters respectively; the weight coefficient is inversely related to the distance between the wide-angle image coordinate corresponding to the weight coefficient and the wide-angle image coordinate of the calibration point;

s42: and taking the sum of the products of the n control parameters and the corresponding weight coefficients as the calibration control parameter.

The embodiment of the invention provides a distance-based interpolation algorithm, which is characterized in that the distance between a wide-angle image coordinate corresponding to n control parameters (namely, the wide-angle image coordinate corresponding to a weight coefficient) and a wide-angle image coordinate of a calibration point is determined, and the size of the weight coefficient is determined, so that the proportion of the control parameters corresponding to the wide-angle image coordinate closer to the calibration point in the calibration control parameters is larger, and the positioning accuracy of a long-focus camera is effectively improved.

Preferably, in the S10, the method includes:

s11: controlling the binocular imaging system to image the calibration card printed with the two-dimensional code matrix to obtain a wide-angle image and a tele image; each two-dimensional code encodes the row and column coordinates of the two-dimensional code in the two-dimensional code matrix;

s12: determining the corresponding relation between the coordinates of the wide-angle image and the coordinates of a row and a column according to the two-dimensional codes in the wide-angle image, and determining the corresponding relation between the coordinates of the tele-image and the coordinates of the row and the column according to the two-dimensional codes in the tele-image;

s13: determining a row-column coordinate corresponding to a central pixel of the long-focus image according to the corresponding relation between the long-focus image coordinate and the row-column coordinate;

s14: determining a wide-angle image coordinate corresponding to a central pixel of the long-focus image according to a row-column coordinate corresponding to the central pixel of the long-focus image and a corresponding relation between the wide-angle image coordinate and the row-column coordinate;

s15: and establishing the mapping table by taking the wide-angle image coordinate corresponding to the central pixel of the tele image as an index and taking the control parameter when the tele camera images as a return value.

The embodiment of the invention images the calibration card printed with the two-dimensional code matrix by controlling the binocular imaging system to obtain the wide-angle image and the tele image. And then, determining the corresponding relation between the wide-angle image coordinates and the row and column coordinates and the corresponding relation between the tele-image coordinates and the row and column coordinates based on the two-dimensional codes on the image. Then, based on the corresponding relationship, the central pixel of the tele image (the imaging central point of the tele camera) and the coordinates of the wide image can be unified to the same coordinate system (a row-column coordinate system), so that the control parameters corresponding to the coordinates of the wide image are determined, the influence caused by optical distortion and production and assembly errors of structural members is eliminated, and the accuracy of the positioning of the tele camera is effectively improved.

Preferably, in S12, the determining, according to the two-dimensional code in the wide-angle image, a correspondence between coordinates of the wide-angle image and row-column coordinates includes:

s121: determining row and column coordinates corresponding to the two-dimensional codes based on the wide-angle image;

s122: determining the corresponding relation between the coordinates of the wide-angle image and the coordinates of the ranks according to the coordinates of the wide-angle image of the central pixel of the two-dimensional code in the wide-angle image and the coordinates of the ranks corresponding to the two-dimensional code;

determining the corresponding relation between the long-focus image coordinates and the row and column coordinates according to the two-dimensional code in the long-focus image, and the method comprises the following steps:

s123: determining row and column coordinates corresponding to the two-dimensional codes based on the tele images;

s124: and determining the corresponding relation between the long-focus image coordinates and the row and column coordinates according to the long-focus image coordinates of the central pixel of the two-dimensional code in the long-focus image and the row and column coordinates corresponding to the two-dimensional code.

According to the embodiment of the invention, when the corresponding relation between the wide-angle image coordinate and the row-column coordinate is established, the row-column coordinate corresponding to the two-dimensional code and the wide-angle image coordinate of the central pixel of the two-dimensional code are adopted for determining, the established corresponding relation is accurate, and the image offset and distortion are small when the coordinate system is transformed. Similarly, the corresponding relation between the long-focus image coordinate and the row-column coordinate established by adopting the row-column coordinate corresponding to the two-dimensional code and the long-focus image coordinate of the central pixel of the two-dimensional code has the advantages.

Preferably, in S121, the method includes:

s1211: converting the wide-angle image into a gray image and carrying out binarization processing;

s1212: performing connected domain analysis on the wide-angle image subjected to binarization processing to obtain a plurality of color blocks;

s1213: eliminating color blocks with areas not meeting preset standards;

s1214: determining color blocks with the centers of gravity being closest to central pixels of the wide-angle image as seed color blocks, and setting row and column coordinates of the seed color blocks to be the same as row and column coordinates coded by the two-dimensional codes at the center of the two-dimensional code matrix;

s1215: and setting corresponding row and column coordinates for each color block based on the seed color blocks so as to determine the row and column coordinates corresponding to each two-dimensional code.

Due to the fact that the wide-angle image coordinate imaging precision is low, row and column coordinates coded by the two-dimensional codes cannot be obtained through analyzing the two-dimensional codes after imaging. Therefore, in the embodiment of the invention, through connected domain analysis, the two-dimensional codes in the wide-angle image are processed into color blocks, then color block screening is performed according to a preset standard, the color block with the gravity center closest to the central pixel of the wide-angle image is determined as the seed color block, and then corresponding row and column coordinates can be set for each color block based on the seed color block so as to determine the row and column coordinates corresponding to each two-dimensional code, thereby solving the technical problem that the row and column coordinates coded by the two-dimensional codes cannot be obtained due to the fact that the imaging precision of a wide-angle camera is low and the two-dimensional codes cannot be analyzed.

Preferably, in S1215, the method includes:

s12151: searching the next color block to the left/right by taking the ordinate of the barycentric coordinate of the seed color block as a reference, after a new color block is searched each time, correspondingly subtracting 1 or adding 1 from the row coordinate of the row-column coordinate of the previous color block to serve as the row-column coordinate of the new color block, and then continuously searching the next color block to the left/right by taking the ordinate of the barycentric coordinate of the new color block as a reference until the boundary of the wide-angle image is reached;

s12152: and for the color block with the determined row coordinate, respectively searching the next color block upwards/downwards by taking the abscissa of the barycentric coordinate of the color block as a reference, after a new color block is searched each time, correspondingly subtracting 1 or adding 1 to the row coordinate of the row-column coordinate of the previous color block to be used as the row-column coordinate of the new color block, and then continuously searching the next color block upwards/downwards by taking the abscissa of the barycentric coordinate of the new color block as a reference until the boundary of the wide-angle image is reached.

In the embodiment of the invention, the ordinate of the barycentric coordinate of the seed color block is taken as a reference, the next color block is searched for to the left/right, after a new color block is searched for each time, 1 is correspondingly subtracted or added from the column coordinate of the row-column coordinate of the previous color block to be taken as the row-column coordinate of the new color block, then the next color block is continuously searched for to the left/right by taking the ordinate of the barycentric coordinate of the new color block as a reference, but not all the color blocks in the same row are searched for by always taking the ordinate of the barycentric coordinate of the seed color block as a reference. Therefore, since the vertical coordinates of the barycentric coordinates of the color patches of the adjacent columns in the color patches of the same row do not change much, even if the calibration card is slightly tilted and/or rotated relative to the wide-angle camera and the lens of the wide-angle camera is slightly distorted, the color patch which is adjacent to the next color patch in the horizontal direction can be reliably found from one color patch. Similarly, since the abscissa of the barycentric coordinates of the color blocks in the adjacent rows in the same column of color blocks does not change much, even if the calibration card is slightly tilted and/or rotated relative to the wide-angle camera and the lens of the wide-angle camera is slightly distorted, the next and closest color blocks in the vertical direction can be reliably found from one color block.

Preferably, the two-dimensional code matrix includes reference rows and reference columns, the two-dimensional codes in the reference rows and reference columns have different colors from other two-dimensional codes in the two-dimensional code matrix, and after S1215, the method further includes:

s1216: calculating the color average value of the row color block for the color blocks with the same row coordinate, and calculating the color average value of the column color block for the color blocks with the same column coordinate;

s1217: determining a reference row and a reference column based on the color average;

s1218: determining a row coordinate correction value and a column coordinate correction value according to a difference value between row and column coordinates pre-coded by the reference row and the reference column and row and column coordinates set when searching through the seed color blocks;

s1219: and correcting the row-column coordinates corresponding to each color block according to the row coordinate correction value and the column coordinate correction value.

Since the imaging center of the wide-angle camera may not be aligned with the center of the calibration card in some cases, the line and row coordinates set by the patch may not be corrected, and may not be matched with the line and row coordinates analyzed in the telephoto image. Therefore, the embodiment of the invention sets the color of the two-dimensional code of the reference row and the reference column in the calibration card to be different from that of other two-dimensional codes. Therefore, the correction value of the row-column coordinates can be determined by calculating the color mean value of each row and each column of color blocks and judging which row and column of color blocks correspond to the reference row and the reference column, so that the row-column coordinates corresponding to each color block are corrected, the application range of the embodiment of the invention is wider, and the imaging center of the wide-angle camera does not need to be ensured to be aligned with the central position of the calibration card.

Preferably, the corresponding relation between the wide-angle image coordinates and the row-column coordinates is an affine transformation matrix or a perspective transformation matrix; and the corresponding relation between the long-focus image coordinates and the row and column coordinates is an affine transformation matrix or a perspective transformation matrix.

In a second aspect, an embodiment of the present invention provides a control apparatus for a binocular imaging system including a wide-angle camera and a telephoto camera, the apparatus including:

the mapping establishing module is used for establishing a mapping table; the index of the mapping table is a wide-angle image coordinate of the wide-angle camera, and the return value of the mapping table is a control parameter of the telephoto camera;

the wide-angle imaging control module is used for controlling the wide-angle camera to image to obtain a calibration image;

the retrieval module is used for determining a pixel point from the calibration image as a calibration point, taking the wide-angle image coordinate of the calibration point as an index, and retrieving n control parameters from the mapping table, wherein n is a positive integer greater than 1;

the interpolation calculation module is used for carrying out interpolation calculation on the basis of the n control parameters obtained by retrieval and the wide-angle image coordinates corresponding to the n control parameters in the mapping table to obtain calibration control parameters;

and the long-focus imaging control module is used for controlling the long-focus camera to image according to the calibration control parameters.

In a third aspect, an embodiment of the present invention provides a binocular imaging system, including: at least one processor, at least one memory, and computer program instructions stored in the memory, which when executed by the processor, implement the method of the first aspect of the embodiments described above.

In a fourth aspect, embodiments of the present invention provide a storage medium having stored thereon computer program instructions, which when executed by a processor, implement the method of the first aspect in the above embodiments.

Drawings

In order to more clearly illustrate the technical solutions of the embodiments of the present invention, the drawings required to be used in the embodiments of the present invention will be briefly described below, and for those skilled in the art, without any creative effort, other drawings may be obtained according to the drawings, and these drawings are all within the protection scope of the present invention.

Fig. 1 is a schematic diagram of a binocular imaging system provided by an embodiment of the present invention.

Fig. 2 is a schematic flowchart of a control method of a binocular imaging system according to an embodiment of the present invention.

Fig. 3 is a schematic flowchart of an interpolation calculation method according to an embodiment of the present invention.

Fig. 4 is a schematic diagram of interpolation calculation according to an embodiment of the present invention.

Fig. 5 is a flowchart illustrating a mapping table establishing method according to an embodiment of the present invention.

Fig. 6 is a schematic diagram of a calibration card according to an embodiment of the present invention.

Fig. 7 is a schematic diagram of a wide-angle image provided by an embodiment of the invention.

Fig. 8 is a schematic flowchart of a method for determining row-column coordinates corresponding to a two-dimensional code according to an embodiment of the present invention.

Fig. 9 is a schematic diagram of another wide-angle image provided by an embodiment of the invention.

Fig. 10 is a schematic flowchart of a method for setting row and column coordinates based on a seed patch according to an embodiment of the present invention.

Fig. 11 is a flowchart illustrating a method for correcting row and column coordinates according to an embodiment of the present invention.

Fig. 12 is a schematic structural diagram of a control device of a binocular imaging system according to an embodiment of the present invention.

Fig. 13 is a schematic diagram of a binocular imaging system provided by an embodiment of the present invention.

Detailed Description

Features and exemplary embodiments of various aspects of the present invention will be described in detail below, and in order to make objects, technical solutions and advantages of the present invention more apparent, the present invention will be further described in detail below with reference to the accompanying drawings and embodiments. It should be understood that the specific embodiments described herein are merely illustrative of the invention and are not to be construed as limiting the invention. It will be apparent to one skilled in the art that the present invention may be practiced without some of these specific details. The following description of the embodiments is merely intended to provide a better understanding of the present invention by illustrating examples of the present invention.

It is noted that, herein, relational terms such as first and second, and the like may be used solely to distinguish one entity or action from another entity or action without necessarily requiring or implying any actual such relationship or order between such entities or actions. Also, the terms "comprises," "comprising," or any other variation thereof, are intended to cover a non-exclusive inclusion, such that a process, method, article, or apparatus that comprises a list of elements does not include only those elements but may include other elements not expressly listed or inherent to such process, method, article, or apparatus. Without further limitation, an element defined by the phrase "comprising … …" does not exclude the presence of other identical elements in a process, method, article, or apparatus that comprises the element. It should be noted that the row coordinate and the column coordinate and the abscissa and the ordinate referred to in the present invention are exemplary descriptions, and those skilled in the art can obtain the other technical solution through simple inference and permutation based on the technical solution disclosed by one of the row coordinate and the column coordinate or based on the technical solution disclosed by one of the abscissa and the ordinate.

As mentioned above, to achieve tracking and high-resolution imaging of objects of interest over a large scene, a binocular imaging system may be formed using two cameras. Wherein one eye camera is a wide-angle camera, and the other eye camera is a telephoto camera.

For ease of understanding, please refer to fig. 1, which is a schematic diagram of a binocular imaging system according to an embodiment of the present invention. The binocular imaging system includes a wide-angle camera 10, a telephoto camera 20, and a control system 30. The control system 30 is used to control the movement of the tele camera 20, or to control the movement of the wide camera 10 and the tele camera 20. The control system 30 may be a motor, a pan-tilt, a mechanical arm, a mechanical shaft, etc., and the present invention is not particularly limited. In a preferred embodiment of the present invention, a pan-tilt motor is used as a control system, and the pan-tilt motor drives the tele camera 20 to move through two angular position parameters.

In the binocular imaging system, the most preferable setting mode of the wide-angle camera and the telephoto camera is as follows: the imaging plane of the wide-angle camera is parallel to the target plane, the long side of the wide-angle camera image sensor pixel matrix is parallel to the long side of the target plane rectangle (the short side of the image sensor pixel matrix is also parallel to the short side of the target plane rectangle), and the center of the wide-angle camera image sensor pixel matrix aims at the center of the target plane rectangle; when the telephoto camera is in a zero position, the long side of the pixel matrix of the image sensor is parallel to the long side of the target plane rectangle, and the short side of the pixel matrix of the image sensor is parallel to the short side of the target plane rectangle.

Theoretically, when the relative positions of a holder motor, a telephoto camera and a wide-angle camera in the binocular imaging system are fixed, holder motor angle position parameters A (alpha, beta) corresponding to wide-angle image coordinates of the wide-angle camera can be calculated, wherein alpha is the first axis motor angle position of the holder, beta is the second axis motor angle position of the holder, and the motor angle position parameters A (alpha, beta) can drive the telephoto camera to aim at a target. However, because the optical path systems of the wide-angle camera and the telephoto camera have unavoidable distortion, and there are also unavoidable errors in the machining and assembly of mechanical components, under the influence of the superposition of various errors, it is difficult to actually align the imaging center point of the telephoto camera to the position corresponding to the wide-angle image coordinate through the angular position parameter of the pan-tilt motor calculated by the wide-angle image coordinate of the wide-angle camera.

Fig. 2 is a schematic flow chart of a control method of a binocular imaging system according to an embodiment of the present invention, where the method includes:

Specifically, a mapping table is established with the wide-angle image coordinates of the wide-angle camera as an index and the control parameters of the telephoto camera as return values. The mapping table can be stored in a nonvolatile memory and can be repeatedly used after being established once. The mapping Table can also be established by using a Look-Up Table (LUT), and the method is more suitable for circuits with higher integration level.

On the premise of establishing a mapping table, firstly, the wide-angle camera is controlled to image to obtain a calibration image. In the calibration image, a pixel point can be determined as a calibration point. Specifically, the calibration point may be selected manually, or one of the pixel points of the specific image may be determined as the calibration point from the calibration image based on machine vision or other recognition methods. Here, the coordinates of the index point are P (x, y).

The mapping table may then be retrieved based on the coordinates P (x, y) of the index point to determine the n control parameters. Wherein n is a positive integer greater than 1. In an embodiment of the present invention, a value of n may be preset, and during the search, n wide-angle image coordinates closest to the index point are found from the index of the mapping table (i.e., the wide-angle image coordinates), so as to determine n control parameters. In another embodiment of the present invention, a distance threshold may be set, and the coordinates of the wide-angle image whose distance from the coordinates P (x, y) of the calibration point is smaller than the threshold may be retrieved to determine the corresponding control parameters. The aforementioned distance may be defined based on x and y coordinates. E.g., the wide-angle image coordinates are (x)₀，y₀) Then the distance between it and the calibration point can be defined as:

after n control parameters are obtained through retrieval, interpolation calculation can be carried out to obtain calibration control parameters. The interpolation calculation may use various algorithms, and the present invention is not particularly limited. Referring to fig. 3, a preferred embodiment of the present invention, in S40, includes:

Specifically, after n control parameters are determined by the method, the weight coefficients corresponding to the n control parameters are determined respectively. For each weight coefficient, the greater the distance between its corresponding wide-angle image coordinate and the wide-angle image coordinate of the calibration point, the smaller the weight coefficient. For convenience of description, let's note that n control parameters are A₁～A_nTheir corresponding weight coefficient is K₁～K_nThen the calibration control parameter is equal to:

the present invention also provides a more preferable specific algorithm, for the convenience of understanding, please refer to fig. 4, wherein a total of 4 wide-angle image coordinates are obtained by retrieving from the mapping table according to the wide-angle image coordinates (x, y) of the index point, which are (x) respectively_a，y_a)，(x_b，y_b)，(x_c，y_c)，(x_d，y_d) The 4 wide-angle image coordinates correspond to the control parameter (alpha) respectively_a,β_a)，(α_b,β_b)，(α_c,β_c),(α_d,β_d). Firstly, calculating the distance between the 4 wide-angle image coordinates and the coordinate of the calibration point wide-angle image to obtain:

then, calculating a weight coefficient corresponding to each control parameter:

wherein sum is 1/L_a+1/L_b+1/L_c+1/L_d。

Based on the algorithm, the calibration control parameters are as follows:

α＝α_a*K_a+α_b*K_b+α_c*K_c+α_d*K_d；

β＝β_a*K_a+β_b*K_b+β_c*K_c+β_d*K_d。

therefore, after the calibration control parameters are obtained, the telephoto camera can be driven to image, and high-resolution shooting of the calibration point is achieved. By adopting the interpolation calculation method provided by the embodiment of the invention, the entries of the mapping table can be effectively reduced, and when the mapping table is established, all reachable points of the telephoto camera do not need to be established, so that the burden of establishing the mapping table is greatly reduced, and the working efficiency is improved. Meanwhile, due to the reduction of mapping table entries, the retrieval speed can be effectively improved. Particularly, in the implementation mode of the LUT, the number of Multiplexers (MUX) can be greatly reduced, the complexity of an integrated circuit is reduced, and the problem of inaccurate positioning caused by optical path distortion can be avoided after pre-correction because control parameters in a mapping table are preset. The embodiment of the invention can be applied to a classroom, the wide-angle image obtained by shooting through the wide-angle camera comprises a plurality of desks, then documents (such as test papers) placed on the desks are obtained through identification based on the wide-angle image, then one pixel (preferably a central pixel) of each document is used as a calibration point one by one, a plurality of control parameters are searched from a mapping table, the calibration control parameters are obtained through interpolation calculation, the telephoto camera is driven to shoot each document one by one, and therefore the high-definition image of the document is obtained.

Please refer to fig. 5, which is a flowchart illustrating a method for establishing a mapping table according to an embodiment of the present invention, in the step S10, the method includes:

For ease of understanding, please refer to the calibration card shown in fig. 6, which has a two-dimensional code matrix printed thereon, each two-dimensional code encoding its own row and column coordinates in the two-dimensional code matrix. The type of the two-dimensional Code is not particularly limited, and may be, for example, QR Code, Data Matrix, Maxi Code, or the like. In one embodiment of the present invention, the row and column coordinates of the two-dimensional code at the center of the two-dimensional code matrix (i.e., the two-dimensional code in row 3 and column 5 as shown in fig. 6) are (0, 0), and then the row and column coordinates of the other two-dimensional codes are encoded based on the two-dimensional code. For example, the row and column coordinates of the two-dimensional code adjacent to the left side (i.e., the two-dimensional code in row 3 and column 4 as shown in fig. 6) are coded to be (0, -1), the row and column coordinates of the two-dimensional code adjacent to the right side (i.e., the two-dimensional code in row 3 and column 6 as shown in fig. 6) are coded to be (0, 1), the row and column coordinates of the two-dimensional code adjacent to the lower side (i.e., the two-dimensional code in row 4 and column 5 as shown in fig. 6) are coded to be (1, 0), the row and column coordinates of the two-dimensional code adjacent to the upper side (i.e., the two-dimensional code in row 2 and column 5 as shown in fig. 6) are coded to be (-1, 0), and so on.

Therefore, in a telephoto image or a wide-angle image obtained by imaging the calibration card, the row and column coordinates of the two-dimensional code can be determined. Then, the corresponding relation between the wide-angle image coordinate and the row and column coordinate and the corresponding relation between the tele-image coordinate and the row and column coordinate can be determined based on the two-dimensional code.

Specifically, according to the two-dimensional code in the wide-angle image, determining the corresponding relationship between the coordinates of the wide-angle image and the coordinates of rows and columns, including:

s122: and determining the corresponding relation between the coordinates of the wide-angle image and the coordinates of the rows and columns according to the coordinates of the wide-angle image of the central pixel of the two-dimensional code in the wide-angle image and the coordinates of the rows and columns corresponding to the two-dimensional code.

And the corresponding relation between the wide-angle image coordinates and the row and column coordinates is an affine transformation matrix or a perspective transformation matrix.

Specifically, determining the corresponding relationship between the tele image coordinate and the row-column coordinate according to the two-dimensional code in the tele image includes:

And the corresponding relation between the tele image coordinate and the row and column coordinate is an affine transformation matrix or a perspective transformation matrix.

Since the principles of establishing the correspondence between the coordinates of the wide-angle image and the coordinates of the row and column and establishing the correspondence between the coordinates of the tele-image and the coordinates of the row and column are similar, the establishment of the correspondence between the coordinates of the wide-angle image and the coordinates of the row and column is taken as an example for explanation.

Firstly, based on the wide-angle image, the row and column coordinates coded by the two-dimensional code can be obtained by analyzing the two-dimensional code. Meanwhile, the wide-angle image coordinates of any pixel point in the two-dimensional code are used as the corresponding coordinates, so that the m groups of corresponding coordinates are determined. Specifically, when determining the coordinates corresponding to the m groups, the wide-angle image coordinates corresponding to the row and column coordinates are preferably the wide-angle image coordinates of the central pixel points of the two-dimensional code, and if the center of the two-dimensional code is located among a plurality of pixel points, the wide-angle image coordinates corresponding to the row and column coordinates are preferably the coordinate average value of the plurality of pixel points, or the coordinates of one pixel point are selected as the wide-angle image coordinates corresponding to the row and column coordinates.

If an affine transformation matrix is to be established, m is a positive integer greater than or equal to 3, and if a perspective transformation matrix is to be established, m is a positive integer greater than or equal to 4. Here, the affine transformation matrix is used for explanation, and when the affine transformation matrix is established, the following formula can be adopted:

it is equivalent to:

wherein x represents the abscissa of the wide-angle image coordinate, y represents the ordinate of the wide-angle image coordinate, u represents the row coordinate of the row-column coordinate, and v represents the ordinate of the row-column coordinate. According to the formula, 6 unknowns need to be solved when the affine transformation matrix is established, and the solution can be carried out by substituting 3 groups of corresponding wide-angle image coordinates and row-column coordinates. Based on similar processes, the correspondence between the tele-image coordinates and the row-column coordinates can be established, which is not further described herein.

After the corresponding relation between the wide-angle image coordinates and the row-column coordinates and the corresponding relation between the tele-image coordinates and the row-column coordinates are established, the row-column coordinates corresponding to the central pixel of the tele-image can be determined according to the corresponding relation between the tele-image coordinates and the row-column coordinates. Specifically, the tele image coordinates of the central pixel of the tele image are substituted into the affine transformation matrix or the perspective transformation matrix between the tele image coordinates and the row-column coordinates, so as to obtain the row-column coordinates corresponding to the central pixel of the tele image.

Further, substituting the row and column coordinates corresponding to the central pixel of the tele image into an affine transformation matrix or a perspective transformation matrix between the wide image coordinates and the row and column coordinates, so as to determine the wide image coordinates corresponding to the central pixel of the tele image.

At this time, the control parameter when the tele camera images is obtained as the return value in the mapping table, and the wide image coordinate corresponding to the central pixel of the tele image is used as the index of the return value, so as to establish a pair of mapping relations in the mapping table. If the next group of mapping relations are to be established, the control parameters are transformed, and the process is repeated.

In some scenes, because the resolution capability of the wide-angle camera is limited, the captured wide-angle image cannot be analyzed by the two-dimensional code as shown in fig. 7, and thus the row and column coordinates corresponding to the two-dimensional code cannot be obtained (the line and column coordinates are not worried about by the tele-image). Therefore, an embodiment of the present invention further provides a method for acquiring row-column coordinates corresponding to a two-dimensional code in a wide-angle image, please refer to fig. 8, where in S121, the method includes:

s1213: eliminating color blocks with areas not meeting preset standards;

Specifically, the wide-angle image shown in fig. 7 may be converted into a grayscale image, and subjected to binarization processing and connected component analysis to obtain a plurality of imagesWide angle image of color block. Then, the rejection area is smaller than A_minOr an area greater than A_maxThe wide-angle image shown in fig. 9 is obtained. Wherein A is_minAnd A_maxIs a preset value.

And calculating the gravity center of each color block based on the wide-angle image shown as 9, wherein the gravity center of each color block is the mean value of x coordinates of all pixel points of the color block and the mean value of y coordinates of all pixels. Then, find out the color lump whose center of gravity is closest to the central pixel of the wide-angle image, regard it as the seed color lump (namely the 3 rd row 5 th column color lump), and presume its line and row coordinates are the same as line and row coordinates encoded by two-dimensional code of the matrix center of two-dimensional code. If the row and column coordinates coded by the two-dimensional code in the center of the two-dimensional code matrix are (0, 0), the row and column coordinates of the seed color blocks are set to be (0, 0), and corresponding row and column coordinates are set for each color block based on the seed color blocks so as to determine the row and column coordinates corresponding to each two-dimensional code. Specifically, if the row-column coordinates of the seed color block are set to be (0, 0), the row-column coordinates of the color block adjacent to the left side (i.e., the color block in the 3 rd row and the 4 th column) are set to be (0, -1), the row-column coordinates of the color block adjacent to the right side (i.e., the color block in the 3 rd row and the 6 th column) are set to be (0, 1), the row-column coordinates of the color block adjacent to the lower side (i.e., the color block in the 4 th row and the 5 th column) are set to be (1, 0), and the row-column coordinates of the color block adjacent to the upper side (i.e., the color block in the 2 nd row and the 5 th column) are set to be (-1, 0), and so on.

In one embodiment of the present invention, the filtering may be performed on the grayscale image first, and then the binarization processing may be performed, and low-pass filtering is preferable.

In one embodiment of the present invention, morphological erosion (erode) may be performed on the binarized image before connected domain analysis, so that thin and small interfering objects may be eliminated without substantially affecting the center of gravity of the color patches.

The embodiment of the present invention further provides a method for more accurately searching for an adjacent color block, please refer to fig. 10, in S1215, the method includes:

Specifically, a scene with row and column coordinates of a seed color block as (0, 0) is described, and barycentric coordinates of the seed color block are recorded as (x)_e，y_e)。

First, find all color blocks with row coordinates of 0: when y is equal to y_eTo the left of the horizontal line of (a) to find the nearest color patch with a barycentric coordinate of (x)_f,y_f) Setting the row and column coordinates to (0, -1), and then setting y to y_fAnd continuously searching the next nearest color block to the left on the horizontal line, and recursively performing the process until the left boundary of the image is reached, thereby finding all color blocks on the left half side of the same row as the seed color block. Then, all color blocks on the right half of the same row as the seed color block are found by the same method, and accordingly, all color blocks with the row coordinate of 0 are obtained. Because the y coordinate of the barycentric coordinates of the color blocks in the adjacent columns in the same row of color blocks does not change greatly, even if the correction card slightly tilts and/or rotates relative to the wide-angle camera and the lens of the wide-angle camera slightly distorts, the next and closest color blocks in the horizontal direction can be reliably found from one color block.

Then, each color block with row coordinate of 0 (i.e., the color block with row and column coordinates determined in the foregoing step) is used as a starting point color block, and all color blocks in the same row with the starting point color block are respectively found. The barycentric coordinate of the starting point color block (0, C) is defined as (x)_g,y_g) Where x is equal to x_gFind the nearest color block upwards on the vertical line of the color block, and the barycentric coordinate is(x_h,y_h) Setting the coordinates of the row and column as (-1, C), and then x as x_hAnd continuously searching the nearest color block upwards on the vertical line, and performing the process recursively until the upper boundary of the image is reached, thereby finding all color blocks on the upper side which belong to the same column with the starting point color block. Then, all the color blocks of the lower half belonging to the same column as the starting point color block are found out by the same method, and accordingly, all the color blocks with the column coordinate of C are obtained. Since the x coordinate of the barycentric coordinates of the color blocks in the adjacent rows in the same column of color blocks does not change greatly, even if the correction card is slightly tilted and/or rotated relative to the wide-angle camera and the lens of the wide-angle camera is slightly distorted, the next and closest color blocks in the vertical direction can be reliably found from one color block.

In some application scenarios, since the center of the wide-angle image does not necessarily correspond to the center of the calibration card, in these cases, if the line and column coordinates are set by the wide-angle image in the aforementioned manner, and the line and column coordinates are obtained by directly analyzing the two-dimensional code by the tele-image, the line and column coordinates will be mismatched. Therefore, an embodiment of the present invention further provides a method for correcting row and column coordinates, please refer to fig. 11, after S1215, the method further includes:

Specifically, the two-dimensional code matrix comprises a reference row and a reference column, and the color of the two-dimensional code in the reference row and the reference column is different from that of other two-dimensional codes in the two-dimensional code matrix. In a preferred embodiment of the present invention, as shown in fig. 6, the reference row is a middle row of the two-dimensional code matrix, and the reference column is a center column of the two-dimensional code matrix. It is obvious that the reference row can be set to any row of the two-dimensional code matrix, and the reference column is any column of the two-dimensional code matrix. Or taking 2 diagonal lines of the two-dimensional code matrix as reference rows and reference columns.

And calculating the color average value of the color blocks in the row with the same coordinates. For example, the RGB color mode is used for calculation, color values of an R channel, a G channel, and a B channel of each pixel point in a color block are determined first, and then a color average value of the R channel, a color average value of the G channel, and a color average value of the B channel are determined respectively according to a mode of independent calculation of each channel. Since the color of the reference line is known in advance, it can be determined which line color block is for the reference line based on the average value of the color of each line. Similarly, it can be determined which column of color blocks corresponds to the reference column. The present invention is not limited to the color mode, and may be a color mode such as CMYK or LAB.

For the reference row, the set row-column coordinates are now known, and the encoded row-column coordinates are known. Therefore, the difference between them can be calculated. If the row coordinate set by the reference row is S and the row coordinate encoded in the two-dimensional code matrix is 0, the difference value between the two rows can be determined to be S, the difference value is used as a row coordinate correction value, and the row coordinate of each color block is subtracted by the S to obtain the final row coordinate of each color block.

Similarly, for the reference column, the set row and column coordinates are known, and the encoded row and column coordinates are known. Therefore, the difference between them can be calculated. If the column coordinate set by the reference column is D and the column coordinate encoded in the two-dimensional code matrix is 0, the difference value between the two column coordinates can be determined to be D, the difference value is used as the column coordinate correction value, and the column coordinate of each color block is subtracted by D to obtain the final column coordinate of each color block.

Therefore, the embodiment of the invention can be applied to more scenes without ensuring that the imaging center of the wide-angle camera is aligned with the calibration card.

Based on the technical solutions described with reference to fig. 7 to 11, when the wide-angle image cannot analyze the two-dimensional code, the row and column coordinates corresponding to each color block are determined, and the row and column coordinates of the color block are used as the row and column coordinates of the corresponding two-dimensional code, so that the corresponding relationship between the wide-angle image coordinates and the row and column coordinates is established based on the wide-angle image. Based on the technical scheme disclosed above, a person skilled in the art can easily and simply deduce another technical scheme: the two-dimensional codes on the calibration card are arranged in other modes, for example, the two-dimensional codes are annularly arranged in a plurality of groups, the central point of each two-dimensional code annularly arranged in each group forms a circle, and the circles form concentric circles. And meanwhile, a polar coordinate system is adopted when the two-dimensional code is coded. Obviously, there is no essential difference between the arrangement mode of the two-dimensional code of the calibration card and the technical scheme disclosed by the embodiment of the invention, and the arrangement mode is regarded as simple transformation of the invention.

Referring to fig. 12, an embodiment of the present invention further provides a control apparatus for a binocular imaging system, where the apparatus includes:

In addition, the control method of the binocular imaging system according to the embodiment of the present invention described with reference to fig. 2 may be implemented by the binocular imaging system. Fig. 13 is a schematic diagram illustrating a hardware structure of a binocular imaging system according to an embodiment of the present invention.

The binocular imaging system may include a processor and a memory storing computer program instructions.

In particular, the processor may include a Central Processing Unit (CPU), or an Application Specific Integrated Circuit (ASIC), or one or more Integrated circuits that may be configured to implement embodiments of the present invention.

The memory may include mass storage for data or instructions. By way of example, and not limitation, memory may include a Hard Disk Drive (HDD), floppy Disk Drive, flash memory, optical Disk, magneto-optical Disk, magnetic tape, or Universal Serial Bus (USB) Drive or a combination of two or more of these. The memory may include removable or non-removable (or fixed) media, where appropriate. The memory may be internal or external to the data processing apparatus, where appropriate. In a particular embodiment, the memory is non-volatile solid-state memory. In a particular embodiment, the memory includes Read Only Memory (ROM). Where appropriate, the ROM may be mask-programmed ROM, Programmable ROM (PROM), Erasable PROM (EPROM), Electrically Erasable PROM (EEPROM), electrically rewritable ROM (EAROM), or flash memory or a combination of two or more of these.

The processor reads and executes the computer program instructions stored in the memory to implement the control method of any of the binocular imaging systems in the above embodiments.

In one example, the binocular imaging system may also include a communication interface and a bus. As shown in fig. 13, the processor, the memory, and the communication interface are connected via a bus to complete communication therebetween.

The communication interface is mainly used for realizing communication among modules, devices, units and/or equipment in the embodiment of the invention.

The bus includes hardware, software, or both that couple the components of the binocular imaging system to each other. By way of example, and not limitation, a bus may include an Accelerated Graphics Port (AGP) or other graphics bus, an Enhanced Industry Standard Architecture (EISA) bus, a Front Side Bus (FSB), a Hypertransport (HT) interconnect, an Industry Standard Architecture (ISA) bus, an infiniband interconnect, a Low Pin Count (LPC) bus, a memory bus, a Micro Channel Architecture (MCA) bus, a Peripheral Component Interconnect (PCI) bus, a PCI-Express (PCI-X) bus, a Serial Advanced Technology Attachment (SATA) bus, a video electronics standards association local (VLB) bus, or other suitable bus or a combination of two or more of these. A bus may include one or more buses, where appropriate. Although specific buses have been described and shown in the embodiments of the invention, any suitable buses or interconnects are contemplated by the invention.

In addition, in combination with the control method of the binocular imaging system in the above embodiments, the embodiments of the present invention may be implemented by providing a computer-readable storage medium. The computer readable storage medium having stored thereon computer program instructions; the computer program instructions, when executed by a processor, implement the control method of any of the binocular imaging systems of the above embodiments.

It is to be understood that the invention is not limited to the specific arrangements and instrumentality described above and shown in the drawings. A detailed description of known methods is omitted herein for the sake of brevity. In the above embodiments, several specific steps are described and shown as examples. However, the method processes of the present invention are not limited to the specific steps described and illustrated, and those skilled in the art can make various changes, modifications and additions or change the order between the steps after comprehending the spirit of the present invention.

The functional blocks shown in the above-described structural block diagrams may be implemented as hardware, software, firmware, or a combination thereof. When implemented in hardware, it may be, for example, an electronic circuit, an Application Specific Integrated Circuit (ASIC), suitable firmware, plug-in, function card, or the like. When implemented in software, the elements of the invention are the programs or code segments used to perform the required tasks. The program or code segments may be stored in a machine-readable medium or transmitted by a data signal carried in a carrier wave over a transmission medium or a communication link. A "machine-readable medium" may include any medium that can store or transfer information. Examples of a machine-readable medium include electronic circuits, semiconductor memory devices, ROM, flash memory, Erasable ROM (EROM), floppy disks, CD-ROMs, optical disks, hard disks, fiber optic media, Radio Frequency (RF) links, and so forth. The code segments may be downloaded via computer networks such as the internet, intranet, etc.

It should also be noted that the exemplary embodiments mentioned in this patent describe some methods or systems based on a series of steps or devices. However, the present invention is not limited to the order of the above-described steps, that is, the steps may be performed in the order mentioned in the embodiments, may be performed in an order different from the order in the embodiments, or may be performed simultaneously.

As described above, only the specific embodiments of the present invention are provided, and it can be clearly understood by those skilled in the art that, for convenience and brevity of description, the specific working processes of the system, the module and the unit described above may refer to the corresponding processes in the foregoing method embodiments, and are not described herein again. It should be understood that the scope of the present invention is not limited thereto, and any person skilled in the art can easily conceive various equivalent modifications or substitutions within the technical scope of the present invention, and these modifications or substitutions should be covered within the scope of the present invention.

Claims

1. A method of controlling a binocular imaging system including a wide camera and a tele camera, the method comprising:

2. The method according to claim 1, wherein in the S40, comprising:

3. The method according to claim 1, wherein in the S10, comprising:

4. The method of claim 3, wherein in the step S12, the determining the correspondence between the coordinates of the wide-angle image and the coordinates of the row and column according to the two-dimensional code in the wide-angle image includes:

5. The method according to claim 4, wherein in the S121, the method comprises:

s1213: eliminating color blocks with areas not meeting preset standards;

6. The method according to claim 5, wherein in S1215, the method comprises:

7. The method of claim 5, wherein the two-dimensional code matrix comprises reference rows and reference columns, and wherein the two-dimensional codes in the reference rows and reference columns have different colors from other two-dimensional codes in the two-dimensional code matrix, and after the step S1215, the method further comprises:

8. The method according to any one of claims 3-7, wherein the correspondence between the wide-angle image coordinates and the row and column coordinates is an affine transformation matrix or a perspective transformation matrix; and the corresponding relation between the long-focus image coordinates and the row and column coordinates is an affine transformation matrix or a perspective transformation matrix.

9. A control apparatus for a binocular imaging system including a wide angle camera and a telephoto camera, the apparatus comprising:

10. A binocular-based imaging system, comprising: at least one processor, at least one memory, and computer program instructions stored in the memory that, when executed by the processor, implement the method of any of claims 1-8.

11. A storage medium having computer program instructions stored thereon, which when executed by a processor implement the method of any one of claims 1-8.