CN116342706A - Calibration device, method and storage medium for multi-eye heterogeneous camera - Google Patents

Abstract

The application relates to a calibration device, a calibration method and a storage medium of a multi-view heterogeneous camera, wherein a first calibration plate, a second calibration plate, an optical alignment platform, a plane mirror, a first six-axis adjustment platform, a second six-axis adjustment platform, an optical prism, an optical alignment mirror, a first locking structural member and a second locking structural member are arranged; the first calibration plate is arranged on the optical collimation platform through a first locking structural member; the first six-axis adjusting platform is arranged on the optical collimating platform, and the second calibration plate is arranged on the first six-axis adjusting platform through the second locking structural member; the second six-axis adjusting platform is arranged on the optical collimating platform, and the optical collimating mirror is arranged on the second six-axis adjusting platform; the plane mirror is arranged in parallel with the first calibration plate; the optical prism is arranged in parallel with the second calibration plate; the emergent light of the optical collimating lens passes through the optical prism and reaches the first calibration plate, so that the accuracy of the calibration result of the multi-view heterogeneous camera is effectively improved.

Description

Calibration device, method and storage medium for multi-eye heterogeneous camera

Technical Field

The present disclosure relates to the field of computer vision, and in particular, to a calibration device and method for a multi-view heterogeneous camera, a storage medium, and an assembling method for the calibration device for the multi-view heterogeneous camera.

Background

With the development of scientific technology, the multi-view camera is widely applied in the field of video monitoring and the field of eye tracking technology. The multi-view camera has a plurality of cameras that are rigidly connected in structure. Before the multi-camera is used, in order to establish the connection between the cameras, the multi-camera needs to be calibrated to obtain an external parameter conversion matrix between the cameras. If a common viewing angle exists among a plurality of cameras of the multi-view camera and the depth of field difference is not large, the multi-view camera can be calibrated by only using one calibration plate, so that the external parameter conversion matrix among the cameras can be easily obtained, but if the common viewing angle does not exist among the cameras, the mode cannot be applied.

In the prior art, by introducing a third-party camera as an intermediate medium, the third-party camera and a camera without a common viewing angle in the multi-camera to be calibrated are required to have a common viewing angle, and an external parameter conversion matrix between the cameras without the common viewing angle in the multi-camera is solved by utilizing the common viewing angle.

However, when the depth of field of the camera without the common view angle in the multi-view camera is far different, the coverage area of the third-party camera and the field of view of the camera without the common view angle in the multi-view camera are far different, and the calibration result accuracy is low, aiming at the problem that the calibration result accuracy of the multi-view heterogeneous camera in the related technology is low, no effective solution has been proposed at present.

Disclosure of Invention

Based on the above, it is necessary to provide a calibration device and method for a multi-eye heterogeneous camera, a storage medium and an assembling method for the calibration device for the multi-eye heterogeneous camera, so as to solve the problem that the accuracy of the calibration result is low in the multi-eye heterogeneous camera calibration in the related art.

In a first aspect, an embodiment of the present application provides a calibration device for a multi-view heterogeneous camera, where the device includes a first calibration plate, a second calibration plate, an optical collimating platform, a plane mirror, a first six-axis adjustment platform, a second six-axis adjustment platform, an optical prism, an optical collimating mirror, a first locking structure and a second locking structure;

the first calibration plate is arranged on the optical collimation platform through the first locking structural member;

the first six-axis adjusting platform is arranged on the optical collimation platform, the second calibration plate is arranged on the first six-axis adjusting platform through the second locking structural member, and the first six-axis adjusting platform is used for adjusting the directions of the second locking structural member and the second calibration plate;

the second six-axis adjusting platform is arranged on the optical collimating platform, the optical collimating mirror is arranged on the second six-axis adjusting platform, and the second six-axis adjusting platform is used for adjusting the emergent light direction of the optical collimating mirror;

The plane mirror is arranged in parallel with the first calibration plate; the optical prism is arranged in parallel with the second calibration plate; the emergent light of the optical collimating lens penetrates through the optical prism to reach the first calibration plate.

In some embodiments, the calibration device of the multi-view heterogeneous camera further comprises a third calibration plate mounted on the first six-axis adjustment platform through the second locking structure.

In some of these embodiments, the apparatus further comprises a locator;

the second locking structural member comprises an extension structure, wherein a plurality of positioning and aligning holes are formed in the extension structure and used for installing the positioner.

In some embodiments, the locator is a spider laser pen.

In a second aspect, in this embodiment, there is provided a method for assembling a calibration device of a multi-view heterogeneous camera, which is used for assembling the calibration device of the multi-view heterogeneous camera in the first aspect, where the method includes:

adjusting the fixing direction of the first locking structural member on the optical collimation platform to enable the first calibration plate to be perpendicular to the optical collimation platform;

adjusting the first six-axis adjusting platform to enable the second calibration plate to be perpendicular to the optical collimation platform;

Adjusting a second six-axis adjusting platform and translating a plane mirror to enable an emergent light path of the optical collimating mirror to coincide with an incident light path after the emergent light is reflected by the plane mirror;

and adjusting the first six-axis adjusting platform to enable emergent light of the optical collimating lens to penetrate through the optical prism and be projected onto the plane lens, and enabling an emergent light path to coincide with an incident light path after being reflected by the plane lens.

In some of these embodiments, the method further comprises:

installing a positioner on one of positioning and aligning holes on an extending structure of the second locking structural member, and translating the plane mirror to enable emergent light of the positioner to be projected onto the plane mirror;

and adjusting the screw on the positioning and aligning hole to enable the light path of the emergent light of the positioner to coincide with the light path reflected by the plane mirror.

In a third aspect, in this embodiment, there is provided a calibration method of a multi-view heterogeneous camera, where the multi-view heterogeneous camera includes at least two cameras without a common field of view, the calibration method of the multi-view heterogeneous camera is implemented based on the assembly method of the calibration device of the multi-view heterogeneous camera described in the second aspect, and the method includes:

Acquiring a calibration plate image shot by a camera without a public view field; each camera without the public view field independently shoots a corresponding plane calibration plate, and the calibration plates are provided with calibration points;

determining image coordinates of a calibration point on the calibration plate image;

and obtaining an initial external parameter conversion matrix between every two cameras without the common field of view according to the camera internal parameters of each camera without the common field of view, the image coordinates of the calibration points on the images of the calibration plates and the world coordinates of the calibration points.

In some embodiments, the obtaining an initial extrinsic transformation matrix between the cameras without the common field of view according to the camera intrinsic parameters of each camera without the common field of view, the image coordinates of the calibration points on the images of the calibration plates and the world coordinates of the calibration points comprises the following steps:

establishing a world coordinate system by taking a calibration point on one of the calibration plates as an origin, and acquiring world coordinates of each calibration point;

and obtaining an initial external parameter conversion matrix between every two cameras without the common field of view according to the camera internal parameters of the cameras without the common field of view, the image coordinates of the calibration points on the images of the calibration plates and the world coordinates of the calibration points.

In some of these embodiments, the method further comprises:

setting preset times, and indicating the multi-view heterogeneous camera to perform pose change of the preset times, wherein the multi-view heterogeneous camera obtains a calibration plate image shot by the camera without the public view field once every time the multi-view heterogeneous camera performs pose change;

constructing a linear constraint condition according to the acquired image coordinates of the calibration points on the calibration plate image and the world coordinates of each calibration point;

and carrying out iterative computation on the linear constraint conditions by using an optimization method, thereby obtaining the optimal external parameter conversion matrix between every two cameras without the public view field.

In some of these embodiments, the optimization method comprises a Levenberg-Marquard algorithm or Newton's iterative method.

In a fourth aspect, in this embodiment, a computer-readable storage medium is provided, on which a computer program is stored, which computer program, when being executed by a processor, implements the steps of the method according to the third aspect.

The calibration device and method of the multi-eye heterogeneous camera, the storage medium and the assembly method of the calibration device of the multi-eye heterogeneous camera are characterized in that a first calibration plate, a second calibration plate, an optical collimation platform, a plane mirror, a first six-axis adjustment platform, a second six-axis adjustment platform, an optical prism, an optical collimation mirror, a first locking structural member and a second locking structural member are arranged; the first calibration plate is arranged on the optical collimation platform through a first locking structural member; the first six-axis adjusting platform is arranged on the optical collimating platform, the second calibration plate is arranged on the first six-axis adjusting platform through the second locking structural member, and the first six-axis adjusting platform is used for adjusting the directions of the second locking structural member and the second calibration plate; the second six-axis adjusting platform is arranged on the optical collimating platform, the optical collimating mirror is arranged on the second six-axis adjusting platform, and the second six-axis adjusting platform is used for adjusting the emergent light direction of the optical collimating mirror; the plane mirror is arranged in parallel with the first calibration plate; the optical prism is arranged in parallel with the second calibration plate, and the emergent light of the optical collimating lens passes through the optical prism to reach the first calibration plate. According to the multi-view heterogeneous camera calibration device, the first calibration plate and the second calibration plate which are independent of each other are arranged respectively, and then the plane mirror, the first six-axis adjustment platform, the second six-axis adjustment platform, the optical prism, the optical collimating mirror, the first locking structural member and the second locking structural member are utilized to realize fine adjustment of the relative spatial position relation of the first calibration plate and the second calibration plate, and according to the size of the positioning hole of the optical collimating platform and the size of each checkerboard on the first calibration plate and the second calibration plate, the relative spatial coordinates of each calibration point on the first calibration plate and the second calibration plate can be accurately obtained, so that the external parameter conversion matrix between every two cameras without a common visual angle is accurately obtained, and the accuracy of the calibration result of the multi-view heterogeneous camera is effectively improved.

Drawings

The accompanying drawings, which are included to provide a further understanding of the application and are incorporated in and constitute a part of this application, illustrate embodiments of the application and together with the description serve to explain the application and do not constitute an undue limitation to the application. In the drawings:

fig. 1 is a schematic structural diagram of a calibration device of a multi-view heterogeneous camera according to an embodiment of the present application;

FIG. 2 is a schematic structural view of a planar calibration plate provided according to an embodiment of the present application;

fig. 3 is a schematic structural view of a second locking structure provided in accordance with an embodiment of the present application;

FIG. 4 is a flow chart of an assembly method of a calibration device of a multi-view heterogeneous camera provided according to an embodiment of the present application;

FIG. 5a is a schematic view of the optical path effect of an optical collimator according to an embodiment of the present application;

fig. 5b is a schematic diagram of a second optical path effect of the optical collimator provided in the embodiment of the present application;

FIG. 6 is a flow chart of a calibration method for a multi-view heterogeneous camera provided according to an embodiment of the present application;

fig. 7 is a schematic structural diagram of a computer device according to an embodiment of the present application.

Detailed Description

In order to make the objects, technical solutions and advantages of the present application more apparent, the present application is described and illustrated below with reference to the accompanying drawings and examples. It should be understood that the specific embodiments described herein are for purposes of illustration only and are not intended to limit the present application. All other embodiments, which can be made by one of ordinary skill in the art without undue burden on the person of ordinary skill in the art based on the embodiments provided herein, are intended to be within the scope of the present application.

It is apparent that the drawings in the following description are only some examples or embodiments of the present application, and it is possible for those of ordinary skill in the art to apply the present application to other similar situations according to these drawings without inventive effort. Moreover, it should be appreciated that while such a development effort might be complex and lengthy, it would nevertheless be a routine undertaking of design, fabrication, or manufacture for those of ordinary skill having the benefit of this disclosure, and thus should not be construed as having the benefit of this disclosure.

Reference in the specification to "an embodiment" means that a particular feature, structure, or characteristic described in connection with the embodiment may be included in at least one embodiment of the application. The appearances of such phrases in various places in the specification are not necessarily all referring to the same embodiment, nor are separate or alternative embodiments mutually exclusive of other embodiments. It is to be expressly and implicitly understood by those of ordinary skill in the art that the embodiments described herein can be combined with other embodiments without conflict.

Unless defined otherwise, technical or scientific terms used herein should be given the ordinary meaning as understood by one of ordinary skill in the art to which this application belongs. Reference to "a," "an," "the," and similar terms herein do not denote a limitation of quantity, but rather denote the singular or plural. The terms "comprising," "including," "having," and any variations thereof, are intended to cover a non-exclusive inclusion; for example, a process, method, system, article, or apparatus that comprises a list of steps or modules (elements) is not limited to only those steps or elements but may include other steps or elements not expressly listed or inherent to such process, method, article, or apparatus. The terms "connected," "coupled," and the like in this application are not limited to physical or mechanical connections, but may include electrical connections, whether direct or indirect. The term "plurality" as used herein refers to two or more. "and/or" describes an association relationship of an association object, meaning that there may be three relationships, e.g., "a and/or B" may mean: a exists alone, A and B exist together, and B exists alone. The character "/" generally indicates that the context-dependent object is an "or" relationship. The terms "first," "second," "third," and the like, as used herein, are merely distinguishing between similar objects and not representing a particular ordering of objects.

Fig. 1 is a calibration device 10 of a multi-view heterogeneous camera according to an embodiment of the present application. As shown in fig. 1, the calibration device 10 of the multi-view heterogeneous camera includes a first calibration plate 101, a second calibration plate 102, an optical alignment platform 103, a plane mirror 104, a first six-axis adjustment platform 105, a second six-axis adjustment platform 106, an optical prism 107, an optical alignment mirror 108, a first locking structure 109, and a second locking structure 110.

The sizes of the first calibration plate 101 and the second calibration plate 102 can be adjusted according to actual requirements, so as to meet the requirements of the field of view, the object distance and the depth of field of the camera to be calibrated. As shown in fig. 2, the first calibration plate 101 and the second calibration plate 102 in this embodiment are both planar calibration plates. Further, the plane calibration plate is provided with a plurality of black and white checkerboards, wherein cross points of the black and white checkerboards are calibration points, and the size of the black and white checkerboards can be adjusted according to actual requirements. The first calibration plate 101 is mounted on the optical alignment platform 103 by a first locking structure 109; the first six-axis adjusting platform 105 is mounted on the optical collimating platform 103, the second calibration plate 102 is mounted on the first six-axis adjusting platform 105 through the second locking structural member 110, and the first six-axis adjusting platform 105 is used for adjusting the directions of the second locking structural member 110 and the second calibration plate 102; the first locking structure 109 and the second locking structure 110 are both common mounting means for the fixed mounting of objects. The second six-axis adjusting platform 106 is mounted on the optical collimating platform 103, the optical collimating mirror 108 is mounted on the second six-axis adjusting platform 106, and the second six-axis adjusting platform 106 is used for adjusting the emergent light direction of the optical collimating mirror 108. The first six-axis adjustment platform 105 and the second six-axis adjustment platform 106 belong to precise azimuth adjustment instruments, and can perform direction adjustment from six dimensions, including a first translation direction (X), a second translation direction (Y), a third translation direction (Z), and a first rotation direction (pitch), a second rotation direction (yaw ), and a third rotation direction (roll ). The plane mirror 104 is any mirror that can be used to reflect light, and is used to reflect light projected onto the plane mirror 104, where the plane mirror 104 is parallel to the first calibration plate 101; the optical prism 107 is arranged parallel to the second calibration plate 102, preferably, the optical prism 107 is horizontally placed on the second locking structure 110, so that the optical prism 107 is parallel to the second locking structure 110, that is, the optical prism 107 is parallel to the second calibration plate 102, and the additional installation cost of the optical prism 107 is effectively saved. The light emitted from the light source in the optical collimating lens 108 is emitted in parallel from the light emitting surface of the optical collimating lens 108, and the emitted light of the optical collimating lens 108 reaches the first calibration plate 101 through the optical prism 107.

Because the calibration device 10 of the multi-view heterogeneous camera includes the first calibration plate 101 and the second calibration plate 102, when calibrating the cameras without the common view angle, each camera without the common view angle is required to correspond to one calibration plate, the calibration device 10 of the multi-view heterogeneous camera can calibrate two cameras without the common view angle at a time, if the multi-view heterogeneous camera has a plurality of cameras without the common view angle, the calibration device 10 of the multi-view heterogeneous camera can be utilized to select two cameras without the common view angle at a time to calibrate, and after multiple calibrations, the external parameter conversion matrix between every two cameras without the common view angle is obtained.

In the prior art, by introducing a third-party camera as an intermediate medium, the third-party camera and a camera without a common viewing angle in the multi-camera to be calibrated are required to have a common viewing angle, and an external parameter conversion matrix between the cameras without the common viewing angle in the multi-camera is solved by utilizing the common viewing angle. However, when the depth of field of the camera without the common view angle in the multi-view camera is far different, the coverage area of the third-party camera and the coverage area of the field of the camera without the common view angle in the multi-view camera are far different, and the calibration result has the problem of low accuracy. Based on this, this application is through setting up mutually independent first calibration board 101 and second calibration board 102 respectively, and then utilize plane mirror 104, first six-axis adjustment platform 105, second six-axis adjustment platform 106, optical prism 107, optical collimating mirror 108, first locking structure 109 and second locking structure 110, realize carrying out the fine adjustment to first calibration board 101 and second calibration board 102 relative position relation, and according to the size of the locating hole of optical collimating platform 103 and the size of each checkerboard on first calibration board 101 and the second calibration board 102, can acquire the relative space coordinate of each calibration point on first calibration board 101 and the second calibration board 102 accurately. The relative space coordinates of the calibration points on the first calibration plate 101 and the second calibration plate 102 are accurately obtained, so that the cameras without the common view angle are accurately calibrated, the external parameter conversion matrix between every two cameras without the common view angle is accurately obtained, and the accuracy of the calibration result of the multi-view heterogeneous camera is effectively improved.

With continued reference to fig. 1, in one embodiment, the calibration device 10 of the multi-camera includes a third calibration plate 111, the third calibration plate 111 being mounted on the first six-axis adjustment 105 platform by a second locking structure 110.

Taking the existing helmet type eye tracking device as an example, the helmet type eye tracking device comprises a world camera and two eye tracking cameras, wherein the world camera faces to eyes of people to observe the world azimuth, and the eye tracking cameras face to left and right eyeballs of eyes respectively. There is no common viewing angle between the world camera and the eye tracking camera, and the two eye tracking cameras are the same type of camera and have the same azimuth of viewing angle, but also have no common viewing angle. By arranging the third calibration plate 111 on the calibration device 10 of the multi-eye heterogeneous camera, the calibration device is matched with the structure of the helmet type eye tracking equipment, and can calibrate one world camera and two eye tracking cameras at a time, thereby obtaining the external parameter conversion matrix between the world camera and the two eye tracking cameras.

Further, in one embodiment, the calibration device 10 of the multi-view heterogeneous camera of the present application further includes a positioner. Referring to fig. 3, the second locking structure 110 includes an extension structure 112, and a plurality of positioning holes 113 are formed in the extension structure 112, and are used for installing a positioner.

Among them, the cameras without common view angle in the multi-eye heterogeneous camera may have large differences among the fields of view, object distance and depth of field, and the above-mentioned helmet type eye tracking device is taken as an example, the field of view range of the eye tracking camera is small, the object distance is generally within 30mm, the depth of field is small (generally 19 mm-32 mm), the field of view range of the world camera is large, the object distance is generally outside 50cm, and the depth of field is large (generally 40 cm-120 cm). In order to satisfy the field of view, object distance, and depth of field of a camera without a common viewing angle, the dimensions of the first calibration plate 101 and the second calibration plate 102 are often widely different. Since the first calibration plate 101 and the second calibration plate 102 have large structural dimensions, it is difficult to intuitively determine the relative spatial positional relationship between the first calibration plate 101 and the second calibration plate 102. Due to the presence of the extension structure 112, the plane of the second calibration plate 102 is enlarged, so that the positional relationship between the first calibration plate 101 and the second calibration plate 102 can be observed more clearly. In addition, the spacing between the positioning holes 113 is the same, the positioning holes 113 can be used for installing the positioners, the positioners can be installed on the positioning holes 113 by using screws, and the positioners can be used for accurately positioning the corresponding relation between the points on the plane of the second calibration plate 102 and the points on the plane of the first calibration plate 101 under the condition that the measurement of an instrument is not needed.

As one of the embodiments, the positioner is a spider laser pen. Specifically, because the light emitted by the cross star laser pen is projected onto any plane, a cross star appears, and the projection point of the light emitted by the current cross star laser pen can be accurately known through the position of the cross star.

Fig. 4 is a flowchart of an assembling method of a calibration device of a multi-view heterogeneous camera according to an embodiment of the present application, configured to assemble the calibration device of the multi-view heterogeneous camera, as shown in fig. 4, where the flowchart includes the following steps:

step S210, adjusting the fixing direction of the first locking structural member on the optical collimation platform to enable the first calibration plate to be perpendicular to the optical collimation platform.

In particular, the first calibration plate may be secured perpendicular to the optical alignment platform in conjunction with the measurement of the level during the securing of the first locking structure to the optical alignment platform.

Step S220, the first six-axis adjusting platform is adjusted to enable the second calibration plate to be perpendicular to the optical collimation platform.

Specifically, the first six-axis adjusting platform can be adjusted in six directions, and the second calibration plate can be vertical to the optical collimation platform by adjusting the first six-axis adjusting platform and combining with the measurement of the level meter.

Step S230, adjusting a second six-axis adjusting platform and translating the plane mirror to enable an emergent light path of the optical collimating mirror to coincide with an incident light path after the emergent light of the optical collimating mirror is reflected by the plane mirror;

specifically, the plane mirror is arranged in parallel with the first calibration plate, and can translate along the direction of the plane where the first calibration plate is located. As one of the embodiments, the plane mirror can be translated against the first calibration plate. The outgoing target surface of the optical collimating mirror is provided with a cross star, outgoing light of the optical collimating mirror is projected onto the plane mirror through the cross star, and is reflected by the plane mirror and then is incident onto the outgoing target surface of the optical collimating mirror again, and if the cross star of the outgoing light path coincides with the cross star of the incident light path, the outgoing light path and the incident light path can be ensured to coincide.

Fig. 5a and 5b are schematic diagrams of an optical path effect of the optical collimating lens according to the embodiment of the present application, fig. 5a is an effect that an outgoing optical path of the optical collimating lens does not coincide with an incoming optical path, and fig. 5b is an effect that an outgoing optical path of the optical collimating lens coincides with an incoming optical path. When the emergent light of the optical collimating mirror is reflected by the plane mirror, the emergent light path coincides with the incident light path, and at the moment, the direction of the light emitted by the optical collimating mirror is vertical to the plane mirror, namely, the direction of the light emitted by the optical collimating mirror is vertical to the first calibration plate.

Step S240, the first six-axis adjusting platform is adjusted, so that emergent light of the optical collimating lens is projected onto the plane lens through the optical prism, and after being reflected by the plane lens, an emergent light path is overlapped with an incident light path.

Specifically, when the emergent light of the optical collimating mirror is projected onto the plane mirror through the optical prism, after being reflected by the plane mirror, the emergent light path coincides with the incident light path, so that the direction of the light emitted by the optical collimating mirror is ensured to be vertical to the plane mirror and the optical prism at the same time. Since the optical prism is parallel to the second calibration plate, the direction of the light emitted by the optical collimating mirror is perpendicular to the plane mirror and the second calibration plate at the same time, i.e. the direction of the light emitted by the optical collimating mirror is perpendicular to the first calibration plate and the second calibration plate at the same time. Because the direction of the light emitted by the optical collimating lens is vertical to the first calibration plate and the second calibration plate at the same time, the first calibration plate and the second calibration plate are parallel according to the position relation between the mathematical center line and the surface.

Through the steps S210 to S240, the first calibration plate and the first calibration plate can be precisely adjusted to be perpendicular to the optical alignment platform at the same time, and the first calibration plate and the second calibration plate are parallel to each other.

Still further, in one embodiment, the method for assembling the calibration device of the multi-view heterogeneous camera further includes the steps of:

Step S250, the positioner is arranged on one of the positioning and aligning holes on the extending structure of the second locking structure, and the plane mirror is translated, so that the emergent light of the positioner is projected onto the plane mirror.

Step S260, adjusting the screw on the positioning and aligning hole to enable the light path of the emergent light of the positioner to coincide with the light path reflected by the plane mirror.

Specifically, the number of turns of the screw on the positioning alignment hole can be adjusted to adjust the outgoing light direction of the positioner. When the light path of the emergent light of the positioner coincides with the light path reflected by the plane mirror, the direction of the emergent light of the positioner is perpendicular to the plane mirror, the direction of the emergent light of the positioner can be determined to be perpendicular to the first calibration plate and the second calibration plate at the same time, and at the moment, the point on the first calibration plate can be accurately determined from the optical angle to correspond to the current positioning alignment hole. According to the corresponding relation, the size of each checkerboard on the first calibration plate and the size of each locating and locating hole on the second calibration plate and the size between every two locating and locating holes, the relative space coordinates of each calibration point on the first calibration plate and each calibration point on the second calibration plate can be accurately obtained. After the space coordinates of the calibration points on the first calibration plate are obtained, the coordinates of the calibration points on the second calibration plate can be obtained according to the relative space coordinates.

The method for assembling the calibration device of the multi-view heterogeneous camera is realized by adjusting the calibration device 10 of the multi-view heterogeneous camera, specifically, the adjustment can be performed by an automatic mechanical arm, or the adjustment can be performed manually, which is not limited herein.

In one embodiment, the multi-view heterogeneous camera includes at least two cameras without a common field of view, and the calibration method of the multi-view heterogeneous camera is implemented based on the assembly method of the calibration device of the multi-view heterogeneous camera, as shown in fig. 6, and includes the following steps:

step S310, obtaining a calibration plate image shot by a camera without a public view field; each camera without a public view field independently shoots a corresponding plane calibration plate, and calibration points are arranged on the calibration plates;

step S320, determining the image coordinates of the calibration points on the calibration plate image;

step S330, obtaining an initial external parameter conversion matrix between every two cameras without a common field of view according to the internal parameters of each camera without the common field of view, the image coordinates of the calibration points on the images of the calibration plates and the world coordinates of the calibration points.

Specifically, after the calibration device of the multi-eye heterogeneous camera is assembled by the assembly method of the calibration device of the multi-eye heterogeneous camera, the first calibration plate and the second calibration plate can be ensured to be perpendicular to the optical collimation platform at the same time, and the first calibration plate and the second calibration plate are parallel to each other. And setting the cameras without the common field of view in the field of view of the corresponding calibration plates, so that the calibration plate images shot by the cameras without the common field of view can be acquired. Preferably, in order to accurately acquire the coordinates of each calibration point on the calibration plate image and to have a plurality of groups of calibration points as references, before the calibration plate image shot by the camera without the common field of view is acquired, the heterogeneous multi-camera is adjusted, so that the calibration plate image shot by the camera without the common field of view is clear and the calibration plate occupies more than 70% of the calibration plate image. The camera internal references refer to a transformation matrix that transforms coordinates on an image coordinate system into coordinates in a camera coordinate system, and the camera internal references of each camera without a common field of view can be obtained using existing camera internal reference calibration methods. Assuming that cameras without common fields of view exist in the multi-view heterogeneous cameras as cameras A1 and B respectively, camera internal references obtained according to the existing camera internal reference calibration method are K respectively _A1 And K _B . According to camera internal reference K _A1 And K _B And calibrating the image coordinates of the calibration points on the plate image to obtain the coordinates of the calibration points under the camera coordinate system of each camera without the common field of view.

The camera external parameters refer to a transformation matrix for transforming the coordinates in the camera coordinate system into the coordinates in the world coordinate system, and the camera external parameters of each camera without the common field of view can be obtained according to the coordinates of each calibration point in the camera coordinate system of each camera without the common field of view and the world coordinates of each calibration point

And->

Wherein A1 and B are coordinates in a camera coordinate system, and W is coordinates in a world coordinate system. As one implementation mode, a world coordinate system is established at any point in space, and world coordinates of each calibration point on the first calibration plate and the second calibration plate can be obtained through the distance between each positioning hole on the optical collimation platform, the size of the checkerboard in the first calibration plate and the second calibration plate and additional measurement. Preferably, by the above assembling method of the calibration device for the multi-view heterogeneous camera, the relative spatial coordinates of each calibration point on the second calibration plate and each calibration point on the first calibration plate are already obtained, and after the world coordinates of each calibration point on the first calibration plate are obtained, the world coordinates of each calibration point on the second calibration plate are obtained according to the relative spatial coordinates, so that the relative coordinates with high precision between the first calibration plate and the second calibration plate can be constructed. Due to- >

At->

And->

Under certain conditions, the initial extrinsic transformation matrix between cameras A1 and B can be obtained by mathematical methods>

In addition, if the heterogeneous multi-camera includes more than two cameras without a common viewing angle, for example, there is no common viewing angle between A2 and B, the cameras A2 and B may be calibrated again through steps S310 to S330, so as to obtain an initial extrinsic transformation matrix between A2 and B

After obtaining->

And->

After that, according to->

The initial external parameter conversion matrix between A1 and A2 can be obtained>

Therefore, no matter how many heterogeneous cameras include cameras without a common viewing angle, the initial extrinsic transformation matrix between every two cameras without a common viewing angle can be accurately obtained through the steps S310 to S330.

As one embodiment, taking the above helmet-type eye tracking device as an example, the third calibration board may be fully utilized, and the calibration board images captured by the world camera and the two eye tracking cameras may be obtained at the same time, so that the helmet-type eye tracking device may be calibrated once through steps S310 to S330, and the initial external parameter transformation matrix between the world camera and the two eye tracking cameras may be obtained.

Further, the step S330 obtains an initial external parameter conversion matrix between every two cameras without a common field of view according to the internal parameters of each camera without a common field of view, the image coordinates of the calibration points on the images of the calibration plates and the world coordinates of the calibration points, and the method comprises the following steps:

Step S331, setting up a world coordinate system by taking a calibration point on one of the calibration plates as an origin, and obtaining world coordinates of each calibration point;

step S332, obtaining an initial external parameter conversion matrix between every two cameras without a common field of view according to the internal parameters of the cameras without the common field of view, the image coordinates of the calibration points on the images of the calibration plates and the world coordinates of the calibration points.

Specifically, the world coordinate system is established by taking the calibration point on one of the calibration plates as the origin, and the world coordinates of the calibration points on each calibration plate can be accurately obtained according to the distance between the positioning holes on the optical collimation platform and the size of the checkerboard in the first calibration plate and the second calibration plate without additional measurement. The world coordinates of the first calibration plate and the second calibration plate are more accurate, so that the initial external parameter conversion matrix between every two cameras without a common field of view is more accurate to solve.

As one embodiment, the calibration method of the multi-view heterogeneous camera further includes the following steps:

step S340, setting preset times, and indicating the multi-view heterogeneous camera to perform pose change of the preset times, wherein each time the multi-view heterogeneous camera performs pose change, a calibration plate image shot by the camera without a public view field is obtained;

Step S350, constructing a linear constraint condition according to the acquired image coordinates of the calibration points on all the calibration plate images and the world coordinates of each calibration point;

and step S360, performing iterative computation on the linear constraint conditions by using an optimization method, so as to obtain an optimal external parameter conversion matrix between every two cameras without a common field of view.

Specifically, for a scene where the accuracy requirement is not particularly high, the initial extrinsic transformation matrix obtained through the above steps S310 to S330 is fully satisfied with the actual requirement. In order to pursue higher precision, pose transformation is required to be carried out on the multi-view heterogeneous camera, calibration plate images under multiple poses are obtained, linear constraint conditions are built according to image coordinates of calibration points of the calibration plate images under the multiple poses and world coordinates of all the calibration points, and iterative calculation is carried out on the linear constraint conditions by using an optimization method, so that an optimal external parameter conversion matrix between every two cameras without a common field of view is obtained. The pose conversion times can be set in advance according to actual requirements, namely, the preset times are set. For example, the preset number of times is N, the cameras without common view fields are A1 and B respectively, then the acquired N groups of calibration plate image pairs are { Img }, respectively _A1 ，Img _B } ⁱ I represents different pose serial numbers, and the value range is i epsilon [1, N]. Let the camera B acquire an image Img in the ith pose _B The kth mark point in (a)Image coordinates of (a) are as follows

D represents a first calibration plate, and k represents a value range k epsilon [1, Q]Q represents the number of marking points on the second calibration plate. Image Img acquired by camera A1 in the ith pose _A1 The image coordinate of the j-th mark point in (a) is +.>

C1 represents a second calibration plate, and j represents a value range j epsilon [1, M]M represents the number of calibration points on the first calibration plate; the following linear constraints are constructed:

lambda in the formula ₁ 、λ ₂ 、λ ₃ The penalty coefficients of the items are set to fixed values, for example 17, 1 and 10, respectively. Lambda (lambda) ₁ 、λ ₂ Balancing the minimum resolution precision of cameras A1 and B, respectively; lambda (lambda) ₃ Is a value preset empirically. ΔP _C1,j The absolute difference between the theoretical three-dimensional coordinate and the actual three-dimensional coordinate of the jth calibration point on the second calibration plate is represented, and the initial value is 0; ΔP _D,k The absolute difference between the theoretical three-dimensional coordinates and the actual three-dimensional coordinates of the kth calibration point on the first calibration plate is shown as 0.

Is->

Is>

Is->

Is between the projected pixel coordinates and the back projected pixel coordinates The camera internal parameter conversion formula is satisfied. Iteration is carried out on the linear constraint condition by utilizing an optimization method, the iteration process is required to be compared with a set threshold according to the accumulated sum of the last iteration result and the errors of the projection pixel coordinates and the back projection pixel coordinates, and if the iteration result is larger than the set threshold, the penalty coefficient lambda is properly amplified ₁ And lambda (lambda) ₃ General lambda ₃ And preferentially amplifying, and stopping iteration until the sum of error accumulation of the projection pixel coordinates and the back projection pixel coordinates is smaller than a set threshold value. And finally, obtaining the optimal value of the params as the final result of iteration, and obtaining the optimal external parameter conversion matrix between the cameras A1 and B without the common view field.

In one embodiment, the optimization method includes a Levenberg-Marquard algorithm or Newton's iteration method.

In one embodiment, a computer device is provided, which may be a server, the internal structure of which may be as shown in fig. 7. The computer device includes a processor, a memory, a network interface, and a database connected by a system bus. Wherein the processor of the computer device is configured to provide computing and control capabilities. The memory of the computer device includes a non-volatile storage medium and an internal memory. The non-volatile storage medium stores an operating system, computer programs, and a database. The memory provides an environment for the operation of the operating system and computer programs in the non-volatile storage media. The database of the computer device is used for storing a preset configuration information set. The network interface of the computer device is used for communicating with an external terminal through a network connection. The computer program, when executed by a processor, implements the calibration method of the multi-view heterogeneous camera.

In one embodiment, a computer device is provided, which may be a terminal. The computer device includes a processor, a memory, a network interface, a display screen, and an input device connected by a system bus. Wherein the processor of the computer device is configured to provide computing and control capabilities. The memory of the computer device includes a non-volatile storage medium and an internal memory. The non-volatile storage medium stores an operating system and a computer program. The memory provides an environment for the operation of the operating system and computer programs in the non-volatile storage media. The network interface of the computer device is used for communicating with an external terminal through a network connection. The computer program, when executed by a processor, implements a method for calibrating a multi-view heterogeneous camera. The display screen of the computer equipment can be a liquid crystal display screen or an electronic ink display screen, and the input device of the computer equipment can be a touch layer covered on the display screen, can also be keys, a track ball or a touch pad arranged on the shell of the computer equipment, and can also be an external keyboard, a touch pad or a mouse and the like.

It will be appreciated by those skilled in the art that the structure shown in fig. 7 is merely a block diagram of some of the structures associated with the present application and is not limiting of the computer device to which the present application may be applied, and that a particular computer device may include more or fewer components than shown, or may combine certain components, or have a different arrangement of components.

In one embodiment, a computer readable storage medium is provided having a computer program stored thereon, which when executed by a processor, performs the steps of:

acquiring a calibration plate image shot by a camera without a public view field; each camera without a public view field independently shoots a corresponding plane calibration plate, and calibration points are arranged on the calibration plates;

determining image coordinates of a calibration point on the calibration plate image;

and obtaining an initial external parameter conversion matrix between every two cameras without the common field of view according to the camera internal parameters of each camera without the common field of view, the image coordinates of the calibration points on the images of the calibration plates and the world coordinates of the calibration points.

In one embodiment, the processor when executing the computer program further performs the steps of:

establishing a world coordinate system by taking a calibration point on one of the calibration plates as an origin, and acquiring world coordinates of each calibration point;

and obtaining an initial external parameter conversion matrix between every two cameras without the common field of view according to the internal parameters of the cameras without the common field of view, the image coordinates of the calibration points on the images of the calibration plates and the world coordinates of the calibration points.

In one embodiment, the processor when executing the computer program further performs the steps of:

Setting preset times, and indicating the multi-eye heterogeneous camera to perform pose change of the preset times, wherein each time the multi-eye heterogeneous camera performs pose change, a calibration plate image shot by the camera without a public view field is obtained;

constructing a linear constraint condition according to the acquired image coordinates of the calibration points on all the calibration plate images and the world coordinates of all the calibration points;

and carrying out iterative computation on the linear constraint conditions by using an optimization method, thereby obtaining an optimal external parameter conversion matrix between every two cameras without a common view field.

In one embodiment, the optimization method includes a Levenberg-Marquard algorithm or Newton's iterative method.

According to the storage medium, after the relative spatial position relation of the first calibration plate and the second calibration plate is finely adjusted, and the relative spatial coordinates of all positioning points on the first calibration plate and the second calibration plate are accurately obtained, the initial external parameter conversion matrix between every two cameras without a common field of view is obtained by utilizing the camera internal parameters of the cameras without the common field of view, the image coordinates of the calibration points on the images of all the calibration plates and the world coordinates of all the calibration points. Because the relative space coordinates of the calibration points on the first calibration plate and the second calibration plate are accurate, the accuracy of the calibration result of the multi-view heterogeneous camera is effectively improved.

It should be understood that the specific embodiments described herein are merely illustrative of this application and are not intended to be limiting. All other embodiments, which can be made by one of ordinary skill in the art without undue burden from the present application, are within the scope of the present application in light of the embodiments provided herein.

It is evident that the drawings are only examples or embodiments of the present application, from which the present application can also be adapted to other similar situations by a person skilled in the art without the inventive effort. In addition, it should be appreciated that while the development effort might be complex and lengthy, it would nevertheless be a routine undertaking of design, fabrication, or manufacture for those of ordinary skill having the benefit of this disclosure, and thus should not be construed as an admission of insufficient detail.

The term "embodiment" in this application means that a particular feature, structure, or characteristic described in connection with the embodiment may be included in at least one embodiment of the application. The appearances of such phrases in various places in the specification are not necessarily all referring to the same embodiment, nor are separate or alternative embodiments mutually exclusive. It will be clear or implicitly understood by those of ordinary skill in the art that the embodiments described in this application can be combined with other embodiments without conflict.

The above examples only represent a few embodiments of the present application, which are described in more detail and are not to be construed as limiting the scope of the patent. It should be noted that it would be apparent to those skilled in the art that various modifications and improvements could be made without departing from the spirit of the present application, which would be within the scope of the present application. Accordingly, the scope of protection of the present application shall be subject to the appended claims.

Claims (11)

1. The calibrating device of the multi-eye heterogeneous camera is characterized by comprising a first calibrating plate, a second calibrating plate, an optical collimating platform, a plane mirror, a first six-axis adjusting platform, a second six-axis adjusting platform, an optical prism, an optical collimating mirror, a first locking structural member and a second locking structural member;

the first calibration plate is arranged on the optical collimation platform through the first locking structural member;

the first six-axis adjusting platform is arranged on the optical collimation platform, the second calibration plate is arranged on the first six-axis adjusting platform through the second locking structural member, and the first six-axis adjusting platform is used for adjusting the directions of the second locking structural member and the second calibration plate;

The second six-axis adjusting platform is arranged on the optical collimating platform, the optical collimating mirror is arranged on the second six-axis adjusting platform, and the second six-axis adjusting platform is used for adjusting the emergent light direction of the optical collimating mirror;

the plane mirror is arranged in parallel with the first calibration plate; the optical prism is arranged in parallel with the second calibration plate; the emergent light of the optical collimating lens penetrates through the optical prism to reach the first calibration plate.

2. The calibration device of a multi-camera according to claim 1, further comprising a third calibration plate mounted on the first six-axis adjustment platform by the second locking structure.

3. Calibration device of a multi-view heterogeneous camera according to claim 1 or 2, characterized in that the device further comprises a locator;

the second locking structural member comprises an extension structure, wherein a plurality of positioning and aligning holes are formed in the extension structure and used for installing the positioner.

4. A calibration device for a multi-view heterogeneous camera according to claim 3, wherein the positioner is a spider laser pen.

5. A method for assembling a calibration device for a multi-view heterogeneous camera according to any one of claims 1 to 4, comprising:

adjusting the fixing direction of the first locking structural member on the optical collimation platform to enable the first calibration plate to be perpendicular to the optical collimation platform;

adjusting the first six-axis adjusting platform to enable the second calibration plate to be perpendicular to the optical collimation platform;

adjusting a second six-axis adjusting platform and translating a plane mirror to enable an emergent light path of the optical collimating mirror to coincide with an incident light path after the emergent light is reflected by the plane mirror;

and adjusting the first six-axis adjusting platform to enable emergent light of the optical collimating lens to penetrate through the optical prism and be projected onto the plane lens, and enabling an emergent light path to coincide with an incident light path after being reflected by the plane lens.

6. The method of assembling a calibration device for a multi-view heterogeneous camera of claim 5, further comprising:

installing a positioner on one of positioning and aligning holes on an extending structure of the second locking structural member, and translating the plane mirror to enable emergent light of the positioner to be projected onto the plane mirror;

And adjusting the screw on the positioning and aligning hole to enable the light path of the emergent light of the positioner to coincide with the light path reflected by the plane mirror.

7. A method for calibrating a multi-view heterogeneous camera, the multi-view heterogeneous camera comprising at least two cameras without a common field of view, the method for calibrating a multi-view heterogeneous camera being implemented based on the method for assembling a calibration device for a multi-view heterogeneous camera according to any of the preceding claims 5 or 6, the method comprising:

acquiring a calibration plate image shot by a camera without a public view field; each camera without the public view field independently shoots a corresponding plane calibration plate, and the calibration plates are provided with calibration points;

determining image coordinates of a calibration point on the calibration plate image;

and obtaining an initial external parameter conversion matrix between every two cameras without the common field of view according to the camera internal parameters of each camera without the common field of view, the image coordinates of the calibration points on the images of the calibration plates and the world coordinates of the calibration points.

8. The method for calibrating a multi-view heterogeneous camera according to claim 7, wherein the obtaining the initial extrinsic transformation matrix between every two cameras without a common field of view according to the camera intrinsic parameters of each camera without a common field of view, the image coordinates of the calibration points on the images of the calibration plates and the world coordinates of the calibration points comprises the following steps:

Establishing a world coordinate system by taking a calibration point on one of the calibration plates as an origin, and acquiring world coordinates of each calibration point;

and obtaining an initial external parameter conversion matrix between every two cameras without the common field of view according to the camera internal parameters of the cameras without the common field of view, the image coordinates of the calibration points on the images of the calibration plates and the world coordinates of the calibration points.

9. The method of calibrating a multi-view heterogeneous camera of claim 7, further comprising:

setting preset times, and indicating the multi-view heterogeneous camera to perform pose change of the preset times, wherein the multi-view heterogeneous camera obtains a calibration plate image shot by the camera without the public view field once every time the multi-view heterogeneous camera performs pose change;

constructing a linear constraint condition according to the acquired image coordinates of the calibration points on the calibration plate image and the world coordinates of each calibration point;

and carrying out iterative computation on the linear constraint conditions by using an optimization method, thereby obtaining the optimal external parameter conversion matrix between every two cameras without the public view field.

10. The method of calibrating a multi-view heterogeneous camera according to claim 9, wherein the optimization method comprises a Levenberg-Marquard algorithm or newton's iteration method.

11. A computer readable storage medium, on which a computer program is stored, characterized in that the computer program, when being executed by a processor, implements the steps of the method of any of claims 7 to 10.

Images