CN117372543A

CN117372543A - Camera calibration method and device, calibration plate and electronic equipment

Info

Publication number: CN117372543A
Application number: CN202311343907.3A
Authority: CN
Inventors: 唐修东
Original assignee: Vivo Mobile Communication Co Ltd
Current assignee: Vivo Mobile Communication Co Ltd
Priority date: 2023-10-17
Filing date: 2023-10-17
Publication date: 2024-01-09

Abstract

The application discloses a camera calibration method, a camera calibration device, a calibration plate and electronic equipment, and belongs to the technical field of camera calibration. The calibration method comprises the following steps: according to the calibration image of the calibration plate shot by the camera, determining the size information, the coordinate information and the shape information of the feature points in the calibration image; determining a marking point and a corner point set from the characteristic points according to the size information, the coordinate information and the shape information; determining a feature matrix according to world coordinates of the mark points and image coordinates of the mark points in the calibration image; determining a calibration result of the camera according to the feature matrix, the world coordinates of the corner sets and the image coordinates of the corner sets; under the condition that the camera is a first type camera, the marking points and the corner sets comprise feature points with the smallest size in the calibration plate; under the condition that the camera is a second type camera, the marking points and the corner point sets do not comprise the feature points with the smallest size in the calibration plate; the resolution of the first type of camera is greater than the resolution of the second type of camera.

Description

Camera calibration method and device, calibration plate and electronic equipment

Technical Field

The application belongs to the technical field of camera calibration, and particularly relates to a camera calibration method, a camera calibration device, a calibration plate and electronic equipment.

Background

Currently, with the development of image technology, the number and variety of cameras are increasing. Based on this, besides the currently mainstream multi-RGB camera system, TOF (Time of Flight, depth perception technology based on the Time of Flight principle) cameras are adopted by more and more manufacturers, especially, the combination of TOF cameras and RGB cameras is adopted by more and more manufacturers, and on this basis, the combined calibration of the RGB cameras and the TOF cameras becomes increasingly important.

However, the calibration requirements for RGB cameras and TOF cameras are different. When calibrating an RGB camera, a calibration plate usually used needs to have enough feature blocks or feature points because of the distortion parameters of the camera to be calibrated. When the TOF camera is calibrated, the single feature block or feature point in the calibration plate shot by the TOF camera is required to be significant enough due to low resolution and high noise of the TOF camera, otherwise, the calibration algorithm cannot extract the features.

On the basis, when the RGB camera and the TOF camera are subjected to combined calibration, different calibration plates are required to be used for calibration according to the characteristics of different cameras, so that the calibration efficiency of the cameras is reduced, and the calibration cost of the cameras is increased.

Disclosure of Invention

The embodiment of the application aims to provide a camera calibration method, a camera calibration device, a calibration plate and electronic equipment, which can improve the calibration efficiency of a camera and reduce the calibration cost of the camera.

In a first aspect, an embodiment of the present application provides a camera calibration method, including: according to the calibration image of the calibration plate shot by the camera, determining the size information, the coordinate information and the shape information of the feature points in the calibration image; determining a mark point from the feature points according to the size information, the coordinate information and the shape information, and determining a corner set from the feature points according to the mark point; determining a feature matrix according to world coordinates of the mark points and image coordinates of the mark points in the calibration image; determining a calibration result of the camera according to the feature matrix, the world coordinates of the corner sets and the image coordinates of the corner sets in the calibration image; under the condition that the camera is a first type camera, the marking points and the corner sets comprise feature points with the smallest size in the calibration plate; under the condition that the camera is a second type camera, the marking points and the corner point sets do not comprise the feature points with the smallest size in the calibration plate; the resolution of the first type of camera is greater than the resolution of the second type of camera.

In a second aspect, an embodiment of the present application provides a camera calibration device, including: the processing unit is used for determining the size information, the coordinate information and the shape information of the characteristic points in the calibration image according to the calibration image of the calibration plate shot by the camera; the processing unit is also used for determining mark points from the feature points according to the size information, the coordinate information and the shape information, and determining a corner set from the feature points according to the mark points; the processing unit is also used for determining a feature matrix according to the world coordinates of the mark points and the image coordinates of the mark points in the calibration image; the processing unit is also used for determining the calibration result of the camera according to the feature matrix, the world coordinates of the corner sets and the image coordinates of the corner sets in the calibration image; under the condition that the camera is a first type camera, the marking points and the corner sets comprise feature points with the smallest size in the calibration plate; under the condition that the camera is a second type camera, the marking points and the corner point sets do not comprise the feature points with the smallest size in the calibration plate; the resolution of the first type of camera is greater than the resolution of the second type of camera.

In a third aspect, embodiments of the present application provide a calibration plate, including: a first set of feature points, the feature points in the first set of feature points being used as marker points for the first type of camera; the second characteristic point set is used as a mark point of the second type camera; the characteristic points in the third characteristic point set are used for determining the angular point set of the first type or the second type of cameras; the first feature point concentration feature point has a first size, the second feature point concentration feature point has a second size, and the first size is smaller than the second size; the size of the feature points in the third feature point set is less than or equal to the first size.

In a fourth aspect, embodiments of the present application provide an electronic device comprising a processor and a memory storing programs or instructions executable on the processor, which when executed by the processor implement the steps of the camera calibration method as in the first aspect.

In a fifth aspect, embodiments of the present application provide a readable storage medium having stored thereon a program or instructions which, when executed by a processor, implement the steps of the camera calibration method as in the first aspect.

In a sixth aspect, embodiments of the present application provide a chip comprising a processor and a communication interface, the communication interface and the processor being coupled, the processor being configured to execute programs or instructions for implementing the steps of the camera calibration method as in the first aspect.

In a seventh aspect, embodiments of the present application provide a computer program product stored in a storage medium, the program product being executable by at least one processor to perform the steps of the camera calibration method as in the first aspect.

In the camera calibration method provided by the embodiment of the application, the size information, the coordinate information and the shape information of the feature points in the calibration image are determined according to the calibration image of the calibration plate shot by the camera; determining a mark point from the feature points according to the size information, the coordinate information and the shape information, and determining a corner set from the feature points according to the mark point; determining a feature matrix according to world coordinates of the mark points and image coordinates of the mark points in the calibration image; and determining a calibration result of the camera according to the feature matrix, the world coordinates of the corner sets and the image coordinates of the corner sets in the calibration image. Under the condition that the camera is a first type camera, the marking points and the corner sets comprise feature points with the smallest size in the calibration plate; under the condition that the camera is a second type camera, the marking points and the corner point sets do not comprise the feature points with the smallest size in the calibration plate; the resolution of the first type of camera is greater than the resolution of the second type of camera. According to the camera calibration method, the characteristic points with different coordinates, sizes and shapes are arranged on the calibration plate, in the process of calibrating the camera by using the calibration plate, the mark points and the corner sets corresponding to the type of the camera are selected from the characteristic points of the calibration image according to the size information, the shape information and the coordinate information of the characteristic points in the calibration image shot by the camera, and the camera is calibrated based on the selected mark points and the corner sets. Therefore, in the process of calibrating the cameras, the calibration schemes of the cameras of different types are compatible, the calibration of the cameras of different types can be realized only through one calibration plate, and the calibration cost of the cameras can be reduced while the calibration efficiency of the cameras is improved.

Drawings

Fig. 1 is a flow chart of a camera calibration method according to an embodiment of the present application;

fig. 2 is one of schematic diagrams of a camera calibration method according to an embodiment of the present application;

FIG. 3 (a) is a second schematic diagram of a camera calibration method according to an embodiment of the present disclosure;

FIG. 3 (b) is a third schematic diagram of a camera calibration method according to an embodiment of the present disclosure;

FIG. 4 is a schematic diagram of a camera calibration method according to an embodiment of the present disclosure;

FIG. 5 is a schematic diagram of a camera calibration method according to an embodiment of the present disclosure;

FIG. 6 is a schematic diagram of a camera calibration method according to an embodiment of the present disclosure;

FIG. 7 (a) is a schematic diagram of a camera calibration method according to an embodiment of the present disclosure;

FIG. 7 (b) is a schematic diagram eighth of a camera calibration method according to an embodiment of the present application;

fig. 8 is a schematic diagram of a camera calibration method according to an embodiment of the present application;

FIG. 9 (a) is a schematic diagram of a camera calibration method according to an embodiment of the present application;

FIG. 9 (b) is an eleventh schematic diagram of a camera calibration method according to an embodiment of the present application;

FIG. 10 is a block diagram of a camera calibration device according to an embodiment of the present disclosure;

Fig. 11 is a block diagram of an electronic device according to an embodiment of the present application;

fig. 12 is a schematic hardware structure of an electronic device according to an embodiment of the present application.

Detailed Description

Technical solutions in the embodiments of the present application will be clearly described below with reference to the drawings in the embodiments of the present application, and it is apparent that the described embodiments are some embodiments of the present application, but not all embodiments. All other embodiments obtained by a person of ordinary skill in the art based on the embodiments in the present application are within the scope of the protection of the present application.

The terms first, second and the like in the description and in the claims, are used for distinguishing between similar objects and not necessarily for describing a particular sequential or chronological order. It is to be understood that the terms so used are interchangeable under appropriate circumstances such that the embodiments of the application are capable of operation in sequences other than those illustrated or otherwise described herein, and that the objects identified by "first," "second," etc. are generally of a type and do not limit the number of objects, for example, the first object may be one or more. Furthermore, in the description and claims, "and/or" means at least one of the connected objects, and the character "/", generally means that the associated object is an "or" relationship.

The camera calibration method provided by the embodiment of the application is described in detail through specific embodiments and application scenes thereof with reference to the accompanying drawings.

In the related art, besides the currently mainstream multi-RGB camera system, TOF sensors are adopted by more and more manufacturers. The combination of the TOF sensor and the RGB sensor brings promotion to various fields of images, and particularly in the field of 3D images, the combination effect of the TOF sensor and the RGB sensor is more remarkable. Specifically, on the currently released head display and mobile phone, the TOF sensor is an indispensable device, the combination of the TOF sensor and the RGB sensor can bring great improvement to a plurality of traditional image fields such as blurring, focusing and the like, and the combination effect of the TOF sensor and the RGB sensor is more prominent in the aspects of 3D image, space calculation and the like. Based on this, joint calibration of RGB cameras and TOF cameras becomes particularly important. However, the requirements of the RGB camera and the TOF camera on the characteristic points in the calibration process are different, and different calibration plates are required to be used for calibration respectively, so that the calibration cost is increased. Therefore, the embodiment of the application provides a camera calibration method based on the calibration plate, which can realize the joint calibration of the RGB camera and the TOF camera through one calibration plate, thereby reducing the calibration cost.

As shown in fig. 1, an embodiment of the present application provides a camera calibration method, which may include the following steps S102 to S108:

s102: and determining the size information, the coordinate information and the shape information of the feature points in the calibration image according to the calibration image of the calibration plate shot by the camera.

The cameras can specifically include a first type of camera and a second type of camera. The resolution of the first type of camera is greater than the resolution of the second type of camera.

In an actual application process, the first type of camera may be a high-resolution camera such as an RGB camera, and the second type of camera may be a low-resolution camera such as a TOF camera, which is not limited herein.

Further, feature points of different coordinates, sizes and shapes are arranged in the calibration plate.

It will be appreciated that in calibrating a camera by means of the feature points in the calibration plate, the effect of the feature points in the calibration plate may include two types: one is mark points, namely mark points, used for calculating an initial feature matrix of the camera, and the other is common angular points used for iteratively fitting a final calibration result of the camera. In addition, for cameras with different resolutions, namely a first type camera and a second type camera, in the process of calibrating the cameras through the characteristic points in the calibration plate, the requirements of the cameras with different resolutions on the characteristic points are different.

Specifically, for mark points, i.e., marker points, used for calculating the initial feature matrix of the camera, since the resolution of the second type camera, e.g., the TOF camera, is low, for an image of the same content, the number of pixels occupied by each feature point in the TOF image is far smaller than that of the RGB image, and the calibration algorithm cannot successfully extract the feature points lower than a certain number of pixels, therefore, the mark points, i.e., marker points, used for calculating the initial feature matrix of the second type camera, e.g., the TOF camera, need to be larger than the outline of the mark points, i.e., marker points, used for calculating the initial feature matrix of the first type camera, e.g., the RGB camera.

Further, for normal corner points used for iteratively fitting the final calibration result of the cameras, since the first type of cameras, such as RGB cameras, require calibration distortion parameters and the RGB images require enough feature points to fit accurate distortion parameters, a sufficient number of normal corner points must be provided in the calibration plate when the first type of cameras, such as RGB cameras, are calibrated by the calibration plate. Further, since the resolution of the second type of camera, such as a TOF camera, is low, the number of common corner points used to fit the final calibration result of the TOF camera needs to be large enough, and since the second type of camera, such as a TOF camera, does not need to calibrate distortion parameters, the number of common corner points used to fit the final calibration result of the TOF camera may be small relative to the number of common corner points used to fit the final calibration result of the first type of camera, such as an RGB camera.

On the basis, in the camera calibration method provided by the embodiment of the application, the characteristic points with different coordinates, sizes and shapes are arranged in the calibration plate, and the acquisition results of the characteristic points in the calibration plate by the cameras with different types are different, so that the characteristic points are provided for the calibration of the cameras with different types through the calibration plate.

Specifically, in the camera calibration method provided by the embodiment of the application, the calibration plate includes a first feature point set, a second feature point set and a third feature point set. Wherein the feature points in the first feature point set are used as the marker points for a first type of camera, such as an RGB camera, the feature points in the second feature point set are used as the marker points for a second type of camera, such as a TOF camera, and the feature points in the third feature point set are used to determine the set of corner points for the first type or the second type of camera. Further, the size of the feature points in the first feature point set is a first size, the size of the feature points in the second feature point set is a second size, the first size is smaller than the second size, and the size of the feature points in the third feature point set is smaller than or equal to the first size.

Specifically, in the camera calibration method provided in the embodiment of the present application, the first feature point set includes a first feature point and a second feature point, where the shape of the first feature point is a first shape, the shape of the second feature point is a second shape, and the number of the second feature points is smaller than that of the first feature point. Further, the second feature point set includes third feature points and fourth feature points, wherein the shape of the third feature points is the first shape, the shape of the fourth feature points is the second shape, and the number of the fourth feature points is smaller than the number of the third feature points. The third feature point set includes first feature points and fifth feature points, the shape of the fifth feature points is the first shape, the size of the fifth feature points is the third size, the third size is smaller than the first size, and the number of the fifth feature points in the third feature point set is larger than the number of the first feature points.

On the basis, the size characteristics and the number characteristics of each characteristic point in the calibration plate and the different requirements of the first type camera and the second type camera on the characteristic points in the calibration process are combined, and in the process of calibrating the first type camera and the second type camera through the characteristic points in the calibration plate, the first characteristic points and the second characteristic points in the first characteristic point set can be particularly used for calculating the initial characteristic matrix of the first type camera such as the RGB camera; the third feature point and the fourth feature point in the second feature point set may be specifically used to calculate an initial feature matrix of a second type camera, such as a TOF camera; the feature points in the second feature point set and the third feature point set can be specifically used for fitting the final calibration result of the first type camera such as an RGB camera; the first feature points in the third feature point set may be used in particular to fit the final calibration result of a second type of camera, such as a TOF camera.

Further, the first shape is a dot shape, and the second shape is a ring shape. In the practical application process, the first shape may be a circular dot, a square dot, a triangular dot, etc., and the second shape may be a circular ring, a square ring, a triangular ring, etc., and those skilled in the art may set the specific shapes of the first shape and the second shape according to the practical situation, which is not limited herein.

In this embodiment, as shown in fig. 2, the calibration plate and the camera calibration method provided in this embodiment are explained by taking the shape of the feature point in the calibration plate as a round dot and a circular ring as examples, that is, taking the first shape as a round dot and the second shape as a circular ring as examples.

Specifically, as shown in fig. 2, in the embodiment of the present application, the following five feature points are arranged in the calibration plate: small circle a, medium circle B, large circle C, small circle D, and large circle E. And numbering the characteristic points in the calibration plate from the upper left corner to the right and downwards by using the coordinates of the small dots A at the upper left corner of the calibration plate as (1, 1). On this basis, the coordinate numbers of the respective medium-sized dots B are, in order from left to right and from above and below, as follows: (4, 3), (4, 7), (4, 11), (4, 15), (4, 19), (4, 23), (4, 27), (8, 3), (8, 7), (8, 11), (8, 15), (8, 23), (8, 27), (11, 16), (12, 3), (12, 7), (12, 19), (12, 23), (12, 27), (13, 16), (16, 3), (16,7), (16, 11), (16, 15), (16, 23), (16, 27), (20, 3), (20, 7), (20, 11), (20, 15), (20, 19), (20, 23) and (20, 27); the coordinate numbers of the large dots C are as follows: (8, 19), (16, 19); the coordinates of the small ring D are numbered: (12, 14); the coordinates of the large ring E are numbered: (12, 11).

As shown in fig. 2, because the positions occupied by the large-size dots C and the large-size ring E in the calibration plate are larger, the small-size dots at four positions adjacent to the positions of the large-size dots C and the large-size ring E in the up-down, left-right directions are missing, and the missing small-size dots are recorded in the set T.

Further, based on the requirements of the first type camera and the second type camera on the feature points in the calibration process, the small-sized ring D in the calibration plate is equivalent to the second feature point, and the middle two middle-sized dots B (i.e., the middle-sized dots B at the positions of (11, 16) and (13, 16)) are equivalent to the first feature point in the first feature point set. That is, the small-sized ring D and the middle two medium-sized dots B in the calibration plate are used as mark points, i.e., marker points, for calculating the initial feature matrix of the first type camera, e.g., the RGB camera. Further, the large circle E corresponds to the fourth feature point, and the two large dots C correspond to the third feature point, i.e. the large circle E and the two large dots C are used as mark points for calculating the initial feature matrix of the second type camera, such as the TOF camera. Further, the medium-size dots B (excluding the medium-size dots B at the two positions (11, 16) and (13, 16)) are the first feature points corresponding to the third feature point set, that is, the medium-size dots B (excluding the medium-size dots B at the two positions (11, 16) and (13, 16)) are used as a set of corner points for iteratively fitting a second type camera such as a TOF image to obtain a final calibration result. Further, the small-size circular dots a are equivalent to the fifth feature points, and feature points except the small-size circular ring D and the middle two middle-size circular dots B in the calibration plate are used as corner sets for iteratively fitting the first type camera, such as the RGB image, to obtain the final calibration result.

Specifically, in the method for calibrating the camera provided in the embodiment of the present application, the size and the characteristics of the feature points in the calibration plate are different, and the acquisition results of the feature points in the calibration plate by cameras of different types, that is, cameras of different resolutions, are different. On the basis, in the camera calibration method provided by the embodiment of the application, in the process of calibrating the camera through the calibration plate, after a calibration image corresponding to the camera is obtained by shooting the calibration plate through the camera, image analysis is carried out on the calibration image shot by the camera, and the size information, the coordinate information and the shape information of each feature point in the calibration image are determined. For example, when the first shape is a dot and the second shape is a ring, the radius, center coordinates, i.e., center coordinates, and shape of each feature point in the calibration image are determined by performing image analysis on the calibration image captured by the camera.

S104: and determining mark points from the feature points according to the size information, the coordinate information and the shape information, and determining a corner set from the feature points according to the mark points.

In the embodiment of the application, based on different resolution cameras, namely, the first type camera and the second type camera, the requirements on the feature points in the calibration process are different, the feature points with different coordinates, sizes and shapes are arranged in the calibration plate, and the feature points with different coordinates, sizes and shapes correspond to different calibration processes of the cameras with different resolutions in the process of calibrating the cameras with different resolutions.

Specifically, in the process of calibrating the first type camera and the second type camera through the feature points in the calibration plate, the first feature points and the second feature points in the first feature point set are used as marking points for calculating an initial feature matrix of the first type camera such as an RGB camera; the third feature point and the fourth feature point in the second feature point set are used as marker points for calculating an initial feature matrix of a second type camera, such as a TOF camera; the feature points in the second feature point set and the third feature point set are used as corner point sets for fitting the final calibration result of the first type camera such as an RGB camera; the first feature points in the third feature point set are used as a set of corner points for fitting the final calibration result of a second type of camera, such as a TOF camera.

On the basis, in the camera calibration method provided by the embodiment of the application, after the size information, the coordinate information and the shape information of each feature point in the calibration image shot by the camera are determined, based on the requirements of different types of cameras on the calibration feature points, the feature points corresponding to the types of the cameras are selected from the feature points of the calibration image to serve as the marking points and the corner sets of the calibration camera respectively according to the size information, the coordinate information and the shape information of each feature point in the calibration image.

S106: and determining a feature matrix according to the world coordinates of the mark points and the image coordinates of the mark points in the calibration image.

Specifically, in the camera calibration method provided in the embodiment of the present application, after determining a mark point corresponding to a camera from feature points of a calibration image, a feature matrix of the camera is determined according to a correspondence between world coordinates of the mark point and image coordinates of the mark point in the calibration image, and specifically, a feature matrix of the camera is determined according to a correspondence between world coordinates and center coordinates of the mark point.

S108: and determining a calibration result of the camera according to the feature matrix, the world coordinates of the corner sets and the image coordinates of the corner sets in the calibration image.

Specifically, in the camera calibration method provided by the embodiment of the application, after the feature matrix of the camera is obtained, iterative fitting operation is performed on the calibration image shot by the camera according to the feature matrix of the camera, the world coordinates of the corner set corresponding to the camera and the corresponding relation of the image coordinates of the corner set in the calibration image, so as to obtain the final calibration result of the camera. Specifically, according to the feature matrix of the camera, the corresponding relationship between the world coordinates and the center coordinates of the corner sets corresponding to the camera, the final calibration result of the camera is calculated.

According to the camera calibration method provided by the embodiment of the application, the size information, the coordinate information and the shape information of the feature points in the calibration image are determined according to the calibration image of the camera shooting calibration plate; determining a mark point from the feature points according to the size information, the coordinate information and the shape information, and determining a corner set from the feature points according to the mark point; determining a feature matrix according to world coordinates of the mark points and image coordinates of the mark points in the calibration image; and determining a calibration result of the camera according to the feature matrix, the world coordinates of the corner sets and the image coordinates of the corner sets in the calibration image. Under the condition that the camera is a first type camera, the marking points and the corner sets comprise feature points with the smallest size in the calibration plate; under the condition that the camera is a second type camera, the marking points and the corner point sets do not comprise the feature points with the smallest size in the calibration plate; the resolution of the first type of camera is greater than the resolution of the second type of camera. According to the camera calibration method, the characteristic points with different coordinates, sizes and shapes are arranged on the calibration plate, in the process of calibrating the camera by using the calibration plate, the mark points and the corner sets corresponding to the type of the camera are selected from the characteristic points of the calibration image according to the size information, the shape information and the coordinate information of the characteristic points in the calibration image shot by the camera, and the camera is calibrated based on the selected mark points and the corner sets. Therefore, in the process of calibrating the cameras, the calibration schemes of the cameras of different types are compatible, the calibration of the cameras of different types can be realized only through one calibration plate, and the calibration cost of the cameras can be reduced while the calibration efficiency of the cameras is improved.

In the embodiment of the present application, the above S102 may specifically include the following S102a to S102d:

s102a: and determining the size information of each feature point according to the image contour of each feature point.

Specifically, in the camera calibration method provided in the embodiment of the present application, after a calibration image corresponding to a camera is obtained by shooting a calibration plate with the camera, an image contour of each feature point in the calibration image is extracted, and size information of each feature point is determined according to the image contour of each feature point, for example, if the first shape is a dot and the second shape is a ring, a radius of each feature point in the calibration image is determined according to the image contour of each feature point.

The different types of cameras acquire different results of characteristic points in the calibration plate. In the process of extracting the image contour of each feature point in the calibration image, when the first shape is a dot and the second shape is a ring, as shown in fig. 2, for the calibration image captured by the first type camera with higher resolution, for example, the RGB camera, the contour of each dot in the calibration image may be extracted, and the inner contour and the outer contour of each ring in the calibration image may be extracted; for the calibration image captured by the second type of camera with lower resolution, such as the TOF camera, the image captured by the TOF camera has low resolution and large noise, and only the image contour of the feature point in the calibration image captured by the TOF camera can be successfully extracted as shown in fig. 6 The contours of the medium-size dots B and the large-size dots C are O, the inner contour and the outer contour of the large-size ring D are E, and the outer contour of the small-size ring E is O, and irregular noise contours are as followsEtc., while the outline of the small circle a and the inner outline of the small circle E cannot be extracted.

S102b: fitting each image contour to obtain a first coordinate corresponding to each image contour.

Specifically, in the camera calibration method provided by the embodiment of the application, after the image contour of each feature point in the calibration image is extracted, fitting is performed on each image contour, so as to obtain a first coordinate corresponding to each image contour. For example, when the first shape is a circular dot and the second shape is a circular ring, the image contour of the feature point is a circular contour, and the circle center coordinates of each circular contour are obtained by fitting each extracted circular contour.

Under the condition that the first shape is a round dot and the second shape is a circular ring, the outline of each circular ring in the calibration image shot by the first type camera such as an RGB camera and the outline of the large circular ring D in the calibration image shot by the second type camera such as a TOF camera are all in the form of excellent, namely the image outlines of the circular rings comprise two circular outlines, and after the image outlines of the circular rings are fitted, two circle center coordinates with the same coincidence or smaller distance difference, namely the first coordinates, can be obtained. For the small-sized ring E in the calibration image shot by the second type camera, such as the TOF camera, the inner contour of the small-sized ring E cannot be detected, but only the outer contour o of the small-sized ring E can be extracted, so that only one first coordinate can be fitted according to the image contour of the small-sized ring E.

S102c: and when the distance between the first coordinate and any other coordinate is greater than or equal to a first threshold value, determining the characteristic point of the first shape corresponding to the first coordinate, and determining the first coordinate as the center coordinate of the corresponding characteristic point.

Specifically, in the camera calibration method provided in the embodiment of the present application, after each extracted image contour is fitted to obtain a first coordinate corresponding to each image contour, when the distance between a certain first coordinate and any one of other coordinates is greater than or equal to a first threshold, a feature point corresponding to a first shape by the first coordinate is determined, that is, a feature point corresponding to a point by the first coordinate is determined, and the first coordinate is determined as a center coordinate of the corresponding feature point.

The first threshold is a smaller value, and a person skilled in the art may set a specific value of the first threshold according to actual situations, which is not limited herein.

S102d: and when the distance between the two first coordinates is smaller than the first threshold value, determining the feature point of which the two first coordinates correspond to one second shape, and determining the average value of the two first coordinates as the center coordinate of the corresponding feature point.

Specifically, in the camera calibration method provided in the embodiment of the present application, after each extracted image contour is fitted to obtain first coordinates corresponding to each image contour, if there are two first coordinates with a distance smaller than a first threshold, it is determined that the two first coordinates correspond to a feature point of a second shape, that is, it is determined that the two first coordinates correspond to a feature point of the same ring. At this time, the average value of the two first coordinates is determined as the center coordinate of its corresponding feature point.

In the above embodiment provided by the present application, in the process of determining the size information, the coordinate information and the shape information of the feature points in the calibration image, the size information of each feature point is determined according to the image contour of each feature point; fitting each image contour to obtain a first coordinate corresponding to each image contour; under the condition that the distance between the first coordinate and any other coordinate is larger than or equal to a first threshold value, determining a characteristic point of the first shape corresponding to the first coordinate, and determining the first coordinate as a center coordinate of the corresponding characteristic point; and when the distance between the two first coordinates is smaller than the first threshold value, determining the feature point of which the two first coordinates correspond to one second shape, and determining the average value of the two first coordinates as the center coordinate of the corresponding feature point. In this way, the accuracy of determining the size information, the coordinate information and the shape information of the feature points is ensured, and the accuracy of selecting the mark points and the corner sets based on the size information, the coordinate information and the shape information of the feature points is further ensured.

In this embodiment of the present application, the camera is a first type camera, and on the basis of this, the step of determining the mark point from the feature point according to the size information, the coordinate information, and the shape information may specifically include the following steps S110 and S112:

s110: and determining the feature point with the first size and the second shape in the calibration image as a first mark point of the first type camera according to the size information and the shape information of the feature point in the calibration image.

The first type of camera can be an RGB camera, the RGB camera needs to calibrate distortion parameters, and the RGB image needs enough feature points to fit accurate distortion parameters, so that when the RGB camera is calibrated through the calibration board, a sufficient number of common corner points must be provided in the calibration board. In addition, the resolution ratio of the RGB camera is larger, and the size of the characteristic point is not required to be too large when the RGB camera is calibrated.

On the basis, in the camera calibration method provided by the embodiment of the application, when the first type of camera such as an RGB camera is calibrated, after the size information, the coordinate information and the shape information of each feature point in the calibration image shot by the RGB camera are determined, the feature points possibly serving as the marking points of the calibration RGB camera in the calibration image are initially screened according to the size information of each feature point in the calibration image. Specifically, the size information of each feature point in the calibration image is compared with a second threshold value, feature points with the size information larger than the second threshold value in the calibration image are counted into a set N, and the feature points in the set N are ordered according to the sequence of the center coordinates of the feature points in the set N from left to right and from top to bottom, so that a first point set is obtained.

The second threshold is greater than the size of the fifth feature point, and the second threshold is smaller than the size of the first feature point, that is, the second threshold has a value between the third size and the first size. On the basis, the first point set only comprises a first feature point, a second feature point, a third feature point and a fourth feature point in the calibration plate.

For example, when the first shape is a circle and the second shape is a circle, the value of the second threshold is between the radius of the small circle a and the radius of the medium circle B in the calibration plate, and then, as shown in fig. 3 (a), the first point set includes only four feature points, i.e., the medium circle B, the large circle C, the small circle D, and the large circle E.

Further, in the calibration method of the camera provided in the embodiment of the present application, after the first point set is obtained, according to the size information and the shape information of each feature point in the first point set, the feature point with the first size and the feature point with the second shape in the first point set is determined as the first mark point of the first type camera.

For example, when the first shape is a circle point and the second shape is a circle, after the first point set is obtained, a coordinate difference between the center coordinates of each adjacent two circular outlines in the first point set is calculated, and when the coordinate difference between the center coordinates of the adjacent two circular outlines is smaller than a third threshold value, it is considered that the two circular outlines correspond to one circle feature point, and at this time, an average value of the center coordinates of the two circular outlines is taken as the center coordinates of the corresponding circle feature point. Based on this, as shown in fig. 3 (a), it is possible to detect that two characteristic points in the shape of a circle, i.e., a small-size circle D and a large-size circle E, are concentrated at the first point. At this time, a small ring D whose center coordinates are closer to the center position is selected as the first mark point mark1.

The value of the third threshold is related to the accuracy capability of extracting the image contour and fitting the center coordinates, and a person skilled in the art can set a specific value of the third threshold according to the actual situation, which is not limited herein.

S112: and determining the feature point with the first size in the calibration image as a first size and a first distance from the first mark point as a second mark point of the first type camera according to the center coordinates of the first mark point, the coordinate information and the size information of the feature point in the calibration image.

Wherein the first dimension is smaller than the second dimension.

Specifically, in the calibration method of the camera provided in the embodiment of the present application, after determining the first mark point of the first type camera, the size in the calibration image is determined to be the first size according to the center coordinate of the first mark point, the coordinate information and the size information of the feature point in the calibration image, and the feature point spaced from the first mark point by the first distance is determined to be the second mark point of the first type camera.

For example, in the case where the first shape is a circular dot and the second shape is a circular ring, as shown in fig. 3 (b), after the first mark point mark1 is determined, the center coordinates of the first mark point mark1 are moved upward by 1 unit value d1 and then moved rightward by 2 unit values, that is, (2×d1) according to the positional relationship between the feature points in the first point set and the first mark point mark1, and then the feature point having the smallest distance from the first mark point mark1 is determined as the first and second mark points mark2. On this basis, the center coordinates of the first mark point mark1 are further moved downward by two unit values (2×d1), and the feature point at this time, which is the smallest distance from the first mark point mark1, is determined as the second mark point mark3. Thus, three marker points mark1, mark2 and mark3 are selected from the first set for calibrating a first type of camera, such as an RGB camera.

The unit value d1 is determined according to an ideal calibration plate corresponding to the first type of camera, namely the RGB camera. As shown in fig. 4, in the case that the first shape is a dot and the second shape is a ring, the ideal calibration plate corresponding to the RGB camera is a calibration plate with full-image full-size dots, the space between the small-size dots is the same as the space between the small-size dots in the current actual calibration plate, and the ideal calibration plate corresponding to the RGB camera corresponds to the dot set M0. On the basis, the unit value d1 is the distance between two adjacent small-size dots in the ideal calibration plate.

In the above embodiment provided by the present application, the camera is a first type camera, and in the process of determining the mark point from the feature point according to the size information, the coordinate information and the shape information, the feature point with the size of the first size and the shape of the second shape in the calibration image is determined as the first mark point of the first type camera according to the size information and the shape information of the feature point in the calibration image; according to the center coordinates of the first mark points, the coordinate information and the size information of the feature points in the calibration image, determining the feature points which are in the first size in the calibration image and are spaced from the first mark points by a first distance as second mark points of the first type camera; wherein the first dimension is smaller than the second dimension. Therefore, the accuracy of determining the corresponding mark points of the first type camera is ensured, and the accuracy of calibrating the first type camera is ensured.

In this embodiment of the present application, the step of determining the corner set from the feature points according to the mark points may specifically include the following step S114:

s114: and determining characteristic points except the first mark point and the second mark point in the calibration image as a corner set corresponding to the first type camera.

Specifically, in the camera calibration method provided in the embodiment of the present application, after determining the first mark point and the second mark point corresponding to the first type camera, for example, the RGB camera, feature points in the calibration image except for the first mark point and the second mark point are determined as the corner set corresponding to the first type camera.

In the practical application process, as shown in fig. 5, the feature points in the set T may be further removed from the point set M0, and the feature points in the processed point set M0 may be used as a set of corner points for calibrating the first type camera, such as an RGB camera, which is not limited herein.

In the above embodiment provided by the present application, in a process of determining a set of corner points from feature points according to the mark points, feature points in the calibration image except for the first mark point and the second mark point are determined as the set of corner points corresponding to the first type camera. Therefore, the accuracy of determining the angular point set corresponding to the first type camera is ensured, and the accuracy of calibrating the first type camera is ensured.

In this embodiment of the present application, the camera is a second type camera, and on the basis of this, the step of determining the marker point from the feature point according to the size information, the coordinate information, and the shape information may specifically include the following steps S116 to S120:

s116: and carrying out noise reduction treatment on the calibration image according to the shape information of each characteristic point in the calibration image.

The second type of camera may specifically be a TOF camera.

It can be appreciated that the second type of camera, such as a TOF camera, is low in resolution and high in noise, and when extracting the image profile of the feature points in the calibration image, such as an irregular noise profileTherefore, in the camera calibration method provided in the embodiment of the present application, when a second type of camera such as a TOF camera is calibrated, after the image contour of the feature points in the calibration image captured by the TOF camera is extracted, noise reduction processing is performed on the calibration image according to the shape of each feature point in the calibration image, so as to filter noise information in the calibration image.

S118: and determining the feature point with the second size and the second shape in the calibration image as a third mark point of the second type camera according to the size information and the shape information of the feature point in the calibration image after the noise reduction treatment.

The second type of camera may specifically be a TOF camera. The resolution of the TOF camera is low, mark points used for calculating a feature matrix of the TOF camera are marking points, and common angular points used for fitting a final calibration result of the TOF camera need to be large enough, and the TOF camera does not need to calibrate distortion parameters, so that the number of common angular points used for fitting the final calibration result of the TOF camera can be small.

On the basis, in the camera calibration method provided by the embodiment of the application, when a second type of camera such as a TOF camera is calibrated, noise reduction processing is performed on a calibration image shot by the TOF camera, size information, coordinate information and shape information of each feature point in the calibration image are determined, and then the feature point with the second size and the second shape in the calibration image is determined as a third mark point of the second type of camera according to the size information and the shape information of the feature point in the calibration image after the noise reduction processing. Wherein the second dimension is greater than the first dimension.

In an exemplary embodiment, when the first shape is a dot and the second shape is a circle, after the noise reduction processing is performed on the calibration image, a coordinate difference of center coordinates of every two adjacent circular outlines in the noise reduction processed calibration image is calculated, and when the coordinate difference of center coordinates of the two adjacent circular outlines is smaller than a third threshold, the two circular outlines are considered to correspond to one circle feature point, and at this time, an average value of the center coordinates of the two circular outlines is taken as the center coordinates of the corresponding circle feature point. Based on this, as shown in fig. 7 (a), it is possible to detect that only one large-sized ring E, which is a ring-shaped feature point, exists in the calibration image after the noise reduction processing, and determine the large-sized ring E as the third mark point 4.

S120: and determining the feature point with the second distance from the third mark point as a fourth mark point of the second type camera, wherein the size in the calibration image is the second size according to the center coordinates of the third mark point, the coordinate information and the size information of the feature point in the calibration image.

Specifically, in the calibration method of the camera provided in the embodiment of the present application, after the third mark point of the second type camera is determined, the size in the calibration image is the second size according to the center coordinate of the third mark point, the coordinate information and the size information of the feature point in the calibration image, and the feature point spaced from the third mark point by the second distance is determined as the fourth mark point of the second type camera.

For example, in the case where the first shape is a circular dot and the second shape is a circular ring, as shown in fig. 7 (b), after the third mark point mark4 is determined, the center coordinate of the third mark point mark4 is moved upward by 1 unit value d2 and then moved rightward by 2 unit values, that is, (2×d2), and then the feature point having the smallest distance from the third mark point mark4 is determined as the first fourth mark point mark5. On this basis, the center coordinates of the third mark point mark4 are further moved downward by two unit values (2×d2), and the feature point at this time, which is the smallest distance from the third mark point mark4, is determined as the second fourth mark point mark6. In this way, three mark points mark4, mark5 and mark6 are selected from the calibration image for calibrating a second type camera such as a TOF camera.

The unit value d2 is determined according to an ideal calibration plate corresponding to the second type of camera, namely the TOF camera. As shown in fig. 9 (a), when the first shape is a dot and the second shape is a ring, the ideal calibration plate corresponding to the TOF camera is a calibration plate with full-scale medium dots, the pitch of the medium dots is the same as the pitch of the medium dots in the current actual calibration plate, and the ideal calibration plate corresponding to the TOF camera corresponds to the dot set M1. On the basis, the unit value d2 is the distance between two adjacent medium-sized dots in the ideal calibration plate.

According to the embodiment, the camera is a second type camera, and noise reduction processing is performed on the calibration image according to the shape information of each feature point in the calibration image in the process of determining the mark point from the feature points according to the size information, the coordinate information and the shape information; according to the size information and the shape information of the feature points in the calibrated image after the noise reduction treatment, determining the feature points with the second size and the second shape in the calibrated image as third mark points of the second type camera; according to the center coordinates of the third mark points, the coordinate information and the size information of the feature points in the calibration image, determining the feature points which are in the second size in the calibration image and are spaced from the third mark points by a second distance as fourth mark points of the second type camera; wherein the second dimension is greater than the first dimension. Therefore, the accuracy of determining the corresponding mark points of the second type camera is ensured, and the accuracy of calibrating the second type camera is ensured.

In this embodiment of the present application, the calibration image does not include feature points of a third size, where the third size is smaller than the first size, and on the basis of this, the step of determining the corner set from the feature points according to the mark points may specifically include the following step S122:

s122: and determining characteristic points except the first mark point, the second mark point, the third mark point and the fourth mark point in the calibration image as a corner set corresponding to the second type camera.

Specifically, in the method for calibrating a camera provided in the embodiments of the present application, in the process of calibrating a second type of camera, such as a TOF camera, by using the calibration board, a calibration image captured by the second type of camera does not include a feature point of a third size, where the third size is smaller than the first size. For example, in the case where the first shape is a dot and the second shape is a circle, the calibration image captured by the second type camera does not include a small dot a. On the basis, in the process of determining the corner set corresponding to the second type camera, after determining a third mark point and a fourth mark point for calibrating the second type camera such as the TOF camera, determining characteristic points except the third mark point, the fourth mark point and the first mark point and the second mark point in the calibration image as the corner set corresponding to the second type camera.

In the process of eliminating the first mark point and the second mark point in the calibration image, as shown in fig. 8, when the first shape is a dot and the second shape is a ring, the center coordinate of the third mark point mark4 is moved rightward by 1 unit value d2, then three feature points closest to the third mark point mark4 are the first feature point and the second feature point, and at this time, three feature points closest to the third mark point mark4 are eliminated from the calibration image.

In addition, in the practical application process, as shown in fig. 9 (b), the center medium dots may be removed from the point set M1, and the feature points in the processed point set M1 may be used as the corner set for calibrating the second type camera, such as the TOF camera, which is not limited herein.

According to the embodiment, the calibration image does not include the feature points with the third size, the third size is smaller than the first size, and in the process of determining the corner sets from the feature points according to the mark points, the feature points except the first mark point, the second mark point, the third mark point and the fourth mark point in the calibration image are determined to be the corner sets corresponding to the second type cameras. Therefore, the accuracy of determining the angular point set corresponding to the second type of camera is ensured, and the accuracy of calibrating the second type of camera is ensured.

According to the camera calibration method provided by the embodiment of the application, the execution main body can be a camera calibration device. In this embodiment of the present application, a camera calibration device provided in this embodiment of the present application is described by taking an example in which the camera calibration device executes the above-mentioned camera calibration method.

As shown in fig. 10, an embodiment of the present application provides a camera calibration apparatus 900, which may include a processing unit 902 described below.

The processing unit 902 is configured to determine size information, coordinate information and shape information of feature points in the calibration image according to the calibration image of the calibration plate captured by the camera;

the processing unit 902 is further configured to determine a marker point from the feature points according to the size information, the coordinate information, and the shape information, and determine a corner set from the feature points according to the marker point;

the processing unit 902 is further configured to determine a feature matrix according to world coordinates of the marker points and image coordinates of the marker points in the calibration image;

the processing unit 902 is further configured to determine a calibration result of the camera according to the feature matrix, world coordinates of the corner set, and image coordinates of the corner set in the calibration image;

under the condition that the camera is a first type camera, the marking points and the corner sets comprise feature points with the smallest size in the calibration plate;

Under the condition that the camera is a second type camera, the marking points and the corner point sets do not comprise the feature points with the smallest size in the calibration plate;

the resolution of the first type of camera is greater than the resolution of the second type of camera.

According to the camera calibration device 900 provided by the embodiment of the application, the size information, the coordinate information and the shape information of the feature points in the calibration image are determined according to the calibration image of the calibration plate shot by the camera; determining a mark point from the feature points according to the size information, the coordinate information and the shape information, and determining a corner set from the feature points according to the mark point; determining a feature matrix according to world coordinates of the mark points and image coordinates of the mark points in the calibration image; and determining a calibration result of the camera according to the feature matrix, the world coordinates of the corner sets and the image coordinates of the corner sets in the calibration image. Under the condition that the camera is a first type camera, the marking points and the corner sets comprise feature points with the smallest size in the calibration plate; under the condition that the camera is a second type camera, the marking points and the corner point sets do not comprise the feature points with the smallest size in the calibration plate; the resolution of the first type of camera is greater than the resolution of the second type of camera. Through the camera calibration device 900, feature points with different coordinates, sizes and shapes are set on the calibration plate, in the process of calibrating the camera by using the calibration plate, according to the size information, the shape information and the coordinate information of the feature points in the calibration image shot by the camera, a mark point and a corner point set corresponding to the type of the camera are selected from the feature points of the calibration image, and the camera is calibrated based on the selected mark point and corner point set. Therefore, in the process of calibrating the cameras, the calibration schemes of the cameras of different types are compatible, the calibration of the cameras of different types can be realized only through one calibration plate, and the calibration cost of the cameras can be reduced while the calibration efficiency of the cameras is improved.

In the embodiment of the present application, the processing unit 902 is specifically configured to: determining the size information of each feature point according to the image contour of each feature point; fitting each image contour to obtain a first coordinate corresponding to each image contour; under the condition that the distance between the first coordinate and any other coordinate is larger than or equal to a first threshold value, determining a characteristic point of the first shape corresponding to the first coordinate, and determining the first coordinate as a center coordinate of the corresponding characteristic point; and when the distance between the two first coordinates is smaller than the first threshold value, determining the feature point of which the two first coordinates correspond to one second shape, and determining the average value of the two first coordinates as the center coordinate of the corresponding feature point.

In this embodiment, the camera is a first type of camera, and the processing unit 902 is specifically configured to: according to the size information and the shape information of the feature points in the calibration image, determining the feature points with the first size and the second shape in the calibration image as first mark points of the first type camera; according to the center coordinates of the first mark points, the coordinate information and the size information of the feature points in the calibration image, determining the feature points which are in the first size in the calibration image and are spaced from the first mark points by a first distance as second mark points of the first type camera; wherein the first dimension is smaller than the second dimension.

In the embodiment of the present application, the processing unit 902 is specifically configured to: and determining characteristic points except the first mark point and the second mark point in the calibration image as a corner set corresponding to the first type camera.

In this embodiment, the camera is a second type camera, and the processing unit 902 is specifically configured to: according to the shape information of each characteristic point in the calibration image, carrying out noise reduction treatment on the calibration image; according to the size information and the shape information of the feature points in the calibrated image after the noise reduction treatment, determining the feature points with the second size and the second shape in the calibrated image as third mark points of the second type camera; according to the center coordinates of the third mark points, the coordinate information and the size information of the feature points in the calibration image, determining the feature points which are in the second size in the calibration image and are spaced from the third mark points by a second distance as fourth mark points of the second type camera; wherein the second dimension is greater than the first dimension.

In this embodiment of the present application, the calibration image does not include feature points of a third size, and the third size is smaller than the first size, and the processing unit 902 is specifically configured to: and determining characteristic points except the first mark point, the second mark point, the third mark point and the fourth mark point in the calibration image as a corner set corresponding to the second type camera.

The camera calibration device 900 in the embodiment of the present application may be an electronic device, or may be a component in an electronic device, for example, an integrated circuit or a chip. The electronic device may be a terminal, or may be other devices than a terminal. By way of example, the electronic device may be a mobile phone, tablet computer, notebook computer, palm computer, vehicle-mounted electronic device, mobile internet appliance (Mobile Internet Device, MID), augmented reality (augmented reality, AR)/Virtual Reality (VR) device, robot, wearable device, ultra-mobile personal computer, UMPC, netbook or personal digital assistant (personal digital assistant, PDA), etc., but may also be a server, network attached storage (Network Attached Storage, NAS), personal computer (personal computer, PC), television (TV), teller machine or self-service machine, etc., and the embodiments of the present application are not limited in particular.

The camera calibration apparatus 900 in the embodiment of the present application may be an apparatus with an operating system. The operating system may be an Android operating system, an iOS operating system, or other possible operating systems, which are not specifically limited in the embodiments of the present application.

The camera calibration apparatus 900 provided in this embodiment of the present application can implement each process implemented by the method embodiment of fig. 1, and in order to avoid repetition, a description is omitted here.

Optionally, as shown in fig. 11, the embodiment of the present application further provides an electronic device 1000, including a processor 1002 and a memory 1004, where a program or an instruction that can be executed on the processor 1002 is stored in the memory 1004, and the program or the instruction when executed by the processor 1002 implements each step of the embodiment of the camera calibration method, and can achieve the same technical effect, so that repetition is avoided, and no further description is given here.

The electronic device in the embodiment of the application includes the mobile electronic device and the non-mobile electronic device.

Fig. 12 is a schematic hardware structure of an electronic device implementing an embodiment of the present application.

The electronic device 1100 includes, but is not limited to: radio frequency unit 1101, network module 1102, audio output unit 1103, input unit 1104, sensor 1105, display unit 1106, user input unit 1107, interface unit 1108, memory 1109, and processor 1110.

Those skilled in the art will appreciate that the electronic device 1100 may further include a power source (e.g., a battery) for powering the various components, which may be logically connected to the processor 1110 by a power management system, such as to perform functions such as managing charging, discharging, and power consumption by the power management system. The electronic device structure shown in fig. 12 does not constitute a limitation of the electronic device, and the electronic device may include more or less components than illustrated, or may combine certain components, or may be arranged in different components, which are not described in detail herein.

The processor 1110 is configured to determine size information, coordinate information, and shape information of feature points in the calibration image according to the calibration image of the calibration plate captured by the camera.

The processor 1110 is further configured to determine a marker point from the feature points according to the size information, the coordinate information, and the shape information, and determine a set of corner points from the feature points according to the marker point.

The processor 1110 is further configured to determine a feature matrix according to world coordinates of the marker points and image coordinates of the marker points in the calibration image.

The processor 1110 is further configured to determine a calibration result of the camera according to the feature matrix, the world coordinates of the corner set, and the image coordinates of the corner set in the calibration image.

Under the condition that the camera is a first type camera, the marking points and the corner sets comprise feature points with the smallest size in the calibration plate; under the condition that the camera is a second type camera, the marking points and the corner point sets do not comprise the feature points with the smallest size in the calibration plate; the resolution of the first type of camera is greater than the resolution of the second type of camera.

In the embodiment of the application, according to a calibration image of a calibration plate shot by a camera, size information, coordinate information and shape information of feature points in the calibration image are determined; determining a mark point from the feature points according to the size information, the coordinate information and the shape information, and determining a corner set from the feature points according to the mark point; determining a feature matrix according to world coordinates of the mark points and image coordinates of the mark points in the calibration image; and determining a calibration result of the camera according to the feature matrix, the world coordinates of the corner sets and the image coordinates of the corner sets in the calibration image. Under the condition that the camera is a first type camera, the marking points and the corner sets comprise feature points with the smallest size in the calibration plate; under the condition that the camera is a second type camera, the marking points and the corner point sets do not comprise the feature points with the smallest size in the calibration plate; the resolution of the first type of camera is greater than the resolution of the second type of camera. In the embodiment of the application, feature points with different coordinates, sizes and shapes are arranged on the calibration plate, in the process of calibrating the camera by using the calibration plate, according to the size information, the shape information and the coordinate information of the feature points in the calibration image shot by the camera, a mark point and a corner point set corresponding to the type of the camera are selected from the feature points of the calibration image, and the camera is calibrated based on the selected mark point and corner point set. Therefore, in the process of calibrating the cameras, the calibration schemes of the cameras of different types are compatible, the calibration of the cameras of different types can be realized only through one calibration plate, and the calibration cost of the cameras can be reduced while the calibration efficiency of the cameras is improved.

Optionally, the processor 1110 is specifically configured to: determining the size information of each feature point according to the image contour of each feature point; fitting each image contour to obtain a first coordinate corresponding to each image contour; under the condition that the distance between the first coordinate and any other coordinate is larger than or equal to a first threshold value, determining a characteristic point of the first shape corresponding to the first coordinate, and determining the first coordinate as a center coordinate of the corresponding characteristic point; and when the distance between the two first coordinates is smaller than the first threshold value, determining the feature point of which the two first coordinates correspond to one second shape, and determining the average value of the two first coordinates as the center coordinate of the corresponding feature point.

Optionally, the camera is a first type of camera, and the processor 1110 is specifically configured to: according to the size information and the shape information of the feature points in the calibration image, determining the feature points with the first size and the second shape in the calibration image as first mark points of the first type camera; according to the center coordinates of the first mark points, the coordinate information and the size information of the feature points in the calibration image, determining the feature points which are in the first size in the calibration image and are spaced from the first mark points by a first distance as second mark points of the first type camera; wherein the first dimension is smaller than the second dimension.

Optionally, the processor 1110 is specifically configured to: and determining characteristic points except the first mark point and the second mark point in the calibration image as a corner set corresponding to the first type camera.

Optionally, the camera is a second type of camera, and the processor 1110 is specifically configured to: according to the shape information of each characteristic point in the calibration image, carrying out noise reduction treatment on the calibration image; according to the size information and the shape information of the feature points in the calibrated image after the noise reduction treatment, determining the feature points with the second size and the second shape in the calibrated image as third mark points of the second type camera; and determining characteristic points except the first mark point, the second mark point, the third mark point and the fourth mark point in the calibration image as a corner set corresponding to the second type camera.

According to the embodiment, the camera is a second type camera, and noise reduction processing is performed on the calibration image according to the shape information of each feature point in the calibration image in the process of determining the mark point from the feature points according to the size information, the coordinate information and the shape information; according to the size information and the shape information of the feature points in the calibrated image after the noise reduction treatment, determining the feature points with the second size and the second shape in the calibrated image as third mark points of the second type camera; and determining characteristic points except the first mark point, the second mark point, the third mark point and the fourth mark point in the calibration image as a corner set corresponding to the second type camera. Therefore, the accuracy of determining the angular point set corresponding to the second type of camera is ensured, and the accuracy of calibrating the second type of camera is ensured.

Optionally, the calibration image does not include feature points of a third size, the third size being smaller than the first size, and the processor 1110 is specifically configured to: and determining characteristic points except the first mark point, the second mark point, the third mark point and the fourth mark point in the calibration image as a corner set corresponding to the second type camera.

It should be appreciated that in embodiments of the present application, the input unit 1104 may include a graphics processor (Graphics Processing Unit, GPU) 11041 and a microphone 11042, the graphics processor 11041 processing image data of still pictures or video obtained by an image capturing device (e.g., a camera) in a video capturing mode or an image capturing mode. The display unit 1106 may include a display panel 11061, and the display panel 11061 may be configured in the form of a liquid crystal display, an organic light emitting diode, or the like. The user input unit 1107 includes at least one of a touch panel 11071 and other input devices 11072. The touch panel 11071 is also referred to as a touch screen. The touch panel 11071 may include two parts, a touch detection device and a touch controller. Other input devices 11072 may include, but are not limited to, a physical keyboard, function keys (e.g., volume control keys, switch keys, etc.), a trackball, a mouse, a joystick, and so forth, which are not described in detail herein.

The memory 1109 may be used to store software programs as well as various data. The memory 1109 may mainly include a first memory area storing programs or instructions and a second memory area storing data, wherein the first memory area may store an operating system, application programs or instructions (such as a sound playing function, an image playing function, etc.) required for at least one function, and the like. Further, the memory 1109 may include volatile memory or nonvolatile memory, or the memory 1109 may include both volatile and nonvolatile memory. The nonvolatile Memory may be a Read-Only Memory (ROM), a Programmable ROM (PROM), an Erasable PROM (EPROM), an Electrically Erasable EPROM (EEPROM), or a flash Memory. The volatile memory may be random access memory (Random Access Memory, RAM), static RAM (SRAM), dynamic RAM (DRAM), synchronous DRAM (SDRAM), double Data Rate SDRAM (ddr SDRAM), enhanced SDRAM (Enhanced SDRAM), synchronous DRAM (SLDRAM), and Direct RAM (DRRAM). Memory 1109 in embodiments of the present application includes, but is not limited to, these and any other suitable types of memory.

Processor 1110 may include one or more processing units; optionally, the processor 1110 integrates an application processor that primarily processes operations involving an operating system, user interface, application programs, and the like, and a modem processor that primarily processes wireless communication signals, such as a baseband processor. It will be appreciated that the modem processor described above may not be integrated into the processor 1110.

The embodiment of the application further provides a readable storage medium, and the readable storage medium stores a program or an instruction, which when executed by a processor, implements each process of the embodiment of the camera calibration method, and can achieve the same technical effect, so that repetition is avoided, and no further description is provided here.

The processor is a processor in the electronic device in the above embodiment. Readable storage media include computer readable storage media such as computer readable memory ROM, random access memory RAM, magnetic or optical disks, and the like.

The embodiment of the application further provides a chip, the chip includes a processor and a communication interface, the communication interface is coupled with the processor, the processor is used for running a program or instructions, each process of the embodiment of the camera calibration method can be achieved, the same technical effect can be achieved, and in order to avoid repetition, the description is omitted here.

It should be understood that the chips referred to in the embodiments of the present application may also be referred to as system-on-chip chips, chip systems, or system-on-chip chips, etc.

The embodiments of the present application provide a computer program product, which is stored in a storage medium, and executed by at least one processor to implement the respective processes of the embodiments of the camera calibration method described above, and achieve the same technical effects, and are not described herein in detail for avoiding repetition.

It should be noted that, in this document, the terms "comprises," "comprising," or any other variation thereof, are intended to cover a non-exclusive inclusion, such that a process, method, article, or apparatus that comprises a list of elements does not include only those elements but may include other elements not expressly listed or inherent to such process, method, article, or apparatus. Without further limitation, an element defined by the phrase "comprising one … …" does not exclude the presence of other like elements in a process, method, article, or apparatus that comprises the element. Furthermore, it should be noted that the scope of the methods and apparatus in the embodiments of the present application is not limited to performing the functions in the order shown or discussed, but may also include performing the functions in a substantially simultaneous manner or in an opposite order depending on the functions involved, e.g., the described methods may be performed in an order different from that described, and various steps may also be added, omitted, or combined. Additionally, features described with reference to certain examples may be combined in other examples.

From the above description of the embodiments, it will be clear to those skilled in the art that the above-described embodiment method may be implemented by means of software plus a necessary general hardware platform, but of course may also be implemented by means of hardware, but in many cases the former is a preferred embodiment. Based on such understanding, the technical solutions of the present application may be embodied essentially or in a part contributing to the prior art in the form of a computer software product stored in a storage medium (such as ROM/RAM, magnetic disk, optical disk), comprising several instructions for causing a terminal (which may be a mobile phone, a computer, a server, or a network device, etc.) to perform the methods of the embodiments of the present application.

The embodiments of the present application have been described above with reference to the accompanying drawings, but the present application is not limited to the above-described embodiments, which are merely illustrative and not restrictive, and many forms may be made by those of ordinary skill in the art without departing from the spirit of the present application and the scope of the claims, which are also within the protection of the present application.

Claims

1. A method of camera calibration, the method comprising:

according to a calibration image of a calibration plate shot by a camera, determining size information, coordinate information and shape information of feature points in the calibration image;

determining a mark point from the feature points according to the size information, the coordinate information and the shape information, and determining a corner set from the feature points according to the mark point;

determining a feature matrix according to the world coordinates of the mark points and the image coordinates of the mark points in the calibration image;

determining a calibration result of the camera according to the feature matrix, the world coordinates of the corner sets and the image coordinates of the corner sets in the calibration image;

wherein, in the case that the camera is a first type camera, the marker points and the corner point set include feature points of a minimum size in the calibration plate;

in the case that the camera is a second type camera, the marker points and the corner point set do not include feature points of the minimum size in the calibration plate;

2. The camera calibration method according to claim 1, wherein the determining size information, coordinate information, and shape information of feature points in the calibration image includes:

determining the size information of each characteristic point according to the image contour of each characteristic point;

fitting each image contour to obtain a first coordinate corresponding to each image contour;

determining a feature point of a first shape corresponding to the first coordinate under the condition that the distance between the first coordinate and any other coordinate is larger than or equal to a first threshold value, and determining the first coordinate as a center coordinate of the corresponding feature point;

and when the distance between the two first coordinates is smaller than the first threshold value, determining that the two first coordinates correspond to one feature point of the second shape, and determining the average value of the two first coordinates as the center coordinate of the corresponding feature point.

3. The camera calibration method according to claim 2, wherein the camera is a first type camera, and the determining a marker point from the feature points according to the size information, the coordinate information, and the shape information includes:

According to the size information and the shape information of the feature points in the calibration image, determining the feature points with the first size and the second shape in the calibration image as first mark points of the first type camera;

according to the center coordinates of the first mark points, the coordinate information and the size information of the feature points in the calibration image, determining the feature points which are in the first size in the calibration image and are spaced from the first mark points by a first distance as second mark points of the first type camera;

wherein the first dimension is smaller than the second dimension.

4. A camera calibration method according to claim 3, wherein the determining a set of corner points from the feature points from the marker points comprises:

and determining characteristic points except the first mark points and the second mark points in the calibration image as a corner set corresponding to the first type camera.

5. The camera calibration method according to claim 2, wherein the camera is a second type camera, and the determining a marker point from the feature points according to the size information, the coordinate information, and the shape information includes:

According to the shape information of each characteristic point in the calibration image, carrying out noise reduction treatment on the calibration image;

according to the size information and the shape information of the feature points in the calibration image after the noise reduction treatment, the size in the calibration image is a second size, and the feature points with the shape being the second shape are determined to be third mark points of the second type camera;

according to the center coordinates of the third mark points, the coordinate information and the size information of the feature points in the calibration image, determining the feature points which are in the second size in the calibration image and are spaced from the third mark points by a second distance as fourth mark points of the second type camera;

wherein the second dimension is greater than the first dimension.

6. The camera calibration method according to claim 5, wherein the calibration image does not include feature points of a third size, the third size being smaller than the first size, the determining a set of corner points from the feature points according to the marker points comprising:

and determining characteristic points except the first mark point, the second mark point, the third mark point and the fourth mark point in the calibration image as a corner set corresponding to the second type camera.

7. A camera calibration apparatus, the apparatus comprising:

the processing unit is used for determining the size information, the coordinate information and the shape information of the characteristic points in the calibration image according to the calibration image of the calibration plate shot by the camera;

the processing unit is further used for determining a mark point from the feature points according to the size information, the coordinate information and the shape information, and determining a corner point set from the feature points according to the mark point;

the processing unit is further used for determining a feature matrix according to world coordinates of the marking points and image coordinates of the marking points in the calibration image;

the processing unit is further used for determining a calibration result of the camera according to the feature matrix, the world coordinates of the corner sets and the image coordinates of the corner sets in the calibration image;

8. A calibration plate, the calibration plate comprising:

a first set of feature points, the feature points in the first set of feature points being used as marker points for a first type of camera;

a second set of feature points, the feature points in the second set of feature points being used as marker points for a second type of camera;

a third set of feature points, wherein the feature points in the third set of feature points are used for determining a set of corner points of the first type or the second type of cameras;

the size of the feature points in the first feature point set is a first size, the size of the feature points in the second feature point set is a second size, and the first size is smaller than the second size;

the size of the feature points in the third feature point set is less than or equal to the first size.

9. The calibration plate of claim 8, wherein the first set of feature points comprises:

the first characteristic point is in a first shape;

the second characteristic points are in a second shape;

wherein the number of the second feature points is smaller than the number of the first feature points; or alternatively, the first and second heat exchangers may be,

The second set of feature points includes:

a third feature point, the shape of which is the first shape;

a fourth feature point, the shape of the fourth feature point being the second shape;

wherein the number of fourth feature points is smaller than the number of third feature points.

10. The calibration plate of claim 9, wherein the third set of feature points comprises:

the first feature point;

and a fifth feature point, wherein the shape of the fifth feature point is the first shape, the size of the fifth feature point is a third size, and the third size is smaller than the first size.

11. An electronic device comprising a processor and a memory storing a program or instructions executable on the processor, which when executed by the processor, implement the steps of the camera calibration method of any one of claims 1 to 6.