CN114897996A - Vehicle-mounted camera calibration method and device, computer equipment and storage medium - Google Patents
Vehicle-mounted camera calibration method and device, computer equipment and storage medium Download PDFInfo
- Publication number
- CN114897996A CN114897996A CN202210616046.0A CN202210616046A CN114897996A CN 114897996 A CN114897996 A CN 114897996A CN 202210616046 A CN202210616046 A CN 202210616046A CN 114897996 A CN114897996 A CN 114897996A
- Authority
- CN
- China
- Prior art keywords
- vehicle
- position information
- transformation matrix
- reference position
- mounted camera
- Prior art date
- Legal status (The legal status is an assumption and is not a legal conclusion. Google has not performed a legal analysis and makes no representation as to the accuracy of the status listed.)
- Pending
Links
Images
Classifications
-
- G—PHYSICS
- G06—COMPUTING; CALCULATING OR COUNTING
- G06T—IMAGE DATA PROCESSING OR GENERATION, IN GENERAL
- G06T7/00—Image analysis
- G06T7/80—Analysis of captured images to determine intrinsic or extrinsic camera parameters, i.e. camera calibration
-
- G—PHYSICS
- G06—COMPUTING; CALCULATING OR COUNTING
- G06F—ELECTRIC DIGITAL DATA PROCESSING
- G06F17/00—Digital computing or data processing equipment or methods, specially adapted for specific functions
- G06F17/10—Complex mathematical operations
- G06F17/16—Matrix or vector computation, e.g. matrix-matrix or matrix-vector multiplication, matrix factorization
-
- G—PHYSICS
- G06—COMPUTING; CALCULATING OR COUNTING
- G06T—IMAGE DATA PROCESSING OR GENERATION, IN GENERAL
- G06T7/00—Image analysis
- G06T7/70—Determining position or orientation of objects or cameras
Abstract
The embodiment of the disclosure discloses a calibration method and a calibration device for a vehicle-mounted camera, computer equipment and a storage medium, wherein the method comprises the following steps: acquiring first position information of a current position of a vehicle-mounted camera relative to a reference position; the vehicle-mounted camera is arranged on the steering wheel column; determining a first transformation matrix of the current position relative to the reference position based on first position information of the onboard camera; acquiring third position information of the reference position in a cabin coordinate system and rotation angle information of the steering wheel column at the reference position; determining a second transformation matrix of the current position of the onboard camera with respect to the cabin coordinate system based on the first transformation matrix, the third position information, and the rotation angle information. Through the scheme, the calibration of the vehicle-mounted camera with the variable position based on the method and the system can improve the accuracy of state evaluation of the driver in a DMS system.
Description
Technical Field
The present disclosure relates to the field of camera technologies, and in particular, to a calibration method and apparatus for a vehicle-mounted camera, a computer device, and a storage medium.
Background
With the pursuit of beautification and individualization of the interior of the vehicle cabin by different brands and different vehicle types, the installation position of the vehicle-mounted camera is more flexible, for example, the position is changed from a fixed position to a variable position in the vehicle cabin. How to apply the position-variable vehicle-mounted camera to a Driver Monitoring System (DMS) System to determine the state of the Driver is still to be solved.
Disclosure of Invention
The embodiment of the disclosure is expected to provide a calibration method and device for a vehicle-mounted camera, computer equipment and a storage medium.
In a first aspect, an embodiment of the present disclosure provides a calibration method for a vehicle-mounted camera, where the method includes:
acquiring first position information of a current position of a vehicle-mounted camera relative to a reference position; the vehicle-mounted camera is arranged on the steering wheel column;
determining a first transformation matrix of the current position relative to the reference position based on first position information of the onboard camera;
acquiring third position information of the reference position in a cabin coordinate system and rotation angle information of the steering wheel column at the reference position;
determining a second transformation matrix of the current position of the onboard camera with respect to the cabin coordinate system based on the first transformation matrix, the third position information, and the rotation angle information.
In a second aspect, an embodiment of the present disclosure provides a calibration apparatus for an on-vehicle camera, where the apparatus includes:
the first acquisition module is used for acquiring first position information of the current position of the vehicle-mounted camera relative to a reference position; the vehicle-mounted camera is arranged on the steering wheel column;
a first determination module, configured to determine a first transformation matrix of the current position relative to the reference position based on first position information of the vehicle-mounted camera;
a second obtaining module, configured to obtain third position information of the reference position in a cabin coordinate system and rotation angle information of the steering wheel column at the reference position;
a second determination module for determining a second transformation matrix of the current position of the onboard camera with respect to the cabin coordinate system based on the first transformation matrix, the third position information, and the rotation angle information.
In a third aspect, an embodiment of the present disclosure provides a computer device, including: a processor; a memory for storing processor-executable instructions;
wherein the processor is configured to perform the method of the first aspect.
In a fourth aspect, the embodiments of the present disclosure provide a storage medium on which a computer program is stored, which when executed by a processor implements the method described in the first aspect.
The technical scheme provided by the embodiment of the disclosure can have the following beneficial effects:
according to the method, the vehicle-mounted camera is mounted on the steering wheel column, the reference position is preset as the middle item, the current position of the vehicle-mounted camera is mapped to the reference position in a two-step mode, and then the conversion of the current position of the vehicle-mounted camera relative to the vehicle cabin coordinate system is completed based on the third position information from the reference position to the vehicle cabin coordinate system and the rotation angle information of the steering wheel column under the reference position, so that the calibration of the vehicle-mounted camera with the variable position based on the method can be realized, and the accuracy of state evaluation of a driver in a DMS system is improved.
It is to be understood that both the foregoing general description and the following detailed description are exemplary and explanatory only and are not restrictive of the disclosure.
Drawings
Fig. 1 is a flowchart illustrating a calibration method for a vehicle-mounted camera according to an embodiment of the present disclosure;
FIG. 2 is an exemplary illustration of mechanical information for a steering wheel column in an embodiment of the present disclosure;
FIGS. 3 and 4 are exemplary diagrams of three-dimensional data of the center of rotation of a steering wheel column in a cabin coordinate system in an embodiment of the present disclosure;
FIGS. 5 and 6 are exemplary diagrams of three-dimensional data of a reference position in a cabin coordinate system in an embodiment of the disclosure;
FIG. 7 is a schematic diagram of a calibration method for a vehicle-mounted camera according to an embodiment of the present disclosure;
fig. 8 is a diagram illustrating an example of a camera calibration apparatus provided in an embodiment of the present disclosure;
fig. 9 is a hardware entity diagram of a computer device according to an embodiment of the present disclosure.
Detailed Description
The technical solution of the present disclosure is further described in detail below with reference to the drawings and specific embodiments of the specification.
The embodiment of the disclosure provides a camera calibration method, an execution main body of which may be a camera calibration device, and the camera calibration device may be a computer device such as a server, a notebook computer, or the like, or may be a vehicle-mounted device. The vehicle-mounted device may be a vehicle machine in a vehicle cabin, or a device host machine in the vehicle that can be used to perform data processing operations such as images, and the like.
Fig. 1 is a flowchart of a calibration method for a vehicle-mounted camera according to an embodiment of the present disclosure, and as shown in fig. 1, the calibration method for the vehicle-mounted camera includes the following steps:
s11, acquiring first position information of the current position of the vehicle-mounted camera relative to a reference position; the vehicle-mounted camera is arranged on the steering wheel column;
s12, determining a first transformation matrix of the current position relative to the reference position based on first position information of the vehicle-mounted camera;
s13, acquiring third position information of the reference position in a cabin coordinate system and rotation angle information of the steering wheel column at the reference position;
s14, determining a second transformation matrix of the current position of the vehicle-mounted camera relative to the cabin coordinate system according to the first transformation matrix, the third position information and the rotation angle information.
In the disclosed embodiment, the height and the rotation angle of the steering wheel can be changed through manual operation of the steering wheel on the vehicle, and the change of the height and the rotation angle of the steering wheel is also variable based on the length and the rotation angle of the steering wheel column. For example, the position of the steering wheel in the longitudinal direction of the column is changed based on the change in the length of the steering wheel column, and the steering wheel is rotatable with respect to the rotation center point of the column based on the change in the rotation angle of the steering wheel column. When the onboard camera is positioned on the steering wheel column, the position of the onboard camera may change based on the change in position of the steering wheel column.
In the embodiment of the present disclosure, a reference position of the vehicle-mounted camera may be preset, where the reference position may refer to a position of the steering wheel column at a preset length and a preset rotation angle, where the preset length may be a length of the steering wheel column when the vehicle leaves a factory, and the preset rotation angle may be a column angle of the steering wheel column at a specified position, for example, an included angle between the column angle and a plane at the bottom of the cabin is 45 degrees.
In step S11, the camera calibration device obtains first position information of the vehicle-mounted camera at a current position relative to a reference position, where the current position may be a position determined based on a position of a steering wheel column during driving of the vehicle.
In step S12, the camera calibration device determines a first transformation matrix of the current position with respect to the reference position based on the first position information of the onboard camera. If the transformation matrix of the current position relative to the reference position is M, the reference position can be obtained by multiplying the current position by M, that is, the first transformation matrix is a mapping matrix from the current position to the reference position.
In the embodiment of the disclosure, because the position of the vehicle-mounted camera is variable, the position of the vehicle-mounted camera under the vehicle cabin coordinate system needs to be calibrated after the position of the vehicle-mounted camera is changed, so that the condition in the vehicle is analyzed based on the image in the vehicle collected by the vehicle-mounted camera. The length of the steering wheel column when the onboard camera is at the current position may be different from the length of the steering wheel column when the onboard camera is at the reference position, and the rotation angle of the steering wheel column when the onboard camera is at the current position may be different from the rotation angle of the steering wheel column when the onboard camera is at the reference position, so it can be understood that the first transformation matrix includes translation and rotation information of the current position with respect to the reference position.
In step S13, the camera calibration device acquires third position information of the reference position in the cabin coordinate system and rotation angle information of the steering wheel column at the reference position. The vehicle cabin coordinate system is a vehicle cabin coordinate system pre-established by taking a certain fixed position in the vehicle cabin as a vehicle cabin original point, where the vehicle cabin original point is, for example, a place where a glasses box is placed in the vehicle cabin, or a central position of a vehicle-mounted central control display screen, and the like.
In the present disclosure, the reference position belongs to the mechanical design information with respect to the third position of the vehicle cabin coordinate system, and the position information (i.e., the third position information) of the reference position in the vehicle cabin coordinate system and the rotation angle information of the reference position with respect to the steering wheel column may be different for different vehicle types. The camera calibration device can directly read the third position information of the corresponding vehicle type and the rotation angle information of the steering wheel column at the reference position.
In step S14, the camera calibration device determines a second transformation matrix of the current position of the onboard camera with respect to the cabin coordinate system based on the first transformation matrix, the third position information, and the rotation angle information. In the embodiment of the present disclosure, assuming that the second matrix of the current position of the vehicle-mounted camera relative to the vehicle cabin coordinate system is E, the position of the vehicle-mounted camera in the vehicle cabin coordinate system can be obtained by multiplying the current position of the vehicle-mounted camera by E, that is, the second transformation matrix is a mapping matrix of the current position of the vehicle-mounted camera to the vehicle cabin coordinate system.
If the vehicle-mounted camera according to the embodiment of the present disclosure detects the state of the driver, for example, detects the sight area of the driver, the positions of pupils of both eyes in a face image of the driver collected by the vehicle-mounted camera at the current position may be located first, and then, for example, the positions of the centers of the pupils of the left eye and the right eye in the cabin coordinate system are determined according to the internal parameters of the vehicle-mounted camera and the calibration matrix determined by the present disclosure, so as to detect the sight direction of the driver, determine the landing point of the sight direction in the cabin, and obtain the sight area of the driver. For example, for detecting a motion such as a head nod, a gesture, or blinking of eyes of a driver, it is necessary to analyze an acquired video with a fixed coordinate system to detect the corresponding motion. Taking the motion of nodding as an example, after the position of the head of a person in the image frame of the video is detected, multiplying the position of the head of the person under the camera coordinate system of the vehicle-mounted camera at the current position by the internal reference matrix of the camera, then multiplying the position of the head of the person by the second transformation matrix of the present disclosure to obtain the position of the head of the person in the vehicle cabin, and determining whether the driver has the motion of nodding or not based on the position of the head of the person in the vehicle cabin determined by the continuous multi-frame images. If the driver has a nodding action, fatigue driving of the driver can be judged, and prompt information may need to be output so that the driver can drive safely.
In the related art, the position of the onboard camera is fixed, and is mounted on, for example, an "a" pillar, which refers to a connecting pillar of a windshield and left and right front doors. Most of the cameras mounted on the "a" pillar take images of the side of the driver, and thus errors are likely to occur in analyzing the facial expression of the driver or analyzing the behavior of the driver based on the acquired driver side images in the DMS system, for example, resulting in poor accuracy of state evaluation of the driver, and the in-vehicle camera located on the "a" pillar also affects the interior appearance of the vehicle.
In contrast, according to the vehicle-mounted camera calibration method, the vehicle-mounted camera is mounted on the steering wheel column, so that the interference to people in the vehicle is small, the preset reference position serves as an intermediary, and the current position of the vehicle-mounted camera is mapped to the reference position in a two-step mode, and then the conversion of the current position of the vehicle-mounted camera relative to the vehicle cabin coordinate system is completed based on the third position information of the reference position in the vehicle cabin coordinate system and the rotation angle information of the steering wheel column in the reference position, so that the calibration of the vehicle-mounted camera with the variable position based on the vehicle-mounted camera calibration method can be achieved, and the accuracy of state evaluation of a driver in a DMS system is improved.
In some embodiments, the acquiring first position information of the vehicle-mounted camera relative to the reference position at the current position includes:
and reading the mechanical signal of the steering wheel column to acquire the telescopic length data and the deflection angle data of the steering wheel column as the first position information.
In the embodiment of the disclosure, the telescopic length data of the steering wheel column, that is, the length of the steering wheel column corresponding to the current position of the vehicle-mounted camera, is length change data corresponding to the preset length of the steering wheel column corresponding to the reference position of the vehicle-mounted camera; the angle change data of the steering wheel column is angle change data of a preset rotation angle of the steering wheel column corresponding to the reference position of the vehicle-mounted camera relative to the angle of the steering wheel column corresponding to the current position of the vehicle-mounted camera.
When the camera calibration device of the present disclosure acquires the first position information based on the mechanical signal of the steering wheel column, in some embodiments, the mechanical signal may include a length and a rotation angle of the steering wheel column, and the camera calibration device may calculate and obtain the telescopic length data of the steering wheel column based on the length of the steering wheel column corresponding to the current position of the vehicle-mounted camera and the length of the steering wheel column corresponding to the reference position of the vehicle-mounted camera. Similarly, the deflection angle data of the steering wheel column is calculated and obtained based on the rotation angle of the steering wheel column corresponding to the current position of the vehicle-mounted camera and the rotation angle of the steering wheel column corresponding to the reference position of the vehicle-mounted camera.
In the disclosed embodiment, the mechanical signal of the steering wheel column can be obtained based on the positioning device which is used for controlling the driving direction of the vehicle and is arranged in the vehicle. For example, the vehicle driving direction positioning device determines the radius of the vehicle steering based on the length of the steering wheel column, controls the actual steering angle of the vehicle based on the rotation angle of the steering wheel column, and the vehicle control system can jointly determine the driving direction of the vehicle based on the radius of the vehicle steering and the steering angle.
In other embodiments, the mechanical signals may also directly include telescoping length data and yaw angle data of the steering wheel column.
It should be noted that, in the embodiment of the present disclosure, the steering column has a rotation center point, and the telescopic length and the deflection angle of the steering column may be determined relative to the rotation center point.
Fig. 2 is a mechanical information example diagram of a steering wheel column according to an embodiment of the present disclosure, and as shown in fig. 2, 21 denotes the steering wheel column, O is a rotation center point, 22 denotes an on-vehicle camera located at a reference position, and 23 denotes an on-vehicle camera located at a current position. In the present disclosure, the data of the telescopic length of the steering wheel column is Δ L shown in fig. 2, and the data of the yaw angle of the steering wheel column is Δ θ shown in fig. 2.
It will be appreciated that in this embodiment, the calibration efficiency of the onboard camera of the present disclosure is facilitated since the mechanical signal of the steering wheel column can be obtained, for example, based on a positioning device already in the vehicle for controlling the direction of travel of the vehicle, without additional complex calculations.
In some embodiments, said determining a first transformation matrix of said current position relative to said reference position based on said in-vehicle camera first position information comprises:
performing rigid body transformation based on the telescopic length data, the deflection angle data and a preset angle to obtain a first transformation matrix; the preset angle is an angle of a lens surface of the vehicle-mounted camera relative to the axis of the steering wheel pipe column.
In the disclosed embodiments, the rigid body transformation is a transformation that includes translation and rotation. In this embodiment, considering that the lens surface of the vehicle-mounted camera may not be parallel to the steering wheel column axis when the vehicle-mounted camera is installed, that is, the vehicle-mounted camera may have a certain elevation angle or depression angle with respect to the steering wheel column axis, when the rigid body transformation is performed, the angle of the lens surface of the vehicle-mounted camera with respect to the steering wheel column axis may also be introduced to more accurately determine the first transformation matrix so as to adapt to the calibration of the vehicle-mounted camera with different installation modes.
It can be understood that, in this embodiment, in combination with the preset angle of the lens surface of the vehicle-mounted camera with respect to the axis of the tubular column, since the preset angle is determined when the vehicle-mounted camera is installed, and is not required to be obtained through real-time calculation, the way of performing rigid body transformation on the telescopic length data, the deflection angle data and the preset angle to determine the first transformation matrix can be applied to different installation ways of the vehicle-mounted camera on the basis of not affecting the efficiency, and the universality of the calibration scheme of the present disclosure can be improved.
In some embodiments, the method further comprises:
acquiring a first image acquired by the vehicle-mounted camera at the current position and a second image acquired by the vehicle-mounted camera at the reference position;
performing feature element identification and matching on the first image and the second image after determining a first transformation matrix of the current position relative to the reference position based on the in-vehicle camera first position information;
determining a mapping relation matrix of the first image and the second image according to the matching result of the characteristic elements in the first image and the second image;
updating the first transformation matrix based on the mapping relation matrix.
In this embodiment, after the camera calibration device acquires a first image acquired by the vehicle-mounted camera at the current position and a second image acquired by the vehicle-mounted camera at the reference position, the first image and the second image are subjected to feature element recognition and matching. The feature elements may be objects or feature points of a specific type in the image, such as "B" pillars, also called center pillars, located between the front and rear doors of the vehicle, roof boundary points, etc. Features in each image can be extracted, for example, through a neural network model, matching is performed based on the features in the first image and the features in the second image, and then a mapping relation matrix of the first image and the second image is determined based on the positions of the features with the matching degree larger than a preset matching threshold value, which correspond to the first image and the second image.
It should be noted that, when determining the mapping relationship matrix of the first image and the second image, the present disclosure may preset a transformation matrix, for example, a 3 × 3 zero matrix, use the preset transformation matrix to map the position (first position) in the first image corresponding to the feature whose matching degree is greater than the preset matching threshold, and use the position (second position) in the second image corresponding to the feature whose matching degree is greater than the preset matching threshold as the mapping target, continuously adjust the element values in the preset transformation matrix until the product of the preset transformation matrix and the first position is the same as the second position, and at this time, obtain the mapping relationship matrix.
After determining the mapping relationship matrix of the first image and the second image based on the matching result of the feature elements in the first image and the second image, the first transformation matrix may be updated based on the mapping relationship matrix. When the first transformation matrix is updated based on the mapping relation matrix, for example, a matrix obtained by weighting and adding the mapping relation matrix and the first transformation matrix is determined as the updated first transformation matrix. The setting of the weight can be preset according to experience, the sum of the weights of the mapping relation matrix and the first transformation matrix is 1, and the larger the weight is, the higher the importance degree of the corresponding matrix is represented.
It can be understood that, after the mapping relation matrix is determined based on the visual image information, the first transformation matrix is updated, and since the first transformation matrix is obtained based on the mechanical signal and the mapping relation matrix is obtained based on the visual signal, in this embodiment, an error in the mechanical signal, for example, due to assembly, can be compensated through the visual signal, thereby achieving determination of the first transformation matrix with higher precision, and thus improving calibration accuracy of the vehicle-mounted camera.
In some embodiments, the acquiring first position information of the current position of the vehicle-mounted camera relative to the reference position includes:
respectively carrying out target detection on the first image and the second image, and determining the position of a target object in the first image and the position of the target object in the second image;
determining the first position information as a deviation of the position of the target object in the first image relative to the position in the second image;
the determining a first transformation matrix of the current position relative to the reference position based on the in-vehicle camera first position information includes:
determining the first transformation matrix based on the deviation.
In this embodiment, the first image acquired by the camera calibration device at the current position of the vehicle-mounted camera and the second image acquired by the vehicle-mounted camera at the reference position may be images including the same target object whose position is fixed in the vehicle cabin. For example, the same target object with a fixed position in the vehicle cabin may be a sunroof frame or a "B" pillar, etc. in the vehicle cabin.
The present disclosure performs target detection on the first image and the second image, respectively, and after determining the position of the target object in the first image and the position of the target object in the second image, the present disclosure may determine the deviation of the positions of the target object in the first image and the second image, and use the deviation of the positions as the first position information.
It should be noted that, in this embodiment, the deviation of the position may be caused by translation and rotation of the current position of the onboard camera with respect to the reference position, so that the deviation of the position also includes translation information and rotation information, and correspondingly, the first transformation matrix determined based on the deviation of the position also includes translation information and rotation information.
When determining the first transformation matrix based on the deviation of the position, the disclosed embodiments may determine the first transformation matrix of the current position relative to the reference position based on the deviation of the position multiplied by an internal reference matrix of the camera, for example.
It will be appreciated that in this embodiment, the solution is simple and effective based on the detection of a target object in the visual-based image information and determining the first transformation matrix based on a detected positional deviation of the same target object in different images, i.e. determining the first transformation matrix using machine vision.
In some embodiments, the obtaining third position information of the reference position in a cabin coordinate system includes:
acquiring fourth position information of a rotating center point of the steering wheel column under the cabin coordinate system;
acquiring fifth position information of the reference position relative to a rotation central point of the steering wheel column;
and determining third position information of the reference position in a vehicle cabin coordinate system according to the fourth position information and the fifth position information.
Fig. 3 and 4 are exemplary diagrams of three-dimensional data of a rotation center point of a steering wheel column in a cabin coordinate system according to an embodiment of the present disclosure, where 31 is a cabin coordinate origin, 32 is a rotation center of the steering wheel column, a distance Z1 of the rotation center 32 of the steering wheel column in a height direction from the cabin origin 31, a distance Y1 in a front axle direction, and a distance X1 in a front advancing direction are fourth position information, as shown in fig. 3 and 4. The direction of the front axle of the vehicle can be the direction between two front wheels when the vehicle runs in the forward direction; the height direction can be the direction vertical to the direction of the front axle of the vehicle, and the height of the vehicle roof from the horizontal ground can be measured in the height direction; the vehicle advancing direction can be the direction in which the vehicle is driving, and the distance between a front row seat and a rear row seat in the vehicle can be measured in the vehicle advancing direction.
Further, the position information of the center point of the in-vehicle camera 22 of the reference position with respect to the rotation center O point of the steering wheel column in fig. 2 is fifth position information of the reference position with respect to the rotation center point of the steering wheel column. The third position information of the reference position in the vehicle cabin coordinate system can be determined based on the fourth position information and the fifth position information.
In this embodiment, the third position information of the reference position in the cabin coordinate system is determined with intermediate conversion of the rotation center point of the steering wheel column. In another embodiment, since the reference position is also a position determined in advance, the third position information is also determined directly from the positional relationship between the reference position in the mechanical design information and the cabin origin of the cabin coordinate system.
Fig. 5 and 6 are exemplary diagrams of three-dimensional data of the reference position in the cabin coordinate system in the embodiment of the present disclosure, and as shown in fig. 5 and 6, 31 is the cabin coordinate origin, 33 is the reference position, a distance Z2 of the reference position 33 in the height direction from the cabin origin 31, a distance Y2 in the vehicle front axis direction, and a distance X2 in the vehicle front direction are the third position information.
It can be understood that, in this embodiment, since the fourth position information and the fifth position information are both mechanical design information, and are obtained without performing additional complex calculation, the calibration efficiency of the vehicle-mounted camera disclosed by the disclosure is facilitated.
In some embodiments, the determining a second transformation matrix of the current position of the onboard camera with respect to the cabin coordinate system based on the first transformation matrix, the third position information, and the rotation angle information includes:
determining a translation matrix and a rotation matrix of the reference position relative to the cabin coordinate system according to the third position information and the rotation angle information;
and combining the translation matrix and the rotation matrix and multiplying the combined translation matrix and rotation matrix by the first transformation matrix to obtain the second transformation matrix.
In the disclosed embodiment, a translation matrix of the reference position with respect to the cabin coordinate system may be determined according to the third position information, and the translation matrix is, for example, a matrix composed of X2, Y2, and Z2. In addition, when the rotation matrix is determined according to the third position information and the rotation angle information, three-dimensional space modeling can be performed, a connecting line of the reference position and the cabin origin of the cabin coordinate system is determined, included angles between the connecting line and each coordinate plane of the cabin coordinate system are obtained, active rotation matrixes of the connecting line around each axis of the cabin coordinate system are obtained, and the rotation matrix of the disclosure is obtained by multiplying the active rotation matrixes of the connecting line around each axis.
As mentioned above, the first transformation matrix includes translation and rotation information of the current position relative to the reference position, and the present disclosure may combine the translation matrix and the rotation matrix after obtaining the translation matrix and the rotation matrix of the reference position relative to the cabin coordinate system, for example, the translation matrix is B, and the rotation matrix is D, and then combine to obtain a matrix [ B, D ] including translation and rotation information of the reference position relative to the cabin coordinate system, and multiply [ B, D ] and the first rotation matrix M of the current position relative to the reference position to obtain the second transformation matrix, for example, the second transformation matrix is E, and E is [ B, D ] × M.
Fig. 7 is a schematic diagram of a calibration method of a vehicle-mounted camera in an embodiment of the present disclosure, and as shown in fig. 7, the calibration method of the camera of the present disclosure includes two steps, where in the first step, a real-time position of a vehicle-mounted camera identified by 71 is mapped to a reference position of the vehicle-mounted camera identified by 72, and in the mapping process, a mechanical extension and retraction and rotation signal of a directional pipe column may be used, and a picture of an interior of a vehicle cabin photographed by the vehicle-mounted camera may also be used. The steering wheel column mechanical telescopic signal comprises telescopic length data of the steering wheel column, and the rotation signal comprises deflection angle data; the images of the interior of the vehicle cabin, which are shot by the vehicle-mounted camera, comprise the first image and the second image of the present disclosure, and the first transformation matrix of the present disclosure is obtained through the mapping. The second step is to map the reference position of the vehicle-mounted camera identified by 72 to the passenger vehicle cabin space coordinate system identified by 73, and in the mapping process, a digital model can be designed by utilizing a vehicle cabin machine, for example, the fourth position information of the rotation center point of the steering wheel column in the vehicle cabin coordinate system and the fifth position information of the reference position relative to the rotation center point of the steering wheel column mentioned in the disclosure can be utilized, the third position information of the reference position in the vehicle cabin coordinate system is determined, and the translation matrix and the rotation matrix of the disclosure are determined based on the rotation angle information in the reference position.
And obtaining a second transformation matrix of the real-time position of the vehicle-mounted camera relative to the space coordinate system of the vehicle cabin in the embodiment of the disclosure based on the first transformation matrix, the translation matrix and the rotation matrix obtained in the two steps. It should be noted that, since the matrix transformation is reversible, if the position of the vehicle-mounted camera relative to the cabin space coordinate system is obtained in advance, the real-time position of the vehicle-mounted camera, that is, the real-time position of the steering wheel column, can be obtained based on the second transformation matrix.
Fig. 8 shows an exemplary diagram of a camera calibration apparatus provided in an embodiment of the present disclosure, and as can be seen from fig. 8, the camera calibration apparatus 800 includes:
a first acquiring module 801, configured to acquire first position information of a current position of the vehicle-mounted camera relative to a reference position; the vehicle-mounted camera is arranged on the steering wheel column;
a first determining module 802, configured to determine a first transformation matrix of the current position relative to the reference position based on first position information of the vehicle-mounted camera;
a second obtaining module 803, configured to obtain third position information of the reference position in a cabin coordinate system and rotation angle information of the steering wheel column at the reference position;
a second determining module 804, configured to determine a second transformation matrix of the current position of the vehicle-mounted camera with respect to the cabin coordinate system according to the first transformation matrix, the third position information, and the rotation angle information.
In some embodiments, the first obtaining module 801 is configured to read a mechanical signal of the steering wheel column to obtain telescopic length data and yaw angle data of the steering wheel column as the first position information.
In some embodiments, the first determining module 802 is configured to perform rigid body transformation based on the telescopic length data, the deflection angle data, and a preset angle to obtain the first transformation matrix; the preset angle is an angle of a lens surface of the vehicle-mounted camera relative to the axis of the steering wheel pipe column.
In some embodiments, the apparatus further comprises:
the third acquisition module is used for acquiring a first image acquired by the vehicle-mounted camera at the current position and a second image acquired by the vehicle-mounted camera at the reference position;
a third determination module for performing feature element recognition and matching on the first image and the second image after determining a first transformation matrix of the current position with respect to the reference position based on the in-vehicle camera first position information;
a fourth determining module, configured to determine a mapping relationship matrix between the first image and the second image according to a matching result of feature elements in the first image and the second image;
and the updating module is used for updating the first transformation matrix based on the mapping relation matrix.
In some embodiments, the first obtaining module 801 is configured to perform target detection on the first image and the second image, and determine a position of a target object in the first image and a position of the target object in the second image; determining the first position information as a deviation of the position of the target object in the first image relative to the position in the second image;
the first determining module 802 is configured to determine the first transformation matrix based on the deviation.
In some embodiments, the second obtaining module 802 is configured to obtain fourth position information of a rotation center point of the steering wheel column in the cabin coordinate system; acquiring fifth position information of the reference position relative to a rotation central point of the steering wheel column; and determining third position information of the reference position in a vehicle cabin coordinate system according to the fourth position information and the fifth position information.
In some embodiments, the second determining module 804 is configured to determine a translation matrix and a rotation matrix of the reference position with respect to the cabin coordinate system according to the third position information and the rotation angle information; and combining the translation matrix and the rotation matrix and multiplying the combined translation matrix and rotation matrix by the first transformation matrix to obtain the second transformation matrix.
The above description of the apparatus embodiments, similar to the above description of the method embodiments, has similar beneficial effects as the method embodiments. For technical details not disclosed in the embodiments of the apparatus of the present disclosure, reference is made to the description of the embodiments of the method of the present disclosure.
It should be noted that, in the embodiment of the present disclosure, if the calibration method of the vehicle-mounted camera is implemented in the form of a software functional module and is sold or used as an independent product, the calibration method may also be stored in a computer-readable storage medium. Based on such understanding, the technical solutions of the embodiments of the present disclosure may be embodied in the form of a software product, which is stored in a storage medium and includes several instructions for causing a computer device (which may be a personal computer, a server, or a network device) to execute all or part of the methods described in the embodiments of the present disclosure. And the aforementioned storage medium includes: various media capable of storing program codes, such as a usb disk, a removable hard disk, a Read Only Memory (ROM), a magnetic disk, or an optical disk. Thus, embodiments of the present disclosure are not limited to any specific combination of hardware and software.
Correspondingly, the embodiment of the present disclosure provides a computer device, which includes a memory and a processor, wherein the memory stores a computer program that can be run on the processor, and the processor implements the steps of the method when executing the program.
Correspondingly, the disclosed embodiments provide a computer-readable storage medium, on which a computer program is stored, which, when executed by a processor, implements the steps in the above-described method. The computer readable storage medium may be transitory or non-transitory.
Accordingly, embodiments of the present disclosure provide a computer program product comprising a non-transitory computer readable storage medium storing a computer program, which when read and executed by a computer, implements some or all of the steps of the above method. The computer program product may be embodied in hardware, software or a combination thereof. In an alternative embodiment, the computer program product is embodied in a computer storage medium, and in another alternative embodiment, the computer program product is embodied in a Software product, such as a Software Development Kit (SDK), or the like.
Here, it should be noted that: the above description of the storage medium, the computer program product and the device embodiments is similar to the description of the method embodiments described above, with similar advantageous effects as the method embodiments. For technical details not disclosed in the embodiments of the storage medium, the computer program product and the device of the present disclosure, reference is made to the description of the embodiments of the method of the present disclosure.
It should be noted that fig. 9 is a schematic diagram of a hardware entity of a computer device in an embodiment of the present disclosure, and as shown in fig. 9, the hardware entity of the computer device 900 includes: a processor 901, a communication interface 902, and a memory 903, wherein:
the processor 901 generally controls the overall operation of the computer device 900.
The communication interface 902 may enable the computer device to communicate with other terminals or servers via a network.
The Memory 903 is configured to store instructions and applications executable by the processor 901, and may also buffer data (e.g., image data, audio data, voice communication data, and video communication data) to be processed or already processed by the processor 901 and modules in the computer apparatus 900, and may be implemented by a FLASH Memory (FLASH) or a Random Access Memory (RAM). Data may be transferred between the processor 901, the communication interface 902, and the memory 903 via the bus 904.
It should be appreciated that reference throughout this specification to "one embodiment" or "an embodiment" means that a particular feature, structure or characteristic described in connection with the embodiment is included in at least one embodiment of the present disclosure. Thus, the appearances of the phrases "in one embodiment" or "in an embodiment" in various places throughout this specification are not necessarily all referring to the same embodiment. Furthermore, the particular features, structures, or characteristics may be combined in any suitable manner in one or more embodiments. It should be understood that, in various embodiments of the present disclosure, the sequence numbers of the above-mentioned processes do not mean the execution sequence, and the execution sequence of each process should be determined by its function and inherent logic, and should not constitute any limitation on the implementation process of the embodiments of the present disclosure. The above-mentioned serial numbers of the embodiments of the present disclosure are merely for description and do not represent the merits of the embodiments.
It should be noted that, in this document, the terms "comprises," "comprising," or any other variation thereof, are intended to cover a non-exclusive inclusion, such that a process, method, article, or apparatus that comprises a list of elements does not include only those elements but may include other elements not expressly listed or inherent to such process, method, article, or apparatus. Without further limitation, an element defined by the phrase "comprising an … …" does not exclude the presence of other like elements in a process, method, article, or apparatus that comprises the element.
In the several embodiments provided in the present disclosure, it should be understood that the disclosed apparatus and method may be implemented in other ways. The above-described device embodiments are merely illustrative, for example, the division of the unit is only a logical functional division, and there may be other division ways in actual implementation, such as: multiple units or components may be combined, or may be integrated into another system, or some features may be omitted, or not implemented. In addition, the coupling, direct coupling or communication connection between the components shown or discussed may be through some interfaces, and the indirect coupling or communication connection between the devices or units may be electrical, mechanical or other forms.
The units described as separate parts may or may not be physically separate, and parts displayed as units may or may not be physical units; can be located in one place or distributed on a plurality of network units; some or all of the units can be selected according to actual needs to achieve the purpose of the solution of the embodiment.
In addition, all the functional units in the embodiments of the present disclosure may be integrated into one processing unit, or each unit may be separately regarded as one unit, or two or more units may be integrated into one unit; the integrated unit can be realized in a form of hardware, or in a form of hardware plus a software functional unit.
Those of ordinary skill in the art will understand that: all or part of the steps for realizing the method embodiments can be completed by hardware related to program instructions, the program can be stored in a computer readable storage medium, and the program executes the steps comprising the method embodiments when executed; and the aforementioned storage medium includes: a removable storage device, a ROM, a magnetic or optical disk, or other various media that can store program code.
Alternatively, the integrated unit of the present disclosure may be stored in a computer-readable storage medium if it is implemented in the form of a software functional module and sold or used as a separate product. Based on such understanding, the technical solutions of the present disclosure may be embodied in the form of a software product, which is stored in a storage medium and includes several instructions for causing a computer device (which may be a personal computer, a server, or a network device) to execute all or part of the methods described in the embodiments of the present disclosure. And the aforementioned storage medium includes: a removable storage device, a ROM, a magnetic or optical disk, or other various media that can store program code.
The above description is only an embodiment of the present disclosure, but the scope of the present disclosure is not limited thereto, and any person skilled in the art can easily conceive of changes or substitutions within the technical scope of the present disclosure, and all the changes or substitutions should be covered by the scope of the present disclosure. Therefore, the protection scope of the present disclosure shall be subject to the protection scope of the claims.
Claims (10)
1. A calibration method of a vehicle-mounted camera is characterized by comprising the following steps:
acquiring first position information of a current position of a vehicle-mounted camera relative to a reference position; the vehicle-mounted camera is arranged on the steering wheel column;
determining a first transformation matrix of the current position relative to the reference position based on first position information of the onboard camera;
acquiring third position information of the reference position in a cabin coordinate system and rotation angle information of the steering wheel column at the reference position;
determining a second transformation matrix of the current position of the onboard camera with respect to the cabin coordinate system based on the first transformation matrix, the third position information, and the rotation angle information.
2. The method of claim 1, wherein the obtaining first position information of the vehicle-mounted camera relative to the reference position at the current position comprises:
and reading the mechanical signal of the steering wheel column to acquire the telescopic length data and the deflection angle data of the steering wheel column as the first position information.
3. The method of claim 2, wherein determining a first transformation matrix of the current position relative to the reference position based on the in-vehicle camera first position information comprises:
performing rigid body transformation based on the telescopic length data, the deflection angle data and a preset angle to obtain a first transformation matrix; the preset angle is an angle of a lens surface of the vehicle-mounted camera relative to the axis of the steering wheel pipe column.
4. The method of claim 3, further comprising:
acquiring a first image acquired by the vehicle-mounted camera at the current position and a second image acquired by the vehicle-mounted camera at the reference position;
performing feature element identification and matching on the first image and the second image after determining a first transformation matrix of the current position relative to the reference position based on the in-vehicle camera first position information;
determining a mapping relation matrix of the first image and the second image according to the matching result of the characteristic elements in the first image and the second image;
updating the first transformation matrix based on the mapping relation matrix.
5. The method of claim 4, wherein the obtaining first position information of a current position of the onboard camera relative to a reference position comprises:
respectively carrying out target detection on the first image and the second image, and determining the position of a target object in the first image and the position of the target object in the second image;
determining the first position information as a deviation of the position of the target object in the first image relative to the position in the second image;
the determining a first transformation matrix of the current position relative to the reference position based on the in-vehicle camera first position information includes:
determining the first transformation matrix based on the deviation.
6. The method according to any one of claims 1 to 5, wherein the acquiring third position information of the reference position in a cabin coordinate system comprises:
acquiring fourth position information of a rotating center point of the steering wheel column under the cabin coordinate system;
acquiring fifth position information of the reference position relative to a rotation central point of the steering wheel column;
and determining third position information of the reference position in a vehicle cabin coordinate system according to the fourth position information and the fifth position information.
7. The method according to any one of claims 1 to 6, wherein the determining a second transformation matrix of the current position of the onboard camera relative to the cabin coordinate system from the first transformation matrix, the third position information, and the rotation angle information comprises:
determining a translation matrix and a rotation matrix of the reference position relative to the cabin coordinate system according to the third position information and the rotation angle information;
and combining the translation matrix and the rotation matrix and multiplying the combined translation matrix and rotation matrix by the first transformation matrix to obtain the second transformation matrix.
8. A calibration device of a vehicle-mounted camera is characterized by comprising:
the first acquisition module is used for acquiring first position information of the current position of the vehicle-mounted camera relative to a reference position; the vehicle-mounted camera is arranged on the steering wheel column;
a first determination module, configured to determine a first transformation matrix of the current position relative to the reference position based on first position information of the vehicle-mounted camera;
a second obtaining module, configured to obtain third position information of the reference position in a cabin coordinate system and rotation angle information of the steering wheel column at the reference position;
a second determination module for determining a second transformation matrix of the current position of the onboard camera with respect to the cabin coordinate system based on the first transformation matrix, the third position information, and the rotation angle information.
9. A computer device, comprising: a processor; a memory for storing processor-executable instructions;
wherein the processor is configured to perform the method of any one of claims 1 to 7.
10. A computer-readable storage medium, on which a computer program is stored, which, when being executed by a processor, carries out the method of any one of claims 1 to 7.
Priority Applications (2)
| Application Number | Priority Date | Filing Date | Title |
|---|---|---|---|
| CN202210616046.0A CN114897996A (en) | 2022-05-31 | 2022-05-31 | Vehicle-mounted camera calibration method and device, computer equipment and storage medium |
| PCT/CN2023/090821 WO2023231653A1 (en) | 2022-05-31 | 2023-04-26 | Vehicle-mounted camera calibration method and apparatus, computer device, storage medium and product |
Applications Claiming Priority (1)
| Application Number | Priority Date | Filing Date | Title |
|---|---|---|---|
| CN202210616046.0A CN114897996A (en) | 2022-05-31 | 2022-05-31 | Vehicle-mounted camera calibration method and device, computer equipment and storage medium |
Publications (1)
| Publication Number | Publication Date |
|---|---|
| CN114897996A true CN114897996A (en) | 2022-08-12 |
Family
ID=82725226
Family Applications (1)
| Application Number | Title | Priority Date | Filing Date |
|---|---|---|---|
| CN202210616046.0A Pending CN114897996A (en) | 2022-05-31 | 2022-05-31 | Vehicle-mounted camera calibration method and device, computer equipment and storage medium |
Country Status (2)
| Country | Link |
|---|---|
| CN (1) | CN114897996A (en) |
| WO (1) | WO2023231653A1 (en) |
Cited By (2)
| Publication number | Priority date | Publication date | Assignee | Title |
|---|---|---|---|---|
| WO2023231653A1 (en) * | 2022-05-31 | 2023-12-07 | 上海商汤智能科技有限公司 | Vehicle-mounted camera calibration method and apparatus, computer device, storage medium and product |
| CN117437288A (en) * | 2023-12-19 | 2024-01-23 | 先临三维科技股份有限公司 | Photogrammetry method, device, equipment and storage medium |
Family Cites Families (5)
| Publication number | Priority date | Publication date | Assignee | Title |
|---|---|---|---|---|
| TWI398160B (en) * | 2009-12-01 | 2013-06-01 | Ind Tech Res Inst | Camera calibration system and coordinate data generation system and method thereof |
| CN112686958A (en) * | 2019-10-18 | 2021-04-20 | 上海商汤智能科技有限公司 | Calibration method and device and electronic equipment |
| CN112738487B (en) * | 2020-12-24 | 2022-10-11 | 阿波罗智联(北京)科技有限公司 | Image projection method, device, equipment and storage medium |
| CN114332242A (en) * | 2021-12-29 | 2022-04-12 | 黑芝麻智能科技有限公司 | Method, device and equipment for calibrating coordinate system of object outside camera view field and storage medium |
| CN114897996A (en) * | 2022-05-31 | 2022-08-12 | 上海商汤临港智能科技有限公司 | Vehicle-mounted camera calibration method and device, computer equipment and storage medium |
-
2022
- 2022-05-31 CN CN202210616046.0A patent/CN114897996A/en active Pending
-
2023
- 2023-04-26 WO PCT/CN2023/090821 patent/WO2023231653A1/en unknown
Cited By (3)
| Publication number | Priority date | Publication date | Assignee | Title |
|---|---|---|---|---|
| WO2023231653A1 (en) * | 2022-05-31 | 2023-12-07 | 上海商汤智能科技有限公司 | Vehicle-mounted camera calibration method and apparatus, computer device, storage medium and product |
| CN117437288A (en) * | 2023-12-19 | 2024-01-23 | 先临三维科技股份有限公司 | Photogrammetry method, device, equipment and storage medium |
| CN117437288B (en) * | 2023-12-19 | 2024-05-03 | 先临三维科技股份有限公司 | Photogrammetry method, device, equipment and storage medium |
Also Published As
| Publication number | Publication date |
|---|---|
| WO2023231653A1 (en) | 2023-12-07 |
Similar Documents
| Publication | Publication Date | Title |
|---|---|---|
| CN109767475B (en) | External parameter calibration method and system for sensor | |
| CN114897996A (en) | Vehicle-mounted camera calibration method and device, computer equipment and storage medium | |
| CN109690623B (en) | System and method for recognizing pose of camera in scene | |
| CN107554430B (en) | Vehicle blind area visualization method, device, terminal, system and vehicle | |
| CN106981082B (en) | Vehicle-mounted camera calibration method and device and vehicle-mounted equipment | |
| CN106476728B (en) | Motion compensation for vehicle-mounted vehicle sensors | |
| CN111731187A (en) | Automobile A-pillar blind area image display system and method | |
| JP2017078923A (en) | Calibration device, calibration method, and calibration program | |
| US20160224852A1 (en) | Vehicle operator monitoring system and method | |
| US10964059B2 (en) | Static camera calibration using motion of vehicle portion | |
| CN111027381A (en) | Method, device, equipment and storage medium for recognizing obstacle by monocular camera | |
| KR101271639B1 (en) | A extrinsic parameter calibration method and system for camera on mobile device | |
| CN111381248A (en) | Obstacle detection method and system considering vehicle bump | |
| CN109255329A (en) | Determine method, apparatus, storage medium and the terminal device of head pose | |
| CN109345591B (en) | Vehicle posture detection method and device | |
| CN110555885A (en) | calibration method and device of vehicle-mounted camera and terminal | |
| CN113661495A (en) | Sight line calibration method, sight line calibration device, sight line calibration equipment, sight line calibration system and sight line calibration vehicle | |
| US20230154197A1 (en) | A system and method to track a coupled vehicle | |
| CN116543032B (en) | Impact object ranging method, device, ranging equipment and storage medium | |
| CN112132902B (en) | Vehicle-mounted camera external parameter adjusting method and device, electronic equipment and medium | |
| CN110751598B (en) | Vehicle hinge point coordinate calibration method and device, computer equipment and storage medium | |
| CN113822593A (en) | Security situation assessment method and device, storage medium and electronic equipment | |
| CN113888463A (en) | Wheel rotation angle detection method and device, electronic device and storage medium | |
| US20200090307A1 (en) | Surround View System for a Vehicle | |
| CN116403188A (en) | Method, device, equipment and storage medium for generating automobile A column blind area image |
Legal Events
| Date | Code | Title | Description |
|---|---|---|---|
| PB01 | Publication | ||
| PB01 | Publication | ||
| SE01 | Entry into force of request for substantive examination | ||
| SE01 | Entry into force of request for substantive examination |