WO2022141324A1 - 摄像头的硬件在环标定、靶点设置方法、系统及相关设备 - Google Patents

摄像头的硬件在环标定、靶点设置方法、系统及相关设备 Download PDF

Info

Publication number
WO2022141324A1
WO2022141324A1 PCT/CN2020/141855 CN2020141855W WO2022141324A1 WO 2022141324 A1 WO2022141324 A1 WO 2022141324A1 CN 2020141855 W CN2020141855 W CN 2020141855W WO 2022141324 A1 WO2022141324 A1 WO 2022141324A1
Authority
WO
WIPO (PCT)
Prior art keywords
target
camera
dimensional coordinates
screen
freedom
Prior art date
Application number
PCT/CN2020/141855
Other languages
English (en)
French (fr)
Inventor
张德明
洪峰
陈曦
黄成凯
郭洪强
Original Assignee
华为技术有限公司
Priority date (The priority date is an assumption and is not a legal conclusion. Google has not performed a legal analysis and makes no representation as to the accuracy of the date listed.)
Filing date
Publication date
Application filed by 华为技术有限公司 filed Critical 华为技术有限公司
Priority to CN202080004242.1A priority Critical patent/CN112753047B/zh
Priority to PCT/CN2020/141855 priority patent/WO2022141324A1/zh
Publication of WO2022141324A1 publication Critical patent/WO2022141324A1/zh

Links

Images

Classifications

    • G—PHYSICS
    • G06—COMPUTING; CALCULATING OR COUNTING
    • G06T—IMAGE DATA PROCESSING OR GENERATION, IN GENERAL
    • G06T7/00—Image analysis
    • G06T7/80—Analysis of captured images to determine intrinsic or extrinsic camera parameters, i.e. camera calibration

Definitions

  • the present application relates to the field of hardware-in-the-loop (HIL) technology, and in particular, to a method, system, and related equipment for hardware-in-the-loop calibration and target setting of a camera.
  • HIL hardware-in-the-loop
  • Hardware-in-the-loop is an important method for validating Advanced Driving Assistance System (ADAS), and video injection is an important step to build hardware-in-the-loop.
  • ADAS Advanced Driving Assistance System
  • traditional video injection equipment is expensive and has poor applicability.
  • different cameras need to use different video injection equipment, which greatly increases the cost of video injection.
  • Camera-in-the-loop is a technology that achieves the video injection function by shooting the screen directly through the camera in the camera obscura. Camera-in-the-loop does not require specific video injection equipment, is cheap and has strong applicability, and can complete video injection of different models of cameras.
  • the camera-in-the-loop can achieve video reprocessing by simulating various environments in the camera obscura, such as rainy days and foggy days, which greatly enriches the needs of users. Therefore, camera-in-the-loop has become an important technology for realizing video injection function.
  • the essence of camera-in-the-loop is to make the image captured by the camera exactly the same as the actual image, and its ultimate goal is to achieve pixel-level alignment, that is, to achieve the exact same pixel coordinates as the actual pixel coordinates.
  • the alignment accuracy of the camera is mostly guaranteed by machining or simple manual adjustment, so the accuracy is low and it depends greatly on the experience of the adjuster, the adjustment time is long, and the efficiency is low. Therefore, how to improve the alignment accuracy and calibration efficiency of the camera has become an urgent technical problem to be solved.
  • the embodiments of the present application disclose a hardware-in-the-loop calibration of a camera, a method, a system for setting a target point, and related equipment, which are beneficial to improve alignment accuracy and calibration efficiency.
  • a first aspect of the embodiments of the present application discloses a hardware-in-the-loop calibration method for a camera.
  • the method includes: determining a weight corresponding to each target point in a plurality of target points, wherein the plurality of target points are targets on a display device point, the weight corresponding to the target point located in the target area on the display device is greater than the weight corresponding to the target point located outside the target area, and the target area of the display device is used to display a part of the calibration image; the calibration image is the hardware in the loop of the camera in progress During calibration, the image displayed by the device is displayed; according to the weight, multiple target points are sampled with replacement for P times to obtain P target point groups.
  • P is a positive integer; the hardware-in-the-loop calibration of the camera is performed according to the P target point groups and the current pose of the camera.
  • different weights are set for the target points in different regions, and the target points are resampled with replacement based on the weights, which not only ensures the independence of each group of target point data, but also focuses on improving the camera's performance.
  • the probability of the target points corresponding to the focus area being extracted, and the hardware-in-the-loop calibration of the camera based on the sampled target points and the current pose of the camera can effectively improve the alignment accuracy of the camera to the focus area.
  • the hardware for the camera has a plurality of target points on a display device for ring calibration, and a calibration image is displayed on the display device, the calibration image has the focus area of the camera, and the calibration image is displayed on the display device.
  • the display area used to display the focus area is the target area, and the weight corresponding to the target point within the target area is set to be greater than the weight corresponding to the target point outside the target area; then based on the weight probability Perform P sampling of multiple targets with replacement to obtain P target groups, that is, the greater the weight corresponding to the target, the greater the probability of the target being extracted; due to the weight of the target in the target area is greater than the weight corresponding to the target points outside the target area, so each time you sample, the probability of the target points in the target area being extracted is greater, and the number of target points in the target area is relatively larger.
  • the embodiments of the present application can effectively improve the alignment accuracy of the camera to the focus area, and ensure the personalized alignment requirement of the user.
  • the hardware-in-the-loop calibration of the camera is performed according to the P target point groups and the current pose of the camera, including: determining P first target poses according to the P target point groups, wherein the P first target poses are The target group is in one-to-one correspondence with the P first target poses; the display device currently displaying the calibration image is captured by the camera to obtain the target image, and the current pose of the camera is determined according to the multiple target points and the target image; The hardware-in-the-loop calibration of the camera is performed on the average of the P first target poses and the current pose.
  • P first target poses are determined according to the P target groups obtained by re-sampling; then, the camera is used to perform image acquisition on the display device currently displaying the calibration image to obtain the target image, And determine the current pose of the camera according to the multiple target points on the display device and the target image; then take the average value of the P first target poses as the pose to be adjusted, and perform hardware-in-the-loop calibration of the camera, That is to say, the current pose of the camera is adjusted to the average value of the P first target poses; since the target pose of the camera is calculated for many times by the method of resampling, the pose that needs to be adjusted is obtained by the method of averaging.
  • the pose that needs to be adjusted not only provides a reference benchmark for the camera's pose adjustment direction, but also focuses on ensuring the alignment accuracy of the camera's key focus area, which is conducive to improving the calibration efficiency and alignment accuracy of the camera alignment.
  • the method further includes: capturing images of the calibration board through a camera to obtain Q calibration images, where Q is A positive integer; perform corner detection on each calibration image in the Q calibration images to obtain the pixel coordinates of each corner in each calibration image; The pixel coordinates and the third three-dimensional coordinates of each corner point in each calibration image obtain the camera's internal parameter matrix and the camera's distortion coefficient, where the third three-dimensional coordinates are the coordinates in the calibration board coordinate system.
  • the camera is used to collect images of the calibration board to obtain multiple calibration images, and the internal parameter matrix of the camera and the distortion coefficient of the camera are obtained with the multiple calibration images, and the pose calculation in the calibration process needs to use
  • the internal parameter matrix of the camera and the distortion coefficient of the camera can provide a basis for calculating the pose that the camera needs to adjust to and the current pose of the camera, which is beneficial to improve the hardware-in-the-loop alignment accuracy of the camera.
  • determining the P first target poses according to the P target point groups includes: for each target point group in the P target point groups, performing the following steps to obtain the P first targets Pose: Obtain the two-dimensional coordinates of each target point in the target target group, where the target target group is any one of the P target point groups, the two-dimensional coordinates are the coordinates in the second coordinate system, and the second The coordinate system is a coordinate system with an outer boundary point of the screen frame of the display device as the origin, the vertical direction of the screen frame as the first coordinate axis, and the horizontal direction of the screen frame as the second coordinate axis; The two-dimensional coordinates of each target point, the first three-dimensional coordinates, the internal parameter matrix of the camera and the distortion coefficient of the camera are calculated to obtain the first target pose corresponding to the target target group, wherein the first three-dimensional coordinates are in the first coordinate system.
  • the first coordinate system takes a boundary point of the screen of the display device as the origin, the vertical direction of the screen as the first coordinate axis, the horizontal direction of the screen as the second coordinate axis, and the vertical direction of the screen as the third coordinate axis coordinate system.
  • the first target position corresponding to the target target group is calculated and obtained according to the two-dimensional coordinates of each target point in the target target point group, the first three-dimensional coordinate, the internal parameter matrix of the camera, and the distortion coefficient of the camera.
  • the two-dimensional coordinates are the coordinates in the second coordinate system
  • the first three-dimensional coordinates are the coordinates in the first coordinate system
  • the two-dimensional coordinates of each target point in the second coordinate system can be calculated according to the pixel coordinates of the calibration image
  • the three-dimensional coordinates of each target point in the total station coordinate system can be obtained through the total station,
  • the three-dimensional coordinates of each target point in the total station coordinate system are converted into three-dimensional coordinates in the first coordinate system; thus, it is beneficial to obtain the first target pose of each target point group, and then the camera needs to be adjusted to. 's pose.
  • determining the current pose of the camera according to the multiple target points and the target image includes: performing circle center detection on the target image to obtain the pixel of each target point in the target image among the multiple target points Coordinates; according to the pixel coordinates of each target point in the target image, the first three-dimensional coordinate of each target point in the plurality of target points, the internal parameter matrix of the camera, and the distortion coefficient of the camera, the current pose, wherein the first three-dimensional coordinates are coordinates in a first coordinate system, and the first coordinate system is a boundary point of the screen of the display device as the origin, the vertical direction of the screen as the first coordinate axis, and the horizontal direction of the screen as the first coordinate axis is a coordinate system with the second coordinate axis and the vertical direction of the screen as the third coordinate axis.
  • the camera is used to collect images of the display device currently displaying the calibration image, and after obtaining the target image, each target point among the multiple target points on the display device can be obtained by performing circle center detection on the target image.
  • the pixel coordinates in the target image are helpful to calculate the current pose according to the pixel coordinates of each target point in the target image, the first three-dimensional coordinates of each target point, the camera's internal parameter matrix, and the camera's distortion coefficient.
  • the plurality of target points include M first target points in the screen of the display device and N second target points on the screen frame, wherein the M first target points and the Nth target points The two target points are not coplanar, and M and N are integers greater than or equal to 3.
  • the method further includes: acquiring M first The second three-dimensional coordinates of the target point and the second three-dimensional coordinates of the N second target points, wherein the second three-dimensional coordinates are the coordinates under the coordinate system of the total station; the second three-dimensional coordinates of the M first target points are coordinated.
  • the transformation is performed to obtain the first three-dimensional coordinates of the M first target points, and the coordinate transformation is performed on the second three-dimensional coordinates of the N second target points to obtain the first three-dimensional coordinates of the N second target points.
  • M first target points are set in the screen of the display device
  • N second target points are set on the screen frame of the display device
  • the M first target points and the Nth are not coplanar, so there is no need to use multiple non-coplanar calibration boards when calibrating, which solves the problem of limited space for setting the target points in the hardware-in-the-loop system of the camera.
  • Converting the coordinates in the total station coordinate system to the first coordinate system lays the foundation for subsequent calibration feedback prompts, which is a key step to improve the camera alignment effect.
  • the hardware-in-the-loop calibration of the camera is performed according to the average value of the P first target poses and the current pose, including: making a difference between the second target pose and the current pose to obtain the current pose pose residuals, where the second target pose is the average of P first target poses, and the current pose residuals include 3 current position DOF residuals and 3 current posture DOF residuals;
  • the positional degrees of freedom residuals are adjusted in order from large to small in order to adjust each positional degree of freedom of the camera, so that the difference between the first positional degree of freedom of the camera and the first positional degree of freedom of the second target pose is smaller than the preset first degree of freedom.
  • the positional degree of freedom residual threshold the difference between the second positional degree of freedom of the camera and the second positional degree of freedom of the second target pose is less than the preset second positional degree of freedom residual threshold, the third positional degree of freedom of the camera and The difference between the third position degrees of freedom of the second target pose is less than the preset third position freedom degree of freedom threshold; and then adjust the camera's various attitude degrees of freedom in descending order of the current attitude degree of freedom residuals, So that the difference between the first attitude degree of freedom of the camera and the first attitude degree of freedom of the second target attitude is less than the preset first attitude degree of freedom residual threshold, the second attitude degree of freedom of the camera and the second target attitude The difference of the second attitude degree of freedom is less than the preset second attitude degree of freedom residual threshold, and the difference between the third attitude degree of freedom of the camera and the third attitude degree of freedom of the second target attitude is less than the preset third degree of freedom.
  • Pose DOF residual threshold when the posture of the camera is adjusted from the current posture to the posture that needs to be adjusted, the component with the larger residual error in the position is fed back first, and the component with the larger residual error is adjusted in a corresponding direction.
  • a second aspect of the embodiments of the present application discloses a method for setting a hardware-in-the-loop calibration target point for a camera, including: setting M first target points on a screen of a display device, and setting N number of first target points on a screen frame of the display device Two target points, wherein the display device is used to display the calibration image when the hardware of the camera is performing ring calibration, the M first target points and the N second target points are not coplanar, and M and N are integers greater than or equal to 3 .
  • M first target points are set in the screen of the display device
  • N second target points are set on the screen frame of the display device
  • the M first target points and the Nth are not coplanar, so there is no need to use multiple non-coplanar calibration boards during calibration, which solves the problem of limited space for target setting in the hardware-in-the-loop system of the camera.
  • the method further includes: acquiring the second three-dimensional coordinates of the M first target points and the second three-dimensional coordinates of the N second target points, wherein the second three-dimensional coordinates are total station coordinates Coordinates under the system; coordinate transformation is performed on the second three-dimensional coordinates of the M first target points and the second three-dimensional coordinates of the N second target points, and the first three-dimensional coordinates of the M first target points and the N second three-dimensional coordinates are obtained.
  • the first three-dimensional coordinates of the target point wherein the first three-dimensional coordinates are the coordinates under the first coordinate system, and the first coordinate system is a boundary point of the screen as the origin, the vertical direction of the screen as the first coordinate axis, and the screen as the first coordinate axis.
  • the horizontal direction is the second coordinate axis
  • the vertical direction of the screen is the coordinate system of the third coordinate axis; the first three-dimensional coordinates of the M first target points and the first three-dimensional coordinates of the N second target points are used for the hardware of the camera Calibration in the ring.
  • the coordinates of the target point in the total station coordinate system are converted into the first coordinate system, and the first three-dimensional coordinates of the target point in the first coordinate system can be used for the hardware of the camera.
  • the ring calibration is used to provide a basis for the subsequent hardware-in-the-loop calibration of the camera, which is beneficial to improve the camera alignment effect.
  • coordinate transformation is performed on the second three-dimensional coordinates of the M first target points and the second three-dimensional coordinates of the N second target points to obtain the first three-dimensional coordinates of the M first target points and
  • the first three-dimensional coordinates of the N second target points include: obtaining M first feature vectors and M first eigenvectors according to the second three-dimensional coordinates of the M first target points and the second three-dimensional coordinates of the M first target points Two eigenvectors; adopt the least squares method to calculate the intermediate vector according to the M first eigenvectors and the M second eigenvectors; obtain the control matrix according to the intermediate vector, and obtain the rotation vector and the translation vector according to the control matrix; According to the rotation vector and the translation The vector performs coordinate transformation on the second three-dimensional coordinates of the M first target points and the second three-dimensional coordinates of the N second target points to obtain the first three-dimensional coordinates of the M first target points and the first three-dimensional coordinates of the N second target points.
  • a three-dimensional coordinate In the embodiment of the present application, the first target point in the screen is used as the reference point to solve the rotation matrix and translation vector from the coordinate system of the total station to the first coordinate system;
  • the coordinates in the station coordinate system are converted into the first coordinate system, and the first three-dimensional coordinates of the target point in the first coordinate system are used for the hardware-in-the-loop calibration of the camera, so as to provide the subsequent hardware-in-the-loop calibration of the camera. It is helpful to improve the camera alignment effect.
  • the aspect ratio of the calibration image and the aspect ratio of the screen are the same as the aspect ratio of the display device, and the method further includes: performing circle center detection on the display device displaying the calibration image to obtain M The pixel coordinates of each first target point in the first target points; the pixel coordinates of each first target point in the M first target points are converted into the second coordinate system to obtain M first target points The two-dimensional coordinates of each first target point in the, wherein, the second coordinate system is based on an outer boundary point of the screen frame of the display device as the origin, the vertical direction of the screen frame as the first coordinate axis, and the screen frame as the first coordinate axis.
  • the horizontal direction is the coordinate system of the second coordinate axis; according to the two-dimensional coordinates of at least one first target point in the M first target points, the aspect ratio of the screen and the aspect ratio of the display device, it is determined that the N second target points are at Two-dimensional coordinates in the second coordinate system.
  • the pixel coordinates of the first target point are obtained by detecting the center of the circle; then the two-dimensional coordinates of the first target point are obtained by converting the pixel coordinates of the first target point to the second coordinate system;
  • the two-dimensional coordinates of the second target point are calculated from the two-dimensional coordinates of the first target point, so that the two-dimensional coordinates of all the target points used for the hardware-in-ring calibration of the camera can be obtained.
  • the method further includes: according to the proportional relationship between the size of the calibration image and the size of the outer border of the screen frame, performing a calibration on the calibration image.
  • a third aspect of the embodiments of the present application discloses a hardware-in-the-loop calibration device for a camera.
  • the device includes: a determination unit configured to determine a weight corresponding to each target point in a plurality of target points, wherein the plurality of target points are The target point on the display device, the weight corresponding to the target point located in the target area on the display device is greater than the weight corresponding to the target point located outside the target area, and the target area of the display device is used to display a part of the calibration image; the calibration image is in progress
  • the hardware of the camera displays the image displayed by the device when the ring is calibrated; the sampling unit is used to perform P sampling of multiple targets with replacement according to the weight, so as to obtain P target groups, wherein, for each sampling, the target The greater the weight corresponding to the point, the greater the probability that the target point is extracted, and P is a positive integer; the calibration unit is used to perform hardware-in-the-loop calibration of the camera according to the P target point groups and the current
  • the calibration unit is specifically configured to: determine the P first target poses according to the P target point groups, wherein the P target point groups correspond to the P first target poses one-to-one;
  • the camera is used to collect images of the display device currently displaying the calibration image to obtain the target image, and the current pose of the camera is determined according to multiple target points and target images;
  • the camera's hardware-in-the-loop calibration is specifically configured to: determine the P first target poses according to the P target point groups, wherein the P target point groups correspond to the P first target poses one-to-one;
  • the camera is used to collect images of the display device currently displaying the calibration image to obtain the target image, and the current pose of the camera is determined according to multiple target points and target images;
  • the camera's hardware-in-the-loop calibration is specifically configured to: determine the P first target poses according to the P target point groups, wherein the P target point groups correspond to the P first target poses one-to-one;
  • the camera is used to collect images of the display device currently displaying the calibration
  • the calibration unit is further configured to: before determining the P first target poses according to the P target point groups; perform image acquisition on the calibration board through a camera to obtain Q calibration images, wherein, Q is a positive integer; corner detection is performed on each calibration image in the Q calibration images to obtain the pixel coordinates of each corner in each calibration image; according to each corner in each calibration image in the Q calibration images
  • the pixel coordinates of the point and the third three-dimensional coordinates of each corner point in each calibration image obtain the camera's internal parameter matrix and the camera's distortion coefficient, where the third three-dimensional coordinates are the coordinates in the calibration board coordinate system.
  • the calibration unit is specifically configured to: for each target point group in the P target point groups, perform the following steps to obtain the P first target poses: obtain the The two-dimensional coordinates of each target point, wherein the target target point group is any one of the P target point groups, the two-dimensional coordinates are the coordinates under the second coordinate system, and the second coordinate system is based on the screen frame of the display device.
  • An outer boundary point is the origin, the vertical direction of the screen frame is the first coordinate axis, and the horizontal direction of the screen frame is the second coordinate system; according to the two-dimensional coordinates of each target point in the target target group, the first coordinate system A three-dimensional coordinate, the camera's internal parameter matrix, and the camera's distortion coefficient are calculated to obtain the first target pose corresponding to the target target group, wherein the first three-dimensional coordinate is the coordinate in the first coordinate system, and the first coordinate system is displayed in A boundary point of the screen of the device is the origin, the vertical direction of the screen is the first coordinate axis, the horizontal direction of the screen is the second coordinate axis, and the vertical direction of the screen is the coordinate system of the third coordinate axis.
  • the calibration unit is specifically configured to: perform circle center detection on the target image to obtain the pixel coordinates of each target point in the target image;
  • the pixel coordinates of the target points in the target image, the first three-dimensional coordinates of each target point in the multiple target points, the internal parameter matrix of the camera, and the distortion coefficient of the camera are calculated to obtain the current pose, where the first three-dimensional coordinate is Coordinates in the first coordinate system, where the first coordinate system takes a boundary point of the screen of the display device as the origin, the vertical direction of the screen as the first coordinate axis, the horizontal direction of the screen as the second coordinate axis, and the vertical direction of the screen as the second coordinate axis.
  • the direction is the coordinate system of the third coordinate axis.
  • the plurality of target points include M first target points in the screen of the display device and N second target points on the screen frame, wherein the M first target points and the Nth target points The two target points are not coplanar, and M and N are integers greater than or equal to 3.
  • the calibration unit is also used for: before performing the hardware on-ring calibration of the camera according to the P target point groups and the current pose of the camera, obtain M The second three-dimensional coordinates of the first target point and the second three-dimensional coordinates of the N second target points, wherein the second three-dimensional coordinates are the coordinates under the coordinate system of the total station; the second three-dimensional coordinates of the M first target points Coordinate transformation is performed to obtain the first three-dimensional coordinates of the M first target points, and coordinate transformation is performed on the second three-dimensional coordinates of the N second target points to obtain the first three-dimensional coordinates of the N second target points.
  • the calibration unit is specifically used for: making a difference between the second target pose and the current pose to obtain the current pose residual, where the second target pose is P first target positions
  • the average value of the pose, the current pose residual includes 3 current position DOF residuals and 3 current attitude DOF residuals; first, adjust the camera's position freedom in descending order of the current position DOF residuals degree of freedom, so that the difference between the first positional degree of freedom of the camera and the first positional degree of freedom of the second target pose is less than the preset first positional degree of freedom residual threshold, the second positional degree of freedom of the camera and the second target
  • the difference between the second positional degrees of freedom of the pose is less than the preset second positional degree of freedom residual threshold
  • the difference between the third positional degree of freedom of the camera and the third positional degree of freedom of the second target pose is less than the preset
  • the third position degree of freedom residual threshold then adjust the camera's various attitude degrees of freedom in order of the current attitude degree of freedom residual in descending order, so that the camera
  • a fourth aspect of the embodiments of the present application discloses a camera hardware-in-the-loop calibration target setting device, including: a setting unit for setting M first target points in a screen of a display device, and a screen frame of the display device. N second target points are set on the device, wherein the display device is used to display the calibration image when the hardware of the camera is calibrated in the ring, the M first target points and the N second target points are not coplanar, and M and N are greater than or an integer equal to 3.
  • the setting unit is further configured to: acquire the second three-dimensional coordinates of the M first target points and the second three-dimensional coordinates of the N second target points, wherein the second three-dimensional coordinates are the total station Coordinates under the instrument coordinate system; coordinate transformation is performed on the second three-dimensional coordinates of the M first target points and the second three-dimensional coordinates of the N second target points to obtain the first three-dimensional coordinates of the M first target points and N
  • the first three-dimensional coordinates of the second target point wherein the first three-dimensional coordinates are the coordinates in the first coordinate system, and the first coordinate system is a boundary point of the screen as the origin, the vertical direction of the screen as the first coordinate axis, Taking the horizontal direction of the screen as the second coordinate axis and the vertical direction of the screen as the coordinate system of the third coordinate axis; the first three-dimensional coordinates of the M first target points and the first three-dimensional coordinates of the N second target points are used for the camera hardware-in-the-loop calibration.
  • coordinate transformation is performed on the second three-dimensional coordinates of the M first target points and the second three-dimensional coordinates of the N second target points to obtain the first three-dimensional coordinates of the M first target points.
  • the setting unit is specifically used for: according to the second three-dimensional coordinates of the M first target points and the second three-dimensional coordinates of the M first target points, to obtain the M th An eigenvector and M second eigenvectors; an intermediate vector is obtained by least squares calculation according to the M first eigenvectors and the M second eigenvectors; a control matrix is obtained according to the intermediate vector, and a rotation vector and a translation are obtained according to the control matrix vector; according to the rotation vector and the translation vector, the second three-dimensional coordinates of the M first target points and the second three-dimensional coordinates of the N second target points are subjected to coordinate conversion, and the first three-dimensional coordinates of the M first target points and N
  • the aspect ratio of the calibration image and the aspect ratio of the screen are the same as the aspect ratio of the display device, and the setting unit is further configured to: perform circle center detection on the display device displaying the calibration image to obtain pixel coordinates of each of the M first target points; converting the pixel coordinates of each of the M first target points to the second coordinate system to obtain M first target points The two-dimensional coordinates of each first target point in the target point, wherein the second coordinate system is based on an outer boundary point of the screen frame of the display device as the origin, the vertical direction of the screen frame as the first coordinate axis, and the screen frame.
  • the lateral direction of the frame is the coordinate system of the second coordinate axis; N second targets are determined according to the two-dimensional coordinates of at least one first target point in the M first target points, the aspect ratio of the screen and the aspect ratio of the display device The two-dimensional coordinates of the point in the second coordinate system.
  • the setting unit is further configured to: according to the proportional relationship between the size of the calibration image and the size of the outer border of the screen frame, to calibrate the The image is enlarged to obtain an enlarged calibration image, wherein the size of the enlarged calibration image is equal to the size of the outer boundary of the screen frame; the enlarged calibration image is projected onto the second coordinate system, wherein the second coordinate
  • the system is a coordinate system with an outer boundary point of the screen frame of the display device as the origin, the vertical direction of the screen frame as the first coordinate axis, and the horizontal direction of the screen frame as the second coordinate axis;
  • the pixel coordinates of the coincident center positions of the target points are used as the two-dimensional coordinates of the target point, so as to obtain the two-dimensional coordinates of the M first target points and the two-dimensional coordinates of the N second target points.
  • a fifth aspect of the embodiments of the present application discloses a hardware-in-the-loop calibration system for a camera, the system includes an electronic device and a display device, the electronic device includes the camera, or the electronic device is communicatively connected to the camera, and the display device includes The screen and the screen frame, M first target points are set in the screen, or M first target points are displayed on the screen; N second target points are set on the screen frame; among them, the M first target points and N The second target points are not coplanar, and M and N are integers greater than or equal to 3.
  • M first target points are set in the screen of the display device
  • N second target points are set on the screen frame of the display device
  • the M first target points and the Nth are not coplanar, so there is no need to use multiple non-coplanar calibration boards during calibration, which solves the problem of limited space for target setting in the hardware-in-the-loop system of the camera.
  • a sixth aspect of the embodiments of the present application discloses an electronic device, including a processor, a memory, a communication interface, and one or more programs, where the one or more programs are stored in the memory and configured by the processor Executing, the above program includes instructions for performing the steps in the method of any one of the above first or second aspects.
  • a seventh aspect of the embodiments of the present application discloses a chip, including: a processor, configured to call and run a computer program from a memory, so that a device installed with the chip executes any one of the first aspect or the second aspect above method described in item.
  • An eighth aspect of the embodiments of the present application discloses a computer-readable storage medium, which stores a computer program for electronic data exchange, wherein the computer program causes a computer to execute any one of the first aspect or the second aspect above the method described.
  • a ninth aspect of the embodiments of the present application discloses a computer program, where the computer program causes a computer to execute the method according to any one of the first aspect or the second aspect.
  • FIG. 1 is a schematic diagram of a focus area of a camera provided by an embodiment of the present application.
  • FIG. 2 is a schematic structural diagram of a hardware-in-the-loop calibration system for a camera provided by an embodiment of the present application.
  • FIG. 3 is a schematic diagram of a screen target provided by an embodiment of the present application.
  • FIG. 4 is a schematic diagram of a screen frame target provided by an embodiment of the present application.
  • FIG. 5 is a schematic flowchart of a method for setting a hardware-in-the-loop calibration target point of a camera according to an embodiment of the present application.
  • FIG. 6 is a schematic diagram of a first coordinate system provided by an embodiment of the present application.
  • FIG. 7 is a schematic diagram of a first coordinate system and a second coordinate system provided by an embodiment of the present application.
  • FIG. 8 is a schematic flowchart of a hardware-in-the-loop calibration method for a camera provided by an embodiment of the present application.
  • FIG. 9 is a schematic diagram of a corner detection provided by an embodiment of the present application.
  • FIG. 10 is a schematic flowchart of an optimal pose solution based on a focus area provided by an embodiment of the present application.
  • FIG. 11 is a schematic flowchart of another hardware-in-the-loop calibration method for a camera provided by an embodiment of the present application.
  • FIG. 12 is a schematic flowchart of another hardware-in-the-loop calibration method for a camera provided by an embodiment of the present application.
  • FIG. 13 is a schematic diagram of a generation principle of an error map provided by an embodiment of the present application.
  • FIG. 14 is a schematic structural diagram of a camera hardware-in-the-loop calibration target setting device provided by an embodiment of the present application.
  • FIG. 15 is a schematic structural diagram of a hardware-in-the-loop calibration device for a camera provided by an embodiment of the present application.
  • FIG. 16 is a schematic structural diagram of an electronic device provided by an embodiment of the present application.
  • the current camera-in-the-loop calibration method achieves camera alignment by aligning the center of the camera with the centerline of the LED display; or by adjusting the position of the camera, the camera can completely capture the entire screen, and the camera alignment allow.
  • the camera can be fixed and adjusted through a metal bracket so that the camera just captures the entire screen, and the metal bracket can be moved to match cameras with different field of view (FOV) to ensure that the image captured by the camera is exactly the complete screen image.
  • FOV field of view
  • the center of the camera is vertically aligned with the center of the screen, and the hardware-in-the-loop of cameras with different FOVs is realized by adjusting the distance from the center of the camera to the center of the screen.
  • the basic idea of the above methods is to adjust the camera so that the camera is aimed at the center of the screen, and then adjust the camera according to whether the entire screen is completely captured to achieve the effect of alignment. Since the center of the camera is related to the internal parameters of the camera, the evaluation criteria for aligning the center of the screen are difficult to grasp, and the vertical alignment to the screen can only be roughly estimated, with low accuracy. In addition, in the process of adjusting the camera, due to the image distortion of the camera, the correction model to eliminate the distortion cannot be completely consistent with the actual model, so it is almost impossible to capture the entire screen with complete accuracy. There is no quantitative evaluation standard for how to adjust to the optimal position, which will greatly affect the adjustment efficiency and accuracy during actual adjustment. In addition, different cameras have different functions, and pixel matching for key areas should be more accurate. The above alignment scheme does not take into account the function matching with specific cameras, and the matching accuracy is usually relatively low.
  • the camera calibration feedback method can feed back the current position and attitude of the camera in real time, which is used in the camera hardware-in-the-loop alignment in the embodiment of the present application, and the main purpose is to obtain the current position and attitude of the camera and the target position that needs to be adjusted. Therefore, the camera pose can be quantitatively adjusted to improve the adjustment efficiency and accuracy; however, the camera pose solving algorithm needs to set non-coplanar target points.
  • the existing technology uses two non-coplanar calibration plates to achieve the target point. Not coplanar, which is not allowed in the alignment of the camera on the ring, how to set the target point for the alignment of the camera on the ring becomes a technical problem to be solved by the present application.
  • the function of the camera determines the possible area of interest of the camera. As shown in Figure 1, the camera that recognizes traffic lights focuses on the upper-middle area of the image, and the camera that recognizes lane lines focuses on the middle-lower area of the image. How to adjust the pose of the camera so that the alignment accuracy of the camera in the region of interest is higher has become another key problem to be solved in this application.
  • the technical solutions provided in the embodiments of the present application can be applied to, but not limited to, the following scenarios: when the HIL system leaves the factory, the camera needs to be aligned and adjusted; when the HIL system changes the camera pose during transportation and use, the camera needs to be adjusted. In order to achieve the consistency of the image captured by the camera and the actual image, and ensure the normal use of the HIL function; in addition, when the function of the camera changes and the user's focus area changes, the camera needs to be adjusted. Pose and pose adjustment in order to meet the accuracy requirements of the corresponding perception algorithm.
  • FIG. 2 is a schematic structural diagram of a hardware-in-the-loop calibration system for a camera provided by an embodiment of the present application.
  • the system includes an electronic device 10 and a display device 20, the electronic device 10 includes a camera 101, or the electronic device 10 is connected to the camera 101 in communication, the display device 20 includes a screen 201 and a screen frame 202, the screen 201 M first target points are arranged inside, or M first target points are displayed on the screen 201; N second target points are arranged on the screen frame 202; wherein, M first target points and N second target points Not coplanar, M and N are integers greater than or equal to 3.
  • the screen 201 is used to display the calibration image when the hardware of the camera 101 is calibrated in the ring.
  • the first target point set in the screen 201 or the first target point displayed on the screen 201 is shown in FIG. 3 , the meaning of the target point is to use at the calibrated key space points.
  • the screen 201 and the screen frame 202 may not be coplanar, for example, the screen frame 202 protrudes outward relative to the screen 201; or the screen 201 Coplanar with the screen frame 202 , the second target set on the screen frame 202 is a boss target, and the boss target is shown in FIG. 4 .
  • the electronic device 10 can be a smart camera, a camera, a mobile phone, an electronic device with a camera function, or an electronic device with a camera, etc.;
  • the electronic device 10 uses the camera 101 to capture images of the display device 20; this application does not specifically limit this.
  • the electronic device 10 can be connected in communication with the display device 20, and the electronic device 10 can control the display device 20 to display the calibration image, for example, the electronic device 10 sends the calibration image to the display device 20, instructing the display device 20 to display the calibration image.
  • the hardware-in-the-loop calibration process of the camera is performed by the hardware-in-the-loop calibration system of the camera as follows: first, the two-dimensional coordinates and the three-dimensional coordinates of each target point in all the target points are obtained; then, on the screen 201 of the display device 20 The calibration image is displayed on the display device 10, and the electronic device 10 captures the image of the display device 20 through the camera 101, and needs to completely shoot the display device 20 into the image collected by the camera 101, that is, the image collected by the camera 101 needs to completely include the The display device 20, so the image collected by the camera 101 includes all the target points; then obtain the pixel coordinates of each target point in all the target points in the image collected by the camera 101; finally, according to the pixel coordinates of each target point The two-dimensional coordinates and the three-dimensional coordinates are calculated to obtain the pose that the camera 101 needs to be adjusted to, and the current pose of the camera 101 is calculated according to the pixel coordinates of each target point in the image collected by the camera 101 and the three
  • M first target points are set in the screen of the display device 20, and N second target points are set on the screen frame of the display device 20, and the M first target points and the N The second target points are not coplanar, so there is no need to use multiple non-coplanar calibration boards during calibration, which solves the problem of limited space for setting target points in the hardware-in-the-loop system of the camera 101 .
  • FIG. 5 is a schematic flowchart of a method for setting a hardware-in-the-loop calibration target point of a camera provided by an embodiment of the present application.
  • the method can be applied to the hardware-in-the-loop calibration system for a camera shown in FIG. 2.
  • the method Executed by an electronic device, the electronic device includes the camera, or the electronic device is communicatively connected to the camera, and the method includes but is not limited to the following steps:
  • step 501 M first target points are set on the screen of the display device, and N second target points are set on the screen frame of the display device, wherein the display device is used to display the calibration when the hardware of the camera is calibrated in the ring.
  • the M first target points and the N second target points are not coplanar, and M and N are integers greater than or equal to 3.
  • the camera may be applied to the camera shown in FIG. 2 , and the display device may be the display device shown in FIG. 2 ; the M first target points in the screen may be virtual target points displayed on the screen.
  • M first target points are set in the screen of the display device for the hardware-in-ring calibration of the camera, and on the screen frame of the display device.
  • N second target points are set, and the M first target points and N second target points are not coplanar, as shown in FIG. 3 and FIG. 4 .
  • M first target points are set in the screen of the display device
  • N second target points are set on the screen frame of the display device
  • the M first target points and the Nth are not coplanar, so there is no need to use multiple non-coplanar calibration boards during calibration, which solves the problem of limited space for target setting in the hardware-in-the-loop system of the camera.
  • the three-dimensional coordinates of the target points need to be used for pose calculation. Therefore, after the target points are set, the three-dimensional coordinates of each target point need to be obtained, and each target point needs to be acquired.
  • the three-dimensional coordinates of each target point are saved for the hardware-in-the-loop calibration of the camera. Among them, the three-dimensional coordinates of each target point can be measured by a total station. However, due to the randomness of the placement of the total station, the actual direction of the second three-dimensional coordinate of the target point in the total station coordinate system is random, and the adjustment degree of freedom given by the camera calibration algorithm is the total station coordinate system.
  • the second three-dimensional coordinates of all target points in the total station coordinate system can be converted to the first coordinate system (that is, the screen coordinate system), and the first three-dimensional coordinates of all target points in the first coordinate system can be obtained, And save the first three-dimensional coordinates of all target points for the hardware-in-the-loop calibration of the camera.
  • the embodiment of the present application defines a first coordinate system, where a boundary point of the screen is the origin, the vertical direction of the screen is the first coordinate axis, the horizontal direction of the screen is the second coordinate axis, A coordinate system with the vertical direction of the screen as the third coordinate axis.
  • FIG. 6 is a schematic diagram of a first coordinate system provided by an embodiment of the present application.
  • the first coordinate system takes point o in the upper left corner of the screen as the origin, takes the horizontal direction of the screen as the y-axis, takes the vertical direction of the screen as the x-axis, and takes the vertical outward direction of the screen as the z-axis.
  • the method further includes: acquiring the second three-dimensional coordinates of the M first target points and the second three-dimensional coordinates of the N second target points, wherein the second three-dimensional coordinates are total station coordinates Coordinates under the system; coordinate transformation is performed on the second three-dimensional coordinates of the M first target points and the second three-dimensional coordinates of the N second target points, and the first three-dimensional coordinates of the M first target points and the N second three-dimensional coordinates are obtained.
  • the first three-dimensional coordinates of the target point wherein the first three-dimensional coordinates are the coordinates under the first coordinate system, and the first coordinate system is a boundary point of the screen as the origin, the vertical direction of the screen as the first coordinate axis, and the screen as the first coordinate axis.
  • the horizontal direction is the second coordinate axis
  • the vertical direction of the screen is the coordinate system of the third coordinate axis; the first three-dimensional coordinates of the M first target points and the first three-dimensional coordinates of the N second target points are used for the hardware of the camera Calibration in the ring.
  • the first three-dimensional coordinates of all the target points can be obtained through the following steps: first, measure the target points with a total station to obtain the second three-dimensional coordinates of the target points in the total station coordinate system, and then place the target points in the total station coordinate system.
  • the second three-dimensional coordinates in the total station coordinate system are converted into the first coordinate system to obtain the first three-dimensional coordinates of the target point.
  • the second three-dimensional coordinates of the four target points on the screen frame that is, the four boss target points set at the edge of the screen in Figure 4
  • the second three-dimensional coordinates of other target points can be determined by the pixel relationship
  • the difference value is obtained; after the second three-dimensional coordinates of all the target points are obtained, they are converted into the first coordinate system to obtain the first three-dimensional coordinates of all the target points.
  • the coordinates of the target point in the total station coordinate system are converted into the first coordinate system, and the first three-dimensional coordinates of the target point in the first coordinate system can be used for the hardware of the camera.
  • the ring calibration is used to provide a basis for the subsequent hardware-in-the-loop calibration of the camera, which is beneficial to improve the camera alignment effect.
  • coordinate transformation is performed on the second three-dimensional coordinates of the M first target points and the second three-dimensional coordinates of the N second target points to obtain the first three-dimensional coordinates of the M first target points and
  • the first three-dimensional coordinates of the N second target points include: obtaining M first feature vectors and M first eigenvectors according to the second three-dimensional coordinates of the M first target points and the second three-dimensional coordinates of the M first target points Two eigenvectors; adopt the least squares method to calculate the intermediate vector according to the M first eigenvectors and the M second eigenvectors; obtain the control matrix according to the intermediate vector, and obtain the rotation vector and the translation vector according to the control matrix; according to the rotation vector and translation
  • the vector performs coordinate transformation on the second three-dimensional coordinates of the M first target points and the second three-dimensional coordinates of the N second target points to obtain the first three-dimensional coordinates of the M first target points and the first three-dimensional coordinates of the N second target points.
  • the second three-dimensional coordinates in the system are converted into first three-dimensional coordinates in the first coordinate system. Specifically include the following steps:
  • the least squares method is used to obtain the intermediate vector p, wherein the intermediate vector p is calculated according to formulas (2) and (3).
  • the equation system shown in the formula (2) is an overdetermined equation, so p can be solved by the least square method through the formula (3).
  • the rotation vector R and the translation vector T can be obtained from the total station coordinate system converted to the first coordinate system, wherein the calculation formulas of the rotation vector R and the translation vector T are shown in formula (5).
  • Y represents the second three-dimensional coordinate of the target point in the total station coordinate system
  • X represents the first three-dimensional coordinate of the target point in the first coordinate system
  • the first target point in the screen is used as the reference point to solve the rotation matrix and translation vector from the coordinate system of the total station to the first coordinate system;
  • the coordinates in the station coordinate system are converted into the first coordinate system, and the first three-dimensional coordinates of the target point in the first coordinate system are used for the hardware-in-the-loop calibration of the camera, so as to provide the subsequent hardware-in-the-loop calibration of the camera. It is helpful to improve the camera alignment effect.
  • the two-dimensional coordinates of the target points are also used for pose calculation. Therefore, after setting the target points, it is necessary to obtain the two-dimensional coordinates of each target point. coordinate.
  • the two-dimensional coordinates of the target point in the screen can be obtained through the following steps: first, the pixel coordinates of the target point are acquired by performing a circle center detection on the screen displaying the calibration image, and then the pixel coordinates of the target point are converted to the second coordinate In the system (that is, the screen frame coordinate system), the two-dimensional coordinates of the target point in the screen are obtained; wherein, the circle center detection is an algorithm for detecting the circle center in image detection.
  • the two-dimensional coordinates of the target point on the screen frame can be calculated according to the two-dimensional coordinates of the target point in the screen.
  • the embodiment of the present application defines a second coordinate system, and the second coordinate system takes an outer boundary point of the screen frame of the display device as the origin, the vertical direction of the screen frame as the first coordinate axis, and the horizontal direction of the screen frame as the first coordinate axis. is the coordinate system of the second coordinate axis.
  • FIG. 7 is a schematic diagram of a first coordinate system and a second coordinate system provided by an embodiment of the present application.
  • the second coordinate system takes the upper left corner o' point of the screen border as the origin, takes the horizontal direction of the screen border as the y' axis, and takes the vertical direction of the screen border as the x' axis.
  • the aspect ratio of the calibration image and the aspect ratio of the screen are the same as the aspect ratio of the display device, and the method further includes: performing circle center detection on the display device displaying the calibration image to obtain M The pixel coordinates of each first target point in the first target points; the pixel coordinates of each first target point in the M first target points are converted into the second coordinate system to obtain M first target points The two-dimensional coordinates of each first target point in the, wherein, the second coordinate system is based on an outer boundary point of the screen frame of the display device as the origin, the vertical direction of the screen frame as the first coordinate axis, and the screen frame as the first coordinate axis.
  • the horizontal direction is the coordinate system of the second coordinate axis; according to the two-dimensional coordinates of at least one first target point in the M first target points, the aspect ratio of the screen and the aspect ratio of the display device, it is determined that the N second target points are at Two-dimensional coordinates in the second coordinate system.
  • the aspect ratio of the calibration image and the aspect ratio of the screen are the same, that is, the pixel size of the calibration image is consistent with the pixel size of the screen.
  • the pixels of the calibration image can cover the entire screen. of pixels.
  • the aspect ratio of the calibration image and the aspect ratio of the screen are the same as the aspect ratio of the display device, that is, there is a proportional relationship between the size of the calibration image, the size of the screen and the size of the display device.
  • the pixels of the calibration image are 1920 ⁇ 1080, and the resolution of the screen is 1080P (that is, 1920 ⁇ 1080).
  • the calibration image is displayed on the screen, all the pixels on the screen are covered.
  • the screen of the image is detected by the center of the circle, so that the pixel coordinates of all the first target points can be obtained; and the screen aspect ratio of the display device is the same as the aspect ratio of the display device (or the outer border of the screen frame), such as the screen of the display device.
  • the ratio of the size to the size of the display device (or the outer border of the screen frame) is 1:1.2, and the position of the second target point on the screen frame is fixed, and the pixel coordinates of the first target point can be converted through a mathematical model into two-dimensional coordinates in the second coordinate system, and calculate the two-dimensional coordinates of the second target point according to the two-dimensional coordinates of the first target point.
  • the aspect ratio of the display device calculates the two-dimensional coordinates of the second target point in the upper left corner of the screen frame; since the second target point in the lower left corner, upper right corner and lower right corner of the screen frame and the second target point in the upper left corner of the screen frame are on the screen
  • the relative position on the frame is determined, so that the two-dimensional coordinates of the second target at the lower left corner, upper right corner and lower right corner of the screen frame can be calculated according to the two-dimensional coordinates of the second target at the upper left corner of the screen frame.
  • the pixel coordinates of the first target point are obtained by detecting the center of the circle; then the two-dimensional coordinates of the first target point are obtained by converting the pixel coordinates of the first target point to the second coordinate system; The two-dimensional coordinates of the second target point are calculated from the two-dimensional coordinates of the first target point, so that the two-dimensional coordinates of all the target points used for the hardware-in-ring calibration of the camera can be obtained.
  • the method further includes: according to the proportional relationship between the size of the calibration image and the size of the outer border of the screen frame, performing a calibration on the calibration image.
  • the aspect ratio of the calibration image is the same as the aspect ratio of the outer border of the screen frame.
  • FIG. 8 is a schematic flowchart of a hardware-in-the-loop calibration method for a camera provided by an embodiment of the present application.
  • the method can be applied to the hardware-in-the-loop calibration system for a camera shown in FIG. 2.
  • the method is performed by an electronic device.
  • the electronic device includes the camera, or the electronic device is connected to the camera in communication, and the method includes but is not limited to the following steps:
  • Step 801 Determine the weight corresponding to each target point in the multiple target points, wherein the multiple target points are the target points on the display device, and the corresponding weights of the target points located in the target area on the display device are greater than those located outside the target area.
  • the target area of the display device is used to display a part of the calibration image; the calibration image is the image displayed by the display device when the hardware of the camera is performing ring calibration.
  • the mathematical meaning of the weight corresponding to the target point indicates the importance of the focus area of the camera relative to other areas; a part of the calibration image displayed by the target area of the display device, the part of the calibration image is the focus area of the camera; the camera
  • the focus area of the camera can be automatically identified and determined by the electronic device, and then the electronic device assigns a higher weight to the target point corresponding to the focus area of the camera; the focus area of the camera can also be manually selected by the user.
  • User frame selection interface when the user selects a key area, the electronic device automatically assigns a higher weight to the target point corresponding to the frame selection area.
  • weights can be defined according to user needs.
  • Step 802 Perform P sampling of multiple targets with replacement according to the weight to obtain P target groups, wherein, for each sampling, the greater the weight corresponding to the target, the higher the probability of the target being extracted. large, P is a positive integer.
  • sampling with replacement is performed on the target points with different weights to obtain P target point groups.
  • a bootstrap method can be used.
  • Step 803 Perform hardware-in-the-loop calibration of the camera according to the P target point groups and the current pose of the camera.
  • the two-dimensional coordinates and the three-dimensional coordinates of each target point in any one of the P target point groups can be determined, so as to obtain the two-dimensional coordinates and the three-dimensional coordinates of all the target points, and according to the Two-dimensional coordinates and three-dimensional coordinates are calculated to obtain the pose that the camera needs to be adjusted to, and then take the pose that needs to be adjusted as the direction, adjust from the current pose of the camera to the pose that needs to be adjusted, so as to realize the hardware of the camera Calibration in the ring.
  • the target points are resampled with replacement based on the weights, which not only ensures the independence of each group of target point data, but also focuses on improving the camera's performance.
  • the probability of the target points corresponding to the focus area being extracted, and the hardware-in-the-loop calibration of the camera based on the sampled target points and the current pose of the camera can effectively improve the alignment accuracy of the camera to the focus area.
  • the hardware for the camera has a plurality of target points on a display device for ring calibration, and a calibration image is displayed on the display device, the calibration image has the focus area of the camera, and the calibration image is displayed on the display device.
  • the display area used to display the focus area is the target area, and the weight corresponding to the target point within the target area is set to be greater than the weight corresponding to the target point outside the target area; then based on the weight probability Perform P sampling of multiple targets with replacement to obtain P target groups, that is, the greater the weight corresponding to the target, the greater the probability of the target being extracted; due to the weight of the target in the target area is greater than the weight corresponding to the target points outside the target area, so each time you sample, the probability of the target points in the target area being extracted is greater, and the number of target points in the target area is relatively larger.
  • the embodiments of the present application can effectively improve the alignment accuracy of the camera to the focus area, and ensure the personalized alignment requirement of the user.
  • the hardware-in-the-loop calibration of the camera is performed according to the P target point groups and the current pose of the camera, including: determining P first target poses according to the P target point groups, wherein the P first target poses are The target group is in one-to-one correspondence with the P first target poses; the display device currently displaying the calibration image is captured by the camera to obtain the target image, and the current pose of the camera is determined according to the multiple target points and the target image; The hardware-in-the-loop calibration of the camera is performed on the average of the P first target poses and the current pose.
  • the purpose of the hardware-in-the-loop calibration of the camera is that the image captured by the camera is completely consistent with the picture of the actual display device displaying the calibrated image. Therefore, the aspect ratio of the target image and the aspect ratio of the display device can be the same. In this way, the image acquisition of the display device through the camera can realize that the camera just shoots the entire display device into the target image, that is, in the target image, the display
  • the boundary of the device is also the boundary of the target image; further, the aspect ratio of the target image, the aspect ratio of the calibration image, the aspect ratio of the screen and the aspect ratio of the display device can all be the same.
  • the average value of the P first target poses can be used as the pose to which the camera needs to be adjusted, that is, the average value of the P first target poses can be used as the pose the camera should be in after the calibration is successful.
  • P first target poses are determined according to the P target groups obtained by re-sampling; then, the camera is used to perform image acquisition on the display device currently displaying the calibration image to obtain the target image, And determine the current pose of the camera according to the multiple target points on the display device and the target image; then take the average value of the P first target poses as the pose to be adjusted, and perform hardware-in-the-loop calibration of the camera, That is to say, the current pose of the camera is adjusted to the average value of the P first target poses; since the target pose of the camera is calculated for many times by the method of resampling, the pose that needs to be adjusted is obtained by the method of averaging.
  • the pose that needs to be adjusted not only provides a reference benchmark for the camera's pose adjustment direction, but also focuses on ensuring the alignment accuracy of the camera's key focus area, which is conducive to improving the calibration efficiency and alignment accuracy of the camera alignment.
  • the method further includes: capturing images of the calibration board through a camera to obtain Q calibration images, where Q is A positive integer; perform corner detection on each calibration image in the Q calibration images to obtain the pixel coordinates of each corner in each calibration image; The pixel coordinates and the third three-dimensional coordinates of each corner point in each calibration image obtain the camera's internal parameter matrix and the camera's distortion coefficient, where the third three-dimensional coordinates are the coordinates in the calibration board coordinate system.
  • the pose calculation needs to use the camera's internal parameter matrix and the camera's distortion coefficient, so before determining the first target pose, the camera's internal parameter matrix and the camera's distortion coefficient need to be obtained;
  • the distortion coefficient can be determined according to the image collected by the camera by collecting the image by the camera.
  • corner detection is an image detection algorithm, which mainly refers to detecting the vertices on the black and white corners of the checkerboard in the calibration board.
  • the corner detection process is shown in Figure 9.
  • the distortion coefficients of the camera including K1, K2, D1, D2 and K3, where K1, K2 and K3 are radial distortion coefficients, and D1 and D2 are tangential distortion coefficients
  • the internal parameter matrix can be obtained by using Zhang's calibration method.
  • Zhang's calibration method including Fx, Fy, Cx and Cy, where Fx and Fy are focal lengths, and Cx and Cy are optical centers.
  • the input of Zhang's calibration method is many images, such as 50 images, that is, Q is equal to 50; then through the iteration and calculation of its own algorithm, the optimal distortion coefficient and internal parameter matrix that are finally applicable to all images are obtained as the final result.
  • the camera is used to collect images of the calibration board to obtain multiple calibration images, and the internal parameter matrix of the camera and the distortion coefficient of the camera are obtained with the multiple calibration images, and the pose calculation in the calibration process needs to use
  • the internal parameter matrix of the camera and the distortion coefficient of the camera can provide a basis for calculating the pose that the camera needs to adjust to and the current pose of the camera, which is beneficial to improve the hardware-in-the-loop alignment accuracy of the camera.
  • determining the P first target poses according to the P target point groups includes: for each target point group in the P target point groups, performing the following steps to obtain the P first targets Pose: Obtain the two-dimensional coordinates of each target point in the target target group, where the target target group is any one of the P target point groups, the two-dimensional coordinates are the coordinates in the second coordinate system, the second The coordinate system is a coordinate system with an outer boundary point of the screen frame of the display device as the origin, the vertical direction of the screen frame as the first coordinate axis, and the horizontal direction of the screen frame as the second coordinate axis; The two-dimensional coordinates, the first three-dimensional coordinates of each target point, the internal parameter matrix of the camera and the distortion coefficient of the camera are calculated to obtain the first target pose corresponding to the target target group, wherein the first three-dimensional coordinates are in the first coordinate system.
  • the first coordinate system takes a boundary point of the screen of the display device as the origin, the vertical direction of the screen as the first coordinate axis, the horizontal direction of the screen as the second coordinate axis, and the vertical direction of the screen as the third coordinate
  • the coordinate system of the axes takes a boundary point of the screen of the display device as the origin, the vertical direction of the screen as the first coordinate axis, the horizontal direction of the screen as the second coordinate axis, and the vertical direction of the screen as the third coordinate The coordinate system of the axes.
  • FIG. 10 is a schematic flowchart of an optimal pose solution based on a focus area provided by an embodiment of the present application.
  • the algorithm automatically assigns weights corresponding to the target points, among which, the target points in the focus area are given higher weights; all targets After the weight of the point is determined, the idea of resampling is used to perform bootstrap sampling with replacement on the target points of different weights based on the weight probability to obtain n groups of sampling results; the sampling results of each group are used as the input, that is, the sampling results of each group are taken as the input.
  • the two-dimensional coordinates and the first three-dimensional coordinates of the target point in , as well as the camera's internal parameter matrix and the camera's distortion coefficient are used as input, and the UPNP algorithm is used to solve the pose and obtain a series of first target poses, that is, the nth A target pose; by averaging the n first target poses, the second target pose can be obtained, that is, the pose to which the camera needs to be adjusted.
  • the weighted probability-based re-sampling with replacement not only ensures the independence between the target data of each group, but also focuses on improving the probability of the occurrence of the target in the key focus area, which can effectively improve the camera's ability to focus on the key focus area. Alignment accuracy.
  • the UPNP (unified Perspective-n-Point) algorithm is an improvement of the Perspective-n-Point (PNP) algorithm, which is an existing camera calibration algorithm; when solving the pose, in addition to using the UPNP algorithm , the PNP algorithm and all its improved algorithms can also be used, such as the EPNP algorithm, among which, the EPNP (efficient Perspective-n-Point) algorithm is also an improvement of the PNP algorithm.
  • the first target position corresponding to the target target group is calculated and obtained according to the two-dimensional coordinates of each target point in the target target point group, the first three-dimensional coordinate, the internal parameter matrix of the camera, and the distortion coefficient of the camera.
  • the two-dimensional coordinates are the coordinates in the second coordinate system
  • the first three-dimensional coordinates are the coordinates in the first coordinate system
  • the two-dimensional coordinates of each target point in the second coordinate system can be calculated according to the pixel coordinates of the calibration image
  • the three-dimensional coordinates of each target point in the total station coordinate system can be obtained through the total station,
  • the three-dimensional coordinates of each target point in the total station coordinate system are converted into three-dimensional coordinates in the first coordinate system; thus, it is beneficial to obtain the first target pose of each target point group, and then the camera needs to be adjusted to. 's pose.
  • determining the current pose of the camera according to the multiple target points and the target image includes: performing circle center detection on the target image to obtain the pixel of each target point in the target image among the multiple target points Coordinates; according to the pixel coordinates of each target point in the target image, the first three-dimensional coordinate of each target point in the plurality of target points, the internal parameter matrix of the camera, and the distortion coefficient of the camera, the current pose, wherein the first three-dimensional coordinates are coordinates in a first coordinate system, and the first coordinate system is a boundary point of the screen of the display device as the origin, the vertical direction of the screen as the first coordinate axis, and the horizontal direction of the screen as the first coordinate axis is a coordinate system with the second coordinate axis and the vertical direction of the screen as the third coordinate axis.
  • the process of calculating the current pose of the camera and the process of calculating the first target pose are the same, and the difference between the two is that the input is different, and the input for calculating the first target pose is the two-dimensional coordinates and the first three-dimensional coordinates of each target point.
  • the camera's internal parameter matrix and the camera's distortion coefficient the input to calculate the current pose is the pixel coordinates of the target point in the target image, the first three-dimensional coordinate of the target point, the camera's internal parameter matrix, and the camera's distortion coefficient.
  • the camera is used to collect images of the display device currently displaying the calibration image, and after obtaining the target image, each target point among the multiple target points on the display device can be obtained by performing circle center detection on the target image.
  • the pixel coordinates in the target image are helpful to calculate the current pose according to the pixel coordinates of each target point in the target image, the first three-dimensional coordinates of each target point, the camera's internal parameter matrix, and the camera's distortion coefficient.
  • the plurality of target points include M first target points in the screen of the display device and N second target points on the screen frame, wherein the M first target points and the Nth target points The two target points are not coplanar, and M and N are integers greater than or equal to 3.
  • the method further includes: acquiring M first The second three-dimensional coordinates of the target point and the second three-dimensional coordinates of the N second target points, wherein the second three-dimensional coordinates are the coordinates under the coordinate system of the total station; the second three-dimensional coordinates of the M first target points are coordinated.
  • the transformation is performed to obtain the first three-dimensional coordinates of the M first target points
  • the coordinate transformation is performed on the second three-dimensional coordinates of the N second target points to obtain the first three-dimensional coordinates of the N second target points.
  • M first target points are set in the screen of the display device
  • N second target points are set on the screen frame of the display device
  • the M first target points and the Nth are not coplanar, so there is no need to use multiple non-coplanar calibration boards when calibrating, which solves the problem of limited space for setting the target points in the hardware-in-the-loop system of the camera.
  • Converting the coordinates in the total station coordinate system to the first coordinate system lays the foundation for subsequent calibration feedback prompts, which is a key step to improve the camera alignment effect.
  • the hardware-in-the-loop calibration of the camera is performed according to the average value of the P first target poses and the current pose, including: making a difference between the second target pose and the current pose to obtain the current pose pose residuals, where the second target pose is the average of P first target poses, and the current pose residuals include 3 current position DOF residuals and 3 current posture DOF residuals;
  • the positional degrees of freedom residuals are adjusted in order from large to small in order to adjust each positional degree of freedom of the camera, so that the difference between the first positional degree of freedom of the camera and the first positional degree of freedom of the second target pose is smaller than the preset first degree of freedom.
  • the positional degree of freedom residual threshold the difference between the second positional degree of freedom of the camera and the second positional degree of freedom of the second target pose is less than the preset second positional degree of freedom residual threshold, the third positional degree of freedom of the camera and The difference between the third position degrees of freedom of the second target pose is less than the preset third position freedom degree of freedom threshold; and then adjust the camera's various attitude degrees of freedom in descending order of the current attitude degree of freedom residuals, So that the difference between the first attitude degree of freedom of the camera and the first attitude degree of freedom of the second target attitude is less than the preset first attitude degree of freedom residual threshold, the second attitude degree of freedom of the camera and the second target attitude The difference of the second attitude degree of freedom is less than the preset second attitude degree of freedom residual threshold, and the difference between the third attitude degree of freedom of the camera and the third attitude degree of freedom of the second target attitude is less than the preset third degree of freedom. Pose DOF residual threshold.
  • the pose includes 3 position degrees of freedom and 3 attitude degrees of freedom
  • the pose residual includes 3 position degrees of freedom residuals and 3 attitude degrees of freedom residuals.
  • some degrees of freedom have a greater impact on the alignment accuracy, and the corresponding degree of freedom residual threshold should be as small as possible; some degrees of freedom have little impact on the alignment accuracy, and the corresponding degrees of freedom
  • the degree of freedom residual threshold can be relatively large. Therefore, the degree of freedom residual threshold corresponding to each degree of freedom may be different, so there are 3 position degrees of freedom corresponding to 3 position degrees of freedom residual thresholds, and 3 attitude degrees of freedom corresponding to 3 attitude degrees of freedom residuals difference threshold.
  • the calibration process is to adjust each degree of freedom of the camera in turn, so that the degree of freedom residual corresponding to each degree of freedom is smaller than the degree of freedom residual threshold corresponding to the degree of freedom.
  • the component with the larger residual error in the position is fed back first, and the component with the larger residual error is adjusted in a corresponding direction.
  • FIG. 11 is a schematic flowchart of another hardware-in-the-loop calibration method for a camera provided by an embodiment of the present application. This method can be applied to the hardware-in-the-loop calibration system for a camera shown in FIG. 2. The method is performed by electronic Device execution, the electronic device includes the camera, or the electronic device is communicatively connected to the camera, the method includes but is not limited to the following steps:
  • Step 1101 calibration target setting.
  • the set target points are non-coplanar, which can provide the required non-coplanar target point coordinate information for the hardware-in-the-loop calibration algorithm of the camera.
  • Step 1102 Calculate the three-dimensional coordinates of the target point.
  • the three-dimensional coordinates (that is, the first three-dimensional coordinates) of all target points in the first coordinate system can be obtained through the total station marking and three-dimensional coordinate conversion, that is, the total station marking first obtains the target points at the total station.
  • the three-dimensional coordinates under the instrument coordinate system and then convert the three-dimensional coordinates of the target points in the total station coordinate system to the first coordinate system to obtain the three-dimensional coordinates of all target points in the first coordinate system; all target points are in the first coordinate system.
  • the three-dimensional coordinates in the coordinate system can be used as the input of the camera's hardware-in-the-loop calibration algorithm (including the calculation of the target pose and the calculation of the current pose).
  • Step 1103 Calculation of camera internal parameters.
  • Step 1104 the two-dimensional coordinates of the target point are generated.
  • the two-dimensional coordinates of the target can be obtained by detecting the center of the circle, and the two-dimensional coordinates of the target It can be used as the 2D coordinate input for the target pose calculation of the camera.
  • Step 1105 focus on region selection.
  • the display device displays the calibration image
  • the electronic device can automatically frame the focus area of the camera in the calibration image, or provide a user frame selection interface, and the user manually frame the focus area of the camera in the calibration image.
  • Step 1106 Calculate the target pose of the camera.
  • the target pose calculation of the camera is performed.
  • the target points in the key focus area assign a higher weight to the target points in the key focus area, complete the bootstrap sampling grouping of the target points based on the weight, and divide the three-dimensional coordinates, two-dimensional coordinates of each group of target points in the first coordinate system, and
  • the internal parameter information of the camera obtained by the calibration method (including the internal parameter matrix of the camera and the distortion coefficient of the camera) is brought into the UPNP algorithm, and the target pose corresponding to each group of target points (that is, the first target pose) is calculated.
  • the target poses corresponding to a set of target points are averaged to obtain the final target pose of the camera (ie, the second target pose).
  • the obtained target pose of the camera not only provides a reference benchmark for the orientation adjustment direction of the camera, but also focuses on ensuring the alignment accuracy of key areas of interest, which is crucial to improving the calibration efficiency and accuracy of camera alignment; that is, It can improve the alignment accuracy of key areas of interest, meet user needs, and significantly improve alignment efficiency and alignment accuracy.
  • Step 1107 Calculate the current pose of the camera.
  • the position of the camera is adjusted.
  • the detection of the center of the image captured by the camera is completed, and the pixel coordinates of the center of the target point in the captured image are obtained, and the pixel coordinates of the center of the target in the captured image are obtained.
  • the three-dimensional coordinates of the target point in the first coordinate system and the internal reference information of the camera are brought into the UPNP algorithm, and the current pose of the camera can be obtained.
  • Step 1108 whether the pose residual is greater than a threshold.
  • the direction and size of the degrees of freedom that should be adjusted can be clarified according to the pose residual.
  • step 1109 is performed; if the pose residual is not greater than the threshold, step 1111 is performed.
  • Step 1109 adjust the camera pose.
  • Step 1110 Detect pixel coordinates of the target in the image captured by the camera.
  • Step 1107 is repeated for the pixel coordinates of the center of the target point in the new captured image.
  • Step 1111 an error map of the camera is generated.
  • the camera error map is generated, and after the camera error map is generated, the electronic device evaluates the effect of the error map on the accuracy of the ADAS algorithm; or The error map will be fed back to the user to provide the user with full accuracy information, that is, the user will evaluate the effect of the error map on the accuracy of the ADAS algorithm.
  • a non-coplanar target is provided for the hardware-in-the-loop calibration of the camera; under the premise of considering the focus area of the camera, based on the idea of resampling and
  • the camera calibration algorithm obtains the camera's target pose; when the camera's hardware is calibrated in the loop, the target pose provides a reference for the camera's pose adjustment, which can effectively improve alignment efficiency and alignment accuracy.
  • FIG. 12 is a schematic flowchart of another hardware-in-the-loop calibration method for a camera provided by an embodiment of the present application.
  • the method can be applied to the hardware-in-the-loop calibration system for the camera shown in FIG. 2.
  • the method is performed by electronic Device execution, the electronic device includes the camera, or the electronic device is communicatively connected to the camera, the method includes but is not limited to the following steps:
  • Step 1201 setting non-coplanar target points.
  • Step 1202 check the total station.
  • Step 1203 Calculate the three-dimensional coordinates of the target point.
  • steps 1201-1203 are target setting and coordinate conversion (1): first, set non-coplanar target points, including the target points in the screen of the display device and the target points on the screen frame; then use the total station to make points, Obtain the three-dimensional coordinates of the target point in the total station coordinate system (that is, the second three-dimensional coordinates); then convert the three-dimensional coordinates of the target point in the total station coordinate system to the first coordinate system through the coordinate conversion formula to obtain the target.
  • the three-dimensional coordinates of the point in the first coordinate system is target setting and coordinate conversion (1): first, set non-coplanar target points, including the target points in the screen of the display device and the target points on the screen frame; then use the total station to make points, Obtain the three-dimensional coordinates of the target point in the total station coordinate system (that is, the second three-dimensional coordinates); then convert the three-dimensional coordinates of the target point in the total station coordinate system to the first coordinate system through the coordinate conversion formula to obtain the target.
  • Step 1204 focus on region selection.
  • Step 1205 Set the weight corresponding to the target point.
  • Step 1206 target bootstrap resampling.
  • Step 1207 target pose calculation.
  • steps 1204-1207 are target attitude calculation (2): first, the area of interest of the camera is selected in the calibration image, that is, the focus area of the camera; then the weight is set for the target point, wherein the focus area of the camera corresponds to The weight of the target point is greater than the weight of the target point corresponding to the non-focus area of the camera; then random sampling with replacement is performed on the target point through the bootstrap sampling method, and a series of target point combinations are obtained; finally, the target point is used in the first The three-dimensional coordinates in a coordinate system and the two-dimensional coordinates of the target point, combined with the internal parameters of the camera, obtain the target pose corresponding to each group of target points, and average the target poses corresponding to all groups of target points to obtain the target pose of the camera. .
  • the specific calculation process of the target pose is as follows:
  • the checkerboard calibration plate is collected from different angles of calibration images through the camera, and a certain number of calibration images are obtained; among them, when the calibration image is collected, the calibration plate should appear and fill the camera field of view as much as possible, that is, the calibration plate is in the calibration image.
  • the proportion of ⁇ should be as large as possible, and the angle coverage of the calibration plate should be as large as possible.
  • Select some calibration images from the collected calibration images For example, the number of selected calibration images is greater than 50, and perform corner detection on these selected calibration images to obtain the pixels of each corner in each image in the selected calibration images.
  • the internal parameters of the camera can be obtained by using Zhang's calibration method, That is, the distortion coefficient and internal parameter matrix of the camera are obtained.
  • the internal parameters of the camera After the internal parameters of the camera are obtained, the internal parameters of the camera, the distortion coefficient of the camera, the three-dimensional coordinates of the target point in the first coordinate system, and the two-dimensional coordinates of the target point are brought into the UPNP algorithm to obtain the external parameters of the camera (ie The position and attitude of the camera in the first coordinate system), that is, the target pose of the camera.
  • Step 1208 detecting the center of the circle.
  • a circle center detection is performed on the image currently captured by the camera to obtain the pixel coordinates of the target in the captured image.
  • Step 1209 current pose calculation.
  • the pixel coordinates of the target point in the captured image, the three-dimensional coordinates of the target point in the first coordinate system, the internal parameter matrix of the camera, and the distortion coefficient are brought into the UPNP algorithm to obtain the current pose of the camera.
  • Step 1210 pose residual calculation.
  • the current pose (position includes: x, y, z; pose includes: yaw, pitch, roll) and the target pose are different to obtain the residual of each component, and the residual of each component constitutes the residual of the pose .
  • Step 1211 whether the pose residual is greater than a threshold.
  • step 1212 if the pose residual is greater than the threshold, step 1212 is performed; when the pose residual is not greater than the threshold, step 1214 is performed.
  • Step 1212 prompting the orientation of the pose adjustment.
  • the direction and size of adjustment that need to be adjusted for each degree of freedom of the current pose are prompted.
  • Step 1213 pose adjustment.
  • Step 1214 generating an error map.
  • the system will pre-set the degree of freedom residual threshold corresponding to each degree of freedom.
  • the residual of all degrees of freedom is less than the corresponding degree of freedom residual threshold, the error map under the current pose is calculated.
  • FIG. 13 is a schematic diagram of a generation principle of an error map provided by an embodiment of the present application.
  • the electronic device controls the display device to play a frame of calibration image on the screen through the control module, and records the two-dimensional coordinates of the target point of the display device currently displaying the calibration image through the control module, and passes the center of the circle through the detection module.
  • the pixel coordinates of the target point in the image captured by the camera are detected, and for each target point, the Euclidean distance between the two-dimensional coordinates of the target point and the pixel coordinates of the target point in the image captured by the camera can be obtained.
  • the control module controls to continue playing the next frame of the calibration image, repeating The above operation obtains the pixel error information of all the calibration images to form the final error map.
  • Step 1215 Check whether the error map meets the requirements.
  • step 1216 is executed to complete the alignment; if the error map does not meet the requirements, step 1217 is executed.
  • Step 1216 the alignment ends.
  • Step 1217 whether to adjust the focus area.
  • the error map does not meet the requirements, it will prompt whether to adjust the focus area of the camera. If the focus area of the camera is adjusted, go back to 2 to select the focus area of the camera and calculate the target pose; if the focus area of the camera is not adjusted, go to step 1218 .
  • Step 1218 threshold adjustment.
  • the system provides a threshold adjustment window to allow the user to re-select the threshold; then return to calculate whether the residual between the camera's target pose and the current pose meets the threshold requirements; if not If satisfied, it is necessary to continue to adjust the pose of the camera until all the degrees of freedom of the pose residuals between the pose of the camera and the target pose meet the set threshold.
  • the calibration method proposed in this application can be applied to electronic equipment, and is automatically calibrated by electronic equipment.
  • a 6-DOF motor can be set, and the pose of the camera can be controlled and adjusted by using an appropriate control algorithm to realize automatic alignment;
  • the pose of the camera can also be manually adjusted manually.
  • the target points are set in the screen of the display device and on the screen border, which can avoid the problem that the calibration algorithm needs non-coplanar target points; Alignment cannot take into account all pixels.
  • different weights are set for the targets in different areas. Based on the weights, the target points are resampled with replacement to ensure that each group
  • the independence between the target point data also focuses on improving the probability of the target points corresponding to the focus area of the camera being extracted.
  • the hardware-in-the-loop calibration of the camera based on the current pose can effectively improve the alignment accuracy of the camera to the focus area.
  • FIG. 14 is a schematic structural diagram of a hardware-in-the-loop calibration target setting device for a camera provided by an embodiment of the present application.
  • the hardware-in-the-loop calibration target setting device 1400 for a camera can be applied to electronic equipment.
  • the device includes the camera, or the electronic device is communicatively connected to the camera.
  • the hardware-in-the-loop calibration target setting device 1400 of the camera may include a setting unit 1401, wherein the detailed description of each unit is as follows:
  • the setting unit 1401 is used to set M first target points in the screen of the display device, and set N second target points on the screen frame of the display device, wherein the display device is used to perform the camera's hardware in the ring mark.
  • the calibration image is displayed periodically, the M first target points and the N second target points are not coplanar, and M and N are integers greater than or equal to 3.
  • the setting unit 1401 is further configured to: obtain the second three-dimensional coordinates of the M first target points and the second three-dimensional coordinates of the N second target points, wherein the second three-dimensional coordinates are the full Coordinates in the station coordinate system; coordinate transformation is performed on the second three-dimensional coordinates of the M first target points and the second three-dimensional coordinates of the N second target points to obtain the first three-dimensional coordinates of the M first target points and N
  • the first three-dimensional coordinates of the second target point wherein the first three-dimensional coordinates are the coordinates in the first coordinate system, and the first coordinate system is a boundary point of the screen as the origin and the vertical direction of the screen as the first coordinate axis , take the horizontal direction of the screen as the second coordinate axis, take the vertical direction of the screen as the coordinate system of the third coordinate axis; the first three-dimensional coordinates of the M first target points and the first three-dimensional coordinates of the N second target points are used for The camera's hardware-in-the-l
  • coordinate transformation is performed on the second three-dimensional coordinates of the M first target points and the second three-dimensional coordinates of the N second target points to obtain the first three-dimensional coordinates of the M first target points.
  • the setting unit 1401 is specifically used to: obtain M according to the second three-dimensional coordinates of the M first target points and the second three-dimensional coordinates of the M first target points The first eigenvector and M second eigenvectors; the intermediate vector is obtained by least squares calculation according to the M first eigenvectors and the M second eigenvectors; the control matrix is obtained according to the intermediate vector, and the rotation vector sum is obtained according to the control matrix Translation vector; coordinate transformation is performed on the second three-dimensional coordinates of the M first target points and the second three-dimensional coordinates of the N second target points according to the rotation vector and the translation vector, and the first three-dimensional coordinates of the M first target points and The first three-dimensional coordinates of the N
  • the aspect ratio of the calibration image and the aspect ratio of the screen are the same as the aspect ratio of the display device
  • the setting unit 1401 is further configured to: perform circle center detection on the display device displaying the calibration image, so as to Obtain the pixel coordinates of each first target point in the M first target points; convert the pixel coordinates of each first target point in the M first target points to the second coordinate system to obtain the M first target points
  • Two-dimensional coordinates of each first target point in a target point wherein the second coordinate system is an outer boundary point of the screen frame of the display device as the origin, the vertical direction of the screen frame as the first coordinate axis, and the The horizontal direction of the screen frame is the coordinate system of the second coordinate axis; according to the two-dimensional coordinates of at least one first target point in the M first target points, the aspect ratio of the screen and the aspect ratio of the display device, N second target points are determined.
  • the setting unit 1401 is further configured to: according to the proportional relationship between the size of the calibration image and the size of the outer border of the screen frame, to The calibration image is enlarged to obtain an enlarged calibration image, wherein the size of the enlarged calibration image is equal to the size of the outer boundary of the screen frame; the enlarged calibration image is projected on the second coordinate system, wherein the second The coordinate system is a coordinate system with an outer boundary point of the screen frame of the display device as the origin, the vertical direction of the screen frame as the first coordinate axis, and the horizontal direction of the screen frame as the second coordinate axis; The pixel coordinates coincident with the center position of the target point are used as the two-dimensional coordinates of the target point, so as to obtain the two-dimensional coordinates of the M first target points and the two-dimensional coordinates of the N second target points.
  • each unit may also correspond to the corresponding descriptions of the embodiments shown in FIG. 2 to FIG. 13 .
  • the camera hardware-in-the-loop calibration target setting device 1400 provided in the embodiment of the present application includes but is not limited to the above-mentioned unit modules.
  • the camera hardware-in-the-loop calibration target setting device 1400 may also include a storage unit 1402.
  • the storage unit 1402 may be used to store the program codes and data of the hardware-in-the-loop calibration target setting device 1400 of the camera.
  • M first targets are set on the screen of the display device
  • N second targets are set on the screen frame of the display device
  • the M first target points and the N second target points are not coplanar, so there is no need to use multiple non-coplanar calibration plates during calibration, which solves the problem that the camera hardware is limited by the target setting space in the ring system. limit issue.
  • FIG. 15 is a schematic structural diagram of a hardware-in-the-loop calibration device for a camera provided by an embodiment of the present application.
  • the hardware-in-the-loop calibration device 1500 for a camera may include a determination unit 1501, a sampling unit 1502, and a calibration unit 1503.
  • the camera's hardware-in-the-loop calibration device 1500 is applied to an electronic device, and the electronic device includes the camera, or the electronic device is communicatively connected to the camera, wherein the detailed description of each unit is as follows:
  • the determining unit 1501 is used to determine the weight corresponding to each target point in the multiple target points, wherein the multiple target points are the target points on the display device, and the weight corresponding to the target points located in the target area on the display device is greater than that located in the target area.
  • the weight corresponding to the target point outside the target area, the target area of the display device is used to display a part of the calibration image; the calibration image is the image displayed by the device when the hardware of the camera is performing ring calibration;
  • Sampling unit 1502 configured to perform P sampling of multiple targets with replacement according to the weights to obtain P target groups, wherein, for each sampling, the larger the weight corresponding to the target, the more the target is selected.
  • the greater the probability of , P is a positive integer
  • the calibration unit 1503 is configured to perform hardware-in-the-loop calibration of the camera according to the P target point groups and the current pose of the camera.
  • the calibration unit 1503 is specifically configured to: determine the P first target poses according to the P target point groups, wherein the P target point groups are in one-to-one correspondence with the P first target poses ; Carry out image acquisition on the display device currently displaying the calibration image through the camera to obtain the target image, and determine the current pose of the camera according to multiple target points and the target image; According to the average value of the P first target poses and the current pose Perform hardware-in-the-loop calibration of the camera.
  • the calibration unit 1503 is further configured to: before determining the P first target poses according to the P target point groups; perform image acquisition on the calibration board through a camera to obtain Q calibration images, wherein , Q is a positive integer; corner detection is performed on each calibration image in the Q calibration images to obtain the pixel coordinates of each corner in each calibration image; The pixel coordinates of the corner points and the third three-dimensional coordinates of each corner point in each calibration image obtain the camera's internal parameter matrix and the camera's distortion coefficient, where the third three-dimensional coordinates are the coordinates in the calibration board coordinate system.
  • the calibration unit 1503 is specifically configured to: for each target point group in the P target point groups, perform the following steps to obtain the P first target poses:
  • the two-dimensional coordinates of each target point of An outer boundary point is the origin, the vertical direction of the screen frame is the first coordinate axis, and the horizontal direction of the screen frame is the coordinate system of the second coordinate axis; according to the two-dimensional coordinates of each target point in the target target group,
  • the first three-dimensional coordinates, the internal parameter matrix of the camera, and the distortion coefficient of the camera are calculated to obtain the first target pose corresponding to the target target group, wherein the first three-dimensional coordinates are the coordinates in the first coordinate system, and the first coordinate system is the coordinates in the first coordinate system.
  • a boundary point of the screen of the display device is the origin, the vertical direction of the screen is the first coordinate axis, the horizontal direction of the screen is the second coordinate axis, and the vertical direction of the screen is the third coordinate system.
  • the calibration unit 1503 is specifically configured to: perform circle center detection on the target image to obtain the pixel coordinates of each target point in the target image among the multiple target points;
  • the pixel coordinates of each target point in the target image, the first three-dimensional coordinates of each target point in the multiple target points, the internal parameter matrix of the camera, and the distortion coefficient of the camera are calculated to obtain the current pose, where the first three-dimensional coordinate is the coordinates in the first coordinate system, where the first coordinate system takes a boundary point of the screen of the display device as the origin, takes the vertical direction of the screen as the first coordinate axis, takes the horizontal direction of the screen as the second coordinate axis, and takes the screen's vertical direction as the second coordinate axis.
  • the vertical direction is the coordinate system of the third coordinate axis.
  • the plurality of target points include M first target points in the screen of the display device and N second target points on the screen frame, wherein the M first target points and the Nth target points The two target points are not coplanar, and M and N are integers greater than or equal to 3.
  • the calibration unit 1503 is further configured to: obtain M before performing the camera's hardware in-ring calibration according to the P target point groups and the current pose of the camera.
  • the second three-dimensional coordinates of the first target points and the second three-dimensional coordinates of the N second target points wherein the second three-dimensional coordinates are the coordinates in the coordinate system of the total station; for the second three-dimensional coordinates of the M first target points Coordinate transformation is performed on the coordinates to obtain the first three-dimensional coordinates of the M first target points, and coordinate transformation is performed on the second three-dimensional coordinates of the N second target points to obtain the first three-dimensional coordinates of the N second target points.
  • the calibration unit 1503 is specifically configured to: make a difference between the second target pose and the current pose to obtain the current pose residual, where the second target pose is P first targets
  • the average value of the pose, the current pose residual includes 3 current position DOF residuals and 3 current posture DOF residuals; first adjust the camera positions in the order of the current position DOF residuals from large to small degrees of freedom, so that the difference between the first positional degree of freedom of the camera and the first positional degree of freedom of the second target pose is less than the preset first positional degree of freedom residual threshold, the second positional degree of freedom of the camera and the second
  • the difference between the second position degree of freedom of the target pose is less than the preset second position degree of freedom residual threshold
  • the difference between the third position freedom of the camera and the third position freedom of the second target pose is less than the preset
  • the third position degree of freedom residual threshold value of The difference of a degree of freedom of attitude is less than the preset residual threshold of the first degree of freedom of attitude, and the difference between the second degree of freedom of
  • the hardware-in-the-loop calibration device 1500 for the camera provided in the embodiment of the present application includes but is not limited to the above-mentioned unit modules.
  • the hardware-in-the-loop calibration device 1500 for the camera may further include a storage unit 1504 .
  • the storage unit 1504 may be used to store program codes and data of the hardware-in-the-loop calibration device 1500 of the camera.
  • the hardware-in-the-loop calibration device 1500 of the camera described in FIG. 15 different weights are set for the target points in different regions, and the target points are resampled with replacement based on the weights, which not only ensures the data of each group of target points Independent, it also focuses on improving the probability of the target points corresponding to the focus area of the camera being extracted.
  • the hardware-in-the-loop calibration of the camera is performed according to the sampled target points and the current pose of the camera, which can effectively improve the camera's focus on the focus. Alignment accuracy of the area.
  • the hardware for the camera has a plurality of target points on a display device for ring calibration, and a calibration image is displayed on the display device, the calibration image has the focus area of the camera, and the calibration image is displayed on the display device.
  • the display area used to display the focus area is the target area, and the weight corresponding to the target point within the target area is set to be greater than the weight corresponding to the target point outside the target area; then based on the weight probability Perform P sampling of multiple targets with replacement to obtain P target groups, that is, the greater the weight corresponding to the target, the greater the probability of the target being extracted; due to the weight of the target in the target area is greater than the weight corresponding to the target points outside the target area, so each time you sample, the probability of the target points in the target area being extracted is greater, and the number of target points in the target area is relatively larger.
  • the embodiments of the present application can effectively improve the alignment accuracy of the camera to the focus area, and ensure the personalized alignment requirement of the user.
  • FIG. 16 is a schematic structural diagram of an electronic device provided by an embodiment of the present application.
  • the electronic device 1610 includes a transceiver 1611, a processor 1612, and a memory 1613.
  • the transceiver 1611, the processor 1612, and the memory 1613 pass through a bus 1614 are connected to each other.
  • the memory 1613 includes, but is not limited to, random access memory (RAM), read-only memory (ROM), erasable programmable read only memory (EPROM), or A portable read-only memory (compact disc read-only memory, CD-ROM), the memory 1613 is used for related instructions and data.
  • RAM random access memory
  • ROM read-only memory
  • EPROM erasable programmable read only memory
  • CD-ROM compact disc read-only memory
  • the transceiver 1611 is used to receive and transmit data.
  • the processor 1612 may be one or more central processing units (central processing units, CPUs). In the case where the processor 1612 is a CPU, the CPU may be a single-core CPU or a multi-core CPU.
  • the processor 1612 in the electronic device 1610 is configured to read program codes stored in the memory 1613, and execute the methods described in the embodiments of this application.
  • each operation may also correspond to the corresponding descriptions of the embodiments shown in FIG. 2 to FIG. 13 .
  • non-coplanar target points are set in the screen of the display device and on the screen frame, so there is no need to use multiple non-coplanar calibration boards during calibration, which solves the problem of the hardware of the camera.
  • the problem of limited space for target setting in the ring system; and setting different weights for targets in different regions, and resampling the targets with replacement based on the weights not only ensures the independence of each group of target data, It also focuses on improving the probability of the target points corresponding to the focus area of the camera being extracted.
  • the hardware-in-the-loop calibration of the camera is performed according to the sampled target points and the current pose of the camera, which can effectively improve the camera's accuracy to the focus area. Quasi-accuracy.
  • An embodiment of the present application further provides a chip, the chip includes at least one processor, a memory and an interface circuit, the memory, the transceiver and the at least one processor are interconnected through a line, and a computer program is stored in the at least one memory;
  • the computer program is executed by the above-mentioned processor, the method flow shown in the above-mentioned method embodiment is realized.
  • Embodiments of the present application further provide a computer-readable storage medium, where a computer program is stored in the computer-readable storage medium, and when the computer program is executed on an electronic device, the method flow shown in the foregoing method embodiments is realized.
  • the embodiments of the present application further provide a computer program, when the computer program is executed on an electronic device, the method flow shown in the above method embodiments can be realized.
  • the existing camera calibration methods basically have no quantitative indicators, and only qualitatively give the initial position of the camera (such as on the central axis of the screen), and then manually adjust the camera to adjust the camera so that it can be completely captured. screen; since the pose adjustment of the camera greatly depends on the user's experience, this qualitative pose adjustment method is usually inefficient (fine adjustment takes about 1 hour); in addition, the accuracy of this traditional calibration method is difficult to guarantee, and It is impossible to perform focused pose adjustment according to the function of the camera, and the applicability is poor.
  • the embodiment of the present application can perform precise adjustment of the camera pose.
  • the method utilizes the target point in the screen and the target point on the screen frame.
  • the characteristics of non-coplanarity avoid the disadvantages of using multiple calibration boards; and a hardware-in-the-loop calibration method for cameras is proposed.
  • the target combination is generated by the bootstrap resampling method based on weight assignment, and Zhang’s calibration method and UPNP are used.
  • the algorithm and the known coordinate information of the target point calculate the target pose of the camera, which provides a benchmark for the alignment direction of the camera, which can greatly improve the alignment efficiency and alignment accuracy.
  • the embodiment of the present application also provides the evaluation index of the final alignment result and its calculation method (error map), provides the user with the full amount of final error information, and provides the original error input for the user to evaluate the ADAS algorithm.
  • the embodiment of the present application provides a method for setting a hardware-in-the-loop calibration target point of a camera.
  • Using the camera calibration method for the camera's hardware-in-the-loop calibration can effectively improve the alignment efficiency and alignment accuracy of the camera.
  • the calibration algorithm of the camera needs to use non-coplanar target information, and usually two non-coplanar calibration plates are used to complete the calibration of the camera.
  • the hardware-in-the-loop system of the camera cannot use the calibration board. For this reason, a target point is designed in the screen and the target point is set on the screen frame, so as to realize the setting of the non-coplanar target point and give the calculation of the three-dimensional coordinates of the target point. and conversion method.
  • the embodiment of the present application provides a target pose solution method based on the selection of a focus area. Cameras with different functions focus on different areas. The function of designing and selecting key areas can effectively improve the alignment accuracy of key areas and reduce the error of the original data layer of the camera.
  • the present application proposes a weight-based bootstrap resampling target combination generation method.
  • the camera calibration technology is used for each set of target data to solve the target pose of the camera, and the target pose value of the camera is obtained by averaging. This sampling method can ensure the independence of each group of target data, and at the same time focuses on the targets in the focus area of the alignment frame, ensuring the user's personalized alignment needs.
  • the camera pose adjustment benchmark is quantified, which can effectively improve the calibration efficiency and calibration accuracy of the camera.
  • Beneficial effect 1 The hardware-in-the-loop calibration target setting method of the camera provided by the embodiment of the present application does not require the use of multiple non-coplanar calibration plates, which solves the problem of the limited space for setting the target in the hardware-in-the-loop calibration system of the camera. Problem; through coordinate conversion, the target coordinates can be converted to the screen coordinate system, which lays the foundation for the subsequent adjustment feedback prompts, which is conducive to improving the camera alignment effect.
  • a hardware-in-the-loop calibration method for a camera provided by the embodiment of the present application, combined with a weight resampling method and a camera calibration algorithm, solves the target pose of the camera while ensuring the alignment accuracy of the focus area of the camera , the target pose provides a quantitative benchmark for camera alignment, which ensures alignment accuracy and improves alignment efficiency.
  • processors mentioned in the embodiments of the present application may be a central processing unit (Central Processing Unit, CPU), and may also be other general-purpose processors, digital signal processors (Digital Signal Processor, DSP), application-specific integrated circuits ( Application Specific Integrated Circuit, ASIC), off-the-shelf Programmable Gate Array (Field Programmable Gate Array, FPGA) or other programmable logic devices, discrete gate or transistor logic devices, discrete hardware components, etc.
  • a general purpose processor may be a microprocessor or the processor may be any conventional processor or the like.
  • the memory mentioned in the embodiments of the present application may be volatile memory or non-volatile memory, or may include both volatile and non-volatile memory.
  • the non-volatile memory may be a read-only memory (Read-Only Memory, ROM), a programmable read-only memory (Programmable ROM, PROM), an erasable programmable read-only memory (Erasable PROM, EPROM), an electrically programmable read-only memory (Erasable PROM, EPROM). Erase programmable read-only memory (Electrically EPROM, EEPROM) or flash memory.
  • Volatile memory may be Random Access Memory (RAM), which acts as an external cache.
  • RAM Static RAM
  • DRAM Dynamic RAM
  • SDRAM Synchronous DRAM
  • SDRAM double data rate synchronous dynamic random access memory
  • Double Data Rate SDRAM DDR SDRAM
  • enhanced SDRAM ESDRAM
  • synchronous link dynamic random access memory Synchlink DRAM, SLDRAM
  • Direct Rambus RAM Direct Rambus RAM
  • the processor is a general-purpose processor, DSP, ASIC, FPGA or other programmable logic devices, discrete gate or transistor logic devices, or discrete hardware components
  • the memory storage module
  • the size of the sequence numbers of the above-mentioned processes does not mean the sequence of execution, and the execution sequence of each process should be determined by its functions and internal logic, and should not be dealt with in the embodiments of the present application. implementation constitutes any limitation.
  • the disclosed system, apparatus and method may be implemented in other manners.
  • the apparatus embodiments described above are only illustrative.
  • the division of the above units is only a logical function division.
  • multiple units or components may be combined or may be Integration into another system, or some features can be ignored, or not implemented.
  • the shown or discussed mutual coupling or direct coupling or communication connection may be through some interfaces, indirect coupling or communication connection of devices or units, and may be in electrical, mechanical or other forms.
  • the above-mentioned units described as separate components may or may not be physically separated, and components displayed as units may or may not be physical units, that is, may be located in one place, or may be distributed to multiple network units. Some or all of the units may be selected according to actual needs to achieve the purpose of the solution in this embodiment.
  • each functional unit in each embodiment of the present application may be integrated into one processing unit, or each unit may exist physically alone, or two or more units may be integrated into one unit.
  • the above functions are implemented in the form of software functional units and sold or used as independent products, they may be stored in a computer-readable storage medium.
  • the technical solution of the present application can be embodied in the form of a software product in essence, or the part that contributes to the prior art or the part of the technical solution, and the computer software product is stored in a storage medium, including Several instructions are used to cause a computer device (which may be a personal computer, a server, or a network device, etc.) to execute all or part of the steps of the above-mentioned methods of the various embodiments of the present application.
  • the aforementioned storage medium includes: U disk, mobile hard disk, read-only memory (Read-Only Memory, ROM), random access memory (Random Access Memory, RAM), magnetic disk or optical disk and other media that can store program codes .
  • the modules in the apparatus of the embodiment of the present application may be combined, divided and deleted according to actual needs.

Landscapes

  • Engineering & Computer Science (AREA)
  • Computer Vision & Pattern Recognition (AREA)
  • Physics & Mathematics (AREA)
  • General Physics & Mathematics (AREA)
  • Theoretical Computer Science (AREA)
  • Studio Devices (AREA)
  • Closed-Circuit Television Systems (AREA)

Abstract

本申请实施例提供一种摄像头的硬件在环标定、靶点设置方法、系统及相关设备，其中，方法包括：确定多个靶点中的每个靶点对应的权重，其中，多个靶点为显示设备上的靶点，显示设备上位于目标区域内的靶点对应的权重大于位于目标区域外的靶点对应的权重，显示设备的目标区域用于显示标定图像的一部分；标定图像为在进行摄像头的硬件在环标定时显示设备显示的图像；根据权重对多个靶点进行P次有放回的抽样，以得到P个靶点组，其中，针对每次抽样，靶点对应的权重越大，该靶点被抽取的概率越大，P为正整数；根据P个靶点组和摄像头的当前位姿进行摄像头的硬件在环标定。采用本申请实施例，有利于提升对准精度和标定效率。

Description

摄像头的硬件在环标定、靶点设置方法、系统及相关设备 技术领域
本申请涉及硬件在环(Hardware in the Loop，HIL)技术领域，尤其涉及一种摄像头的硬件在环标定、靶点设置方法、系统及相关设备。
背景技术
硬件在环是验证高级驾驶辅助系统(Advanced Driving Assistance System，ADAS)的重要方法，视频注入是构建硬件在环的重要步骤。而传统的视频灌注设备价格昂贵，且适用性较差，例如不同的相机需要使用不同的视频注入设备，因此极大地提高了视频注入的成本。相机在环是一种在暗箱中直接通过相机拍摄屏幕的方式达到视频注入功能的技术。相机在环无需特定的视频注入设备，价格便宜且适用性强，可完成不同型号相机的视频灌注。相机在环通过在暗箱中模拟各种环境，例如雨天、雾天等，可达到视频再处理功能，极大地丰富了用户的需求。因此，相机在环已成为实现视频注入功能的重要技术。
相机在环的本质是使得相机拍摄图像和实际图像完全相同，其最终目标是实现像素级的对准，也即实现实际的像素坐标和拍摄所得的像素坐标完全相同。目前，相机的对准精度大多是基于机加工或者简单手工调节保证，故精度较低且极大依赖调节者的经验，调节时间长，效率低下。因此，如何提升相机的对准精度和标定效率已成为亟需解决的技术问题。
在高级驾驶辅助系统功能集成中，不同的相机通常承担不同的功能，也即不同的相机有不同重点关注区域；从而相机对不同区域的精度要求也不尽相同，而由于相机模组加工误差及相机建模与实际相机的差异，例如畸变校正模型无法完全拟合实际模型，使得相机对准通常无法做到绝对的准确，通常只能保证局部的高精度；故相机对重点关注区域的精度应尽可能的高，相机对非重点关注区域的精度相对于重点关注区域可以低些。因此，如何提升相机对重点关注区域的对准精度也成为亟需解决的技术问题。
发明内容
本申请实施例公开了一种摄像头的硬件在环标定、靶点设置方法、系统及相关设备，有利于提升对准精度和标定效率。
本申请实施例第一方面公开了一种摄像头的硬件在环标定方法，该方法包括：确定多个靶点中的每个靶点对应的权重，其中，多个靶点为显示设备上的靶点，显示设备上位于目标区域内的靶点对应的权重大于位于目标区域外的靶点对应的权重，显示设备的目标区域用于显示标定图像的一部分；标定图像为在进行摄像头的硬件在环标定时显示设备显示的图像；根据权重对多个靶点进行P次有放回的抽样，以得到P个靶点组，其中，针对每次抽样，靶点对应的权重越大，该靶点被抽取的概率越大，P为正整数；根据P个靶点组和摄像头的当前位姿进行摄像头的硬件在环标定。在本申请实施例中，给不同区域的靶点 设置不同的权重，基于权重对靶点进行有放回的重采样既保证了各组靶点数据之间的独立性，又侧重提升了摄像头的重点关注区域对应的靶点被抽取的概率，根据采样得到的各组靶点和摄像头的当前位姿进行摄像头的硬件在环标定，可有效提升摄像头对重点关注区域的对准精度。具体地，用于摄像头的硬件在环标定的显示设备上有多个靶点，在该显示设备上显示标定图像，该标定图像有该摄像头的重点关注区域，该标定图像被显示在该显示设备上时，该显示设备上用于显示该重点关注区域的显示区域为目标区域，先设置该目标区域内的靶点对应的权重大于位于该目标区域外的靶点对应的权重；然后基于权重概率对多个靶点进行P次有放回的抽样以得到P个靶点组，也即靶点对应的权重越大，该靶点被抽取的概率越大；由于目标区域内的靶点的权重大于位于该目标区域外的靶点对应的权重，故每次抽样时，目标区域内的靶点被抽取的概率更大，目标区域内的靶点被抽取的数量相对就越多，在根据P个靶点组和摄像头的当前位姿进行摄像头的硬件在环标定时，目标区域内的靶点所起的决定作用占比就越大，摄像头对重点关注区域的对准精度就越高。综上，本申请实施例，可有效提升摄像头对重点关注区域的对准精度，保证用户的个性化对准需求。
在一种可能的实现方式中，根据P个靶点组和摄像头的当前位姿进行摄像头的硬件在环标定，包括：根据P个靶点组确定P个第一目标位姿，其中，P个靶点组与P个第一目标位姿一一对应；通过摄像头对当前显示标定图像的显示设备进行图像采集以得到目标图像，并根据多个靶点和目标图像确定摄像头的当前位姿；根据P个第一目标位姿的平均值和当前位姿进行摄像头的硬件在环标定。在本申请实施例中，先根据又放回的重采样得到的P个靶点组确定P个第一目标位姿；然后通过摄像头对当前显示标定图像的显示设备进行图像采集以得到目标图像，并根据显示设备上的多个靶点和该目标图像确定摄像头的当前位姿；再将P个第一目标位姿的平均值作为需要调整到的位姿，进行该摄像头的硬件在环标定，也即将该摄像头的当前位姿调整至该P个第一目标位姿的平均值；由于利用重采样的方法多次计算摄像头的目标姿态，通过求平均的方法得到需要调整到的位姿，该需要调整到的位姿不仅为摄像头的位姿调整方向提供参考标杆，同时着重保证了摄像头的重点关注区域的对准精度，有利于提升摄像头对准的标定效率、对准精度。
在一种可能的实现方式中，在根据P个靶点组确定P个第一目标位姿之前，该方法还包括：通过摄像头对标定板进行图像采集以得到Q张标定图像，其中，Q为正整数；对Q张标定图像中的每张标定图像进行角点检测以得到每张标定图像中的各个角点的像素坐标；根据Q张标定图像中的每张标定图像中的各个角点的像素坐标、每张标定图像中的各个角点的第三三维坐标得到摄像头的内参矩阵和摄像头的畸变系数，其中，第三三维坐标为标定板坐标系下的坐标。在本申请实施例中，通过摄像头对标定板进行图像采集以得到多张标定图像，同该多张标定图像得到摄像头的内参矩阵和摄像头的畸变系数，而标定过程中的位姿计算需要用到摄像头的内参矩阵和摄像头的畸变系数，从而可以为计算摄像头的需要调整到的位姿以及摄像头的当前位姿提供基础，有利于提升摄像头的硬件在环对准精度。
在一种可能的实现方式中，根据P个靶点组确定P个第一目标位姿，包括：针对P个靶点组中的每个靶点组，执行以下步骤，得到P个第一目标位姿：获取目标靶点组中的每 个靶点的二维坐标，其中，目标靶点组为P个靶点组中的任意一个，二维坐标为第二坐标系下的坐标，第二坐标系为以显示设备的屏幕边框的一个外边界点为原点、以屏幕边框的竖向为第一坐标轴、以屏幕边框的横向为第二坐标轴的坐标系；根据目标靶点组中的每个靶点的二维坐标、第一三维坐标、摄像头的内参矩阵和摄像头的畸变系数，计算得到目标靶点组对应的第一目标位姿，其中，第一三维坐标为第一坐标系下的坐标，第一坐标系为以显示设备的屏幕的一个边界点为原点、以屏幕的竖向为第一坐标轴、以屏幕的横向为第二坐标轴、以屏幕的垂直方向为第三坐标轴的坐标系。在本申请实施例中，根据目标靶点组中的每个靶点的二维坐标、第一三维坐标、摄像头的内参矩阵和摄像头的畸变系数，计算得到目标靶点组对应的第一目标位姿，其中，该二维坐标为第二坐标系下的坐标，该第一三维坐标为第一坐标系下的坐标；由于显示设备的屏幕和屏幕边框的尺寸是固定的，当屏幕上显示有标定图像时，可以根据标定图像的像素坐标推算得到每个靶点在第二坐标系上的二维坐标；并且可以通过全站仪获取每个靶点在全站仪坐标系下的三维坐标，再将每个靶点在全站仪坐标系下的三维坐标转换成在第一坐标系下的三维坐标；从而有利于得到每个靶点组的第一目标位姿，进而得到摄像头需要调整到的位姿。
在一种可能的实现方式中，根据多个靶点和目标图像确定摄像头的当前位姿，包括：对目标图像进行圆心检测以得到多个靶点中的每个靶点在目标图像中的像素坐标；根据多个靶点中的每个靶点在目标图像中的像素坐标、多个靶点中的每个靶点的第一三维坐标、摄像头的内参矩阵、摄像头的畸变系数，计算得到当前位姿，其中，第一三维坐标为第一坐标系下的坐标，第一坐标系为以显示设备的屏幕的一个边界点为原点、以屏幕的竖向为第一坐标轴、以屏幕的横向为第二坐标轴、以屏幕的垂直方向为第三坐标轴的坐标系。在本申请实施例中，通过摄像头对当前显示标定图像的显示设备进行图像采集，得到目标图像后，可以通过对目标图像进行圆心检测，获得显示设备上的多个靶点中的每个靶点在目标图像中的像素坐标，从而有利于根据每个靶点在目标图像中的像素坐标、每个靶点的第一三维坐标、摄像头的内参矩阵、摄像头的畸变系数，计算得到当前位姿。
在一种可能的实现方式中，多个靶点包括显示设备的屏幕内的M个第一靶点和屏幕边框上的N个第二靶点，其中，M个第一靶点与N个第二靶点不共面，M和N为大于或等于3的整数，在根据P个靶点组和摄像头的当前位姿进行摄像头的硬件在环标定之前，该方法还包括：获取M个第一靶点的第二三维坐标和N个第二靶点的第二三维坐标，其中，第二三维坐标为全站仪坐标系下的坐标；对M个第一靶点的第二三维坐标进行坐标转换以得到M个第一靶点的第一三维坐标，以及对N个第二靶点的第二三维坐标进行坐标转换以得到N个第二靶点的第一三维坐标。在本申请实施例中，在显示设备的屏幕内设置M个第一靶点，以及在显示设备的屏幕边框上设置N个第二靶点，且该M个第一靶点与该N个第二靶点是不共面的，因此在标定时无需使用多块不共面的标定板，解决了摄像头的硬件在环系统中靶点设置空间受限的问题；并且通过坐标转换，将靶点在全站仪坐标系下的坐标转换到第一坐标系下，为后续的标定反馈提示奠定基础，是提升摄像头对准效果的关键步骤。
在一种可能的实现方式中，根据P个第一目标位姿的平均值和当前位姿进行摄像头的硬件在环标定，包括：将第二目标位姿与当前位姿作差以得到当前位姿残差，其中，第二 目标位姿为P个第一目标位姿的平均值，当前位姿残差包括3个当前位置自由度残差和3个当前姿态自由度残差；先按照当前位置自由度残差从大到小的顺序依次调节摄像头的各个位置自由度，以使摄像头的第一位置自由度与第二目标位姿的第一位置自由度的差值小于预设的第一位置自由度残差阈值、摄像头的第二位置自由度与第二目标位姿的第二位置自由度的差值小于预设的第二位置自由度残差阈值、摄像头的第三位置自由度与第二目标位姿的第三位置自由度的差值小于预设的第三位置自由度残差阈值；再按照当前姿态自由度残差从大到小的顺序依次调节摄像头的各个姿态自由度，以使摄像头的第一姿态自由度与第二目标位姿的第一姿态自由度的差值小于预设的第一姿态自由度残差阈值、摄像头的第二姿态自由度与第二目标位姿的第二姿态自由度的差值小于预设的第二姿态自由度残差阈值、摄像头的第三姿态自由度与第二目标位姿的第三姿态自由度的差值小于预设的第三姿态自由度残差阈值。在本申请实施例中，在将摄像头的姿态从当前位姿调节到需要调整到的位姿时，先反馈位置中残差较大的分量，对该残差较大的分量向对应方向调节某一数值；后反馈位置中残差较小的分量，对该残差较小的分量向对应方向调节某一数值；待调节好3个位置自由度后，再调节姿态分量中较大的分量，且按照残差从大到小依次调节各姿态分量；实践表明，先调节位置，再调节姿态，可提高调节效率。
本申请实施例第二方面公开了一种摄像头的硬件在环标定靶点设置方法，包括：在显示设备的屏幕内设置M个第一靶点，以及在显示设备的屏幕边框上设置N个第二靶点，其中，显示设备用于在进行摄像头的硬件在环标定时显示标定图像，M个第一靶点与N个第二靶点不共面，M和N为大于或等于3的整数。在本申请实施例中，在显示设备的屏幕内设置M个第一靶点，以及在显示设备的屏幕边框上设置N个第二靶点，且该M个第一靶点与该N个第二靶点是不共面的，因此在标定时无需使用多块不共面的标定板，解决了摄像头的硬件在环系统中靶点设置空间受限的问题。
在一种可能的实现方式中，该方法还包括：获取M个第一靶点的第二三维坐标和N个第二靶点的第二三维坐标，其中，第二三维坐标为全站仪坐标系下的坐标；对M个第一靶点的第二三维坐标和N个第二靶点的第二三维坐标进行坐标转换，得到M个第一靶点的第一三维坐标和N个第二靶点的第一三维坐标，其中，第一三维坐标为第一坐标系下的坐标，第一坐标系为以屏幕的一个边界点为原点、以屏幕的竖向为第一坐标轴、以屏幕的横向为第二坐标轴、以屏幕的垂直方向为第三坐标轴的坐标系；M个第一靶点的第一三维坐标和N个第二靶点的第一三维坐标用于摄像头的硬件在环标定。在本申请实施例中，通过坐标转换，将靶点在全站仪坐标系下的坐标转换到第一坐标系下，靶点在第一坐标系下的第一三维坐标可以供摄像头的硬件在环标定使用，从而为后续的摄像头的硬件在环标定提供基础，有利于提升摄像头对准效果。
在一种可能的实现方式中，对M个第一靶点的第二三维坐标和N个第二靶点的第二三维坐标进行坐标转换，得到M个第一靶点的第一三维坐标和N个第二靶点的第一三维坐标，包括：根据M个第一靶点的第二三维坐标和M个第一靶点的第二三维坐标，得到M个第一特征向量和M个第二特征向量；根据M个第一特征向量和M个第二特征向量采用最小二乘法计算得到中间向量；根据中间向量得到控制矩阵，以及根据控制矩阵得到旋转向量和平移向量； 根据旋转向量和平移向量对M个第一靶点的第二三维坐标和N个第二靶点的第二三维坐标进行坐标转换，得到M个第一靶点的第一三维坐标和N个第二靶点的第一三维坐标。在本申请实施例中，以屏幕内的第一靶点作为参考点，求解全站仪坐标系到第一坐标系的旋转矩阵和平移向量；再依据该旋转矩阵和平移向量将靶点在全站仪坐标系下的坐标转换到第一坐标系下，并将靶点在第一坐标系下的第一三维坐标供摄像头的硬件在环标定使用，从而为后续的摄像头的硬件在环标定提供基础，有利于提升摄像头对准效果。
在一种可能的实现方式中，标定图像的高宽比、屏幕的高宽比与显示设备的高宽比相同，该方法还包括：对显示标定图像的显示设备进行圆心检测，以得到M个第一靶点中的每个第一靶点的像素坐标；将M个第一靶点中的每个第一靶点的像素坐标转换到第二坐标系下，以得到M个第一靶点中的每个第一靶点的二维坐标，其中，第二坐标系为以显示设备的屏幕边框的一个外边界点为原点、以屏幕边框的竖向为第一坐标轴、以屏幕边框的横向为第二坐标轴的坐标系；根据M个第一靶点中的至少一个第一靶点的二维坐标、屏幕的高宽比与显示设备的高宽比确定N个第二靶点在第二坐标系下的二维坐标。在本申请实施例中，先通过圆心检测获取第一靶点的像素坐标；然后通过将第一靶点的像素坐标转换到第二坐标系下，得到第一靶点的二维坐标；再依据第一靶点的二维坐标推算出第二靶点的二维坐标，从而可以得到用于摄像头的硬件在环标定的所有靶点的二维坐标。
在一种可能的实现方式中，标定图像的尺寸与屏幕边框的外边界尺寸存在比例关系，该方法还包括：根据标定图像的尺寸与屏幕边框的外边界尺寸存在的比例关系，对标定图像进行放大处理，以得到放大后的标定图像，其中，放大后的标定图像的尺寸与屏幕边框的外边界尺寸相等；将放大后的标定图像投影到第二坐标系上，其中，第二坐标系为以显示设备的屏幕边框的一个外边界点为原点、以屏幕边框的竖向为第一坐标轴、以屏幕边框的横向为第二坐标轴的坐标系；将放大后的标定图像中与靶点中心位置重合的像素坐标作为该靶点的二维坐标，以得到M个第一靶点的二维坐标和N个第二靶点的二维坐标。在本申请实施例中，通过将标定图像放大后投影到第二坐标系上，并将放大后的标定图像中与靶点中心位置重合的像素坐标作为该靶点的二维坐标，从而可以得到用于摄像头的硬件在环标定的所有靶点的二维坐标。
本申请实施例第三方面公开了一种摄像头的硬件在环标定装置，该装置包括：确定单元，用于确定多个靶点中的每个靶点对应的权重，其中，多个靶点为显示设备上的靶点，显示设备上位于目标区域内的靶点对应的权重大于位于目标区域外的靶点对应的权重，显示设备的目标区域用于显示标定图像的一部分；标定图像为在进行摄像头的硬件在环标定时显示设备显示的图像；抽样单元，用于根据权重对多个靶点进行P次有放回的抽样，以得到P个靶点组，其中，针对每次抽样，靶点对应的权重越大，该靶点被抽取的概率越大，P为正整数；标定单元，用于根据P个靶点组和摄像头的当前位姿进行摄像头的硬件在环标定。
在一种可能的实现方式中，标定单元，具体用于：根据P个靶点组确定P个第一目标位姿，其中，P个靶点组与P个第一目标位姿一一对应；通过摄像头对当前显示标定图像的显示设备进行图像采集以得到目标图像，并根据多个靶点和目标图像确定摄像头的当前位姿；根据P个第一目标位姿的平均值和当前位姿进行摄像头的硬件在环标定。
在一种可能的实现方式中，标定单元，还用于：在根据P个靶点组确定P个第一目标位姿之前；通过摄像头对标定板进行图像采集以得到Q张标定图像，其中，Q为正整数；对Q张标定图像中的每张标定图像进行角点检测以得到每张标定图像中的各个角点的像素坐标；根据Q张标定图像中的每张标定图像中的各个角点的像素坐标、每张标定图像中的各个角点的第三三维坐标得到摄像头的内参矩阵和摄像头的畸变系数，其中，第三三维坐标为标定板坐标系下的坐标。
在一种可能的实现方式中，标定单元，具体用于：针对P个靶点组中的每个靶点组，执行以下步骤，得到P个第一目标位姿：获取目标靶点组中的每个靶点的二维坐标，其中，目标靶点组为P个靶点组中的任意一个，二维坐标为第二坐标系下的坐标，第二坐标系为以显示设备的屏幕边框的一个外边界点为原点、以屏幕边框的竖向为第一坐标轴、以屏幕边框的横向为第二坐标轴的坐标系；根据目标靶点组中的每个靶点的二维坐标、第一三维坐标、摄像头的内参矩阵和摄像头的畸变系数，计算得到目标靶点组对应的第一目标位姿，其中，第一三维坐标为第一坐标系下的坐标，第一坐标系为以显示设备的屏幕的一个边界点为原点、以屏幕的竖向为第一坐标轴、以屏幕的横向为第二坐标轴、以屏幕的垂直方向为第三坐标轴的坐标系。
在一种可能的实现方式中，标定单元，具体用于：对目标图像进行圆心检测以得到多个靶点中的每个靶点在目标图像中的像素坐标；根据多个靶点中的每个靶点在目标图像中的像素坐标、多个靶点中的每个靶点的第一三维坐标、摄像头的内参矩阵、摄像头的畸变系数，计算得到当前位姿，其中，第一三维坐标为第一坐标系下的坐标，第一坐标系为以显示设备的屏幕的一个边界点为原点、以屏幕的竖向为第一坐标轴、以屏幕的横向为第二坐标轴、以屏幕的垂直方向为第三坐标轴的坐标系。
在一种可能的实现方式中，多个靶点包括显示设备的屏幕内的M个第一靶点和屏幕边框上的N个第二靶点，其中，M个第一靶点与N个第二靶点不共面，M和N为大于或等于3的整数，标定单元，还用于：在根据P个靶点组和摄像头的当前位姿进行摄像头的硬件在环标定之前，获取M个第一靶点的第二三维坐标和N个第二靶点的第二三维坐标，其中，第二三维坐标为全站仪坐标系下的坐标；对M个第一靶点的第二三维坐标进行坐标转换以得到M个第一靶点的第一三维坐标，以及对N个第二靶点的第二三维坐标进行坐标转换以得到N个第二靶点的第一三维坐标。
在一种可能的实现方式中，标定单元，具体用于：将第二目标位姿与当前位姿作差以得到当前位姿残差，其中，第二目标位姿为P个第一目标位姿的平均值，当前位姿残差包括3个当前位置自由度残差和3个当前姿态自由度残差；先按照当前位置自由度残差从大到小的顺序依次调节摄像头的各个位置自由度，以使摄像头的第一位置自由度与第二目标位姿的第一位置自由度的差值小于预设的第一位置自由度残差阈值、摄像头的第二位置自由度与第二目标位姿的第二位置自由度的差值小于预设的第二位置自由度残差阈值、摄像头的第三位置自由度与第二目标位姿的第三位置自由度的差值小于预设的第三位置自由度残差阈值；再按照当前姿态自由度残差从大到小的顺序依次调节摄像头的各个姿态自由度，以使摄像头的第一姿态自由度与第二目标位姿的第一姿态自由度的差值小于预设的第一姿态自由度残差阈值、摄像头的第二姿态自由度与第二目标位姿的第二姿态自由度的差值小于 预设的第二姿态自由度残差阈值、摄像头的第三姿态自由度与第二目标位姿的第三姿态自由度的差值小于预设的第三姿态自由度残差阈值。
本申请实施例第四方面公开了一种摄像头的硬件在环标定靶点设置装置，包括：设置单元，用于在显示设备的屏幕内设置M个第一靶点，以及在显示设备的屏幕边框上设置N个第二靶点，其中，显示设备用于在进行摄像头的硬件在环标定时显示标定图像，M个第一靶点与N个第二靶点不共面，M和N为大于或等于3的整数。
在一种可能的实现方式中，设置单元，还用于：获取M个第一靶点的第二三维坐标和N个第二靶点的第二三维坐标，其中，第二三维坐标为全站仪坐标系下的坐标；对M个第一靶点的第二三维坐标和N个第二靶点的第二三维坐标进行坐标转换，得到M个第一靶点的第一三维坐标和N个第二靶点的第一三维坐标，其中，第一三维坐标为第一坐标系下的坐标，第一坐标系为以屏幕的一个边界点为原点、以屏幕的竖向为第一坐标轴、以屏幕的横向为第二坐标轴、以屏幕的垂直方向为第三坐标轴的坐标系；M个第一靶点的第一三维坐标和N个第二靶点的第一三维坐标用于摄像头的硬件在环标定。
在一种可能的实现方式中，在对M个第一靶点的第二三维坐标和N个第二靶点的第二三维坐标进行坐标转换，得到M个第一靶点的第一三维坐标和N个第二靶点的第一三维坐标方面，设置单元，具体用于：根据M个第一靶点的第二三维坐标和M个第一靶点的第二三维坐标，得到M个第一特征向量和M个第二特征向量；根据M个第一特征向量和M个第二特征向量采用最小二乘法计算得到中间向量；根据中间向量得到控制矩阵，以及根据控制矩阵得到旋转向量和平移向量；根据旋转向量和平移向量对M个第一靶点的第二三维坐标和N个第二靶点的第二三维坐标进行坐标转换，得到M个第一靶点的第一三维坐标和N个第二靶点的第一三维坐标。
在一种可能的实现方式中，标定图像的高宽比、屏幕的高宽比与显示设备的高宽比相同，设置单元，还用于：对显示标定图像的显示设备进行圆心检测，以得到M个第一靶点中的每个第一靶点的像素坐标；将M个第一靶点中的每个第一靶点的像素坐标转换到第二坐标系下，以得到M个第一靶点中的每个第一靶点的二维坐标，其中，第二坐标系为以显示设备的屏幕边框的一个外边界点为原点、以屏幕边框的竖向为第一坐标轴、以屏幕边框的横向为第二坐标轴的坐标系；根据M个第一靶点中的至少一个第一靶点的二维坐标、屏幕的高宽比与显示设备的高宽比确定N个第二靶点在第二坐标系下的二维坐标。
在一种可能的实现方式中，标定图像的尺寸与屏幕边框的外边界尺寸存在比例关系，设置单元，还用于：根据标定图像的尺寸与屏幕边框的外边界尺寸存在的比例关系，对标定图像进行放大处理，以得到放大后的标定图像，其中，放大后的标定图像的尺寸与屏幕边框的外边界尺寸相等；将放大后的标定图像投影到第二坐标系上，其中，第二坐标系为以显示设备的屏幕边框的一个外边界点为原点、以屏幕边框的竖向为第一坐标轴、以屏幕边框的横向为第二坐标轴的坐标系；将放大后的标定图像中与靶点中心位置重合的像素坐标作为该靶点的二维坐标，以得到M个第一靶点的二维坐标和N个第二靶点的二维坐标。
本申请实施例第五方面公开了一种摄像头的硬件在环标定系统，该系统包括电子设备 和显示设备，该电子设备包括该摄像头，或所该电子设备与该摄像头通信连接，该显示设备包括屏幕和屏幕边框，屏幕内设置有M个第一靶点，或屏幕显示有M个第一靶点；屏幕边框上设置有N个第二靶点；其中，M个第一靶点与N个第二靶点不共面，M和N为大于或等于3的整数。在本申请实施例中，在显示设备的屏幕内设置M个第一靶点，以及在显示设备的屏幕边框上设置N个第二靶点，且该M个第一靶点与该N个第二靶点是不共面的，因此在标定时无需使用多块不共面的标定板，解决了摄像头的硬件在环系统中靶点设置空间受限的问题。
本申请实施例第六方面公开了一种电子设备，包括处理器、存储器、通信接口，以及一个或多个程序，上述一个或多个程序被存储在上述存储器中，并且被配置由上述处理器执行，上述程序包括用于执行如上述第一方面或第二方面中任一项所述的方法中的步骤的指令。
本申请实施例第七方面公开了一种芯片，包括：处理器，用于从存储器中调用并运行计算机程序，使得安装有所述芯片的设备执行如上述第一方面或第二方面中任一项所述的方法。
本申请实施例第八方面公开了一种计算机可读存储介质，其存储用于电子数据交换的计算机程序，其中，所述计算机程序使得计算机执行如上述第一方面或第二方面中任一项所述的方法。
本申请实施例第九方面公开了一种计算机程序，所述计算机程序使得计算机执行如上述第一方面或第二方面中任一项所述的方法。
附图说明
图1是本申请实施例提供的一种相机的重点关注区域的示意图。
图2是本申请实施例提供的一种摄像头的硬件在环标定系统的架构示意图。
图3是本申请实施例提供的一种屏幕靶点的示意图。
图4是本申请实施例提供的一种屏幕边框靶点的示意图。
图5是本申请实施例提供的一种摄像头的硬件在环标定靶点设置方法的流程示意图。
图6是本申请实施例提供的一种第一坐标系的示意图。
图7是本申请实施例提供的一种第一坐标系和第二坐标系的示意图。
图8是本申请实施例提供的一种摄像头的硬件在环标定方法的流程示意图。
图9是本申请实施例提供的一种角点检测的示意图。
图10是本申请实施例提供的一种基于重点关注区域的最优位姿求解的流程示意图。
图11是本申请实施例提供的另一种摄像头的硬件在环标定方法的流程示意图。
图12是本申请实施例提供的又一种摄像头的硬件在环标定方法的流程示意图。
图13是本申请实施例提供的一种误差地图的生成原理的示意图。
图14是本申请实施例提供的一种摄像头的硬件在环标定靶点设置装置的结构示意图。
图15是本申请实施例提供的一种摄像头的硬件在环标定装置的结构示意图。
图16是本申请实施例提供的一种电子设备的结构示意图。
具体实施方式
为了使本技术领域的人员更好地理解本申请方案，下面将结合本申请实施例中的附图，对本申请实施例中的技术方案进行清楚、完整地描述，显然，所描述的实施例仅仅是本申请一部分实施例，而不是全部的实施例。基于本申请中的实施例，本领域普通技术人员在没有作出创造性劳动前提下所获得的所有其他实施例，都属于本申请保护的范围。
本申请的说明书和权利要求书及上述附图中的术语“包括”和“具有”以及它们任何变形，意图在于覆盖不排他的包含。例如包含了一系列步骤或单元的过程、方法、系统、产品或设备没有限定于已列出的步骤或单元，而是可选地还包括没有列出的步骤或单元，或可选地还包括对于这些过程、方法、产品或设备固有的其他步骤或单元。
在本说明书中提及“实施例”意味着，结合实施例描述的特定特征、结构或特性可以包含在本申请的至少一个实施例中。在说明书中的各个位置出现该短语并不一定均是指相同的实施例，也不是与其它实施例互斥的独立的或备选的实施例。本领域技术人员显式地和隐式地理解的是，本说明书所描述的实施例可以与其它实施例相结合。
为了便于理解本申请实施例，进一步分析并提出本申请所具体要解决的技术问题。目前的相机在环标定方法，通过使得相机的中心对准LED显示屏的中心线达到相机对准的作用；或者通过调节相机的位置，使得相机可以完整拍摄整个屏幕，以此为依据完成相机对准。例如，通过金属支架固定和调节相机，使相机刚好拍摄到全部屏幕，可通过移动金属支架匹配不同视场角(field of view，FOV)的相机，保证相机拍摄图像刚好是完整的屏幕图像，其本质上是将相机的中心垂直对准屏幕中心，通过调节相机中心到屏幕中心的距离实现不同FOV的相机的硬件在环。
可以看出，上述方法的基本思想均是通过调节相机，使得相机对准屏幕中心，然后以是否完整拍摄到整个屏幕进行相机调节，达到对准的效果。由于相机中心是和相机的内参相关的，对准屏幕中心的评价标准很难把握，而且垂直对准屏幕也只能大概估计，精度较低。除此之外，在调节相机过程中，由于相机存在图像畸变，消除畸变的校正模型与实际模型不可能完全一致，因此完全精确的拍摄到整个屏幕几乎不可能做到。如何调节到最优的位置没有一个量化的评价标准，导致在实际调节的时候会极大影响调节效率和精度。另外，不同相机的功能各异，对于关键区域的像素匹配应该更加精确，上述对准方案没有考虑到与具体相机的功能匹配，匹配精度通常相对较低。
因此，本申请所要解决的技术问题可以包括如下：
1)相机标定反馈方法可以实时反馈相机的当前位置和姿态，本申请实施例将其用于相机硬件在环的对准中，主要目的是求取相机的当前位姿与需要调整到的目标位姿的误差，从而可以定量调节相机位姿，提升调节效率和精度；而相机的位姿求解算法需要设置不共面的靶点，现有技术是使用两块不共面的标定板实现靶点不共面，这在相机在环对准中不 允许的，如何设置相机在环对准的靶点成为本申请的需要的解决的技术问题。
2)相机的功能决定了相机可能的感兴趣区域。如图1所示，识别红绿灯的相机重点关注图像的中上区域，识别车道线的相机重点关注图像的中下区域。如何调整相机的位姿，使得相机在感兴趣区域中的对准精度更高，成为本申请又一重点需要的解决的问题。
本申请实施例提供的技术方案可以应用于但不仅限于如下场景：当HIL系统出厂时，需要进行相机的对准调教；当HIL系统在搬运、使用过程中发生相机位姿改变之后，需要进行相机的位姿对准，以实现相机拍摄图像和实际图像的一致性，保证HIL的功能正常使用；除此之外，当相机的功能发生改变，用户的重点关注区域发生改变后，需要进行相机的位姿调整，以期满足对应感知算法的精度要求。
下面结合具体实施方式对本申请提供的技术方案进行详细的介绍。
请参阅图2，图2是本申请实施例提供的一种摄像头的硬件在环标定系统的架构示意图。如图2所示，该系统包括电子设备10和显示设备20，该电子设备10包括摄像头101，或该电子设备10与摄像头101通信连接，该显示设备20包括屏幕201和屏幕边框202，屏幕201内设置有M个第一靶点，或屏幕201显示有M个第一靶点；屏幕边框202上设置有N个第二靶点；其中，M个第一靶点与N个第二靶点不共面，M和N为大于或等于3的整数。
其中，屏幕201用于在进行摄像头101的硬件在环标定时显示标定图像，屏幕201内设置的第一靶点或屏幕201显示的第一靶点如图3所示，靶点的含义为用于标定的关键空间点。应理解，为使所有的第一靶点与所有的第二靶点不共面，可以使屏幕201和屏幕边框202不共面，例如屏幕边框202相对于屏幕201是往外突出的；或者屏幕201和屏幕边框202是共面的，屏幕边框202上设置的第二靶点是凸台靶点，凸台靶点如图4所示。
应理解，电子设备10可以为智能摄像头、相机、手机、带有摄像功能的电子设备或带摄像头的电子设备等；或者，电子设备10与摄像头101通信连接，电子设备10可以通过摄像头101采集图像，例如电子设备10通过摄像头101对显示设备20进行图像采集；本申请对此不作具体限定。电子设备10可以与显示设备20通信连接，电子设备10可以控制显示设备20显示标定图像，例如电子设备10向显示设备20发送标定图像，指示显示设备20显示该标定图像。
其中，通过该摄像头的硬件在环标定系统进行摄像头的硬件在环标定过程如下：首先，获取所有靶点中的每个靶点的二维坐标和三维坐标；然后，在显示设备20的屏幕201上显示标定图像，电子设备10通过摄像头101对显示设备20进行图像采集，并且需要将显示设备20完全拍摄进摄像头101采集到的图像中，也即摄像头101采集到的图像中需要完整的包括该显示设备20，故摄像头101采集到的图像中包括了所有的靶点；再获取所有靶点中的每个靶点在摄像头101采集到的图像中的像素坐标；最后，依据每个靶点的二维坐标和三维坐标计算得到摄像头101需要调整到的位姿，依据每个靶点在摄像头101采集到的图像中的像素坐标、每个靶点的三维坐标计算得到摄像头101的当前位姿，从而将摄像头101从当前位姿向需要调整到的位姿进行调节，如此实现摄像头的硬件在环标定。
在本申请实施例中，在显示设备20的屏幕内设置M个第一靶点，以及在显示设备20 的屏幕边框上设置N个第二靶点，且该M个第一靶点与该N个第二靶点是不共面的，因此在标定时无需使用多块不共面的标定板，解决了摄像头101的硬件在环系统中靶点设置空间受限的问题。
请参阅图5，图5是本申请实施例提供的一种摄像头的硬件在环标定靶点设置方法的流程示意图，该方法可以应用于图2所示的摄像头的硬件在环标定系统，该方法由电子设备执行，该电子设备包括该摄像头，或该电子设备与该摄像头通信连接，该方法包括但不限于如下步骤：
步骤501、在显示设备的屏幕内设置M个第一靶点，以及在显示设备的屏幕边框上设置N个第二靶点，其中，显示设备用于在进行摄像头的硬件在环标定时显示标定图像，M个第一靶点与N个第二靶点不共面，M和N为大于或等于3的整数。
其中，该摄像头可以应用于图2所示的摄像头，该显示设备可以是图2所示的显示设备；屏幕内的M个第一靶点可以是由屏幕显示的虚拟靶点。
应理解，相机的标定算法需要用到不共面的靶点，为此在用于摄像头的硬件在环标定的显示设备的屏幕内设置M个第一靶点，以及在显示设备的屏幕边框上设置N个第二靶点，且这M个第一靶点和N个第二靶点是不共面的，具体如图3和图4所示。
在本申请实施例中，在显示设备的屏幕内设置M个第一靶点，以及在显示设备的屏幕边框上设置N个第二靶点，且该M个第一靶点与该N个第二靶点是不共面的，因此在标定时无需使用多块不共面的标定板，解决了摄像头的硬件在环系统中靶点设置空间受限的问题。
需要说明的是，在进行摄像头的硬件在环标定过程中，需要用到靶点的三维坐标进行位姿计算，故在设置完靶点后，需要获取每个靶点的三维坐标，并且将每个靶点的三维坐标保存，供摄像头的硬件在环标定使用。其中，每个靶点的三维坐标可以通过全站仪测量得到。但由于全站仪摆放的随机性，会导致靶点在全站仪坐标系下的第二三维坐标的实际方向是随机的，而相机标定算法给定的调节自由度是全站仪坐标系下的自由度，这往往需要通过尝试才能明确应该调节的方向，不利于用户使用。因此，可以将所有靶点在全站仪坐标系下的第二三维坐标转换到第一坐标系(也即屏幕坐标系)下，得到所有靶点在第一坐标系下的第一三维坐标，并且保存所有靶点的第一三维坐标，供摄像头的硬件在环标定使用。
为此，本申请实施例定义了第一坐标系，该第一坐标系为以屏幕的一个边界点为原点、以屏幕的竖向为第一坐标轴、以屏幕的横向为第二坐标轴、以屏幕的垂直方向为第三坐标轴的坐标系。
请参阅图6，图6是本申请实施例提供的一种第一坐标系的示意图。如图6所示，该第一坐标系以屏幕的左上角o点为原点，以屏幕的横向为y轴，以屏幕的竖向为x轴，以屏幕的垂直向外为z轴。
在一种可能的实现方式中，该方法还包括：获取M个第一靶点的第二三维坐标和N个第二靶点的第二三维坐标，其中，第二三维坐标为全站仪坐标系下的坐标；对M个第一靶点的第二三维坐标和N个第二靶点的第二三维坐标进行坐标转换，得到M个第一靶点的第一三维坐标和N个第二靶点的第一三维坐标，其中，第一三维坐标为第一坐标系下的坐标，第 一坐标系为以屏幕的一个边界点为原点、以屏幕的竖向为第一坐标轴、以屏幕的横向为第二坐标轴、以屏幕的垂直方向为第三坐标轴的坐标系；M个第一靶点的第一三维坐标和N个第二靶点的第一三维坐标用于摄像头的硬件在环标定。
具体地，所有靶点的第一三维坐标可以通过以下步骤获取：先通过全站仪对靶点进行测量，以获得靶点在全站仪坐标系下的第二三维坐标，再将靶点在全站仪坐标系下的第二三维坐标转换到第一坐标系，得到靶点的第一三维坐标。
举例来说，结合图3和图4来说明如何获取所有靶点的第一三维坐标，首先只需通过全站仪打出屏幕上的4个边界靶点(也即图3中4个角上的靶点)、屏幕边框上的4个靶点(也即图4中屏幕边缘设置的4个凸台靶点)的第二三维坐标即可，然后其他靶点的第二三维坐标可以通过像素关系差值得到；获取到所有靶点的第二三维坐标后，将其转换到第一坐标系下，得到所有靶点的第一三维坐标。
在本申请实施例中，通过坐标转换，将靶点在全站仪坐标系下的坐标转换到第一坐标系下，靶点在第一坐标系下的第一三维坐标可以供摄像头的硬件在环标定使用，从而为后续的摄像头的硬件在环标定提供基础，有利于提升摄像头对准效果。
在一种可能的实现方式中，对M个第一靶点的第二三维坐标和N个第二靶点的第二三维坐标进行坐标转换，得到M个第一靶点的第一三维坐标和N个第二靶点的第一三维坐标，包括：根据M个第一靶点的第二三维坐标和M个第一靶点的第二三维坐标，得到M个第一特征向量和M个第二特征向量；根据M个第一特征向量和M个第二特征向量采用最小二乘法计算得到中间向量；根据中间向量得到控制矩阵，以及根据控制矩阵得到旋转向量和平移向量；根据旋转向量和平移向量对M个第一靶点的第二三维坐标和N个第二靶点的第二三维坐标进行坐标转换，得到M个第一靶点的第一三维坐标和N个第二靶点的第一三维坐标。
其中，以屏幕内的第一靶点作为参考点，求解全站仪坐标系到第一坐标系的旋转矩阵R和平移向量T，根据旋转矩阵R和平移向量T将靶点在全站仪坐标系下的第二三维坐标转换成在第一坐标系下的第一三维坐标。具体包括如下步骤：
1)将屏幕内的所有的第一靶点在全站仪坐标系下的第二三维坐标进行两两相减，以及将屏幕内的所有的第一靶点在第一坐标系下的第一三维坐标进行两两相减，可得到系列对应的特征向量X i、Y i，其中，X i表示第一坐标系下的第i个向量，Y i表示全站仪坐标系下与X i对应的向量。具体地，根据公式(1)求解特征向量。
Figure PCTCN2020141855-appb-000001
2)根据特征向量X i、Y i，采用最小二乘法计算得到中间向量p，其中，根据公式(2)和(3)计算中间向量p。
Figure PCTCN2020141855-appb-000002
其中，公式(2)所示的方程组为超定方程，因此p可以采用最小二乘法通过公式(3)进行求解。
p＝(V TV) -1V TH  (3)
其中，公式(3)可以表示为VP＝H，公式(3)中的V、H与公式(2)对应。
3)中间向量p得到控制矩阵S：
Figure PCTCN2020141855-appb-000003
4)根据控制矩阵S可得到全站仪坐标系向第一坐标系转换的旋转向量R和平移向量T，其中，旋转向量R和平移向量T的计算公式如公式(5)所示。
Figure PCTCN2020141855-appb-000004
5)得到旋转向量R和平移向量T后，可将全站仪坐标系下的所有坐标均转换到第一坐标系下，其计算公式如公式(6)所示。
X＝RY+T (6)
在公式(6)中，Y表示靶点在全站仪坐标系下的第二三维坐标，X表示靶点在第一坐标系下的第一三维坐标。
需要指出的是，该过程只需在出厂时完成一次即可，靶点在第一坐标系下的第一三维坐标X应保存成文件供摄像头的硬件在环标定使用。
在本申请实施例中，以屏幕内的第一靶点作为参考点，求解全站仪坐标系到第一坐标系的旋转矩阵和平移向量；再依据该旋转矩阵和平移向量将靶点在全站仪坐标系下的坐标转换到第一坐标系下，并将靶点在第一坐标系下的第一三维坐标供摄像头的硬件在环标定使用，从而为后续的摄像头的硬件在环标定提供基础，有利于提升摄像头对准效果。
需要说明的是，在进行摄像头的硬件在环标定过程中，还需要用到靶点的二维坐标进行位姿计算，故在设置完靶点后，还需要获取到每个靶点的二维坐标。具体地，屏幕内的靶点的二维坐标可以通过以下步骤获取：先通过对显示标定图像的屏幕进行圆心检测以获取到靶点的像素坐标，再将靶点的像素坐标转换到第二坐标系(也即屏幕边框坐标系)下，得到屏幕内的靶点的二维坐标；其中，圆心检测是图像检测中一种检测圆心的算法。而屏幕边框上的靶点的二维坐标可以根据屏幕内的靶点的二维坐标推算出。
为此，本申请实施例定义了第二坐标系，第二坐标系为以显示设备的屏幕边框的一个外边界点为原点、以屏幕边框的竖向为第一坐标轴、以屏幕边框的横向为第二坐标轴的坐标系。
请参阅图7，图7是本申请实施例提供的一种第一坐标系和第二坐标系的示意图。如图7所示，该第二坐标系以屏幕边框的左上角o’点为原点，以屏幕边框的横向为y’轴，以屏幕边框的竖向为x’轴。
在一种可能的实现方式中，标定图像的高宽比、屏幕的高宽比与显示设备的高宽比相同，该方法还包括：对显示标定图像的显示设备进行圆心检测，以得到M个第一靶点中的 每个第一靶点的像素坐标；将M个第一靶点中的每个第一靶点的像素坐标转换到第二坐标系下，以得到M个第一靶点中的每个第一靶点的二维坐标，其中，第二坐标系为以显示设备的屏幕边框的一个外边界点为原点、以屏幕边框的竖向为第一坐标轴、以屏幕边框的横向为第二坐标轴的坐标系；根据M个第一靶点中的至少一个第一靶点的二维坐标、屏幕的高宽比与显示设备的高宽比确定N个第二靶点在第二坐标系下的二维坐标。
应理解，标定图像的高宽比、屏幕的高宽比相同，也即标定图像像素大小和屏幕的像素大小是一致的，当标定图像显示在屏幕上时，标定图像的像素可以铺满整个屏幕的像素。其中，标定图像的高宽比、屏幕的高宽比与显示设备的高宽比相同，也即标定图像的尺寸、屏幕的尺寸与显示设备的尺寸存在比例关系。
举例来说，标定图像的像素是1920×1080，屏幕的分辨率为1080P(也即1920×1080)，将标定图像显示在屏幕上时，屏幕上的像素全部被铺满，对当前显示该标定图像的屏幕进行圆心检测，从而可以得到所有第一靶点的像素坐标；而显示设备的屏幕宽高比与显示设备(或者说屏幕边框的外边界)的宽高比相同，例如显示设备的屏幕尺寸与显示设备(或者说屏幕边框的外边界)的尺寸的比例为1:1.2，且第二靶点在屏幕边框上的位置是固定的，可以通过数学模型将第一靶点的像素坐标转换成第二坐标系下的二维坐标，并且根据第一靶点的二维坐标推算出第二靶点的二维坐标。下面结合图7来举例说明，如何根据第一靶点的二维坐标推算出第二靶点的二维坐标，根据屏幕内左上角的第一靶点的二维坐标、屏幕的高宽比与显示设备的高宽比推算出屏幕边框左上角的第二靶点的二维坐标；由于屏幕边框左下角、右上角、右下角的第二靶点与屏幕边框左上角的第二靶点在屏幕边框上的相对位置是确定的，从而可以根据屏幕边框左上角的第二靶点的二维坐标推算出屏幕边框左下角、右上角、右下角的第二靶点的二维坐标。
在本申请实施例中，先通过圆心检测获取第一靶点的像素坐标；然后通过将第一靶点的像素坐标转换到第二坐标系下，得到第一靶点的二维坐标；再依据第一靶点的二维坐标推算出第二靶点的二维坐标，从而可以得到用于摄像头的硬件在环标定的所有靶点的二维坐标。
在一种可能的实现方式中，标定图像的尺寸与屏幕边框的外边界尺寸存在比例关系，该方法还包括：根据标定图像的尺寸与屏幕边框的外边界尺寸存在的比例关系，对标定图像进行放大处理，以得到放大后的标定图像，其中，放大后的标定图像的尺寸与屏幕边框的外边界尺寸相等；将放大后的标定图像投影到第二坐标系上，其中，第二坐标系为以显示设备的屏幕边框的一个外边界点为原点、以屏幕边框的竖向为第一坐标轴、以屏幕边框的横向为第二坐标轴的坐标系；将放大后的标定图像中与靶点中心位置重合的像素坐标作为该靶点的二维坐标，以得到M个第一靶点的二维坐标和N个第二靶点的二维坐标。
应理解，标定图像的尺寸与屏幕边框的外边界尺寸存在比例关系，也即标定图像的高宽比与屏幕边框的外边界的高宽比相同。
在本申请实施例中，通过将标定图像放大后投影到第二坐标系上，并将放大后的标定图像中与靶点中心位置重合的像素坐标作为该靶点的二维坐标，从而可以得到用于摄像头的硬件在环标定的所有靶点的二维坐标。
请参阅图8，图8是本申请实施例提供的一种摄像头的硬件在环标定方法的流程示意图，该方法可以应用于图2所示的摄像头的硬件在环标定系统，该方法由电子设备执行，该电子设备包括该摄像头，或该电子设备与该摄像头通信连接，该方法包括但不限于如下步骤：
步骤801、确定多个靶点中的每个靶点对应的权重，其中，多个靶点为显示设备上的靶点，显示设备上位于目标区域内的靶点对应的权重大于位于目标区域外的靶点对应的权重，显示设备的目标区域用于显示标定图像的一部分；标定图像为在进行摄像头的硬件在环标定时显示设备显示的图像。
其中，靶点对应的权重的数学含义表示摄像头的重点关注区域相对于其他区域的重要程度；显示设备的目标区域显示的标定图像的一部分，该标定图像的一部分为该摄像头的重点关注区域；摄像头的重点关注区域可以由电子设备自动识别并框选确定，然后电子设备为摄像头的重点关注区域对应的靶点赋予较高的权重；摄像头的重点关注区域也可以由用户手动框选确定，例如提供用户框选接口，当用户选择重点区域后，电子设备自动将框选区域对应的靶点赋予较高的权重。此外，权重可根据用户需要进行定义。
步骤802、根据权重对多个靶点进行P次有放回的抽样，以得到P个靶点组，其中，针对每次抽样，靶点对应的权重越大，该靶点被抽取的概率越大，P为正整数。
具体地，所有靶点的权重确定之后，利用重采样的思想，对不同权重的靶点进行有放回的抽样以得到P个靶点组。其中，在进行有放回的抽样时，可以采用自展法(bootstrap)。
步骤803、根据P个靶点组和摄像头的当前位姿进行摄像头的硬件在环标定。
具体地，可以确定P个靶点组中的任意一个靶点组中的每个靶点的二维坐标和三维坐标，从而得到所有靶点的二维坐标和三维坐标，并根据这些靶点的二维坐标和三维坐标计算得到摄像头需要调整到的位姿，然后以该需要调整到的位姿为方向，从摄像头的当前位姿调节至该需要调整到的位姿，如此实现该摄像头的硬件在环标定。
在本申请实施例中，给不同区域的靶点设置不同的权重，基于权重对靶点进行有放回的重采样既保证了各组靶点数据之间的独立性，又侧重提升了摄像头的重点关注区域对应的靶点被抽取的概率，根据采样得到的各组靶点和摄像头的当前位姿进行摄像头的硬件在环标定，可有效提升摄像头对重点关注区域的对准精度。具体地，用于摄像头的硬件在环标定的显示设备上有多个靶点，在该显示设备上显示标定图像，该标定图像有该摄像头的重点关注区域，该标定图像被显示在该显示设备上时，该显示设备上用于显示该重点关注区域的显示区域为目标区域，先设置该目标区域内的靶点对应的权重大于位于该目标区域外的靶点对应的权重；然后基于权重概率对多个靶点进行P次有放回的抽样以得到P个靶点组，也即靶点对应的权重越大，该靶点被抽取的概率越大；由于目标区域内的靶点的权重大于位于该目标区域外的靶点对应的权重，故每次抽样时，目标区域内的靶点被抽取的概率更大，目标区域内的靶点被抽取的数量相对就越多，在根据P个靶点组和摄像头的当前位姿进行摄像头的硬件在环标定时，目标区域内的靶点所起的决定作用占比就越大，摄像头对重点关注区域的对准精度就越高。综上，本申请实施例，可有效提升摄像头对重点关注区域的对准精度，保证用户的个性化对准需求。
在一种可能的实现方式中，根据P个靶点组和摄像头的当前位姿进行摄像头的硬件在 环标定，包括：根据P个靶点组确定P个第一目标位姿，其中，P个靶点组与P个第一目标位姿一一对应；通过摄像头对当前显示标定图像的显示设备进行图像采集以得到目标图像，并根据多个靶点和目标图像确定摄像头的当前位姿；根据P个第一目标位姿的平均值和当前位姿进行摄像头的硬件在环标定。
应理解，摄像头的硬件在环标定的目的是：摄像头采集的图像与实际的显示设备显示标定图像的画面完全一致。因此，目标图像的高宽比与显示设备的高宽比可以相同，如此通过摄像头对显示设备进行图像采集，可以实现摄像头刚好将整个显示设备拍摄进目标图像中，也即在目标图像中，显示设备的边界也即目标图像的边界；进一步地，目标图像的高宽比、标定图像的高宽比、屏幕的高宽比与显示设备的高宽比均可以相同。
其中，可以将P个第一目标位姿的平均值作为摄像头需要调整到的位姿，也即将P个第一目标位姿的平均值作为标定成功后摄像头应处的位姿。
在本申请实施例中，先根据又放回的重采样得到的P个靶点组确定P个第一目标位姿；然后通过摄像头对当前显示标定图像的显示设备进行图像采集以得到目标图像，并根据显示设备上的多个靶点和该目标图像确定摄像头的当前位姿；再将P个第一目标位姿的平均值作为需要调整到的位姿，进行该摄像头的硬件在环标定，也即将该摄像头的当前位姿调整至该P个第一目标位姿的平均值；由于利用重采样的方法多次计算摄像头的目标姿态，通过求平均的方法得到需要调整到的位姿，该需要调整到的位姿不仅为摄像头的位姿调整方向提供参考标杆，同时着重保证了摄像头的重点关注区域的对准精度，有利于提升摄像头对准的标定效率、对准精度。
在一种可能的实现方式中，在根据P个靶点组确定P个第一目标位姿之前，该方法还包括：通过摄像头对标定板进行图像采集以得到Q张标定图像，其中，Q为正整数；对Q张标定图像中的每张标定图像进行角点检测以得到每张标定图像中的各个角点的像素坐标；根据Q张标定图像中的每张标定图像中的各个角点的像素坐标、每张标定图像中的各个角点的第三三维坐标得到摄像头的内参矩阵和摄像头的畸变系数，其中，第三三维坐标为标定板坐标系下的坐标。
应理解，位姿计算需要用到摄像头的内参矩阵和摄像头的畸变系数，因此在确定第一目标位姿之前，需要获取到摄像头的内参矩阵和摄像头的畸变系数；而摄像头的内参矩阵和摄像头的畸变系数可以通过摄像头采集图像，依据摄像头采集的图像来确定。
其中，角点检测是一种图像检测算法，主要是指检测标定板中的棋盘格黑白相间的角上的顶点，角点检测过程如图9所示。
其中，可以利用张氏标定法得到相机的畸变系数(包括K1、K2、D1、D2和K3，其中，K1、K2和K3为径向畸变系数，D1和D2为切向畸变系数)和内参矩阵(包括Fx、Fy、Cx和Cy，其中，Fx和Fy为焦距，Cx和Cy为光心)。张氏标定法的输入是很多张图像，例如50张，也即Q等于50；然后通过自身算法的迭代、计算，得到最终适用于所有图像的最优的畸变系数和内参矩阵，作为最终结果。
在本申请实施例中，通过摄像头对标定板进行图像采集以得到多张标定图像，同该多张标定图像得到摄像头的内参矩阵和摄像头的畸变系数，而标定过程中的位姿计算需要用到摄像头的内参矩阵和摄像头的畸变系数，从而可以为计算摄像头的需要调整到的位姿以 及摄像头的当前位姿提供基础，有利于提升摄像头的硬件在环对准精度。
在一种可能的实现方式中，根据P个靶点组确定P个第一目标位姿，包括：针对P个靶点组中的每个靶点组，执行以下步骤，得到P个第一目标位姿：获取目标靶点组中的每个靶点的二维坐标，其中，目标靶点组为P个靶点组中的任意一个，二维坐标为第二坐标系下的坐标，第二坐标系为以显示设备的屏幕边框的一个外边界点为原点、以屏幕边框的竖向为第一坐标轴、以屏幕边框的横向为第二坐标轴的坐标系；根据目标靶点组中的每个靶点的二维坐标、第一三维坐标、摄像头的内参矩阵和摄像头的畸变系数，计算得到目标靶点组对应的第一目标位姿，其中，第一三维坐标为第一坐标系下的坐标，第一坐标系为以显示设备的屏幕的一个边界点为原点、以屏幕的竖向为第一坐标轴、以屏幕的横向为第二坐标轴、以屏幕的垂直方向为第三坐标轴的坐标系。
请参阅图10，图10是本申请实施例提供的一种基于重点关注区域的最优位姿求解的流程示意图。如图10所示，先选择摄像头的重点关注区域；在选择摄像头的重点关注区域之后，算法自动给靶点对应的权重赋值，其中，重点关注区域内的靶点赋予较高的权重；所有靶点的权重确定之后，利用重采样的思想，基于权重概率对不同权重的靶点进行有放回的bootstrap抽样，得到n组抽样结果；以各组抽样结果为输入，也即以各组抽样结果中的靶点的二维坐标和第一三维坐标以及摄像头的内参矩阵和摄像头的畸变系数作为输入，利用UPNP算法，进行位姿求解，得到一系列的第一目标位姿，也即n个第一目标位姿；对该n个第一目标位姿进行求平均，可以得到第二目标位姿，也即得到摄像头需要调整到的位姿。
需要指出的，基于权重概率的有放回重采样既保证了各组靶点数据之间的独立性，又侧重提升了重点关注区域的靶点出现的概率，可有效提升摄像头对重点关注区域的对准精度。UPNP(unified Perspective-n-Point)算法是N点透视(Perspective-n-Point，PNP)算法的改进，其是一种现有的相机标定算法；在进行位姿求解时，除采用UPNP算法外，也可以采用PNP算法及其所有改进算法都可以使用，例如EPNP算法等，其中，EPNP(efficient Perspective-n-Point)算法也是PNP算法的改进。
在本申请实施例中，根据目标靶点组中的每个靶点的二维坐标、第一三维坐标、摄像头的内参矩阵和摄像头的畸变系数，计算得到目标靶点组对应的第一目标位姿，其中，该二维坐标为第二坐标系下的坐标，该第一三维坐标为第一坐标系下的坐标；由于显示设备的屏幕和屏幕边框的尺寸是固定的，当屏幕上显示有标定图像时，可以根据标定图像的像素坐标推算得到每个靶点在第二坐标系上的二维坐标；并且可以通过全站仪获取每个靶点在全站仪坐标系下的三维坐标，再将每个靶点在全站仪坐标系下的三维坐标转换成在第一坐标系下的三维坐标；从而有利于得到每个靶点组的第一目标位姿，进而得到摄像头需要调整到的位姿。
在一种可能的实现方式中，根据多个靶点和目标图像确定摄像头的当前位姿，包括：对目标图像进行圆心检测以得到多个靶点中的每个靶点在目标图像中的像素坐标；根据多个靶点中的每个靶点在目标图像中的像素坐标、多个靶点中的每个靶点的第一三维坐标、摄像头的内参矩阵、摄像头的畸变系数，计算得到当前位姿，其中，第一三维坐标为第一坐标系下的坐标，第一坐标系为以显示设备的屏幕的一个边界点为原点、以屏幕的竖向为 第一坐标轴、以屏幕的横向为第二坐标轴、以屏幕的垂直方向为第三坐标轴的坐标系。
应理解，计算摄像头的当前位姿和计算第一目标位姿的过程相同，两者的区别在于输入不同，计算第一目标位姿的输入为每个靶点的二维坐标、第一三维坐标、摄像头的内参矩阵和摄像头的畸变系数，计算当前位姿的输入为靶点在目标图像中的像素坐标、靶点的第一三维坐标、摄像头的内参矩阵、摄像头的畸变系数。
在本申请实施例中，通过摄像头对当前显示标定图像的显示设备进行图像采集，得到目标图像后，可以通过对目标图像进行圆心检测，获得显示设备上的多个靶点中的每个靶点在目标图像中的像素坐标，从而有利于根据每个靶点在目标图像中的像素坐标、每个靶点的第一三维坐标、摄像头的内参矩阵、摄像头的畸变系数，计算得到当前位姿。
在一种可能的实现方式中，多个靶点包括显示设备的屏幕内的M个第一靶点和屏幕边框上的N个第二靶点，其中，M个第一靶点与N个第二靶点不共面，M和N为大于或等于3的整数，在根据P个靶点组和摄像头的当前位姿进行摄像头的硬件在环标定之前，该方法还包括：获取M个第一靶点的第二三维坐标和N个第二靶点的第二三维坐标，其中，第二三维坐标为全站仪坐标系下的坐标；对M个第一靶点的第二三维坐标进行坐标转换以得到M个第一靶点的第一三维坐标，以及对N个第二靶点的第二三维坐标进行坐标转换以得到N个第二靶点的第一三维坐标。
在本申请实施例中，在显示设备的屏幕内设置M个第一靶点，以及在显示设备的屏幕边框上设置N个第二靶点，且该M个第一靶点与该N个第二靶点是不共面的，因此在标定时无需使用多块不共面的标定板，解决了摄像头的硬件在环系统中靶点设置空间受限的问题；并且通过坐标转换，将靶点在全站仪坐标系下的坐标转换到第一坐标系下，为后续的标定反馈提示奠定基础，是提升摄像头对准效果的关键步骤。
在一种可能的实现方式中，根据P个第一目标位姿的平均值和当前位姿进行摄像头的硬件在环标定，包括：将第二目标位姿与当前位姿作差以得到当前位姿残差，其中，第二目标位姿为P个第一目标位姿的平均值，当前位姿残差包括3个当前位置自由度残差和3个当前姿态自由度残差；先按照当前位置自由度残差从大到小的顺序依次调节摄像头的各个位置自由度，以使摄像头的第一位置自由度与第二目标位姿的第一位置自由度的差值小于预设的第一位置自由度残差阈值、摄像头的第二位置自由度与第二目标位姿的第二位置自由度的差值小于预设的第二位置自由度残差阈值、摄像头的第三位置自由度与第二目标位姿的第三位置自由度的差值小于预设的第三位置自由度残差阈值；再按照当前姿态自由度残差从大到小的顺序依次调节摄像头的各个姿态自由度，以使摄像头的第一姿态自由度与第二目标位姿的第一姿态自由度的差值小于预设的第一姿态自由度残差阈值、摄像头的第二姿态自由度与第二目标位姿的第二姿态自由度的差值小于预设的第二姿态自由度残差阈值、摄像头的第三姿态自由度与第二目标位姿的第三姿态自由度的差值小于预设的第三姿态自由度残差阈值。
应理解，位姿包括3个位置自由度和3个姿态自由度，那么位姿残差包括3个位置自由度残差和3个姿态自由度残差。而对于标定来说，有的自由度对对准精度影响较大，该自由度对应的自由度残差阈值应尽可能小；有的自由度对对准精度影响较小，该自由度对应的自由度残差阈值相对可以大些。因此，每个自由度对应的自由度残差阈值可能不同， 从而有3个位置自由度一一对应3个位置自由度残差阈值，以及3个姿态自由度一一对应3个姿态自由度残差阈值。标定过程也即依次调节摄像头的各个自由度，使其各个自由度对应的自由度残差小于该自由度对应的自由度残差阈值。
实践表明，在进行位姿调整时，先调节摄像头的位置，再调节摄像头的姿态，可提高调节效率，因此，在计算得到第二目标位姿和当前位姿后，先反馈位置中残差较大的分量，提示向对应方向调节某一数值，待调节好3个位置自由度后，提示调节姿态分量中较大的分量，直至6个自由度全部调节完毕。
在本申请实施例中，在将摄像头的姿态从当前位姿调节到需要调整到的位姿时，先反馈位置中残差较大的分量，对该残差较大的分量向对应方向调节某一数值；后反馈位置中残差较小的分量，对该残差较小的分量向对应方向调节某一数值；待调节好3个位置自由度后，再调节姿态分量中较大的分量，且按照残差从大到小依次调节各姿态分量；实践表明，先调节位置，再调节姿态，可提高调节效率。
请参阅图11，图11是本申请实施例提供的另一种摄像头的硬件在环标定方法的流程示意图，该方法可以应用于图2所示的摄像头的硬件在环标定系统，该方法由电子设备执行，该电子设备包括该摄像头，或该电子设备与该摄像头通信连接，该方法包括但不限于如下步骤：
步骤1101、标定靶点设置。
其中，设置的靶点是不共面的，可以为摄像头的硬件在环标定算法提供所需的不共面靶点坐标信息。
步骤1102、靶点三维坐标计算。
具体地，通过全站仪打点及三维坐标转换，可得到所有靶点在第一坐标系下的三维坐标(也即第一三维坐标)，也即通过全站仪打点先得到靶点在全站仪坐标系下的三维坐标，再将靶点在全站仪坐标系下的三维坐标转换到第一坐标系下，得到所有靶点在第一坐标系下的三维坐标；所有靶点在第一坐标系下的三维坐标可作为摄像头的硬件在环标定算法(包括目标位姿的计算、当前位姿的计算)的输入。
步骤1103、摄像头内参计算。
步骤1104、靶点的二维坐标生成。
其中，由于摄像头的硬件在环标定的目的是使摄像头拍摄的图像与实际的屏幕显示图像完全一致，设置屏幕靶点后，可以通过圆心检测得到靶点的二维坐标，靶点的二维坐标可作为摄像头的目标位姿计算的二维坐标输入。
步骤1105、重点关注区域选择。
具体地，显示设备显示有标定图像，电子设备可以自动框选标定图像中该摄像头的重点关注区域，或者提供用户框选接口，用户手动框选标定图像中该摄像头的重点关注区域。
步骤1106、摄像头的目标位姿计算。
其中，框选得到摄像头的重点关注区域后，执行摄像头的目标位姿计算。
具体地，对重点关注区域中的靶点赋予更高的权重，基于权重完成靶点bootstrap抽样分组，将每一组靶点的在第一坐标系下的三维坐标、二维坐标以及基于张氏标定法得到的 摄像头的内参信息(包括摄像头的内参矩阵和摄像头的畸变系数)带入UPNP算法，计算得到每一组靶点所对应的目标位姿(也即第一目标位姿)，将每一组靶点对应的目标位姿进行求平均，得到最终的摄像头的目标位姿(也即第二目标位姿)。
其中，所得的摄像头的目标位姿不仅为摄像头的位姿调整方向提供参考标杆，同时着重保证了重点关注区域的对准精度，对提升相机对准的标定效率、标定精度至关重要；也即可以提高重点关注区域的对准精度，满足用户需求，显著提升对准效率及对准精度。
步骤1107、摄像头的当前位姿计算。
具体地，得到摄像头的目标位姿后，调整摄像头的位置，调整过程中完成摄像头拍摄图像的圆心检测，得到靶点圆心在拍摄图像中的像素坐标，将靶点圆心在拍摄图像中的像素坐标、靶点的在第一坐标系下的三维坐标以及摄像头的内参信息带入UPNP算法，可以得到摄像头的当前位姿。
步骤1108、位姿残差是否大于阈值。
具体地，通过计算摄像头当前位姿与目标位姿的位姿残差，根据位姿残差可以明确应该调整的自由度方向及其大小。
其中，若位姿残差大于阈值，执行步骤1109；若位姿残差不大于阈值，执行步骤1111。
步骤1109、摄像头位姿调整。
步骤1110、靶点在摄像头拍摄图像中的像素坐标检测。
具体地，摄像头位姿调整后，再次通过摄像头拍摄显示设备，以得到新的拍摄图像，并完成新的拍摄图像的圆心检测，得到靶点圆心在新的拍摄图像中的像素坐标；并根据该靶点圆心在新的拍摄图像中的像素坐标重复执行步骤1107。
步骤1111、摄像头的误差地图生成。
具体地，当摄像头的当前位姿和目标位姿的残差在阈值之内后，进行摄像头误差地图的生成，摄像头误差地图生成后，电子设备评估该误差地图对ADAS算法的精度影响；或者将误差地图将反馈给用户，为用户提供全量精度信息，也即由用户评估该误差地图对ADAS算法的精度影响。
在图11所描述的摄像头的硬件在环标定方法中，提供了不共面的靶点，用于摄像头的硬件在环标定；在考虑了摄像头的重点关注区域的前提下，基于重采样思想及摄像头标定算法得到摄像头的目标位姿；在摄像头的硬件在环标定时，该目标位姿为摄像头的位姿调节提供参考标杆，可有效提升对准效率和对准精度。
请参阅图12，图12是本申请实施例提供的又一种摄像头的硬件在环标定方法的流程示意图，该方法可以应用于图2所示的摄像头的硬件在环标定系统，该方法由电子设备执行，该电子设备包括该摄像头，或该电子设备与该摄像头通信连接，该方法包括但不限于如下步骤：
步骤1201、不共面靶点设置。
步骤1202、全站仪打点。
步骤1203、靶点三维坐标计算。
其中，步骤1201-1203为靶点设置及坐标转换(①)：首先设置不共面的靶点，包括 显示设备的屏幕内的靶点、屏幕边框上的靶点；然后利用全站仪打点，获得靶点在全站仪坐标系下的三维坐标(也即第二三维坐标)；再通过坐标转换公式将靶点在全站仪坐标系下的三维坐标转换到第一坐标系下，得到靶点在第一坐标系下的三维坐标。
步骤1204、重点关注区域选择。
步骤1205、设置靶点对应的权重。
步骤1206、靶点bootstrap重采样。
步骤1207、目标位姿计算。
其中，步骤1204-1207为目标姿态计算(②)：首先在标定图像中框选摄像头的感兴趣区域，也即摄像头的重点关注区域；然后给靶点设置权重，其中，摄像头的重点关注区域对应的靶点的权重大于摄像头的非重点关注区域对应的靶点的权重；再通过bootstrap采样方法对靶点进行随机的有放回的采样，得到一系列的靶点组合；最后利用靶点在第一坐标系下的三维坐标及靶点的二维坐标，结合摄像头的内参得到每一组靶点对应的目标位姿，对所有组靶点对应的目标位姿求平均，得到摄像头的目标位姿。目标位姿的具体计算过程如下：
1)通过摄像头对棋盘格标定板进行不同角度的标定图像采集，得到一定数量的标定图像；其中，在进行标定图像采集时，标定板应该出现并尽量充满摄像头视野，也即标定板在标定图像中所占的比例应尽可能的大，且标定板的角度覆盖范围应该尽量大。从采集到的标定图像中选择一些标定图像，例如选用的标定图像数量大于50张，对这些选用的标定图像进行角点检测，得到选用的标定图像中的每张图像中的各个角点的像素坐标，结合各个角点在标定板坐标系下的三维坐标(也即第三三维坐标)，其中，标定板坐标系下的坐标值为定值)，利用张氏标定法可得到摄像头的内参，也即得到摄像头的畸变系数和内参矩阵。
2)得到摄像头的内参后，将摄像头的内参矩阵、摄像头的畸变系数、靶点在第一坐标系下的三维坐标、靶点的二维坐标带入UPNP算法即可得到摄像头的外参(即摄像头在第一坐标系下的位置和姿态)，也即摄像头的目标位姿。
步骤1208、圆心检测。
具体地，得到摄像头的目标位姿后，对摄像头当前拍摄所得的图像进行圆心检测，得到靶点在拍摄图像中的像素坐标。
步骤1209、当前位姿计算。
具体地，将靶点在拍摄图像中的像素坐标、靶点的在第一坐标系下的三维坐标、摄像头的内参矩阵、畸变系数带入UPNP算法，即可得到摄像头当前的位姿情况。
步骤1210、位姿残差计算。
具体地，将当前位姿(位置包括：x，y，z；姿态包括：yaw，pitch，roll)和目标位姿作差，得到各个分量的残差，各个分量的残差组成位姿残差。
步骤1211、位姿残差是否大于阈值。
其中，若位姿残差大于阈值，执行步骤1212；当位姿残差不大于阈值，执行步骤1214。
步骤1212、位姿调整方向提示。
具体地，依据位姿残差提示当前位姿的各个自由度需要调整的方向及调整的大小。
步骤1213、位姿调整。
其中，实践表明，先调节位置，再调节姿态可提高调节效率。因此，先反馈位置中残差较大的分量，提示向对应方向调节某一数值，然后调节位置中残差次大的分量，再调节位置中残差最小的分量；待调节好3个位置自由度后，提示调节姿态分量中较大的分量，然后调节姿态中残差次大的分量，再调节姿态中残差最小的分量。
步骤1214、误差地图生成。
其中，系统会预先设定好各个自由度对应的自由度残差阈值，当所有自由度的残差小于对应的自由度残差阈值后，计算当前位姿下的误差地图。
请参阅图13，图13是本申请实施例提供的一种误差地图的生成原理的示意图。如图13所示，电子设备通过控制模块控制显示设备在屏幕上播放一帧标定图像，并通过控制模块记录当前显示该标定图像的显示设备的靶点的二维坐标，以及通过检测模块通过圆心检测得到靶点在摄像头拍摄图像中的像素坐标，针对每个靶点，求取该靶点的二维坐标与该靶点在摄像头拍摄图像中的像素坐标的欧氏距离，即可得到该靶点的误差信息，从而得到当前播放的标定图像对应的所有靶点的误差信息；得到当前播放的标定图像对应的所有靶点的误差信息后，通过控制模块控制继续播放下一帧标定图像，重复上述操作，得到所有标定图像的像素误差信息，构成最终的误差地图。
步骤1215、误差地图是否满足要求。
其中，得到误差地图后，会判断该误差地图是否满足要求，如果该误差地图满足要求，则执行步骤1216，对准结束；如果该误差地图不满足要求，则执行步骤1217。
步骤1216、对准结束。
步骤1217、是否调整重点关注区域。
其中，如果该误差地图不满足要求，会提示是否调整摄像头的重点关注区域。如果调整了摄像头的重点关注区域，则返回②进行摄像头的重点关注区域选择，以及目标位姿计算；如果不调整摄像头的重点关注区域，则执行步骤1218。
步骤1218、阈值调整。
其中，如果不用调整摄像头的重点关注区域，则调整阈值，例如系统提供阈值调整窗口，让用户重新选择阈值；然后返回计算摄像头的目标位姿与当前位姿的残差是否满足阈值要求；如果不满足，则需要继续调节摄像头的位姿，直到摄像头的位姿与目标位姿的位姿残差的所有自由度符合设定的阈值。
需要指出的是，本申请提出的标定方法可以应用于电子设备，由电子设备自动标定，例如可设置6自由度电机，利用适当的控制算法即可控制调整摄像头的位姿，实现自动化对准；也可以通过人工手动调节摄像头的位姿。
在图12所描述的摄像头的硬件在环标定方法中，在显示设备的屏幕内和屏幕边框上设置靶点，可规避标定算法需要不共面靶点的问题；除此之外，由于靶点对准不可能兼顾所有像素，为解决不同摄像头对不同区域的对准精度的要求，给不同区域的靶点设置不同的权重，基于权重对靶点进行有放回的重采样既保证了各组靶点数据之间的独立性，又侧重提升了摄像头的重点关注区域对应的靶点被抽取的概率，根据采样得到的各组靶点计算摄像头的目标位姿，根据摄像头的目标位姿和摄像头的当前位姿进行摄像头的硬件在环标定， 可有效提升摄像头对重点关注区域的对准精度。
请参阅图14，图14是本申请实施例提供的一种摄像头的硬件在环标定靶点设置装置的结构示意图，该摄像头的硬件在环标定靶点设置装置1400可以应用于电子设备，该电子设备包括该摄像头，或该电子设备与该摄像头通信连接，该摄像头的硬件在环标定靶点设置装置1400可以包括设置单元1401，其中，各个单元的详细描述如下：
设置单元1401，用于在显示设备的屏幕内设置M个第一靶点，以及在显示设备的屏幕边框上设置N个第二靶点，其中，显示设备用于在进行摄像头的硬件在环标定时显示标定图像，M个第一靶点与N个第二靶点不共面，M和N为大于或等于3的整数。
在一种可能的实现方式中，设置单元1401，还用于：获取M个第一靶点的第二三维坐标和N个第二靶点的第二三维坐标，其中，第二三维坐标为全站仪坐标系下的坐标；对M个第一靶点的第二三维坐标和N个第二靶点的第二三维坐标进行坐标转换，得到M个第一靶点的第一三维坐标和N个第二靶点的第一三维坐标，其中，第一三维坐标为第一坐标系下的坐标，第一坐标系为以屏幕的一个边界点为原点、以屏幕的竖向为第一坐标轴、以屏幕的横向为第二坐标轴、以屏幕的垂直方向为第三坐标轴的坐标系；M个第一靶点的第一三维坐标和N个第二靶点的第一三维坐标用于摄像头的硬件在环标定。
在一种可能的实现方式中，在对M个第一靶点的第二三维坐标和N个第二靶点的第二三维坐标进行坐标转换，得到M个第一靶点的第一三维坐标和N个第二靶点的第一三维坐标方面，设置单元1401，具体用于：根据M个第一靶点的第二三维坐标和M个第一靶点的第二三维坐标，得到M个第一特征向量和M个第二特征向量；根据M个第一特征向量和M个第二特征向量采用最小二乘法计算得到中间向量；根据中间向量得到控制矩阵，以及根据控制矩阵得到旋转向量和平移向量；根据旋转向量和平移向量对M个第一靶点的第二三维坐标和N个第二靶点的第二三维坐标进行坐标转换，得到M个第一靶点的第一三维坐标和N个第二靶点的第一三维坐标。
在一种可能的实现方式中，标定图像的高宽比、屏幕的高宽比与显示设备的高宽比相同，设置单元1401，还用于：对显示标定图像的显示设备进行圆心检测，以得到M个第一靶点中的每个第一靶点的像素坐标；将M个第一靶点中的每个第一靶点的像素坐标转换到第二坐标系下，以得到M个第一靶点中的每个第一靶点的二维坐标，其中，第二坐标系为以显示设备的屏幕边框的一个外边界点为原点、以屏幕边框的竖向为第一坐标轴、以屏幕边框的横向为第二坐标轴的坐标系；根据M个第一靶点中的至少一个第一靶点的二维坐标、屏幕的高宽比与显示设备的高宽比确定N个第二靶点在第二坐标系下的二维坐标。
在一种可能的实现方式中，标定图像的尺寸与屏幕边框的外边界尺寸存在比例关系，设置单元1401，还用于：根据标定图像的尺寸与屏幕边框的外边界尺寸存在的比例关系，对标定图像进行放大处理，以得到放大后的标定图像，其中，放大后的标定图像的尺寸与屏幕边框的外边界尺寸相等；将放大后的标定图像投影到第二坐标系上，其中，第二坐标系为以显示设备的屏幕边框的一个外边界点为原点、以屏幕边框的竖向为第一坐标轴、以屏幕边框的横向为第二坐标轴的坐标系；将放大后的标定图像中与靶点中心位置重合的像素坐标作为该靶点的二维坐标，以得到M个第一靶点的二维坐标和N个第二靶点的二维坐 标。
需要说明的是，各个单元的实现还可以对应参照图2-图13所示的实施例的相应描述。当然，本申请实施例提供的摄像头的硬件在环标定靶点设置装置1400包括但不限于上述单元模块，例如：该摄像头的硬件在环标定靶点设置装置1400还可以包括存储单元1402。存储单元1402可以用于存储该摄像头的硬件在环标定靶点设置装置1400的程序代码和数据。
在图14所描述的摄像头的硬件在环标定靶点设置装置1400中，在显示设备的屏幕内设置M个第一靶点，以及在显示设备的屏幕边框上设置N个第二靶点，且该M个第一靶点与该N个第二靶点是不共面的，因此在标定时无需使用多块不共面的标定板，解决了摄像头的硬件在环系统中靶点设置空间受限的问题。
请参阅图15，图15是本申请实施例提供的一种摄像头的硬件在环标定装置的结构示意图，该摄像头的硬件在环标定装置1500可以包括确定单元1501、抽样单元1502和标定单元1503，该摄像头的硬件在环标定装置1500应用于电子设备，该电子设备包括该摄像头，或该电子设备与该摄像头通信连接，其中，各个单元的详细描述如下：
确定单元1501，用于确定多个靶点中的每个靶点对应的权重，其中，多个靶点为显示设备上的靶点，显示设备上位于目标区域内的靶点对应的权重大于位于目标区域外的靶点对应的权重，显示设备的目标区域用于显示标定图像的一部分；标定图像为在进行摄像头的硬件在环标定时显示设备显示的图像；
抽样单元1502，用于根据权重对多个靶点进行P次有放回的抽样，以得到P个靶点组，其中，针对每次抽样，靶点对应的权重越大，该靶点被抽取的概率越大，P为正整数；
标定单元1503，用于根据P个靶点组和摄像头的当前位姿进行摄像头的硬件在环标定。
在一种可能的实现方式中，标定单元1503，具体用于：根据P个靶点组确定P个第一目标位姿，其中，P个靶点组与P个第一目标位姿一一对应；通过摄像头对当前显示标定图像的显示设备进行图像采集以得到目标图像，并根据多个靶点和目标图像确定摄像头的当前位姿；根据P个第一目标位姿的平均值和当前位姿进行摄像头的硬件在环标定。
在一种可能的实现方式中，标定单元1503，还用于：在根据P个靶点组确定P个第一目标位姿之前；通过摄像头对标定板进行图像采集以得到Q张标定图像，其中，Q为正整数；对Q张标定图像中的每张标定图像进行角点检测以得到每张标定图像中的各个角点的像素坐标；根据Q张标定图像中的每张标定图像中的各个角点的像素坐标、每张标定图像中的各个角点的第三三维坐标得到摄像头的内参矩阵和摄像头的畸变系数，其中，第三三维坐标为标定板坐标系下的坐标。
在一种可能的实现方式中，标定单元1503，具体用于：针对P个靶点组中的每个靶点组，执行以下步骤，得到P个第一目标位姿：获取目标靶点组中的每个靶点的二维坐标，其中，目标靶点组为P个靶点组中的任意一个，二维坐标为第二坐标系下的坐标，第二坐标系为以显示设备的屏幕边框的一个外边界点为原点、以屏幕边框的竖向为第一坐标轴、以屏幕边框的横向为第二坐标轴的坐标系；根据目标靶点组中的每个靶点的二维坐标、第一三维坐标、摄像头的内参矩阵和摄像头的畸变系数，计算得到目标靶点组对应的第一目标位姿， 其中，第一三维坐标为第一坐标系下的坐标，第一坐标系为以显示设备的屏幕的一个边界点为原点、以屏幕的竖向为第一坐标轴、以屏幕的横向为第二坐标轴、以屏幕的垂直方向为第三坐标轴的坐标系。
在一种可能的实现方式中，标定单元1503，具体用于：对目标图像进行圆心检测以得到多个靶点中的每个靶点在目标图像中的像素坐标；根据多个靶点中的每个靶点在目标图像中的像素坐标、多个靶点中的每个靶点的第一三维坐标、摄像头的内参矩阵、摄像头的畸变系数，计算得到当前位姿，其中，第一三维坐标为第一坐标系下的坐标，第一坐标系为以显示设备的屏幕的一个边界点为原点、以屏幕的竖向为第一坐标轴、以屏幕的横向为第二坐标轴、以屏幕的垂直方向为第三坐标轴的坐标系。
在一种可能的实现方式中，多个靶点包括显示设备的屏幕内的M个第一靶点和屏幕边框上的N个第二靶点，其中，M个第一靶点与N个第二靶点不共面，M和N为大于或等于3的整数，标定单元1503，还用于：在根据P个靶点组和摄像头的当前位姿进行摄像头的硬件在环标定之前，获取M个第一靶点的第二三维坐标和N个第二靶点的第二三维坐标，其中，第二三维坐标为全站仪坐标系下的坐标；对M个第一靶点的第二三维坐标进行坐标转换以得到M个第一靶点的第一三维坐标，以及对N个第二靶点的第二三维坐标进行坐标转换以得到N个第二靶点的第一三维坐标。
在一种可能的实现方式中，标定单元1503，具体用于：将第二目标位姿与当前位姿作差以得到当前位姿残差，其中，第二目标位姿为P个第一目标位姿的平均值，当前位姿残差包括3个当前位置自由度残差和3个当前姿态自由度残差；先按照当前位置自由度残差从大到小的顺序依次调节摄像头的各个位置自由度，以使摄像头的第一位置自由度与第二目标位姿的第一位置自由度的差值小于预设的第一位置自由度残差阈值、摄像头的第二位置自由度与第二目标位姿的第二位置自由度的差值小于预设的第二位置自由度残差阈值、摄像头的第三位置自由度与第二目标位姿的第三位置自由度的差值小于预设的第三位置自由度残差阈值；再按照当前姿态自由度残差从大到小的顺序依次调节摄像头的各个姿态自由度，以使摄像头的第一姿态自由度与第二目标位姿的第一姿态自由度的差值小于预设的第一姿态自由度残差阈值、摄像头的第二姿态自由度与第二目标位姿的第二姿态自由度的差值小于预设的第二姿态自由度残差阈值、摄像头的第三姿态自由度与第二目标位姿的第三姿态自由度的差值小于预设的第三姿态自由度残差阈值。
需要说明的是，各个单元的实现还可以对应参照图2-图13所示的实施例的相应描述。当然，本申请实施例提供的摄像头的硬件在环标定装置1500包括但不限于上述单元模块，例如：该摄像头的硬件在环标定装置1500还可以包括存储单元1504。存储单元1504可以用于存储该摄像头的硬件在环标定装置1500的程序代码和数据。
在图15所描述的摄像头的硬件在环标定装置1500中，给不同区域的靶点设置不同的权重，基于权重对靶点进行有放回的重采样既保证了各组靶点数据之间的独立性，又侧重提升了摄像头的重点关注区域对应的靶点被抽取的概率，根据采样得到的各组靶点和摄像头的当前位姿进行摄像头的硬件在环标定，可有效提升摄像头对重点关注区域的对准精度。具体地，用于摄像头的硬件在环标定的显示设备上有多个靶点，在该显示设备上显示标定图像，该标定图像有该摄像头的重点关注区域，该标定图像被显示在该显示设备上时，该 显示设备上用于显示该重点关注区域的显示区域为目标区域，先设置该目标区域内的靶点对应的权重大于位于该目标区域外的靶点对应的权重；然后基于权重概率对多个靶点进行P次有放回的抽样以得到P个靶点组，也即靶点对应的权重越大，该靶点被抽取的概率越大；由于目标区域内的靶点的权重大于位于该目标区域外的靶点对应的权重，故每次抽样时，目标区域内的靶点被抽取的概率更大，目标区域内的靶点被抽取的数量相对就越多，在根据P个靶点组和摄像头的当前位姿进行摄像头的硬件在环标定时，目标区域内的靶点所起的决定作用占比就越大，摄像头对重点关注区域的对准精度就越高。综上，本申请实施例，可有效提升摄像头对重点关注区域的对准精度，保证用户的个性化对准需求。
请参阅图16，图16是本申请实施例提供的一种电子设备的结构示意图，该电子设备1610包括收发器1611、处理器1612和存储器1613，收发器1611、处理器1612和存储器1613通过总线1614相互连接。
存储器1613包括但不限于是随机存储记忆体(random access memory，RAM)、只读存储器(read-only memory，ROM)、可擦除可编程只读存储器(erasable programmable read only memory，EPROM)、或便携式只读存储器(compact disc read-only memory，CD-ROM)，该存储器1613用于相关指令及数据。
收发器1611用于接收和发送数据。
处理器1612可以是一个或多个中央处理器(central processing unit，CPU)，在处理器1612是一个CPU的情况下，该CPU可以是单核CPU，也可以是多核CPU。
该电子设备1610中的处理器1612用于读取存储器1613中存储的程序代码，执行本申请实施例所描述的方法。
需要说明的是，各个操作的实现还可以对应参照图2-图13所示的实施例的相应描述。
在图16所描述的电子设备1610中，在显示设备的屏幕内和屏幕边框上设置不共面的靶点，因此在标定时无需使用多块不共面的标定板，解决了摄像头的硬件在环系统中靶点设置空间受限的问题；以及给不同区域的靶点设置不同的权重，基于权重对靶点进行有放回的重采样既保证了各组靶点数据之间的独立性，又侧重提升了摄像头的重点关注区域对应的靶点被抽取的概率，根据采样得到的各组靶点和摄像头的当前位姿进行摄像头的硬件在环标定，可有效提升摄像头对重点关注区域的对准精度。
本申请实施例还提供一种芯片，该芯片包括至少一个处理器，存储器和接口电路，上述存储器、上述收发器和上述至少一个处理器通过线路互联，上述至少一个存储器中存储有计算机程序；上述计算机程序被上述处理器执行时，上述方法实施例中所示的方法流程得以实现。
本申请实施例还提供一种计算机可读存储介质，上述计算机可读存储介质中存储有计算机程序，当其在电子设备上运行时，上述方法实施例中所示的方法流程得以实现。
本申请实施例还提供一种计算机程序，当上述计算机程序在电子设备上运行时，上述方法实施例中所示的方法流程得以实现。
综上可知，现有的摄像头标定方法，基本没有量化指标，只有定性的给出摄像头应处的初始位置(如屏幕中心轴线上)，然后通过手工调节的方法，将摄像头调整到可以完整拍摄到屏幕；由于摄像头的位姿调节极大取决于用户的经验，因此这种定性地位姿调节方法通常效率低下(精细调节需要1小时左右)；此外，这种传统标定方法的精度很难保证，且无法做到根据摄像头的功能进行有侧重的位姿调节，适用性较差。本申请实施例可以进行摄像头位姿的精确调节，首先针对摄像头标定的需要，提出了一种摄像头的硬件在环标定靶点设置方法，该方法利用屏幕内的靶点和屏幕边框上的靶点不共面的特点，规避了使用多个标定板的弊端；并提出了一种摄像头的硬件在环标定方法，通过基于权重赋值的bootstrap重采样方法生成靶点组合，利用张氏标定法、UPNP算法和已知的靶点坐标信息计算得到摄像头的目标位姿，为摄像头的对准方向提供标杆，可大大提升对准效率和对准精度。除此之外，本申请实施例还给出了最终对准结果的评价指标及其计算方法(误差地图)，向用户提供了全量的最终误差信息，为用户评价ADAS算法提供原始误差输入。
本申请实施例提供的关键技术点包括：
关键技术点一：本申请实施例提供一种摄像头的硬件在环标定靶点设置方法。将摄像头标定的方法用于摄像头的硬件在环标定可有效提升摄像头的对准效率和对准精度。摄像头的标定算法需要使用不共面的靶点信息，通常采用2块不共面标定板完成摄像头的标定。摄像头的硬件在环系统无法使用标定板，为此设计一种在屏幕内设置靶点，以及在屏幕边框上设置靶点，实现不共面靶点的设置，并给出靶点三维坐标的计算和转换方法。
关键技术点二：本申请实施例提供一种基于重点关注区域选择的目标位姿求解方法。不同功能的摄像头关注的区域不尽相同，设计选择重点区域的功能可有效提升重点区域的对准精度，降低摄像头原始数据层的误差。为此，本申请提出一种基于权重的bootstrap重采样靶点组合生成方法，对每一组靶点数据使用摄像头标定技术进行摄像头的目标位姿求解，通过求平均得到摄像头的目标位姿数值。该抽样方法可保证每一组靶点数据的独立性，同时侧重对准框重点关注区域内的靶点，保证了用户的个性化对准需求。结合标定算法的重点关注区域对准方法量化给定了摄像头位姿调整标杆，可有效提升摄像头的标定效率和标定精度。
本申请实施例的有益效果包括：
有益效果一：本申请实施例提供的一种摄像头的硬件在环标定靶点设置方法，无需使用多块不共面的标定板，解决了摄像头的硬件在环标定系统中靶点设置空间受限问题；通过坐标转换，可将靶点坐标转换到屏幕坐标系下，为后续的调节反馈提示奠定基础，有利于提升摄像头对准效果。
有益效果二：本申请实施例提供的一种摄像头的硬件在环标定方法，结合权重重采样方法、摄像头标定算法，在保证摄像头的重点关注区域的对准精度下，求解出摄像头的目标位姿，该目标位姿为摄像头的对准提供了量化标杆，是保证对准精度、提升对准效率。
应理解，本申请实施例中提及的处理器可以是中央处理单元(Central Processing Unit，CPU)，还可以是其他通用处理器、数字信号处理器(Digital Signal Processor，DSP)、专用集成电路(Application Specific Integrated Circuit，ASIC)、现成可编程门阵列(Field  Programmable Gate Array，FPGA)或者其他可编程逻辑器件、分立门或者晶体管逻辑器件、分立硬件组件等。通用处理器可以是微处理器或者该处理器也可以是任何常规的处理器等。
还应理解，本申请实施例中提及的存储器可以是易失性存储器或非易失性存储器，或可包括易失性和非易失性存储器两者。其中，非易失性存储器可以是只读存储器(Read-Only Memory，ROM)、可编程只读存储器(Programmable ROM，PROM)、可擦除可编程只读存储器(Erasable PROM，EPROM)、电可擦除可编程只读存储器(Electrically EPROM，EEPROM)或闪存。易失性存储器可以是随机存取存储器(Random Access Memory，RAM)，其用作外部高速缓存。通过示例性但不是限制性说明，许多形式的RAM可用，例如静态随机存取存储器(Static RAM，SRAM)、动态随机存取存储器(Dynamic RAM，DRAM)、同步动态随机存取存储器(Synchronous DRAM，SDRAM)、双倍数据速率同步动态随机存取存储器(Double Data Rate SDRAM，DDR SDRAM)、增强型同步动态随机存取存储器(Enhanced SDRAM，ESDRAM)、同步连接动态随机存取存储器(Synchlink DRAM，SLDRAM)和直接内存总线随机存取存储器(Direct Rambus RAM，DR RAM)。
需要说明的是，当处理器为通用处理器、DSP、ASIC、FPGA或者其他可编程逻辑器件、分立门或者晶体管逻辑器件、分立硬件组件时，存储器(存储模块)集成在处理器中。
应注意，本说明书描述的存储器旨在包括但不限于这些和任意其它适合类型的存储器。
应理解，在本申请的各种实施例中，上述各过程的序号的大小并不意味着执行顺序的先后，各过程的执行顺序应以其功能和内在逻辑确定，而不应对本申请实施例的实施过程构成任何限定。
本领域普通技术人员可以意识到，结合本说明书中所公开的实施例描述的各示例的单元及算法步骤，能够以电子硬件、或者计算机软件和电子硬件的结合来实现。这些功能究竟以硬件还是软件方式来执行，取决于技术方案的特定应用和设计约束条件。专业技术人员可以对每个特定的应用来使用不同方法来实现所描述的功能，但是这种实现不应认为超出本申请的范围。
所属领域的技术人员可以清楚地了解到，为描述的方便和简洁，上述描述的系统、装置和单元的具体工作过程，可以参考前述方法实施例中的对应过程，在此不再赘述。
在本申请所提供的几个实施例中，应该理解到，所揭露的系统、装置和方法，可以通过其它的方式实现。例如，以上所描述的装置实施例仅仅是示意性的，例如，上述单元的划分，仅仅为一种逻辑功能划分，实际实现时可以有另外的划分方式，例如多个单元或组件可以结合或者可以集成到另一个系统，或一些特征可以忽略，或不执行。另一点，所显示或讨论的相互之间的耦合或直接耦合或通信连接可以是通过一些接口，装置或单元的间接耦合或通信连接，可以是电性，机械或其它的形式。
上述作为分离部件说明的单元可以是或者也可以不是物理上分开的，作为单元显示的部件可以是或者也可以不是物理单元，即可以位于一个地方，或者也可以分布到多个网络单元上。可以根据实际的需要选择其中的部分或者全部单元来实现本实施例方案的目的。
另外，在本申请各个实施例中的各功能单元可以集成在一个处理单元中，也可以是各个单元单独物理存在，也可以两个或两个以上单元集成在一个单元中。
上述功能如果以软件功能单元的形式实现并作为独立的产品销售或使用时，可以存储 在一个计算机可读取存储介质中。基于这样的理解，本申请的技术方案本质上或者说对现有技术做出贡献的部分或者该技术方案的部分可以以软件产品的形式体现出来，该计算机软件产品存储在一个存储介质中，包括若干指令用以使得一台计算机设备(可以是个人计算机，服务器，或者网络设备等)执行本申请各个实施例上述方法的全部或部分步骤。而前述的存储介质包括：U盘、移动硬盘、只读存储器(Read-Only Memory，ROM)、随机存取存储器(Random Access Memory，RAM)、磁碟或者光盘等各种可以存储程序代码的介质。
本申请实施例方法中的步骤可以根据实际需要进行顺序调整、合并和删减。此外，本申请各实施例中的术语、解释说明，可以参照其他实施例中相应的描述。
本申请实施例装置中的模块可以根据实际需要进行合并、划分和删减。
以上描述，以上实施例仅用以说明本申请的技术方案，而非对其限制；尽管参照前述实施例对本申请进行了详细的说明，本领域的普通技术人员应当理解：其依然可以对前述各实施例所记载的技术方案进行修改，或者对其中部分技术特征进行等同替换；而这些修改或者替换，并不使相应技术方案的本质脱离本申请各实施例技术方案的范围。

Claims (21)

  1. 一种摄像头的硬件在环标定方法，其特征在于，所述方法包括：
    确定多个靶点中的每个靶点对应的权重，其中，所述多个靶点为显示设备上的靶点，所述显示设备上位于目标区域内的靶点对应的权重大于位于所述目标区域外的靶点对应的权重，所述显示设备的目标区域用于显示标定图像的一部分；所述标定图像为在进行所述摄像头的硬件在环标定时所述显示设备显示的图像；
    根据所述权重对所述多个靶点进行P次有放回的抽样，以得到P个靶点组，其中，针对每次抽样，靶点对应的权重越大，该靶点被抽取的概率越大，所述P为正整数；
    根据所述P个靶点组和所述摄像头的当前位姿进行所述摄像头的硬件在环标定。
  2. 根据权利要求1所述的方法，其特征在于，所述根据所述P个靶点组和所述摄像头的当前位姿进行所述摄像头的硬件在环标定，包括：
    根据所述P个靶点组确定P个第一目标位姿，其中，所述P个靶点组与所述P个第一目标位姿一一对应；
    通过所述摄像头对当前显示所述标定图像的所述显示设备进行图像采集以得到目标图像，并根据所述多个靶点和所述目标图像确定所述摄像头的当前位姿；
    根据所述P个第一目标位姿的平均值和所述当前位姿进行所述摄像头的硬件在环标定。
  3. 根据权利要求2所述的方法，其特征在于，所述根据所述P个靶点组确定P个第一目标位姿，包括：
    针对所述P个靶点组中的每个靶点组，执行以下步骤，得到P个第一目标位姿：
    获取所述目标靶点组中的每个靶点的二维坐标，其中，所述目标靶点组为所述P个靶点组中的任意一个，
    所述二维坐标为第二坐标系下的坐标，所述第二坐标系为以所述显示设备的屏幕边框的一个外边界点为原点、以所述屏幕边框的竖向为第一坐标轴、以所述屏幕边框的横向为第二坐标轴的坐标系；
    根据所述目标靶点组中的每个靶点的二维坐标、第一三维坐标、所述摄像头的内参矩阵和所述摄像头的畸变系数，计算得到所述目标靶点组对应的第一目标位姿，
    其中，所述第一三维坐标为第一坐标系下的坐标，所述第一坐标系为以所述显示设备的屏幕的一个边界点为原点、以所述屏幕的竖向为第一坐标轴、以所述屏幕的横向为第二坐标轴、以所述屏幕的垂直方向为第三坐标轴的坐标系。
  4. 根据权利要求2或3所述的方法，其特征在于，所述根据所述多个靶点和所述目标图像确定所述摄像头的当前位姿，包括：
    对所述目标图像进行圆心检测以得到所述多个靶点中的每个靶点在所述目标图像中的像素坐标；
    根据所述多个靶点中的每个靶点在所述目标图像中的像素坐标、所述多个靶点中的每个靶点的第一三维坐标、所述摄像头的内参矩阵、所述摄像头的畸变系数，计算得到所述当前位姿，
    其中，所述第一三维坐标为第一坐标系下的坐标，所述第一坐标系为以所述显示设备的屏幕的一个边界点为原点、以所述屏幕的竖向为第一坐标轴、以所述屏幕的横向为第二坐标轴、以所述屏幕的垂直方向为第三坐标轴的坐标系。
  5. 根据权利要求2-4中任一项所述的方法，其特征在于，所述多个靶点包括所述显示设备的屏幕内的M个第一靶点和屏幕边框上的N个第二靶点，其中，所述M个第一靶点与所述N个第二靶点不共面，所述M和所述N为大于或等于3的整数，在所述根据所述P个靶点组和所述摄像头的当前位姿进行所述摄像头的硬件在环标定之前，所述方法还包括：
    获取所述M个第一靶点的第二三维坐标和所述N个第二靶点的第二三维坐标，其中，所述第二三维坐标为全站仪坐标系下的坐标；
    对所述M个第一靶点的第二三维坐标进行坐标转换以得到所述M个第一靶点的第一三维坐标，以及对所述N个第二靶点的第二三维坐标进行坐标转换以得到所述N个第二靶点的第一三维坐标。
  6. 根据权利要求2-5中任一项所述的方法，其特征在于，所述根据所述P个第一目标位姿的平均值和所述当前位姿进行所述摄像头的硬件在环标定，包括：
    将第二目标位姿与所述当前位姿作差以得到当前位姿残差，其中，所述第二目标位姿为所述P个第一目标位姿的平均值，所述当前位姿残差包括3个当前位置自由度残差和3个当前姿态自由度残差；
    先按照当前位置自由度残差从大到小的顺序依次调节所述摄像头的各个位置自由度，以使所述摄像头的第一位置自由度与所述第二目标位姿的第一位置自由度的差值小于预设的第一位置自由度残差阈值、所述摄像头的第二位置自由度与所述第二目标位姿的第二位置自由度的差值小于预设的第二位置自由度残差阈值、所述摄像头的第三位置自由度与所述第二目标位姿的第三位置自由度的差值小于预设的第三位置自由度残差阈值；
    再按照当前姿态自由度残差从大到小的顺序依次调节所述摄像头的各个姿态自由度，以使所述摄像头的第一姿态自由度与所述第二目标位姿的第一姿态自由度的差值小于预设的第一姿态自由度残差阈值、所述摄像头的第二姿态自由度与所述第二目标位姿的第二姿态自由度的差值小于预设的第二姿态自由度残差阈值、所述摄像头的第三姿态自由度与所述第二目标位姿的第三姿态自由度的差值小于预设的第三姿态自由度残差阈值。
  7. 一种摄像头的硬件在环标定靶点设置方法，其特征在于，包括：
    在显示设备的屏幕内设置M个第一靶点，以及在所述显示设备的屏幕边框上设置N个第二靶点，其中，所述显示设备用于在进行摄像头的硬件在环标定时显示标定图像，所述M个第一靶点与所述N个第二靶点不共面，所述M和所述N为大于或等于3的整数。
  8. 根据权利要求7所述的方法，其特征在于，所述方法还包括：
    获取所述M个第一靶点的第二三维坐标和所述N个第二靶点的第二三维坐标，其中，所述第二三维坐标为全站仪坐标系下的坐标；
    对所述M个第一靶点的第二三维坐标和所述N个第二靶点的第二三维坐标进行坐标转换，得到所述M个第一靶点的第一三维坐标和所述N个第二靶点的第一三维坐标，其中，所述第一三维坐标为第一坐标系下的坐标，所述第一坐标系为以所述屏幕的一个边界点为原点、以所述屏幕的竖向为第一坐标轴、以所述屏幕的横向为第二坐标轴、以所述屏幕的垂直方向为第三坐标轴的坐标系；
    所述M个第一靶点的第一三维坐标和所述N个第二靶点的第一三维坐标用于摄像头的硬件在环标定。
  9. 一种摄像头的硬件在环标定装置，其特征在于，所述装置包括：
    确定单元，用于确定多个靶点中的每个靶点对应的权重，其中，所述多个靶点为显示设备上的靶点，所述显示设备上位于目标区域内的靶点对应的权重大于位于所述目标区域外的靶点对应的权重，所述显示设备的目标区域用于显示标定图像的一部分；所述标定图像为在进行所述摄像头的硬件在环标定时所述显示设备显示的图像；
    抽样单元，用于根据所述权重对所述多个靶点进行P次有放回的抽样，以得到P个靶点组，其中，针对每次抽样，靶点对应的权重越大，该靶点被抽取的概率越大，所述P为正整数；
    标定单元，用于根据所述P个靶点组和所述摄像头的当前位姿进行所述摄像头的硬件在环标定。
  10. 根据权利要求9所述的装置，其特征在于，所述标定单元，具体用于：
    根据所述P个靶点组确定P个第一目标位姿，其中，所述P个靶点组与所述P个第一目标位姿一一对应；
    通过所述摄像头对当前显示所述标定图像的所述显示设备进行图像采集以得到目标图像，并根据所述多个靶点和所述目标图像确定所述摄像头的当前位姿；
    根据所述P个第一目标位姿的平均值和所述当前位姿进行所述摄像头的硬件在环标定。
  11. 根据权利要求10所述的装置，其特征在于，所述标定单元，具体用于：
    针对所述P个靶点组中的每个靶点组，执行以下步骤，得到P个第一目标位姿：
    获取所述目标靶点组中的每个靶点的二维坐标，其中，所述目标靶点组为所述P个靶点组中的任意一个，
    所述二维坐标为第二坐标系下的坐标，所述第二坐标系为以所述显示设备的屏幕边框的一个外边界点为原点、以所述屏幕边框的竖向为第一坐标轴、以所述屏幕边框的横向为第二坐标轴的坐标系；
    根据所述目标靶点组中的每个靶点的二维坐标、第一三维坐标、所述摄像头的内参矩 阵和所述摄像头的畸变系数，计算得到所述目标靶点组对应的第一目标位姿，
    其中，所述第一三维坐标为第一坐标系下的坐标，所述第一坐标系为以所述显示设备的屏幕的一个边界点为原点、以所述屏幕的竖向为第一坐标轴、以所述屏幕的横向为第二坐标轴、以所述屏幕的垂直方向为第三坐标轴的坐标系。
  12. 根据权利要求10或11所述的装置，其特征在于，所述标定单元，具体用于：
    对所述目标图像进行圆心检测以得到所述多个靶点中的每个靶点在所述目标图像中的像素坐标；
    根据所述多个靶点中的每个靶点在所述目标图像中的像素坐标、所述多个靶点中的每个靶点的第一三维坐标、所述摄像头的内参矩阵、所述摄像头的畸变系数，计算得到所述当前位姿，
    其中，所述第一三维坐标为第一坐标系下的坐标，所述第一坐标系为以所述显示设备的屏幕的一个边界点为原点、以所述屏幕的竖向为第一坐标轴、以所述屏幕的横向为第二坐标轴、以所述屏幕的垂直方向为第三坐标轴的坐标系。
  13. 根据权利要求10-12中任一项所述的装置，其特征在于，所述多个靶点包括所述显示设备的屏幕内的M个第一靶点和屏幕边框上的N个第二靶点，其中，所述M个第一靶点与所述N个第二靶点不共面，所述M和所述N为大于或等于3的整数，所述标定单元，还用于：
    在所述根据所述P个靶点组和所述摄像头的当前位姿进行所述摄像头的硬件在环标定之前，
    获取所述M个第一靶点的第二三维坐标和所述N个第二靶点的第二三维坐标，其中，所述第二三维坐标为全站仪坐标系下的坐标；
    对所述M个第一靶点的第二三维坐标进行坐标转换以得到所述M个第一靶点的第一三维坐标，以及对所述N个第二靶点的第二三维坐标进行坐标转换以得到所述N个第二靶点的第一三维坐标。
  14. 根据权利要求10-13中任一项所述的装置，其特征在于，所述标定单元，具体用于：
    将第二目标位姿与所述当前位姿作差以得到当前位姿残差，其中，所述第二目标位姿为所述P个第一目标位姿的平均值，所述当前位姿残差包括3个当前位置自由度残差和3个当前姿态自由度残差；
    先按照当前位置自由度残差从大到小的顺序依次调节所述摄像头的各个位置自由度，以使所述摄像头的第一位置自由度与所述第二目标位姿的第一位置自由度的差值小于预设的第一位置自由度残差阈值、所述摄像头的第二位置自由度与所述第二目标位姿的第二位置自由度的差值小于预设的第二位置自由度残差阈值、所述摄像头的第三位置自由度与所述第二目标位姿的第三位置自由度的差值小于预设的第三位置自由度残差阈值；
    再按照当前姿态自由度残差从大到小的顺序依次调节所述摄像头的各个姿态自由度， 以使所述摄像头的第一姿态自由度与所述第二目标位姿的第一姿态自由度的差值小于预设的第一姿态自由度残差阈值、所述摄像头的第二姿态自由度与所述第二目标位姿的第二姿态自由度的差值小于预设的第二姿态自由度残差阈值、所述摄像头的第三姿态自由度与所述第二目标位姿的第三姿态自由度的差值小于预设的第三姿态自由度残差阈值。
  15. 一种摄像头的硬件在环标定靶点设置装置，其特征在于，包括：
    设置单元，用于在显示设备的屏幕内设置M个第一靶点，以及在所述显示设备的屏幕边框上设置N个第二靶点，其中，所述显示设备用于在进行摄像头的硬件在环标定时显示标定图像，所述M个第一靶点与所述N个第二靶点不共面，所述M和所述N为大于或等于3的整数。
  16. 根据权利要求15所述的装置，其特征在于，所述设置单元，还用于：
    获取所述M个第一靶点的第二三维坐标和所述N个第二靶点的第二三维坐标，其中，所述第二三维坐标为全站仪坐标系下的坐标；
    对所述M个第一靶点的第二三维坐标和所述N个第二靶点的第二三维坐标进行坐标转换，得到所述M个第一靶点的第一三维坐标和所述N个第二靶点的第一三维坐标，其中，所述第一三维坐标为第一坐标系下的坐标，所述第一坐标系为以所述屏幕的一个边界点为原点、以所述屏幕的竖向为第一坐标轴、以所述屏幕的横向为第二坐标轴、以所述屏幕的垂直方向为第三坐标轴的坐标系；
    所述M个第一靶点的第一三维坐标和所述N个第二靶点的第一三维坐标用于摄像头的硬件在环标定。
  17. 一种摄像头的硬件在环标定系统，所述系统包括电子设备和显示设备，所述电子设备包括所述摄像头，或所述电子设备与所述摄像头通信连接，所述显示设备包括屏幕和屏幕边框，其特征在于：
    所述屏幕内设置有M个第一靶点，或
    所述屏幕显示有M个第一靶点；
    所述屏幕边框上设置有N个第二靶点；
    其中，所述M个第一靶点与所述N个第二靶点不共面，所述M和所述N为大于或等于3的整数。
  18. 一种电子设备，其特征在于，包括处理器、存储器、通信接口，以及一个或多个程序，所述一个或多个程序被存储在所述存储器中，并且被配置由所述处理器执行，所述程序包括用于执行如权利要求1-6或7或8中任一项所述的方法中的步骤的指令。
  19. 一种芯片，其特征在于，包括：处理器，用于从存储器中调用并运行计算机程序，使得安装有所述芯片的设备执行如权利要求1-6或7或8中任一项所述的方法。
  20. 一种计算机可读存储介质，其特征在于，其存储用于电子数据交换的计算机程序，其中，所述计算机程序使得计算机执行如权利要求1-6或7或8中任一项所述的方法。
  21. 一种计算机程序，所述计算机程序使得计算机执行如权利要求1-6或7或8中任一项所述的方法。
PCT/CN2020/141855 2020-12-30 2020-12-30 摄像头的硬件在环标定、靶点设置方法、系统及相关设备 WO2022141324A1 (zh)

Priority Applications (2)

Application Number Priority Date Filing Date Title
CN202080004242.1A CN112753047B (zh) 2020-12-30 2020-12-30 摄像头的硬件在环标定、靶点设置方法、系统及相关设备
PCT/CN2020/141855 WO2022141324A1 (zh) 2020-12-30 2020-12-30 摄像头的硬件在环标定、靶点设置方法、系统及相关设备

Applications Claiming Priority (1)

Application Number Priority Date Filing Date Title
PCT/CN2020/141855 WO2022141324A1 (zh) 2020-12-30 2020-12-30 摄像头的硬件在环标定、靶点设置方法、系统及相关设备

Publications (1)

Publication Number Publication Date
WO2022141324A1 true WO2022141324A1 (zh) 2022-07-07

Family

ID=75651198

Family Applications (1)

Application Number Title Priority Date Filing Date
PCT/CN2020/141855 WO2022141324A1 (zh) 2020-12-30 2020-12-30 摄像头的硬件在环标定、靶点设置方法、系统及相关设备

Country Status (2)

Country Link
CN (1) CN112753047B (zh)
WO (1) WO2022141324A1 (zh)

Cited By (3)

* Cited by examiner, † Cited by third party
Publication number Priority date Publication date Assignee Title
CN115239795A (zh) * 2022-09-23 2022-10-25 山东工程职业技术大学 射箭靶环命中位置记环检测方法、检测装置及电子设备
CN115979121A (zh) * 2022-10-26 2023-04-18 成都清正公路工程试验检测有限公司 一种提高自动测量系统点位测量精度的方法
CN116563394A (zh) * 2023-07-10 2023-08-08 国科天成科技股份有限公司 一种异面点空间坐标标定系统和方法

Citations (6)

* Cited by examiner, † Cited by third party
Publication number Priority date Publication date Assignee Title
US20150341629A1 (en) * 2014-05-21 2015-11-26 GM Global Technology Operations LLC Automatic calibration of extrinsic and intrinsic camera parameters for surround-view camera system
CN108038886A (zh) * 2017-12-06 2018-05-15 智车优行科技（北京）有限公司 双目摄像系统标定方法、装置及其汽车
CN207529431U (zh) * 2017-11-20 2018-06-22 北京经纬恒润科技有限公司 一种摄像头标定系统
CN109712198A (zh) * 2018-12-26 2019-05-03 珠海亿智电子科技有限公司 一种高级驾驶辅助系统的标定方法
CN111652942A (zh) * 2020-05-29 2020-09-11 维沃移动通信有限公司 摄像模组的标定方法、第一电子设备和第二电子设备
CN111681286A (zh) * 2020-06-09 2020-09-18 商汤集团有限公司 一种标定方法、装置、系统、电子设备及存储介质

Family Cites Families (5)

* Cited by examiner, † Cited by third party
Publication number Priority date Publication date Assignee Title
WO2012019322A1 (zh) * 2010-08-13 2012-02-16 Xu Hong 利用单目摄像机标定技术的视觉指向型鼠标输入方法、输入系统和输入装置
CN103559707B (zh) * 2013-10-30 2016-04-20 同济大学 基于运动方靶标定物的工业定焦相机参数标定方法
CN105678748B (zh) * 2015-12-30 2019-01-15 清华大学 三维监控系统中基于三维重构的交互式标定方法和装置
CN106548477B (zh) * 2017-01-24 2019-03-29 长沙全度影像科技有限公司 一种基于立体标定靶的多路鱼眼相机标定装置及方法
CN107092751B (zh) * 2017-04-24 2019-11-26 厦门大学 基于Bootstrap的变权重模型组合预测方法

Patent Citations (6)

* Cited by examiner, † Cited by third party
Publication number Priority date Publication date Assignee Title
US20150341629A1 (en) * 2014-05-21 2015-11-26 GM Global Technology Operations LLC Automatic calibration of extrinsic and intrinsic camera parameters for surround-view camera system
CN207529431U (zh) * 2017-11-20 2018-06-22 北京经纬恒润科技有限公司 一种摄像头标定系统
CN108038886A (zh) * 2017-12-06 2018-05-15 智车优行科技（北京）有限公司 双目摄像系统标定方法、装置及其汽车
CN109712198A (zh) * 2018-12-26 2019-05-03 珠海亿智电子科技有限公司 一种高级驾驶辅助系统的标定方法
CN111652942A (zh) * 2020-05-29 2020-09-11 维沃移动通信有限公司 摄像模组的标定方法、第一电子设备和第二电子设备
CN111681286A (zh) * 2020-06-09 2020-09-18 商汤集团有限公司 一种标定方法、装置、系统、电子设备及存储介质

Cited By (5)

* Cited by examiner, † Cited by third party
Publication number Priority date Publication date Assignee Title
CN115239795A (zh) * 2022-09-23 2022-10-25 山东工程职业技术大学 射箭靶环命中位置记环检测方法、检测装置及电子设备
CN115239795B (zh) * 2022-09-23 2022-12-30 山东工程职业技术大学 射箭靶环命中位置记环检测方法、检测装置及电子设备
CN115979121A (zh) * 2022-10-26 2023-04-18 成都清正公路工程试验检测有限公司 一种提高自动测量系统点位测量精度的方法
CN116563394A (zh) * 2023-07-10 2023-08-08 国科天成科技股份有限公司 一种异面点空间坐标标定系统和方法
CN116563394B (zh) * 2023-07-10 2023-09-22 国科天成科技股份有限公司 一种异面点空间坐标标定系统和方法

Also Published As

Publication number Publication date
CN112753047B (zh) 2022-08-26
CN112753047A (zh) 2021-05-04

Similar Documents

Publication Publication Date Title
WO2022141324A1 (zh) 摄像头的硬件在环标定、靶点设置方法、系统及相关设备
US20200288059A1 (en) Image processor, image processing method and program, and imaging system
CN108765498B (zh) 单目视觉跟踪方法、装置及存储介质
US20230027389A1 (en) Distance determination method, apparatus and system
CN112232279B (zh) 一种人员间距检测方法和装置
JP5074322B2 (ja) 画像処理装置、画像処理方法、画像処理プログラム、及び、撮像装置
CN102194223B (zh) 一种变焦镜头的畸变系数标定方法及系统
CN106570907B (zh) 一种相机标定方法及装置
CN111627073B (zh) 一种基于人机交互的标定方法、标定装置和存储介质
CN109727292B (zh) 基于多摄像头-投影仪的互动投影系统及自动化标定方法
CN110136207B (zh) 鱼眼相机标定系统、方法、装置、电子设备及存储介质
KR101111503B1 (ko) 전방향 피티지 카메라 제어 장치 및 그 방법
CN111461963A (zh) 一种鱼眼图像拼接方法及装置
CN113920206A (zh) 透视移轴相机的标定方法
CN112598747A (zh) 一种单目相机与投影仪的联合标定方法
CN115830131A (zh) 一种固定相位偏差的确定方法、装置和设备
CN111696141B (zh) 一种三维全景扫描采集方法、设备及存储设备
CN111353945B (zh) 鱼眼图像校正方法、装置及存储介质
CN110838147A (zh) 摄像模组检测方法及装置
CN115239816A (zh) 一种相机标定方法、系统、电子设备及存储介质
CN113421300A (zh) 确定鱼眼相机图像中物体实际位置的方法及装置
CN112446928B (zh) 一种拍摄装置的外参确定系统和方法
CN115278071B (zh) 图像处理方法、装置、电子设备和可读存储介质
CN115937321B (zh) 一种电子设备的姿态检测方法和装置
CN117406185B (zh) 雷达与相机间的外参标定方法、装置、设备及存储介质

Legal Events

Date Code Title Description
121 Ep: the epo has been informed by wipo that ep was designated in this application

Ref document number: 20967663

Country of ref document: EP

Kind code of ref document: A1

NENP Non-entry into the national phase

Ref country code: DE

122 Ep: pct application non-entry in european phase

Ref document number: 20967663

Country of ref document: EP

Kind code of ref document: A1