WO2022267444A1 - 用于相机标定的方法及装置 - Google Patents

用于相机标定的方法及装置 Download PDF

Info

Publication number
WO2022267444A1
WO2022267444A1 PCT/CN2022/070534 CN2022070534W WO2022267444A1 WO 2022267444 A1 WO2022267444 A1 WO 2022267444A1 CN 2022070534 W CN2022070534 W CN 2022070534W WO 2022267444 A1 WO2022267444 A1 WO 2022267444A1
Authority
WO
WIPO (PCT)
Prior art keywords
point cloud
target frame
frame point
target
transformation matrix
Prior art date
Application number
PCT/CN2022/070534
Other languages
English (en)
French (fr)
Inventor
黄超
张�浩
杨嘉靖
Original Assignee
上海仙途智能科技有限公司
Priority date (The priority date is an assumption and is not a legal conclusion. Google has not performed a legal analysis and makes no representation as to the accuracy of the date listed.)
Filing date
Publication date
Application filed by 上海仙途智能科技有限公司 filed Critical 上海仙途智能科技有限公司
Publication of WO2022267444A1 publication Critical patent/WO2022267444A1/zh

Links

Images

Classifications

    • G—PHYSICS
    • G06—COMPUTING; CALCULATING OR COUNTING
    • G06T—IMAGE DATA PROCESSING OR GENERATION, IN GENERAL
    • G06T7/00—Image analysis
    • G06T7/20—Analysis of motion
    • G06T7/246—Analysis of motion using feature-based methods, e.g. the tracking of corners or segments
    • G—PHYSICS
    • G06—COMPUTING; CALCULATING OR COUNTING
    • G06T—IMAGE DATA PROCESSING OR GENERATION, IN GENERAL
    • G06T7/00—Image analysis
    • G06T7/80—Analysis of captured images to determine intrinsic or extrinsic camera parameters, i.e. camera calibration
    • G—PHYSICS
    • G06—COMPUTING; CALCULATING OR COUNTING
    • G06T—IMAGE DATA PROCESSING OR GENERATION, IN GENERAL
    • G06T2207/00—Indexing scheme for image analysis or image enhancement
    • G06T2207/10—Image acquisition modality
    • G06T2207/10028—Range image; Depth image; 3D point clouds
    • G—PHYSICS
    • G06—COMPUTING; CALCULATING OR COUNTING
    • G06T—IMAGE DATA PROCESSING OR GENERATION, IN GENERAL
    • G06T2207/00—Indexing scheme for image analysis or image enhancement
    • G06T2207/30—Subject of image; Context of image processing
    • G06T2207/30244—Camera pose

Definitions

  • One or more embodiments of this specification relate to the technical field of computer vision, and in particular to a method and device for camera calibration.
  • the camera can provide information such as color and texture
  • the position sensor can provide information such as distance and speed
  • a relatively common way is to configure the camera and position sensor on devices such as smart driving cars and drones at the same time, so as to obtain the information captured by the camera.
  • the pixel coordinate system of the image collected by the camera and the point cloud coordinate system of the point cloud collected by the position sensor are independent of each other, in order to comprehensively use the information of the two, it is necessary to determine the relationship between the pixel coordinate system and the point cloud coordinate system.
  • the conversion relationship, the process of determining the conversion relationship belongs to camera calibration. However, at present, the accuracy of realizing the joint calibration between the position sensor and the camera in related technologies still needs to be improved.
  • one or more embodiments of this specification provide a camera calibration method and device.
  • a camera calibration method is proposed, which is applied to an electronic device equipped with a position sensor and a camera.
  • the device includes: acquiring a target frame image collected by a camera at a target time point, and multiple single-frame point clouds respectively collected by a position sensor at multiple time points including the target time point; wherein, the multiple single-frame point clouds
  • the frame point cloud includes the non-target frame point cloud collected at the non-target time point and the target frame point cloud collected at the target time point;
  • a camera calibration device configured with a position sensor and a camera, and includes an acquisition unit, a superposition unit, and a calibration unit: the acquisition A unit, configured to acquire a target frame image collected by the camera at the target time point, and a plurality of single-frame point clouds respectively collected by the position sensor at multiple time points including the target time point; wherein, the multiple single-frame point clouds
  • the frame point cloud includes a non-target frame point cloud collected at a non-target time point and a target frame point cloud collected at a target time point;
  • the overlay unit is configured to superimpose the non-target frame point cloud on the In the target frame point cloud, an integrated frame point cloud corresponding to the target time point is obtained;
  • the calibration unit is configured to perform camera calibration based on the target frame image and the integrated frame point cloud.
  • an electronic device including a camera and a position sensor, a processor, and a memory for storing instructions executable by the processor; wherein, the processor executes the The executable instructions implement the steps in the method described in the first aspect above.
  • the electronic device superimposes one or more non-target frame point clouds collected by the position sensor at one or more non-target time points on the point cloud collected by the position sensor at the target time point.
  • the integrated frame point cloud corresponding to the target time point is obtained, and the electronic device uses the integrated frame point cloud and the target frame image collected by the camera at the target time point to realize the position sensor and camera Joint calibration between, in view of the fact that the integrated frame point cloud contains more dense and farther feature points at the target time point than the target frame point cloud, based on the integrated frame point cloud and the target
  • the accuracy of camera calibration with frame images is thus improved.
  • Fig. 1 is a schematic diagram of camera calibration between a point cloud coordinate system of a position sensor and a pixel coordinate system of a camera according to an exemplary embodiment of the present specification.
  • Fig. 2 is a flowchart of a camera calibration method provided by an exemplary embodiment of the present specification.
  • Fig. 3 is a flowchart of a method for an electronic device to superimpose a point cloud of a non-target frame onto a point cloud of a target frame according to an exemplary embodiment of the present specification.
  • Fig. 4 is a flow chart of a method for camera calibration of an electronic device based on an integrated frame point cloud and a target frame image according to an exemplary embodiment of the present specification.
  • Fig. 5 is a schematic structural diagram of an electronic device in which a camera calibration device is provided according to an exemplary embodiment.
  • Fig. 6 is a block diagram of a camera calibration device provided by an exemplary embodiment.
  • the steps of the corresponding methods are not necessarily performed in the order shown and described in this specification.
  • the method may include more or less steps than those described in this specification.
  • a single step described in this specification may be decomposed into multiple steps for description in other embodiments; multiple steps described in this specification may also be combined into a single step in other embodiments describe.
  • the camera can provide information such as color and texture
  • the position sensor can provide information such as distance and speed
  • a relatively common way is to configure cameras and position sensors on devices such as smart driving cars and drones at the same time to comprehensively utilize the two information collected by the
  • the camera configured on the device can collect two-dimensional images, which contain the color and texture information of objects within the camera's field of view; wherein, the two-dimensional images are constructed based on the preset pixel coordinate system in the camera, and the camera collects
  • the obtained image is composed of a plurality of pixel points in the pixel coordinate system, and the coordinate value of each pixel point is the coordinate value in the pixel coordinate system, and the pixel value of each pixel point can represent information such as color.
  • the position sensor configured on the device can collect a three-dimensional point cloud, which contains the distance and motion state information of the object in the position sensor's field of view; wherein, the three-dimensional point cloud is based on the point cloud coordinates preset in the position sensor System construction, the point cloud collected by the position sensor each time is composed of a plurality of feature points under the point cloud coordinate system, the coordinate value of each feature point is the coordinate value under the point cloud coordinate system, based on the feature point The coordinate value can obtain the information of the distance and motion state of the object.
  • the pixel coordinate system of the camera and the point cloud coordinate system of the position sensor are independent of each other. After the device obtains the 2D image collected by the camera and the 3D point cloud collected by the position sensor, in order to comprehensively utilize the two information, It is necessary to determine the transformation relationship between the pixel coordinate system and the point cloud coordinate system so that the information of the pixel points in the image and the information of the feature points in the point cloud can maintain spatial synchronization, and obtain the representation of this transformation relationship
  • the process of the coordinate transformation matrix is the joint calibration process between the camera and the position sensor, which belongs to the camera calibration.
  • this specification proposes a camera calibration method, which is applied to an electronic device equipped with a camera and a position sensor.
  • the electronic device may be a smart driving car, a drone, or the like.
  • the camera is a camera capable of collecting two-dimensional images, including but not limited to a monocular camera;
  • the position sensor is a sensor capable of collecting a three-dimensional point cloud, including but not limited to single-line lidar and multi-line lidar.
  • FIG. 2 is a flow chart of a camera calibration method shown in an exemplary embodiment of this specification.
  • the camera calibration method may include the following specific steps: Step 202, the electronic device obtains the target frame image collected by the camera at the target time point, and the multiple frame images collected by the position sensor at multiple time points including the target time point respectively.
  • a single-frame point cloud, the multiple single-frame point clouds include a non-target frame point cloud collected at a non-target time point and a target frame point cloud collected at a target time point.
  • the camera and the position sensor When the electronic device performs camera calibration, the camera and the position sensor will be triggered to collect images and point clouds respectively for a target time point;
  • the target time point can be a time point manually controlled by the technician, for example, the technician passes The switch or button triggers the time point of image and point cloud acquisition.
  • the target time point can also be a time point preset in the electronic device that is automatically triggered after it starts to perform camera calibration. For example, the electronic device starts to perform camera calibration. The timing of image and point cloud acquisition is automatically triggered 10 seconds after calibration.
  • the electronic device will acquire images and point clouds respectively acquired by the camera and the position sensor after the acquisition is completed.
  • the single-frame two-dimensional image collected by the camera at the target time point is constructed based on the established pixel coordinate system of the camera.
  • the pixel coordinate system can be the factory setting of the camera, or it can be manually set.
  • the target frame image may consist of 1920*1080 pixels.
  • the single-frame three-dimensional point cloud synchronously collected by the position sensor at the target time point, that is, the target frame point cloud, and the single-frame three-dimensional point cloud collected at one or more non-target time points outside the target time point Point cloud, i.e. non-target frame point cloud; optionally, the position sensor can collect one or more non-target frames at one or more non-target time points according to a preset time interval after the target time point point cloud.
  • Each single-frame three-dimensional point cloud collected by the position sensor is constructed based on the established point cloud coordinate system of the position sensor.
  • the point cloud coordinate system can be the factory setting of the position sensor, or can be manually set, for example , a single-frame 3D point cloud can be composed of n feature points including three-axis coordinate values of a Cartesian coordinate system.
  • Step 204 the electronic device superimposes the point cloud of the non-target frame into the point cloud of the target frame to obtain a comprehensive frame point cloud collected by the position sensor at the target time point.
  • the multiple single-frame point clouds collected by the position sensor are constructed based on the same point cloud coordinate system. Based on step 202, after the electronic device acquires the multiple single-frame point clouds collected by the position sensor, it can One or more non-target frame point clouds collected at a non-target time point are superimposed on a target frame point cloud collected at a target time point to synthesize information contained in multiple single-frame point clouds that can be converted to the target time point The object information of the target time point is obtained to obtain the comprehensive frame point cloud corresponding to the target time point.
  • the target frame point cloud collected at the target time point includes the range of 0 to 10 meters from the position sensor at the target time point object information within.
  • the non-target frame point cloud collected by the position sensor at the non-target time point 0.1 second after the target time point contains the non-target frame point cloud.
  • Object information within the range of 0 to 10 meters from the position sensor at the target time point; through the conversion of speed and time, the non-target frame point cloud can be obtained from the position sensor 1 to 11 at the target time point object information within a meter range, and superimpose it into the target frame point cloud.
  • the integrated frame point cloud may contain more object information within the range of 0 to 13 meters or more from the position sensor at the target time point, that is, compared with the target frame point cloud, the The integrated frame point cloud includes feature points that are denser at the target time point and located at a greater distance from the position sensor.
  • Step 206 the electronic device performs camera calibration based on the target frame image and the integrated frame point cloud.
  • the electronic device performs joint calibration between the camera and the position sensor based on the target frame image acquired in step 202 and the integrated frame point cloud obtained in step 204 .
  • each set of mapping points includes a pixel point in the target frame image and a feature point in the integrated frame point cloud, between them There is a mapping relationship between them, and the pixel points and the feature points correspond to the same point in the physical world.
  • the multiple sets of mapping points may be determined based on calibration boards, or may be manually selected.
  • the specific way of determining multiple sets of mapping points based on the calibration board can be referred to related technologies, and will not be repeated here; when manually selecting multiple sets of mapping points, it is necessary to select as evenly as possible from the target frame image and the integrated frame point cloud. Mapping points to avoid the decrease of calibration accuracy caused by over-fitting during camera calibration.
  • the electronic device will perform joint calibration between the camera and the position sensor based on multiple sets of mapping points between the target frame image and the integrated frame point cloud. Specifically, the electronic device will determine the pixel coordinate system of the camera and the point of the position sensor based on the coordinate values of the pixel points in each group of mapping points in the pixel coordinate system and the coordinate values of the feature points in the point cloud coordinate system.
  • a coordinate transformation matrix between cloud coordinate systems, the coordinate transformation matrix is used to realize the spatial synchronization between the image collected by the subsequent camera and the point cloud collected by the position sensor after being determined.
  • the electronic device superimposes one or more non-target frame point clouds collected by the position sensor at one or more non-target time points on the point cloud collected by the position sensor at the target time point.
  • the comprehensive frame point cloud corresponding to the target time point is obtained.
  • the comprehensive frame point cloud contains more dense and farther objects at the target time point feature points, avoiding the problem of sparse point cloud feature points and short coverage distance caused by the limited scanning range of the position sensor, based on the integrated frame point cloud and the target frame image collected by the camera at the target time point The joint calibration between the position sensor and the camera is thus more accurate.
  • step 204 For the electronic device described in step 204 to superimpose the point cloud of the non-target frame on the point cloud of the target frame to obtain the comprehensive frame point cloud corresponding to the target time point, there are many optional implementations.
  • step 204 may include the following specific steps: Step 2042, the electronic device determines the relationship between the non-target frame point cloud and the non-target frame point cloud for each non-target frame point cloud Point cloud transformation matrix between target frame point clouds.
  • the point cloud transformation matrix is used to make the non-target frame point cloud approach the target frame point cloud, for example, the point cloud transformation matrix can be a 6-degree-of-freedom transformation matrix; determine the non-target frame point cloud and
  • the point cloud transformation matrix between target frame point clouds can be based on such as ICP (Iterative Closest Point, iterative closest point algorithm), GICP (Generalized Iterative Closest Point, generalized iterative closest point algorithm), NDT (Normal Distributions Transform, normal distribution transformation algorithm) and other point cloud conversion algorithms, and the specific point cloud conversion algorithm used is not limited.
  • the point cloud collected by the position sensor at 0.1 seconds after the target time point is the first non-target frame point cloud
  • the point cloud collected at 0.2 seconds after the target time point is the second non-target frame point cloud
  • the point cloud collected in 0.3 seconds is the third non-target frame point cloud
  • the electronic device can respectively determine the point cloud transformation matrix between the point cloud of the first, second, and third non-target frame and the point cloud of the target frame based on the GICP algorithm.
  • the electronic device can first determine the prediction matrix corresponding to the point cloud of each non-target frame, and determine the relationship between the point cloud of each non-target frame and the point cloud based on the prediction matrix Point cloud transformation matrix between target frame point clouds.
  • the electronic device may first determine an inter-frame transformation matrix between two adjacent frames of point clouds among the multiple single-frame point clouds collected by the position sensor.
  • the electronic device After determining the inter-frame transformation matrix between all adjacent two frame point clouds, for each non-target frame point cloud, the electronic device can be based on the transition matrix between the non-target frame point cloud and the target frame point cloud One or more inter-frame transformation matrices to determine a prediction matrix corresponding to the point cloud of the non-target frame; based on the prediction matrix, the electronic device determines the relationship between the point cloud of the non-target frame and the point cloud of the target frame The point cloud transformation matrix between .
  • the electronic device may first determine the third inter-frame transformation matrix between the third non-target frame point cloud and the second non-target frame point cloud, the second non-target frame point cloud and the first non-target frame point cloud based on the GICP algorithm.
  • the electronic device may determine the predicted value corresponding to the third non-target frame point cloud based on the first, second, and third inter-frame transformation matrices. an estimation matrix, and determine an estimation matrix corresponding to the point cloud of the second non-target frame based on the first and second inter-frame transformation matrices. For example, the electronic device may obtain the prediction matrix by multiplying the inter-frame transformation matrix.
  • the electronic device can use the prediction matrix corresponding to the point cloud of the third non-target frame as an initial value, based on the GICP algorithm in the prediction matrix Iterates on the basis of to determine the point cloud transformation matrix between the third non-target frame point cloud and the target frame point cloud; obtain the point cloud between the second non-target frame point cloud and the target frame point cloud The same is true for the point cloud transformation matrix. It can be understood that the point cloud transformation matrix between the point cloud of the first non-target frame and the point cloud of the target frame can directly adopt the first inter-frame transformation matrix.
  • Step 2044 the electronic device superimposes the point cloud of the non-target frame into the point cloud of the target frame based on the determined point cloud transformation matrix.
  • the feature points in the non-target frame point cloud can be converted into corresponding feature points at the target time point and then superimposed on the target frame point in the cloud.
  • the electronic device may multiply the non-target frame point cloud with its corresponding point cloud transformation matrix, so that the feature points in the non-target frame point cloud are transformed by their coordinate values at the non-target time point to its coordinate value at the target time point, and then superimpose the non-target frame point cloud after multiplication and transformation into the target frame point cloud, so that the information of the target frame point cloud after superposition can be enriched.
  • the position sensor collects information about an object 9.5 meters away from it in the physical world 0.1 seconds after the target time point, that is, 0.1 seconds after the target time point exists in the point cloud of the first non-target frame.
  • a feature point at a distance of 9.5 meters from the position sensor by multiplying the first non-target frame point cloud with its corresponding point cloud transformation matrix, the feature point in the first non-target frame point cloud is transformed It is a feature point 10.5 meters away from the position sensor at the target time point, and the electronic device superimposes the multiplied and converted first non-target frame point cloud into the target frame point cloud, so that after superposition
  • the target frame point cloud includes the feature points that are 10.5 meters away from the position sensor, and the superimposed target frame point cloud can include beyond the scanning range of the position sensor at the target time point object information.
  • step 2042 to step 2044 only superimposes a non-target frame point cloud on the target frame point cloud.
  • the electronic device needs to collect all non-target frame point clouds
  • the point cloud of the target frame is superimposed on the point cloud of the target frame in the above manner, and the integrated frame point cloud is obtained after the superposition of all non-target frame point clouds to the point cloud of the target frame is completed.
  • step 206 may include the following specific steps: step 2062, the electronic device selects from multiple sets of mapping points between the target frame image and the integrated frame point cloud A partial set of mapping points, based on the selected mapping points, determines a coordinate transformation matrix between the target frame image and the integrated frame point cloud.
  • mapping points from the multiple sets of mapping points between the target frame image and the comprehensive frame point cloud.
  • the pixels and feature points in the same set of mapping points represent the same point in the physical world, based on the pixels with a mapping relationship and the coordinate values of the feature points in the pixel coordinate system and the point cloud coordinate system respectively, using a preset coordinate transformation algorithm to determine the coordinate transformation matrix between the pixel coordinate system and the point cloud coordinate system, the coordinate transformation
  • the matrices include a 6-degree-of-freedom transformation matrix.
  • the coordinate transformation algorithm for determining the coordinate transformation matrix based on the mapping points including EPnP (Efficient Perspective-n-Point, 3D-2D point pair motion efficient algorithm), BA (Bundle Adjustment, beam adjustment) optimization, etc.;
  • the coordinate transformation algorithm is not limited.
  • the electronic device may first select 6 sets of mapping points, and determine the coordinate transformation matrix based on the BA optimization algorithm .
  • Step 2064 based on the determined coordinate transformation matrix, the electronic device projects the integrated frame point cloud into the target frame image.
  • the coordinate transformation between the point cloud coordinate system of the position sensor and the pixel coordinate system of the camera can be realized.
  • the feature points are projected into the target frame image to determine whether the feature points are projected on the pixel points that have a mapping relationship with the feature points.
  • the electronic device may project all the feature points in the integrated frame point cloud to the target frame image, or may project the feature points in the integrated frame point cloud belonging to the plurality of sets of mapping points Points are projected into the target frame image.
  • Step 2066 the electronic device determines the projection error, and when the projection error does not meet the camera calibration requirements, selects a new mapping point from the plurality of sets of mapping points, and based on the added and selected mapping points, calculates the coordinates
  • the transformation matrix is re-iterated to obtain a new coordinate transformation matrix, and the projection error is determined again based on the new coordinate transformation matrix.
  • Step 2068 when the projection error meets the camera calibration requirement, the electronic device determines the coordinate transformation matrix as a coordinate transformation matrix for camera calibration.
  • the electronic device can determine the projection error based on the distance deviation between the feature points in the comprehensive frame point cloud and the corresponding pixel points after they are projected onto the target frame image.
  • the electronic device can use a visualization method based on the coordinate
  • the transformation matrix projects the integrated frame point cloud and displays it in the target frame image for technicians to determine projection errors.
  • the electronic device can map from multiple groups Select or re-select new mapping points from the points, and determine a new coordinate transformation matrix again based on the added or re-selected mapping points.
  • the electronic device can add 6 sets of new mapping points to the 50 sets of mapping points, based on the 6 sets of mapping points selected last time and this selection 6 sets of mapping points, use the BA optimization algorithm to iteratively determine the new coordinate transformation matrix; the electronic device can also re-select 10 sets of new mapping points from the 50 sets of mapping points, based on the 10 sets of re-selected this time Mapping points, using the BA optimization algorithm to iteratively determine a new coordinate transformation matrix.
  • the electronic device After re-iteratively determining the new coordinate transformation matrix, the electronic device projects the comprehensive frame point cloud into the target frame image again based on the new coordinate transformation matrix, and checks whether the projection error meets the camera calibration requirements Confirm again, if the camera calibration requirements are still not met, then repeat the above step 2066 until the camera calibration requirements can be met.
  • the determined coordinate transformation matrix can make the projection of the comprehensive frame point cloud on the target frame image meet the camera calibration requirements, it can be determined as the coordinate transformation matrix for subsequent camera calibration.
  • the electronic device may select some groups of mapping points from multiple groups of mapping points, and use the BA optimization algorithm
  • the simpler EPnP algorithm estimates the coordinate transformation matrix to obtain its estimated value, and uses the estimated value as the initial value in the BA optimization algorithm, and executes the above step 2062 on the basis of the estimated value Get the coordinate transformation matrix determined for the first time.
  • the coordinate transformation between the point cloud coordinate system of the position sensor and the pixel coordinate system of the camera in the electronic device actually includes converting the point cloud coordinate system to the camera coordinate system through a coordinate transformation matrix, and then transforming the camera coordinate system Convert to the pixel coordinate system through the internal reference matrix and the distortion matrix of the camera.
  • the point cloud coordinate system and the camera coordinate system are a three-dimensional coordinate system
  • the pixel coordinate system is a two-dimensional coordinate system
  • the internal parameter matrix and the distortion matrix of the camera are used to realize the camera coordinate system
  • the coordinate transformation between the pixel coordinate systems is generally a predetermined matrix in the camera calibration described in this manual, that is, the internal reference matrix and the distortion matrix whose values have been determined are generally used in steps 2062 to 2066 to realize the points Coordinate transformation between the cloud coordinate system and the pixel coordinate system.
  • the electronic device may determine a new coordinate transformation matrix based on the added or re-selected mapping points. Correcting the internal reference matrix and distortion matrix of the camera.
  • Fig. 5 is a schematic structural diagram of an electronic device in which a camera calibration device is provided according to an exemplary embodiment.
  • the device includes a processor 502 , an internal bus 504 , a network interface 506 , a memory 508 and a non-volatile memory 510 , and of course it may also include hardware required by other services.
  • the processor 502 reads a corresponding computer program from the non-volatile memory 510 into the memory 508 and executes it.
  • one or more embodiments of this specification do not exclude other implementations, such as logic devices or a combination of software and hardware, etc., that is to say, the execution subject of the following processing flow is not limited to each A logic unit, which can also be a hardware or logic device.
  • FIG. 6 is a block diagram of a camera calibration device shown in an exemplary embodiment.
  • the camera calibration device shown in FIG. 6 can be applied to the electronic device shown in FIG. 5, and the electronic device is also equipped with a position sensor and a camera.
  • the camera calibration device may include an acquisition unit 610, a superposition unit 620, and a calibration unit 630: the acquisition unit 610 is used to acquire the target frame image collected by the camera at the target time point, and the position sensor includes the target frame image A plurality of single-frame point clouds collected at multiple time points of time points; wherein, the plurality of single-frame point clouds include non-target frame point clouds collected at non-target time points and collected at target time points The target frame point cloud; the superposition unit 620 is used to superimpose the non-target frame point cloud into the target frame point cloud to obtain the comprehensive frame point cloud corresponding to the target time point; the calibration unit 630 , for performing camera calibration based on the target frame image and the integrated frame point cloud.
  • the superimposing unit 620 when the superimposing unit 620 superimposes the non-target frame point cloud on the target frame point cloud to obtain the comprehensive frame point cloud corresponding to the target time point, it is specifically configured to: for each non-target frame point cloud The target frame point cloud, determining the point cloud transformation matrix between the non-target frame point cloud and the target frame point cloud; based on the determined point cloud transformation matrix, superimposing the non-target frame point cloud on the target frame point cloud.
  • the superposition unit 620 is also used to: determine the inter-frame transformation matrix between two adjacent frame point clouds in the plurality of single-frame point clouds; the superposition unit 620 determines the non-target frame point
  • the point cloud transformation matrix between the cloud and the target frame point cloud is specifically used for: for each non-target frame point cloud, based on the one or between the non-target frame point cloud and the target frame point cloud A plurality of inter-frame transformation matrices, determining an estimation matrix corresponding to the non-target frame point cloud; based on the estimation matrix, determining a point cloud transformation matrix between the non-target frame point cloud and the target frame point cloud .
  • each set of mapping points includes a pixel in the target frame image and a feature in the integrated frame point cloud point;
  • the calibration unit 630 when performing camera calibration based on the target frame image and the integrated frame point cloud, is specifically configured to: select a part of the mapping points from the multiple groups of mapping points, and based on the selected mapping points , determine the coordinate transformation matrix between the target frame image and the integrated frame point cloud; based on the determined coordinate transformation matrix, project the integrated frame point cloud into the target frame image; determine the projection error, in the When the projection error does not meet the camera calibration requirements, a new mapping point is selected from the multiple sets of mapping points, and based on the added mapping points, the coordinate transformation matrix is re-iterated to obtain a new coordinate transformation matrix, based on The new coordinate transformation matrix determines the projection error again; when the projection error meets the camera calibration requirements, the coordinate transformation matrix is determined as a coordinate transformation matrix for camera calibration.
  • a typical implementing device is a computer, which may take the form of a personal computer, laptop computer, cellular phone, camera phone, smart phone, personal digital assistant, media player, navigation device, e-mail device, game control device, etc. desktops, tablets, wearables, or any combination of these.
  • a computer includes one or more processors (CPUs), input/output interfaces, network interfaces and memory.
  • processors CPUs
  • input/output interfaces network interfaces
  • memory volatile and non-volatile memory
  • Memory may include non-permanent storage in computer-readable media, in the form of random access memory (RAM) and/or nonvolatile memory such as read-only memory (ROM) or flash RAM. Memory is an example of computer readable media.
  • RAM random access memory
  • ROM read-only memory
  • flash RAM flash random access memory
  • Computer-readable media including both permanent and non-permanent, removable and non-removable media, can be implemented by any method or technology for storage of information.
  • Information may be computer readable instructions, data structures, modules of a program, or other data.
  • Examples of computer storage media include, but are not limited to, phase change memory (PRAM), static random access memory (SRAM), dynamic random access memory (DRAM), other types of random access memory (RAM), read only memory (ROM), Electrically Erasable Programmable Read-Only Memory (EEPROM), Flash memory or other memory technology, Compact Disc Read-Only Memory (CD-ROM), Digital Versatile Disc (DVD) or other optical storage, Magnetic cassettes, disk storage, quantum memory, graphene-based storage media or other magnetic storage devices or any other non-transmission media that can be used to store information that can be accessed by computing devices.
  • computer-readable media excludes transitory computer-readable media, such as modulated data signals and carrier waves.
  • first, second, third, etc. may be used in one or more embodiments of the present specification to describe various information, the information should not be limited to these terms. These terms are only used to distinguish information of the same type from one another. For example, without departing from the scope of one or more embodiments of this specification, first information may also be called second information, and similarly, second information may also be called first information. Depending on the context, the word “if” as used herein may be interpreted as “at” or "when” or "in response to a determination.”

Landscapes

  • Engineering & Computer Science (AREA)
  • Computer Vision & Pattern Recognition (AREA)
  • Physics & Mathematics (AREA)
  • General Physics & Mathematics (AREA)
  • Theoretical Computer Science (AREA)
  • Multimedia (AREA)
  • Image Processing (AREA)
  • Studio Devices (AREA)

Abstract

本说明书一个或多个实施例提供一种相机标定方法及装置；所述方法包括：获取相机在目标时间点采集到的目标帧图像，以及位置传感器在包括目标时间点的多个时间点分别采集到的多个单帧点云；其中，在非目标时间点采集到的单帧点云为非目标帧点云，在目标时间点采集到的单帧点云为目标帧点云；将非目标帧点云叠加至目标帧点云中得到目标时间点对应的综合帧点云；基于目标帧图像和综合帧点云进行相机标定。本方案中，将非目标帧点云叠加至目标帧点云中得到了综合帧点云，其相较于目标帧点云包含更为密集、所处距离更远的特征点，使用所述综合帧点云与目标帧图像实现位置传感器和相机间的联合标定，提高了相机标定的准确性。

Description

用于相机标定的方法及装置 技术领域
本说明书一个或多个实施例涉及计算机视觉技术领域，尤其涉及一种用于相机标定的方法及装置。
背景技术
随着智能驾驶、无人机等应用的不断推广，相机往往需要和各类传感器配合以提供更加准确全面的视觉数据。因为相机能够提供色彩和纹理等信息，而位置传感器能够提供距离和速度等信息，一种比较常见的方式是在智能驾驶汽车和无人机等设备上同时配置相机和位置传感器，从而获得相机采集的二维图像中的物体色彩和纹理信息，以及位置传感器采集的三维点云中的物体距离和运动状态信息。
由于相机所采集图像的像素坐标系与位置传感器所采集点云的点云坐标系是彼此独立的，为了综合使用二者的信息，需要确定所述像素坐标系与所述点云坐标系之间的转换关系，所述转换关系的确定过程就属于相机标定。不过，目前，相关技术中实现位置传感器和相机间联合标定的准确性还有待提高。
发明内容
有鉴于此，本说明书一个或多个实施例提供一种相机标定方法及装置。
为实现上述目的，本说明书一个或多个实施例提供技术方案如下：根据本说明书一个或多个实施例的第一方面，提出了一种相机标定方法，应用于配置有位置传感器和相机的电子设备，包括：获取相机在目标时间点采集到的目标帧图像，以及位置传感器在包括所述目标时间点的多个时间点分别采集到的多个单帧点云；其中，所述多个单帧点云包括，在非目标时间点采集到的非目标帧点云以及在目标时间点采集到的目标帧点云；
将所述非目标帧点云叠加至所述目标帧点云中，得到所述目标时间点对应的综合帧点云；基于所述目标帧图像和所述综合帧点云进行相机标定。
根据本说明书一个或多个实施例的第二方面，提出了一种相机标定装置，所述装置应用于配置有位置传感器和相机的电子设备，包括获取单元、叠加单元和标定单元：所述获取单元，用于获取相机在目标时间点采集到的目标帧图像，以及位置传感器在包括 所述目标时间点的多个时间点分别采集到的多个单帧点云；其中，所述多个单帧点云包括，在非目标时间点采集到的非目标帧点云以及在目标时间点采集到的目标帧点云；所述叠加单元，用于将所述非目标帧点云叠加至所述目标帧点云中，得到所述目标时间点对应的综合帧点云；所述标定单元，用于基于所述目标帧图像和所述综合帧点云进行相机标定。
根据本说明书一个或多个实施例的第三方面，提出了一种电子设备，包括相机和位置传感器、处理器以及用于存储处理器可执行指令的存储器；其中，所述处理器通过运行所述可执行指令实现如上述第一方面所述方法中的步骤。
由以上描述可以看出，本说明书中，电子设备将位置传感器在一或多个非目标时间点采集到的一或多个非目标帧点云叠加至所述位置传感器在目标时间点采集到的目标帧点云中，得到了所述目标时间点对应的综合帧点云，所述电子设备使用所述综合帧点云与相机在所述目标时间点采集到的目标帧图像实现位置传感器和相机间的联合标定，鉴于所述综合帧点云相较于所述目标帧点云包含了在目标时间点上更为密集且距离更远的特征点，基于所述综合帧点云与所述目标帧图像进行的相机标定的准确性因而得以提升。
附图说明
图1是本说明书一示例性实施例示出的在位置传感器的点云坐标系和相机的像素坐标系之间进行相机标定的示意图。
图2是本说明书一示例性实施例提供的一种相机标定方法的流程图。
图3是本说明书一示例性实施例示出的电子设备将非目标帧点云叠加至目标帧点云的方法流程图。
图4是本说明书一示例性实施例示出的电子设备基于综合帧点云和目标帧图像进行相机标定的方法流程图。
图5是一示例性实施例提供的一种相机标定装置所在电子设备的结构示意图。
图6是一示例性实施例提供的一种相机标定装置的框图。
具体实施方式
这里将详细地对示例性实施例进行说明，其示例表示在附图中。下面的描述涉及附 图时，除非另有表示，不同附图中的相同数字表示相同或相似的要素。以下示例性实施例中所描述的实施方式并不代表与本说明书一个或多个实施例相一致的所有实施方式。相反，它们仅是与如所附权利要求书中所详述的、本说明书一个或多个实施例的一些方面相一致的装置和方法的例子。
需要说明的是：在其他实施例中并不一定按照本说明书示出和描述的顺序来执行相应方法的步骤。在一些其他实施例中，其方法所包括的步骤可以比本说明书所描述的更多或更少。此外，本说明书中所描述的单个步骤，在其他实施例中可能被分解为多个步骤进行描述；而本说明书中所描述的多个步骤，在其他实施例中也可能被合并为单个步骤进行描述。
随着智能驾驶、无人机等应用的不断推广，相机往往需要和各类传感器配合以提供更加准确全面的视觉数据。因为相机能够提供色彩和纹理等信息，而位置传感器能够提供距离和速度等信息，一种比较常见的方式是在智能驾驶汽车和无人机等设备上同时配置相机和位置传感器，以综合利用二者采集到的信息。
设备上配置的相机可以采集二维图像，所述二维图像中含有相机视野内物体的色彩和纹理信息；其中，所述二维图像基于相机中预设的像素坐标系构建，相机每次采集到的图像由所述像素坐标系下的多个像素点构成，每个像素点的坐标值即所述像素坐标系下的坐标值，而每个像素点的像素值则可以表征色彩等信息。
而设备上配置的位置传感器则可以采集三维点云，所述三维点云中含有位置传感器视野内物体的距离和运动状态信息；其中，所述三维点云基于位置传感器中预设的点云坐标系构建，位置传感器每次采集到的点云由所述点云坐标系下的多个特征点构成，每个特征点的坐标值即所述点云坐标系下的坐标值，基于特征点的坐标值可以得到物体距离和运动状态的信息。
虽然部分相机也具有采集视野内物体的距离和运动状态信息的功能，但是都极易受到天气和外界环境影响，鲁棒性差，因而设备中一般也不会使用相机提供的物体距离和运动状态信息。
如图1所示，相机的像素坐标系和位置传感器的点云坐标系是彼此独立的，设备在得到相机采集的二维图像和位置传感器采集的三维点云后，为了综合利用二者信息，需要确定所述像素坐标系和所述点云坐标系之间的转换关系以使所述图像中像素点的信息和所述点云中特征点的信息能够保持空间同步，得到表征这一转换关系的坐标转换矩 阵的过程即相机和位置传感器之间的联合标定过程，该过程属于相机标定。
目前，相关技术中位置传感器和相机之间进行联合相机标定的方法大多较为粗糙，其准确性亟待提升。
有鉴于此，本说明书提出一种相机标定方法，应用于配置有相机和位置传感器的电子设备。
所述电子设备可以是智能驾驶汽车、无人机等。所述相机，即能够采集得到二维图像的相机，包括但不限于单目相机；所述位置传感器，即能够采集得到三维点云的传感器，包括但不限于单线激光雷达、多线激光雷达。
本说明书对于所述电子设备、相机和位置传感器的具体类型不做限制。不过，可以理解的是，装配于电子设备上的相机和位置传感器的相对位置一般是固定的而非动态变化的。
请参考图2，图2为本说明书一示例性实施例示出的一种相机标定方法的流程图。
所述相机标定方法可以包括如下具体步骤：步骤202，电子设备获取相机在目标时间点采集到的目标帧图像，以及位置传感器在包括所述目标时间点的多个时间点分别采集到的多个单帧点云，所述多个单帧点云包括非目标时间点采集到的非目标帧点云以及目标时间点采集到的目标帧点云。
当电子设备执行相机标定时，将针对一个目标时间点触发相机和位置传感器分别进行图像和点云的采集；所述目标时间点可以是由技术人员人工操控的一个时间点，例如，技术人员通过开关或按钮触发图像和点云采集的时间点，所述目标时间点也可以是预设于所述电子设备中在其开始执行相机标定后自动触发的一个时间点，例如，电子设备开始执行相机标定后10秒自动触发图像和点云采集的时间点。
所述电子设备将在所述相机和所述位置传感器完成采集后获取二者所分别采集到的图像和点云。
所述相机在目标时间点上采集到的单帧二维图像，即目标帧图像，它基于相机既定的像素坐标系构建，所述像素坐标系可以是相机的出厂设定，也可以是人工设定，举例来说，所述目标帧图像可以由1920*1080个像素点构成。
所述位置传感器在所述目标时间点上同步采集到的单帧三维点云，即目标帧点云，在所述目标时间点外的一或多个非目标时间点上采集到的单帧三维点云，即非目标帧点 云；可选择地，所述位置传感器可以在目标时间点后，根据预设的时间间隔，在一或多个非目标时间点上采集一或多个非目标帧点云。
所述位置传感器采集到的各个单帧三维点云，基于位置传感器既定的点云坐标系构建，所述点云坐标系可以是位置传感器的出厂设定，也可以是人工设定，举例来说，一个单帧三维点云可以由n个包括直角坐标系三轴坐标值的特征点构成。
步骤204，所述电子设备将所述非目标帧点云叠加至所述目标帧点云中，得到所述位置传感器在目标时间点采集到的综合帧点云。
所述位置传感器采集到的多个单帧点云基于相同的点云坐标系构建，基于步骤202，所述电子设备在获取到所述位置传感器采集到的多个单帧点云后，可以将非目标时间点采集到的一或多个非目标帧点云叠加至目标时间点采集到的一个目标帧点云中，以综合多个单帧点云中所包涵的可转换至目标时间点上的物体信息，得到所述目标时间点对应的综合帧点云。
举例来说，假设电子设备上配置的位置传感器的扫描范围为10米，其在目标时间点上采集到的目标帧点云中包涵所述目标时间点上处于所述位置传感器0至10米范围内的物体信息。
假设电子设备的移动速度为10米/秒，根据预设的时间间隔0.1秒，位置传感器在目标时间点后0.1秒这一非目标时间点上采集到的非目标帧点云中包涵所述非目标时间点上处于所述位置传感器0至10米范围内的物体信息；通过速度和时间的换算，可以由这一非目标帧点云得到所述目标时间点上处于所述位置传感器1至11米范围内的物体信息，并将其叠加至所述目标帧点云中。
以此类推，可以将目标时间点后0.2秒、0.3秒等非目标时间点采集到的非目标帧点云转换并叠加至目标帧点云中，从而得到目标时间点对应的一个综合帧点云，所述综合帧点云中可以包涵所述目标时间点上处于所述位置传感器0至13米或更远范围内、更多的物体信息，即，相较于所述目标帧点云，所述综合帧点云中包含所述目标时间点上更为密集且位于所述位置传感器更远距离的特征点。
步骤206，所述电子设备基于所述目标帧图像和所述综合帧点云进行相机标定。
所述电子设备基于步骤202获取到的目标帧图像以及基于步骤204得到的综合帧点云，进行相机和位置传感器之间的联合标定。
所述目标帧图像和所述综合帧点云之间具有多组映射点，每组映射点包括所述目标 帧图像中的一个像素点以及所述综合帧点云中的一个特征点，它们之间具有映射关系，所述像素点和所述特征点对应于物理世界中的同一点。
所述多组映射点，可以基于标定板确定，也可以人工选取。基于标定板确定多组映射点的具体方式，可以参见相关技术，此处不再赘述；而人工选取多组映射点时，需要在所述目标帧图像和所述综合帧点云中选取尽量均匀的映射点，以避免相机标定时由过拟合造成的标定准确性下降。
所述电子设备将基于所述目标帧图像和所述综合帧点云之间的多组映射点进行相机和位置传感器之间的联合标定。具体地，所述电子设备将基于各组映射点中的像素点在像素坐标系中的坐标值，以及特征点在点云坐标系中的坐标值，确定相机的像素坐标系以及位置传感器的点云坐标系之间的坐标转换矩阵，所述坐标转换矩阵在确定后被用于实现后续相机所采集图像和位置传感器所采集点云之间的空间同步。
由以上描述可以看出，本说明书中，电子设备将位置传感器在一或多个非目标时间点采集到的一或多个非目标帧点云叠加至所述位置传感器在目标时间点采集到的目标帧点云中，得到了所述目标时间点对应的综合帧点云，所述综合帧点云相较于所述目标帧点云包含了在目标时间点上更为密集且距离更远的特征点，避免了因位置传感器扫描范围有限造成的点云特征点稀疏以及覆盖距离范围近的问题，基于所述综合帧点云与相机在所述目标时间点采集到的目标帧图像所进行的位置传感器和相机之间的联合标定因而具有更高的准确性。
针对步骤204所述的电子设备将非目标帧点云叠加至目标帧点云中，得到目标时间点对应的综合帧点云，存在多种可选择的实现方式。
请参考图3，在一种可选择的实现方式下，步骤204可以包括如下具体步骤：步骤2042，所述电子设备针对每个非目标帧点云，确定所述非目标帧点云与所述目标帧点云之间的点云转换矩阵。
为将各个非目标帧点云叠加至目标帧点云中，针对每个非目标帧点云，确定其与目标帧点云之间的点云转换矩阵。所述点云转换矩阵用于使所述非目标帧点云逼近所述目标帧点云，举例来说，所述点云转换矩阵可以为6-自由度转换矩阵；确定非目标帧点云和目标帧点云之间的点云转换矩阵，可以基于诸如ICP(Iterative Closest Point，迭代最近点算法)、GICP(Generalized Iterative Closest Point，泛化迭代最近点算法)、NDT(Normal Distributions Transform，正态分布变换算法)等点云转换算法实现，具体采用 的点云转换算法不做限制。
举例来说，假设位置传感器在目标时间点后0.1秒采集到的为第一非目标帧点云，在目标时间点后0.2秒采集到的为第二非目标帧点云，在目标时间点后0.3秒采集到的为第三非目标帧点云，电子设备可以基于GICP算法分别确定第一、二、三非目标帧点云与目标帧点云之间的点云转换矩阵。
考虑到非目标时间点与目标时间点之间的时间间隔越大，基于预设的点云转换算法确定非目标帧点云与目标帧点云之间的点云转换矩阵的准确性越低的问题，为进一步提高所确定的点云转换矩阵的准确性，所述电子设备可以先行确定各个非目标帧点云对应的预估矩阵，并基于所述预估矩阵确定各个非目标帧点云与目标帧点云之间的点云转换矩阵。
具体地，所述电子设备可以先行确定位置传感器采集到的多个单帧点云中相邻两帧点云之间的帧间转换矩阵。
在确定所有相邻两帧点云之间的帧间转换矩阵后，针对每个非目标帧点云，所述电子设备可以基于所述非目标帧点云与所述目标帧点云之间的一或多个帧间转换矩阵，确定所述非目标帧点云对应的预估矩阵；基于所述预估矩阵，所述电子设备确定所述非目标帧点云与所述目标帧点云之间的点云转换矩阵。
基于前例，所述电子设备可以基于GICP算法，先分别确定第三非目标帧点云与第二非目标帧点云之间的第三帧间转换矩阵、第二非目标帧点云与第一非目标帧点云之间的第二帧间转换矩阵，以及第一非目标帧点云与目标帧点云之间的第一帧间转换矩阵。由于采集相邻两帧点云之间的时间间隔较小，确定所述帧间转换矩阵时可以使用缺省矩阵作为初始值，基于GICP算法在所述缺省矩阵的基础上进行迭代以确定所述帧间转换矩阵。
得到所述第一、第二、第三帧间转换矩阵后，所述电子设备可以基于所述第一、第二、第三帧间转换矩阵确定所述第三非目标帧点云对应的预估矩阵，基于所述第一、第二帧间转换矩阵确定所述第二非目标帧点云对应的预估矩阵。例如，所述电子设备可以通过将所述帧间转换矩阵相乘得到所述预估矩阵。
得到所述第三非目标帧点云对应的预估矩阵后，所述电子设备可以使用所述第三非目标帧点云对应的预估矩阵作为初始值，基于GICP算法在所述预估矩阵的基础上进行迭代以确定所述第三非目标帧点云与所述目标帧点云之间的点云转换矩阵；得到所述第 二非目标帧点云与所述目标帧点云之间的点云转换矩阵的方式亦然，可以理解的是，所述第一非目标帧点云与所述目标帧点云之间的点云转换矩阵可以直接采用所述第一帧间转换矩阵。
步骤2044，所述电子设备基于确定的点云转换矩阵，将所述非目标帧点云叠加至所述目标帧点云中。
基于步骤2042得到的非目标帧点云与目标帧点云之间的点云转换矩阵，可以将非目标帧点云中的特征点转换为目标时间点上对应的特征点再叠加至目标帧点云中。
可选择地，所述电子设备可以将非目标帧点云与其对应的点云转换矩阵相乘，以使所述非目标帧点云中的特征点由其在非目标时间点上的坐标值转换至其在目标时间点上的坐标值，然后将相乘转换后的非目标帧点云叠加至目标帧点云中，以使所述目标帧点云在叠加后的信息得以丰富。
基于前例，假设所述位置传感器在目标时间点后0.1秒采集到物理世界中与其相距9.5米的一物体信息，即所述第一非目标帧点云中存在所述目标时间点后0.1秒与所述位置传感器相距9.5米的一特征点，通过将所述第一非目标帧点云与其对应的点云转换矩阵相乘，所述第一非目标帧点云中的所述特征点被转换为所述目标时间点时上与所述位置传感器相距10.5米的一特征点，所述电子设备将相乘转换后的第一非目标帧点云叠加至目标帧点云中，以使叠加后的所述目标帧点云中包括所述与位置传感器相距10.5米的特征点，所述叠加后的所述目标帧点云中即可包涵在所述目标时间点上超出所述位置传感器扫描范围的物体信息。
要说明的是，上述步骤2042至步骤2044只是将一个非目标帧点云叠加至目标帧点云中，为得到目标时间点对应的综合帧点云，电子设备需要将位置传感器采集到的所有非目标帧点云均以上述方式叠加至所述目标帧点云中，完成所有非目标帧点云至所述目标帧点云的叠加后得到的才是所述综合帧点云。
针对步骤206所述的电子设备基于目标帧图像和综合帧点云进行相机标定，存在多种可选择的实现方式。
请参考图4，在一种可选择的实现方式下，步骤206可以包括如下具体步骤：步骤2062，所述电子设备从所述目标帧图像和综合帧点云之间的多组映射点中选取部分组映射点，基于选取的映射点，确定所述目标帧图像和所述综合帧点云之间的坐标转换矩阵。
从目标帧图像和综合帧点云之间具有的多组映射点中选取部分组映射点，同组映射 点中的像素点和特征点表征物理世界中的同一点，基于具有映射关系的像素点和特征点分别在像素坐标系中和点云坐标系中的坐标值，利用预设的坐标转换算法确定所述像素坐标系以及所述点云坐标系之间的坐标转换矩阵，所述坐标转换矩阵包括6-自由度转换矩阵。
基于映射点确定坐标转换矩阵的坐标转换算法，包括EPnP(Efficient Pespective-n-Point，3D-2D点对运动高效求解算法)、BA(Bundle Adjustment，光束平差)优化等；本申请对于具体采用的坐标转换算法不做限制。
举例来说，假设预先确定了所述综合帧点云与所述目标帧图像之间的50组映射点，所述电子设备可以先行选取6组映射点，基于BA优化算法确定所述坐标转换矩阵。
步骤2064，基于确定的坐标转换矩阵，所述电子设备将所述综合帧点云投影至所述目标帧图像中。
基于本次确定的坐标转换矩阵，可以实现位置传感器的点云坐标系和相机的像素坐标系之间的坐标转换，所述电子设备将基于本次确定的坐标转换矩阵将综合帧点云中的特征点投影至目标帧图像中，以确定特征点是否投影于与所述特征点具有映射关系的像素点上。可选择地，所述电子设备可以将所述综合帧点云中的特征点均投影至所述目标帧图像中，也可以将所述综合帧点云中属于所述多组映射点中的特征点投影至所述目标帧图像中。
步骤2066，所述电子设备确定投影误差，在所述投影误差不满足相机标定要求时，从所述多组映射点中增加选取新的映射点，基于增加选取后的映射点，对所述坐标转换矩阵进行重新迭代得到新的坐标转换矩阵，基于所述新的坐标转换矩阵再次对投影误差进行确定。
步骤2068，在所述投影误差满足相机标定要求时，所述电子设备确定所述坐标转换矩阵为用于相机标定的坐标转换矩阵。
所述电子设备可以基于综合帧点云中的特征点在投影至目标帧图像后与其对应的像素点之间的距离偏差确定投影误差，可选择地，所述电子设备可以采用可视化方式，基于坐标转换矩阵将所述综合帧点云投影并显示于所述目标帧图像中以供技术人员确定投影误差。
在投影误差不满足相机标定要求时，即所述特征点在投影至目标帧图像后与其对应的像素点之间的距离偏差超出相机标定要求的距离偏差时，所述电子设备可以从多组映 射点中增加选取或重新选取新的映射点，并基于增加选取或重新选取后的映射点再次确定新的坐标转换矩阵。
基于前例，所述电子设备可以在已选取的6组映射点之外，再于所述50组映射点中增加选取6组新的映射点，基于上一次选取的6组映射点以及本次选取的6组映射点，利用BA优化算法重新迭代确定新的坐标转换矩阵；所述电子设备也可以于所述50组映射点中重新选取10组新的映射点，基于本次重新选取的10组映射点，利用BA优化算法重新迭代确定新的坐标转换矩阵。
在重新迭代确定新的坐标转换矩阵后，所述电子设备基于所述新的坐标转换矩阵，再次将所述综合帧点云投影至所述目标帧图像中，并对投影误差是否满足相机标定要求进行再次确定，若仍不满足相机标定要求，则重复上述步骤2066直至能够满足相机标定要求。
若所确定的坐标转换矩阵能够使综合帧点云在目标帧图像上的投影满足相机标定要求，则可以确定其为后续用于相机标定的坐标转换矩阵。
为进一步提高所确定的坐标转换矩阵的准确性，在一种可选择的实现方式下，执行步骤2062之前，所述电子设备可以从多组映射点中选取部分组映射点，利用较BA优化算法更为简单的EPnP算法，对坐标转换矩阵进行预估以得到其预估值，并以所述预估值作为BA优化算法中的初始值，在所述预估值的基础上执行上述步骤2062得到首次确定的坐标转换矩阵。
电子设备中实现位置传感器的点云坐标系和相机的像素坐标系之间的坐标转换，实际上包括将所述点云坐标系通过坐标转换矩阵转换至相机坐标系，再将所述相机坐标系通过所述相机的内参矩阵和畸变矩阵转换至所述像素坐标系。其中，所述点云坐标系和所述相机坐标系为三维坐标系，所述像素坐标系为二维坐标系；所述相机的内参矩阵和畸变矩阵用于实现相机内部所述相机坐标系和所述像素坐标系之间的坐标转换，它们在本说明书所述的相机标定中一般为既定矩阵，即步骤2062至步骤2066中一般使用已经确定数值的所述内参矩阵和畸变矩阵实现所述点云坐标系和所述像素坐标系之间的坐标转换。不过，为了进一步提高相机标定的准确性，在一种可选择的实现方式下，执行步骤2064时，所述电子设备可以基于增加选取或重新选取的映射点，在确定新的坐标转换矩阵的同时对所述相机的内参矩阵和畸变矩阵进行修正。
图5是一示例性实施例提供的一种相机标定装置所在电子设备的示意结构图。请参 考图5，在硬件层面，该设备包括处理器502、内部总线504、网络接口506、内存508以及非易失性存储器510，当然还可能包括其他业务所需要的硬件。本说明书一个或多个实施例可以基于软件方式来实现，比如由处理器502从非易失性存储器510中读取对应的计算机程序到内存508中然后运行。当然，除了软件实现方式之外，本说明书一个或多个实施例并不排除其他实现方式，比如逻辑器件抑或软硬件结合的方式等等，也就是说以下处理流程的执行主体并不限定于各个逻辑单元，也可以是硬件或逻辑器件。
请参考图6，图6为一示例性实施例示出的一种相机标定装置的框图。
图6所示的相机标定装置可以应用于如图5所示的电子设备中，所述电子设备还配置有位置传感器和相机。其中，所述相机标定装置可以包括获取单元610、叠加单元620和标定单元630：所述获取单元610，用于获取相机在目标时间点采集到的目标帧图像，以及位置传感器在包括所述目标时间点的多个时间点分别采集到的多个单帧点云；其中，所述多个单帧点云包括，在非目标时间点采集到的非目标帧点云以及在目标时间点采集到的目标帧点云；所述叠加单元620，用于将所述非目标帧点云叠加至所述目标帧点云中，得到所述目标时间点对应的综合帧点云；所述标定单元630，用于基于所述目标帧图像和所述综合帧点云进行相机标定。
可选择地，所述叠加单元620在将所述非目标帧点云叠加至所述目标帧点云中，得到所述目标时间点对应的综合帧点云时，具体用于：针对每个非目标帧点云，确定所述非目标帧点云与所述目标帧点云之间的点云转换矩阵；基于确定的点云转换矩阵，将所述非目标帧点云叠加至所述目标帧点云中。
进一步地，所述叠加单元620，还用于：确定所述多个单帧点云中相邻两帧点云之间的帧间转换矩阵；所述叠加单元620在确定所述非目标帧点云与所述目标帧点云之间的点云转换矩阵时，具体用于：针对每个非目标帧点云，基于所述非目标帧点云与所述目标帧点云之间的一或多个帧间转换矩阵，确定所述非目标帧点云对应的预估矩阵；基于所述预估矩阵，确定所述非目标帧点云与所述目标帧点云之间的点云转换矩阵。
可选择地，所述目标帧图像和所述综合帧点云之间具有多组映射点，每组映射点包括所述目标帧图像中的一个像素点以及所述综合帧点云中的一个特征点；所述标定单元630，在基于所述目标帧图像和所述综合帧点云进行相机标定时，具体用于：从所述多组映射点中选取部分组映射点，基于选取的映射点，确定所述目标帧图像和所述综合帧点云之间的坐标转换矩阵；基于确定的坐标转换矩阵，将所述综合帧点云投影至所述目标帧图像中；确定投影误差，在所述投影误差不满足相机标定要求时，从所述多组映射 点中增加选取新的映射点，基于增加选取后的映射点，对所述坐标转换矩阵进行重新迭代得到新的坐标转换矩阵，基于所述新的坐标转换矩阵再次对投影误差进行确定；在所述投影误差满足相机标定要求时，确定所述坐标转换矩阵为用于相机标定的坐标转换矩阵。
上述实施例阐明的系统、装置、模块或单元，具体可以由计算机芯片或实体实现，或者由具有某种功能的产品来实现。一种典型的实现设备为计算机，计算机的具体形式可以是个人计算机、膝上型计算机、蜂窝电话、相机电话、智能电话、个人数字助理、媒体播放器、导航设备、电子邮件收发设备、游戏控制台、平板计算机、可穿戴设备或者这些设备中的任意几种设备的组合。
在一个典型的配置中，计算机包括一个或多个处理器(CPU)、输入/输出接口、网络接口和内存。
内存可能包括计算机可读介质中的非永久性存储器，随机存取存储器(RAM)和/或非易失性内存等形式，如只读存储器(ROM)或闪存(flash RAM)。内存是计算机可读介质的示例。
计算机可读介质包括永久性和非永久性、可移动和非可移动媒体可以由任何方法或技术来实现信息存储。信息可以是计算机可读指令、数据结构、程序的模块或其他数据。计算机的存储介质的例子包括，但不限于相变内存(PRAM)、静态随机存取存储器(SRAM)、动态随机存取存储器(DRAM)、其他类型的随机存取存储器(RAM)、只读存储器(ROM)、电可擦除可编程只读存储器(EEPROM)、快闪记忆体或其他内存技术、只读光盘只读存储器(CD-ROM)、数字多功能光盘(DVD)或其他光学存储、磁盒式磁带、磁盘存储、量子存储器、基于石墨烯的存储介质或其他磁性存储设备或任何其他非传输介质，可用于存储可以被计算设备访问的信息。按照本文中的界定，计算机可读介质不包括暂存电脑可读媒体(transitory media)，如调制的数据信号和载波。
还需要说明的是，术语“包括”、“包含”或者其任何其他变体意在涵盖非排他性的包含，从而使得包括一系列要素的过程、方法、商品或者设备不仅包括那些要素，而且还包括没有明确列出的其他要素，或者是还包括为这种过程、方法、商品或者设备所固有的要素。在没有更多限制的情况下，由语句“包括一个……”限定的要素，并不排除在包括所述要素的过程、方法、商品或者设备中还存在另外的相同要素。
上述对本说明书特定实施例进行了描述。其它实施例在所附权利要求书的范围内。 在一些情况下，在权利要求书中记载的动作或步骤可以按照不同于实施例中的顺序来执行并且仍然可以实现期望的结果。另外，在附图中描绘的过程不一定要求示出的特定顺序或者连续顺序才能实现期望的结果。在某些实施方式中，多任务处理和并行处理也是可以的或者可能是有利的。
在本说明书一个或多个实施例使用的术语是仅仅出于描述特定实施例的目的，而非旨在限制本说明书一个或多个实施例。在本说明书一个或多个实施例和所附权利要求书中所使用的单数形式的“一种”、“所述”和“该”也旨在包括多数形式，除非上下文清楚地表示其他含义。还应当理解，本文中使用的术语“和/或”是指并包含一个或多个相关联的列出项目的任何或所有可能组合。
应当理解，尽管在本说明书一个或多个实施例可能采用术语第一、第二、第三等来描述各种信息，但这些信息不应限于这些术语。这些术语仅用来将同一类型的信息彼此区分开。例如，在不脱离本说明书一个或多个实施例范围的情况下，第一信息也可以被称为第二信息，类似地，第二信息也可以被称为第一信息。取决于语境，如在此所使用的词语“如果”可以被解释成为“在……时”或“当……时”或“响应于确定”。
以上所述仅为本说明书一个或多个实施例的较佳实施例而已，并不用以限制本说明书一个或多个实施例，凡在本说明书一个或多个实施例的精神和原则之内，所做的任何修改、等同替换、改进等，均应包含在本说明书一个或多个实施例保护的范围之内。

Claims (10)

  1. 一种相机标定方法，其特征在于，所述方法应用于配置有位置传感器和相机的电子设备，包括：
    获取相机在目标时间点采集到的目标帧图像，以及位置传感器在包括所述目标时间点的多个时间点分别采集到的多个单帧点云；其中，所述多个单帧点云包括在非目标时间点采集到的非目标帧点云、以及在目标时间点采集到的目标帧点云；
    将所述非目标帧点云叠加至所述目标帧点云中，得到所述目标时间点对应的综合帧点云；
    基于所述目标帧图像和所述综合帧点云进行相机标定。
  2. 根据权利要求1所述的方法，其特征在于，所述将所述非目标帧点云叠加至所述目标帧点云中，得到所述目标时间点对应的综合帧点云，包括：
    针对每个非目标帧点云，
    确定所述非目标帧点云与所述目标帧点云之间的点云转换矩阵；
    基于确定的点云转换矩阵，将所述非目标帧点云叠加至所述目标帧点云中。
  3. 根据权利要求2所述的方法，其特征在于，
    所述方法还包括：
    确定所述多个单帧点云中相邻两帧点云之间的帧间转换矩阵；
    所述确定所述非目标帧点云与所述目标帧点云之间的点云转换矩阵，包括：
    针对每个非目标帧点云，
    基于所述非目标帧点云与所述目标帧点云之间的一或多个帧间转换矩阵，确定所述非目标帧点云对应的预估矩阵；
    基于所述预估矩阵，确定所述非目标帧点云与所述目标帧点云之间的点云转换矩阵。
  4. 根据权利要求1所述的方法，其特征在于，所述目标帧图像和所述综合帧点云之间具有多组映射点，每组映射点包括所述目标帧图像中的一个像素点以及所述综合帧点云中的一个特征点；
    所述基于所述目标帧图像和所述综合帧点云进行相机标定，包括：
    从所述多组映射点中选取部分组映射点，基于选取的映射点，确定所述目标帧图像和所述综合帧点云之间的坐标转换矩阵；
    基于确定的坐标转换矩阵，将所述综合帧点云投影至所述目标帧图像中；
    确定投影误差；
    在所述投影误差不满足相机标定要求时，
    从所述多组映射点中增加选取新的映射点，
    基于增加选取后的映射点，对所述坐标转换矩阵进行重新迭代得到新的坐标转换矩阵，
    基于所述新的坐标转换矩阵再次对投影误差进行确定；
    在所述投影误差满足相机标定要求时，确定所述坐标转换矩阵为用于相机标定的坐标转换矩阵。
  5. 根据权利要求1所述的方法，其特征在于，所述位置传感器包括单线激光雷达、多线激光雷达。
  6. 一种相机标定装置，其特征在于，所述装置应用于配置有位置传感器和相机的电子设备，包括获取单元、叠加单元和标定单元：
    所述获取单元，用于获取相机在目标时间点采集到的目标帧图像，以及位置传感器在包括所述目标时间点的多个时间点分别采集到的多个单帧点云；其中，所述多个单帧点云包括，在非目标时间点采集到的非目标帧点云以及在目标时间点采集到的目标帧点云；
    所述叠加单元，用于将所述非目标帧点云叠加至所述目标帧点云中，得到所述目标时间点对应的综合帧点云；
    所述标定单元，用于基于所述目标帧图像和所述综合帧点云进行相机标定。
  7. 根据权利要求6所述的装置，其特征在于，所述叠加单元在将所述非目标帧点云叠加至所述目标帧点云中，得到所述目标时间点对应的综合帧点云时，具体用于：
    针对每个非目标帧点云，
    确定所述非目标帧点云与所述目标帧点云之间的点云转换矩阵；
    基于确定的点云转换矩阵，将所述非目标帧点云叠加至所述目标帧点云中。
  8. 根据权利要求7所述的装置，其特征在于，
    所述叠加单元，还用于：
    确定所述多个单帧点云中相邻两帧点云之间的帧间转换矩阵；
    所述叠加单元在确定所述非目标帧点云与所述目标帧点云之间的点云转换矩阵时，具体用于：
    针对每个非目标帧点云，
    基于所述非目标帧点云与所述目标帧点云之间的一或多个帧间转换矩阵，确定所述非目标帧点云对应的预估矩阵；
    基于所述预估矩阵，确定所述非目标帧点云与所述目标帧点云之间的点云转换矩阵。
  9. 根据权利要求6所述的装置，其特征在于，所述目标帧图像和所述综合帧点云之间具有多组映射点，每组映射点包括所述目标帧图像中的一个像素点以及所述综合帧点云中的一个特征点；
    所述标定单元，在基于所述目标帧图像和所述综合帧点云进行相机标定时，具体用于：
    从所述多组映射点中选取部分组映射点，
    基于选取的映射点，确定所述目标帧图像和所述综合帧点云之间的坐标转换矩阵；
    基于确定的坐标转换矩阵，将所述综合帧点云投影至所述目标帧图像中；
    确定投影误差；
    在所述投影误差不满足相机标定要求时，
    从所述多组映射点中增加选取新的映射点，
    基于增加选取后的映射点，对所述坐标转换矩阵进行重新迭代得到新的坐标转换矩阵，
    基于所述新的坐标转换矩阵再次对投影误差进行确定；
    在所述投影误差满足相机标定要求时，确定所述坐标转换矩阵为用于相机标定的坐标转换矩阵。
  10. 一种电子设备，其特征在于，包括：
    相机和位置传感器；
    处理器；
    用于存储处理器可执行指令的存储器；
    其中，所述处理器通过运行所述可执行指令以实现如权利要求1至5中任一项所述方法中的步骤。
PCT/CN2022/070534 2021-06-23 2022-01-06 用于相机标定的方法及装置 WO2022267444A1 (zh)

Applications Claiming Priority (2)

Application Number Priority Date Filing Date Title
CN202110699270.6A CN115239815B (zh) 2021-06-23 2021-06-23 一种相机标定方法及装置
CN202110699270.6 2021-06-23

Publications (1)

Publication Number Publication Date
WO2022267444A1 true WO2022267444A1 (zh) 2022-12-29

Family

ID=83666319

Family Applications (1)

Application Number Title Priority Date Filing Date
PCT/CN2022/070534 WO2022267444A1 (zh) 2021-06-23 2022-01-06 用于相机标定的方法及装置

Country Status (2)

Country Link
CN (1) CN115239815B (zh)
WO (1) WO2022267444A1 (zh)

Citations (3)

* Cited by examiner, † Cited by third party
Publication number Priority date Publication date Assignee Title
CN109949371A (zh) * 2019-03-18 2019-06-28 北京智行者科技有限公司 一种用于激光雷达和相机数据的标定方法
CN112017251A (zh) * 2020-10-19 2020-12-01 杭州飞步科技有限公司 标定方法、装置、路侧设备和计算机可读存储介质
CN112698302A (zh) * 2020-12-16 2021-04-23 南京航空航天大学 一种颠簸路况下的传感器融合目标检测方法

Family Cites Families (6)

* Cited by examiner, † Cited by third party
Publication number Priority date Publication date Assignee Title
CN108648240B (zh) * 2018-05-11 2022-09-23 东南大学 基于点云特征地图配准的无重叠视场相机姿态标定方法
CN111208492B (zh) * 2018-11-21 2022-04-19 长沙智能驾驶研究院有限公司 车载激光雷达外参标定方法及装置、计算机设备及存储介质
CN112446915B (zh) * 2019-08-28 2024-03-29 北京初速度科技有限公司 一种基于图像组的建图方法及装置
CN111127563A (zh) * 2019-12-18 2020-05-08 北京万集科技股份有限公司 联合标定方法、装置、电子设备及存储介质
CN112270713A (zh) * 2020-10-14 2021-01-26 北京航空航天大学杭州创新研究院 标定方法以及装置、存储介质、电子装置
CN112819889B (zh) * 2020-12-30 2024-05-10 浙江大华技术股份有限公司 位置信息的确定方法及装置、存储介质、电子装置

Patent Citations (3)

* Cited by examiner, † Cited by third party
Publication number Priority date Publication date Assignee Title
CN109949371A (zh) * 2019-03-18 2019-06-28 北京智行者科技有限公司 一种用于激光雷达和相机数据的标定方法
CN112017251A (zh) * 2020-10-19 2020-12-01 杭州飞步科技有限公司 标定方法、装置、路侧设备和计算机可读存储介质
CN112698302A (zh) * 2020-12-16 2021-04-23 南京航空航天大学 一种颠簸路况下的传感器融合目标检测方法

Also Published As

Publication number Publication date
CN115239815B (zh) 2023-10-27
CN115239815A (zh) 2022-10-25

Similar Documents

Publication Publication Date Title
US11704833B2 (en) Monocular vision tracking method, apparatus and non-transitory computer-readable storage medium
EP3707674B1 (en) Method and apparatus for performing depth estimation of object
US10586352B2 (en) Camera calibration
CN107705333B (zh) 基于双目相机的空间定位方法及装置
CN113570721B (zh) 三维空间模型的重建方法、装置和存储介质
CN107230225B (zh) 三维重建的方法和装置
US8355565B1 (en) Producing high quality depth maps
US11816810B2 (en) 3-D reconstruction using augmented reality frameworks
CN111862179A (zh) 三维对象建模方法与设备、图像处理装置及介质
US20170278302A1 (en) Method and device for registering an image to a model
JP2018511874A (ja) ３次元モデリング方法及び装置
EP3304500B1 (en) Smoothing 3d models of objects to mitigate artifacts
CN111986472B (zh) 车辆速度确定方法及车辆
CN113643414B (zh) 一种三维图像生成方法、装置、电子设备及存储介质
CN111507132A (zh) 一种定位方法、装置及设备
CN114013449A (zh) 针对自动驾驶车辆的数据处理方法、装置和自动驾驶车辆
US20200134389A1 (en) Rolling shutter rectification in images/videos using convolutional neural networks with applications to sfm/slam with rolling shutter images/videos
US20210118172A1 (en) Target detection method, target detection apparatus, and unmanned aerial vehicle
CN113610702A (zh) 一种建图方法、装置、电子设备及存储介质
WO2022267444A1 (zh) 用于相机标定的方法及装置
JP2016070884A (ja) 映像データ処理方法、映像データ処理装置及び映像データ処理プログラム
CN107845108A (zh) 一种光流值计算方法、装置及电子设备
CN115222815A (zh) 障碍物距离检测方法、装置、计算机设备和存储介质
JP2021131377A (ja) 生成装置、生成方法およびプログラム
JP2005063012A (ja) 全方位カメラ運動と３次元情報の復元方法とその装置及びプログラム並びにこれを記録した記録媒体

Legal Events

Date Code Title Description
121 Ep: the epo has been informed by wipo that ep was designated in this application

Ref document number: 22826955

Country of ref document: EP

Kind code of ref document: A1

NENP Non-entry into the national phase

Ref country code: DE