WO2021098081A1 - Trajectory feature alignment-based multispectral stereo camera self-calibration algorithm - Google Patents

Trajectory feature alignment-based multispectral stereo camera self-calibration algorithm Download PDF

Info

Publication number
WO2021098081A1
WO2021098081A1 PCT/CN2020/077952 CN2020077952W WO2021098081A1 WO 2021098081 A1 WO2021098081 A1 WO 2021098081A1 CN 2020077952 W CN2020077952 W CN 2020077952W WO 2021098081 A1 WO2021098081 A1 WO 2021098081A1
Authority
WO
WIPO (PCT)
Prior art keywords
image
camera
trajectory
matrix
distortion
Prior art date
Application number
PCT/CN2020/077952
Other languages
French (fr)
Chinese (zh)
Inventor
仲维
李豪杰
柳博谦
王智慧
刘日升
罗钟铉
樊鑫
Original Assignee
大连理工大学
Priority date (The priority date is an assumption and is not a legal conclusion. Google has not performed a legal analysis and makes no representation as to the accuracy of the date listed.)
Filing date
Publication date
Application filed by 大连理工大学 filed Critical 大连理工大学
Priority to US17/283,772 priority Critical patent/US11575873B2/en
Publication of WO2021098081A1 publication Critical patent/WO2021098081A1/en

Links

Images

Classifications

    • G—PHYSICS
    • G06—COMPUTING; CALCULATING OR COUNTING
    • G06T—IMAGE DATA PROCESSING OR GENERATION, IN GENERAL
    • G06T7/00—Image analysis
    • G06T7/20—Analysis of motion
    • G06T7/246—Analysis of motion using feature-based methods, e.g. the tracking of corners or segments
    • G—PHYSICS
    • G06—COMPUTING; CALCULATING OR COUNTING
    • G06T—IMAGE DATA PROCESSING OR GENERATION, IN GENERAL
    • G06T7/00—Image analysis
    • G06T7/80—Analysis of captured images to determine intrinsic or extrinsic camera parameters, i.e. camera calibration
    • G—PHYSICS
    • G06—COMPUTING; CALCULATING OR COUNTING
    • G06F—ELECTRIC DIGITAL DATA PROCESSING
    • G06F18/00—Pattern recognition
    • G06F18/20—Analysing
    • G06F18/22—Matching criteria, e.g. proximity measures
    • G—PHYSICS
    • G06—COMPUTING; CALCULATING OR COUNTING
    • G06T—IMAGE DATA PROCESSING OR GENERATION, IN GENERAL
    • G06T5/00—Image enhancement or restoration
    • G06T5/80—Geometric correction
    • G—PHYSICS
    • G06—COMPUTING; CALCULATING OR COUNTING
    • G06T—IMAGE DATA PROCESSING OR GENERATION, IN GENERAL
    • G06T7/00—Image analysis
    • G06T7/20—Analysis of motion
    • G—PHYSICS
    • G06—COMPUTING; CALCULATING OR COUNTING
    • G06T—IMAGE DATA PROCESSING OR GENERATION, IN GENERAL
    • G06T7/00—Image analysis
    • G06T7/30—Determination of transform parameters for the alignment of images, i.e. image registration
    • G06T7/33—Determination of transform parameters for the alignment of images, i.e. image registration using feature-based methods
    • G—PHYSICS
    • G06—COMPUTING; CALCULATING OR COUNTING
    • G06T—IMAGE DATA PROCESSING OR GENERATION, IN GENERAL
    • G06T7/00—Image analysis
    • G06T7/80—Analysis of captured images to determine intrinsic or extrinsic camera parameters, i.e. camera calibration
    • G06T7/85—Stereo camera calibration
    • G—PHYSICS
    • G06—COMPUTING; CALCULATING OR COUNTING
    • G06V—IMAGE OR VIDEO RECOGNITION OR UNDERSTANDING
    • G06V10/00—Arrangements for image or video recognition or understanding
    • G06V10/10—Image acquisition
    • G06V10/12—Details of acquisition arrangements; Constructional details thereof
    • G06V10/14—Optical characteristics of the device performing the acquisition or on the illumination arrangements
    • G06V10/143—Sensing or illuminating at different wavelengths
    • G—PHYSICS
    • G06—COMPUTING; CALCULATING OR COUNTING
    • G06V—IMAGE OR VIDEO RECOGNITION OR UNDERSTANDING
    • G06V10/00—Arrangements for image or video recognition or understanding
    • G06V10/20—Image preprocessing
    • G06V10/24—Aligning, centring, orientation detection or correction of the image
    • G06V10/243—Aligning, centring, orientation detection or correction of the image by compensating for image skew or non-uniform image deformations
    • G—PHYSICS
    • G06—COMPUTING; CALCULATING OR COUNTING
    • G06V—IMAGE OR VIDEO RECOGNITION OR UNDERSTANDING
    • G06V10/00—Arrangements for image or video recognition or understanding
    • G06V10/70—Arrangements for image or video recognition or understanding using pattern recognition or machine learning
    • G06V10/74—Image or video pattern matching; Proximity measures in feature spaces
    • H—ELECTRICITY
    • H04—ELECTRIC COMMUNICATION TECHNIQUE
    • H04N—PICTORIAL COMMUNICATION, e.g. TELEVISION
    • H04N13/00—Stereoscopic video systems; Multi-view video systems; Details thereof
    • H04N13/20—Image signal generators
    • H04N13/204—Image signal generators using stereoscopic image cameras
    • H04N13/246—Calibration of cameras
    • H—ELECTRICITY
    • H04—ELECTRIC COMMUNICATION TECHNIQUE
    • H04N—PICTORIAL COMMUNICATION, e.g. TELEVISION
    • H04N13/00—Stereoscopic video systems; Multi-view video systems; Details thereof
    • H04N13/20—Image signal generators
    • H04N13/204—Image signal generators using stereoscopic image cameras
    • H04N13/25—Image signal generators using stereoscopic image cameras using two or more image sensors with different characteristics other than in their location or field of view, e.g. having different resolutions or colour pickup characteristics; using image signals from one sensor to control the characteristics of another sensor
    • H—ELECTRICITY
    • H04—ELECTRIC COMMUNICATION TECHNIQUE
    • H04N—PICTORIAL COMMUNICATION, e.g. TELEVISION
    • H04N13/00—Stereoscopic video systems; Multi-view video systems; Details thereof
    • H04N13/20—Image signal generators
    • H04N13/204—Image signal generators using stereoscopic image cameras
    • H04N13/254—Image signal generators using stereoscopic image cameras in combination with electromagnetic radiation sources for illuminating objects
    • G—PHYSICS
    • G06—COMPUTING; CALCULATING OR COUNTING
    • G06T—IMAGE DATA PROCESSING OR GENERATION, IN GENERAL
    • G06T2207/00—Indexing scheme for image analysis or image enhancement
    • G06T2207/10—Image acquisition modality
    • G06T2207/10048—Infrared image
    • H—ELECTRICITY
    • H04—ELECTRIC COMMUNICATION TECHNIQUE
    • H04N—PICTORIAL COMMUNICATION, e.g. TELEVISION
    • H04N13/00—Stereoscopic video systems; Multi-view video systems; Details thereof
    • H04N2013/0074—Stereoscopic image analysis
    • H04N2013/0085—Motion estimation from stereoscopic image signals
    • Y—GENERAL TAGGING OF NEW TECHNOLOGICAL DEVELOPMENTS; GENERAL TAGGING OF CROSS-SECTIONAL TECHNOLOGIES SPANNING OVER SEVERAL SECTIONS OF THE IPC; TECHNICAL SUBJECTS COVERED BY FORMER USPC CROSS-REFERENCE ART COLLECTIONS [XRACs] AND DIGESTS
    • Y02—TECHNOLOGIES OR APPLICATIONS FOR MITIGATION OR ADAPTATION AGAINST CLIMATE CHANGE
    • Y02A—TECHNOLOGIES FOR ADAPTATION TO CLIMATE CHANGE
    • Y02A90/00—Technologies having an indirect contribution to adaptation to climate change
    • Y02A90/10—Information and communication technologies [ICT] supporting adaptation to climate change, e.g. for weather forecasting or climate simulation

Definitions

  • the invention belongs to the field of image processing and computer vision, and relates to a multispectral stereo camera self-calibration algorithm based on trajectory feature registration.
  • Infrared is an electromagnetic wave with a wavelength between microwave and visible light, and the wavelength is longer than red light. Any substance above absolute zero (-273.15°C) can produce infrared rays. Infrared images are widely used in different fields such as military and national defense, resource exploration, weather forecasting, environmental monitoring, medical diagnosis and treatment, and marine research because of their ability to observe through fog and rain. Infrared can be used to shoot scenes through mist and smoke, and infrared photography can also be carried out at night.
  • the advantage of infrared camera imaging is that it can also image in extreme scenes (low light, rain, snow, dense fog, etc.), but the disadvantage is low resolution and blurry image details.
  • the advantages of visible light cameras are high resolution and clear image details, but they cannot be imaged in extreme scenes. Therefore, it is of great practical significance to combine the infrared camera and the visible light camera.
  • Stereo vision is an important subject in the field of computer vision. Its purpose is to reconstruct the 3D geometric information of the scene. Binocular stereo vision is an important field of stereo vision. In binocular stereo vision, the left and right cameras are used to simulate two eyes. Calculate the depth image by calculating the difference between the binocular images. Binocular stereo vision has the advantages of high efficiency, high accuracy, simple system structure and low cost. Since binocular stereo vision needs to match the same point on the left and right image capture points, the focal length and image capture center of the two lenses of the camera, as well as the positional relationship between the left and right lenses. In order to obtain the above data, the camera is calibrated. Obtaining the positional relationship between the visible light camera and the infrared camera is called joint calibration.
  • the two lens parameters and relative position parameters of the camera are obtained, but these parameters are not stable.
  • the internal parameters of the camera lens will also change.
  • the positional relationship between the two lenses may change. Therefore, every time you use the camera, you must modify the internal and external parameters, which is self-calibration.
  • the positional relationship between the infrared lens and the visible light lens is corrected by extracting the infrared image characteristics and the visible light image characteristics respectively, that is, the joint self-calibration of the infrared camera and the visible light camera.
  • the imaging of an infrared camera is different from that of a visible light camera, the effective point contrast obtained by directly extracting matching feature points from the two cameras is less.
  • the trajectory of the moving object can be used, because the trajectory of the moving object will not be different due to different camera modes.
  • the invention aims to solve the change of the positional relationship between the infrared camera and the visible light camera due to factors such as temperature, humidity, vibration and the like.
  • a multispectral stereo camera self-calibration algorithm based on trajectory feature registration including the following steps:
  • the correction of the original image in the step 2) specifically includes the following steps:
  • the pixel coordinate system takes the upper left corner of the picture as the origin, and its x-axis and y-axis are parallel to the x-axis and y-axis of the image coordinate system; pixel coordinates The unit of the system is the pixel; the pixel is the basic and indivisible unit of image display; the optical center of the camera is taken as the origin of the image coordinate system, and the distance from the optical center to the image plane is scaled to 1; the relationship between pixel coordinates and normal coordinates as follows:
  • Non-linear distortion can be roughly divided into radial distortion and tangential distortion.
  • the image radial distortion is the position deviation of the image pixel points along the radial direction with the distortion center as the center point, which causes the image formed in the image to be deformed.
  • the general expression of radial distortion is as follows:
  • x d x(1+k 1 r 2 + k 2 r 4 + k 3 r 6 )
  • y d y(1+k 1 r 2 +k 2 r 4 +k 3 r 6 )
  • r 2 x 2 +y 2 , k 1 , k 2 , and k 3 are radial distortion parameters.
  • the tangential distortion of the image is caused by the defect in the manufacturing of the camera that makes the lens itself not parallel to the image plane. It can be quantitatively described as:
  • x d x+(2p 1 xy+p 2 (r 2 +2x 2 ))
  • p 1 and p 2 are tangential distortion coefficients.
  • x d x(1+k 1 r 2 + k 2 r 4 + k 3 r 6 )+(2p 1 xy+p 2 (r 2 +2x 2 ))
  • y d y(1+k 1 r 2 +k 2 r 4 +k 3 r 6 )+(p 1 (r 2 +2y 2 )+2p 2 xy)
  • (x, y) are the normal coordinates in an ideal state
  • (x d , y d ) are the actual normal coordinates with distortion.
  • X l represents the normal coordinates of the infrared camera
  • X r represents the normal coordinates of the visible light camera.
  • obtaining the corresponding points of the best trajectory specifically includes the following steps:
  • the correction of the calibration result in the step 7) specifically includes the following steps:
  • Random Sampling Consistency (RANSAC) to further screen point pairs.
  • K l and K r are the internal parameter matrices of the infrared camera and the visible light camera, respectively.
  • the rotation matrix before distortion is R 0
  • the rotation matrix calculated in the previous step is R
  • the new R new and t new are as follows:
  • the invention solves the change of the positional relationship between the infrared camera and the visible light camera due to factors such as temperature, humidity, vibration and the like. It has the advantages of fast speed, accurate results, and simple operation. Compared with ordinary methods, we use the trajectory of the moving object as the feature required for self-calibration. The advantage of using the trajectory is because it has good cross-modal robustness; in addition, directly matching the trajectory also saves the feature point extraction and matching steps. .
  • Figure 1 is the overall flow chart.
  • Figure 2 shows the calibration flow chart
  • the invention solves the change of the positional relationship between the infrared camera and the visible light camera due to factors such as temperature, humidity, vibration and the like.
  • the detailed description is as follows in conjunction with the drawings and embodiments:
  • Coordinate System The pixel coordinate system takes the upper left corner of the picture as the origin, and its x-axis and y-axis are parallel to the x-axis and y-axis of the image coordinate system, respectively.
  • the unit of the pixel coordinate system is the pixel.
  • the relationship between pixel coordinates and normal coordinates is as follows:
  • Non-linear distortion can be roughly divided into radial distortion and tangential distortion.
  • the image radial distortion is the position deviation of the image pixel points along the radial direction with the distortion center as the center point, which causes the image formed in the image to be deformed.
  • the general expression of radial distortion is as follows:
  • x d x(1+k 1 r 2 + k 2 r 4 + k 3 r 6 )
  • y d y(1+k 1 r 2 +k 2 r 4 +k 3 r 6 )
  • r 2 x 2 +y 2 , k 1 , k 2 , and k 3 are radial distortion parameters.
  • the tangential distortion of the image is caused by the defect in the manufacturing of the camera that makes the lens itself not parallel to the image plane. It can be quantitatively described as:
  • x d x+(2p 1 xy+p 2 (r 2 +2x 2 ))
  • p 1 and p 2 are tangential distortion coefficients.
  • x d x(1+k 1 r 2 + k 2 r 4 + k 3 r 6 )+(2p 1 xy+p 2 (r 2 +2x 2 ))
  • y d y(1+k 1 r 2 +k 2 r 4 +k 3 r 6 )+(p 1 (r 2 +2y 2 )+2p 2 xy)
  • (x, y) are the normal coordinates in an ideal state
  • (x d , y d ) are the actual normal coordinates with distortion.
  • X l represents the normal coordinates of the infrared camera
  • X r represents the normal coordinates of the visible light camera.
  • Random Sampling Consistency (RANSAC) to further screen point pairs.
  • K l and K r are the internal parameter matrices of the infrared camera and the visible light camera, respectively.
  • the rotation matrix before distortion is R 0
  • the rotation matrix calculated in the previous step is R
  • the new R new and t new are as follows:

Landscapes

  • Engineering & Computer Science (AREA)
  • Theoretical Computer Science (AREA)
  • Physics & Mathematics (AREA)
  • General Physics & Mathematics (AREA)
  • Computer Vision & Pattern Recognition (AREA)
  • Multimedia (AREA)
  • Signal Processing (AREA)
  • Data Mining & Analysis (AREA)
  • Evolutionary Computation (AREA)
  • Artificial Intelligence (AREA)
  • General Engineering & Computer Science (AREA)
  • Evolutionary Biology (AREA)
  • Bioinformatics & Computational Biology (AREA)
  • Life Sciences & Earth Sciences (AREA)
  • Bioinformatics & Cheminformatics (AREA)
  • Electromagnetism (AREA)
  • Computing Systems (AREA)
  • Medical Informatics (AREA)
  • Software Systems (AREA)
  • General Health & Medical Sciences (AREA)
  • Databases & Information Systems (AREA)
  • Health & Medical Sciences (AREA)
  • Image Processing (AREA)
  • Geometry (AREA)
  • Studio Devices (AREA)
  • Length Measuring Devices By Optical Means (AREA)

Abstract

A trajectory feature alignment-based multispectral stereo camera self-calibration method, relating to the fields of image processing and computer vision. In the method, an optimal matching point is obtained by extracting and matching an object movement trajectory, and an external parameter is corrected accordingly. Compared with an ordinary method, a trajectory of a moving object is used as a feature required for self-calibration, and the advantage of using the trajectory is that the trajectory has good cross-modal robustness. In addition, directly matching the trajectory further omits feature point extracting and matching steps, the operation is simple and convenient, and the result is accurate.

Description

基于轨迹特征配准的多光谱立体相机自标定算法Multispectral stereo camera self-calibration algorithm based on trajectory feature registration 技术领域Technical field
本发明属于图像处理和计算机视觉领域，涉及基于轨迹特征配准的多光谱立体相机自标定算法。The invention belongs to the field of image processing and computer vision, and relates to a multispectral stereo camera self-calibration algorithm based on trajectory feature registration.
背景技术Background technique
红外线(Infrared)是波长介于微波与可见光之间的电磁波，波长比红光要长。高于绝对零度(-273.15℃)的物质都可以产生红外线。红外图像由于其具有透过雾、雨等进行观察的能力而被广泛用于军事国防、资源勘探、气象预报、环境监测、医学诊治、海洋研究等不同领域。利用红外线可以隔着薄雾和烟雾拍摄景物，而且在夜间也可以进行红外摄影。红外相机成像的优点是在极端场景(低光、雨雪、浓雾等)也可以成像，缺点是分辨率低、图像细节较模糊。相比之下，可见光相机的优点是分辨率高、图像细节清晰，但是在极端场景下不能成像。因此，将红外相机和可见光相机结合起来具有重大的现实意义。Infrared (Infrared) is an electromagnetic wave with a wavelength between microwave and visible light, and the wavelength is longer than red light. Any substance above absolute zero (-273.15°C) can produce infrared rays. Infrared images are widely used in different fields such as military and national defense, resource exploration, weather forecasting, environmental monitoring, medical diagnosis and treatment, and marine research because of their ability to observe through fog and rain. Infrared can be used to shoot scenes through mist and smoke, and infrared photography can also be carried out at night. The advantage of infrared camera imaging is that it can also image in extreme scenes (low light, rain, snow, dense fog, etc.), but the disadvantage is low resolution and blurry image details. In contrast, the advantages of visible light cameras are high resolution and clear image details, but they cannot be imaged in extreme scenes. Therefore, it is of great practical significance to combine the infrared camera and the visible light camera.
立体视觉是计算机视觉领域的重要主题。其目的是重建场景的3D几何信息。双目立体视觉是立体视觉的重要领域。在双目立体视觉中，左右摄像头用于模拟两只眼睛。通过计算双目图像之间的差异来计算深度图像。双目立体视觉具有效率高，准确度高，系统结构简单，成本低的优点。由于双目立体视觉需要匹配左右图像捕获点上的相同点，因此相机两个镜头的焦距和图像捕获中心，以及左右两个镜头之间的位置关系。为了得到以上数据，对相机进行标定。获取可见光相机和红外相机之间的位置关系称为联合标定。Stereo vision is an important subject in the field of computer vision. Its purpose is to reconstruct the 3D geometric information of the scene. Binocular stereo vision is an important field of stereo vision. In binocular stereo vision, the left and right cameras are used to simulate two eyes. Calculate the depth image by calculating the difference between the binocular images. Binocular stereo vision has the advantages of high efficiency, high accuracy, simple system structure and low cost. Since binocular stereo vision needs to match the same point on the left and right image capture points, the focal length and image capture center of the two lenses of the camera, as well as the positional relationship between the left and right lenses. In order to obtain the above data, the camera is calibrated. Obtaining the positional relationship between the visible light camera and the infrared camera is called joint calibration.
在标定过程中获得了相机的两个镜头参数和相对位置参数，但这些参数不稳定。当温度、湿度等发生变化时，相机镜头的内部参数也会发生变化。另外，由于意外的相机碰撞，两个镜头之间的位置关系可能会改变。因此，每次使用摄像机时，都必须修改内部和外部参数，这就是自标定。在已知相机内部参数的情况 下，通过分别提取红外图像特征和可见光图像特征来对红外镜头和可见光镜头的位置关系进行修正，即红外相机与可见光相机的联合自标定。In the calibration process, the two lens parameters and relative position parameters of the camera are obtained, but these parameters are not stable. When temperature, humidity, etc. change, the internal parameters of the camera lens will also change. In addition, due to accidental camera collisions, the positional relationship between the two lenses may change. Therefore, every time you use the camera, you must modify the internal and external parameters, which is self-calibration. When the internal parameters of the camera are known, the positional relationship between the infrared lens and the visible light lens is corrected by extracting the infrared image characteristics and the visible light image characteristics respectively, that is, the joint self-calibration of the infrared camera and the visible light camera.
由于红外相机的成像与可见光相机不同，直接从两台相机提取匹配特征点得到的有效点对比较少。为了解决这个问题，可以利用运动物体的轨迹，这是因为运动物体的轨迹不会因相机模态不同而不同。Since the imaging of an infrared camera is different from that of a visible light camera, the effective point contrast obtained by directly extracting matching feature points from the two cameras is less. In order to solve this problem, the trajectory of the moving object can be used, because the trajectory of the moving object will not be different due to different camera modes.
发明内容Summary of the invention
本发明旨在解决由于温湿度、震动等因素造成红外相机和可见光相机位置关系的改变。使用红外相机和可见光相机同时拍摄一组运动物体。从运动物体中提取并匹配运动轨迹，从而得到红外相机和可见光相机之间的成像关系，并获得若干对应特征点，通过这些特征点来对原有的标定结果进行修正。The invention aims to solve the change of the positional relationship between the infrared camera and the visible light camera due to factors such as temperature, humidity, vibration and the like. Use infrared camera and visible light camera to shoot a group of moving objects at the same time. Extract and match the motion trajectory from the moving object, thus obtain the imaging relationship between the infrared camera and the visible light camera, and obtain a number of corresponding feature points, and use these feature points to correct the original calibration results.
具体技术方案为：基于轨迹特征配准的多光谱立体相机自标定算法，包括步骤如下：The specific technical solution is: a multispectral stereo camera self-calibration algorithm based on trajectory feature registration, including the following steps:
1)使用红外相机和可见光相机同时拍摄一组有运动物体的场景连续帧。1) Use an infrared camera and a visible light camera to simultaneously shoot a set of continuous frames of a scene with moving objects.
2)原图校正：将原图根据红外相机和可见光相机各自内参和原来的外参进行去畸变和双目校正。流程如图2所示。2) Original image correction: De-distortion and binocular correction are performed on the original image according to the respective internal parameters of the infrared camera and the visible light camera and the original external parameters. The process is shown in Figure 2.
3)计算运动物体的轨迹。3) Calculate the trajectory of the moving object.
4)获取最佳轨迹对应点，并据此获取红外图像到可见光图像的变换矩阵。4) Obtain the corresponding points of the best trajectory, and obtain the transformation matrix from the infrared image to the visible light image accordingly.
5)进一步优化轨迹对应点的匹配结果：选取误差较低的配准点对数作为候选特征点对。5) Further optimize the matching results of the corresponding points of the trajectory: select the registration point pairs with lower error as the candidate feature point pairs.
6)判断特征点覆盖区域：将图像分成m*n个格子，如果特征点覆盖到所有格子，则进行下一步，否则继续拍摄图像，重复步骤1)～步骤5)。6) Determine the feature point coverage area: divide the image into m*n grids, if the feature points cover all grids, proceed to the next step, otherwise continue to take the image and repeat steps 1) to 5).
7)修正标定结果：使用所有特征点的图像坐标来计算校正之后的两相机之间的位置关系，然后与原来的外参相叠加。7) Correct the calibration result: use the image coordinates of all the feature points to calculate the positional relationship between the two cameras after correction, and then superimpose it with the original external parameters.
所述步骤2)中原图校正，具体包括以下步骤：The correction of the original image in the step 2) specifically includes the following steps:
2-1)计算图像的像素点对应的正规坐标系下的坐标；像素坐标系以图片的 左上角为原点，其x轴和y轴分别与图像坐标系的x轴和y轴平行；像素坐标系的单位是像素；像素是图像显示的基本且不可分割的单位；以相机的光心作为图像坐标系的原点，且将光心到图像平面的距离缩放到1；像素坐标与正规坐标的关系如下：2-1) Calculate the coordinates in the normal coordinate system corresponding to the pixels of the image; the pixel coordinate system takes the upper left corner of the picture as the origin, and its x-axis and y-axis are parallel to the x-axis and y-axis of the image coordinate system; pixel coordinates The unit of the system is the pixel; the pixel is the basic and indivisible unit of image display; the optical center of the camera is taken as the origin of the image coordinate system, and the distance from the optical center to the image plane is scaled to 1; the relationship between pixel coordinates and normal coordinates as follows:
u＝KXu=KX
Figure PCTCN2020077952-appb-000001
Figure PCTCN2020077952-appb-000001
其中，
Figure PCTCN2020077952-appb-000002
表示图像的像素坐标；
Figure PCTCN2020077952-appb-000003
表示相机的内参矩阵，f x和f y分别表示图像x方向和y方向的焦距，单位是像素，(c x,c y)表示相机的主点位置，即相机中心在图像上的对应位置；
Figure PCTCN2020077952-appb-000004
是正规坐标系下的坐标。已知图像的像素坐标系以及相机的内参计算出像素点对应的正规坐标系，即X＝K -1u； among them,
Figure PCTCN2020077952-appb-000002
Indicates the pixel coordinates of the image;
Figure PCTCN2020077952-appb-000003
Indicates the internal parameter matrix of the camera, f x and f y respectively represent the focal length of the image in the x direction and y direction, the unit is pixel, (c x , c y ) represents the position of the principal point of the camera, that is, the corresponding position of the camera center on the image;
Figure PCTCN2020077952-appb-000004
Are the coordinates in the normal coordinate system. Knowing the pixel coordinate system of the image and the internal parameters of the camera to calculate the normal coordinate system corresponding to the pixel, that is, X=K -1 u;
2-2)去除图像畸变：由于镜头生产工艺的限制，实际情况下的镜头会存在一些失真现象导致非线性的畸变。因此纯线性模型不能完全准确地描述成像几何关系。非线性畸变可大致分为径向畸变和切向畸变。2-2) Removal of image distortion: Due to the limitation of the lens production process, there will be some distortion phenomena in the lens under actual conditions, resulting in non-linear distortion. Therefore, the pure linear model cannot describe the imaging geometric relationship completely and accurately. Non-linear distortion can be roughly divided into radial distortion and tangential distortion.
图像径向畸变是图像像素点以畸变中心为中心点，沿着径向产生的位置偏差，从而导致图像中所成的像发生形变。径向畸变的大致表述如下：The image radial distortion is the position deviation of the image pixel points along the radial direction with the distortion center as the center point, which causes the image formed in the image to be deformed. The general expression of radial distortion is as follows:
x d＝x(1+k 1r 2+k 2r 4+k 3r 6) x d = x(1+k 1 r 2 + k 2 r 4 + k 3 r 6 )
y d＝y(1+k 1r 2+k 2r 4+k 3r 6) y d = y(1+k 1 r 2 +k 2 r 4 +k 3 r 6 )
其中，r 2＝x 2+y 2，k 1、k 2、k 3为径向畸变参数。 Among them, r 2 =x 2 +y 2 , k 1 , k 2 , and k 3 are radial distortion parameters.
图像切向畸变是由于摄像机制造上的缺陷使得透镜本身与图像平面不平行而产生的，可定量描述为：The tangential distortion of the image is caused by the defect in the manufacturing of the camera that makes the lens itself not parallel to the image plane. It can be quantitatively described as:
x d＝x+(2p 1xy+p 2(r 2+2x 2)) x d = x+(2p 1 xy+p 2 (r 2 +2x 2 ))
y d＝y+(p 1(r 2+2y 2)+2p 2xy) y d =y+(p 1 (r 2 +2y 2 )+2p 2 xy)
其中，p 1、p 2为切向畸变系数。 Among them, p 1 and p 2 are tangential distortion coefficients.
综上，畸变前后的坐标关系如下：In summary, the coordinate relationship before and after the distortion is as follows:
x d＝x(1+k 1r 2+k 2r 4+k 3r 6)+(2p 1xy+p 2(r 2+2x 2)) x d = x(1+k 1 r 2 + k 2 r 4 + k 3 r 6 )+(2p 1 xy+p 2 (r 2 +2x 2 ))
y d＝y(1+k 1r 2+k 2r 4+k 3r 6)+(p 1(r 2+2y 2)+2p 2xy) y d = y(1+k 1 r 2 +k 2 r 4 +k 3 r 6 )+(p 1 (r 2 +2y 2 )+2p 2 xy)
其中，(x,y)是理想状态下的正规坐标，(x d,y d)是实际带有畸变的正规坐标。 Among them, (x, y) are the normal coordinates in an ideal state, and (x d , y d ) are the actual normal coordinates with distortion.
2-3)根据原来两相机的旋转关系将两图转回来：已知原来两个相机之间的旋转矩阵R和平移向量t，使得：2-3) Turn the two images back according to the original rotation relationship of the two cameras: Knowing the original rotation matrix R and translation vector t between the two cameras, such that:
X r＝RX l+t X r ＝RX l +t
其中，X l表示红外相机的正规坐标，X r表示可见光相机的正规坐标。将红外图像向R正方向旋转一半的角度，将可见光图像向R反方向旋转一半的角度； Among them, X l represents the normal coordinates of the infrared camera, and X r represents the normal coordinates of the visible light camera. Rotate the infrared image by a half angle in the positive direction of R, and rotate the visible light image by a half angle in the reverse direction of R;
2-4)根据公式u＝KX将去畸旋转后的图像还原至像素坐标系。2-4) Restore the deformed and rotated image to the pixel coordinate system according to the formula u=KX.
所述步骤4)中获取最佳轨迹对应点，具体包括以下步骤：In the step 4), obtaining the corresponding points of the best trajectory specifically includes the following steps:
4-1)随机选择一对轨迹，并重复以下步骤直到误差足够小：4-1) Randomly select a pair of trajectories, and repeat the following steps until the error is small enough:
a.在已选的轨迹对中随机选取4对点；a. Randomly select 4 pairs of points in the selected trajectory pairs;
b.计算红外图像点到可见光图像点的变换矩阵H；b. Calculate the transformation matrix H from infrared image points to visible light image points;
c.加入使用变换矩阵H求得的误差足够小的点对；c. Add a point pair with a sufficiently small error obtained by using the transformation matrix H;
d.重新计算H；d. Recalculate H;
e.计算并评估误差；e. Calculate and evaluate the error;
4-2)加入使用变换矩阵H求得的误差足够小的轨迹对。4-2) Add the trajectory pair obtained by using the transformation matrix H with a sufficiently small error.
4-3)重新计算H。4-3) Recalculate H.
4-4)计算并评估误差，如果误差不够小则重复步骤4-1)。4-4) Calculate and evaluate the error, if the error is not small enough, repeat step 4-1).
所述步骤7)中修正标定结果，具体包括以下步骤：The correction of the calibration result in the step 7) specifically includes the following steps:
7-1)使用随机抽样一致性(RANSAC)对点对做进一步筛选。7-1) Use Random Sampling Consistency (RANSAC) to further screen point pairs.
7-2)求解基础矩阵F和本质矩阵E：红外和可见光对应像素点对u l、u r和基 础矩阵F的关系是： 7-2) Solve the basic matrix F and the essential matrix E: the relationship between the infrared and visible light corresponding pixel pairs u l , u r and the basic matrix F is:
Figure PCTCN2020077952-appb-000005
Figure PCTCN2020077952-appb-000005
将对应点坐标代入上式，构建齐次线性方程组求解F。Substitute the coordinates of the corresponding points into the above formula, construct a homogeneous linear equation system to solve F.
基础矩阵和本质矩阵的关系是：The relationship between the fundamental matrix and the essential matrix is:
Figure PCTCN2020077952-appb-000006
Figure PCTCN2020077952-appb-000006
其中，K l、K r分别是红外相机和可见光相机的内参矩阵。 Among them, K l and K r are the internal parameter matrices of the infrared camera and the visible light camera, respectively.
7-3)从本质矩阵分解出旋转和平移关系：本质矩阵E与旋转R和平移t的关系如下：7-3) Decompose the relationship between rotation and translation from the essential matrix: The relationship between the essential matrix E, rotation R and translation t is as follows:
E＝[t] ×R E=[t] × R
其中[t] ×表示t的叉乘矩阵。 Where [t] × represents the cross product matrix of t.
将E做奇异值分解，得Taking E into singular value decomposition, we get
Figure PCTCN2020077952-appb-000007
Figure PCTCN2020077952-appb-000007
定义两个矩阵Define two matrices
Figure PCTCN2020077952-appb-000008
和
Figure PCTCN2020077952-appb-000009
ZW＝Σ
Figure PCTCN2020077952-appb-000008
with
Figure PCTCN2020077952-appb-000009
ZW=Σ
所以E可以写成以下两种形式So E can be written in the following two forms
(1)E＝UZU TUWV T (1) E = UZU T UWV T
令[t] ×＝UZU T，R＝UWV T Let [t] × = UZU T , R = UWV T
(2)E＝-UZU TUW TV T (2) E=-UZU T UW T V T
令[t] ×＝-UZU T，R＝UW TV T Let [t] × = -UZU T , R = UW T V T
7-4)将分解出的旋转和平移关系叠加到原来的红外相机和可见光相机的位置关系里面；7-4) Superimpose the decomposed rotation and translation relationship into the original position relationship between the infrared camera and the visible light camera;
记去畸变前的旋转矩阵为R 0，平移向量为t 0＝(t x,t y,t z) T；上一步计算出的旋转矩阵为R，平移向量为t＝(t′ x,t′ y,t′ z) T；则新的R new和t new如下： Remember that the rotation matrix before distortion is R 0 , the translation vector is t 0 =(t x ,t y ,t z ) T ; the rotation matrix calculated in the previous step is R, the translation vector is t=(t′ x ,t ′ Y ,t′ z ) T ; then the new R new and t new are as follows:
Figure PCTCN2020077952-appb-000010
Figure PCTCN2020077952-appb-000010
Figure PCTCN2020077952-appb-000011
Figure PCTCN2020077952-appb-000011
此外，还需要将t new乘一个系数，使得t new在x方向上的分量
Figure PCTCN2020077952-appb-000012
In addition, we need to multiply t new by a coefficient so that the component of t new in the x direction
Figure PCTCN2020077952-appb-000012
本发明的有益效果是：The beneficial effects of the present invention are:
本发明解决了由于温湿度、震动等因素造成红外相机和可见光相机位置关系的改变。具有速度快、结果精确、操作简单等优点。相比于普通方法，我们使用运动物体的轨迹作为自标定所需的特征，使用轨迹的优点是因为其具有良好的跨模态鲁棒性；此外直接匹配轨迹还节省了特征点提取和匹配步骤。The invention solves the change of the positional relationship between the infrared camera and the visible light camera due to factors such as temperature, humidity, vibration and the like. It has the advantages of fast speed, accurate results, and simple operation. Compared with ordinary methods, we use the trajectory of the moving object as the feature required for self-calibration. The advantage of using the trajectory is because it has good cross-modal robustness; in addition, directly matching the trajectory also saves the feature point extraction and matching steps. .
附图说明Description of the drawings
图1为整体流程图。Figure 1 is the overall flow chart.
图2为校正流程图。Figure 2 shows the calibration flow chart.
具体实施方式Detailed ways
本发明解决了由于温湿度、震动等因素造成红外相机和可见光相机位置关系的改变。结合附图及实施例详细说明如下：The invention solves the change of the positional relationship between the infrared camera and the visible light camera due to factors such as temperature, humidity, vibration and the like. The detailed description is as follows in conjunction with the drawings and embodiments:
1)使用红外相机和可见光相机同时拍摄一组有运动物体的场景连续帧。1) Use an infrared camera and a visible light camera to simultaneously shoot a set of continuous frames of a scene with moving objects.
2)原图校正：将原图根据红外相机和可见光相机各自内参和原来的外参进行去畸变和双目校正。流程如图2所示。2) Original image correction: De-distortion and binocular correction are performed on the original image according to the respective internal parameters of the infrared camera and the visible light camera and the original external parameters. The process is shown in Figure 2.
2-1)计算图像的像素点对应的正规坐标系下的坐标。其中，正规坐标系是相机坐标系在平面Z＝1的投影；而相机坐标系是以相机的中心作为图像坐标系的原点，以图片方向为XY轴方向，以垂直于图像为Z轴方向的坐标系。像素坐标系以图片的左上角为原点，其x轴和y轴分别与图像坐标系的x轴和y轴平行。像素坐标系的单位是像素。像素坐标与正规坐标的关系如下：2-1) Calculate the coordinates in the normal coordinate system corresponding to the pixels of the image. Among them, the normal coordinate system is the projection of the camera coordinate system on the plane Z=1; while the camera coordinate system is based on the center of the camera as the origin of the image coordinate system, the direction of the picture is the XY axis direction, and the direction perpendicular to the image is the Z axis direction. Coordinate System. The pixel coordinate system takes the upper left corner of the picture as the origin, and its x-axis and y-axis are parallel to the x-axis and y-axis of the image coordinate system, respectively. The unit of the pixel coordinate system is the pixel. The relationship between pixel coordinates and normal coordinates is as follows:
u＝KXu=KX
Figure PCTCN2020077952-appb-000013
Figure PCTCN2020077952-appb-000013
其中，
Figure PCTCN2020077952-appb-000014
表示图像的像素坐标；
Figure PCTCN2020077952-appb-000015
表示相机的内参矩阵，f x和f y分别表示图像x方向和y方向的焦距，单位是像素，(c x,c y)表示相机的主点位置，即相机中心在图像上的对应位置；
Figure PCTCN2020077952-appb-000016
是正规坐标系下的坐标。已知图像的像素坐标系以及相机的内参计算出像素点对应的正规坐标系，即X＝K -1u； among them,
Figure PCTCN2020077952-appb-000014
Indicates the pixel coordinates of the image;
Figure PCTCN2020077952-appb-000015
Indicates the internal parameter matrix of the camera, f x and f y respectively represent the focal length of the image in the x direction and y direction, the unit is pixel, (c x , c y ) represents the position of the principal point of the camera, that is, the corresponding position of the camera center on the image;
Figure PCTCN2020077952-appb-000016
Are the coordinates in the normal coordinate system. Knowing the pixel coordinate system of the image and the internal parameters of the camera to calculate the normal coordinate system corresponding to the pixel, that is, X=K -1 u;
2-2)去除图像畸变：由于镜头生产工艺的限制，实际情况下的镜头会存在一些失真现象导致非线性的畸变。因此纯线性模型不能完全准确地描述成像几何关系。非线性畸变可大致分为径向畸变和切向畸变。2-2) Removal of image distortion: Due to the limitation of the lens production process, there will be some distortion phenomena in the lens under actual conditions, resulting in non-linear distortion. Therefore, the pure linear model cannot describe the imaging geometric relationship completely and accurately. Non-linear distortion can be roughly divided into radial distortion and tangential distortion.
图像径向畸变是图像像素点以畸变中心为中心点，沿着径向产生的位置偏差，从而导致图像中所成的像发生形变。径向畸变的大致表述如下：The image radial distortion is the position deviation of the image pixel points along the radial direction with the distortion center as the center point, which causes the image formed in the image to be deformed. The general expression of radial distortion is as follows:
x d＝x(1+k 1r 2+k 2r 4+k 3r 6) x d = x(1+k 1 r 2 + k 2 r 4 + k 3 r 6 )
y d＝y(1+k 1r 2+k 2r 4+k 3r 6) y d = y(1+k 1 r 2 +k 2 r 4 +k 3 r 6 )
其中，r 2＝x 2+y 2，k 1、k 2、k 3为径向畸变参数。 Among them, r 2 =x 2 +y 2 , k 1 , k 2 , and k 3 are radial distortion parameters.
图像切向畸变是由于摄像机制造上的缺陷使得透镜本身与图像平面不平行而产生的，可定量描述为：The tangential distortion of the image is caused by the defect in the manufacturing of the camera that makes the lens itself not parallel to the image plane. It can be quantitatively described as:
x d＝x+(2p 1xy+p 2(r 2+2x 2)) x d = x+(2p 1 xy+p 2 (r 2 +2x 2 ))
y d＝y+(p 1(r 2+2y 2)+2p 2xy) y d =y+(p 1 (r 2 +2y 2 )+2p 2 xy)
其中，p 1、p 2为切向畸变系数。 Among them, p 1 and p 2 are tangential distortion coefficients.
综上，畸变前后的坐标关系如下：In summary, the coordinate relationship before and after the distortion is as follows:
x d＝x(1+k 1r 2+k 2r 4+k 3r 6)+(2p 1xy+p 2(r 2+2x 2)) x d = x(1+k 1 r 2 + k 2 r 4 + k 3 r 6 )+(2p 1 xy+p 2 (r 2 +2x 2 ))
y d＝y(1+k 1r 2+k 2r 4+k 3r 6)+(p 1(r 2+2y 2)+2p 2xy) y d = y(1+k 1 r 2 +k 2 r 4 +k 3 r 6 )+(p 1 (r 2 +2y 2 )+2p 2 xy)
其中，(x,y)是理想状态下的正规坐标，(x d,y d)是实际带有畸变的正规坐标。 Among them, (x, y) are the normal coordinates in an ideal state, and (x d , y d ) are the actual normal coordinates with distortion.
2-3)根据原来两相机的旋转关系将两图转回来：已知原来两个相机之间的 旋转矩阵R和平移向量t，使得：2-3) Turn the two images back according to the original rotation relationship between the two cameras: the original rotation matrix R and the translation vector t between the two cameras are known, so that:
X r＝RX l+t X r ＝RX l +t
其中，X l表示红外相机的正规坐标，X r表示可见光相机的正规坐标。将红外图像向R正方向旋转一半的角度，将可见光图像向R反方向旋转一半的角度； Among them, X l represents the normal coordinates of the infrared camera, and X r represents the normal coordinates of the visible light camera. Rotate the infrared image by a half angle in the positive direction of R, and rotate the visible light image by a half angle in the reverse direction of R;
2-4)根据公式u＝KX将去畸旋转后的图像还原至像素坐标系。2-4) Restore the deformed and rotated image to the pixel coordinate system according to the formula u=KX.
3)计算运动物体的轨迹。3) Calculate the trajectory of the moving object.
4)获取最佳轨迹对应点，并据此获取红外图像到可见光图像的变换矩阵。4) Obtain the corresponding points of the best trajectory, and obtain the transformation matrix from the infrared image to the visible light image accordingly.
4-1)随机选择一对轨迹，并重复以下步骤直到误差足够小：4-1) Randomly select a pair of trajectories, and repeat the following steps until the error is small enough:
●在已选的轨迹对中随机选取4对点。● Randomly select 4 pairs of points in the selected trajectory pairs.
●计算红外图像点到可见光图像点的变换矩阵H。●Calculate the transformation matrix H from infrared image point to visible light image point.
●加入使用变换矩阵H求得的误差足够小的点对。●Add a point pair with a sufficiently small error obtained by using the transformation matrix H.
●重新计算H。● Recalculate H.
●计算并评估误差。●Calculate and evaluate errors.
4-2)加入使用变换矩阵H求得的误差足够小的轨迹对。4-2) Add the trajectory pair obtained by using the transformation matrix H with a sufficiently small error.
4-3)重新计算H。4-3) Recalculate H.
4-4)计算并评估误差，如果误差不够小则重复步骤4-1)。4-4) Calculate and evaluate the error, if the error is not small enough, repeat step 4-1).
5)进一步优化轨迹对应点的匹配结果：选取误差较低的配准点对数作为候选特征点对。5) Further optimize the matching results of the corresponding points of the trajectory: select the registration point pairs with lower error as the candidate feature point pairs.
6)判断特征点覆盖区域：将图像分成m*n个格子，如果特征点覆盖到所有格子，则进行下一步，否则继续拍摄图像，重复步骤1)～步骤5)。6) Determine the feature point coverage area: divide the image into m*n grids, if the feature points cover all grids, proceed to the next step, otherwise continue to take the image and repeat steps 1) to 5).
7)修正标定结果：使用所有特征点的图像坐标来计算校正之后的两相机之间的位置关系，然后与原来的外参相叠加。7) Correct the calibration result: use the image coordinates of all the feature points to calculate the positional relationship between the two cameras after correction, and then superimpose it with the original external parameters.
7-1)使用随机抽样一致性(RANSAC)对点对做进一步筛选。7-1) Use Random Sampling Consistency (RANSAC) to further screen point pairs.
7-2)求解基础矩阵F和本质矩阵E：红外和可见光对应像素点对u l、u r和基础矩阵F的关系是： 7-2) Solve the basic matrix F and the essential matrix E: the relationship between the infrared and visible light corresponding pixel pairs u l , u r and the basic matrix F is:
Figure PCTCN2020077952-appb-000017
Figure PCTCN2020077952-appb-000017
可以将对应点坐标代入上式，构建齐次线性方程组求解F。The coordinates of the corresponding points can be substituted into the above formula to construct a homogeneous linear equation system to solve F.
基础矩阵和本质矩阵的关系是：The relationship between the fundamental matrix and the essential matrix is:
Figure PCTCN2020077952-appb-000018
Figure PCTCN2020077952-appb-000018
其中，K l、K r分别是红外相机和可见光相机的内参矩阵。 Among them, K l and K r are the internal parameter matrices of the infrared camera and the visible light camera, respectively.
7-3)从本质矩阵分解出旋转和平移关系：本质矩阵E与旋转R和平移t的关系如下：7-3) Decompose the relationship between rotation and translation from the essential matrix: The relationship between the essential matrix E, rotation R and translation t is as follows:
E＝[t] ×R E=[t] × R
其中[t] ×表示t的叉乘矩阵。 Where [t] × represents the cross product matrix of t.
将E做奇异值分解，得Taking E into singular value decomposition, we get
Figure PCTCN2020077952-appb-000019
Figure PCTCN2020077952-appb-000019
定义两个矩阵Define two matrices
Figure PCTCN2020077952-appb-000020
和
Figure PCTCN2020077952-appb-000021
ZW＝Σ
Figure PCTCN2020077952-appb-000020
with
Figure PCTCN2020077952-appb-000021
ZW=Σ
所以E可以写成以下两种形式So E can be written in the following two forms
(1)E＝UZU TUWV T (1) E = UZU T UWV T
令[t] ×＝UZU T，R＝UWV T Let [t] × = UZU T , R = UWV T
(2)E＝-UZU TUW TV T (2) E=-UZU T UW T V T
令[t] ×＝-UZU T，R＝UW TV T Let [t] × = -UZU T , R = UW T V T
7-4)将分解出的旋转和平移关系叠加到原来的红外相机和可见光相机的位置关系里面；7-4) Superimpose the decomposed rotation and translation relationship into the original position relationship between the infrared camera and the visible light camera;
记去畸变前的旋转矩阵为R 0，平移向量为t 0＝(t x,t y,t z) T；上一步计算出的旋转矩阵为R，平移向量为t＝(t′ x,t′ y,t′ z) T；则新的R new和t new如下： Remember that the rotation matrix before distortion is R 0 , the translation vector is t 0 =(t x ,t y ,t z ) T ; the rotation matrix calculated in the previous step is R, the translation vector is t=(t′ x ,t ′ Y ,t′ z ) T ; then the new R new and t new are as follows:
Figure PCTCN2020077952-appb-000022
Figure PCTCN2020077952-appb-000022
Figure PCTCN2020077952-appb-000023
Figure PCTCN2020077952-appb-000023
此外，还需要将t new乘一个系数，使得t new在x方向上的分量
Figure PCTCN2020077952-appb-000024
In addition, we need to multiply t new by a coefficient so that the component of t new in the x direction
Figure PCTCN2020077952-appb-000024

Claims (4)

  1. 基于轨迹特征配准的多光谱立体相机自标定算法，其特征在于，包括下列步骤：The self-calibration algorithm of multispectral stereo camera based on trajectory feature registration is characterized in that it includes the following steps:
    1)使用红外相机和可见光相机同时拍摄一组有运动物体的场景连续帧；1) Use infrared camera and visible light camera to shoot a group of continuous frames with moving objects at the same time;
    2)原图校正：将原图根据红外相机和可见光相机各自内参和原来的外参进行去畸变和双目校正；2) Original image correction: De-distortion and binocular correction are performed on the original image according to the respective internal parameters of the infrared camera and the visible light camera and the original external parameters;
    3)计算运动物体的轨迹；3) Calculate the trajectory of the moving object;
    4)获取最佳轨迹对应点，并据此获取红外图像到可见光图像的变换矩阵；4) Obtain the corresponding points of the best trajectory, and obtain the transformation matrix from infrared image to visible light image accordingly;
    5)进一步优化轨迹对应点的匹配结果：选取误差低的配准点对数作为候选特征点对；5) Further optimize the matching results of the corresponding points of the trajectory: select the registration point pairs with low error as the candidate feature point pairs;
    6)判断特征点覆盖区域：将图像分成m*n个格子，如果特征点覆盖到所有格子，则进行下一步，否则继续拍摄图像，重复步骤1)～步骤5)；6) Determine the feature point coverage area: divide the image into m*n grids, if the feature points cover all grids, proceed to the next step, otherwise continue to take the image and repeat steps 1) to 5);
    7)修正标定结果：使用所有特征点的图像坐标来计算校正之后的两相机之间的位置关系，然后与原来的外参相叠加。7) Correct the calibration result: use the image coordinates of all the feature points to calculate the positional relationship between the two cameras after correction, and then superimpose it with the original external parameters.
  2. 根据权利要求1所述的基于轨迹特征配准的多光谱立体相机自标定算法，其特征在于，步骤2)中原图校正，具体包括以下步骤：The multi-spectral stereo camera self-calibration algorithm based on trajectory feature registration according to claim 1, characterized in that the original image correction in step 2) specifically includes the following steps:
    2-1)计算图像的像素点对应的正规坐标系下的坐标；像素坐标系以图片的左上角为原点，其x轴和y轴分别与图像坐标系的x轴和y轴平行；像素坐标系的单位是像素；以相机的光心作为图像坐标系的原点，且将光心到图像平面的距离缩放到1；像素坐标与正规坐标的关系如下：2-1) Calculate the coordinates in the normal coordinate system corresponding to the pixels of the image; the pixel coordinate system takes the upper left corner of the picture as the origin, and its x-axis and y-axis are parallel to the x-axis and y-axis of the image coordinate system; pixel coordinates The unit of the system is pixels; the optical center of the camera is taken as the origin of the image coordinate system, and the distance from the optical center to the image plane is scaled to 1; the relationship between pixel coordinates and normal coordinates is as follows:
    u＝KXu=KX
    Figure PCTCN2020077952-appb-100001
    Figure PCTCN2020077952-appb-100001
    其中，
    Figure PCTCN2020077952-appb-100002
    表示图像的像素坐标；
    Figure PCTCN2020077952-appb-100003
    表示相机的内参矩阵，f x和f y分别表示图像x方向和y方向的焦距，单位是像素，(c x,c y)表示相机主点的位置；
    Figure PCTCN2020077952-appb-100004
    是正规坐标系下的坐标；已知图像的像素坐标系以及相机的内参计算出像素点对应的正规坐标系，即     among them,
    Figure PCTCN2020077952-appb-100002
    Indicates the pixel coordinates of the image;
    Figure PCTCN2020077952-appb-100003
    Indicates the internal parameter matrix of the camera, f x and f y respectively represent the focal length of the image in the x direction and y direction, the unit is pixel, (c x , c y ) represents the position of the principal point of the camera;
    Figure PCTCN2020077952-appb-100004
    Is the coordinates in the normal coordinate system; the pixel coordinate system of the image and the camera's internal parameters are known to calculate the normal coordinate system corresponding to the pixel point, namely
    X＝K -1u X=K -1 u
    2-2)去除图像畸变：图像径向畸变是图像像素点以畸变中心为中心点，沿着径向产生的位置偏差，从而导致图像中所成的像发生形变；径向畸变的表述如下：2-2) Removal of image distortion: Image radial distortion is the position deviation of image pixels along the radial direction with the distortion center as the center point, which causes the image formed in the image to be deformed; the expression of radial distortion is as follows:
    x d＝x(1+k 1r 2+k 2r 4+k 3r 6) x d = x(1+k 1 r 2 + k 2 r 4 + k 3 r 6 )
    y d＝y(1+k 1r 2+k 2r 4+k 3r 6) y d = y(1+k 1 r 2 +k 2 r 4 +k 3 r 6 )
    其中，r 2＝x 2+y 2，k 1、k 2、k 3为径向畸变参数； Among them, r 2 =x 2 +y 2 , k 1 , k 2 , and k 3 are radial distortion parameters;
    图像切向畸变是由于摄像机制造上的缺陷使得透镜本身与图像平面不平行而产生的，定量描述为：The tangential distortion of the image is caused by the defect in the manufacturing of the camera that makes the lens itself not parallel to the image plane. It is quantitatively described as:
    x d＝x+(2p 1xy+p 2(r 2+2x 2)) x d = x+(2p 1 xy+p 2 (r 2 +2x 2 ))
    y d＝y+(p 1(r 2+2y 2)+2p 2xy) y d =y+(p 1 (r 2 +2y 2 )+2p 2 xy)
    其中，p 1、p 2为切向畸变系数； Among them, p 1 and p 2 are tangential distortion coefficients;
    畸变前后的坐标关系如下：The coordinate relationship before and after the distortion is as follows:
    x d＝x(1+k 1r 2+k 2r 4+k 3r 6)+(2p 1xy+p 2(r 2+2x 2)) x d = x(1+k 1 r 2 + k 2 r 4 + k 3 r 6 )+(2p 1 xy+p 2 (r 2 +2x 2 ))
    y d＝y(1+k 1r 2+k 2r 4+k 3r 6)+(p 1(r 2+2y 2)+2p 2xy) y d = y(1+k 1 r 2 +k 2 r 4 +k 3 r 6 )+(p 1 (r 2 +2y 2 )+2p 2 xy)
    其中，(x,y)是理想状态下的正规坐标，(x d,y d)是实际带有畸变的正规坐标； Among them, (x, y) are the normal coordinates in an ideal state, and (x d , y d ) are the actual normal coordinates with distortion;
    2-3)根据原来两相机的旋转关系将两图转回来：已知原来两个相机之间的旋转矩阵R和平移向量t，使得2-3) Turn the two images back according to the original rotation relationship between the two cameras: the original rotation matrix R and the translation vector t between the two cameras are known, so that
    X r＝RX l+t X r ＝RX l +t
    其中，X l表示红外相机的正规坐标，X r表示可见光相机的正规坐标；将红外图像旋转R正方向一半的角度，将可见光图像旋转R反方向一半的角度； Among them, X l represents the normal coordinates of the infrared camera, X r represents the normal coordinates of the visible light camera; rotate the infrared image by half the angle in the positive direction of R, and rotate the visible light image by half the angle in the reverse direction of R;
    2-4)根据公式u＝KX将去畸旋转后的图像还原至像素坐标系。2-4) Restore the deformed and rotated image to the pixel coordinate system according to the formula u=KX.
  3. 根据权利要求1所述的基于轨迹特征配准的多光谱立体相机自标定算法，其特征在于，步骤4)中获取最佳轨迹对应点，包括以下步骤：The multi-spectral stereo camera self-calibration algorithm based on trajectory feature registration according to claim 1, wherein the step 4) obtaining the best trajectory corresponding point includes the following steps:
    4-1)随机选择一对轨迹，并重复以下步骤直到误差足够小：4-1) Randomly select a pair of trajectories, and repeat the following steps until the error is small enough:
    a.在已选的轨迹对中随机选取4对点；a. Randomly select 4 pairs of points in the selected trajectory pairs;
    b.计算红外图像点到可见光图像点的变换矩阵H；b. Calculate the transformation matrix H from infrared image points to visible light image points;
    c.加入使用变换矩阵H求得的误差足够小的点对；c. Add a point pair with a sufficiently small error obtained by using the transformation matrix H;
    d.重新计算H；d. Recalculate H;
    e.计算并评估误差；e. Calculate and evaluate the error;
    4-2)加入使用变换矩阵H求得的误差足够小的轨迹对；4-2) Add the trajectory pair obtained by using the transformation matrix H with a sufficiently small error;
    4-3)重新计算H；4-3) Recalculate H;
    4-4)计算并评估误差，如果误差不够小则重复步骤4-1)。4-4) Calculate and evaluate the error, if the error is not small enough, repeat step 4-1).
  4. 根据权利要求1所述的基于轨迹特征配准的多光谱立体相机自标定算法，其特征在于，步骤7)中修正标定结果，包括以下步骤：The multispectral stereo camera self-calibration algorithm based on trajectory feature registration according to claim 1, wherein the correction of the calibration result in step 7) comprises the following steps:
    7-1)使用随机抽样一致性对点对做进一步筛选；7-1) Use random sampling consistency to further screen point pairs;
    7-2)求解基础矩阵F和本质矩阵E：红外和可见光对应像素点对u l、u r和基础矩阵F的关系是： 7-2) Solve the basic matrix F and the essential matrix E: the relationship between the infrared and visible light corresponding pixel pairs u l , u r and the basic matrix F is:
    Figure PCTCN2020077952-appb-100005
    Figure PCTCN2020077952-appb-100005
    将对应点坐标代入上式，构建齐次线性方程组求解F；Substitute the coordinates of the corresponding point into the above formula to construct a homogeneous linear equation system to solve F;
    基础矩阵和本质矩阵的关系是：The relationship between the fundamental matrix and the essential matrix is:
    Figure PCTCN2020077952-appb-100006
    Figure PCTCN2020077952-appb-100006
    其中，K l、K r分别是红外相机和可见光相机的内参矩阵； Among them, K l and K r are the internal parameter matrices of the infrared camera and the visible light camera respectively;
    7-3)从本质矩阵分解出旋转和平移关系：本质矩阵E与旋转R和平移t的关系如下：7-3) Decompose the relationship between rotation and translation from the essential matrix: The relationship between the essential matrix E, rotation R and translation t is as follows:
    E＝[t] ×R E=[t] × R
    其中[t] ×表示t的叉乘矩阵； Where [t] × represents the cross product matrix of t;
    将E做奇异值分解，得Taking E into singular value decomposition, we get
    Figure PCTCN2020077952-appb-100007
    Figure PCTCN2020077952-appb-100007
    定义两个矩阵Define two matrices
    Figure PCTCN2020077952-appb-100008
    和
    Figure PCTCN2020077952-appb-100009
    ZW＝Σ
    Figure PCTCN2020077952-appb-100008
    with
    Figure PCTCN2020077952-appb-100009
    ZW=Σ
    所以E写成以下两种形式So E is written in the following two forms
    (1)E＝UZU TUWV T (1) E = UZU T UWV T
    令[t] ×＝UZU T，R＝UWV T Let [t] × = UZU T , R = UWV T
    (2)E＝-UZU TUW TV T (2) E=-UZU T UW T V T
    令[t] ×＝-UZU T，R＝UW TV T Let [t] × = -UZU T , R = UW T V T
    7-4)将分解出的旋转和平移关系叠加到原来的红外相机和可见光相机的位置关系里面；7-4) Superimpose the decomposed rotation and translation relationship into the original position relationship between the infrared camera and the visible light camera;
    记去畸变前的旋转矩阵为R 0，平移向量为t 0＝(t x,t y,t z) T；上一步计算出的旋转矩阵为R，平移向量为t＝(t′ x,t′ y,t′ z) T；则新的R new和t new如下： Remember that the rotation matrix before distortion is R 0 , the translation vector is t 0 =(t x ,t y ,t z ) T ; the rotation matrix calculated in the previous step is R, the translation vector is t=(t′ x ,t ′ Y ,t′ z ) T ; then the new R new and t new are as follows:
    Figure PCTCN2020077952-appb-100010
    Figure PCTCN2020077952-appb-100010
    Figure PCTCN2020077952-appb-100011
    Figure PCTCN2020077952-appb-100011
    此外，还需要将t new乘一个系数，使得t new在x方向上的分量
    Figure PCTCN2020077952-appb-100012
    In addition, we need to multiply t new by a coefficient so that the component of t new in the x direction
    Figure PCTCN2020077952-appb-100012
PCT/CN2020/077952 2019-11-22 2020-03-05 Trajectory feature alignment-based multispectral stereo camera self-calibration algorithm WO2021098081A1 (en)

Priority Applications (1)

Application Number Priority Date Filing Date Title
US17/283,772 US11575873B2 (en) 2019-11-22 2020-03-05 Multispectral stereo camera self-calibration algorithm based on track feature registration

Applications Claiming Priority (2)

Application Number Priority Date Filing Date Title
CN201911153778.5A CN111080709B (en) 2019-11-22 2019-11-22 Multispectral stereo camera self-calibration algorithm based on track feature registration
CN201911153778.5 2019-11-22

Publications (1)

Publication Number Publication Date
WO2021098081A1 true WO2021098081A1 (en) 2021-05-27

Family

ID=70311826

Family Applications (1)

Application Number Title Priority Date Filing Date
PCT/CN2020/077952 WO2021098081A1 (en) 2019-11-22 2020-03-05 Trajectory feature alignment-based multispectral stereo camera self-calibration algorithm

Country Status (3)

Country Link
US (1) US11575873B2 (en)
CN (1) CN111080709B (en)
WO (1) WO2021098081A1 (en)

Cited By (3)

* Cited by examiner, † Cited by third party
Publication number Priority date Publication date Assignee Title
CN114677430A (en) * 2022-01-12 2022-06-28 深圳市路远智能装备有限公司 Combined multi-camera vision alignment method
CN114708331A (en) * 2022-02-24 2022-07-05 合肥的卢深视科技有限公司 Calibration method and device of depth camera, electronic equipment and storage medium
CN117994865A (en) * 2024-04-01 2024-05-07 杭州海康威视数字技术股份有限公司 Binocular face matching method and device, electronic equipment and storage medium

Families Citing this family (5)

* Cited by examiner, † Cited by third party
Publication number Priority date Publication date Assignee Title
CN111028281B (en) * 2019-10-22 2022-10-18 清华大学 Depth information calculation method and device based on light field binocular system
CN111932622B (en) * 2020-08-10 2022-06-28 浙江大学 Device, method and system for determining flight altitude of unmanned aerial vehicle
CN114858186B (en) * 2022-04-21 2023-03-31 武汉大学 On-satellite geometric calibration method for linear array camera under fixed star observation mode
CN115719320B (en) * 2023-01-09 2023-04-14 中国人民解放军63921部队 Tilt correction dense matching method based on remote sensing image
CN117934600B (en) * 2024-03-25 2024-06-14 南京信息工程大学 Method for quickly identifying remote markers and resolving three-dimensional positions based on unmanned aerial vehicle

Citations (6)

* Cited by examiner, † Cited by third party
Publication number Priority date Publication date Assignee Title
CN104359464A (en) * 2014-11-02 2015-02-18 天津理工大学 Mobile robot positioning method based on stereoscopic vision
US20150199814A1 (en) * 2011-05-16 2015-07-16 Sony Corporation Device for determining disappearing direction and method thereof, apparatus for video camera calibration and method thereof
CN105701827A (en) * 2016-01-15 2016-06-22 中林信达（北京）科技信息有限责任公司 Method and device for jointly calibrating parameters of visible light camera and infrared camera
CN106485735A (en) * 2015-09-01 2017-03-08 南京理工大学 Human body target recognition and tracking method based on stereovision technique
CN106981077A (en) * 2017-03-24 2017-07-25 中国人民解放军国防科学技术大学 Infrared image and visible light image registration method based on DCE and LSS
CN110472658A (en) * 2019-07-05 2019-11-19 哈尔滨工程大学 A kind of the level fusion and extracting method of the detection of moving-target multi-source

Family Cites Families (20)

* Cited by examiner, † Cited by third party
Publication number Priority date Publication date Assignee Title
DE4439742C1 (en) * 1994-10-28 1996-03-07 Daimler Benz Ag Automatic combinatorial optimisation of arrangements for multiple target tracking radar
US7317812B1 (en) * 2002-11-15 2008-01-08 Videomining Corporation Method and apparatus for robustly tracking objects
US20060125658A1 (en) * 2004-12-10 2006-06-15 Deutsches Zentrum For Luft-Und Raumfahrt E.V. System for assisting navigation operation of moving objects
CN101563709B (en) * 2006-12-18 2013-07-31 皇家飞利浦电子股份有限公司 Calibrating a camera system
CN102750697B (en) * 2012-06-08 2014-08-20 华为技术有限公司 Parameter calibration method and device
US9361520B2 (en) * 2014-04-10 2016-06-07 Disney Enterprises, Inc. Method and system for tracking objects
JP6027070B2 (en) * 2014-09-24 2016-11-16 富士フイルム株式会社 Area detection apparatus, area detection method, image processing apparatus, image processing method, program, and recording medium
US9792664B2 (en) * 2015-01-29 2017-10-17 Wipro Limited System and method for mapping object coordinates from a video to real world coordinates using perspective transformation
US9898677B1 (en) * 2015-10-13 2018-02-20 MotionDSP, Inc. Object-level grouping and identification for tracking objects in a video
CN106204574B (en) * 2016-07-07 2018-12-21 兰州理工大学 Camera pose self-calibrating method based on objective plane motion feature
WO2018049514A1 (en) * 2016-09-13 2018-03-22 Genetec Inc. Auto-calibration of tracking systems
US10198829B2 (en) * 2017-04-25 2019-02-05 Symbol Technologies, Llc Systems and methods for extrinsic calibration of a plurality of sensors
CN107358633A (en) * 2017-07-12 2017-11-17 北京轻威科技有限责任公司 Join scaling method inside and outside a kind of polyphaser based on 3 points of demarcation things
CN108010085B (en) * 2017-11-30 2019-12-31 西南科技大学 Target identification method based on binocular visible light camera and thermal infrared camera
CN108288294A (en) * 2018-01-17 2018-07-17 视缘(上海)智能科技有限公司 A kind of outer ginseng scaling method of a 3D phases group of planes
RU2696548C1 (en) * 2018-08-29 2019-08-02 Александр Владимирович Абрамов Method of constructing a video surveillance system for searching and tracking objects
EP3867795A1 (en) * 2018-10-16 2021-08-25 Advertima AG A calibration method for a recording device and a method for an automatic setup of a multi-camera system
US10861172B2 (en) * 2018-11-28 2020-12-08 Battelle Memorial Institute Sensors and methods for monitoring flying objects
US10957074B2 (en) * 2019-01-29 2021-03-23 Microsoft Technology Licensing, Llc Calibrating cameras using human skeleton
US20210033722A1 (en) * 2019-07-29 2021-02-04 Trackman A/S System and method for inter-sensor calibration

Patent Citations (6)

* Cited by examiner, † Cited by third party
Publication number Priority date Publication date Assignee Title
US20150199814A1 (en) * 2011-05-16 2015-07-16 Sony Corporation Device for determining disappearing direction and method thereof, apparatus for video camera calibration and method thereof
CN104359464A (en) * 2014-11-02 2015-02-18 天津理工大学 Mobile robot positioning method based on stereoscopic vision
CN106485735A (en) * 2015-09-01 2017-03-08 南京理工大学 Human body target recognition and tracking method based on stereovision technique
CN105701827A (en) * 2016-01-15 2016-06-22 中林信达（北京）科技信息有限责任公司 Method and device for jointly calibrating parameters of visible light camera and infrared camera
CN106981077A (en) * 2017-03-24 2017-07-25 中国人民解放军国防科学技术大学 Infrared image and visible light image registration method based on DCE and LSS
CN110472658A (en) * 2019-07-05 2019-11-19 哈尔滨工程大学 A kind of the level fusion and extracting method of the detection of moving-target multi-source

Cited By (4)

* Cited by examiner, † Cited by third party
Publication number Priority date Publication date Assignee Title
CN114677430A (en) * 2022-01-12 2022-06-28 深圳市路远智能装备有限公司 Combined multi-camera vision alignment method
CN114677430B (en) * 2022-01-12 2024-04-16 深圳市路远智能装备有限公司 Combined multi-camera visual alignment method
CN114708331A (en) * 2022-02-24 2022-07-05 合肥的卢深视科技有限公司 Calibration method and device of depth camera, electronic equipment and storage medium
CN117994865A (en) * 2024-04-01 2024-05-07 杭州海康威视数字技术股份有限公司 Binocular face matching method and device, electronic equipment and storage medium

Also Published As

Publication number Publication date
US11575873B2 (en) 2023-02-07
CN111080709A (en) 2020-04-28
CN111080709B (en) 2023-05-05
US20220046220A1 (en) 2022-02-10

Similar Documents

Publication Publication Date Title
WO2021098081A1 (en) Trajectory feature alignment-based multispectral stereo camera self-calibration algorithm
WO2021098080A1 (en) Multi-spectral camera extrinsic parameter self-calibration algorithm based on edge features
WO2021098083A1 (en) Multispectral camera dynamic stereo calibration algorithm based on salient feature
US11830222B2 (en) Bi-level optimization-based infrared and visible light fusion method
Wang et al. 360sd-net: 360 stereo depth estimation with learnable cost volume
CN111243033B (en) Method for optimizing external parameters of binocular camera
CN110956661B (en) Method for calculating dynamic pose of visible light and infrared camera based on bidirectional homography matrix
WO2021139176A1 (en) Pedestrian trajectory tracking method and apparatus based on binocular camera calibration, computer device, and storage medium
CN110969669B (en) Visible light and infrared camera combined calibration method based on mutual information registration
CN110910456B (en) Three-dimensional camera dynamic calibration method based on Harris angular point mutual information matching
CN107038714B (en) Multi-type visual sensing cooperative target tracking method
CN111461963B (en) Fisheye image stitching method and device
CN110992409B (en) Multispectral stereo camera dynamic registration method based on Fourier transform registration
CN110874854B (en) Camera binocular photogrammetry method based on small baseline condition
WO2024045632A1 (en) Binocular vision and imu-based underwater scene three-dimensional reconstruction method, and device
CN110880191B (en) Infrared stereo camera dynamic external parameter calculation method based on histogram equalization
CN110807815B (en) Quick underwater calibration method based on corresponding vanishing points of two groups of mutually orthogonal parallel lines
CN112470192A (en) Dual-camera calibration method, electronic device and computer-readable storage medium
CN110910457B (en) Multispectral three-dimensional camera external parameter calculation method based on angular point characteristics
CN108805937B (en) Single-camera polarization information prediction method
CN112396687B (en) Binocular stereoscopic vision three-dimensional reconstruction system and method based on infrared micro-polarizer array
CN114972536B (en) Positioning and calibrating method for aviation area array swing scanning type camera
Ma et al. Image Dehazing With Polarization Boundary Constraints of Transmission
Xu et al. Automatic Point Cloud Colorization of Ground-based LiDAR Data Using Video Imagery Without POS
Mo et al. A Robust Infrared and Visible Image Registration Method for Dual Sensor UAV System

Legal Events

Date Code Title Description
121 Ep: the epo has been informed by wipo that ep was designated in this application

Ref document number: 20890525

Country of ref document: EP

Kind code of ref document: A1

NENP Non-entry into the national phase

Ref country code: DE

122 Ep: pct application non-entry in european phase

Ref document number: 20890525

Country of ref document: EP

Kind code of ref document: A1