CN111784586B - Panoramic image level correction method and system based on self-supervised learning - Google Patents
Panoramic image level correction method and system based on self-supervised learning Download PDFInfo
- Publication number
- CN111784586B CN111784586B CN202010446601.0A CN202010446601A CN111784586B CN 111784586 B CN111784586 B CN 111784586B CN 202010446601 A CN202010446601 A CN 202010446601A CN 111784586 B CN111784586 B CN 111784586B
- Authority
- CN
- China
- Prior art keywords
- training
- image
- angle
- panoramic image
- horizontal
- Prior art date
- Legal status (The legal status is an assumption and is not a legal conclusion. Google has not performed a legal analysis and makes no representation as to the accuracy of the status listed.)
- Active
Links
- 238000012937 correction Methods 0.000 title claims abstract description 34
- 238000000034 method Methods 0.000 title claims abstract description 24
- 238000012549 training Methods 0.000 claims abstract description 85
- 238000013527 convolutional neural network Methods 0.000 claims abstract description 28
- 239000011159 matrix material Substances 0.000 claims abstract description 25
- 230000006870 function Effects 0.000 claims abstract description 9
- 238000009795 derivation Methods 0.000 claims description 4
- 238000004364 calculation method Methods 0.000 claims description 3
- 230000002194 synthesizing effect Effects 0.000 claims 1
- 238000010586 diagram Methods 0.000 description 5
- 238000013528 artificial neural network Methods 0.000 description 4
- 238000000605 extraction Methods 0.000 description 2
- 238000013507 mapping Methods 0.000 description 2
- 239000000463 material Substances 0.000 description 2
- 238000007781 pre-processing Methods 0.000 description 2
- 238000011160 research Methods 0.000 description 2
- 238000003702 image correction Methods 0.000 description 1
- 238000012986 modification Methods 0.000 description 1
- 230000004048 modification Effects 0.000 description 1
- 230000001537 neural effect Effects 0.000 description 1
- 238000006467 substitution reaction Methods 0.000 description 1
Images
Classifications
-
- G—PHYSICS
- G06—COMPUTING; CALCULATING OR COUNTING
- G06T—IMAGE DATA PROCESSING OR GENERATION, IN GENERAL
- G06T3/00—Geometric image transformations in the plane of the image
- G06T3/60—Rotation of whole images or parts thereof
- G06T3/608—Rotation of whole images or parts thereof by skew deformation, e.g. two-pass or three-pass rotation
-
- G—PHYSICS
- G06—COMPUTING; CALCULATING OR COUNTING
- G06N—COMPUTING ARRANGEMENTS BASED ON SPECIFIC COMPUTATIONAL MODELS
- G06N3/00—Computing arrangements based on biological models
- G06N3/02—Neural networks
- G06N3/04—Architecture, e.g. interconnection topology
- G06N3/045—Combinations of networks
-
- G—PHYSICS
- G06—COMPUTING; CALCULATING OR COUNTING
- G06N—COMPUTING ARRANGEMENTS BASED ON SPECIFIC COMPUTATIONAL MODELS
- G06N3/00—Computing arrangements based on biological models
- G06N3/02—Neural networks
- G06N3/08—Learning methods
-
- G—PHYSICS
- G06—COMPUTING; CALCULATING OR COUNTING
- G06T—IMAGE DATA PROCESSING OR GENERATION, IN GENERAL
- G06T7/00—Image analysis
- G06T7/70—Determining position or orientation of objects or cameras
- G06T7/73—Determining position or orientation of objects or cameras using feature-based methods
Landscapes
- Engineering & Computer Science (AREA)
- Physics & Mathematics (AREA)
- Theoretical Computer Science (AREA)
- General Physics & Mathematics (AREA)
- General Health & Medical Sciences (AREA)
- Molecular Biology (AREA)
- Biophysics (AREA)
- Computational Linguistics (AREA)
- Data Mining & Analysis (AREA)
- Evolutionary Computation (AREA)
- Artificial Intelligence (AREA)
- Biomedical Technology (AREA)
- Computing Systems (AREA)
- General Engineering & Computer Science (AREA)
- Life Sciences & Earth Sciences (AREA)
- Mathematical Physics (AREA)
- Software Systems (AREA)
- Health & Medical Sciences (AREA)
- Computer Vision & Pattern Recognition (AREA)
- Image Analysis (AREA)
Abstract
本发明公开了一种自监督学习的全景图像水平矫正方法及系统,其中,该方法包括:构建训练图像库;利用训练图像库中的训练图像对卷积神经网络进行自监督式训练,直至卷积神经网络的误差函数收敛;将训练图像库中的待矫正图像输入至训练完成的卷积神经网络中,推导出拍摄当前全景图像时相机的俯仰角和滚筒角;对俯仰角和滚筒角进行欧拉角推算旋转矩阵计算,得到相机位姿的旋转矩阵,根据相机位姿的旋转矩阵矫正待矫正图像,合成水平图像。该方法有效解决相机姿态不竖直而导致的全景图像扭曲畸变问题。
The invention discloses a self-supervised learning panorama image level correction method and system, wherein the method includes: constructing a training image library; using the training images in the training image library to perform self-supervised training on a convolutional neural network until the volume The error function of the convolutional neural network converges; the to-be-corrected image in the training image library is input into the trained convolutional neural network, and the pitch angle and roll angle of the camera when the current panoramic image is captured are derived; The Euler angle is used to calculate the rotation matrix, and the rotation matrix of the camera pose is obtained, and the image to be corrected is corrected according to the rotation matrix of the camera pose, and the horizontal image is synthesized. This method effectively solves the problem of distortion and distortion of panoramic images caused by the non-vertical camera posture.
Description
技术领域technical field
本发明涉及图像矫正技术领域,特别涉及一种自监督学习的全景图像水平矫正方法及系统。The invention relates to the technical field of image correction, in particular to a self-supervised learning method and system for horizontal correction of panoramic images.
背景技术Background technique
现有全景图像可以应用在虚拟现实、三维重构等场景中。拍摄全景图像时,全景相机的姿态,会对采集到的图像造成较大的影响。在理想情况下,全景相机的姿态应保持和地面垂直,此时采集到的全景图像是水平的。可以直观理解为,图像中的地平线近似成一条水平直线。然而,在实际情况中,全景相机摆放不一定是竖直的,亦或是存在相机抖动等干扰情况,导致拍摄到的全景图像不是水平的。如图1所示,图像中的地平线近似成一条正弦曲线,并且会伴随较大的扭曲和畸变,这会对全景图像后续的使用带来较大的麻烦。Existing panoramic images can be applied in scenarios such as virtual reality and 3D reconstruction. When shooting a panoramic image, the posture of the panoramic camera will have a great influence on the collected image. Ideally, the posture of the panoramic camera should be kept perpendicular to the ground, and the panoramic image collected at this time is horizontal. It can be intuitively understood that the horizon in the image is approximately a horizontal line. However, in an actual situation, the panoramic camera placement is not necessarily vertical, or there is interference such as camera shake, so that the captured panoramic image is not horizontal. As shown in Figure 1, the horizon in the image is approximately a sine curve, and will be accompanied by large distortion and distortion, which will bring great trouble to the subsequent use of the panoramic image.
相关技术中,使用额外的陀螺仪来获取相机的三轴位姿,并对采集到图像进行旋转校正,生成一张水平图像。但这种方式会使系统的硬件成本和复杂度提高,同时,有些全景图采集时并没有与之匹配的陀螺仪数据。In the related art, an additional gyroscope is used to obtain the three-axis pose of the camera, and rotation correction is performed on the collected image to generate a horizontal image. However, this method will increase the hardware cost and complexity of the system, and at the same time, some panorama images do not have matching gyroscope data when they are collected.
因此,亟待一种可以解决相机姿态不竖直而导致的全景图像扭曲畸变的问题。Therefore, there is an urgent need for a solution that can solve the problem of distortion and distortion of panoramic images caused by the non-vertical camera posture.
发明内容SUMMARY OF THE INVENTION
本发明旨在至少在一定程度上解决相关技术中的技术问题之一。The present invention aims to solve one of the technical problems in the related art at least to a certain extent.
为此,本发明的一个目的在于提出一种自监督学习的全景图像水平矫正方法,该方法无需额外辅助设备,就可解决相机竖直导致全景图像扭曲畸变的问题,且硬件成本低,复杂度低。Therefore, an object of the present invention is to propose a self-supervised learning method for horizontal correction of panoramic images, which can solve the problem of distortion of panoramic images caused by vertical cameras without additional auxiliary equipment, and has low hardware cost and complexity. Low.
本发明的另一个目的在于提出一种自监督学习的全景图像水平矫正系统。Another object of the present invention is to propose a self-supervised learning panorama image level correction system.
为达到上述目的,本发明一方面实施例提出了自监督学习的全景图像水平矫正方法,包括以下步骤:步骤S1,构建训练图像库;步骤S2,利用所述训练图像库的训练图像对卷积神经网络进行自监督式训练,直至所述卷积神经网络的误差函数收敛;步骤S3,将所述训练图像库中的待矫正图像输入至训练完成的卷积神经网络中,推导出拍摄所述待矫正图像时相机的俯仰角和滚筒角;步骤S4,对所述俯仰角和所述滚筒角进行欧拉角推算旋转矩阵计算,得到相机位姿的旋转矩阵,根据所述相机位姿的旋转矩阵矫正所述待矫正图像,合成水平图像。In order to achieve the above purpose, an embodiment of the present invention proposes a self-supervised learning panorama image level correction method, including the following steps: step S1, constructing a training image database; step S2, using the training images of the training image database to convolve The neural network performs self-supervised training until the error function of the convolutional neural network converges; step S3, input the to-be-corrected image in the training image library into the trained convolutional neural network, and derive the The pitch angle and roll angle of the camera when the image is to be corrected; step S4, Euler angle reckoning rotation matrix calculation is performed on the pitch angle and the roll angle to obtain the rotation matrix of the camera pose, according to the rotation of the camera pose Matrix corrects the to-be-corrected image to synthesize a horizontal image.
本发明实施例的自监督学习的全景图像水平矫正方法,获取足够数量的水平全景图像,使用大量的水平全景图像对神经网络进行自监督式的训练,使得网络具备推理相机姿态角度的能力,估计全景图像拍摄时相机的旋转角度,根据旋转角度对相机位姿进行补偿,得到水平的全景图像,有效解决了相机姿态不竖直而导致全景图扭曲畸变的问题。The panoramic image level correction method for self-supervised learning according to the embodiment of the present invention acquires a sufficient number of horizontal panoramic images, and uses a large number of horizontal panoramic images to perform self-supervised training on the neural network, so that the network has the ability to infer the camera pose angle, and estimates The rotation angle of the camera when the panoramic image is captured, the camera pose is compensated according to the rotation angle, and a horizontal panoramic image is obtained, which effectively solves the problem of distortion and distortion of the panoramic image caused by the non-vertical camera attitude.
另外,根据本发明上述实施例的自监督学习的全景图像水平矫正方法还可以具有以下附加的技术特征:In addition, the panoramic image level correction method of self-supervised learning according to the above-mentioned embodiment of the present invention may also have the following additional technical features:
进一步地,在本发明的一个实施例中,所述训练图像库中的每幅全景图像均为水平角度。Further, in an embodiment of the present invention, each panoramic image in the training image library is a horizontal angle.
进一步地,在本发明的一个实施例中,所述步骤S2包括:利用欧拉角中的俯仰角和滚筒角作为监督信息;通过所述监督信息对所述训练图像库中的每张全景图像进行迭代训练,生成若干张旋转后的非水平图像;当所述非水平图像的旋转角度与所述监督信息中的旋转角度的误差最小,完成所述卷积神经网络的训练。Further, in an embodiment of the present invention, the step S2 includes: using the pitch angle and the roll angle in the Euler angles as supervision information; Perform iterative training to generate several rotated non-horizontal images; when the error between the rotation angle of the non-horizontal image and the rotation angle in the supervision information is the smallest, the training of the convolutional neural network is completed.
进一步地,在本发明的一个实施例中,在对所述全景图像进行迭代训练前,将所述全景图像映射到球面上。Further, in an embodiment of the present invention, before performing iterative training on the panoramic image, the panoramic image is mapped onto a spherical surface.
为达到上述目的,本发明另一方面实施例提出了自监督学习的全景图像水平矫正系统,包括:构建模块,用于构建训练图像库;训练模块,用于利用所述训练图像库的训练图像对卷积神经网络进行自监督式训练,直至所述卷积神经网络的误差函数收敛;推导模块,用于将所述训练图像库中的待矫正图像输入至训练完成的卷积神经网络中,推导出拍摄所述待矫正图像时相机的俯仰角和滚筒角;矫正模块,用于对所述俯仰角和所述滚筒角进行欧拉角推算旋转矩阵计算,得到相机位姿的旋转矩阵,根据所述相机位姿的旋转矩阵矫正所述待矫正图像,合成水平图像。In order to achieve the above object, another embodiment of the present invention proposes a panoramic image level correction system for self-supervised learning, including: a building module for building a training image library; a training module for using the training images in the training image library Carry out self-supervised training on the convolutional neural network until the error function of the convolutional neural network converges; the derivation module is used to input the to-be-corrected image in the training image library into the trained convolutional neural network, Deriving the pitch angle and roll angle of the camera when shooting the to-be-corrected image; the correction module is used to calculate the Euler angle reckoning rotation matrix for the pitch angle and the roll angle, and obtain the rotation matrix of the camera pose, according to The rotation matrix of the camera pose corrects the to-be-corrected image to synthesize a horizontal image.
本发明实施例的自监督学习的全景图像水平矫正系统,获取足够数量的水平全景图像,使用大量的水平全景图像对神经网络进行自监督式的训练,使得网络具备推理相机姿态角度的能力,估计全景图像拍摄时相机的旋转角度,根据旋转角度对相机位姿进行补偿,得到水平的全景图像,有效解决了相机姿态不竖直而导致全景图扭曲畸变的问题。The panoramic image level correction system for self-supervised learning according to the embodiment of the present invention acquires a sufficient number of horizontal panoramic images, and uses a large number of horizontal panoramic images to perform self-supervised training on the neural network, so that the network has the ability to infer the camera pose angle, and estimates The rotation angle of the camera when the panoramic image is captured, the camera pose is compensated according to the rotation angle, and a horizontal panoramic image is obtained, which effectively solves the problem of distortion and distortion of the panoramic image caused by the non-vertical camera attitude.
另外,根据本发明上述实施例的自监督学习的全景图像水平矫正系统还可以具有以下附加的技术特征:In addition, the panoramic image level correction system for self-supervised learning according to the above embodiments of the present invention may also have the following additional technical features:
进一步地,在本发明的一个实施例中,所述训练图像库中的每幅全景图像均为水平角度。Further, in an embodiment of the present invention, each panoramic image in the training image library is a horizontal angle.
进一步地,在本发明的一个实施例中,所述训练模块进一步用于:确定单元,用于利用欧拉角中的俯仰角和滚筒角作为监督信息;训练单元,用于通过所述监督信息对所述训练图像库中的每幅全景图像进行迭代训练,生成若干张旋转后的非水平图像;判断单元,用于当所述非水平图像的旋转角度与所述监督信息中的旋转角度的误差最小,完成所述卷积神经网络的训练。Further, in an embodiment of the present invention, the training module is further used for: a determination unit for using the pitch angle and the roll angle in Euler angles as supervision information; a training unit for passing the supervision information Perform iterative training on each panorama image in the training image library to generate several rotated non-horizontal images; the judgment unit is used for when the rotation angle of the non-horizontal image and the rotation angle in the supervision information are different. With the smallest error, the training of the convolutional neural network is completed.
进一步地,在本发明的一个实施例中,在对所述全景图像进行迭代训练前,将所述全景图像映射到球面上。Further, in an embodiment of the present invention, before performing iterative training on the panoramic image, the panoramic image is mapped onto a spherical surface.
本发明附加的方面和优点将在下面的描述中部分给出,部分将从下面的描述中变得明显,或通过本发明的实践了解到。Additional aspects and advantages of the present invention will be set forth, in part, from the following description, and in part will be apparent from the following description, or may be learned by practice of the invention.
附图说明Description of drawings
本发明上述的和/或附加的方面和优点从下面结合附图对实施例的描述中将变得明显和容易理解,其中:The above and/or additional aspects and advantages of the present invention will become apparent and readily understood from the following description of embodiments taken in conjunction with the accompanying drawings, wherein:
图1为非水平的全景图像中存在大量扭曲和畸形的示意图;Figure 1 is a schematic diagram of a large number of distortions and deformities in a non-horizontal panoramic image;
图2为根据本发明一个实施例的自监督学习的全景图像水平矫正方法的流程图;FIG. 2 is a flow chart of a panoramic image level correction method for self-supervised learning according to an embodiment of the present invention;
图3为根据本发明一个实施例的全景图像平面到球体的对应关系的示意图;3 is a schematic diagram of the correspondence between a panoramic image plane and a sphere according to an embodiment of the present invention;
图4为根据本发明一个实施例的自监督学习的全景图像水平矫正方法简化流程图;FIG. 4 is a simplified flowchart of a panoramic image level correction method for self-supervised learning according to an embodiment of the present invention;
图5为根据本发明一个实施例的矫正前后图像对比示意图;5 is a schematic diagram of image comparison before and after correction according to an embodiment of the present invention;
图6为根据本发明一个实施例的自监督学习的全景图像水平矫正系统的结构示意图。FIG. 6 is a schematic structural diagram of a panoramic image level correction system for self-supervised learning according to an embodiment of the present invention.
具体实施方式Detailed ways
下面详细描述本发明的实施例,所述实施例的示例在附图中示出,其中自始至终相同或类似的标号表示相同或类似的元件或具有相同或类似功能的元件。下面通过参考附图描述的实施例是示例性的,旨在用于解释本发明,而不能理解为对本发明的限制。The following describes in detail the embodiments of the present invention, examples of which are illustrated in the accompanying drawings, wherein the same or similar reference numerals refer to the same or similar elements or elements having the same or similar functions throughout. The embodiments described below with reference to the accompanying drawings are exemplary, and are intended to explain the present invention and should not be construed as limiting the present invention.
下面参照附图描述根据本发明实施例提出的自监督学习的全景图像水平矫正方法及系统,首先将参照附图描述根据本发明实施例提出的自监督学习的全景图像水平矫正方法。The method and system for level correction of panoramic images according to self-supervised learning according to the embodiments of the present invention will be described below with reference to the accompanying drawings.
图1是本发明一个实施例的自监督学习的全景图像水平矫正方法的流程图。FIG. 1 is a flow chart of a panoramic image level correction method for self-supervised learning according to an embodiment of the present invention.
如图1所示,该自监督学习的全景图像水平矫正方法包括以下步骤:As shown in Figure 1, the self-supervised learning panorama image level correction method includes the following steps:
在步骤S1中,构建训练图像库,其中,训练图像库中的每幅全景图像均为水平角度。In step S1, a training image library is constructed, wherein each panoramic image in the training image library is a horizontal angle.
具体地,收集大量水平拍摄的全景图像,可以将全景支在三脚架或者小车上进行采集,也可以使用一些公开的全景图像数据库,或者从街景地图网站上爬取。无论图像来源如何,必须要保证图像库中每一幅全景图像都是水平的。可适当的对图像进行一定预处理,例如裁剪拍摄设备或支撑设备在图像中出现的区域等。步骤S1通过获取足够数量的水平全景图像,以用于后续的模型训练。Specifically, to collect a large number of panoramic images shot horizontally, the panoramic images can be collected on a tripod or a trolley, or some public panoramic image databases can be used, or crawled from a street view map website. Regardless of the image source, it must be ensured that each panorama image in the image library is horizontal. The image can be properly pre-processed, such as cropping the area where the shooting device or the support device appears in the image. Step S1 obtains a sufficient number of horizontal panoramic images for subsequent model training.
在步骤S2中,利用训练图像库中的训练图像对卷积神经网络进行自监督式训练,直至卷积神经网络的误差函数收敛。In step S2, self-supervised training is performed on the convolutional neural network using the training images in the training image library until the error function of the convolutional neural network converges.
进一步地,在本发明的一个实施例中,步骤S2包括:利用欧拉角中的俯仰角和滚筒角作为监督信息;通过监督信息对训练图像库中的每张全景图像进行迭代训练,生成若干张旋转后的非水平图像;当非水平图像的旋转角度与监督信息中的旋转角度的误差最小,完成卷积神经网络的训练。Further, in an embodiment of the present invention, step S2 includes: using the pitch angle and the roll angle in the Euler angles as supervision information; performing iterative training on each panoramic image in the training image library through the supervision information, and generating several A rotated non-horizontal image; when the error between the rotation angle of the non-horizontal image and the rotation angle in the supervision information is the smallest, the training of the convolutional neural network is completed.
也就是说,步骤S2是对每张训练图像,施加随机的三维旋转,并记录旋转的角度。其中,该三维旋转并非指简单的平面旋转,而是对图像拍摄时相机的姿态进行三维空间的旋转。该旋转需要将全景图像映射到设备的球面上,再用旋转对应的矩阵和坐标相乘,再映射回平面,具体步骤下述进行描述。That is to say, step S2 is to apply random three-dimensional rotation to each training image, and record the rotation angle. Wherein, the three-dimensional rotation does not refer to a simple plane rotation, but a three-dimensional space rotation of the posture of the camera when the image is captured. The rotation needs to map the panoramic image to the spherical surface of the device, multiply the matrix and coordinates corresponding to the rotation, and then map it back to the plane. The specific steps are described below.
其中,三维旋转可以用欧拉角——即三个角度来表示,而其中一个角度是水平方向的旋转,对于水平矫正来说没有意义,因此只选用其中两个角度:俯仰角pitch和滚筒角roll。至此,每张水平图像可以生成若干张旋转后的非水平图像,并且其经历的两个旋转角pitch和roll是已知的。使用已知的两个旋转角度作为训练的监督信息,尽可能地使输入对应非水平图像时网络输出的旋转角度和真实旋转角度的误差最小。经过反复迭代训练,直至卷积神经网络的误差函数收敛,保存此时的模型参数。Among them, the three-dimensional rotation can be represented by Euler angles—that is, three angles, and one of them is the rotation in the horizontal direction, which is meaningless for horizontal correction, so only two of them are selected: pitch angle and drum angle. roll. So far, each horizontal image can generate several rotated non-horizontal images, and the two rotation angles pitch and roll experienced by it are known. Using the two known rotation angles as training supervision information, try to minimize the error between the rotation angle output by the network and the real rotation angle when the input corresponds to a non-horizontal image. After repeated iterative training, until the error function of the convolutional neural network converges, the model parameters at this time are saved.
在步骤S3中,将训练图像库中的待矫正图像输入至训练完成的卷积神经网络中,推导出拍摄待矫正图像时相机的俯仰角和滚筒角。In step S3, the images to be corrected in the training image library are input into the trained convolutional neural network, and the pitch angle and the roll angle of the camera when shooting the to-be-corrected image are derived.
也就是说,为网络模型加载上一步中训练好的模型参数。将全景图像进行缩放、裁剪等预处理后,送入卷积神经网络中。神经网络推理得到全景图像拍摄时相机的俯仰角pitch和滚筒角roll。That is, load the model parameters trained in the previous step for the network model. After preprocessing, such as scaling and cropping, the panoramic image is sent to the convolutional neural network. The neural network inference obtains the pitch angle of the camera and the roll angle of the camera when the panoramic image is captured.
在步骤S4中,对俯仰角和滚筒角进行欧拉角推算旋转矩阵计算,得到相机位姿的旋转矩阵,根据相机位姿的旋转矩阵矫正待矫正图像,合成水平图像。In step S4, the Euler angle reckoning rotation matrix is calculated for the pitch angle and the drum angle to obtain the rotation matrix of the camera pose, and the image to be corrected is corrected according to the rotation matrix of the camera pose to synthesize a horizontal image.
也就是话,在估计出输入全景图像拍摄时相机的俯仰角pitch和滚筒角roll后,参考由欧拉角推算的相关资料推算出代表相机位姿的旋转矩阵R。如图3所示,一幅全景图像可以映射到一个全景球面上,水平方向可以看作经度方向,垂直方向可以看作纬度方向。一般地,全景图像上某一像素(u,v)和球面上坐标(x,y,z)映射关系如下,其中Uc,Vc,f分别为全景图像的中心点横坐标、纵坐标和像素。That is to say, after estimating the pitch angle pitch and roll angle of the camera when the input panoramic image is taken, the rotation matrix R representing the camera pose is calculated with reference to the relevant data calculated from the Euler angle. As shown in Figure 3, a panoramic image can be mapped onto a panoramic spherical surface, the horizontal direction can be regarded as the longitude direction, and the vertical direction can be regarded as the latitude direction. Generally, the mapping relationship between a certain pixel (u, v) on the panoramic image and the coordinates (x, y, z) on the spherical surface is as follows, where U c , V c , f are the abscissa, ordinate and y of the center point of the panoramic image, respectively. pixel.
在得到全景图像上每一像素对应的球面上点坐标之后,使用相机位姿旋转矩阵R的逆矩阵R-1对球面坐标进行补偿,得到新的球面坐(x-,y_,z_)=R-1(x,y,z)T。进而利用上述映射关系,可以反算出矫正后的全景图像中对应像素坐标(u-,v_)。根据矫正前后全景图像中的像素对应关系(u,v)(u-,v_),可以合成出一幅新的矫正后的水平全景图像。After obtaining the point coordinates on the spherical surface corresponding to each pixel on the panoramic image, use the inverse matrix R -1 of the camera pose rotation matrix R to compensate the spherical coordinates to obtain the new spherical coordinates (x - , y _ , z _ ) =R -1 (x,y,z) T . Furthermore, by using the above mapping relationship, the corresponding pixel coordinates (u - , v _ ) in the corrected panoramic image can be inversely calculated. According to the pixel correspondence (u, v) (u - , v _ ) in the panoramic images before and after correction, a new corrected horizontal panoramic image can be synthesized.
简单来讲,如图4所示,发明实施例的自监督学习的全景图像水平矫正方法的工作原理为:先构建训练图像库,利用训练图像库中的图像进行自监督式训练,获得训练好的神经模型,向训练好的神经模型中输入待矫正图像,推算出相机的旋转角度,根据该角度对待矫正图像进行矫正,合成一幅新的水平全景图像。To put it simply, as shown in FIG. 4 , the working principle of the panoramic image level correction method for self-supervised learning according to the embodiment of the invention is as follows: first, a training image library is constructed, and images in the training image library are used to perform self-supervised training, and the trained images are obtained. Input the image to be corrected into the trained neural model, calculate the rotation angle of the camera, correct the image to be corrected according to the angle, and synthesize a new horizontal panoramic image.
根据本发明实施例提出的自监督学习的全景图像水平矫正方法,通过构建大规模训练图像数据库,使用自监督学习的方法,训练得到一个能够对全景图像拍摄时相机三轴姿态进行预测的网络模型,使用该模型可以估计出输入全景图像拍摄时相机的俯仰角pitch和滚筒角roll,进而对全景图像进行三维旋转,生成水平全景图。如图5所示,水平的全景图和未处理过的全景图相比,扭曲和畸变程度较小,使得后续的检索、去重、特征提取、特征匹配等操作效果更好,可作为一种全景图像强有力的预处理手段,对于本领域技术研究具有较大的理论和实际价值。According to the self-supervised learning panorama image level correction method proposed in the embodiment of the present invention, by constructing a large-scale training image database, and using the self-supervised learning method, a network model capable of predicting the camera's three-axis posture during panoramic image shooting is obtained by training. , using this model, the pitch angle pitch and roll angle of the camera can be estimated when the input panoramic image is taken, and then the panoramic image is rotated three-dimensionally to generate a horizontal panorama. As shown in Figure 5, compared with the unprocessed panorama, the horizontal panorama has a smaller degree of distortion and distortion, which makes the subsequent retrieval, deduplication, feature extraction, feature matching and other operations more effective, which can be used as a kind of A powerful preprocessing method for panoramic images has great theoretical and practical value for technical research in this field.
其次参照附图描述根据本发明实施例提出的自监督学习的全景图像水平矫正系统。Next, the panoramic image level correction system for self-supervised learning proposed according to the embodiment of the present invention will be described with reference to the accompanying drawings.
图6是本发明一个实施例的自监督学习的全景图像水平矫正系统的结构示意图。FIG. 6 is a schematic structural diagram of a panoramic image level correction system for self-supervised learning according to an embodiment of the present invention.
如图6所示,该系统10包括:构建模块100、训练模块200、推导模块300和矫正模块400。As shown in FIG. 6 , the
其中,构建模块100用于构建训练图像库,训练图像库中的每幅全景图像均为水平角度。训练模块200用于利用训练图像库中的训练图像对卷积神经网络进行自监督式训练,直至卷积神经网络的误差函数收敛。推导模块300用于将训练图像库中的待矫正图像输入至训练完成的卷积神经网络中,推导出拍摄待矫正图像时相机的俯仰角和滚筒角。矫正模块400用于对俯仰角和滚筒角进行欧拉角推算旋转矩阵计算,得到相机位姿的旋转矩阵,根据相机位姿的旋转矩阵矫正待矫正图像,合成水平图像。The
进一步地,在本发明的一个实施例中,训练模块200进一步用于:Further, in an embodiment of the present invention, the
确定单元,用于利用欧拉角中的俯仰角和滚筒角作为监督信息;Determining unit for using pitch and roll angles in Euler angles as supervision information;
训练单元,用于通过监督信息对训练图像库中的每幅全景图像进行迭代训练,生成若干张旋转后的非水平图像;The training unit is used to iteratively train each panoramic image in the training image library through the supervision information, and generate several rotated non-horizontal images;
判断单元,用于当非水平图像的旋转角度与监督信息中的旋转角度的误差最小,完成卷积神经网络的训练。The judgment unit is used to complete the training of the convolutional neural network when the error between the rotation angle of the non-horizontal image and the rotation angle in the supervision information is the smallest.
进一步地,在本发明的一个实施例中,在对全景图像进行迭代训练前,将全景图像映射到球面上。Further, in an embodiment of the present invention, before performing iterative training on the panoramic image, the panoramic image is mapped onto a spherical surface.
根据本发明实施例提出的自监督学习的全景图像水平矫正系统,通过构建大规模训练图像数据库,使用自监督学习的方法,训练得到一个能够对全景图像拍摄时相机三轴姿态进行预测的网络模型,使用该模型可以估计出输入全景图像拍摄时相机的俯仰角pitch和滚筒角roll,进而对全景图像进行三维旋转,生成水平全景图。水平的全景图和未处理过的全景图相比,扭曲和畸变程度较小,使得后续的检索、去重、特征提取、特征匹配等操作效果更好,可作为一种全景图像强有力的预处理手段,对于本领域技术研究具有较大的理论和实际价值。According to the self-supervised learning panorama image level correction system proposed in the embodiment of the present invention, by constructing a large-scale training image database and using the self-supervised learning method, a network model capable of predicting the three-axis pose of the camera when the panoramic image is captured is obtained by training. , using this model, the pitch angle pitch and roll angle of the camera can be estimated when the input panoramic image is taken, and then the panoramic image is rotated three-dimensionally to generate a horizontal panorama. Compared with the unprocessed panorama, the horizontal panorama has a smaller degree of distortion and distortion, which makes the subsequent retrieval, deduplication, feature extraction, feature matching and other operations more effective, and can be used as a powerful preview of the panorama image. The treatment method has great theoretical and practical value for technical research in this field.
此外,术语“第一”、“第二”仅用于描述目的,而不能理解为指示或暗示相对重要性或者隐含指明所指示的技术特征的数量。由此,限定有“第一”、“第二”的特征可以明示或者隐含地包括至少一个该特征。在本发明的描述中,“多个”的含义是至少两个,例如两个,三个等,除非另有明确具体的限定。In addition, the terms "first" and "second" are only used for descriptive purposes, and should not be construed as indicating or implying relative importance or implying the number of indicated technical features. Thus, a feature delimited with "first", "second" may expressly or implicitly include at least one of that feature. In the description of the present invention, "plurality" means at least two, such as two, three, etc., unless otherwise expressly and specifically defined.
在本说明书的描述中,参考术语“一个实施例”、“一些实施例”、“示例”、“具体示例”、或“一些示例”等的描述意指结合该实施例或示例描述的具体特征、结构、材料或者特点包含于本发明的至少一个实施例或示例中。在本说明书中,对上述术语的示意性表述不必须针对的是相同的实施例或示例。而且,描述的具体特征、结构、材料或者特点可以在任一个或多个实施例或示例中以合适的方式结合。此外,在不相互矛盾的情况下,本领域的技术人员可以将本说明书中描述的不同实施例或示例以及不同实施例或示例的特征进行结合和组合。In the description of this specification, description with reference to the terms "one embodiment," "some embodiments," "example," "specific example," or "some examples", etc., mean specific features described in connection with the embodiment or example , structure, material or feature is included in at least one embodiment or example of the present invention. In this specification, schematic representations of the above terms are not necessarily directed to the same embodiment or example. Furthermore, the particular features, structures, materials or characteristics described may be combined in any suitable manner in any one or more embodiments or examples. Furthermore, those skilled in the art may combine and combine the different embodiments or examples described in this specification, as well as the features of the different embodiments or examples, without conflicting each other.
尽管上面已经示出和描述了本发明的实施例,可以理解的是,上述实施例是示例性的,不能理解为对本发明的限制,本领域的普通技术人员在本发明的范围内可以对上述实施例进行变化、修改、替换和变型。Although the embodiments of the present invention have been shown and described above, it should be understood that the above embodiments are exemplary and should not be construed as limiting the present invention. Embodiments are subject to variations, modifications, substitutions and variations.
Claims (4)
Priority Applications (1)
Application Number | Priority Date | Filing Date | Title |
---|---|---|---|
CN202010446601.0A CN111784586B (en) | 2020-05-25 | 2020-05-25 | Panoramic image level correction method and system based on self-supervised learning |
Applications Claiming Priority (1)
Application Number | Priority Date | Filing Date | Title |
---|---|---|---|
CN202010446601.0A CN111784586B (en) | 2020-05-25 | 2020-05-25 | Panoramic image level correction method and system based on self-supervised learning |
Publications (2)
Publication Number | Publication Date |
---|---|
CN111784586A CN111784586A (en) | 2020-10-16 |
CN111784586B true CN111784586B (en) | 2022-10-18 |
Family
ID=72753202
Family Applications (1)
Application Number | Title | Priority Date | Filing Date |
---|---|---|---|
CN202010446601.0A Active CN111784586B (en) | 2020-05-25 | 2020-05-25 | Panoramic image level correction method and system based on self-supervised learning |
Country Status (1)
Country | Link |
---|---|
CN (1) | CN111784586B (en) |
Families Citing this family (1)
Publication number | Priority date | Publication date | Assignee | Title |
---|---|---|---|---|
CN114581338A (en) * | 2022-03-21 | 2022-06-03 | 上海商汤科技开发有限公司 | An image correction method, device, computer equipment and storage medium |
Citations (3)
Publication number | Priority date | Publication date | Assignee | Title |
---|---|---|---|---|
CN109472739A (en) * | 2018-10-29 | 2019-03-15 | 深圳云行智能科技有限公司 | A kind of fish eye images antidote and device for wide-angle camera |
CN110248079A (en) * | 2019-02-14 | 2019-09-17 | 南京泓众电子科技有限公司 | A kind of full view image generating system and panorama image generation method |
CN110675453A (en) * | 2019-10-16 | 2020-01-10 | 北京天睿空间科技股份有限公司 | Self-positioning method for moving target in known scene |
Family Cites Families (1)
Publication number | Priority date | Publication date | Assignee | Title |
---|---|---|---|---|
KR102227583B1 (en) * | 2018-08-03 | 2021-03-15 | 한국과학기술원 | Method and apparatus for camera calibration based on deep learning |
-
2020
- 2020-05-25 CN CN202010446601.0A patent/CN111784586B/en active Active
Patent Citations (3)
Publication number | Priority date | Publication date | Assignee | Title |
---|---|---|---|---|
CN109472739A (en) * | 2018-10-29 | 2019-03-15 | 深圳云行智能科技有限公司 | A kind of fish eye images antidote and device for wide-angle camera |
CN110248079A (en) * | 2019-02-14 | 2019-09-17 | 南京泓众电子科技有限公司 | A kind of full view image generating system and panorama image generation method |
CN110675453A (en) * | 2019-10-16 | 2020-01-10 | 北京天睿空间科技股份有限公司 | Self-positioning method for moving target in known scene |
Also Published As
Publication number | Publication date |
---|---|
CN111784586A (en) | 2020-10-16 |
Similar Documents
Publication | Publication Date | Title |
---|---|---|
CN106875451B (en) | Camera calibration method and device and electronic equipment | |
CN112434709A (en) | Aerial survey method and system based on real-time dense three-dimensional point cloud and DSM of unmanned aerial vehicle | |
US20200082611A1 (en) | Generating three-dimensional geo-registered maps from image data | |
CN111355884B (en) | Monitoring method, device, system, electronic equipment and storage medium | |
CN109035327B (en) | Panoramic camera pose estimation method based on deep learning | |
CN111123962A (en) | Rotor unmanned aerial vehicle repositioning photographing method for power tower inspection | |
CN113808269B (en) | Map generation method, positioning method, system and computer readable storage medium | |
CN116883251B (en) | Image orientation splicing and three-dimensional modeling method based on unmanned aerial vehicle video | |
CN113436267A (en) | Visual inertial navigation calibration method and device, computer equipment and storage medium | |
CN113205591B (en) | Method and device for acquiring three-dimensional reconstruction training data and electronic equipment | |
CN104457758A (en) | Video-acquisition-based Visual Map database establishing method and indoor visual positioning method using database | |
CN111784586B (en) | Panoramic image level correction method and system based on self-supervised learning | |
CN101545775A (en) | Method for calculating orientation elements of photo and the height of building by utilizing digital map | |
CN113436269A (en) | Image dense stereo matching method and device and computer equipment | |
CN110503604B (en) | A Real-time Ortho-Mosaic Method of Aerial Area Array Image Based on High-precision POS | |
CN111062875B (en) | Coordinate conversion method and device for air panoramic roaming data | |
CN115131494A (en) | Optical remote sensing satellite imaging simulation method and device | |
JP3490774B2 (en) | How to generate geospecific textures | |
CN118509713A (en) | Video processing method, device, computer equipment and storage medium | |
CN114445583A (en) | Data processing method, device, electronic device and storage medium | |
CN117036666B (en) | UAV low-altitude positioning method based on inter-frame image stitching | |
CN112184662A (en) | Camera external parameter initial method and system applied to unmanned aerial vehicle image stitching | |
CN117422650A (en) | Panoramic image distortion correction method and device, electronic equipment and medium | |
CN117351047A (en) | Local map point cloud global alignment method and device, electronic equipment and storage medium | |
CN112419172B (en) | Remote sensing image processing method for correcting and deblurring inclined image |
Legal Events
Date | Code | Title | Description |
---|---|---|---|
PB01 | Publication | ||
PB01 | Publication | ||
SE01 | Entry into force of request for substantive examination | ||
SE01 | Entry into force of request for substantive examination | ||
GR01 | Patent grant | ||
GR01 | Patent grant |