CN111784586B - Panoramic image level correction method and system based on self-supervised learning - Google Patents

Abstract

The invention discloses a self-supervised learning panorama image level correction method and system, wherein the method includes: constructing a training image library; using the training images in the training image library to perform self-supervised training on a convolutional neural network until the volume The error function of the convolutional neural network converges; the to-be-corrected image in the training image library is input into the trained convolutional neural network, and the pitch angle and roll angle of the camera when the current panoramic image is captured are derived; The Euler angle is used to calculate the rotation matrix, and the rotation matrix of the camera pose is obtained, and the image to be corrected is corrected according to the rotation matrix of the camera pose, and the horizontal image is synthesized. This method effectively solves the problem of distortion and distortion of panoramic images caused by the non-vertical camera posture.

Description

Panoramic image level correction method and system based on self-supervised learning

technical field

The invention relates to the technical field of image correction, in particular to a self-supervised learning method and system for horizontal correction of panoramic images.

Background technique

Existing panoramic images can be applied in scenarios such as virtual reality and 3D reconstruction. When shooting a panoramic image, the posture of the panoramic camera will have a great influence on the collected image. Ideally, the posture of the panoramic camera should be kept perpendicular to the ground, and the panoramic image collected at this time is horizontal. It can be intuitively understood that the horizon in the image is approximately a horizontal line. However, in an actual situation, the panoramic camera placement is not necessarily vertical, or there is interference such as camera shake, so that the captured panoramic image is not horizontal. As shown in Figure 1, the horizon in the image is approximately a sine curve, and will be accompanied by large distortion and distortion, which will bring great trouble to the subsequent use of the panoramic image.

In the related art, an additional gyroscope is used to obtain the three-axis pose of the camera, and rotation correction is performed on the collected image to generate a horizontal image. However, this method will increase the hardware cost and complexity of the system, and at the same time, some panorama images do not have matching gyroscope data when they are collected.

Therefore, there is an urgent need for a solution that can solve the problem of distortion and distortion of panoramic images caused by the non-vertical camera posture.

SUMMARY OF THE INVENTION

The present invention aims to solve one of the technical problems in the related art at least to a certain extent.

Therefore, an object of the present invention is to propose a self-supervised learning method for horizontal correction of panoramic images, which can solve the problem of distortion of panoramic images caused by vertical cameras without additional auxiliary equipment, and has low hardware cost and complexity. Low.

Another object of the present invention is to propose a self-supervised learning panorama image level correction system.

In order to achieve the above purpose, an embodiment of the present invention proposes a self-supervised learning panorama image level correction method, including the following steps: step S1, constructing a training image database; step S2, using the training images of the training image database to convolve The neural network performs self-supervised training until the error function of the convolutional neural network converges; step S3, input the to-be-corrected image in the training image library into the trained convolutional neural network, and derive the The pitch angle and roll angle of the camera when the image is to be corrected; step S4, Euler angle reckoning rotation matrix calculation is performed on the pitch angle and the roll angle to obtain the rotation matrix of the camera pose, according to the rotation of the camera pose Matrix corrects the to-be-corrected image to synthesize a horizontal image.

The panoramic image level correction method for self-supervised learning according to the embodiment of the present invention acquires a sufficient number of horizontal panoramic images, and uses a large number of horizontal panoramic images to perform self-supervised training on the neural network, so that the network has the ability to infer the camera pose angle, and estimates The rotation angle of the camera when the panoramic image is captured, the camera pose is compensated according to the rotation angle, and a horizontal panoramic image is obtained, which effectively solves the problem of distortion and distortion of the panoramic image caused by the non-vertical camera attitude.

In addition, the panoramic image level correction method of self-supervised learning according to the above-mentioned embodiment of the present invention may also have the following additional technical features:

Further, in an embodiment of the present invention, each panoramic image in the training image library is a horizontal angle.

Further, in an embodiment of the present invention, the step S2 includes: using the pitch angle and the roll angle in the Euler angles as supervision information; Perform iterative training to generate several rotated non-horizontal images; when the error between the rotation angle of the non-horizontal image and the rotation angle in the supervision information is the smallest, the training of the convolutional neural network is completed.

Further, in an embodiment of the present invention, before performing iterative training on the panoramic image, the panoramic image is mapped onto a spherical surface.

In order to achieve the above object, another embodiment of the present invention proposes a panoramic image level correction system for self-supervised learning, including: a building module for building a training image library; a training module for using the training images in the training image library Carry out self-supervised training on the convolutional neural network until the error function of the convolutional neural network converges; the derivation module is used to input the to-be-corrected image in the training image library into the trained convolutional neural network, Deriving the pitch angle and roll angle of the camera when shooting the to-be-corrected image; the correction module is used to calculate the Euler angle reckoning rotation matrix for the pitch angle and the roll angle, and obtain the rotation matrix of the camera pose, according to The rotation matrix of the camera pose corrects the to-be-corrected image to synthesize a horizontal image.

The panoramic image level correction system for self-supervised learning according to the embodiment of the present invention acquires a sufficient number of horizontal panoramic images, and uses a large number of horizontal panoramic images to perform self-supervised training on the neural network, so that the network has the ability to infer the camera pose angle, and estimates The rotation angle of the camera when the panoramic image is captured, the camera pose is compensated according to the rotation angle, and a horizontal panoramic image is obtained, which effectively solves the problem of distortion and distortion of the panoramic image caused by the non-vertical camera attitude.

In addition, the panoramic image level correction system for self-supervised learning according to the above embodiments of the present invention may also have the following additional technical features:

Further, in an embodiment of the present invention, each panoramic image in the training image library is a horizontal angle.

Further, in an embodiment of the present invention, the training module is further used for: a determination unit for using the pitch angle and the roll angle in Euler angles as supervision information; a training unit for passing the supervision information Perform iterative training on each panorama image in the training image library to generate several rotated non-horizontal images; the judgment unit is used for when the rotation angle of the non-horizontal image and the rotation angle in the supervision information are different. With the smallest error, the training of the convolutional neural network is completed.

Further, in an embodiment of the present invention, before performing iterative training on the panoramic image, the panoramic image is mapped onto a spherical surface.

Additional aspects and advantages of the present invention will be set forth, in part, from the following description, and in part will be apparent from the following description, or may be learned by practice of the invention.

Description of drawings

The above and/or additional aspects and advantages of the present invention will become apparent and readily understood from the following description of embodiments taken in conjunction with the accompanying drawings, wherein:

Figure 1 is a schematic diagram of a large number of distortions and deformities in a non-horizontal panoramic image;

FIG. 2 is a flow chart of a panoramic image level correction method for self-supervised learning according to an embodiment of the present invention;

3 is a schematic diagram of the correspondence between a panoramic image plane and a sphere according to an embodiment of the present invention;

FIG. 4 is a simplified flowchart of a panoramic image level correction method for self-supervised learning according to an embodiment of the present invention;

5 is a schematic diagram of image comparison before and after correction according to an embodiment of the present invention;

FIG. 6 is a schematic structural diagram of a panoramic image level correction system for self-supervised learning according to an embodiment of the present invention.

Detailed ways

The following describes in detail the embodiments of the present invention, examples of which are illustrated in the accompanying drawings, wherein the same or similar reference numerals refer to the same or similar elements or elements having the same or similar functions throughout. The embodiments described below with reference to the accompanying drawings are exemplary, and are intended to explain the present invention and should not be construed as limiting the present invention.

The method and system for level correction of panoramic images according to self-supervised learning according to the embodiments of the present invention will be described below with reference to the accompanying drawings.

FIG. 1 is a flow chart of a panoramic image level correction method for self-supervised learning according to an embodiment of the present invention.

As shown in Figure 1, the self-supervised learning panorama image level correction method includes the following steps:

In step S1, a training image library is constructed, wherein each panoramic image in the training image library is a horizontal angle.

Specifically, to collect a large number of panoramic images shot horizontally, the panoramic images can be collected on a tripod or a trolley, or some public panoramic image databases can be used, or crawled from a street view map website. Regardless of the image source, it must be ensured that each panorama image in the image library is horizontal. The image can be properly pre-processed, such as cropping the area where the shooting device or the support device appears in the image. Step S1 obtains a sufficient number of horizontal panoramic images for subsequent model training.

In step S2, self-supervised training is performed on the convolutional neural network using the training images in the training image library until the error function of the convolutional neural network converges.

Further, in an embodiment of the present invention, step S2 includes: using the pitch angle and the roll angle in the Euler angles as supervision information; performing iterative training on each panoramic image in the training image library through the supervision information, and generating several A rotated non-horizontal image; when the error between the rotation angle of the non-horizontal image and the rotation angle in the supervision information is the smallest, the training of the convolutional neural network is completed.

That is to say, step S2 is to apply random three-dimensional rotation to each training image, and record the rotation angle. Wherein, the three-dimensional rotation does not refer to a simple plane rotation, but a three-dimensional space rotation of the posture of the camera when the image is captured. The rotation needs to map the panoramic image to the spherical surface of the device, multiply the matrix and coordinates corresponding to the rotation, and then map it back to the plane. The specific steps are described below.

Among them, the three-dimensional rotation can be represented by Euler angles—that is, three angles, and one of them is the rotation in the horizontal direction, which is meaningless for horizontal correction, so only two of them are selected: pitch angle and drum angle. roll. So far, each horizontal image can generate several rotated non-horizontal images, and the two rotation angles pitch and roll experienced by it are known. Using the two known rotation angles as training supervision information, try to minimize the error between the rotation angle output by the network and the real rotation angle when the input corresponds to a non-horizontal image. After repeated iterative training, until the error function of the convolutional neural network converges, the model parameters at this time are saved.

In step S3, the images to be corrected in the training image library are input into the trained convolutional neural network, and the pitch angle and the roll angle of the camera when shooting the to-be-corrected image are derived.

That is, load the model parameters trained in the previous step for the network model. After preprocessing, such as scaling and cropping, the panoramic image is sent to the convolutional neural network. The neural network inference obtains the pitch angle of the camera and the roll angle of the camera when the panoramic image is captured.

In step S4, the Euler angle reckoning rotation matrix is calculated for the pitch angle and the drum angle to obtain the rotation matrix of the camera pose, and the image to be corrected is corrected according to the rotation matrix of the camera pose to synthesize a horizontal image.

That is to say, after estimating the pitch angle pitch and roll angle of the camera when the input panoramic image is taken, the rotation matrix R representing the camera pose is calculated with reference to the relevant data calculated from the Euler angle. As shown in Figure 3, a panoramic image can be mapped onto a panoramic spherical surface, the horizontal direction can be regarded as the longitude direction, and the vertical direction can be regarded as the latitude direction. Generally, the mapping relationship between a certain pixel (u, v) on the panoramic image and the coordinates (x, y, z) on the spherical surface is as follows, where U _c , V _c , f are the abscissa, ordinate and y of the center point of the panoramic image, respectively. pixel.

After obtaining the point coordinates on the spherical surface corresponding to each pixel on the panoramic image, use the inverse matrix R ^-1 of the camera pose rotation matrix R to compensate the spherical coordinates to obtain the new spherical coordinates (x _- , y _{_} , z _{_} ) =R ^-1 (x,y,z) ^T . Furthermore, by using the above mapping relationship, the corresponding pixel coordinates (u _- , v _{_} ) in the corrected panoramic image can be inversely calculated. According to the pixel correspondence (u, v) (u _- , v _{_} ) in the panoramic images before and after correction, a new corrected horizontal panoramic image can be synthesized.

To put it simply, as shown in FIG. 4 , the working principle of the panoramic image level correction method for self-supervised learning according to the embodiment of the invention is as follows: first, a training image library is constructed, and images in the training image library are used to perform self-supervised training, and the trained images are obtained. Input the image to be corrected into the trained neural model, calculate the rotation angle of the camera, correct the image to be corrected according to the angle, and synthesize a new horizontal panoramic image.

According to the self-supervised learning panorama image level correction method proposed in the embodiment of the present invention, by constructing a large-scale training image database, and using the self-supervised learning method, a network model capable of predicting the camera's three-axis posture during panoramic image shooting is obtained by training. , using this model, the pitch angle pitch and roll angle of the camera can be estimated when the input panoramic image is taken, and then the panoramic image is rotated three-dimensionally to generate a horizontal panorama. As shown in Figure 5, compared with the unprocessed panorama, the horizontal panorama has a smaller degree of distortion and distortion, which makes the subsequent retrieval, deduplication, feature extraction, feature matching and other operations more effective, which can be used as a kind of A powerful preprocessing method for panoramic images has great theoretical and practical value for technical research in this field.

Next, the panoramic image level correction system for self-supervised learning proposed according to the embodiment of the present invention will be described with reference to the accompanying drawings.

FIG. 6 is a schematic structural diagram of a panoramic image level correction system for self-supervised learning according to an embodiment of the present invention.

As shown in FIG. 6 , the system 10 includes: a building module 100 , a training module 200 , a derivation module 300 and a correction module 400 .

The building module 100 is used to construct a training image library, and each panoramic image in the training image library is a horizontal angle. The training module 200 is configured to perform self-supervised training on the convolutional neural network using the training images in the training image library until the error function of the convolutional neural network converges. The derivation module 300 is configured to input the to-be-corrected image in the training image library into the trained convolutional neural network, and derive the pitch angle and the roll angle of the camera when the to-be-corrected image is captured. The correction module 400 is configured to perform Euler angle reckoning and rotation matrix calculation for the pitch angle and the roll angle to obtain a rotation matrix of the camera pose, correct the image to be corrected according to the rotation matrix of the camera pose, and synthesize a horizontal image.

Further, in an embodiment of the present invention, the training module 200 is further used for:

Determining unit for using pitch and roll angles in Euler angles as supervision information;

The training unit is used to iteratively train each panoramic image in the training image library through the supervision information, and generate several rotated non-horizontal images;

The judgment unit is used to complete the training of the convolutional neural network when the error between the rotation angle of the non-horizontal image and the rotation angle in the supervision information is the smallest.

Further, in an embodiment of the present invention, before performing iterative training on the panoramic image, the panoramic image is mapped onto a spherical surface.

According to the self-supervised learning panorama image level correction system proposed in the embodiment of the present invention, by constructing a large-scale training image database and using the self-supervised learning method, a network model capable of predicting the three-axis pose of the camera when the panoramic image is captured is obtained by training. , using this model, the pitch angle pitch and roll angle of the camera can be estimated when the input panoramic image is taken, and then the panoramic image is rotated three-dimensionally to generate a horizontal panorama. Compared with the unprocessed panorama, the horizontal panorama has a smaller degree of distortion and distortion, which makes the subsequent retrieval, deduplication, feature extraction, feature matching and other operations more effective, and can be used as a powerful preview of the panorama image. The treatment method has great theoretical and practical value for technical research in this field.

In addition, the terms "first" and "second" are only used for descriptive purposes, and should not be construed as indicating or implying relative importance or implying the number of indicated technical features. Thus, a feature delimited with "first", "second" may expressly or implicitly include at least one of that feature. In the description of the present invention, "plurality" means at least two, such as two, three, etc., unless otherwise expressly and specifically defined.

In the description of this specification, description with reference to the terms "one embodiment," "some embodiments," "example," "specific example," or "some examples", etc., mean specific features described in connection with the embodiment or example , structure, material or feature is included in at least one embodiment or example of the present invention. In this specification, schematic representations of the above terms are not necessarily directed to the same embodiment or example. Furthermore, the particular features, structures, materials or characteristics described may be combined in any suitable manner in any one or more embodiments or examples. Furthermore, those skilled in the art may combine and combine the different embodiments or examples described in this specification, as well as the features of the different embodiments or examples, without conflicting each other.

Although the embodiments of the present invention have been shown and described above, it should be understood that the above embodiments are exemplary and should not be construed as limiting the present invention. Embodiments are subject to variations, modifications, substitutions and variations.

Claims (4)

1. a panoramic image level correction method of self-supervised learning, is characterized in that, comprises the following steps: Step S1, build a training image library; Step S2, using the training images in the training image library to perform self-supervised training on the convolutional neural network, until the error function of the convolutional neural network converges; Step S3, input the image to be corrected in the training image library into the convolutional neural network that has been trained, and derive the pitch angle and roll angle of the camera when shooting the image to be corrected; and Step S4, performing Euler angle reckoning rotation matrix calculation on the pitch angle and the drum angle to obtain a rotation matrix of the camera pose, correcting the image to be corrected according to the rotation matrix of the camera pose, and synthesizing a horizontal image; Wherein, each panoramic image in the training image library is a horizontal angle; Wherein, the step S2 includes: Use the pitch and roll angles in Euler angles as supervision information; Perform iterative training on each panoramic image in the training image library through the supervision information to generate several rotated non-horizontal images; When the error between the rotation angle of the non-horizontal image and the rotation angle in the supervision information is the smallest, the training of the convolutional neural network is completed. 2 . The panoramic image level correction method for self-supervised learning according to claim 1 , wherein, before performing iterative training on the panoramic image, the panoramic image is mapped onto a spherical surface. 3 . 3. A panoramic image level correction system of self-supervised learning, characterized in that, comprising: building blocks for building a library of training images; A training module for self-supervised training of the convolutional neural network using the training images of the training image library, until the error function of the convolutional neural network converges; A derivation module for inputting the images to be corrected in the training image library into the trained convolutional neural network, and deriving the pitch angle and roll angle of the camera when shooting the to-be-corrected images; and The correction module is used to calculate the Euler angle reckoning and rotation matrix for the pitch angle and the drum angle to obtain the rotation matrix of the camera pose, correct the image to be corrected according to the rotation matrix of the camera pose, and synthesize the horizontal image; Wherein, each panoramic image in the training image library is a horizontal angle; Wherein, the training module is further used for: Determining unit for using pitch and roll angles in Euler angles as supervision information; A training unit, configured to perform iterative training on each panoramic image in the training image library through the supervision information, to generate several rotated non-horizontal images; The judgment unit is configured to complete the training of the convolutional neural network when the error between the rotation angle of the non-horizontal image and the rotation angle in the supervision information is the smallest. 4 . The panoramic image level correction system for self-supervised learning according to claim 3 , wherein, before performing iterative training on the panoramic image, the panoramic image is mapped onto a spherical surface. 5 .

Images