TWI769782B - Method for optimizing image stitching and computer readable storage medium - Google Patents

Abstract

The present invention provides a method for optimizing image stitching and a computer readable storage medium, which obtains a flat fisheye image that projects each pixel of the fisheye image onto a three-dimensional spherical surface and maps it back to a two-dimensional plane by calculating a corresponding non-linear deformation relationship between a fisheye image of an object and entity thereof, and then stitches the continuous fisheye images, such that problems of stitching afterimages and screen breaks that occur when the fisheye image is flattened and stitched will be greatly reduced, and a better image stitching optimization effect can be achieved.

Description

Method and computer-readable storage medium for image stitching optimization

The present invention relates to multimedia technology and its applications, in particular to a method and computer-readable storage medium for image stitching optimization.

With the advent of the 5G era, the problem of insufficient bandwidth for audio and video transmission has been solved, prompting the emergence of new types of multimedia interactive applications. Among them, the high interaction and high immersion provided by 360-degree panoramic images are deeply loved by consumers, and are also used in large-scale activities such as exhibitions and ball games, thereby bringing a sense of presence that traditional flat images cannot match.

However, when a fisheye camera is used to obtain a 360-degree panoramic image, it is usually necessary to perform dewarping and stitching processing on two fisheye images with a viewing angle of 180 degrees. At this time, the quality of the stitched image depends on the degree of distortion of the flattened fisheye image, and there may be problems such as stitching afterimage and screen breakage.

Therefore, the present invention provides a method and a computer-readable medium for image stitching optimization, so as to solve the problems of stitching afterimage and screen breakage when producing panoramic images from fisheye images, thereby providing better optimized 360-degree panoramic images screen effect.

In order to solve the above problems, the present invention provides a method for image stitching optimization, which includes: shooting a fisheye image with a specific angle between the object and the optical axis of the camera, wherein the object has a plurality of marking points; recording each The pairing data of the first distance between each of the plurality of marking points in the fisheye image and the center point of the fisheye image and the specific angle corresponding to the fisheye image when shooting the fisheye image; and calculating a representative of the camera according to the pairing data Non-linear relationship between incident light and imaging radius when shooting.

In the above method, the step of photographing a fisheye image with a specific angle between the optical axis of the subject and the camera further includes the following sub-steps: pivoting the camera around the axis of the tripod at a fixed angle to make the subject The specific angle between the subject and the optical axis of the camera is the cumulative value of the fixed angle each time the camera is pivoted; the subject and the optical axis of the camera are photographed at the specific angle each time the camera is pivoted the fisheye image; and when the specific angle is greater than or equal to 360 degrees, stop the camera from pivoting and shooting.

In the above method, the nonlinear relationship is represented by an N-degree equation, wherein the first distance of the paired data corresponds to a variable value of the N-degree equation, wherein the specific included angle of the paired data corresponds to the N-degree equation The function value of the equation, and wherein, the step of calculating the nonlinear relationship between the incident light and the imaging radius when the camera shoots according to the paired data includes calculating the coefficient value of the Nth-order equation according to the paired data.

In the above-mentioned method, further comprising performing the flattening of the fisheye image according to the nonlinear relationship, it includes the following sub-steps: obtaining two-dimensional plane coordinates of pixels in the fisheye image based on the center point of the fisheye image Calculate the second distance between the pixel and the center point of the fisheye image according to the two-dimensional plane coordinate value of the pixel; calculate the second distance according to the nonlinear relationship to calculate the elevation angle of the pixel to the three-dimensional spherical surface, and according to The two-dimensional plane coordinate value of the pixel calculates the horizontal angle of the pixel projected to the three-dimensional spherical surface; calculates the three-dimensional projection of the pixel to the three-dimensional spherical surface according to the elevation angle and the horizontal angle dimensional coordinate values; project all the pixels of the fisheye image to the 3D sphere according to the corresponding 3D coordinate values; and map all the pixels from the 3D sphere back to a 2D plane by equidistant cylindrical projection to generate Flatten the fisheye image.

In the above-mentioned method, the method further comprises: stitching the two continuous scenes in the flattened fisheye image to obtain a panoramic image.

The present invention further provides a computer-readable storage medium, which is applied to a computer and stores instructions for executing the above-mentioned method for image stitching optimization.

To sum up, the method and computer-readable medium for image stitching optimization of the present invention obtain the fisheye image by calculating the corresponding nonlinear deformation relationship between the fisheye image of the subject captured by the camera and the entity. Each pixel is projected to a 3D spherical surface and mapped back to a flat fisheye image of a 2D plane, and then continuous fisheye images are stitched, which can greatly reduce the stitching afterimage and screen breakage generated when the fisheye image is flattened and stitched. problem, to achieve better image stitching optimization effect.

10: Camera

20: Subject

21: Mark Points

m: pixel

O: ball center

o: center point of fisheye image

Q: Incident light

q: punctuation

r: distance

r _m : distance

S1~S10: Steps

W: constant

θ _m : elevation angle

φ: horizontal angle

The specific embodiments disclosed in this case will be described in detail with the following drawings, and these descriptions are shown in the following drawings:

FIG. 1 is a flowchart showing the steps of the method for image stitching optimization of the present invention;

FIG. 2 is a schematic diagram showing the implementation stage of the method for image stitching optimization of the present invention;

3 is a schematic diagram showing the implementation stage of the method for image stitching optimization of the present invention;

4 is a schematic diagram showing the implementation stage of the method for image stitching optimization of the present invention;

FIG. 5 is a schematic diagram showing the implementation stage of the method for image stitching optimization of the present invention;

6 is a schematic diagram showing the implementation stages of the method for image stitching optimization of the present invention; and

FIG. 7 is a schematic diagram showing the implementation of the method for image stitching optimization of the present invention.

The embodiments of the present invention are described below by specific examples, and those skilled in the art can easily understand other advantages and effects of the present invention from the content disclosed herein. The structures, proportions, sizes, etc. shown in the drawings in this specification are only used to cooperate with the contents disclosed in the specification for the understanding and reading of those skilled in the art, and are not intended to limit the conditions for the implementation of the present invention. Therefore, any modification, change or adjustment should still fall within the scope that the technical content disclosed in the present invention can cover without affecting the effect that the case can produce and the purpose that can be achieved.

FIG. 1 is a flow chart showing the steps of the method for image stitching optimization of the present invention, wherein each step S1 - S10 can be clearly understood by the following description and the schematic diagrams of the implementation stages in the following figures.

At step S1, as shown in FIG. 2, a subject 20 can be prepared in front of the camera (eg, a fisheye camera, or other photographing equipment suitable for generating panoramic images) 10, and the subject 20 has a plurality of Mark the dots 21 (the surface of the checkerboard pattern as indicated by the reference numeral 21) as the subject 20 in the fish-eye shadow captured Deformed or distorted reference in the image. The camera 10 can pivot relative to the axis of the tripod, so that fisheye images of the subject 20 captured by the camera 10 at different shooting angles can be obtained.

At step S2 , as shown in FIG. 3 , the camera 10 can be pivoted in the same direction according to a fixed angle α to capture a fisheye image including the subject 20 . That is, each time the camera 10 shoots, the angle between the subject 20 and the optical axis of the camera 10 will increase by α degree compared with the previous time, and the relationship between the marker points 21 in the obtained fisheye image and the camera 10 is circled as shown in FIG. 3 . The images of the camera 10 are generally distributed in a circular manner. It should be noted that, the value of the included angle α between the optical axis and the subject 20 for each pivoting of the camera 10 can be changed according to operational requirements, which is not particularly limited herein.

In an alternative embodiment, without rotating the camera 10 , the marking point 21 of the subject 20 may face the camera 10 , and move around the camera 10 with a fixed radius around the camera 10 by increasing the included angle α each time. , the fisheye images of different angles between the optical axis of the subject 20 and the camera 10 can also be generated. However, according to the effect requirements of image stitching optimization, the rotation of the camera 10 or the circular movement of the subject 20 is not particularly limited herein. Further, the rotation of the camera 10 or the circular movement of the subject 20 can be performed manually or automatically, which is not particularly limited herein.

At step S3, as shown in FIG. 4, when shooting fisheye images with different angles between the optical axis of the subject 20 and the camera 10, the relative fisheye relative to the fisheye of each marking point 21 of the subject 20 in each fisheye image will be recorded The distance from the center point o of the image (that is, at the optical axis of the camera 10) (represented by the code r at this time), and the angle between the subject 20 and the optical axis of the camera 10 (represented by the code θ at this time), and represented by ( θ ,r ) in the form of paired data records.

In one embodiment, the camera 10 is shot with the angle α of each rotation being 20 degrees, then the recorded distance r of each marked point 21 of the subject 20 relative to the center point o of the fisheye image and the subject The angle θ between the 20 present and the optical axis of the camera 10 can be expressed as the following mathematical formula, for example:

[Mathematical formula 1]

Among them, each included angle θ will have its own distance for each marked point of the subject 20 (for example, if a subject has 20 marked points, then each included angle θ will have 20 distances ( θ, r )).

Following step S4, the above-mentioned steps S2 to S3 of rotating the camera 10 (or moving the subject 20) and recording the pairing data ( θ,r ) will be repeated until the position of the subject 20 in the fisheye image returns to the original position (when the accumulated value of the included angle α of the rotating camera 10 reaches 360 degrees or more) (ie, the camera 10 completes one rotation or the subject 20 completes one circular movement).

It should be noted that the determination of whether to repeat steps S2 to S3 at step S4 is not limited to the completion of one rotation of the camera 10 or the completion of a circular movement (ie, 360 degrees) of the subject 20, and the criteria for the determination are also There are different settings depending on the shooting capability or work requirements of the camera 10 . Specifically, a circle of rotation or a circle of circular movement as defined herein is for the camera 10 with two or more fisheye lenses (for example, two or more fisheye lenses with a 180-degree viewing angle are positioned away from each other However, for the camera 10 with only one fisheye lens, it can be designed only to determine the rotation of the camera 10 or the circumference of the subject 20. The repeated execution of steps S2 to S3 can be stopped when the movement completes half a circle (ie, 180 degrees).

The above steps S1 to S4 are used to obtain calibration data when flattening the fisheye image captured by the camera 10 , so as to find out the relationship between the subject 20 and its imaging deformation when the camera 10 captures the fisheye image in subsequent steps.

In the fisheye image, as the included angle (θ) between the subject 20 and the optical axis of the camera 10 is larger, the distortion of the subject 20 in the fisheye image is more serious, and each marked point of the subject 20 is more serious. 21 is farther from the center point in the fisheye image, so the relationship can be regarded as the incident light angle (that is, the included angle θ) and the imaging radius (that is, the distance of the camera 10 (fisheye camera) when shooting the subject 20 ). r) nonlinear relationship. Therefore, step S5 is to solve the coefficient value of the Nth equation for the collected paired data ( θ, r ) according to the above relationship, as shown in the following mathematical formula:

[Math 2] a ₀ + a ₁ +…+ a _N r ^N = θ

Wherein, the variable value r represents the distance of each marked point 21 of the aforementioned subject 20 relative to the center point (ie, the optical axis of the camera 10 ) in the fisheye image;

Wherein, the function value θ represents the angle between the aforementioned subject 20 and the optical axis of the camera 10;

In one embodiment, the N value of the N-th power may be different according to cameras 10 of different brands and specifications, so as to provide the camera 10 with the best image stitching optimization effect, and it should be noted that the larger the N value is, the larger the N value is. , the effect of image stitching optimization is more significant, but the burden on the computer is heavier, so it will have a more significant effect for devices with strong computer computing power.

Then, all recorded paired data ( θ, r ) (for example, all the values of Equation 1) can be brought into the above Equation 2 to obtain the coefficient values of a ₀ to a _N of the Nth-order equation, and then obtain this The nonlinear relationship between the subject 20 captured by the camera 10 and its imaging deformation. In this embodiment, it is assumed that the N value of the N-th degree equation is 4. However, according to different brands or specifications of the camera 10, the N value can also be any value that contributes to the best image stitching optimization effect. It is not particularly limited herein.

It should be noted that the purpose of the aforementioned steps S1 to S5 is to obtain the relationship between the deformation of the image captured by the camera 10 and the corresponding object by photographing the test object (ie, the subject 20 ), so for a specific camera 10 , it only needs to be performed once In steps S1 to S5 , steps S6 to S10 can be performed according to the mathematical formula 2 obtained in step S5 for subsequent fisheye images obtained through the camera 10 to perform corresponding image stitching optimization.

As shown in the conversion process from left to right in FIG. 5 , steps S6 to S8 reveal the process of projecting each pixel in the fisheye image to a 3D sphere (similar to simulating a fisheye camera that incorporates the image of the scene into its lens to generate an image) . At this time, since the coefficient values of a ₀ to a _N in the Nth-order equation (ie, the mathematical equation 2) have been obtained in step S5, the mathematical equation 2 can be used to calculate any pixel in the fisheye image in step S6. (pixel m in Figure 5) is projected to the elevation angle θ _m of the three-dimensional sphere.

The processing flow of step S6 includes: taking the pixel m shown on the left side of FIG. 5 as a reference to the center point o of the fisheye image to obtain its two-dimensional plane coordinate value ( u, v ) in the fisheye image; The plane coordinate value ( u, v ) calculates its distance r _m relative to the center point o of the fisheye image; the obtained distance r _m is brought into the mathematical formula 2 of the solution to obtain the elevation angle θ of the pixel m projected to the three-dimensional spherical surface _m . Among them, the calculation method of the above distance _rm is shown in the following mathematical formula:

Continuing from step S7, according to the two-dimensional plane coordinate values ( u, v ) of the above-mentioned pixel m, the horizontal angle φ of the pixel m projected to the three-dimensional spherical surface can also be calculated, as shown in the following mathematical formula:

Continue to step S8, according to the calculated elevation angle θ _m and horizontal angle φ, the three-dimensional coordinate value of the coordinate point q of the pixel m in the fisheye image projected to the three-dimensional spherical surface can be calculated, as shown in the following mathematical formula:

[Equation 5] X=W sin θ cos φ , Y=W sin θ sin φ , Z=W cos θ

Among them, W in the mathematical formula 5 is used as a constant, representing the distance of any point (for example, the coordinate point q) on the three-dimensional spherical surface relative to the center of the sphere O, which is also equivalent to the periphery of the fisheye image relative to the center point of the fisheye image o the distance;

And among them, the X, Y and Z values obtained by the above-mentioned mathematical formula 5 represent the three-dimensional coordinate values ( X, Y, Z ) of the coordinate point q of the pixel m projected to the three-dimensional spherical surface in the fisheye image, that is, when the camera 10 When shooting the real object corresponding to the pixel m, the position where the incident light Q enters the three-dimensional spherical surface.

The transformation process shown in the above-mentioned steps S6 to S8 and FIG. 5 is performed for all the pixels in the fisheye image. After calculating the three-dimensional coordinate values of all the pixels in the fisheye image projected to the three-dimensional spherical surface, step S9 can be executed successively, Each pixel of the fisheye image is mapped from the coordinate values of the three-dimensional spherical surface back to a two-dimensional plane by equidistant cylindrical projection, so as to generate a flattened fisheye image. As shown in the transformation process shown in Figure 6, the original fisheye image (left image) is flattened according to the calculated 3D coordinate values of each pixel on the 3D spherical surface using the equidistant cylindrical projection (middle image), and an optimized distribution can be obtained. Flat fisheye image (right image).

Finally, at step S10 , two flattened fisheye images with continuous scenes (eg, two flattened fisheye images with a viewing angle of 180 degrees) may be stitched to obtain an optimized 360-degree panoramic image. FIG. 7 is a schematic diagram showing the implementation before and after flattening and stitching two fisheye images. It can be clearly seen that in the process of using the above steps S1 to S10 to generate the two fisheye images into a 360-degree panoramic image, phenomena such as stitching afterimages and screen breakage have been greatly optimized.

As described previously, since only one evaluation calculation of the coefficient values of a ₀ to a _N in the aforementioned mathematical formula 2 is required for a specific camera 10 , after the mathematical formula 2 is obtained in step S5 , any arbitrary value captured by the camera 10 can be obtained. The flattening and stitching of steps S6 to S10 are repeated for the continuous fisheye images, so as to achieve the effect of image stitching optimization.

The present invention further provides a computer-readable storage medium, which is applied to a computer or computing device having a processor and/or a memory, and stores instructions. The computer or computing device uses the processor (eg, CPU, GPU, etc.) and/or Or the memory can execute the image stitching optimization method described above through the instruction.

To sum up, the method and computer-readable medium for image stitching optimization of the present invention obtain the fisheye image by calculating the corresponding nonlinear deformation relationship between the fisheye image of the subject captured by the camera and the entity. Each pixel is projected to a 3D spherical surface and mapped back to a flat fisheye image of a 2D plane, and then continuous fisheye images are stitched, which can greatly reduce the stitching afterimage and screen breakage generated when the fisheye image is flattened and stitched. problem, to achieve better image stitching optimization effect.

S1~S10: Steps

Claims (7)

A method for image stitching optimization, comprising: shooting a fish-eye image with a specific angle θ between a subject and an optical axis of a camera, wherein the subject has a plurality of marking points; recording the subject in each fish-eye image pairing data (θ, r) of the first distance r from each of the marked points of the subject to the center point of the fisheye image and the specific angle θ corresponding to the fisheye image; and according to the subject The paired data (θ, r) recorded from the first distance r from each marker point of the object to the center point of the fisheye image and the specific angle θ corresponding to the fisheye image when shooting The calculation represents the nonlinear relationship between the incident light and the imaging radius when the camera is shooting, which is represented by the Nth-order equation as a ₀ +a ₁ r+...+a _N r ^N =θ, where the paired data (θ, r ) in the first distance r from the marked point of the subject to the center point of the fisheye image and the specific angle θ corresponding to the fisheye image are brought into the N-th equation as a ₀ +a ₁ r+...+a _N r ^N = θ, so that the N-th equation is calculated as a ₀ +a ₁ r+...+a _N r ^N = θ by using the paired data (θ, r). Coefficient values a ₀ to a _N . The method of claim 1, wherein the step of capturing a fish-eye image with a specific angle between the optical axis of the subject and the camera further comprises the following sub-steps: pivoting the camera around its tripod axis at a fixed angle rotating, so that the specific angle between the subject and the optical axis of the camera is the accumulated value of the fixed angle each time the camera is pivoted; and shooting the subject and the camera each time the camera is pivoted. The fisheye image with the optical axis at the specific angle. The method of claim 2, wherein the step of capturing a fish-eye image with a specific angle between the subject and the optical axis of the camera further comprises the following sub-steps: When the specific angle is greater than or equal to 360 degrees, the camera is stopped from pivoting and shooting. The method of claim 1, wherein the first distance of the paired data corresponds to a variable value of the Nth-degree equation, and the specific included angle of the paired data corresponds to a function value of the Nth-degree equation. The method of claim 1, further comprising performing the flattening of the fisheye image according to the nonlinear relationship, including the following sub-steps: obtaining two pixels of the fisheye image based on the center point of the fisheye image dimensional plane coordinate value; calculate the second distance between the pixel and the center point of the fisheye image according to the two-dimensional plane coordinate value of the pixel; calculate the second distance according to the nonlinear relationship to calculate the elevation angle of the pixel projected to the three-dimensional spherical surface , and calculate the horizontal angle of the pixel projected to the three-dimensional spherical surface according to the two-dimensional plane coordinate value of the pixel; calculate the three-dimensional coordinate value of the pixel projected to the three-dimensional spherical surface according to the elevation angle and the horizontal angle; All the pixels are projected onto the 3D sphere according to the corresponding 3D coordinate values; and all the pixels are mapped from the 3D sphere back to a 2D plane by equidistant cylindrical projection to generate a flattened fisheye image. The method according to claim 5, further comprising: stitching two of the flattened fisheye images with continuous scenes to obtain a panoramic image. A computer-readable storage medium used in a computer and storing instructions for executing the method for image stitching optimization as described in any one of claims 1 to 6.

Images