KR20090078463A - Distorted image correction apparatus and method - Google Patents

Abstract

A distortion apparatus and a method thereof for compensating video signal and producing the image in which the distortion is corrected from the image photographed through fish-eye lens are provided to calculate a distortion compensation coefficient automatically. A two-dimensional compensation index image about a correction index photographed through camera is received. A distortion correction factor comprising the registration error correction algorithm is calculated by using a plurality of feature data(S430, S440). A compensation file is generated by using the calculated coefficient. The image which is corrected by using the correction algorithm is generated.

Description

Distortion image correction device and method {DISTORTED IMAGE CORRECTION APPARATUS AND METHOD}

The present invention relates to an apparatus and method for correcting a distorted image, and more particularly, to an apparatus and method for correcting a distorted image using an image photographed through a fisheye lens and automatically calculating and correcting a distortion correction coefficient. .

The rear camera of the vehicle or the video surveillance camera provides a user's region of interest as an image. In particular, wide-angle or fish-eye lenses with large angles of view provide the user with a lot of information, including a wider area.

A wide-angle lens is a lens having a shorter focal length than a standard lens and has an angle of view of about 60 ° to 120 °. A fish-eye lens is an ultra-wide angle lens made by using spherical aberration, and has an angle of view of 180 ° or more. Have

In particular, the fisheye lens intentionally creates a cylindrical distortion to maintain uniform brightness and sharpness over the entire angle of view of 180 ° or more. . That is, the fisheye lens can acquire a wide image, and in the case of a car, a larger angle of view of the fisheye lens can provide more information to the driver behind the vehicle. In addition, when the fisheye lens is applied to the video phone, a wide viewing angle can be secured at a short distance.

However, the acquired image has severe distortion, and the camera distortion in particular hinders the realism of the driver.

1A to 1C are exemplary views illustrating an image without lens distortion and an image in which lens distortion occurs.

That is, as shown in FIG. 1A, the image having a straight line vertically and horizontally appears as a curved shape according to the position of the focus as shown in FIG. 1B or 1C due to lens distortion. In particular, FIG. 1B illustrates a pincushion distortion in which a distortion constant k representing a distortion of a lens is less than zero, and FIG. 1C illustrates a barrel having a lens distortion constant k of greater than zero representing a distortion of a lens. It shows that barrel distortion has occurred.

In particular, Figure 1c is an image input from the fisheye lens, it can be seen that the distortion appears more toward the outside of the image.

As a method for correcting the lens distortion, there is a method of physically correcting an additional lens for correcting lens distortion or a method for correcting a software-distorted image. Here, the physical correction method has the disadvantage of increasing the manufacturing cost of the camera in that the correction lens must be added, and software correction is mainly used.

According to the prior art, it was possible to manually correct the distortion correction coefficient of the camera until the test pattern in the form of a grid was selected and the image was not displayed distorted. In other words, by dragging the corners of the test image radially from the center of the image, the contour of the image and the correction of the content of the image were repeated until the distortion disappeared.

According to another conventional technology, by inputting an arbitrary image and increasing or decreasing the range of the lens distortion variable k of the image, the k having the smallest representative value is set as the lens distortion variable, so that the distortion-corrected image from the distorted original image is obtained. To obtain.

However, according to the conventional technique of correcting the geometric distortion of a digital image, a user manipulates or selects it on a screen until the image is not distorted with a test pattern or an arbitrary image.

Therefore, there is a problem that the user has to manually correct the distortion correction constant, and there is a problem that the level of geometric distortion correction of the image changes according to the user's selection criteria.

Accordingly, an aspect of the present invention is directed to a distortion image correction apparatus and method for automatically calculating distortion correction coefficients and correcting distortion images in order to minimize distortion of an image resulting from the use of a fisheye lens.

According to an aspect of the present invention, there is provided a method of correcting a distorted image, the method including: receiving a two-dimensional correction index image of a correction index photographed through a camera; Extracting a plurality of feature data from the correction indicator image; Calculating a distortion correction coefficient constituting a distortion correction algorithm using the plurality of feature data; And generating a correction file using the calculated coefficients.

The method may further include generating a corrected image by using the correction algorithm.

The correction indicator image may be generated by photographing the correction indicator having a plurality of square grids through the camera.

The correction index, the plurality of square lattice may be arranged in the form of odd X even or even X odd.

The feature data may include vertex coordinate information and vertex connection information between grids included in the correction indicator.

The distortion correction coefficient includes a parameter constituting the distortion correction algorithm and a distortion constant, the parameter includes a focal length and an optical center point of the lens mounted to the camera, and the distortion constant is a value of the lens mounted to the camera. It can include a radial distortion constant and a tangential distortion constant.

The distortion correction algorithm may be expressed by the following equation.

r _d = atan (2 * r _u * tan (ω / 2)) * 1 / ω

u = f _x * {r _d * cos (θ)} + c _x

v = f _y * {r _d * sin (θ)} + c _y

Here, u and v refer to the coordinates of the correction indicator image on the image plane, x 'and y' refer to the coordinates on the lens plane on which the three-dimensional spatial coordinates are projected, and f _x and f _y refer to the focal lengths. , c _x and c _y are the coordinates of the optical center point, ω is the distortion constant, r _u ² = x ' ² + y' ² , and θ = tan ^{- 1} (y '/ x'). . In particular, ω is a value of 1.0.

The distortion correction algorithm may be expressed by the following equation.

u = f _x * (x '* (1+ k ₁ * r ² + k ₂ * r ⁴ ) + 2p ₁ * x' * y '+ p ₂ (r ² + 2x' ² )} + c _x

v = f _y * {y '* (1+ k ₁ * r ² + k ₂ * r ⁴ ) + p ₁ (r ² + 2y' ² ) + 2p ₂ * x '* y'} + c _y

Where f _x and f _y are focal lengths, c _x and c _y are optical center point coordinates, k ₁ and k ₂ are radial distortion constants, and p ₁ and p ₂ are tangential distortion constants. Means r ² = x ' ² + y' ²

The camera may be equipped with a fisheye lens.

According to one or more exemplary embodiments, an apparatus for correcting a distorted image includes: an image input unit configured to receive a two-dimensional correction index image of a correction index photographed through a camera; A controller for extracting a plurality of feature data from the correction indicator image and calculating a distortion correction coefficient constituting a distortion correction algorithm using the plurality of feature data; And a correction file generation unit generating a correction file using the calculated distortion correction coefficients.

The apparatus may further include a corrected image generator configured to generate a corrected image by using the corrected file.

The angle formed between the optical axis of the camera and the correction index may be photographed while maintaining 15 degrees or more with respect to the vertical line.

As described above, according to the present invention, an image of which distortion is corrected may be generated from an image photographed through the fisheye lens.

DETAILED DESCRIPTION Hereinafter, exemplary embodiments of the present invention will be described in detail with reference to the accompanying drawings so that those skilled in the art may easily implement the present invention.

2 is a diagram showing the configuration of a distortion image correction system according to an embodiment of the present invention.

As shown in FIG. 2, the distorted image correcting system may include a camera 100, a distorted image correcting apparatus 200, and a display device 300.

The camera 100 may include a fisheye lens, and acquires a three-dimensional image having a view angle of 180 ° or more as a two-dimensional image through the fisheye lens. At this time, the image obtained through the camera 100 has a severe distortion, as shown in Figure 1c.

That is, the subject located at the optical center point of the fisheye lens does not have distortion, and the subject located at the periphery has very severe distortion.

The distortion image correcting apparatus 200 includes an image input unit 210, a memory unit 220, a control unit 230, a correction file storage unit 240, and a correction image generating unit 250.

The image input unit 210 receives a two-dimensional correction indicator image for a predetermined correction indicator photographed through a camera having a fisheye lens.

The memory unit 220 stores the plurality of correction indicator images and the distortion correction algorithm program that have been received.

The controller 230 extracts a plurality of feature data from the plurality of correction index images and calculates a distortion correction coefficient constituting the distortion correction algorithm using the plurality of feature data. Here, the distortion correction coefficient includes an internal parameter, a distortion constant, and the like.

Then, the controller 230 generates a correction file consisting of the calculated internal parameter and the distortion constant.

The correction file storage unit 240 is configured as a nonvolatile memory area, and receives and stores a correction file from the control unit 230.

The corrected image generator 250 generates a distortion corrected image based on the distortion correction algorithm stored in the memory 220 and outputs the image to the external display device 300.

The display device 300 is a device capable of displaying a distortion corrected image, and may be implemented as a PC or an LCD monitor, or may be included in the distortion image correcting apparatus 200.

Hereinafter, before the distortion image correcting apparatus 200 corrects the distorted image, a method of photographing the correction index through the camera 100 will be described.

3A to 3D are diagrams showing photographing correction indicators using a camera according to an embodiment of the present invention.

3A shows a predetermined correction index, which is arranged such that black and white squares do not overlap alternately in a lattice form.

As shown in FIG. 3B, the correction indicator image may be generated by photographing the correction indicator in various directions using the camera 100.

At this time, as shown in Figure 3c, the angle formed between the optical axis of the camera and the correction index is preferably at least 15 ° relative to the vertical line (illustrated as 45 ° in Figure 3c).

In addition, as shown in FIG. 3D, the camera 100 may be photographed while being rotated by 90 °, and a sample of various correction index images may be generated for one correction index.

On the other hand, as shown in Fig. 3a, the number of square lattice is configured in the form of odd X even and even X odd. For example, the configuration is 5X6, 7X8, 10X7 (shown in the form of 6X5 in Figure 3a).

As described above, by configuring the horizontal and vertical numbers of the square in odd-numbered and even-numbered odd-numbered forms, it is possible to prevent an error in the vertex connection information of the correction index image photographed at various angles as shown in FIG. 3D.

Therefore, it is possible to easily distinguish the correction indicator image corresponding to each square.

Hereinafter, a method of correcting the distorted image by the distorted image correcting apparatus 200 with respect to the generated correction index image will be described.

4 is a flowchart illustrating a method of correcting a distorted image of the apparatus for correcting a distorted image, according to an exemplary embodiment.

First, a plurality of correction indicator images of the correction indicators photographed by the camera 100 are stored in the memory unit 220 in the same manner as in FIG. 3 (S410).

The controller 230 extracts feature data from the plurality of correction indicator images stored in the memory 220 as shown in FIG. 5 (420).

Here, the feature data includes vertex coordinate information and vertex connection information between a plurality of grids constituting the correction index image.

5 is an exemplary view showing feature data of a correction index image according to an embodiment of the present invention.

Here, the vertex coordinate information indicates coordinates of vertices formed by four adjacent squares as shown in FIG. 5, and the vertex connection information indicates a connection relationship between vertices. Vertex connection information may be generated to prevent generation of vertices that are not recognized in the photographed correction index.

The controller 230 stores a plurality of feature data of the correction indicator in the memory 220 (S430).

When the plurality of feature data extraction is completed from three or more correction index images, the controller 230 automatically calculates various distortion correction coefficients, ie, a camera matrix and a distortion constant, that constitute a distortion correction algorithm using the plurality of feature data. (S440).

The distortion correction algorithm is composed of internal or external parameters and distortion constants in the form of a camera matrix, and a detailed description of how to calculate the parameters and distortion constants of the fisheye lens will be given later.

The controller 230 generates a correction file in the form of an XML or text file from the corrected distortion correction coefficients and stores the correction file in the correction file storage unit 240 which is a nonvolatile memory area (S450).

The information stored in the correction file is information about the distortion correction algorithm. The distortion correction algorithm includes a camera matrix including focal length and distortion center coordinates, a radial distortion constant, a tangential distortion constant, and the like. Pixel error values indicating the number of images and the accuracy of distortion correction are also included.

The controller 230 stores the corrected distortion correction algorithm in the memory unit 220.

If the camera image distortion correction does not satisfy the expected level based on the generated correction file, the correction index is changed to another position to additionally photograph and add the feature data to generate a correction file (S460).

The corrected image generating unit 250 generates an image in which the geometric distortion image of the camera is corrected using the corrected algorithm (S470).

The external display device 300 displays the corrected image uploaded from the corrected image generator 250 (S480).

Here, in the exemplary embodiment of the present invention, the corrected image generator 250 directly generates a corrected image in which the distorted image is corrected from the corrected file. However, the corrected image generator 250 is applied to the external display device 300. If included, only the correction file may be transferred to the external display device 300.

In addition, in the exemplary embodiment of the present invention, the display apparatus 300 is described as being separated from the distortion image correcting apparatus 200. However, when the display apparatus is included in the distortion image correcting apparatus 200, the corrected image generated is directly displayed on the screen. Can be marked on.

Hereinafter, a method of calculating the distortion correction coefficient of the distortion image algorithm by the controller 230 of the distortion image correcting apparatus 200 will be described with reference to FIGS. 6 to 7.

6 is a diagram illustrating a process of converting a world coordinate system into a camera coordinate system.

As shown in FIG. 6, an external parameter matrix is used to convert a world coordinate system, which is virtual coordinates in a three-dimensional space, to a camera coordinate system based on the camera focus C.

That is, a matrix including a rotation matrix R and a motion vector T is required to convert a world coordinate system, which is a virtual coordinate, into a camera coordinate system based on the camera focus C.

Therefore, in order to convert the (X, Y, Z) coordinates in the world coordinate system into the (x, y, z) coordinates of the camera coordinate system, the following equation (1) is applied.

In Equation 1, R and t are expressed in matrix form as in Equation 2 below.

R and t shown in Equation 2, that is, [R | t], are referred to as external parameters, and R and t in matrix form are referred to as rotation-translation matrices.

The external parameter is for converting the coordinates of the world coordinate system into the coordinates of the camera coordinate system, and is a parameter related to the relative movement and rotation of the camera coordinate system with respect to the world coordinate system.

Therefore, in a fixed camera having an arbitrary coordinate system by [R | t], one point P (X, Y, Z) in three-dimensional space is converted into one point p (x, y, z) in three-dimensional space.

7 is a view showing that one vertex of the correction index according to an embodiment of the present invention passes through the lens and forms an image plane.

As shown in FIG. 7, a lens having a focal length f of one point (c _x , c _y ) on the image plane where the optical axis of the camera meets, that is, one point P (x, y, z) on the three-dimensional plane with respect to the center of distortion. When the projection on the two-dimensional image plane (image plane) through, the point p is formed on the two-dimensional image plane can be obtained as shown in Equation 3 below.

Here, one point P (x, y, z) on the three-dimensional image represents one of the vertices of the correction index shown in FIG. 3A.

Here, the first matrix at the right end of the equation is called a camera matrix, and f _x , f _y , c _x , and c _y included in the camera matrix are called internal parameters.

An internal parameter includes a focal length, a distortion factor, and a distortion center associated with a camera matrix, and is a parameter for distortion generated in the camera itself. By calculating such internal parameters, it is possible to remove all distortion elements.

In Equation 3, f _x and f _y represent focal lengths, and c _x and c _y correspond to optical center points and represent distortion center coordinates of the lens. In addition, s represents a scale, which can be expressed as a function related to the parameters f _x , f _y , c _x , and c _y , and according to sampling (up / down), the parameters (f _x , f _y , c) _x , c _y ) multiply or divide by a percentage.

On the other hand, the coordinates (x ', y') (not shown in Fig. 7) on the lens plane on which the three-dimensional spatial coordinates (x, y, z) are projected by the camera coordinate system and the three-dimensional spatial coordinates P (x, y, z) ) Is proportional to the following equation (4). Assume that z is not zero.

x '= x / z

y '= y / z

Here, by calculating Equation 3 and substituting the result obtained in Equation 4, the following Equation 5 can be derived.

u = f _x * x '+ c _x

v = f _y * y '+ c _y

Therefore, p, which is a coordinate on a two-dimensional image plane, can be obtained using the internal parameters f _x , f _y , c _x , and c _y .

However, since the actual lens is not a perfect sphere shape, distortion occurs.

That is, since the coordinate p 'on the image plane is an ideal point where no distortion is generated, the coordinate p' may not coincide with the coordinate p on the image plane where the actual distortion occurs.

Hereinafter, the relationship between p 'and p will be described with reference to Equations 6 to 8.

In FIG. 7, c _x and c _y are points on the Z axis, and represent the distortion center coordinates of the lens, and no distortion occurs.

Here, the lens distortion model has a field of view ω corresponding to the fisheye lens as one distortion constant, and the coordinates x 'and y' on the image plane are x 'by the lens distortion model as shown in Equation 6 below. It can be expressed by changing to ', y' (not shown in FIG. 7).

That is, it is possible to map the coordinates x ', y' on the image plane without distortion by mapping the inverse spatial mapping to the coordinates x '', y '' on the image plane where the distortion has occurred.

r _d = 1 / ω * atan (2 * r _u * tan (ω / 2))

x '' = r _d * cos (θ)

y '' = r _d * sin (θ)

Here, r _u ² = x ' ² + y' ² , θ = tan ^-1 (y '/ x'), and ω represents a distortion constant. The distortion constant ω may include a field of view corresponding to the fisheye lens.

Similarly, if x '' and y '' are substituted for x 'and y' in Equation 5 by applying a lens distortion model, Equation 7 is derived.

u = f _x * x '' + c _x

v = f _y * y '' + c _y

Substituting Equation 6 into Equation 7, an extended u and v value can be derived as shown in Equation 8 below.

u = f _x * {r _d * cos (θ)} + c _x

v = f _y * {r _d * sin (θ)} + c _y

As shown in Equation 8, u and v, which are coordinate values corresponding to the coordinate p 'on the image plane where the distortion is generated, are represented as internal parameters (f _x , f _y , c _x , c _y ) and a distortion constant (ω), respectively. Can be represented.

Therefore, the vertex information (u, v) extracted from a plurality of correction indicator images is substituted into Equation 8, and the internal parameters of the fisheye lens most optimized through a numerical analysis method such as Newton-method. (f _x , f _y , c _x , c _y ) and distortion constant (ω) can be obtained.

In particular, since the distortion constant is maintained the same value regardless of the camera position, it can be set to a fixed value only after the first calculation.

In this way, the controller 230 calculates the internal parameters f _x , f _y , c _x , c _y and distortion constant ω, which are distortion correction coefficients. An ideal fisheye lens follows the characteristic ω = 1.

On the other hand, the lens distortion model can be expressed as an infinite series, the coordinates x ', y' on the lens plane is x '', y '' (not shown in Figure 7) by the lens distortion model according to the following equation (9) Can be expressed as

That is, by the camera lens distortion model considering radial distortion and tangential distortion, the coordinates x '' and y '' on the lens plane may be expressed as a fifth-order function of r as shown in Equation (9).

x '' = x '* (1+ k ₁ * r ² + k ₂ * r ⁴ ) + 2p ₁ * x' * y '+ p ₂ (r ² + 2x' ² )

y '' = y '* (1+ k ₁ * r ² + k ₂ * r ⁴ ) + p ₁ (r ² + 2y' ² ) + 2p ₂ * x '* y'

Where r ² = x ' ² + y' ² , k ₁ and k ₂ represent radial distortion constants, and p ₁ and p ₂ represent tangential distortion constants.

Similarly, by substituting Equation 9 into Equation 7, an extended u and v value can be derived as shown in Equation 10 below.

u = f _x * (x '* (1+ k ₁ * r ² + k ₂ * r ⁴ ) + 2p ₁ * x' * y '+ p ₂ (r ² + 2x' ² )} + c _x

v = f _y * {y '* (1+ k ₁ * r ² + k ₂ * r ⁴ ) + p ₁ (r ² + 2y' ² ) + 2p ₂ * x '* y'} + c _y

As shown in Equation 8, u and v, which are coordinate values corresponding to the coordinates p 'on the image plane where the distortion is not generated, are internal parameters (f _x , f _y , c _x , c _y ) and distortion constants (k _{1), respectively.} , k ₂ , p ₁ , p ₂ ).

The controller 230 generates a correction file from the calculated distortion correction coefficient and stores the correction file in the correction file storage 240.

As described above, according to an exemplary embodiment of the present invention, the distortion correction coefficient may be automatically calculated using a plurality of images photographed through the fisheye lens, and used to correct the distortion image.

Although the preferred embodiments of the present invention have been described in detail above, the scope of the present invention is not limited thereto, and various modifications and improvements of those skilled in the art using the basic concepts of the present invention defined in the following claims are also provided. It belongs to the scope of rights.

1A to 1C are exemplary views illustrating an image without lens distortion and an image in which lens distortion occurs.

2 is a diagram showing the configuration of a distortion image correction system according to an embodiment of the present invention.

3A to 3D are diagrams showing photographing correction indicators using a camera according to an embodiment of the present invention.

4 is a flowchart illustrating a method of correcting a distorted image of the apparatus for correcting a distorted image, according to an exemplary embodiment.

5 is an exemplary view showing feature data of a correction index image according to an embodiment of the present invention.

6 is a diagram illustrating a process of converting a world coordinate system into a camera coordinate system.

7 is a view showing that one vertex of the correction index according to an embodiment of the present invention passes through the lens and forms an image plane.

Claims (16)

Receiving a two-dimensional correction indicator image of the correction indicator photographed through the camera; Extracting a plurality of feature data from the correction indicator image; Calculating a distortion correction coefficient constituting a distortion correction algorithm using the plurality of feature data; And And generating a correction file using the calculated coefficients. The method of claim 1, And generating a corrected image by using the correction algorithm. The method of claim 1, The correction indicator image, The correction image correction method, characterized in that the correction index having a plurality of square grid pattern is generated by shooting with the camera. The method of claim 3, The correction index is The plurality of square lattice pattern is a distortion image correcting method, characterized in that arranged in an odd X even or even X odd shape. The method of claim 4, wherein The feature data, And the vertex coordinate information and the vertex connection information between the grids included in the correction index. The method of claim 5, The distortion correction coefficient includes a parameter constituting the distortion correction algorithm and a distortion constant, The parameter includes a focal length and an optical center point of the lens mounted to the camera, and the distortion constant includes a radial distortion constant and a tangential distortion constant of the lens mounted to the camera. The method of claim 6, The distortion correction algorithm is represented by the following equation: r _d = atan (2 * r _u * tan (ω / 2)) * 1 / ω u = f _x * {r _d * cos (θ)} + c _x v = f _y * {r _d * sin (θ)} + c _y Here, u and v refer to the coordinates of the correction indicator image on the image plane, x 'and y' refer to the coordinates on the lens plane on which the three-dimensional spatial coordinates are projected, and f _x and f _y refer to the focal lengths. , c _x and c _y are the coordinates of the optical center point, ω is the distortion constant, r _u ² = x ' ² + y' ² , and θ = tan ^{- 1} (y '/ x'). . In particular, ω is a value of 1.0. The method of claim 6, The distortion correction algorithm is represented by the following equation: u = f _x * (x '* (1+ k ₁ * r ² + k ₂ * r ⁴ ) + 2p ₁ * x' * y '+ p ₂ (r ² + 2x' ² )} + c _x v = f _y * {y '* (1+ k ₁ * r ² + k ₂ * r ⁴ ) + p ₁ (r ² + 2y' ² ) + 2p ₂ * x '* y'} + c _y Here, u and v refer to the coordinates of the correction indicator image on the image plane, x 'and y' refer to the coordinates on the lens plane on which the three-dimensional spatial coordinates are projected, and f _x and f _y refer to the focal lengths. C _x and c _y are the coordinates of the optical center point, k ₁ and k ₂ are the radial distortion constants, p ₁ and p ₂ are the tangential distortion constants, and r ² = x ' ² + y' ² The method of claim 1, And the camera is equipped with a fisheye lens. An image input unit configured to receive a two-dimensional correction indicator image of the correction indicator photographed by the camera; A controller for extracting a plurality of feature data from the correction indicator image and calculating a distortion correction coefficient constituting a distortion correction algorithm using the plurality of feature data; And And a correction file generator which generates a correction file using the calculated distortion correction coefficients. The method of claim 10, And a corrected image generating unit generating a corrected image by using the corrected file. The method of claim 10, The correction indicator image, The correction image correction apparatus, characterized in that the correction indicator having a plurality of square grid pattern is generated by photographing through the camera. The method of claim 12, The angle formed between the optical axis of the camera and the correction index is photographed while maintaining at least 15 degrees relative to the vertical line. The method of claim 12, The correction index is The plurality of square lattice pattern is a distortion image correcting device, characterized in that arranged in an odd-X even or even-X odd form. The method of claim 14, The feature data, And a vertex coordinate information and vertex connection information between the grids included in the correction index. The method of claim 11, The distortion correction coefficient includes a parameter constituting the distortion correction algorithm and a distortion constant, The parameter includes a focal length and an optical center point of the lens mounted to the camera, And the distortion constant includes a radial distortion constant and a tangential distortion constant of a lens mounted to the camera.

Images