JP2006146760A - Image transformation method and image transformation system - Google Patents

Abstract

<P>PROBLEM TO BE SOLVED: To provide an image transformation method and system for transforming a camera image into a predetermined image very precisely corresponding to the three dimensional space within a camera view without requiring the internal and external parameters of a camera. <P>SOLUTION: The image transformation method and system calculates a homography between a first plane S1 and second plane S2 based on correspondence between four points commonly contained in the first plane S1 included in the three dimensional space within the view of the camera CM and the second plane S2 corresponding to an image inputted through the camera CM. The homography is used for transforming the camera image into a predetermined image corresponding to the three dimensional space within the camera view. <P>COPYRIGHT: (C)2006,JPO&NCIPI

Description

  The present invention relates to an image conversion method and an image conversion apparatus, and more particularly to an image conversion method and an image conversion apparatus for converting a camera image input via a camera that projects a three-dimensional space onto a two-dimensional image into another image.

    Conventionally, various proposals have been made regarding a method of converting a camera image input via a camera that projects a three-dimensional space onto a two-dimensional image into another image. A method for calculating a conversion relationship between a two-dimensional image plane and a three-dimensional space using each of an internal parameter and an external parameter is disclosed. Similarly, Non-Patent Document 2 discloses a method for calibrating the values of the internal parameters and external parameters of the camera. Patent Document 1 discloses a projection relationship between an image plane and a road surface with respect to a vehicle peripheral image processing apparatus.

  Non-Patent Documents 3 to 5 below disclose technical information such as image processing technology that is the basis of the present invention, which will be described later as part of the description of embodiments of the present invention.

JP 2002-87160 A Sundaram Ganapathy "DECOMPOSITION OF TRANSFORMATION MATRICES FOR ROBOT VISION", IEEE Conference Robotics, 1984 Roger Y. Tsai, “An Efficient and Accurate Camera Calibration Technique for 3D Machine Vision”, Proceedings of IEEE Conference on Computer Vision and Pattern Recognition, Miami Beach, FL, 1986, 364-374 "Computer Vision" by Satoshi Sato, Corona, October 10, 2001, first edition, 3rd edition, pp. 11-41 "Geometric Framework for Vision I: Single View and Two-View Geometry" by Andrew Zisserman, [online], April 16, 1997, [searched September 3, 2004], Robotics Research Group. University of Oxford., Internet <URL: http: //homepages.inf.ed.ac.uk/rbf/CVonline/LOCAL_COPIES/EPSRC_SSAZ/epsrc_ssaz.html> "Multiple View Geometry in Computer Vision" by Richard Hartley and Andrew Zisserman, Cambridge University Press. August 2000, pp. 11-16

  In order to use the method described in the above-mentioned Non-Patent Document 1, it is necessary to determine all the parameters of the camera. For example, in the case of targeting a camera mounted on a vehicle, due to the tolerance of each parameter. In addition, high-accuracy projection conversion is difficult. In Non-Patent Document 2, it is assumed that each of the internal parameters and external parameters of the camera is used to calculate the conversion relationship between the two-dimensional image plane and the three-dimensional space. When a camera is used, high-accuracy projection conversion is difficult due to the tolerance of each parameter.

  Also, in the above-mentioned Patent Document 1, it is assumed that each of the internal parameters and external parameters of the camera is used to calculate the conversion relationship between the two-dimensional image plane and the three-dimensional space. When a camera is used, high-accuracy projection conversion is difficult because of the tolerance of each parameter. This problem is caused by assuming that the internal parameters and external parameters of the camera are already known by some calibration method in the coordinate conversion relationship between the image plane and the three-dimensional road surface.

  Therefore, the present invention provides an image conversion method capable of converting a camera image into a predetermined image with high accuracy in accordance with a three-dimensional space in the camera field of view without requiring internal parameters and external parameters of the camera. The task is to do.

  The present invention also provides an image conversion device capable of converting a camera image into the predetermined image with high accuracy without requiring an internal parameter and an external parameter of the camera, and further mounting the camera on a vehicle. Another object is to provide an image conversion apparatus that can easily display a converted image.

  In order to achieve the above object, the present invention provides an image conversion method for converting a camera image input via a camera that projects a three-dimensional space onto a two-dimensional image into another image as described in claim 1. Based on the correspondence of at least four points that are commonly included in the first plane included in the three-dimensional space in the field of view of the camera and the second plane corresponding to the image input via the camera. , Calculating a homography between the first plane and the second plane, and converting the camera image into a predetermined image according to a three-dimensional space in the field of view of the camera by the homography That's what it meant.

  According to a second aspect of the present invention, an image conversion apparatus includes a camera that projects a three-dimensional space onto a two-dimensional image, and an image that converts a camera image input via the camera into another image. In the conversion device, correspondence between at least four points commonly included in the first plane included in the three-dimensional space in the field of view of the camera and the second plane corresponding to the image input via the camera. Based on the relationship, the camera image is obtained by the homography computing means for computing the homography between the first plane and the second plane, and the homography computed by the homography computing means. A conversion means for converting to a predetermined image according to the three-dimensional space in the field of view is provided.

  Furthermore, as described in claim 3, the image conversion apparatus of the present invention includes the camera, the homography calculation unit, and the conversion unit mounted on a vehicle, and a display unit that displays the predetermined image. Can be. In this case, the display means can be configured to change and display the geometric shape of the camera image.

  In addition, as described in claim 5, the conversion unit converts the camera image into an image viewed from a viewpoint different from the viewpoint of the camera with respect to a three-dimensional space in the field of view of the camera. Can be configured to be displayed by the display means.

  Since this invention is comprised as mentioned above, there exist the following effects. That is, in the image conversion method according to claim 1, the first plane included in the three-dimensional space in the field of view of the camera and the second plane corresponding to the image input through the camera are common. A homography between the first plane and the second plane is calculated based on the correspondence relationship of at least four points included, and the camera image is obtained according to the three-dimensional space in the camera field of view by the homography. Therefore, even if both the internal and external parameters of the camera are unknown, the four points in the camera image can be converted to four points on the plane in the original three-dimensional space with high accuracy. Can be restored.

  Further, according to the image conversion apparatus of the present invention, the camera image is converted into a predetermined image according to the three-dimensional space in the field of view of the camera by the homography calculated by the homography calculating means. Even if the internal parameters and external parameters of the camera are unknown, the four points in the camera image can be appropriately restored as the four points on the plane in the original three-dimensional space. In this case, the internal parameters of the camera such as the focal length and the origin position of the camera may be unknown, and the external parameters of the camera that are information on the position and orientation of the camera in the three-dimensional space may also be unknown. Installation and installation conditions are eased.

  Further, it can be configured as described in claim 3 and can be mounted on a vehicle. In this case, even if both the internal parameter and the external parameter of the camera are unknown, A point can be appropriately restored as four points on a plane in the original three-dimensional space.

  Furthermore, if configured as described in claim 4 and claim 5, it is effectively applied to, for example, an image display system for grasping the situation around the vehicle, a system for recognizing a runway, a system for detecting an obstacle, etc. can do.

A specific aspect of the image conversion method of the present invention having the above configuration will be described below with reference to the drawings. First, camera technology on which the present invention is premised will be described with reference to FIGS. 8 to 11, and then embodiments of the present invention will be described with reference to FIGS. First, the camera coordinate system is defined as shown in FIG. That is, as described in Non-Patent Document 3, the optical center C of the camera is the coordinate origin, and the X-axis and Y-axis of the three-dimensional coordinate are taken in parallel with the x-axis and y-axis on the image plane, respectively. The axis is parallel to the optical axis l ₀ . A straight line l ₀ drawn perpendicularly to the image plane π from the optical center C is referred to as an “optical axis”, and an intersection x _C between the optical axis and the image plane is referred to as an “image center”.

In FIG. 8, a point X = [X, Y, Z] ^T having coordinates (X, Y, Z) in a three-dimensional space is a point x = [[x, y) on the two-dimensional image. x, y] ^T. In addition, capital X etc. represent three dimensions, and small letter x etc. represent two dimensions. “ ^T ” means a transposed matrix. If the focal length is f = 1, the position on the two-dimensional image in FIG. 8 is (x, y, 1) when considered as a point in the three-dimensional space. As is apparent from FIG. 8, in the projection by this camera, the ratio of x and f of the point on the two-dimensional image is equal to the ratio of X and Z of the point in the three-dimensional space, and the ratio of y and f is Y. Is equal to the ratio of Z. Therefore, the relationship of the following [Equation 1] is established between the point in the three-dimensional space and the projected image, and such a projection is called “perspective projection”.

  The perspective projection of the above [Equation 1] can be expressed in the form of a matrix operation as shown in the following [Equation 2] using homogeneous coordinates.

  In [Expression 2], [lambda] is an arbitrary real number, and [Expression 2] represents the same projection as [Expression 1]. [Expression 2] is simplified and expressed as the following [Expression 3].

  In the above [Equation 3], Pf is as shown in [Equation 4] below.

  By the way, the image data handled on the computer is where the origin is taken, and the subdivision method and aspect ratio as pixel coordinate values vary depending on the design. That is, the image data handled on the computer is based on image coordinates in pixel units that are unrelated to the physical size and position. Therefore, when modeling an actual camera, it is necessary to convert from physical coordinates x to image coordinates m. This conversion is a combination of parallel movement for origin position alignment, vertical and horizontal scale conversion, and scale conversion according to the focal length, and is expressed as follows.

Here, the preceding stage of the above [Equation 5] is represented by [m ₁ , m ₂ , m ₃ ] ^T and is a homogeneous coordinate of the image coordinates m = [u, v] ^T , and the following [Equation 6] Set so that the equivalence relation of the expression holds. That is, u = m ₁ / m ₃ and v = m ₂ / m ₃ .

A is a matrix for converting physical coordinates x into image coordinates m, and includes focal length f, image center coordinates (u ₀ , v ₀ ), u and v direction scale factors k _u , k _v , and The following [Equation 7] is expressed by the shear coefficient k _s . This “shear coefficient” is a coefficient that causes deformation (shear deformation) that maintains parallelism but does not maintain perpendicularity.

  By the above [Equation 3] and [Equation 5], the point X in the three-dimensional space is projected to the pixel m on the two-dimensional image as shown in the following [Equation 8]. Here, A is referred to as a “camera calibration matrix”, and more specifically, as is apparent from the above [Equation 7], it is a matrix constituted by camera internal parameters. Call.

Next, when considering the relationship between multiple cameras, moving to multiple positions even for the same camera, or when considering the relationship between the camera and the object, reference coordinates Set the system. In this case, as shown in FIG. 9, the coordinates determined in common for the number and positions of the cameras to be considered and the object are defined as “world coordinates”. From the world coordinates ( _Xw ) to the camera coordinates X is converted by the three-dimensional rotation R and translation T as shown in the following [Equation 9].

The three-dimensional rotation R and translation T are respectively the rotations θ _x , θ _y , θ _z around the X axis, the Y axis, and the Z axis, and the translations T _X , X axis, Y axis, and Z axis. It consists of T _Y and T _Z and is expressed as the following [Equation 10] and [Equation 11], respectively.

  [Equation 9] can be expressed as the following [Equation 12] using homogeneous coordinates.

  Here, M is a 4 × 4 matrix in which the rotation R and the translation T are combined, as shown in [Equation 13] below. That is, M is a matrix determined by the attitude (rotation) and position (translation) with respect to the world coordinates. These rotations and translations are referred to as “camera external parameters”, and A is referred to as “camera internal parameter matrix”, whereas M is referred to as “camera external parameter matrix”.

  From the above [Equation 8] and [Equation 12], the point X in the three-dimensional space in the world coordinate system is projected to the point m on the two-dimensional image as follows.

In the above [Expression 14], P is a 3 × 4 matrix expressed by the following [Expression 15]. In the [Equation 14] where it world coordinates X _w is repositioned in X, hereinafter without distinguishing symbol relative to the world coordinate and the camera coordinate, and that both represented by X.

  Thus, the camera model represented by the above [Expression 14] is referred to as a “perspective camera model”. P is referred to as a “perspective camera matrix” and is composed of camera internal parameters and external parameters. Therefore, if P is obtained, it can be decomposed into A, R, and T. When all of A, R, and T are known, that is, when P is known, the camera is said to be “calibrated”, and when these are unknown, the camera is said to be “uncalibrated”. Therefore, obtaining A, R, and T is called “camera calibration”.

  In the above-mentioned “perspective camera model”, the camera matrix P is composed of A, R, and T. When the camera matrix P is generalized by a 3 × 4 matrix, the following [Equation 16] is obtained.

The above P _p is a 3 × 4 matrix as shown in [Equation 17] below. The camera model represented by the generalized 3 × 4 matrix is called “projection camera model”, and P _p is called “projection camera matrix”.

As shown in [Equation 17] above, P _p is a 3 × 4 matrix and the number of elements is twelve. However, the above [Expression 16] is expressed by homogeneous coordinates, and even if P _p is multiplied by a constant, the same camera model is represented, so the degree of freedom is 11 instead of 12. Thus, the “perspective camera” can be generalized by such a “projection camera”.

  Next, as shown in FIG. 10, when the world coordinates are limited to the plane of Z = 0, the above 3 × 4 matrix can be simplified to a 3 × 3 matrix as shown in the following [Equation 18]. However, this 3 × 3 matrix represents a general “plane projection transformation” (homography).

  Here, when two planes π and plane Π as shown in FIG. 11 are considered, the relationship between the points x and x ′ on the respective planes can be expressed as the following [Equation 19]. .

  Alternatively, a 3 × 3 regular matrix (however, the determinant is not 0) can also be expressed as [Equation 20] below, and this H is called homography.

  The above [Equation 20] can also be expressed as the following [Equation 21].

  Further, in the element of the homography H, the following two linear equations [Equation 22] and [Equation 23] are obtained for the corresponding points in each plane. This indicates that any four points on a plane can be projectively converted to four points on another plane.

  When the above relationship is arranged, in the perspective camera model, a plurality of conversions are repeated as shown in FIG. 12, but in the projection camera model, the conversion is performed by a simple calculation as shown in FIG. Note that “D” in FIGS. 12 and 13 is a distortion correction coefficient for performing correction processing on image distortion.

  The present invention performs image conversion using homography as described above, and an embodiment thereof is shown in FIG. The camera CM in FIG. 1 may take any form as long as it is a camera that projects a three-dimensional space onto a two-dimensional image. In this embodiment, at least the first plane S1 included in the three-dimensional space in the field of view of the camera CM and at least the second plane S2 corresponding to the image input via the camera CM are included. Based on the correspondence of the four points, the camera image is obtained by the homography computing means HM for computing the homography between the first plane S1 and the second plane S2, and the homography computed by the homography computing means HM. Is converted to a predetermined image according to a three-dimensional space in the field of view of the camera.

  The camera or the like of the present invention may be mounted on a moving body, and as shown in FIG. 2, the apparatus of FIG. 1 is mounted on a vehicle VH and includes display means DS for displaying the predetermined image. be able to. This display means DS can be configured to change and display the geometric shape of the camera image. The conversion means CV converts the camera image into an image viewed from a viewpoint different from the viewpoint of the camera with respect to the three-dimensional space in the camera field of view, and the display means DS displays the image. You can also.

  Thus, the above image conversion device can be applied to, for example, an image display system for grasping the situation around the vehicle, a system for recognizing a runway, a system for detecting an obstacle, etc. A specific configuration example applied to the support device is shown in FIG. In the present embodiment, a camera CM (for example, a CCD camera) is mounted behind a vehicle (not shown), and the field of view behind the vehicle including the road surface is continuously imaged. The video signal of the camera CM is A / D converted through the video input buffer circuit VB and the sync separation circuit SY and stored in the frame memory FM. The image data stored in the frame memory FM is processed by the image processing unit VC. The image processing unit VC includes an image data control unit VP, a distortion correction unit CP, a feature point detection unit SP, a coordinate conversion unit KP, and a recognition processing unit LP.

  In the image processing unit VC, data addressed by the image data control unit VP is called from the image data in the frame memory FM and sent to the distortion correction unit CP, where distortion correction processing is performed. For example, as shown in FIG. 4, when the image from the camera CM is a distorted image, the distortion correction unit CP performs a correction process to obtain the image shown in the upper side of FIG. The upper and lower stipple parts in FIG. 5 are not camera images). On the distortion corrected image, the feature point detector CP detects four points at the four corners of the parking section.

  In the coordinate conversion unit KP in FIG. 3, the camera image is converted into an image viewed from the sky by the homography parameters calibrated in advance. That is, as shown in the upper side of FIG. 5, four points at the four corners of the parking area are detected from the image obtained by correcting the distortion of the image of FIG. 4, and these four points match the shape of the original parking area. When the above-described conversion by the homography H is performed, as shown in the lower side of FIG. 5, the image is converted into an image viewed from above and restored as an original rectangular parking section. Thus, the mode shown in FIG. 5 is configured to convert and display an image viewed from a viewpoint different from the viewpoint of the camera CM with respect to the three-dimensional space in the field of view of the camera CM. Note that the parking section is recognized by the recognition processing unit LP and an image signal is output via the drawing unit DP and the display controller DC, but these processes are not directly related to the present invention. Is omitted.

  On the other hand, when the above image conversion apparatus is applied to a road recognition system and used for white line recognition of a road, an edge point is detected from a camera image and applied to road surface coordinates to detect it as a road boundary line. it can. In this case, when projecting onto the road surface coordinates, the above-described image conversion by the homography H can be applied.

  Next, another experimental example in which image conversion is performed by the above-described homography H will be described. FIG. 6 and FIG. 7 illustrate another experimental example. After shooting a checkered pattern made of a set of squares of a known size and performing distortion correction on the camera image of FIG. Projective transformation is performed. Also in FIG. 7, the four points that were originally rectangular are restored as rectangles as shown in the lower side of FIG.

  As described above, the internal parameters are unknown (assuming that the distortion of the image is corrected), and any four points whose arrangement on the plane in the three-dimensional space is known, and four points in the corresponding image, respectively. Is performed by the above-described homography H, so that the four points in the image can be restored as the four points on the plane in the original three-dimensional space even if the external parameters of the camera are unknown. . In this case, camera external parameters R and T, which are information about the position and orientation of the camera in the three-dimensional space, such as the camera height and tilt angle, are not required, and these are unknown. Therefore, the installation and mounting conditions of the camera are eased, and the conversion accuracy between the two-dimensional image and the plane in the three-dimensional space is improved.

  In particular, since the four-point conversion relationship between two images is determined by homography H, the internal parameters of the camera such as the focal length and origin position of the camera may be unknown, and the height at which the camera is mounted Since external parameters of the camera such as the angle may be unknown, high-accuracy conversion can be realized by a simple method, and the calculation can be simplified by using the projection camera model. In addition, this invention is not limited to the apparatus mounted in moving bodies, such as the above vehicles, It can apply to the various apparatuses which use a camera image.

1 is a block diagram illustrating a main configuration of an image conversion apparatus according to an embodiment of the present invention. It is a block diagram which shows the main structures of the image converter which concerns on other embodiment of this invention. It is a block diagram which shows one Embodiment which applied the image conversion apparatus of this invention to the parking assistance apparatus of the vehicle. In one Embodiment which applied the image conversion apparatus of this invention to the parking assistance apparatus of the vehicle, it is a front view of the image which shows the condition where the camera image is distorted. In one Embodiment which applied the image conversion device of the present invention to a parking assistance device of vehicles, it is a front view of an image showing a situation where a camera image is converted into an image seen from the sky and a parking section is detected. It is a front view of an image which shows the situation where the picture which checked the checkered pattern as one embodiment which applied the image conversion device of the present invention to a general camera picture is distorted. FIG. 7 is a front view of an image showing a situation in which projective transformation is performed after distortion correction is performed on the image of FIG. 6 as an embodiment in which the image conversion apparatus of the present invention is applied to a general camera image. It is explanatory drawing which shows the perspective projection in a general camera technique. It is explanatory drawing which shows the relationship between the camera coordinate and world coordinate in general camera technology. It is explanatory drawing which shows planar projection conversion in a general camera technique. It is explanatory drawing which shows the relationship of the point on the plane of two plane (pi) and plane Π in a general camera technique. It is explanatory drawing which shows the perspective camera model and the relationship of the conversion by it. It is explanatory drawing which shows the relationship between a projection camera model and the conversion by it.

Explanation of symbols

CM camera HM homography calculation means CV conversion means DS display means VB video input buffer circuit SY sync separation circuit FM frame memory VC image processing section

Claims (5)

  In an image conversion method for converting a camera image input via a camera that projects a three-dimensional space into a two-dimensional image into another image, a first plane included in the three-dimensional space in the field of view of the camera, Calculating a homography between the first plane and the second plane based on a correspondence relationship of at least four points commonly included in the second plane corresponding to the image input via the camera; The image conversion method characterized by converting the camera image into a predetermined image according to a three-dimensional space in the field of view of the camera by the homography.   An image conversion apparatus that includes a camera that projects a three-dimensional space onto a two-dimensional image and converts a camera image input via the camera into another image. Between the first plane and the second plane based on a correspondence relationship of at least four points that are commonly included in the second plane corresponding to the image input via the camera and the second plane corresponding to the image input via the camera. Homography calculation means for calculating homography, and conversion means for converting the camera image into a predetermined image according to a three-dimensional space in the field of view of the camera by the homography calculated by the homography calculation means. An image conversion apparatus characterized by that.   3. The image conversion apparatus according to claim 2, wherein the camera, the homography calculation means, and the conversion means are mounted on a vehicle and further include display means for displaying the predetermined image.   The image conversion apparatus according to claim 3, wherein the display unit is configured to display the camera image by changing a geometric shape thereof. The converting means converts the camera image into an image viewed from a viewpoint different from the viewpoint of the camera with respect to a three-dimensional space in the field of view of the camera, and the display means displays the image. The image conversion apparatus according to claim 3.

Images