KR101351911B1 - Apparatus and method for processing image of camera - Google Patents

Abstract

The present invention relates to an image processing device of a camera. The present invention includes a calculation unit calculating a projection conversion matrix between a real space and a camera image plane and calculating homography between the camera image plane and a virtual space image plane on which a predetermined pattern is formed by using the projection conversion matrix; a conversion unit projecting virtual space images on the camera image plane by using the homography; and a composing unit composing the virtual space images projected on the camera image plane with camera images. [Reference numerals] (110a,110n) Camera; (120) Estimating unit; (130) Calculation unit; (140) Conversion unit; (150) Correction unit; (160) Composing unit; (170) Display unit

Description

Apparatus and Method for Processing Image of Camera

The present invention relates to an image processing apparatus and method of a camera, and more particularly, to an image processing apparatus and method of a camera capable of capturing and displaying an image of the surrounding environment of a vehicle through a camera installed in a vehicle.

In general, a camera imaging system that allows the driver to conveniently check the surrounding environment of the vehicle by visually photographing and displaying the surrounding environment of the vehicle for the convenience of the driver is commonly used.

The camera imaging system photographs the surrounding environment through cameras installed on the front, rear, left and right sides of the car, and displays the synthesized image of the surrounding environment so that the driver can accurately recognize the surrounding situation of the car. Although parking is convenient without looking at the side mirror or the back mirror, the front, rear and left and right images of the car are simply combined and displayed, so the image processing for overlapping areas overlapped naturally. There was a problem that could not be performed.

In addition, since the real world is provided as an image through a camera, there is a problem that the driver does not accurately recognize the surrounding environment of the vehicle due to the distance between the real world and the image.

In order to solve the above problems, the conventional camera imaging system directly marks a guideline on the floor, captures an image of the floor on which the guideline is marked through the camera, and then acquires image coordinates corresponding to the guideline. A guideline was displayed on the camera image.

However, the conventional camera imaging system has a problem that can provide distorted information to the driver because the guide line and the image are distorted and displayed differently according to the mounting position or angle of the camera.

In addition, since it is necessary to directly correct the marking of the guideline according to the type of the vehicle, the mounting position or the angle of the camera, it takes a long time and a high cost.

KR 2009-0016966 A

An object of the present invention is to provide an apparatus and method for processing an image of a camera that can more easily display guide lines on a camera image using a virtual space without directly displaying the guide lines in a real space.

In addition, another problem to be solved by the present invention is to create a camera image of the virtual view using the virtual view transformation model, and to display the camera image of the various views desired by the user by displaying the guide line of the virtual view on the camera image of the virtual view. An image processing apparatus and method for outputting a camera are provided.

The image processing apparatus of the camera according to an embodiment of the present invention for solving this problem is to obtain a projection transformation matrix between the real world space and the camera image plane, and the virtual space image plane having a predetermined pattern formed using the projection transformation matrix A calculating unit for obtaining homography between the camera image planes; A transformation unit projecting the virtual space image to the camera image plane using the homography; And a synthesizer configured to synthesize the virtual space image projected on the camera image plane into the camera image.

The converting unit converts the camera image and the virtual space image into a camera image of the virtual view and a virtual space image of the virtual view by using a virtual view transformation model, and converts the virtual space image of the virtual view to the camera image of the virtual view. Can be projected onto a plane.

The apparatus may further include an estimator configured to estimate a position and a pose of the camera, and the calculator may calculate a projection transformation matrix between the real world space and the camera image plane using camera information including the position and the pose of the camera.

The estimator may receive a calibration index image photographed through the camera and estimate the position and attitude of the camera using the calibration index image.

The virtual space image plane may include at least one of a bottom plane, a front plane, a left plane, a right plane, and a rear plane.

The operation unit may obtain a homography corresponding to at least one of a bottom plane, a front plane, a left plane, a right plane, and a rear plane of the virtual space image plane using the projection transformation matrix.

The apparatus may further include a correction unit configured to correct distortion of the camera image and distortion of the virtual space image by using a distortion correction model.

The camera may be installed in a vehicle, and the predetermined pattern may be a guide line for parking the vehicle.

On the other hand, the image processing method of the camera according to an embodiment of the present invention comprises the steps of obtaining a projection transformation matrix between the real world space and the camera image plane; Obtaining a homography between the virtual space image plane having a predetermined pattern and the camera image plane using the projection transformation matrix; Projecting the virtual spatial image onto the camera image plane using the homography; Compositing the virtual space image projected on the camera image plane to the camera image.

And after the obtaining of the homography, converting the camera image and the virtual space image into a camera image of the virtual view and a virtual space image of the virtual view by using a virtual view transformation model. May project the virtual space image of the virtual viewpoint to the camera image plane of the virtual viewpoint.

Prior to obtaining the projection transformation matrix, the method may further include estimating the position and attitude of the camera, wherein the calculating of the projection transformation matrix may include: using the camera information including the position and the attitude of the camera; The projection transformation matrix between the camera image planes may be obtained.

In the estimating step, the calibration index image photographed by the camera may be received, and the position and attitude of the camera may be estimated using the calibration index image.

After the projecting step, the method may further include correcting the distortion of the camera image and the distortion of the virtual space image using a distortion correction model.

The virtual space image plane may include at least one of a bottom plane, a front plane, a left plane, a right plane, and a rear plane.

The obtaining of the homography may obtain homography corresponding to at least one of a bottom plane, a front plane, a left plane, a right plane, and a rear plane of the virtual space image plane by using the projection transformation matrix.

The camera may be installed in a vehicle, and the predetermined pattern may be a guide line for parking the vehicle.

As described above, according to the apparatus and method for processing an image of a camera according to an exemplary embodiment of the present invention, a guideline may be more easily displayed on a camera image using a virtual space without directly displaying the guideline in a real space. .

More specifically, by forming a guide line on the virtual space and projecting it on the camera image plane without directly displaying the guide line in a real space such as a floor or the like, the guide line of the virtual space can be easily synthesized and displayed on the camera image. There are advantages to it.

In addition, by generating a camera image of the virtual view using the virtual view transformation model, by displaying the guide line of the virtual view on the generated camera image of the virtual view, it is possible to output the camera image of the various views desired by the user. In addition, since the guideline can be displayed at various points of view, the driver can park the car more safely and easily.

In addition, the guideline can be generated by automatically calculating the position and posture of the camera installed in the car, and even when the camera image is converted to the virtual viewpoint, the guideline can be easily converted to the virtual viewpoint and displayed. There is an advantage to improve the convenience of.

In addition, the guideline can be displayed more easily using the virtual space without actually displaying the guideline according to the type of the car, the position or the posture of the camera. It can be expressed, and there is an advantage that can be expressed as a three-dimensional picture using the wall of the virtual space.

In addition, as in the prior art, the process of directly marking the guideline on the floor can be eliminated, thereby reducing the trouble of renting a space for marking the guideline or the time for marking the guideline, and at the same time, the cost. There is an advantage that can be reduced.

In addition, since all the operations of displaying the guideline on the camera image are performed on the computer and do not require a separate working space, there is an advantage of reducing environmental pollution.

1 is a block diagram of an image processing apparatus of a camera according to an embodiment of the present invention.
2 is an exemplary diagram of a virtual space according to an embodiment of the present invention.
3 is an exemplary diagram of a virtual space image in which a predetermined pattern is formed according to an embodiment of the present invention.
4 is a diagram illustrating a process of converting a world coordinate system into a camera coordinate system.
FIG. 5 is a diagram illustrating that vertices on a three-dimensional space pass through a lens to form a two-dimensional image plane.
6 is an exemplary view of a camera image with distortion and a camera image without distortion.
7 is an exemplary diagram of a virtual space image in which distortion is corrected according to an embodiment of the present invention.
8 illustrates a process of synthesizing a virtual space image and a camera image according to an embodiment of the present invention.
9 is an operation flowchart showing an image processing process of a camera according to an embodiment of the present invention.

The terms and words used in the present specification and claims should not be construed as limited to ordinary or dictionary terms and the inventor may appropriately define the concept of the term in order to best describe its invention It should be construed as meaning and concept consistent with the technical idea of the present invention.

Therefore, the embodiments described in the specification and the drawings shown in the drawings are only the most preferred embodiments of the present invention and do not represent all of the technical idea of the present invention, various equivalents that may be substituted for them at the time of the present application It should be understood that there may be water and variations.

Hereinafter, embodiments of the present invention will be described in detail with reference to the accompanying drawings.

1 is a block diagram of an image processing apparatus of a camera according to an embodiment of the present invention.

As shown in FIG. 1, the image processing apparatus 100 of a camera photographs a surrounding image of a vehicle through a plurality of cameras installed in a vehicle, and a predetermined pattern, such as a guide line, on the captured image of the surrounding vehicle. It is a device that controls the display so that the driver can park more safely and easily.

The image processing apparatus 100 of the camera includes a plurality of cameras 110a to 110n, an estimator 120, a calculator 130, a converter 140, a corrector 150, a synthesizer 160, and a display unit ( 170).

The plurality of cameras 110a to 110n may be installed at the front, rear, left and right sides of the vehicle, respectively, and a lens having a large angle of view, such as a wide angle lens or a fisheye lens, may be provided. The cameras 110a to 110n may photograph a subject in a real world space (three-dimensional space) as a two-dimensional image through a lens and provide the photographed image to the estimator 120 or the like.

In order to minimize blind spots of the car, the cameras 110a to 110n need a wide angle of at least 180 degrees, and the cameras must be kept at least 1000 × 1000 mm2 in the overlapping field of view of the two cameras to improve the quality of the surrounding image. The installation height of the can be determined. The higher the installation height of the camera, the better image quality can be secured. Accordingly, it is important for each camera to determine the installation position and the viewing angle to solve the blind spot of the car and minimize the deterioration of the quality of the surrounding image.

In more detail, where the cameras 110a to 110n are installed, the front cameras are installed at the center of the bonnet of the vehicle, and the cameras installed at the left and right sides are positioned at the edges or the bottom of both side mirrors of the vehicle, respectively. It can be installed to be located. In addition, the camera installed at the rear may be installed at the center of the upper portion of the rear bumper, and the camera installed at the front and rear may be installed so that the camera can photograph at least 170 degrees with reference to a vertical line in the direction of the ground.

In general, a wide-angle camera is deteriorated in image quality due to the lack of light amount around the lens, and more distortion occurs in the peripheral portion than the central portion of the lens. In addition, when the image captured through the camera is converted into a viewpoint, the image quality of the peripheral portion is seriously degraded. Therefore, the cameras installed in the front and rear to use the image formed in the central region of the camera lens may be installed so that the optical axis is parallel to the horizon, and the cameras installed on the left and right are perpendicular to the ground.

The estimator 120 may perform a function of estimating the position and attitude of the camera. The estimator 120 may estimate the position and attitude of the camera using a calibration index disposed at a predetermined position.

In more detail, after the calibration index is disposed at a predetermined position (for example, the floor), the calibration index is photographed and transmitted to the estimator 120 through the plurality of cameras 110a to 110n. ), The position and attitude of the camera can be estimated by comparing a plurality of calibration index images with a pre-stored reference image.

In this case, the calibration index may be formed in a grid pattern or an arbitrary pattern in order to minimize the estimation error, and the grid pattern may be formed of a combination of colors with strong color contrast. For example, the calibration indicator may be composed of a grid-shaped pattern made of a combination of white and black.

The calculator 130 may calculate a projection matrix between the real world space and the camera image plane.

In more detail, the calculator 130 may calculate the real-world space (three-dimensional space) by using the camera information such as the position and attitude of the camera estimated by the estimator 120, the camera internal parameters, and the camera external parameters. A projection transformation matrix defining a relationship between camera image planes, which are two-dimensional images captured by the camera, can be obtained.

Here, the internal parameter of the camera is an internal error that the camera has when it is made, for example, the distortion of the lens or the focal length, and the external parameter of the camera is a parameter of how much the camera moves and rotates from the real world space. It can be expressed as a matrix for rotation and movement.

In addition, the projection transformation matrix is a matrix that determines how the subject placed in the coordinate system of the real world space looks when viewed through a plane of a specific viewpoint. The operation unit 130 calculates an image of the subject in the real world space by calculating the projection transformation matrix. It is possible to detect the relationship between the real world space and the camera image plane on how the image can be seen as the subject image photographed by the camera.

In addition, the calculator 130 may obtain a homography between the virtual space image plane and the camera image plane on which a predetermined pattern is formed using the projection transformation matrix.

Here, the virtual space image plane defines a virtual space composed of at least one plane among virtual spaces surrounded by five planes (bottom plane, front plane, left plane, right plane, and rear plane). Predetermined patterns, such as guide lines, can be put. In this case, the size of each plane is variable, and the actual planes may be defined using an image on which a pattern such as a guide line is drawn. In addition, the pattern on the image will represent the texture of each plane, allowing the matching of the texture of each plane to the image between the plane and the final image.

2 is an exemplary view of a virtual space according to an embodiment of the present invention and FIG. 3 is an exemplary view of a virtual space image in which a predetermined pattern is formed according to an embodiment of the present invention. Referring to FIG. When constructing the drawn virtual space image, only the image corresponding to the floor plane is possible. In this case, when the pixel resolution is set to 10 mm per pixel, the actual horizontal distance 8M may correspond to 800 pixels, and the vertical distance 10M may correspond to 1000 pixels.

Then, the car is placed at the center of the image, and the size of the car corresponding to the full width and the full length of the car is drawn, and then a guideline to be expressed is drawn with colors proportional to the distance resolution.

Next, when the thickness of the guide line is narrower in the line width close to the vehicle and the line width in the farther distance is drawn wide, the same line width can be obtained in the image converted to an arbitrary camera viewpoint.

If such a virtual space image is defined on a plane, as shown in FIG. 3, a guideline pattern may be drawn on the virtual space image.

Referring again to the process of obtaining homography with reference to FIG. 1, the operation unit 130 may obtain a homography between a virtual space image plane and a camera image plane on which a predetermined pattern is formed using a projection transformation matrix. That is, since the virtual space image plane may be formed of at least one plane among the five planes, the operation unit 130 may obtain homography corresponding to the at least one plane. For example, if the virtual space image plane consists of a bottom plane, a front plane, a left plane, a right plane, and a back plane, the calculation unit 130 is a bottom plane homography, a front plane homography, a left plane homography, a right plane. Planar homography and back plane homography can be obtained respectively.

FIG. 4 is a diagram illustrating a process of converting a world coordinate system into a camera coordinate system, and a process of obtaining a projection transformation matrix and homography will be described in detail with reference to FIG. 4.

First, an external parameter matrix is used to convert a world coordinate system, which is a real world space (three-dimensional space) coordinate, to a camera coordinate system based on the camera focus C. That is, a matrix including a rotation matrix R and a motion vector T is required to convert a world coordinate system, which is a virtual coordinate in a three-dimensional space, to a camera coordinate system.

Therefore, in order to convert the (X, Y, Z) coordinates in the world coordinate system into the (x, y, z) coordinates of the camera coordinate system, the following (Equation 1) may be applied.

(Equation 1)

When R and T are expressed in matrix form in Equation 1, Equation 2 is shown below.

(Formula 2)

R and T shown in Equation 2, that is, [R | T] are called external parameters, and R and T in matrix form are called rotation transformation matrices.

In this case, the external parameter is for converting the coordinates of the world coordinate system to the coordinates of the camera coordinate system, and indicates a parameter about the relative movement and rotation of the camera coordinate system with respect to the world coordinate system, and has a fixed coordinate system having an arbitrary coordinate system by [R | T]. In the camera, a point P (X, Y, Z) in three-dimensional space may be converted into a point p (x, y, z) in three-dimensional space.

Then, as in Equation 2, the camera model represented by the 3x4 matrix with respect to Equation 1 is called a projection transformation matrix. For example, if the world coordinates are limited to a plane with Z = 0, the 3x4 matrix is defined. This can be simplified to a 3x3 matrix like (3), which can be represented homogeneously, which can also be expressed as (4).

(Equation 3)

(Equation 4)

In other words, the virtual space image plane is necessary for projecting the images on the five walls configured in the plane form to the camera image plane. Since the five walls and the camera image plane are relations between the plane and the plane, homography is established. For example, in the case of the floor plane, the x and y coordinates change but the z coordinate is fixed to 0, so that z is set to 0 and calculated. By using this property, the projection transformation matrix of the 3x4 matrix can be transformed into the homography of the 3x3 matrix.

The converter 140 may project the virtual space image on the camera image plane using homography. That is, the converter 140 may map the virtual space image to the coordinates on the camera image plane in order to project the virtual space image having the predetermined pattern onto the camera image plane.

The correction unit 150 may correct the distortion of the camera image and the distortion of the virtual space image, respectively, by using the distortion correction model.

Here, the distortion correction algorithm is an algorithm for correcting the geometric distortion generated by the camera lens. Since the actual wide-angle lens or the fisheye lens is not a perfect sphere shape and the focal length is short, the geometric distortion of the lens, for example, radioactive distortion or tangential line Directional distortion may occur. In the image photographed by the distortion of the lens, a straight line may be deformed into a curve.

Therefore, the geometric distortion image of the lens may be corrected through the distortion correction algorithm, which may be represented as a function of a correction parameter and a distortion constant. The correction parameter may include a focal length and optical center coordinates of the lens mounted to the camera, and the distortion constant may include a radial distortion constant and a tangential distortion constant.

FIG. 5 is a diagram showing that vertices in a three-dimensional space pass through a lens to form a two-dimensional image plane. Referring to FIG. 5, a process of correcting distortion is described in detail. The straight line should come out as a straight line, but distortion will occur in the lens by using a wide-angle lens to secure a wide viewing angle. In this case, the degree of distortion varies depending on an angle corresponding to a scene point in each three-dimensional image, and parameters for the degree of distortion may be distortion constants.

는 3차원 상의 한 점(scene point)을 나타내며,

는 3차원 상의 한 점(scene point)에 해당하는 각도(angle)를 의미한다. 또한,

는 3차원 상의 한 점(scene point)에 대응하는 2차원의 이미지 평면에 맺히는 이미지 점(image point)이 되며,

는 이미지 상에서 해당 점(point)이 이미지 좌표(coordinate)의 x축에 대해서 얼마만큼의 각도를 가지는지를 나타낸다. 그리고

은 해당 이미지 점(image point)이 이미지 중심으로부터 얼마만큼 떨어져 있는지를 나타내며,

은 왜곡(distortion)이 없는 경우의 3차원 상의 한 점(scene point)에 대응하는 이미지 점(image point)을 나타내며, 마지막으로,

는 카메라의 초점 거리(focal length)를 의미한다.5,

Represents a scene point in three dimensions,

Denotes an angle corresponding to a scene point in 3D. Also,

Becomes an image point in the two-dimensional image plane corresponding to a point in the three-dimensional scene,

Denotes how many angles the point has on the image with respect to the x-axis of the image coordinate. And

Indicates how far the image point is from the center of the image,

Represents an image point corresponding to a point in the three-dimensional scene in the absence of distortion, and finally,

Is the focal length of the camera.

의 위치는

와

, 그리고

에 의해서 정해지게 된다. 그 관계는 아래의 (수식 5)와 (수식 6)을 통해서 정의될 수 있다.From here,

The location of

Wow

, And

It is decided by. The relationship can be defined through Equations 5 and 6 below.

(Equation 5)

(Equation 6)

가 있어서, 왜곡의 보정 정도에 영향을 미치게 된다. An image obtained by using homography is applied to distortion by using a distortion parameter. The distortion constant has five parameters

This affects the degree of distortion correction.

FIG. 6 is a diagram illustrating a camera image having a distortion and a camera image having no distortion, and FIG. 7 is a diagram illustrating a virtual space image in which distortion is corrected according to an embodiment of the present invention. , the y-axis is the distorted image coordinate (distorted image coordinate), and the image x, y 'axis on the right in Fig. 6 represents the image distortion (undistorted image coordinate). The origin of the distorted image coordinates disposed on the left side of FIG. 6 is centered on the upper left side of the image, and the origin of the non-distorted image coordinates arranged on the right side of FIG. 6 is centered on the center of the image.

로 표현하고, 왜곡이 없는 이미지 특징 점의 좌표를

로 표현한다면, 최종적으로 왜곡이 끝났을 때, 왜곡이 있는 이미지에서 영상의 좌표인

에 해당하는 픽셀 값(pixel value)은 그에 대응하는 왜곡이 없는 이미지의 카메라 이미지 평면 좌표(normalized image coordinate)인

에 해당하는 픽셀 값(pixel value)을 가져오면, 왜곡이 있는 영상에서의 픽셀 값(pixel value)을 모두 구할 수 있게 된다. Where the coordinates of the distorted image feature points

The coordinates of an image feature point without distortion

In this case, when the distortion is finally finished, the coordinates of the image in the distorted image

The pixel value corresponding to is the normalized image coordinate of the camera image of the corresponding distortion-free image.

When the pixel value corresponding to the pixel value is obtained, all the pixel values in the distorted image can be obtained.

과 x 축의 단위 거리 당의 픽셀 수를 뜻하는

, y 축의 단위 거리 당의 픽셀 수를 뜻하는

, 그리고 초점 거리(

)를 나타나며, 왜곡 보정 상수인

와 왜곡 상수인

도 또한 미리 구해놓은 후에 저장될 수 있다. At this time, the internal parameters of the camera are stored in advance, and the internal parameters of the camera are centered on the image.

And the number of pixels per unit distance on the x-axis

, the number of pixels per unit distance on the y axis

, And focal length (

) And the distortion correction constant

And distortion constant

Can also be stored after preliminary.

In summary, 1) the distorted image feature point coordinates are moved to the normalized image plane using (Equation 7) and (Equation 8). This can be obtained using camera internal parameters.

(Formula 7)

(Equation 8)

2) Find the distance r from the center of the image using Equation (9).

(Formula 9)

)를 추정한다.3) Using the equation (10), the incidence angle (

).

(Formula 10)

)로부터 계획적인 투영(projective projection)을 수행함으로써 왜곡이 제거된 거리(

)를 얻을 수 있다.4) The incident angle estimated using (Equation 11)

Distance from which distortion is removed by performing projective projection from

) Can be obtained.

(Equation 11)

와

로부터 왜곡이 없는 이미지 특징 점의 좌표

를 각각 찾는다.5) using (Formula 12)

Wow

Of image feature points without distortion from

Find each one.

(Formula 12)

와

를 구한 뒤에

와 대응되는 왜곡이 없는 픽셀 좌표

의 픽셀 값을

의 픽셀 값을 가져와서 적용하면 도 7에서와 같이, 왜곡이 보정된 가상 공간 이미지를 얻을 수 있게 된다. 여기서, 상술한 왜곡 보정 모델 외에 다양한 왜곡 보정 모델을 사용하여 왜곡을 보정할 수 있음은 물론이다.Corresponding to the above format

Wow

After finding

Pixel coordinates without distortion corresponding to

The pixel value of

If the pixel value of is taken and applied, as shown in FIG. 7, a virtual space image having distortion corrected can be obtained. Here, of course, the distortion can be corrected by using various distortion correction models in addition to the above-described distortion correction model.

If the user adjusts the display of the camera image to the virtual view, the conversion unit 140 converts the camera image and the virtual space image into the camera image of the virtual view and the camera image of the virtual view, respectively, using the virtual view transformation model. The virtual space image of the virtual view can be projected onto the camera image plane of the virtual view.

In more detail, the conversion unit 140 converts a camera image into a camera image of a virtual viewpoint using a virtual viewpoint transformation model, and converts a virtual space image into a virtual space image of a virtual viewpoint, and then converts the virtual image into a virtual viewpoint. By mapping the converted virtual space image to coordinates on the camera image plane converted to the virtual viewpoint, the converted camera image may be output to various viewpoints desired by the user. In this case, the virtual viewpoint transformation model may be configured as a model of various methods capable of converting a camera image and a virtual space image into a virtual viewpoint.

FIG. 8 is a view illustrating a process of synthesizing a virtual space image and a camera image according to an embodiment of the present invention. Referring to FIG. 8, the synthesis unit 160 displays a virtual space image projected on a camera image plane. In this case, the virtual space image in which the predetermined pattern is formed and the camera image may be synthesized by an overlay method so as to overlap the camera image.

That is, the synthesis unit 160 transparently processes the black portion of the virtual space image so that the camera image converted to the virtual viewpoint and the virtual space image converted to the same viewpoint as the camera image appear naturally, and the transparent processed virtual space. The image and camera images can be combined by blending.

The display unit 170 displays a camera image obtained by synthesizing a predetermined pattern on a display module such as a liquid crystal display (LCD) and an organic light emitting diode (OLED). Of course, it is possible to receive the surrounding image from the automatic navigation device (not shown) installed in the vehicle and display it on the screen.

Hereinafter, an image processing process of a camera according to an embodiment of the present invention will be described.

9 is an operation flowchart illustrating an image processing process of a camera according to an embodiment of the present invention. Referring to FIG. 9, a projection transformation matrix between a real world space and a camera image plane is obtained (S910).

As described above, before obtaining the projection transformation matrix, the position and attitude of the camera can be estimated by using the calibration index arranged at a predetermined position.

In more detail, after arranging the calibration indicators at a predetermined position (for example, the floor), a comparison between a plurality of calibration indicator images photographed through the plurality of cameras 110a to 110n and a previously stored reference image may be performed. Through this, the position and attitude of the camera can be estimated.

In this case, the calibration index may be formed in a grid pattern or an arbitrary pattern in order to minimize the estimation error, and the grid pattern may be formed of a combination of colors with strong color contrast. For example, the calibration indicator may be composed of a grid-shaped pattern made of a combination of white and black.

Next, a homography between the virtual space image plane and the camera image plane on which a predetermined pattern is formed may be obtained using the projection transformation matrix (S920).

Here, the virtual space image plane may be formed of at least one plane among the five planes, so that the operation unit 130 may obtain homography corresponding to the at least one plane. For example, if the virtual space image plane consists of a bottom plane, a front plane, a left plane, a right plane, and a back plane, the calculation unit 130 is a bottom plane homography, a front plane homography, a left plane homography, a right plane. Planar homography and back plane homography can be obtained respectively.

In operation S930, the virtual space image may be projected onto the camera image plane using homography.

If the user adjusts the camera image to be displayed in the virtual view, the camera image is converted into the camera image of the virtual view using the virtual view transformation model, and the virtual space image is converted into the virtual space image of the virtual view, respectively. By mapping the virtual space image converted into the virtual viewpoint to the coordinates on the camera image plane converted into the virtual viewpoint, the camera image converted to various viewpoints desired by the user may be output. In this case, the virtual viewpoint transformation model is a model capable of converting a camera image and a virtual space image into a virtual viewpoint, and may be defined in various ways in advance.

In addition, the distortion correction model may correct the distortion of the camera image and the distortion of the virtual space image.

Then, the virtual space image may be synthesized with the camera image (S940).

That is, the virtual space image in which a predetermined pattern is formed may be synthesized by an overlay method so that the camera image overlaps, and when the virtual space image and the camera image are converted to the virtual viewpoint, the same as the camera image converted to the virtual viewpoint. The black portion of the virtual space image may be transparently processed so that the virtual space image converted to a viewpoint may appear naturally, and the transparent virtual space image and the camera image may be synthesized by blending.

One embodiment of the present invention includes a computer-readable medium including program instructions for performing various computer-implemented operations. This medium records a program for executing the image processing method of the camera described above. The medium may include program instructions, data files, data structures, etc., alone or in combination. Examples of such media include magnetic media such as hard disks, floppy disks and magnetic tape, optical recording media such as CD and DVD, programmed instructions such as floptical disk and magneto-optical media, ROM, RAM, And a hardware device configured to store and execute the program. Or such medium may be a transmission medium, such as optical or metal lines, waveguides, etc., including a carrier wave that transmits a signal specifying a program command, data structure, or the like. Examples of program instructions include machine language code such as those produced by a compiler, as well as high-level language code that can be executed by a computer using an interpreter or the like.

The foregoing detailed description is illustrative of the present invention. It is also to be understood that the foregoing is illustrative and explanatory of preferred embodiments of the invention only, and that the invention may be used in various other combinations, modifications and environments. That is, it is possible to make changes or modifications within the scope of the concept of the invention disclosed in this specification, the disclosure and the equivalents of the disclosure and / or the scope of the art or knowledge of the present invention. The foregoing embodiments are intended to illustrate the best mode contemplated for carrying out the invention and are not intended to limit the scope of the present invention to other modes of operation known in the art for utilizing other inventions such as the present invention, Various changes are possible. Accordingly, the foregoing description of the invention is not intended to limit the invention to the precise embodiments disclosed. It is also to be understood that the appended claims are intended to cover such other embodiments.

100: image processing device of the camera
110a to 110n: multiple cameras 120: estimator
130: calculator 140: converter
150: correction unit 160: synthesis unit
170:

Claims (16)

A calculation unit obtaining a projection transformation matrix between the real world space and the camera image plane, and obtaining a homography between the virtual space image plane having the predetermined pattern and the camera image plane using the projection transformation matrix;
A transformation unit projecting the virtual space image to the camera image plane using the homography;
Comprising a synthesizer for synthesizing the virtual space image projected on the camera image plane to the camera image,
The converting unit converts the camera image and the virtual space image into a camera image of the virtual view and a virtual space image of the virtual view by using a virtual view transformation model, and converts the virtual space image of the virtual view to the camera image of the virtual view. Project onto a plane,
The camera is installed in the car,
And the predetermined pattern is a guide line for parking the vehicle. delete The method of claim 1,
Further comprising an estimator for estimating the position and attitude of the camera,
And the calculation unit obtains a projection transformation matrix between the real world space and the camera image plane using camera information including the position and attitude of the camera. The method of claim 3, wherein
And the estimating unit receives a calibration index image photographed through the camera and estimates the position and attitude of the camera using the calibration index image. The method of claim 1,
And the virtual space image plane comprises at least one of a bottom plane, a front plane, a left plane, a right plane, and a rear plane. The method of claim 5, wherein
And the calculating unit obtains a homography corresponding to at least one of a bottom plane, a front plane, a left plane, a right plane, and a rear plane of the virtual space image plane by using the projection transformation matrix. The method of claim 1,
And a correction unit configured to correct distortion of the camera image and distortion of the virtual space image by using a distortion correction model. delete Obtaining a projection transformation matrix between the real world space and the camera image plane;
Obtaining a homography between the virtual space image plane having a predetermined pattern and the camera image plane using the projection transformation matrix;
Converting the camera image and the virtual space image into a camera image of the virtual view and a virtual space image of the virtual view by using a virtual view transformation model, respectively.
Projecting the virtual spatial image onto the camera image plane using the homography;
Compositing the virtual space image projected on the camera image plane with the camera image,
The projecting step projects the virtual space image of the virtual viewpoint to the camera image plane of the virtual viewpoint,
The camera is installed in the car,
And the predetermined pattern is a guide line for parking the vehicle. delete The method of claim 9,
Before estimating the projection transformation matrix, further comprising estimating the position and attitude of the camera,
The obtaining of the projection transformation matrix may include obtaining a projection transformation matrix between the real world space and the camera image plane using camera information including the position and the attitude of the camera. The method of claim 11,
In the estimating step, the calibration index image photographed by the camera is received, and the position and posture of the camera is estimated using the calibration index image. The method of claim 9,
And correcting the distortion of the camera image and the distortion of the virtual space image by using the distortion correction model after the projecting step. The method of claim 9,
And the virtual space image plane comprises at least one of a bottom plane, a front plane, a left plane, a right plane, and a rear plane. 15. The method of claim 14,
The obtaining of the homography may include obtaining a homography corresponding to at least one of a bottom plane, a front plane, a left plane, a right plane, and a rear plane of the virtual space image plane by using the projection transformation matrix. . delete

Images