KR101093316B1 - Method and System for Image Matching While Driving Vehicle - Google Patents

Abstract

The present invention relates to an image registration method and a system thereof while driving.

The present invention includes receiving a side image from a side camera and a rear image from a rear camera, extracting a distortion correction parameter from the rear image, calculating homography between adjacent images, and an extension center of the rear image. Comprising a step of performing a region separation for each of the side image and the rear image to the road image and the background image, and the distortion correction parameter, the homography between the adjacent image and the road image and the background image Provided is an image registration method and system for performing image registration, including a registration step of performing an image registration using the FOE to create a registration image.

According to the present invention, when the cameras are installed on the side and the rear of the vehicle, the centers of the cameras do not coincide with each other and the types of the rear and side cameras are different. There is an effect of providing image matching for images obtained from a plurality of cameras.

Image registration, blind spot, side / rear

Description

Image Matching Method and System for Vehicle Driving {Method and System for Image Matching While Driving Vehicle}

The present invention relates to an image registration method and a system thereof while driving. More specifically, the image registration method while driving the vehicle by removing a blind spot by mounting a plurality of cameras on the side / rear of the vehicle, and after generating the image from each camera through the matching step to provide a single image to the driver And to the system.

Blind spots that are invisible to the driver while driving are a major danger to the driver's safety. It is difficult to pay attention to the rear of the vehicle because the driver who drives the vehicle is generally only looking ahead. Various blind spot removal devices have been developed for vehicles to eliminate these hazards. In order to eliminate blind spots that cannot be seen in the side mirrors, a device has been developed and used in actual vehicles to integrate an auxiliary mirror or to detect a rear object with an infrared sensor and alert the driver through an alarm. Recently, there are methods for reducing blind spots by outputting the image of the rear of the vehicle to the driver's display.

However, these methods have the disadvantage of essentially removing only part of the blind spot that the driver has. This alone causes the objects behind to be invisible, and in fact, many drivers are not aware of the vehicles approaching from the rear when they change lanes and are suffering casualties due to large and small traffic accidents.

In the case of the side auxiliary mirror, the driver sees a distorted image of the left and right rear through an additional convex mirror. In the case of the right side far away from the driver, only the existence of an object is detected dimly.

Detecting objects in the blind spot using an infrared sensor does not show objects in the blind spot, but only discriminates the presence or absence of objects in the sensor's area. Considering that the most sensitive sensation among human senses is vision, the driver's perception of dangerous goods is considerably lower than the rear view. Moreover, when there is a large number of dangerous goods, the driver may not know whether there are many dangerous goods or one, and the accident may occur.

In addition, the rear surveillance camera is an efficient blind spot removal method that shows the rear as it is to the driver, but the range is limited to the rear, and side problems cannot be solved. Moreover, the rear surveillance camera has a problem that the image is severely distorted due to the mounting of a wide angle camera.

In order to solve the above problems, the present invention is equipped with a plurality of cameras on the side / rear of the vehicle, remove the blind spots by correcting the image from each camera to generate a single image through the matching step and then provide to the driver The purpose is to.

In order to achieve the above object, the present invention provides a video registration method for driving a vehicle, comprising: (a) an image receiving step of receiving a side image from a side camera and a rear image from a rear camera; (b) extracting distortion correction parameters from the rear image; (c) calculating homography between adjacent images; (d) calculating a focus of expansion (FOE) of the rear image; (e) an area separation step of performing area separation on the side image and the rear image into a roadway image and a background image, respectively; And (f) a matching step of performing a matching operation using the distortion correction parameter, the homography between the adjacent images, and the FOE to the difference image and the background image to form a matching image. To provide.

In addition, the present invention, in order to achieve the above object, an image matching system while driving a vehicle, comprising: a side camera for receiving a side image; A rear camera for receiving a rear image; An image receiver configured to receive the side image and the rear image; A correction parameter extracting unit extracting a distortion correction parameter from the rear image; A homography calculator for calculating homography between adjacent images; A FOE calculator configured to calculate a focus of expansion of the rear image; A region separator configured to separate regions of the side image and the rear image into a roadway image and a background image, respectively; And an image registration unit for performing image registration using the distortion correction parameter, the homography between the adjacent images, and the FOE on the difference image and the background image to form a registration image. .

According to the present invention, when the cameras are installed on the side and the rear of the vehicle, the centers of the cameras do not coincide with each other and the types of the rear and side cameras are different. There is an effect of providing image matching for images obtained from a plurality of cameras.

In addition, the image registration method and the visualization method for removing the blind spot has an effect as a safety device for prevention of lowering the accident rate by easily recognizing the dangerous object of the blind spot threatening the driver's safety. In particular, the attached camera acquires images of a wide area that could not be seen with the existing side mirrors and rearview mirrors, and helps to improve the cognitive ability of dangerous objects by matching these images and visualizing them in the form of a panorama that is easy for the driver to see.

Another effect is better image matching in terms of technology. Conventional image matching methods mostly match only images that have the same projection point (the center of the camera) of the image. When the camera is mounted on a moving object such as a vehicle, motion parallax occurs. This has the effect of solving the impossible match.

In addition, the camera mounted on the vehicle overcomes the limitation that it is impossible to mount the camera so that the projection point of the camera coincides, and furthermore, the rear camera generally overcomes the limitations of different camera types or different viewing areas. There is an effect of matching the images obtained in the current situation that violates all of them.

Hereinafter, embodiments of the present invention will be described in detail with reference to the accompanying drawings. First of all, in adding reference numerals to the components of each drawing, it should be noted that the same reference numerals are used as much as possible even if displayed on different drawings. In the following description of the present invention, a detailed description of known functions and configurations incorporated herein will be omitted when it may make the subject matter of the present invention unclear.

1 is a diagram illustrating a screen displayed to a driver using an image registration method according to an embodiment of the present invention.

As shown in FIG. 1, in order to efficiently remove the blind spot, a rear camera 106 and a left camera 104 and a right camera 102 may be mounted on the left and right sides of the vehicle. The present invention provides a method of matching images (left image, right image, rear image) into a panorama-type registration image and providing the driver with the image.

Accordingly, the present invention can be equipped with three cameras in the rear and side of the vehicle side mirror position to assist and further replace the rear and side mirrors in the rear and side of the vehicle. Each camera provides the driver with more of the side and rear blind spots removed from conventional side and rear mirrors.

If each of these images is provided separately, the driver's cognitive ability is lowered. On roads where most cars run fast, this is a very dangerous problem. Accordingly, the present invention provides a visualization method that allows a driver to recognize an object present in the rear blind spot more quickly and efficiently by extracting a matched image of three images.

2 is a flowchart illustrating an image registration method according to an embodiment of the present invention.

In an image matching method according to an embodiment of the present invention, an image receiving step (S202) of receiving a side image from a side camera and a rear image from a rear camera, and extracting a distortion correction parameter from the rear image (S204) Computing homography between adjacent images (S206), Computing a focus of expansion (FOE) of the rear image (S208), and a side view image and a background image with respect to the lateral image and the rear image A region separation step (S210) for performing region separation, and a matching step of performing image registration using the distortion correction parameter, the homography between the adjacent images, and the FOE on the driveway image and the background image to form a matched image ( S212).

Here, after the matching step (S212), it may further include a cylinder mapping step (S214) for performing a cylinder mapping on the registration image, rolls to include the information of the vehicle in the registration image instead of the cylinder mapping step (S214) The panoramic image generating step S216 may be performed instead.

The homography between adjacent images calculated in step S206 may be lateral posterior homography between the lateral image and the rear image and front and rear homography between the front and rear frames of the rear image. At this time, the side image may be composed of a left image and a right image, in this case, the lateral rear homography may be composed of left and right homography and right and left homography.

In addition, in the area separation step S210, the area separation may use a color-based method.

In the step S202 of receiving a side image from a side camera and receiving a rear image from a rear camera, a right camera (general camera) mounted on the right side of the vehicle, a left camera (general camera) mounted on the left side, and a rear mounted on the rear side A right image, a left image and a rear image are respectively received from a camera (wide angle camera).

Before performing blind spot image registration, a correction step consisting of a distortion correction parameter extraction step (S204), a homography calculation step (S206), and a FOE calculation step (S208) is performed as a task to be calculated before driving for efficiency of calculation. To perform.

First, in the step S204 of extracting the distortion correction parameter from the rear image, a calibration operation is performed on an image obtained from the rear wide-angle camera among the input images. If distortion correction parameters are obtained for a single image, subsequent images can be corrected and used at this stage. Correcting the image of the actual wide-angle camera does not use complex equations, and simply and quickly performs the correction using the relationship between the sphere and the plane, assuming that the distorted wide-angle camera lens is the same as the surface of the sphere.

3 is a diagram illustrating a distortion correction method of an image of a wide-angle camera in the distortion correction parameter extraction step (S204).

Assuming that the surface of the lens is a sphere, the image mapped to the surface of the sphere will be a distorted image. Accordingly plane (X _u, Y _u) on the non-distorted and by using a relational expression between a position projecting in the same plane image is mapped to sphere (X _d, Y _d) a distortion correction position (X _u, Y _u) Can be obtained. Referring to FIG. 3, the relational expression may be represented as in [Equation 1] and [Equation 2].

Using Equations 1 and 2, points at positions corrected by backward mapping may be calculated in advance. By simply applying the position corresponding to the calculated value to all the distorted images as it is, it is possible to quickly obtain the distorted image in the image registration step described later.

After the preprocessing for distortion correction, the homography between adjacent images to be used to match the difference image between the lateral rear image and the front and rear frames of the same position is calculated (S206) (here, using robust RANSAC (Random Sample Consensus) algorithm) . In calculating the homography between adjacent images (S206), homography between adjacent images includes left and right homography between a left image and a rear image, right and right homography between a right image and a rear image, and front and back between front and rear frames of a rear image. Calculate homography.

FIG. 4 is a diagram illustrating the meaning of homography between two images in the homography calculation step S206.

The side / rear image of the same position refers to an image captured by the side camera and the rear camera on the same (at least similar) world coordinate system at a predetermined time interval. Generally, homography is a 3x3 transformation matrix going from one image to another.

As shown in FIG. 4, when the homography H _ab is calculated, the point P _a of the image _a may be mapped to the point P _b of the image b. Generally, homography is a 3x3 transformation matrix going from one image to another. Therefore, the homography between the lateral driveway image and the rear driveway image and the front and rear frames of the rear image can be calculated in advance, and the information can be used in the subsequent image registration step. In this manner, right to rear homography, left to right homography, and front to rear homography are obtained.

After the homography calculation step S206, a step of calculating a focus of expansion (FOE) of the rear image is performed (S208). The FOE refers to a point at which each motion on the image converges when only translation is performed in one direction of the camera position. The FOE is used to separate the driveway image and the background image, and this FOE is calculated in the preprocessing step. The FOE calculates coincidence points between images of different frames and obtains motion of each point from the coincidence points. The position where the motions thus converged is the FOE.

5 is a view showing the meaning of the FOE calculated in the FOE calculation step (S208) and its calculation method.

In operation S208, the final operation of obtaining the FOE is as follows. In general, when the camera is photographed while moving in a certain direction, the optical flow between the captured images is moved based on a specific point. It is the FOE that meets these moving vectors. Figure 5a shows the relationship between the optical flow and FOE in the image plane.

The FOE is created by the movement of the camera, where the fixed camera moves forward at high speed. Therefore, the FOE is obtained in the calibration step, and the obtained result is used while driving.

5B is a diagram showing the relationship between the movement of the camera center and the resulting FOE during two frames of the rear image.

Now, the FOE of the image plane is obtained using the feature points obtained from the image difference. There are various ways to obtain FOE, but it can be classified into three types. The first method is a discrete method, which is obtained by using matching points or lines. This method is tricky and inaccurate but has the advantage of being available with simple information. The second method is a continuous method, which uses optical flow. This method is more accurate than the previous method, but more computational. The third method is to use a change in brightness, which can be used for an image in which there is only a pure shift transform without rotation.

First, when we know at least eight matching feature point pairs, the fundamental matrix (F) represents the relationship between the points in the image coordinates according to [Equation 3]. Can be. In this case, RANSAC can be used to automatically calculate a more accurate fundamental matrix. Here, x is a two-dimensional coordinate of X when the vehicle is at the position O _c , and x 'is a two-dimensional coordinate of X when the vehicle is at the position O _c '.

FOE can be obtained by solving Epipole e by solving Equation 4 using the obtained fundamental matrix (F).

After the step S208, the area separation step (S210) for performing the area separation for the side image and the rear image to the roadway image and the background image, respectively.

For image registration, not only the projection point of the image coincides, but most of the image must exist on one plane. However, more than half of the images taken with the camera mounted on the vehicle occupy an area of the roadway, and the rest occupy the background. Since the background and the roadway do not exist in the same plane, it is impossible to match with a general image matching method (for example, Mosaiking).

Therefore, here, the background and the roadway are separated and matched as a method for matching images. In order to separate the background image and the driveway image, first, the images of the similar positions are used in the side image and the rear image. That is, by using images photographed at similar locations but different time zones, the transformation between the front and rear is minimized and the area is divided using the image set. One is the roadway area and the other is the background image part. This step uses a color-based method.

The color-based method separates the part estimated by the roadway by using the difference between the hugh value and the saturation value in the road and the background in the image, and determines the driveway area based on the separated image.

FIG. 6 is a diagram illustrating a roadway area detected by the color-based method in the area separation step S210. The driveway region of FIG. 6B is separated from the image of FIG. 6A.

If the road image and the background image are separated by performing a process of separating the road image and the background image, it is possible to apply the roadway region separated from the rear side to the left and right images by using homography between the left and right images. Therefore, it is possible to separate the roadway image and the background image for the left and right images without any calculation.

FIG. 7 is a diagram illustrating a result of separating a roadway region of a side image by using homography in a rear image in an area separation step (S210).

As shown in FIG. 7, the roadway region may be separated using left and right homography (H _L ) and right and right homography (H _R ).

After the area separation step S210, a matching step S212 is performed to generate a matched image by performing image registration using a distortion correction parameter, homography between adjacent images, and a FOE on the roadway image and the background image.

The final matching is performed while driving by using the image obtained by separating the road and the background and the parameters obtained in the correction step. In this step, the homograph between the lateral and posterior images at the same point in time (side / rear images at similar positions) and homography between front and rear frames is used to obtain the difference image of the image taken at the same time, not at the same position. Match based on the rear image. After this work, a virtual lateral plane is placed on the left and right sides of the matched image, and the lateral background image is mapped to the plane. In this way the images are matched.

FIG. 8 is a diagram illustrating a form in which a background image is mapped to a virtual plane set on the side in the registration step S212.

After the registration step S212, as a visualization method of the registration image, a cylinder mapping step S214 for performing cylinder mapping on the registration image may be performed. Here, the cylinder projection is performed to generate a panoramic image.

As another visualization method, a roll panorama image generating step (S216) of including the vehicle information in the registration image may be performed, and a natural rear view in which the rear registration image information is provided to the driver along with the vehicle information. Create and provide separately.

9 is a diagram illustrating an image matching system 900 according to an exemplary embodiment.

As shown in FIG. 9, the image registration system 900 according to an embodiment of the present invention includes a side camera 902 for receiving a side image, a rear camera 106 for receiving a rear image, a side image, and a rear image. Image receiving unit 906 for receiving the image, correction parameter extraction unit 908 for extracting the distortion correction parameter from the rear image, homography calculation unit 910 for calculating homography between adjacent images, and an extension center of the rear image (Focus of Expansion: FOE calculation unit 912 for calculating FOE, region separation unit 914 for segmenting the side image and the rear image into the difference image and the background image, the distortion correction parameter in the difference image and the background image, And an image registration unit 916 which performs image registration using homography and neighboring images between adjacent images to form a registration image.

In some cases, the visualizer 918 may further include a cylinder mapping for performing cylinder mapping to the registration image or a visualization unit 918 for generating a roll panorama image including information of the vehicle in the registration image.

9 and the operation of the image registration system 900 will be described below with reference to FIGS.

In order to remove the blind spot behind the vehicle while driving, the camera is mounted on the side (normal camera) and the rear (wide angle camera) of the vehicle. Matching is performed by using an image captured while driving using a mounted camera.

10 is a diagram illustrating a state in which a camera is mounted on a vehicle for image acquisition.

As the side camera 902, a right camera 102 and a left camera 104 are installed on the left and right sides of the vehicle, respectively. As shown in FIG. 10, the left camera 104 is installed below the left side mirror, and the right camera 102 is installed below the right side mirror (the right side mirror side is not shown in FIG. 10), The rear camera 106 is installed on the trunk upper part of the vehicle, but is not limited thereto.

The image receiver 906 performs an image receiving step S202 for receiving the side image and the rear image. First, the driving image, that is, the left image, the right image, and the rear image are captured by the camera installed at the left and right sides and the rear of the vehicle and received as an input. Cameras on the left and right are taken using a wide-angle camera having a field of view of 30 degrees and a rear camera of 120 degrees. Side camera and rear camera are installed at a distance of 2.8m. 11 is a diagram illustrating an input image photographed by a camera.

The correction parameter extractor 908 extracts the distortion correction parameter from the rear image (S204).

The homography calculator 910 calculates homography between adjacent images (S206). As described in step S206, here, homography between adjacent images is lateral and homography between the lateral and rear images and front and rear homography between the front and rear frames of the rear image, and the lateral and rear homography is again performed by the left and right homography and It is made by homography.

The FOE calculator 912 calculates a focus of expansion (FOE) of the rear image (S208).

The area separating unit 914 performs a step of separating the area into the roadway image and the background image for the lateral image and the rear image, respectively (S210). Here, the color-based method is used as the region separation method as described in step S210. After the distortion correction operation and the homography calculation of the rear image are completed based on the photographed image, the operation of separating the road image and the background of the rear image is performed based on the information. 12 is a diagram illustrating results of a background and a roadway image separated by the rear camera 106. Based on the obtained result of the rear image, it is possible to separate the road image and the background image from the side image using homography between adjacent images.

The image matching unit 916 performs a matching step (S212) to create a matching image by performing image registration using the distortion correction parameter, the homography between adjacent images, and the FOE on the roadway image and the background image. After the roadway and the background image are separated, registration is performed using the calculated homography and the separated image. FIG. 13 is a diagram illustrating a result of matching images of FIG. 11.

The visualization unit 918 generates a roll panorama image including information of a vehicle in a cylinder mapping or registration image that performs cylinder mapping on the registration image (S216).

In order to display the matched image to the driver, it is necessary to find an appropriate visualization method. The first solution was a panoramic image using cylinder mapping. 14 is a panoramic image made using cylinder mapping.

The panoramic image can easily express a large area. However, the information of the vehicle and other drivers cannot easily express the necessary information, and there are distorted and removed images to make a panorama. Therefore, in order to reduce such a problem and express the natural image matched with each other, a roll panorama image including information of a vehicle can be made. FIG. 15 is a diagram illustrating a final roll panorama image. FIG. 15 is a visualization screen for representing a virtual model of a vehicle, displaying the driving speed and various information of the vehicle, and providing the driver with rear information around the vehicle.

The foregoing description is merely illustrative of the technical idea of the present invention, and various changes and modifications may be made by those skilled in the art without departing from the essential characteristics of the present invention. Therefore, the embodiments disclosed in the present invention are not intended to limit the technical idea of the present invention but to describe the present invention, and the scope of the technical idea of the present invention is not limited by these embodiments. The protection scope of the present invention should be interpreted by the following claims, and all technical ideas within the equivalent scope should be interpreted as being included in the scope of the present invention.

According to the present invention, when the cameras are installed on the side and the rear of the vehicle, the centers of the cameras do not coincide with each other and the types of the rear and side cameras are different. There is an effect of providing image matching even for images obtained from a plurality of cameras.

In addition, the image registration method and the visualization method for removing the blind spot has an effect as a safety device for prevention of lowering the accident rate by easily recognizing the dangerous object of the blind spot threatening the driver's safety. In particular, the attached camera acquires images of a wide area that could not be seen with the existing side mirrors and rearview mirrors, and helps to improve the cognitive ability of dangerous objects by matching these images and visualizing them in the form of a panorama that is easy for the driver to see.

Another effect is better image matching in terms of technology. Conventional image matching methods mostly match only images that have the same projection point (the center of the camera) of the image. When the camera is mounted on a moving object such as a vehicle, motion parallax occurs. This has the effect of solving the impossible match.

In addition, the camera mounted on the vehicle overcomes the limitation that it is impossible to mount the camera so that the projection point of the camera coincides, and furthermore, in the case of the rear camera, it overcomes the limitation of the different types of cameras or different viewing areas. There is an effect of matching the images obtained in the current situation that violates all of them.

1 is a diagram illustrating a screen displayed to a driver using an image registration method according to an embodiment of the present invention.

2 is a flowchart illustrating an image registration method according to an embodiment of the present invention.

3 is a diagram illustrating a distortion correction method of an image of a wide-angle camera in the distortion correction parameter extraction step (S204).

FIG. 4 is a diagram illustrating the meaning of homography between two images in the homography calculation step S206.

5 is a view showing the meaning of the FOE and the calculation method in the FOE calculation step (S208).

FIG. 6 is a diagram illustrating a roadway area detected by the color-based method in the area separation step S210.

FIG. 7 is a diagram illustrating a result of separating a roadway region of a side image by using homography in a rear image in an area separation step (S210).

FIG. 8 is a diagram illustrating a form in which a background image is mapped to a virtual plane set on the side in the registration step S212.

9 is a diagram illustrating an image matching system 900 according to an exemplary embodiment.

10 is a diagram illustrating a state in which a camera is mounted on a vehicle for image acquisition.

11 is a diagram illustrating an input image photographed by a camera.

12 is a diagram illustrating results of a background and a roadway image separated by the rear camera 106.

FIG. 13 is a diagram illustrating a result of matching images of FIG. 11.

14 is a panoramic image made using cylinder mapping.

FIG. 15 is a diagram illustrating a final roll panorama image.

<Description of Symbols for Main Parts of Drawings>

102: right camera 104: left camera

106: rear camera 900: video registration system

902: side view camera 906: video receiver

908: correction parameter extraction unit 910: homography calculation unit

912: FOE calculation unit 914: region separation unit

916: Image matching unit 918: Visualization unit

Claims (10)

In the image matching method while driving the vehicle, (a) an image receiving step of receiving a side image from a side camera and a rear image from a rear camera; (b) extracting distortion correction parameters from the rear image; (c) calculating homography between adjacent images; (d) calculating a focus of expansion (FOE) of the rear image; (e) an area separation step of performing area separation on the side image and the rear image into a roadway image and a background image, respectively; And (f) a matching step of performing a matching operation on the difference image and the background image by using the distortion correction parameter, the homography between the adjacent images, and the FOE to create a matching image; Image matching method comprising a. The method of claim 1, wherein the homography between adjacent images is And lateral rear homography between the lateral image and the rear image, and front and rear homography between front and rear frames of the rear image. The method of claim 2, wherein the side view image, Including a left image and a right image, The side rear homography, A left and right liver homography and a right rear homography. The method of claim 1, wherein in step (e), The area separation is an image matching method, characterized in that using a color-based method. The method of claim 1, wherein the image registration method comprises: And a cylinder mapping step of performing cylinder mapping on the registration image after the step (f) or a roll panorama image generation step of including information of the vehicle in the registration image. In image matching system while driving a vehicle, A side camera for receiving a side image; A rear camera for receiving a rear image; An image receiver configured to receive the side image and the rear image; A correction parameter extracting unit extracting a distortion correction parameter from the rear image; A homography calculator for calculating homography between adjacent images; A FOE calculator configured to calculate a focus of expansion of the rear image; A region separator configured to separate regions of the side image and the rear image into a roadway image and a background image, respectively; And An image matching unit for generating a registration image by performing image registration using the distortion correction parameter, the homography between the adjacent images, and the FOE on the difference image and the background image. Image matching system comprising a. The method of claim 6, wherein the homography between adjacent images, And a front and rear homography between the front and rear frames of the rear image and the rear and rear homography between the side image and the rear image. The method of claim 7, wherein the side camera, Including a right camera and a left camera, The side image is, Including a left image and a right image, The side rear homography, An image registration system comprising left to right homography and right to left homography. The method of claim 6, wherein the region separator, And a color-based method as the area separation method. The method of claim 6, wherein the image registration system, And a visualization unit configured to generate a cylinder mapping for performing cylinder mapping to the registration image or a roll panorama image including information of a vehicle in the registration image.

Images