KR20180110534A - Apparatus and method for around view monitoring - Google Patents

Abstract

An apparatus for monitoring an around view according to an embodiment of the present invention includes an image receiving part for receiving images from a plurality of cameras, a 2D table correction part which detecting a plurality of straight lines corresponding to at least one of a reference line, a feature point, and a vehicle area from each of the images, calculating a homography matrix between surfaces including the straight line of an adjacent image among the images, and generating a 2D lookup table according to the homography, and a 2D matching part for matching the images according to the 2D lookup table to generate a top view image. The lookup table is automatically corrected even if the shooting angle of the camera is changed.

Description

[0001] APPARATUS AND METHOD FOR AROUND VIEW MONITORING [0002]

BACKGROUND OF THE INVENTION 1. Field of the Invention [0002] The present invention relates to an approach view monitoring technology, and more particularly, to an approach view monitoring technology for automatically correcting an error when a photographing angle of a camera is changed to match images.

The AVM (Around View Monitoring) device is a technology that provides the driver with surrounding environment information by matching four wide-angle camera images installed in the vehicle with one image. Around-view monitoring technology is a technology most applied to flagship model vehicles.

Top View AVM (AVM) is a technology that displays two-dimensional images as seen from the top of a vehicle by matching four images. In the 2010s, 3D AVM was developed that provides AVM in three dimensions.

Korean Patent Laid-Open Publication No. 10-2013-0040595 is a prior art of the present invention.

SUMMARY OF THE INVENTION It is an object of the present invention to provide an apparatus and method for monitoring an ambient view, which automatically corrects a lookup table even when a photographing angle of a camera is changed.

According to an aspect of the present invention, there is provided an image processing apparatus comprising: an image receiving unit for receiving images from a plurality of cameras, respectively; Detecting a plurality of straight lines corresponding to at least one of a reference line, a minutiae, and a vehicle area from each of the images, calculating a homography matrix between surfaces of the adjacent images including the straight line, A 2D table correcting unit for generating a 2D lookup table according to the 2D lookup table; And a 2D matching unit for matching the images according to the 2D look-up table to generate a top view image.

Wherein the 2D table correction unit comprises: a reference line detection unit for detecting the reference line from the image; A feature point detection unit which detects a shadow region and an illumination region corresponding to the feature point from the image and calculates a straight line constituting the shadow region and the outline of the illumination region; A vehicle area detecting unit for detecting a vehicle area from the image and calculating a straight line constituting an outline of the vehicle area; A matrix calculation unit for calculating a homography matrix between surfaces including a plurality of straight lines along at least one of the reference line, the minutiae, and the vehicle area; And a look-up table corrector for generating the 2D look-up table according to the homography matrix.

The reference line may be a line that is projected in a predetermined direction in the vehicle and is reflected by the external environment and appears in the image photographed by the camera.

Wherein the special point detecting unit generates a frame difference image for an out-of-vehicle area in an image when the vehicle is straight ahead during a day, applies a moving average recursive filter to each pixel value of the difference image, To generate a shadow area including pixels whose pixel values are within a predetermined range from 0, generate an inter-frame difference image for the non-vehicle area in the image taken at night, A moving average recursive filter may be applied to the moving average recursive filter and the lighting area including the pixels whose pixel value applying the moving average recursive filter is within a predetermined range from 0 may be calculated.

The vehicle area detecting unit may detect, as the vehicle area, an area including a pixel whose value obtained by applying a moving average recursive filter to a difference between corresponding pixels of a previous frame and a current frame of the image is within a predetermined range from 0 have.

A 3D table correcting unit for detecting a transform region from the top view image and generating a 3D lookup table for performing transform on the transform region; And a 3D matching unit for matching the top view image according to the 3D lookup table to generate a 3D image.

Wherein the 3D table correcting unit comprises: a road surface detecting unit for detecting a road surface area from the top view image; A fitting part for detecting a figure to be fitted to the road surface area; A conversion area calculating unit for selecting, as the conversion area, an area corresponding to the outside of the figure from among the top view images; And a look-up table corrector for generating a 3D look-up table for changing a position of a pixel in the transform area according to a specified curvature.

The road surface detection unit generates a predictive image by moving the pixel of the previous top view image by a distance corresponding to the vehicle speed in the driving direction and calculates a difference between each pixel value of the predictive image and each pixel value of the current top view image And a pixel corresponding to a pixel value within a predetermined range from 0, which is obtained by applying a moving average recursive filter to each pixel value of the predicted difference image, can be detected as the road surface area.

The fitting unit can detect a figure having the smallest sum of distances from the pixels of the outline of the road surface area on the top view image.

According to another aspect of the present invention, there is provided a method of providing an overview view monitoring method, the method comprising: receiving an image from a plurality of cameras, respectively; Detecting a plurality of straight lines corresponding to at least one of a reference line, a feature point, and a vehicle area from the image; Calculating a homography matrix between planes of the adjacent images including the straight line; Generating a 2D look-up table according to the homography matrix; And generating a top view image by matching the images according to the 2D look-up table; A method of monitoring an ambient view is provided.

The step of detecting a plurality of straight lines corresponding to at least one of the reference line, the feature point, and the vehicle area from the image may include: detecting the reference line from the image; Detecting a shadow region and an illumination region corresponding to the feature point from the image and calculating a straight line constituting the shadow region and an outline of the illumination region; And detecting a vehicle area from the image and calculating a straight line constituting an outline of the vehicle area; . ≪ / RTI >

The reference line may be a line that is projected in a predetermined direction in the vehicle and is reflected by the external environment and appears in the image photographed by the camera.

Wherein the step of detecting a shadow region and an illumination region corresponding to a feature point from the image and calculating a straight line constituting the shadow region and an outline of the illumination region includes calculating, Frame difference image, applying a moving average recursive filter to each pixel value of the difference image, and applying the moving average recursive filter to the shadow region including pixels having a pixel value within a predetermined range from 0 Calculating; A moving average recursive filter is applied to each pixel value of the difference image, and the pixel value to which the moving average recursive filter is applied is changed from 0 to Calculating the illumination area including pixels within a specified range; And detecting a straight line corresponding to the shadow region and the outline of the illumination region.

Calculating a straight line constituting an outline of the vehicle area by detecting a vehicle area from the image includes calculating a difference between values of pixels corresponding to a previous frame and a current frame of the image by a value obtained by applying a moving average recursive filter to 0 Detecting a region including a pixel within a predetermined range from the vehicle area as the vehicle area.

The method includes the steps of: detecting a transform region from the top view image and generating a 3D lookup table for transforming the transform region; And generating a 3D image by matching the top view image according to the 3D lookup table.

The step of detecting a transform region from the top view image and generating a 3D lookup table for transforming the transform region may include detecting a road surface region from the top view image, Detecting a figure to be fitted to the road surface area; Selecting an area of the top view image corresponding to the outside of the figure as the conversion area; And generating a 3D lookup table that causes a position of a pixel in the transformation region to change according to a specified curvature.

The step of detecting a road surface area from the top view image may include generating a predicted image by moving a pixel of the previous top view image by a distance corresponding to a vehicle speed in a traveling direction; Generating a prediction difference image including a difference between pixel values of the prediction image and pixel values of a current top view image as pixel values; And detecting the pixel corresponding to the pixel value of the predicted difference image by applying a moving average recursive filter within a predetermined range from 0 to the road surface region.

The step of detecting a figure to be fitted to the road surface area may be a step of detecting a figure having the smallest sum of distances from each pixel of the outline of the road surface area on the top view image.

As described above, according to the present invention, normal surround view monitoring can be automatically provided even when the photographing angle of the camera is changed due to an external factor such as an accident.

1 is a block diagram illustrating an apparatus for monitoring an overview view according to an embodiment of the present invention.
BACKGROUND OF THE INVENTION Field of the Invention [0001]
3 is a view illustrating a reference line projected from a vehicle photographed by an ambient view monitoring apparatus according to an embodiment of the present invention;
FIG. 4 is a diagram illustrating a line formed by the vanishing point generated by the surrounding view monitoring apparatus according to the embodiment of the present invention; FIG.
5 is a diagram of lines in which the surround view monitoring device according to an embodiment of the present invention is made by a vanishing point created by a tilted camera;
6 is a block diagram illustrating a 3D table correction unit of an apparatus for monitoring the surround view according to an embodiment of the present invention;
7 is a flowchart illustrating a method of monitoring an overview view of an overview view monitoring apparatus according to an exemplary embodiment of the present invention;
FIG. 8 is a flowchart illustrating a process of an ambient view monitoring apparatus correcting a 2D look-up table according to an embodiment of the present invention. FIG.
9 is a diagram illustrating a process of generating a 3D lookup table by an ambient view monitoring apparatus according to an embodiment of the present invention.

While the present invention has been described in connection with certain exemplary embodiments, it is to be understood that the invention is not limited to the disclosed embodiments, but, on the contrary, is intended to cover various modifications and similarities. It should be understood, however, that the invention is not intended to be limited to the particular embodiments, but includes all modifications, equivalents, and alternatives falling within the spirit and scope of the invention.

FIG. 1 is a block diagram illustrating an apparatus for monitoring an overview view according to an exemplary embodiment of the present invention. Referring to FIG.

Referring to FIG. 1, an apparatus for monitoring an ambient view according to an exemplary embodiment of the present invention includes an image receiving unit 110, a 2D table correcting unit 120, a 2D matching unit 130, a 3D table correcting unit 140, (150).

The image receiving unit 110 receives images from a plurality of cameras. At this time, each camera is a camera installed to photograph a designated direction, and preferably, it may be four cameras that respectively photograph the front, rear, and side of the vehicle. The image receiving unit 110 may transmit the images received from the cameras to the 2D table correcting unit 120 and the 2D matching unit 130, respectively.

The 2D table corrector 120 analyzes the image received from the image receiver 110 and corrects the 2D geometric look-up table used when the 2D matching unit 130 matches the images. The process of correcting the 2D look-up table by the 2D table correcting unit 120 will be described later in detail with reference to FIG.

The 2D matching unit 130 corrects each of the images according to the 2D look-up table and matches them to generate a two-dimensional image (hereinafter referred to as a top view image) as viewed from above the vehicle. At this time, the 2D matching unit 130 may store the corrected 2D look-up table if the 2D look-up table is corrected according to the 2D table correcting unit 120 after storing the 2D look-up table in advance. The 2D matching unit 130 transmits the top view image to the 3D table correcting unit 140 and the 3D matching unit 150.

The 3D table correction unit 140 analyzes the top view image received from the 2D matching unit 130 and corrects the 3D lookup table stored in the 3D matching unit 150. [ Hereinafter, a process of correcting the 3D lookup table by the 3D table corrector 140 will be described in detail with reference to FIG.

The 2D matching unit 130 corrects each of the images according to the 3D look-up table, and matches them to generate a three-dimensional image (hereinafter referred to as 3D image) that can be viewed at various points around the vehicle. At this time, the 3D matching unit 150 may store the corrected 3D look-up table if the 3D look-up table is corrected according to the 3D table correcting unit 140 after storing the 3D look-up table in advance. The 3D matching unit 150 may output the 3D image to an external device (e.g., a display device).

FIG. 2 is a block diagram illustrating a 2D table correcting unit of an ambient view monitoring apparatus according to an exemplary embodiment of the present invention. FIG. 3 is a block diagram of an ambient view monitoring apparatus according to an embodiment of the present invention. FIG. 4 is a diagram illustrating a line formed by the vanishing point generated by the surrounding view monitoring apparatus according to an exemplary embodiment of the present invention, and FIG. 5 is a view illustrating an example of the surround view monitoring apparatus according to an exemplary embodiment of the present invention. And a line formed by a vanishing point generated by an inclined camera.

2, the 2D table corrector 120 of the apparatus for monitoring the surround view according to an embodiment of the present invention includes a reference line detection unit 210, a feature point detection unit 220, a vehicle region detection unit 230, A vanishing point detecting unit 250, a roll angle calculating unit 260, and a lookup table correcting unit 270.

The reference line detection unit 210 detects a reference line from an image. Referring to FIG. 3, the reference line indicates a line 310 that is projected in a predetermined direction in the vehicle and is reflected by the external environment and appears in the image captured by the camera. In this case, the reference line detection unit 210 can detect a straight line having the smallest variance with respect to pixels of a predetermined color in the image through the Least Mean Square.

The feature point detection unit 220 detects feature points from each image. The minutiae point detecting unit 220 can extract the minutiae during the daytime running or the nighttime running. For example, the feature point detection unit 220 determines whether the vehicle is traveling straight during the daytime running. At this time, the minutia matching point detector 220 can receive rotation information indicating the current steering rotation angle and the like from the vehicle during the weekly running of the vehicle, and can determine whether the vehicle is currently being straightened or rotated. The minutiae point detecting unit 220 calculates a difference image for the remaining area excluding the vehicle area (hereinafter referred to as " out-of-area area ") in the image when the current vehicle is running straight. The process of calculating the vehicle area at this time will be described in detail later with reference to the description of the vehicle area detection unit 230. [ The minutiae point detecting unit 220 may generate a difference image having a pixel value as a difference between pixels in a position on the out-of-vehicle area corresponding to the previous frame and the current frame of the image. The feature point detection unit 220 calculates a shadow area including pixels having a value obtained by applying a moving average recursive filter to a pixel value difference between consecutive frames from 0 to a predetermined range. That is, a pixel whose difference between pixel values between specific frames during a daytime running continuously has a value similar to 0 is an area corresponding to a shadow, and the feature point detector 220 may include the pixel in the shadow area.

In addition, the minutiae point detecting unit 220 can calculate a difference image for the out-of-vehicle area during nighttime driving. For example, the minutia matching point detector 220 may generate a difference image having a pixel value as a difference between pixels in a position on the out-of-vehicle area corresponding to a previous frame and a current frame of the image. The feature point detection unit 220 calculates an illumination region including pixels whose values obtained by applying a moving average recursive filter to a pixel value difference between consecutive frames are within a predetermined range from 0. That is, a pixel whose difference between pixel values between specific frames during nighttime driving continuously has a value similar to 0 is an illuminated area, and the minutia detection unit 220 may include the pixel in the illumination area.

Accordingly, the minutiae point detecting unit 220 can extract the shadow region during the daytime running and extract the illumination region during nighttime driving. At this time, the feature point detection unit 220 can calculate a straight line constituting the outline of the shadow region and the illumination region.

The vehicle area detecting unit 230 detects the vehicle area from the image. For example, the vehicle area detecting unit 230 may detect an average recursive filter (hereinafter referred to as an average recursive filter) for the difference between the previous frame of the current frame and the corresponding pixel of the current frame in all the photographing situations (during stoppage, during straight- Can be detected as the vehicle area. At this time, since the vehicle area detected by the vehicle area detecting unit 220 is detected through the difference value with respect to the image of all the photographing situations, it can be distinguished from the shadow area and the illuminating area which are detected by the image of the limited condition. The vehicle area detecting section 230 can calculate a straight line formed by the outline of the vehicle area.

The matrix calculation unit 240 calculates a homography matrix indicating a deviation between a shadow region, an illumination region, and a vehicle region of a neighboring image among the images. At this time, the matrix calculation unit 240 can calculate the homography matrix according to the following equation (1).

[Equation 1]

는 일 영상의 면 상에 위치하는 점을 의미하고,

는 일 영상과 인접한 타 영상의 면 상에 위치하는 점을 의미하고, s는 x축 및 y 축의 크기 변환 행렬이고, R은 3차원 회전 변환 행렬이고, T는 3차원 이동 변환 행렬이고,

는 호모그래피 행렬을 의미한다.At this time,

Means a point located on a plane of one image,

Where s is the x-axis and y-axis magnitude conversion matrix, R is the three-dimensional rotation transformation matrix, T is the three-dimensional motion transformation matrix,

Denotes a homography matrix.

At this time, the matrix calculating unit 240 receives information indicating a straight line (hereinafter, referred to as linear information) for each image from the reference line detecting unit 210, the minutiae point detecting unit 220 and the vehicle area detecting unit 230, It is possible to detect the three-dimensional surface formed by the above straight line. Accordingly, the matrix calculator 240 can detect the faces corresponding to the respective images, and calculate the homography matrix indicating the deviation between the faces of adjacent images according to Equation (1).

The matrix calculator 240 transmits the homography matrix to the lookup table corrector 270.

The vanishing point detecting unit 250 calculates the vanishing point of each image. At this time, the vanishing point detecting unit 250 can calculate the vanishing point of each image according to the known vanishing point calculating method. The vanishing point detector 250 generates a cumulative histogram by accumulating the positions of vanishing points on each frame of the image. The vanishing point detecting unit 250 transmits the cumulative histogram to the roll angle calculating unit 260.

The roll angle calculating unit 260 calculates the roll angle of each image from the cumulative histogram. At this time, when the camera is mounted on the vehicle horizontally in the photographing direction, the accumulated histogram can be expressed as the vanishing point is located on the horizontal line on the image when the vehicle rotates as shown in 410 of FIG. The line connecting the vanishing point can be a horizontal line. In this case, when the roll angle in the photographing direction of the camera is not 0 degrees (when the camera is turned on and mounted) as shown in Fig. 5, the line connecting the vanishing point according to the yaw motion of the vehicle is a diagonal line Can be formed. Also, when the roll angle in the photographing direction of the camera is not 0 degrees, a line formed by the vehicle as a pitch vanishing point due to the vehicle running on the road surface with irregularities may form a diagonal line instead of a vertical line. The roll angle calculating unit 260 calculates an angle between a line formed as a vanishing point in the yawing motion of the vehicle and an in-plane horizontal line in a cumulative histogram or an angle between a line passing through a vanishing point and a vertical line on the image, can do. The roll angle calculating section 260 transmits the roll angle to the matrix calculating section 240.

At this time, the matrix calculation unit 240 may calculate a homography matrix including a rotation matrix according to the roll angle. That is, the matrix calculation unit 240 includes two unknowns for S, which is the x-axis and y-axis magnitude conversion matrix of Equation (1), and includes three unknowns for the rotation transformation matrix for pitch, yaw and roll , and a homography matrix including three unknowns for the x, y, and z axis shift transformation matrices. At this time, since the component values for the rolls of the rotation conversion matrix are received from the roll angle calculation unit 260, a total of seven unknown values can be calculated. Therefore, the matrix calculation unit 240 calculates the homography matrix using the roll angle, thereby reducing the calculation complexity.

The lookup table corrector 270 corrects the lookup table according to the homography matrix and transmits the corrected lookup table to the 2D matching unit 130. [ For example, the lookup table correcting unit 270 may generate a 2D lookup table storing and matching coordinates obtained by applying a homography matrix transformation to each coordinate of an image. Accordingly, the surround view monitoring apparatus according to an embodiment of the present invention automatically corrects the 2D look-up table even when the shooting direction of the camera is changed due to an accident or the like, unlike the case of production of the vehicle, .

6 is a block diagram illustrating a 3D table correction unit of an apparatus for monitoring the surround view according to an exemplary embodiment of the present invention.

6, the 3D table correcting unit 140 of the surrounding view monitoring apparatus according to an embodiment of the present invention includes a road surface detecting unit 610, a fitting unit 620, a conversion area calculating unit 630, Gt; 640 < / RTI >

The road surface detection unit 610 receives the vehicle speed and the direction of travel from the vehicle. The road surface detecting unit 610 generates a predicted image that predicts a road surface change amount of the previous top view image according to the vehicle speed and the driving direction. That is, the road surface detector 610 can generate the predicted image by moving the pixel of the previous top view image by a distance corresponding to the vehicle speed in the driving direction. The road surface detection unit 610 generates a predicted difference image including a difference between each pixel value of the predicted image and each pixel value of the current top view image as each pixel value. The road surface detection unit 610 can detect, as a road surface area, pixels corresponding to a pixel value within a predetermined range from a value obtained by applying a moving average recursive filter to each pixel value of the predicted difference image. The road surface detection unit 610 transmits the road surface area information indicating the road surface area to the fitting unit 620.

The fitting portion 620 fits the road surface area information with an arbitrary shape (for example, an ellipse). For example, the fitting unit 620 can calculate a figure having the smallest sum of distances from each pixel of the outline of the road surface region among arbitrary figures (ellipses) positioned in the top view image. That is, the fitting unit 620 detects an arbitrary figure according to the Least Mean Square.

The conversion area calculating unit 630 selects an area corresponding to the outside of the figure calculated in the top view image as the conversion area.

The lookup table correcting unit 640 generates a 3D lookup table that matches and stores the position converted from the three-dimensional position of the top view image pixel located on the conversion area according to the predetermined curvature and the pixel position before the conversion, (150).

Therefore, the surrounding view monitoring apparatus according to the embodiment of the present invention can change the 3D lookup table according to the road surface detection pattern of the top view image, and can normally generate the 3D image even when the photographing direction of the camera is changed.

FIG. 7 is a flowchart illustrating a method of monitoring an ambient view by an ambient view monitoring apparatus according to an exemplary embodiment of the present invention. Hereinafter, each of the steps described below is referred to as a surrounding view monitoring apparatus for the process performed by one of the functional units included in the surrounding view monitoring apparatus, or the subject of each step for the sake of brevity and clear explanation of the invention.

Referring to FIG. 7, in step 710, the surrounding view monitoring apparatus receives an image from a plurality of cameras. For example, the surround view monitoring device can receive images from cameras installed on the front, rear, and side of the vehicle, respectively.

In step 720, the surround view monitoring apparatus detects reference lines, feature points, vehicle areas, and vanishing points from each image, and corrects the 2D lookup table according to reference lines, feature points, vehicle areas, and vanishing points. The detailed correction process of step 420 will be described in detail later with reference to FIG.

In step 730, the surrounding view monitoring apparatus generates a top view image by matching each image according to a 2D lookup table.

In step 740, the surrounding view monitoring apparatus detects the conversion area from the top view image and corrects the 3D lookup table according to the conversion area.

In step 750, the surrounding view monitoring apparatus matches the top view image according to the 3D lookup table to generate a 3D image.

FIG. 8 is a flowchart illustrating a process of correcting a 2D look-up table by an ambient view monitoring apparatus according to an embodiment of the present invention.

Referring to FIG. 8, in step 810, the surrounding view monitoring apparatus detects a reference line from each image. For example, a straight line with the smallest variance for pixels of a predetermined color in an image can be detected through a Least Mean Square.

In step 820, the surrounding view monitoring apparatus detects feature points from each image. For example, the surround view monitoring device determines whether the vehicle is traveling straight during the daytime running. At this time, the surrounding view monitoring apparatus can receive rotation information indicating the current steering rotation angle and the like from the vehicle during the daytime running to determine whether the vehicle is currently being straightened or rotated. The surround view monitoring apparatus calculates a difference image for the area outside the vehicle when the vehicle is currently traveling straight. The surround view monitoring apparatus can generate a difference image in which a pixel value is a difference between pixels in a position on the out-of-vehicle area corresponding to the previous frame and the current frame of the image. The surround view monitoring apparatus calculates a shadow area including pixels having a value obtained by applying a moving average recursive filter to a pixel value difference between successive frames within a predetermined range from 0. That is, a pixel whose difference between pixel values between specific frames during a daytime running continuously has a value similar to 0 is an area corresponding to a shadow, and the surrounding view monitoring apparatus can include the pixel in the shadow area. In addition, the surround view monitoring apparatus can calculate a difference image for an area outside the vehicle during night driving. For example, the surround view monitoring apparatus can generate a difference image in which a pixel value is a difference between pixels of a previous frame of the image and a corresponding position on the out-of-vehicle region of the current frame. The surround view monitoring apparatus calculates an illumination area including pixels whose values obtained by applying a moving average recursive filter to pixel difference between consecutive frames are within a predetermined range from 0. That is, a pixel whose difference between pixel values between specific frames during nighttime driving continuously has a value similar to 0 is an area where illumination is projected, and the surrounding view monitoring device can include the pixel in the illumination area. The surround view monitoring device can extract shadow areas during daytime driving and extract lighting areas during nighttime driving. At this time, the surrounding view monitoring apparatus can calculate a straight line constituting the outline of the shadow region and the illumination region.

In step 830, the surrounding view monitoring device detects the vehicle area from each image. For example, the surround view monitoring apparatus can detect, as a vehicle area, an area including pixels whose difference value between the previous frame and the current frame of the image is within a predetermined range from zero. The surround view monitoring apparatus can calculate a straight line formed by the outline of the vehicle area.

In step 840, the surround view monitoring apparatus detects a vanishing point from each image.

In step 850, the surrounding view monitoring apparatus stores a vanishing point corresponding to each frame of each image as a cumulative histogram, and calculates a roll angle from the cumulative histogram. For example, when the vehicle rotates, the cumulative histogram can be expressed as the vanishing point is located on the horizontal line on the image. At this time, the surround view monitoring apparatus can calculate the angle between the line formed by the vanishing point of the vehicle's yawing motion and the horizontal line on the screen or the angle between the line passing through the vanishing point and the vertical line on the image, .

In step 860, the surrounding view monitoring apparatus calculates a homography matrix between adjacent faces of the images. For example, the surround view monitoring device can detect three-dimensional surfaces formed by two or more straight lines according to at least one of reference lines, feature points, and vehicle regions for each image. The surrounding view monitoring apparatus can calculate a homography matrix representing a deviation between adjacent images in accordance with Equation (1).

In step 870, the surrounding view monitoring apparatus generates a lookup table according to the homography matrix. For example, the surrounding view monitoring apparatus can generate a 2D look-up table storing coordinates of coordinates of the image by matching the coordinates to which the homography matrix transformation is applied. Accordingly, the surround view monitoring apparatus according to an embodiment of the present invention automatically corrects the 2D look-up table even when the shooting direction of the camera is changed due to an accident or the like, unlike the case of production of the vehicle, .

9 is a diagram illustrating a process of generating the 3D lookup table by the surrounding view monitoring apparatus according to an embodiment of the present invention.

Referring to FIG. 9, in step 910, the surrounding view monitoring apparatus detects a road surface area from a top view image. For example, the surrounding view monitoring apparatus generates a predicted image that predicts the amount of road surface change of the previous top view image according to the vehicle speed and the driving direction. That is, the road surface detector 610 can generate the predicted image by moving the pixel of the previous top view image by a distance corresponding to the vehicle speed in the driving direction. The surround view monitoring apparatus generates a prediction difference image including a difference between each pixel value of the prediction image and each pixel value of the current top view image as each pixel value. The surround view monitoring apparatus can detect a pixel corresponding to a pixel value within a predetermined range from a value obtained by applying a moving average recursive filter to each pixel value of the predicted difference image as a road surface region.

In step 920, the ambient view monitoring device fits the road surface area and the geometry. For example, the surrounding view monitoring apparatus can calculate a figure having the smallest sum of distances from each pixel of the outline of the road surface region among arbitrary figures (for example, ellipses) located in the top view image. That is, the fitting unit 620 can detect a figure according to the Least Mean Square.

In step 930, the surrounding view monitoring apparatus sets the outer region of the fitted figure of the top view image as the transform region.

In step 940, the surrounding view monitoring apparatus generates a 3D lookup table storing and storing the position converted from the 3D position of the top view image pixel located on the conversion area according to the predetermined curvature and the pixel position before conversion.

Therefore, the surrounding view monitoring apparatus according to the embodiment of the present invention can change the 3D lookup table according to the road surface detection pattern of the top view image, and can normally generate the 3D image even when the photographing direction of the camera is changed.

The present invention has been described above with reference to the embodiments thereof. Many embodiments other than the above-described embodiments are within the scope of the claims of the present invention. It will be understood by those skilled in the art that various changes in form and details may be made therein without departing from the spirit and scope of the invention as defined by the appended claims. The disclosed embodiments should, therefore, be considered in an illustrative rather than a restrictive sense. The scope of the present invention is defined by the appended claims rather than by the foregoing description, and all differences within the scope of equivalents thereof should be construed as being included in the present invention.

Claims (18)

An image receiving unit for receiving images from a plurality of cameras, respectively;
Detecting a plurality of straight lines corresponding to at least one of a reference line, a minutiae, and a vehicle area from each of the images, calculating a homography matrix between surfaces of the adjacent images including the straight line, A 2D table correcting unit for generating a 2D lookup table according to the 2D lookup table; And
A 2D matching unit for matching the images according to the 2D look-up table to generate a top view image;
The monitoring device comprising:
The method according to claim 1,
Wherein the 2D table correction unit comprises:
A reference line detection unit for detecting the reference line from the image;
A feature point detection unit which detects a shadow region and an illumination region corresponding to the feature point from the image and calculates a straight line constituting the shadow region and the outline of the illumination region;
A vehicle area detecting unit for detecting a vehicle area from the image and calculating a straight line constituting an outline of the vehicle area;
A matrix calculation unit for calculating a homography matrix between surfaces including a plurality of straight lines along at least one of the reference line, the minutiae, and the vehicle area; And
A lookup table correcting unit for generating the 2D lookup table according to the homography matrix;
And a monitoring unit for monitoring the surrounding view.
3. The method of claim 2,
Wherein the reference line is a line that is projected in a predetermined direction in the vehicle and is reflected by the external environment and appears in the image captured by the camera.
3. The method of claim 2,
The special point detecting unit
A moving average recursive filter is applied to each pixel value of the difference image, and the pixel value of the pixel value to which the moving average recursive filter is applied Calculates the shadow area including pixels within a predetermined range from 0,
A moving average recursive filter is applied to each pixel value of the difference image, and the pixel value to which the moving average recursive filter is applied is changed from 0 to And calculates the illumination area including the pixels within the specified range.
3. The method of claim 2,
Wherein the vehicle-
Wherein the area detection unit detects, as the vehicle area, an area including a pixel whose value obtained by applying a moving average recursive filter to a difference between corresponding pixels of a previous frame and a current frame of the image is within a predetermined range from 0, Device.
The method according to claim 1,
A 3D table correcting unit for detecting a transform region from the top view image and generating a 3D lookup table for performing transform on the transform region; And
A 3D matching unit for matching the top view images according to the 3D lookup table to generate a 3D image,
Further comprising: means for monitoring the surrounding view.
The method according to claim 6,
Wherein the 3D table correcting unit comprises:
A road surface detecting unit for detecting a road surface area from the top view image;
A fitting part for detecting a figure to be fitted to the road surface area;
A conversion area calculating unit for selecting, as the conversion area, an area corresponding to the outside of the figure from among the top view images; And
A lookup table correcting unit for generating a 3D lookup table for changing a position of a pixel in the conversion area according to a specified curvature;
The monitoring device comprising:
8. The method of claim 7,
The road-
The predicted image is generated by moving the pixels of the previous top view image by a distance corresponding to the vehicle speed in the traveling direction,
Generating a prediction difference image including a difference between each pixel value of the prediction image and each pixel value of the current top view image as each pixel value,
Wherein a pixel corresponding to a value obtained by applying a moving average recursive filter to each pixel value of the predicted difference image is within a predetermined range from 0 to the road surface area.
8. The method of claim 7,
Wherein the fitting unit detects an image having a smallest sum of distances from pixels of the outline of the road surface area on the top view image.
A method of providing an overview view monitoring device, the method comprising:
Receiving images from a plurality of cameras, respectively;
Detecting a plurality of straight lines corresponding to at least one of a reference line, a feature point, and a vehicle area from the image;
Calculating a homography matrix between planes of the adjacent images including the straight line;
Generating a 2D look-up table according to the homography matrix; And
Generating a top view image by matching the images according to the 2D look-up table;
The method comprising the steps of:
11. The method of claim 10,
The step of detecting a plurality of straight lines corresponding to at least one of the reference line, the feature point and the vehicle area from the image
Detecting the reference line from the image;
Detecting a shadow region and an illumination region corresponding to the feature point from the image and calculating a straight line constituting the shadow region and an outline of the illumination region; And
Detecting a vehicle area from the image and calculating a straight line constituting an outline of the vehicle area;
And monitoring the surrounding view.
12. The method of claim 11,
Wherein the reference line is a line that is projected in a predetermined direction in the vehicle and is reflected by the external environment and appears in the image captured by the camera.
12. The method of claim 11,
Detecting a shadow region and an illumination region corresponding to a feature point from the image, and calculating a straight line constituting the shadow region and an outline of the illumination region,
A moving average recursive filter is applied to each pixel value of the difference image, and the pixel value of the pixel value to which the moving average recursive filter is applied Calculating the shadow area including pixels within a predetermined range from 0;
A moving average recursive filter is applied to each pixel value of the difference image, and the pixel value to which the moving average recursive filter is applied is changed from 0 to Calculating the illumination area including pixels within a specified range; And
Detecting a straight line corresponding to the shadow region and the outline of the illumination region;
And monitoring the surrounding view.
12. The method of claim 11,
Wherein the step of detecting a vehicle area from the image and calculating a straight line constituting an outline of the vehicle area comprises:
And detecting, as the vehicle area, an area including a pixel whose value obtained by applying a moving average recursive filter to a difference value between corresponding pixels of a previous frame and a current frame of the image is within a predetermined range from 0, View monitoring method.
11. The method of claim 10,
Detecting a transform region from the top view image and generating a 3D lookup table for performing a transform on the transform region; And
Generating 3D images by matching the top view images according to the 3D lookup table
Further comprising the steps of:
16. The method of claim 15,
The step of detecting a transform region from the top view image and generating a 3D lookup table for performing a transform on the transform region,
Detecting a road surface region from the top view image;
Detecting a figure to be fitted to the road surface area;
Selecting an area of the top view image corresponding to the outside of the figure as the conversion area; And
Generating a 3D lookup table that causes a position of a pixel in the transformation region to change according to a specified curvature;
The method comprising the steps of:
17. The method of claim 16,
The step of detecting a road surface area from the top view image may include:
Generating a prediction image by moving a pixel of the previous top view image by a distance corresponding to a vehicle speed in a traveling direction;
Generating a prediction difference image including a difference between pixel values of the prediction image and pixel values of a current top view image as pixel values; And
Detecting a pixel corresponding to a pixel value of the predicted difference image by applying a moving average recursive filter within a predetermined range from 0 to the road surface region;
And monitoring the surrounding view.
17. The method of claim 16,
Wherein the step of detecting a figure to be fitted to the road surface area comprises:
Detecting an image having a smallest sum of distances from pixels of an outline of the road surface area on the top view image.

Images