KR102089343B1 - Around view monitoring system and calibration method for around view cameras - Google Patents

Abstract

Around view monitoring system and camera tolerance correction method are disclosed. The around view monitoring system includes a camera unit including a plurality of cameras mounted to photograph the front, right, left, and rear of the vehicle, respectively; A tolerance correction unit that determines whether or not a distortion according to a predetermined criterion exists in an individual top view image corresponding to the captured image captured by each camera, and updates a conversion rule for converting the captured image into an individual top view image if the distortion exists; And an image synthesizing unit generating an AVM image by using the updated conversion rule and a plurality of captured images captured by the camera unit.

Description

Around view monitoring system and calibration method for around view cameras}

The present invention relates to an around view monitoring system and a camera tolerance correction method.

In general, the driver who boards the inside of the vehicle mainly looks at the front, and since the left and right and rear clocks of the driver are largely obscured by the vehicle body, the driver is forced to have a very limited clock.

To solve this problem, clock assisting means such as a room mirror and a side mirror are used, and recently, technologies including camera means for photographing an external image of a vehicle and providing it to the driver are being applied to the vehicle.

Among them, there is an Around View Monitoring system (hereinafter referred to as an AVM system) that installs a plurality of cameras in a vehicle and shows a 360-degree omnidirectional image around the vehicle.

The AVM system combines the images around the vehicle captured by the cameras provided at each location of the vehicle, and provides AVM images in the form of a Top View image that looks like the driver is looking at the vehicle from the sky, thereby preventing obstacles around the vehicle. It has the advantage of being able to mark and eliminate blind spots.

In the AVM system, it is essential to correct the tolerance of each camera when the camera is mounted, and the vehicle with the AVM system is shipped after the tolerance is corrected to meet the screen conformance criteria for the generation of the AVM image.

However, if the tolerances accumulate during the operation of the vehicle after the vehicle is shipped, the screen consistency is lowered. In this case, in order to correct the accumulated tolerance, the driver had the inconvenience of having to visit a service center or a business office with a vehicle.

Korean Registered Patent No. 10-0948886 (tolerance correction device and method for camera installed in a vehicle)

The present invention is performed by automatically calibrating tolerances while driving a vehicle without having to visit a service center or a business site with the vehicle even when a cumulative tolerance occurs for various reasons such as an angle change of the AVM camera or a change in tire pressure after the vehicle is shipped. To maximize customer convenience and to provide an around view monitoring system and camera tolerance correction method that prevents the risk of accidents caused by uncorrected AVM images.

In the present invention, since tolerance correction is performed for cameras for the AVM system while the vehicle is moving, not only the AVM system in which tolerance correction for each camera is not performed before the vehicle is shipped, but also due to the accumulation of tolerances after the vehicle is shipped. It is to provide an around view monitoring system and a camera tolerance compensation method that can be widely used in AVM systems that require tolerance compensation.

Objects other than the present invention will be easily understood through the following description.

According to an aspect of the present invention, a camera unit including a plurality of cameras mounted to photograph each of the front, right, left, and rear of a vehicle; Tolerances for determining whether or not there is a distortion according to a predetermined criterion in an individual top view image corresponding to a captured image taken by each camera, and updating the conversion rule for converting the captured image to a corresponding individual top view image if there is a distortion Correction unit; And an image synthesizing unit that generates an AVM image by using the updated conversion rule and a plurality of captured images generated by the camera unit.

The tolerance correction unit includes: an image conversion unit that converts each of a plurality of captured images provided by the camera unit into one or more of a peripheral image and an individual top view image according to a predetermined format; An information recognition unit for recognizing a target object including at least a lane by analyzing a target image of at least one of a captured image, a surrounding image, and an individual top view image; A distortion detection unit to determine whether each individual top-view image is distorted by applying a predetermined distortion detection technique to each individual top-view image; For each individual top view image determined to be distorted, positions of each of a plurality of predefined image registration reference points are moved within a predetermined area range in a predetermined pixel unit, and based on each of the moved positions in a predetermined pixel unit. An image correction unit configured to detect location information of an image registration reference point having a minimum calculated value among the calculated values calculated by the distortion detection unit according to the application of the distortion detection method; And a conversion rule update unit that updates the conversion rule to correspond to the detected position information of the matched reference point.

을 이용하여 각 개별 탑뷰 영상의 틀어짐 추정값(G)인 산출값을 산출하고, 산출된 산출값이 미리 지정된 기준값을 초과하면 틀어짐이 존재하는 것으로 판단할 수 있다. 여기서,

이고, V_F는 전방의 개별 탑뷰 영상이며, V_R은 우측의 개별 탑뷰 영상이며, V_L은 좌측의 개별 탑뷰 영상이고, V_B는 후방의 개별 탑뷰 영상이고, μ_α는 어느 하나의 개별 탑뷰 영상인 α 영상에서의 평균 차선 각도(Radian)일 수 있다.The predetermined distortion detection technique is an equation

It is possible to calculate the calculated value, which is the distortion estimation value G of each individual top-view image, and determine that the distortion exists when the calculated calculated value exceeds a predetermined reference value. here,

, V _F is an individual top view image at the front, V _R is an individual top view image at the right, V _L is an individual top view image at the left, V _B is an individual top view image at the rear, μ _α is any one individual top view image It may be an average lane angle (Radian) in the image α image.

을 이용하여 AVM 영상을 생성하기 위해 각 개별 탑뷰 영상이 겹쳐지는 블렌딩 영역 각각에서의 픽셀값 전체 차이(D)인 산출값을 산출하고, 산출된 산출값이 미리 지정된 기준값을 초과하면 틀어짐이 존재하는 것으로 판단할 수 있다. 여기서, λ₁과 λ₂는 합산값이 1을 만족하도록 각각 지정된 가중치 팩터이고, B는 블렌딩 영역의 좌표 범위이고, β는 블렌딩 영역의 좌표 범위 중 어느 한 위치이며, p_β는 블렌딩 영역을 형성하는 제1 개별 탑뷰 영상의 블렌딩 영역 내의 β위치의 픽셀값이고, q_β는 블렌딩 영역을 형성하는 제2 개별 탑뷰 영상의 블렌딩 영역 내의 β위치의 픽셀값이며, δ는 블렌딩 영역의 좌표 범위 중에서 상기 정보 인식부에서 인식한 주변 사물 중 미리 지정된 임계값 이상의 높이를 가지는 주변 물체가 지면과 맞닿는 위치이며, p_δ는 블렌딩 영역을 형성하는 제1 개별 탑뷰 영상의 블렌딩 영역 내의 δ위치의 픽셀값이고, q_δ는 블렌딩 영역을 형성하는 제2 개별 탑뷰 영상의 블렌딩 영역 내의 δ위치의 픽셀값일 수 있다.The predetermined distortion detection technique is an equation

In order to generate an AVM image, a calculated value that is the total difference (D) of the pixel values in each blending area where each individual top-view image overlaps is calculated, and if the calculated calculated value exceeds a predetermined reference value, there is a distortion. You can judge that. Here, λ ₁ and λ ₂ are weight factors respectively designated so that the sum value satisfies 1, B is a coordinate range of the blending region, β is any one of the coordinate ranges of the blending region, and p _β forms a blending region. Is a pixel value at a β position in the blending region of the first individual top view image, q _β is a pixel value at a β position in the blending region of the second individual top view image forming the blending region, and δ is among the coordinate ranges of the blending region Of the surrounding objects recognized by the information recognition unit, a surrounding object having a height equal to or greater than a predetermined threshold value is in contact with the ground, and p _δ is a pixel value at the δ position in the blending area of the first individual top-view image forming the blending area, q _δ may be a pixel value of the δ position in the blending region of the second individual top-view image forming the blending region.

,

및

을 이용하여 개별 탑뷰 영상의 틀어짐 정도(T)인 산출값을 산출하고, 산출된 산출값이 미리 지정된 기준값을 초과하면 틀어짐이 존재하는 것으로 판단할 수 있다. The predetermined distortion detection technique is an equation

,

And

By calculating the calculated value that is the degree of distortion (T) of the individual top view images, and if the calculated calculated value exceeds a predetermined reference value, it can be determined that the distortion exists.

이고, V_F는 전방의 개별 탑뷰 영상이며, V_R은 우측의 개별 탑뷰 영상이며, V_L은 좌측의 개별 탑뷰 영상이고, V_B는 후방의 개별 탑뷰 영상이고, μ_α는 어느 하나의 개별 탑뷰 영상인 α 영상에서의 평균 차선 각도(Radian)이며, D는 AVM 영상을 생성하기 위해 각 개별 탑뷰 영상이 겹쳐지는 블렌딩 영역 각각에서의 픽셀값 전체 차이이고, λ₁과 λ₂는 합산값이 1을 만족하도록 각각 지정된 가중치 팩터이고, B는 블렌딩 영역의 좌표 범위이고, β는 블렌딩 영역의 좌표 범위 중 어느 한 위치이며, p_β는 블렌딩 영역을 형성하는 제1 개별 탑뷰 영상의 블렌딩 영역 내의 β위치의 픽셀값이고, q_β는 블렌딩 영역을 형성하는 제2 개별 탑뷰 영상의 블렌딩 영역 내의 β위치의 픽셀값이며, δ는 블렌딩 영역의 좌표 범위 중에서 상기 정보 인식부에서 인식한 주변 사물 중 미리 지정된 임계값 이상의 높이를 가지는 주변 물체가 지면과 맞닿는 위치이며, p_δ는 블렌딩 영역을 형성하는 제1 개별 탑뷰 영상의 블렌딩 영역 내의 δ위치의 픽셀값이고, q_δ는 블렌딩 영역을 형성하는 제2 개별 탑뷰 영상의 블렌딩 영역 내의 δ위치의 픽셀값일 수 있다.Here, ω ₁ and ω ₂ are weight factors respectively designated so that the sum value satisfies 1, and G is a distortion estimation value of each individual top view image,

, V _F is an individual top view image at the front, V _R is an individual top view image at the right, V _L is an individual top view image at the left, V _B is an individual top view image at the rear, μ _α is any one individual top view image The average lane angle (Radian) in the α image, which is the image, D is the total difference of pixel values in each blending area where each individual top-view image overlaps to generate an AVM image, and λ ₁ and λ ₂ are sums of 1 , Respectively, B is a coordinate range of the blending region, β is any one of the coordinate ranges of the blending region, and p _β is a β location in the blending region of the first individual top-view image forming the blending region. and a pixel value, _β q is the pixel value of the β position in the blending area of the second individual top view image to form a blended region, δ is the peripheral four recognized by the information identification portion in the coordinate range of the blending area In a position close to the object in contact with the ground with the advance over the specified threshold height, p _δ is the pixel value of the δ position in the blending area of the first individual top view image to form a blended region, q _δ is to form a blended region It may be a pixel value at the δ position in the blending region of the second individual top view image.

The information recognition unit may recognize the lane by classifying a traveling direction and a Hough component vector of the vehicle as a feature vector and classifying them using a predetermined classifier.

The information recognition unit further recognizes at least one of traffic lights, pedestrians, road floor display information, and surrounding objects in the target image, and the recognized location information of the surrounding objects uses the plurality of individual top view images to view the AVM image. It can be used as reference information to generate.

The information recognizing unit may limit a recognition object to a recognition object whose reliability is evaluated to satisfy a predetermined criterion among the surrounding objects by applying a predetermined machine learning technique.

The image correction unit may move the image registration reference point by applying at least one of a predetermined optimization technique and a cost function for finding a position of a corresponding point in an individual top-view image in which a difference between a lane angle and a pixel value in a blending region is minimum. .

According to another aspect of the present invention, a computer program stored in a computer-readable medium for correcting camera tolerance, the computer program causing the computer to perform the following steps, the steps comprising: a plurality of camera units Converting each of the captured images generated by the cameras of the camera into one or more of a peripheral image and an individual top view image according to a predefined format; Recognizing a surrounding object including at least a lane by analyzing a target image of at least one of a captured image, a surrounding image, and an individual top view image; Determining whether each individual top view image is distorted by applying a predetermined distortion detection method to each individual top view image; For each individual top view image determined to be distorted, positions of each of a plurality of predefined image registration reference points are moved within a predetermined area range in a predetermined pixel unit, and based on each of the moved positions in a predetermined pixel unit. Detecting position information of an image registration reference point having a minimum calculated value among calculated values calculated according to the application of the distortion detection technique; And updating a conversion rule for generating an AVM image by using a plurality of captured images, so as to correspond to the detected location information of the matched reference point.

,

,

을 이용하여 개별 탑뷰 영상의 틀어짐 정도(T)인 산출값을 산출하고, 산출된 산출값이 미리 지정된 기준값을 초과하면 틀어짐이 존재하는 것으로 판단할 수 있다. The distortion detection technique is an equation

,

,

By calculating the calculated value that is the degree of distortion (T) of the individual top view images, and if the calculated calculated value exceeds a predetermined reference value, it can be determined that the distortion exists.

이고, V_F는 전방의 개별 탑뷰 영상이며, V_R은 우측의 개별 탑뷰 영상이며, V_L은 좌측의 개별 탑뷰 영상이고, V_B는 후방의 개별 탑뷰 영상이고, μ_α는 어느 하나의 개별 탑뷰 영상인 α 영상에서의 평균 차선 각도(Radian)이며, D는 AVM 영상을 생성하기 위해 각 개별 탑뷰 영상이 겹쳐지는 블렌딩 영역 각각에서의 픽셀값 전체 차이이고, λ₁과 λ₂는 합산값이 1을 만족하도록 각각 지정된 가중치 팩터이고, B는 블렌딩 영역의 좌표 범위이고, β는 블렌딩 영역의 좌표 범위 중 어느 한 위치이며, p_β는 블렌딩 영역을 형성하는 제1 개별 탑뷰 영상의 블렌딩 영역 내의 β위치의 픽셀값이고, q_β는 블렌딩 영역을 형성하는 제2 개별 탑뷰 영상의 블렌딩 영역 내의 β위치의 픽셀값이며, δ는 블렌딩 영역의 좌표 범위 중에서 상기 인식된 주변 사물 중 미리 지정된 임계값 이상의 높이를 가지는 주변 물체가 지면과 맞닿는 위치이며, p_δ는 블렌딩 영역을 형성하는 제1 개별 탑뷰 영상의 블렌딩 영역 내의 δ위치의 픽셀값이고, q_δ는 블렌딩 영역을 형성하는 제2 개별 탑뷰 영상의 블렌딩 영역 내의 δ위치의 픽셀값일 수 있다.Here, ω ₁ and ω ₂ are weight factors respectively designated so that the sum value satisfies 1, and G is a distortion estimation value of each individual top view image,

, V _F is an individual top view image at the front, V _R is an individual top view image at the right, V _L is an individual top view image at the left, V _B is an individual top view image at the rear, μ _α is any one individual top view image The average lane angle (Radian) in the α image, which is the image, D is the total difference of pixel values in each blending area where each individual top-view image overlaps to generate an AVM image, and λ ₁ and λ ₂ are sums of 1 , Respectively, B is a coordinate range of the blending region, β is any one of the coordinate ranges of the blending region, and p _β is a β location in the blending region of the first individual top-view image forming the blending region. Is a pixel value of, q _β is a pixel value of a β position in a blending region of a second individual top-view image forming a blending region, and δ is pre-designated among the recognized peripheral objects among coordinate ranges of the blending region. A position where a peripheral object having a height above a threshold is in contact with the ground, p _δ is a pixel value at a position _δ in the blending area of the first individual top-view image forming a blending area, and q _δ is a second individual forming a blending area It may be a pixel value of δ in the blending area of the top view image.

Other aspects, features, and advantages than those described above will become apparent from the following drawings, claims, and detailed description of the invention.

According to an embodiment of the present invention, even when a cumulative tolerance occurs for various reasons, such as a change in the angle of mounting the AVM camera or a change in tire pressure after leaving the vehicle, the vehicle is automatically operated while driving without having to visit a service center or a business office. By performing tolerance correction, there is an effect of maximizing customer convenience and preventing the risk of accidents due to AVM images that have not been corrected.

In addition, since the tolerance correction for the cameras for the AVM system is performed while the vehicle is moving, the tolerance correction is performed due to the accumulation of tolerances after the vehicle is shipped, as well as the AVM system in which tolerance correction for each camera is not performed before the vehicle is shipped. There is also an effect that can be universally used in the required AVM system.

1 is a view for explaining a tolerance correction method according to the prior art.
2 is a block diagram of an AVM system according to an embodiment of the present invention.
3 to 8 are views for explaining the tolerance correction operation of the AVM system according to an embodiment of the present invention.
9 is a flowchart illustrating a camera tolerance correction method of an AVM system according to an embodiment of the present invention.

The present invention can be applied to various changes and may have various embodiments, and specific embodiments will be illustrated in the drawings and described in detail in the detailed description. However, this is not intended to limit the present invention to a specific embodiment, it should be understood to include all modifications, equivalents, and substitutes included in the spirit and scope of the present invention.

When an element is said to be "connected" or "connected" to another component, it is understood that other components may be directly connected to or connected to the other component, but other components may exist in the middle. It should be. On the other hand, when a component is said to be "directly connected" or "directly connected" to another component, it should be understood that no other component exists in the middle.

Terms such as first and second may be used to describe various components, but the components should not be limited by the terms. The terms are used only for the purpose of distinguishing one component from other components.

The terms used in this specification are only used to describe specific embodiments, and are not intended to limit the present invention. Singular expressions include plural expressions unless the context clearly indicates otherwise. In this specification, terms such as “include” or “have” are intended to indicate that a feature, number, step, operation, component, part, or combination thereof described in the specification exists, and that one or more other features are present. It should be understood that the existence or addition possibilities of fields or numbers, steps, operations, components, parts or combinations thereof are not excluded in advance.

In addition, in the description with reference to the accompanying drawings, the same components, regardless of reference numerals, are assigned the same or related reference numerals, and redundant description thereof will be omitted. In the description of the present invention, when it is determined that detailed descriptions of related known technologies may unnecessarily obscure the subject matter of the present invention, detailed descriptions thereof will be omitted.

1 is a view for explaining a tolerance correction method according to the prior art.

As illustrated in FIG. 1, the cameras 10, 20, 30, and 40 are mounted on the front, right, left, and rear positions of the vehicle, respectively, and the vehicle is installed in a place where the correction plate 50 is installed for tolerance correction. It is placed at a predetermined location.

The compensator 50 may be disposed in front left corners, front right corners, rear left corners, and rear right corners of the vehicle to be spaced a predetermined distance from each wheel axis of the vehicle.

The correction plate 50 may be formed of a grid-shaped pattern to prevent an extraction error of a feature point, and the grid-shaped pattern is a combination of colors having strong color contrast.

For tolerance correction, coordinate information of the feature points in the correction plate 50 is extracted from the captured image generated by each camera 10, 20, 30, 40, and the coordinate information is compared and converted to standard coordinate information corresponding to each feature point The matrix is calculated. At this time, for example, a lookup table may be generated by applying a correction algorithm, an affine transform algorithm, a homography transform algorithm, and a viewpoint transform algorithm.

The generated transformation rules (ie, transformation matrix and / or lookup table) are stored in the storage unit and used to generate the AVM image.

However, in the above-described method, the conversion rule generated before the vehicle is released or generated in a service center may generate a cumulative tolerance, such as a change in the camera mounting angle, in the course of driving the vehicle. In this case, visit the service center or office again to allow the tolerance. There is the inconvenience of having to perform the calibration again.

2 is a block diagram of an AVM system according to an embodiment of the present invention, and FIGS. 3 to 8 are views for explaining a tolerance correction operation of the AVM system according to an embodiment of the present invention.

Referring to FIG. 2, the AVM system may include a camera unit 210, a tolerance correction unit 220, an image synthesis unit 230, and an output unit 240.

The camera unit 200 may include four or more cameras for AVM systems. The cameras can be installed at predetermined positions such as front, rear, left and right of the vehicle, respectively. The cameras may be, for example, a wide-angle lens, a fish-eye lens, or the like, with a large angle of view. In addition, the camera unit 200 may further include a narrow-angle camera that is installed at a predetermined location, such as a front or rear of the vehicle, to photograph a long distance image.

Each camera generates a two-dimensional photographed image (refer to (a) of FIG. 3) by photographing a three-dimensional object corresponding to a designated angle of view and an installation angle, and the generated photographed image is a tolerance correction unit 220 and / or It is provided to the image synthesis unit 230. The captured image generated by each camera is stored in a storage unit (not shown), and the tolerance correction unit 220 may be configured to access the storage unit and use the captured image.

To allow the tolerance correction unit 220 to synthesize a plurality of captured images and generate an AVM image (see (d) of FIG. 3) conforming to a screen matching standard, in the cameras included in the camera unit 200 Each individual top-view images are analyzed by analyzing whether the individual top-view images (refer to FIG. 3 (c), which are individual top-view images corresponding to the captured images taken on the right side of the vehicle) are distorted to correspond to the generated shot images. The conversion rules corresponding to each camera are generated to be synthesized into AVM images meeting the standards.

Here, the conversion rule is generated to convert each shot image into a separate top view image based on a transformation matrix and / or a distortion correction algorithm and a transformation matrix of each camera to convert each shot image into an individual top view image that is a top view image. It may be a look-up table (LUT).

The conversion rule generated by the tolerance correction unit 220 is stored in a storage unit (not shown), and the image synthesis unit 230 synthesizes a plurality of captured images provided by the camera unit 200 (for example, an overlay method, etc.) ) To generate an AVM image of a top-view point of view around the vehicle. The AVM image generated by the image synthesizing unit 230 may be output through the output unit 230, for example, a display device.

Specifically, the tolerance correction unit 220 may include an image conversion unit 221, an information recognition unit 223, a distortion detection unit 225, an image correction unit 227, and a conversion rule update unit 229.

The image converter 221 generates individual top-view images (see FIG. 3 (c)) corresponding to the captured images (see (a) of FIG. 3) provided by each camera of the camera unit 210, respectively. do. The image converting unit 221 further converts the photographed image photographed using a wide-angle lens or the like into a peripheral image having a shape of an image photographed by a normal standard lens by correcting distortion of an edge region (see FIG. 3 (b)). You can also convert.

The information recognizing unit 223 recognizes a captured surrounding object by analyzing a target image of at least one of a captured image, a surrounding image, and an individual top view image. The surrounding objects recognized by the information recognizing unit 223 include, for example, vanishing points, roads, lanes, traffic lights, and road marking information (for example, pedestrian crossings, parking lines, etc.), pedestrians, and surrounding vehicles. And more.

The information recognizing unit 223 uses predetermined recognition techniques (eg, Hough Transform, Fourier Transform, and cross ratio) to recognize surrounding objects in each target image corresponding to the camera. Method, one or more of methods using color information, and the like).

The information recognizing unit 223 may recognize, for example, a vanishing point that is a point where parallel straight lines meet by using lanes, sidewalk blocks around a vehicle, and the like in the target image.

As illustrated in FIG. 4, in the perspective view, all parallel straight lines meet at one point, so the information recognition unit 223 is a straight line element present in the perspective image, which is a captured image and / or a surrounding image. By extracting them, the vanishing point, which is the point where the most straight lines cross, can be detected using, for example, the RANSAC (RANdom SAample Consensus) algorithm, the Mean Square Error (MSE) method, and the like.

In addition, the information recognition unit 223 detects the vanishing point, and then, for example, configures a vehicle's traveling direction, a Hough component vector, etc. as a feature vector and classifies it using a classifier such as a support vector machine (SVM). It can be set to recognize the lane more accurately than the method of recognizing the lane by using only a straight line component on the road.

The information recognition unit 223 may be set to detect a crosswalk using, for example, a binarization method for each region, a method using an x-axis histogram, a morphology dilation operation, a feature vector for crosswalk, and SVM machine learning. Also, surrounding objects may be recognized by methods such as outline extraction and color analysis of objects included in the image.

In order to improve accuracy in recognizing surrounding objects such as pedestrian crossings, parking lines, various symbols displayed on the floor, pedestrians, etc., the information recognition unit 223 includes a predetermined machine learning technique (for example, Adaboost, A shallow learning technique such as a support vector machine (SVM) or a deep learning technique such as a convolution neural network (CNN) may be applied.

Peripheral objects other than the lane may be set in advance to select, as a recognition target, only peripheral objects having high reliability according to a predetermined criterion among peripheral objects recognized using a machine learning technique.

The location information of surrounding objects such as pedestrians and pedestrian crossings recognized by the information recognizing unit 223 may be used as reference information when matching each individual top view image to an AVM image.

The distortion detection unit 225 determines whether surrounding objects (eg, lanes, etc.) included in an individual top view image corresponding to each camera included in the camera unit 210 exist in a wrong shape. In the case of surrounding objects in a distorted form in an individual top view image, it may be caused by various reasons, for example, the camera mounting angle is changed or the tire air pressure is changed.

5, individual top view images 510, 512, 514, and 516 corresponding to each camera of the camera unit 210 are illustrated. When referring to each individual top view image (510, 512, 514, 516), the shape of the lane shown in the front individual top view image (510) is consistent with the shape of the lane shown in other individual top view images (512, 514, 516). You can see that you can't go. In this case, it may be determined that the individual top view image 510 in the front is wrong.

In the distortion detection unit 225, the degree of distortion of the individual top-view images is, for example, the angular deviation of the lanes recognized in the individual top-view images, and the overall difference in pixel values of the blending regions in which the individual top-view images overlap to generate an AVM image. It can be recognized using one or more of the.

As a first technique, if the average angle of the lanes extracted from each individual top view image of V _F (front), V _R (right), V _L (left) and V _B (rear) is μ _α , the distortion detection unit 225 ) Can calculate the distortion estimate value (G) of each individual top-view image by Equation 1 below.

이고, μ_α는 어느 하나의 개별 탑뷰 영상인 α 영상에서의 평균 차선 각도(Radian)이다.here,

, And μ _α is the average lane angle (Radian) in one individual top-view image, α image.

According to Equation 1, the sum of the deviation between the angle of the lane recognized in the individual top view image and 90 degrees (π / 2) is calculated. At this time, only lanes determined by the information recognition unit 223 to have a confidence level greater than or equal to a predetermined threshold may be selected as a calculation target.

Therefore, in the case where there are multiple lanes distributed in various directions in the individual top view images, only the lanes in a similar direction are selected according to a current vehicle's traveling direction and a predetermined criterion, and an average lane angle in the corresponding individual top view images is calculated. In addition, a distortion estimation value of an individual top view image may be calculated using the same.

As a second technique, the distortion detection unit 225 calculates the overall difference (D) of the pixel values in each of the blending regions (see 610 to 640 of FIG. 6) where each individual top-view image overlaps to generate an AVM image. Equation 2 can also calculate the degree of distortion of each individual top view image. Here, the pixel value may be, for example, a pixel brightness value.

Here, λ ₁ and λ ₂ are weight factors, and may be respectively set so that the sum of λ ₁ and λ ₂ satisfies 1. B is a coordinate range of the blending region, β is one of the coordinate ranges of the blending region, p _β is a pixel value at the β location in the blending region of the first individual top-view image forming the blending region, and q _β is the blending region. It may be a pixel value at a β position in the blending region of the second individual top-view image forming the region. In addition, δ is a position where a surrounding object (for example, a pedestrian, a surrounding vehicle, etc.) having a height equal to or higher than a predetermined threshold among the surrounding objects recognized by the information recognition unit 223 among the coordinate ranges of the blending area, is in contact with the ground, p _δ may be a pixel value at a δ position in the blending region of the first individual top-view image forming the blending region, and q _δ may be a pixel value at δ position in the blending region of the second individual top-view image forming the blending region.

The weight factor λ ₁ is a weight factor applied to the sum of pixel value differences of each pixel belonging to the blending area, and the weight factor λ ₂ is a position in contact with the ground of a surrounding object having a height above a threshold among pixels belonging to the blending area. Is a weight factor applied to the sum of the pixel value differences of the pixels of.

Accordingly, as the weight factor λ ₂ is set to a relatively large value within the range of the minimum value 0 to the maximum value 1, an object having a height (pedestrian, vehicle, etc.) among surrounding objects recognized by the information recognition unit 223 is increased. More importantly, the degree of distortion of individual top view images can be calculated. As described above, considering that the location information of the surrounding objects can be used as a reference location when generating the AVM image, the size setting of the weight factor λ ₂ may have a specific meaning. Of course, if necessary, any one of λ ₁ and λ ₂ may be selected as 0 (zero).

λ ₁ and λ ₂ may be set to 0.5, for example, respectively, but may be selected as experimental and statistical optimal values through repeated tests, and after the test result data is accumulated, a predetermined optimization algorithm is used to surround it. Each may be selected as an optimal value adaptive to the environment.

The distortion detection unit 225 uses any one of the distortion estimation value (G) calculated by the first technique and the overall difference (D) of each pixel value of the blending region calculated by the second technique, thereby distorting the individual top view images. Although the degree T may be calculated, the degree of distortion T of individual top view images may be calculated by using a third technique that weights the results of the first and second techniques as shown in Equation 3 below. .

Here, ω ₁ and ω ₂ are weight factors, and are respectively set such that the sum of ω ₁ and ω ₂ satisfies 1.

When ω ₁ is set relatively large compared to ω ₂ , the degree of distortion (T) of the individual top view images is calculated by focusing on the distortion estimation value (G) in the individual top view images calculated in Equation 1 above, and ω ₂ is ω _When it is set relatively large compared to ₁ , the degree of distortion T of individual top-view images, in which the difference in lane distortion between different individual top-view images forming the blending region calculated in Equation 2 above, is emphasized is calculated.

Of course, when only the results of the first technique or the second technique are applied to calculate the degree of distortion (T) of the individual top-view images, _one of ω ₁ and ω ₂ may be selected as 0 (zero).

ω ₁ and ω ₂ may be set to 0.5, for example, respectively, but may be selected as experimental and statistical optimal values through repeated tests, and after the test result data is accumulated, a predetermined optimization algorithm is used to surround it. Each may be selected as an optimal value adaptive to the environment.

The image correction unit 227 performs image correction on individual top-view images having a degree of distortion exceeding a predetermined reference value by the distortion detection unit 225.

As illustrated in FIGS. 7 and 8, the image correction operation is performed in such a manner that each location of each of the four image registration reference points 710 pre-stored to generate an AVM image is assigned to a predetermined pixel unit (for example, 1 pixel) from the location. It moves in the direction of up, down, left or right, and stores the position information of the image registration reference point 710 at the corresponding time point and the calculated value for each individual top view image calculated by the distortion detection unit 225.

The calculated value is, for example, a distortion estimation value (G) of the individual top view images calculated using the average angle of the lane extracted from the individual top view images by the first technique, or superimposed to generate an AVM image by the second technique. It may be a total difference (D) of pixel values in each of the blending regions of the plurality of individual top-view images, or a distortion degree (T) of weighting and summing them by a third technique.

The image correction unit 227 detects an output value having the smallest size among the respective calculation values calculated while changing the positions of the image registration reference points 710 in the individual top-view images within a predetermined area range, so as to correspond to this Position information of each of the four stored image registration reference points 710 is detected.

When moving each image registration reference point 710, a predefined optimization method, such as, for example, RANSAC, Gradient Descendent method, may be used.

In addition, the image correction unit 227 determines the position of the corresponding point in the individual top view image in which the angular difference in the lane and / or the pixel value difference in the blending area is minimized to determine the optimal position to which the image registration reference point 710 is to be moved. You can also use the cost function you are looking for.

The image matching reference point 710 detected through this process is a criterion for generating an AVM image with the highest screen matching when generating an AVM image using a separate top view image corresponding to each camera of the camera unit 210. It may be used as information, and may be updated and stored as a new image registration reference point 710. Of course, when generating an AVM image using each individual top view image, the location information of the surrounding objects recognized by the information recognition unit 223 may be further used as reference information for image matching.

Referring back to FIG. 2, when the position of the image registration reference point 710 is determined by the image correction unit 227, the conversion rule update unit 229 converts the conversion rules for each camera of the camera unit 210 to correspond to this. Produces The generated conversion rule may be stored in a storage unit (not shown).

The conversion rules include a transformation matrix for converting each shot image to a top view image and / or a lookup table (LUT) generated to convert each shot image to a top view image based on a distortion correction algorithm and a transformation matrix of each camera. You can.

The method of generating a conversion rule for converting a captured image taken by a camera to an image of a top view is obvious to those skilled in the art, so a detailed description thereof will be omitted.

The image synthesizing unit 230 generates an AVM image of a tolerance-corrected top view point around the vehicle using the captured image provided in the camera unit 210 and a pre-stored conversion rule, and outputs it through the output unit 240. .

As described above, in the AVM system according to the present embodiment, a captured image captured by each camera (see FIG. 3 (a)), a peripheral image generated using the captured image (see FIG. 3 (b)), or individually In the target image, which is a top view image (see (c) of FIG. 3), surrounding objects such as vanishing points, lanes, and pedestrian crossings are recognized, and whether the individual top view images are distorted is determined by referring to the recognized surrounding objects.

If there is a distortion in the individual top view image, the individual top view image is corrected to improve the distortion, and a corresponding conversion rule is generated to generate and output an AVM image that satisfies the consistency criteria.

Therefore, even if a cumulative tolerance occurs after the vehicle is shipped or the tolerance correction is not performed before the vehicle is shipped, the tolerance correction is automatically performed during the driving of the vehicle, thereby maximizing customer convenience and due to the uncorrected AVM image There is an advantage that the risk of accident occurrence is prevented.

9 is a flowchart illustrating a camera tolerance correction method of an AVM system according to an embodiment of the present invention.

Referring to FIG. 9, in step 910, the AVM system recognizes surrounding objects (eg, lanes, pedestrian crossings, pedestrians, etc.) using an image from a perspective. The perspective view image is each shot image taken by each camera of the camera unit 210, or each peripheral image generated by correcting the distortion of the corners of each shot image so that it has the shape of an image shot by a normal standard lens. Can be

In order to recognize the lane, the AVM system may detect, for example, linear elements in an image at a perspective, and may use a RANSAC algorithm, an MSE method, or the like, to detect a vanishing point, which is the point where the most straight lines intersect, and feature vectors ( For example, the direction of travel of the vehicle, Hough component vectors, etc.) and classifiers can also be used. In addition, a method such as binarization for each area, outline detection, and color analysis may be used for recognition of surrounding objects, and machine learning techniques may be applied for detection of highly reliable surrounding objects.

In step 920, the AVM system calculates a calculated value for determining that the detected surrounding objects (eg, lanes) are distorted in individual top-view images corresponding to each camera.

As described above, the calculated value for determining the individual top view image distortion is, for example, the distortion estimation value G of the individual top view image calculated using the average angle of the lane extracted from the individual top view image by the first technique, or The total difference (D) of pixel values in each of the blending regions of a plurality of individual top-view images superimposed to generate the AVM image by the second technique, or the degree of distortion (T) calculated by weighting and summing them by the third technique Can be

In step 930, the AVM system determines whether an individual top view image with distortion exists. The AVM system, for example, may determine that the corresponding individual top view image is wrong when the calculated output value exceeds a predetermined reference value.

If there is no individual top view image with distortion, the AVM system proceeds to step 950 to generate the AVM image of the top view point around the vehicle using the captured image provided in the camera unit 210 and the pre-stored conversion rule. , Output through the output.

However, if there is an individual top-view image determined to exist, in step 940, the AVM system corrects the individual top-view image so that the output value is minimized, and updates the conversion rule accordingly.

In order to correct individual top view images and update them with corresponding conversion rules, the AVM system, for example, positions each of the four image registration reference points pre-stored to generate an AVM image by a predetermined pixel interval from the location (eg 1 Pixel spacing) It moves in the direction of up, down, left or right, and stores the position information of the image registration reference point at the corresponding time point and the calculated value for each individual top view image calculated by the distortion detection unit. This process is repeated while changing the position of the image registration reference point within a predetermined area range in the individual top view image, respectively.

Subsequently, the AVM system detects the calculated value having the smallest size among the calculated values, detects location information of each of the four image registration reference points stored correspondingly, and positions information of each of the four detected image registration reference points. Create an updated conversion rule with reference to.

In step 950, the AVM system generates a tolerance-corrected top-view AVM image for the vehicle surroundings using the captured image provided by the camera unit 210 and the updated stored conversion rule, and outputs it through the output unit.

It is natural that the above-described tolerance correction method may be implemented as a computer program that performs an automated procedure in a time-series order, and that the implemented program can be driven by being built in or installed in a computer device. The codes and code segments constituting the program may be easily inferred by a computer programmer in the field.

The above has been described with reference to embodiments of the present invention, but those skilled in the art variously modify the present invention within the scope not departing from the spirit and scope of the present invention as set forth in the claims below. You will understand that you can change it.

10, 20, 30, 40: camera 50: calibration board
210: camera unit 220: tolerance correction unit
221: image conversion unit 223: information recognition unit
225: distortion detection unit 227: image correction unit
229: Conversion rule update unit 510, 512, 514, 516: individual top view video
610, 620, 630, 640: blending area
710: image registration reference point

Claims (11)

A camera unit including a plurality of cameras mounted to photograph the front, right side, left side, and rear sides of the vehicle;
Tolerances for determining whether or not a distortion according to a predetermined criterion exists in an individual top view image corresponding to a captured image captured by each camera, and updating a conversion rule for converting a captured image to a corresponding individual top view image if a distortion exists. Correction unit; And
An image synthesis unit for generating an AVM image using the updated conversion rule and a plurality of captured images generated by the camera unit,
The tolerance correction unit,
An image converting unit converting each of the plurality of photographed images provided by the camera unit into one or more of a peripheral image and an individual top view image according to a predetermined format;
An information recognition unit for recognizing a surrounding object including at least a lane by analyzing a target image of at least one of a captured image, a surrounding image, and an individual top view image;
A distortion detection unit to determine whether each individual top-view image is distorted by applying a predetermined distortion detection technique to each individual top-view image;
For each top-view image determined to be distorted, positions of each of a plurality of predefined image registration reference points are moved within a predetermined area range in a predetermined pixel unit, and based on each of the positions moved in a predetermined pixel unit. An image correction unit configured to detect location information of an image registration reference point having a minimum calculated value among the calculated values calculated by the distortion detection unit according to the application of the distortion detection method; And
And a conversion rule update unit that updates the conversion rule to correspond to the detected position information of the matched reference point,
The pre-specified distortion detection technique is characterized by calculating the calculated value that is the degree of distortion (T) of an individual top-view image using the following equation, and determining that the distortion exists when the calculated value exceeds a predetermined reference value. Around view monitoring system.

Here, ω ₁ and ω ₂ are weight factors respectively designated so that the sum value satisfies 1, and G is a distortion estimation value of each individual top view image,

, V _F is an individual top view image at the front, V _R is an individual top view image at the right, V _L is an individual top view image at the left, V _B is an individual top view image at the rear, μ _α is any one individual top view image The average lane angle (Radian) in the α image, which is the image, D is the total difference of the pixel values in each blending area where each individual top-view image overlaps to generate an AVM image, and λ ₁ and λ ₂ have a sum of 1 , Respectively, B is a coordinate range of the blending region, β is any one of the coordinate ranges of the blending region, and p _β is a β location in the blending region of the first individual top-view image forming the blending region. and a pixel value, _β q is the pixel value of the β position in the blending area of the second individual top view image to form a blended region, δ is the peripheral four recognized by the information identification portion in the coordinate range of the blending area In a position close to the object in contact with the ground with the advance over the specified threshold height, p _δ is the pixel value of the δ position in the blending area of the first individual top view image to form a blended region, q _δ is to form a blended region Pixel value at position δ in the blending region of the second individual top view image.
delete According to claim 1,
The pre-specified distortion detection technique is characterized in that it calculates a distortion value (G) of each individual top-view image using the following equation, and determines that a distortion exists when the calculated value exceeds a predetermined reference value. Around view monitoring system.

here,

, V _F is an individual top view image at the front, V _R is an individual top view image at the right, V _L is an individual top view image at the left, V _B is an individual top view image at the rear, μ _α is any one individual top view image The average lane angle (Radian) in the image, α image.
According to claim 1,
The predetermined distortion detection technique calculates a calculated value that is the total difference (D) of pixel values in each blending region where each individual top-view image overlaps to generate an AVM image using the following equation, and the calculated calculated value Around view monitoring system, characterized in that it is determined that there is a distortion when the predetermined reference value is exceeded.

Here, λ ₁ and λ ₂ are weight factors respectively designated so that the sum value satisfies 1, B is a coordinate range of the blending region, β is any one of the coordinate ranges of the blending region, and p _β forms a blending region. Is a pixel value at a β position in the blending region of the first individual top view image, q _β is a pixel value at a β position in the blending region of the second individual top view image forming the blending region, and δ is among the coordinate ranges of the blending region Of the surrounding objects recognized by the information recognition unit, a surrounding object having a height equal to or greater than a predetermined threshold value is in contact with the ground, and p _δ is a pixel value at the δ position in the blending area of the first individual top-view image forming the blending area, q _δ is the pixel value of the δ position in the blending region of the second individual top-view image forming the blending region.
delete According to claim 1,
The information recognition unit is configured to configure the vehicle's traveling direction and Hough component vector as a feature vector, and classify it using a predefined classifier to recognize the lane.
According to claim 1,
The information recognition unit further recognizes at least one of traffic lights, pedestrians, road floor information, and surrounding objects in the target image,
Around view monitoring system, characterized in that the recognized location information of surrounding objects is used as reference information for generating the AVM image using a plurality of individual top view images.
The method of claim 7,
The information recognition unit applies a pre-designated machine learning technique to surround the surrounding view monitoring system, characterized in that to limit only those objects whose reliability is evaluated to satisfy a predetermined criterion among surrounding objects.
According to claim 1,
The image correction unit moves the image registration reference point by applying at least one of a predetermined optimization technique and a cost function for finding a position of a corresponding point in an individual top-view image in which a difference between a lane angle and a pixel value in a blending region is minimized. Around view monitoring system.
A computer program stored in a computer-readable medium for correcting camera tolerance, the computer program causing the computer to perform the following steps, the steps comprising:
Converting each of the captured images generated by the plurality of cameras included in the camera unit into one or more of a peripheral image and an individual top view image according to a predetermined format;
Recognizing a surrounding object including at least a lane by analyzing a target image of at least one of a captured image, a surrounding image, and an individual top view image;
Determining whether each individual top view image is distorted by applying a predetermined distortion detection method to each individual top view image;
For each individual top view image determined to be distorted, positions of each of a plurality of predefined image registration reference points are moved within a predetermined area range in a predetermined pixel unit, and based on each of the moved positions in a predetermined pixel unit. Detecting position information of an image registration reference point having a minimum calculated value among calculated values calculated according to the application of the distortion detection technique; And
And updating a conversion rule for generating an AVM image using a plurality of captured images, so as to correspond to the detected location information of the matched reference point.
The distortion detection method calculates a distortion value (T) of an individual top-view image by using the following equation, and when the calculated calculated value exceeds a predetermined reference value, it is determined that a distortion exists. -A computer program stored on a readable medium.

Here, ω ₁ and ω ₂ are weight factors respectively designated so that the sum value satisfies 1, and G is a distortion estimation value of each individual top view image,

, V _F is an individual top view image at the front, V _R is an individual top view image at the right, V _L is an individual top view image at the left, V _B is an individual top view image at the rear, μ _α is any one individual top view image The average lane angle (Radian) in the α image, which is the image, D is the total difference of pixel values in each blending area where each individual top-view image overlaps to generate an AVM image, and λ ₁ and λ ₂ are sums of 1 , Respectively, B is a coordinate range of the blending region, β is any one of the coordinate ranges of the blending region, and p _β is a β location in the blending region of the first individual top-view image forming the blending region. Is the pixel value of, q _β is the pixel value of the β position in the blending region of the second individual top-view image forming the blending region, and δ is pre-designated among the recognized peripheral objects among the coordinate ranges of the blending region. A position where a peripheral object having a height above a threshold is in contact with the ground, p _δ is a pixel value at a position _δ in the blending area of the first individual top-view image forming a blending area, and q _δ is a second individual forming a blending area Pixel value at position δ in the blending area of the top view image.
delete

Images