KR20180052415A - Method for compensating camera tolerance based on a camera image of a vehicle - Google Patents

Abstract

The present invention relates to a technology for correcting a tolerance of a camera for a vehicle mounted on a car, which applies an image recognizing technology to an image input through the camera provided in the vehicle, recognizes and obtains information on a surrounding environment of the vehicle, calculates a tolerance generated in the corresponding camera according to daily vehicle driving based on the recognized information and automatically corrects the tolerance, thereby significantly reducing inconvenience of a driver and a vehicle producer due to application of an additional device, equipment or a technology required in the conventional technology for correcting the tolerance of the camera, and improving marketability of a product vehicle. The method for correcting a tolerance based on a camera image of a vehicle comprises the following steps of: receiving image information from the camera; detecting feature points P and P′ corresponding to two continuous images within the received image information; calculating and estimating ego-motion of the feature points P and P′, and estimating a position relationship and a moving direction (d) of the camera at the time of capturing the two images based on the estimated ego-motion; projecting the position of the camera on a virtual image plane, and calculating a distance and a direction from the projected camera position to a virtual straight line (g); and estimating the tolerance by calculating the direction and the height of the camera from the distance and the direction from the projected camera position to the straight line (g).

Description

≪ Desc / Clms Page number 1 > METHOD FOR COMPENSATING CAMERA TOLERANCE BASED ON A CAMERA IMAGE OF A VEHICLE &

BACKGROUND OF THE INVENTION 1. Field of the Invention [0001] The present invention relates to a technique for correcting a tolerance of a vehicle camera mounted on a vehicle, and more particularly, The present invention relates to a technique of calculating a tolerance generated in a camera based on recognized information and automatically correcting the calculated tolerance.

Generally, an ADAS (Advanced Driver Assistance System), which is mounted on an automobile and plays a role of assisting the driving of the vehicle, includes at least one camera provided to photograph the surrounding situation of the vehicle and provide it to the driver .

Typically, the camera of the driver's running assist system is configured to photograph an image in a predetermined direction, for example, a fixed direction so as to be directed in a specific direction, for example, the rear of the vehicle, To calculate information necessary for the driver according to a predetermined algorithm or program procedure.

However, the camera of the driver's driving assist system has a problem in that the relationship between the camera and the vehicle, that is, the unintended error of the camera's initially designed fixed position or the direction of the camera due to natural deterioration due to long- . This error is commonly referred to as tolerance.

Such a tolerance may cause not only an error of the image captured by the camera and provided to the driver but also an error in the information calculated by a predetermined image recognition algorithm from the captured image so that it is necessary to periodically correct the tolerance generated in the camera there was.

However, the correction of the tolerance is not only difficult for the driver to recognize himself, but also can inconvenience the driver because there is a restriction to visit a specific workshop equipped with related equipments and techniques to correct the tolerance. Therefore, despite the occurrence of the tolerance, there is a disadvantage in that the information obtained from the camera in which the tolerance is not recognized is provided to the driver as it is, or the correction is delayed due to circumstances in which it is difficult to dispense the workshop even though the tolerance is recognized.

As a conventional technique for solving the above problems, Korean Patent Laid-Open Publication No. 10-2015-0135697 entitled " Camera Parameter Measuring Apparatus and Method "discloses a technique in which reference marks (patterns) A method of calculating the image distance between a projection point corresponding to both end points of a cover sheet and a predetermined reference image point from a plurality of images and calculating a height and a photographing direction of the camera based on the calculated image distances. However, in the above-mentioned conventional technique, there is a problem that, when the camera is calibrated, a target having a specific pattern whose length is determined should be used, so that the target is necessarily required.

Therefore, it is required to develop a more advanced technology for solving the problem of lowering the commerciality of the vehicle due to the tolerance of the camera occurring during the use of the vehicle and the difficulty and inconvenience caused by the tolerance correction.

The present invention has been made to solve the above problems,

The image recognition technology is applied to an image input through a camera installed in a vehicle to recognize and acquire information about the environment around the vehicle. Based on the recognized information, a tolerance generated in the corresponding camera is calculated according to a routine vehicle operation, It is possible to greatly reduce the inconvenience of the driver and the vehicle manufacturer due to the application of the additional apparatus, equipment or technology required in the prior art for the tolerance correction of the camera, and further to improve the commerciality of the product vehicle .

According to an aspect of the present invention,

A camera camera based image tolerance correction method, comprising: receiving image information from a camera; Detecting corresponding feature points P and P 'of two consecutive images in the received image information; Calculating and estimating ego-motion of the minutiae points P and P ', estimating a positional relationship and a moving direction (d) of the camera at the time of photographing two images based on the estimated ego motion; Projecting the position of the camera on the virtual image plane, calculating the distance and direction from the projected camera position to the imaginary straight line (g); And estimating the tolerance by calculating the direction and height of the camera from the distance and direction from the position of the projected camera to the straight line g.

The method may further include tracking additional feature points based on a moving direction (d) of the camera and calculating three-dimensional coordinates of the additional feature points tracked; Projecting additional feature points calculated in three-dimensional coordinates on a virtual image plane perpendicular to the moving direction of the camera; And detecting a straight line g including the largest number of additional feature points on the virtual image plane.

The method further includes correcting distortion of the image based on optical characteristics of the wide angle lens of the camera.

The method further includes estimating a front direction of the vehicle based on the calculated moving direction (d) of the camera.

Receiving information on the moving distance of the vehicle from the vehicle tolerance correcting unit via the CAN communication line; Calculating an error ratio between information on the received movement distance and a movement distance d of the camera calculated from the image information; And estimating the tolerance by calculating the direction and height of the camera by applying the calculated error ratio to the distance and direction from the projected camera position to the straight line g and the calculated error ratio. do.

The method may further include generating and outputting a warning message when determining the occurrence of the tolerance.

The method further includes the step of automatically correcting the tolerance by reflecting the estimated tolerance ratio at the position of the camera on the virtual image plane in the image information acquired from the camera when the tolerance occurrence is determined.

The camera-based image-based tolerance correction method of the present invention provides the following characteristic advantages through the above-described configuration.

1) Automatically analyzing and correcting the camera tolerance that may occur during daily use of the vehicle from the image obtained from the camera by using the image recognition technology, thereby making it possible to continuously change the attitude of the camera caused by external shock or aging after shipment It is possible to actively cope with the tolerance according to the present invention.

2) Automatically correcting the tolerance generated in the camera while driving the vehicle can solve the problem of visiting the vehicle repair shop for tolerance correction.

3) When a rear camera equipped with a recognition function is installed in a vehicle, since the camera posture is closely related to the recognition performance, it is possible to secure the safety of the driver by securing the recognition performance when applying the present technology have.

4) When the tolerance of the camera is generated, a warning message is output to recognize the occurrence of the tolerance, so that the safety of the driver can be increased.

1 is a block diagram schematically illustrating a system for practicing a vehicle camera image-based tolerance correction method according to a preferred embodiment of the present invention.
2 is a flowchart illustrating a method for correcting a camera camera image-based tolerance according to a preferred embodiment of the present invention.
Fig. 3 shows the computed moving direction component of the camera.
FIG. 4 shows feature points P and P 'estimated by the KLT feature tracking and additional feature points according to the estimated moving direction (d) of the camera center, respectively.
Figure 5 shows an example of a linear triangulation algorithm.
FIG. 6A is a schematic view schematically showing the step of projecting three-dimensional feature points on any imaginary image plane perpendicular to the moving direction, FIG. 6B schematically shows a step of calculating the height of the camera from the ground on the virtual image plane of FIG. .

Hereinafter, preferred embodiments of the present invention will be described in detail with reference to the accompanying drawings. Hereinafter, preferred embodiments of the present invention will be described in detail with reference to the accompanying drawings. The word is described based on the principle that the inventor can properly define the concept of a term in order to explain his or her invention in the best way. The word is interpreted as a meaning and concept consistent with the technical idea of the present invention. . Therefore, the embodiments described in this specification and the technical structures shown in the drawings are only the most preferred embodiments of the present invention and do not represent all the technical ideas of the present invention. Therefore, It should be understood that various equivalents and modifications are possible. In addition, the terms used herein are used for the purpose of easy understanding of specific embodiments, and are not intended to limit the present invention. It is to be understood that the elements described in the singular form in this specification include plural forms unless otherwise specified.

The present invention relates to a technology for correcting a tolerance of a camera mounted on an automobile. It is an object of the present invention to recognize and acquire information on the surroundings of a vehicle by applying an image recognition technology to an image input through a camera provided in the vehicle, It is possible to greatly reduce the inconvenience of the driver and the vehicle manufacturer due to the application of additional devices, equipments or technologies required in the prior art for correcting the tolerances of the camera, And further, to improve the merchantability of the product vehicle.

Hereinafter, preferred embodiments of the present invention will be described in detail with reference to the drawings.

FIG. 1 is a block diagram schematically showing a system for implementing a camera-based image-based tolerance correction method according to a preferred embodiment of the present invention, and FIG. 2 is a flowchart illustrating a method for correcting a camera camera-based tolerance according to a preferred embodiment of the present invention Fig.

As shown in FIG. 1, the vehicle camera image-based tolerance correcting method of the present invention includes at least one camera 20 (for example, a camera of an automobile) installed in an automobile and photographing a vehicle peripheral image to acquire and generate image information Rear camera); The camera 20 receives the image information, analyzes and recognizes the tolerance of the camera 20 by applying at least one image recognition algorithm to the image information, and corrects the virtual image plane of the image information at the time of determining the occurrence of the tolerance A tolerance correcting unit 10 for correcting the tolerance of the camera 20; And a CAN communication line 30 for connecting the tolerance correcting unit 10 and a vehicle control unit (for example, a BCM or the like) to allow the tolerance correcting unit 10 to receive vehicle information necessary for determining the occurrence of a tolerance ).

The camera 20 is installed in an automobile and generates image information by photographing an image in a direction of the camera 20. The generated image information is transmitted to the tolerance correcting unit 10. [ Hereinafter, in describing the preferred embodiment of the present invention, the camera 20 will be described on the basis of a rear camera of an automobile, but the present invention is not limited thereto. For example, It should be understood that the invention can be applied to a front camera, or to two or more cameras respectively.

In addition, the camera 20 may generate and output image information including a moving image continuously photographing the surroundings of the vehicle, or may display a photograph of the surroundings of the vehicle at a predetermined cycle or image information including a plurality of moving images To the tolerance correcting unit 10 by periodically generating and outputting the error signal.

The image information includes, for example, image information including a plurality of image sections or still images obtained in a predetermined period in one moving image, image information including at least two moving images, or image including at least two still images It can be information.

The tolerance correcting unit 10 analyzes and recognizes the tolerance of the camera 20 by receiving image information from the camera 20 and applying at least one image recognition algorithm to the image information, And corrects the tolerance of the camera 20 by correcting the virtual image plane of the image information.

For this purpose, the tolerance correcting unit 10 includes at least one image information input from the camera 20, for example, image information including a plurality of image sections or still images obtained at predetermined periods in one moving image, The image recognition algorithm is applied to image information such as image information including at least two moving images, or image information including at least two still images.

The tolerance correcting unit 10 may be installed in the vehicle as a separate component including an electronic control unit (ECU) or a micro control unit (MCU) for the purpose of performing the image processing and the algorithm as described above, (E.g., a vehicle body control module (BCM) or a cluster control unit) equipped for control of the vehicle.

The tolerance correcting unit 10 detects and tracks feature points in the image information by applying an image recognition algorithm to the image information input from the camera 20, corrects the coordinate distortion of the lens of the camera 20, And further estimates the three-dimensional additional feature point information based on the detected feature points. Based on the estimated three-dimensional additional feature point information, the camera 20 To determine the tolerance and to correct it.

A method for estimating and correcting the tolerance in the tolerance correcting unit 10 will be described in more detail with reference to FIG.

First, a camera 20 installed in a vehicle, for example, a rear camera of a vehicle, photographs the surroundings of the vehicle, generates image information, and outputs the image information to the tolerance correcting unit 10 (S001).

The tolerance correcting unit 10 corrects the image information inputted from the camera 20 (for example, image information including a plurality of image sections or still images obtained in a predetermined period in one moving image, at least two moving images (I.e., image information including at least two or more still images, etc.), the feature points P and P 'corresponding to the respective images are detected (S002).

At this stage, for example, the following corner detection algorithm and feature point tracking algorithm can be used.

a) Harris corner detector (Harris corner detector)

- Harris Corner Detection Algorithm is an algorithm for detecting corner points in an image. It is an algorithm mainly used for matching images to track or recognize specific objects in an image. The Harris corner detection algorithm is used to detect feature points for matching each image when matching the images.

In the preferred embodiment of the present invention, the tolerance correcting unit 10 detects feature points in two consecutive images by using the Harris corner detection algorithm for the received image information.

b) KLT feature tracker (KLT feature tracker)

-KLT feature tracking algorithm is an algorithm to track the most efficient feature points for matching of detected images of feature points.

In the preferred embodiment of the present invention, the tolerance correcting unit 10 uses the Harris corner detection algorithm to track the KLT feature traces to track corresponding feature points P and P 'among the feature points in two consecutive images detected and obtained Algorithm.

As a result, the tolerance correcting unit 10 may apply the Harris corner detection algorithm and the KLT feature tracking algorithm to the two images received from the camera 20 to obtain corresponding feature points P and P ' have.

Meanwhile, the tolerance correcting unit 10 performs a step of correcting distortion of the image due to the optical characteristics of the wide angle lens constituting the camera 20 (S003).

In general, a wide angle lens is used for a wide angle of view of a camera. However, the wide angle lens has a disadvantage in that the image is distorted due to its optical characteristics. In order to prevent the distortion of the coordinates of the objects in the recognized image due to the distortion, a step of correcting the distortion of the wide-angle lens is performed.

The step of correcting the distortion of the wide-angle lens can be performed by applying the optical characteristics of the wide-angle lens of the camera applied to the vehicle, and a detailed description thereof will be omitted.

The tolerance correcting unit 10 estimates the front direction of the vehicle based on the ego motion by calculating and estimating ego-motion of the corresponding minutiae points P and P 'detected in step S002 (S004).

In one embodiment of the present invention, the ego motion estimation of the vehicle is estimated by applying epipolar constraints. In other words, the tolerance correcting unit 10 applies a point matching algorithm using the epipolar constraint to the minutiae points P and P ' Estimate motion (ego-motion).

At this time, the tolerance correcting unit 10 derives an epipolar geometry for the images from the feature points P and P 'detected between the two images, and the epipolar constraint derived from the epipolar geometry The positional relationship between the two cameras and the moving direction can be estimated by matching each of the minutiae points P and P '.

The epipolar constraint can be expressed by the following equation.

[Equation 1]

As a result, the basic matrix to which the above-mentioned epipolar constraint is applied is expressed by the following equation.

[Formula 2]

(2) where F is a fundamental matrix, p is a project point, K is a camera intrinsic parameter, t is a translation matrix, and R is a rotation matrix ).

Meanwhile, in another embodiment of the present invention, ego motion can be estimated by applying a Gauss-Newton method. The basic matrix in this embodiment is expressed by the following equation.

[Formula 3]

The estimated parameters are as follows

By applying a least square object function to the estimated parameters as described below, it is possible to estimate the ego motion by determining an optimized parameter among the estimated parameters.

[Formula 4]

In another embodiment of the present invention, the RANSAC algorithm may be applied to estimate ego-motion. The RANSAC algorithm is a well-known algorithm for selecting a model supported from the largest number of data. In this embodiment, the RANSAC algorithm is applied to a plurality of feature points P and P ' .

Based on the estimated echo-motion of the camera 20, the moving direction d of the center of the camera 20 is calculated through the following equation. 3 shows the moving direction component of the camera 20 calculated in this way.

[Formula 5]

Thereafter, the tolerance correcting unit 10 further tracks the feature points on the basis of the moving direction (d) of the camera 20 in the image obtained by estimating the echo-motion of the camera 20 through the step S004, The three-dimensional coordinates of the posterior feature points are calculated (S005).

In this step, with the feature point P and P 'estimated by the KLT feature tracking in step S002 with the edgel tracking algorithm, the estimated moving direction d of the center of the camera 20 acquired in step S004 To estimate the additional feature points.

4 shows the feature points P and P 'estimated by the KLT feature tracking and the additional feature points according to the estimated movement direction d of the center of the camera 20, respectively.

Then, the tolerance correcting unit 10 calculates a three-dimensional coordinate of each feature point by applying a linear triangulation algorithm for calculating three-dimensional coordinates to the additional feature points obtained by the islet tracking algorithm. Figure 5 shows an example of a linear triangulation algorithm.

Estimates the ground surface g based on the moving direction of the camera 20 estimated through the step S005 and estimates the height of the camera 20 from the estimated ground surface g in step S006.

In this step, the three-dimensional feature points calculated in the moving direction (d) of the camera 20 are projected onto a virtual image plane perpendicular to the moving direction of the camera 20 estimated through the step S005, and a hough transform is performed. Algorithm to detect a straight line containing the largest number of feature points on the projection plane.

Thereafter, the position of the camera 20 is projected onto the virtual image plane on which the three-dimensional minutiae are projected, the distance and direction to the straight line including the detected most minutiae points are calculated, and at the same time, And direction.

20A is a schematic view schematically showing the step of projecting three-dimensional feature points on any imaginary image plane perpendicular to the moving direction, FIG. 20B is a view showing a step of calculating the height of the camera 20 from the ground on the virtual image plane of FIG. 20B Fig.

The direction of the vehicle 20 is calculated by calculating a rotation matrix in which the direction of movement d of the camera 20 and the direction of the ground g of the camera 20 are respectively the front direction and the downward direction of the camera 20, (S007),

Receives information on the moving distance of the vehicle from the vehicle control unit via the can communication line 30 and calculates an error ratio between the moving distance of the camera 20 calculated from the image information, The tolerance is estimated by calculating the height at which the actual camera 20 is located from the ground surface g by applying the calculated height to the calculated height of the camera 20 (S008)

If it is determined in step S007 that a tolerance has been generated, the tolerance correcting unit 10 generates a warning message and outputs the warning message to a cluster in the vehicle or the like, thereby providing a tolerance occurrence warning message to the driver (S009). At this time, the output of the warning message may be output through any output device included in the cluster provided in the vehicle, for example, a display means or the like.

Lastly, the tolerance correcting unit 10 reflects the tolerance ratio calculated from the steps at the time of determining the tolerance to the position of the camera 20 on the virtual image plane in the image information acquired from the camera 20, It can be corrected automatically (S009).

According to the present invention, the tolerance of the camera 20, which may occur during daily use of the vehicle, is automatically analyzed and corrected from the image obtained from the camera 20 using the image recognition technology, It is possible to actively cope with the tolerance according to the continuous attitude change of the camera 20 caused by external shock or aging after shipment, to solve the troubles to visit the car repair shop for the tolerance correction, and furthermore, When the rear camera 20 is provided in the vehicle, since the posture of the camera 20 is closely related to the recognition performance, safety of the recognition performance is ensured when applying the present technology, thereby providing the advantage of securing the safety of the driver Finally, when the tolerance of the camera 20 is generated, a warning message is output to recognize the occurrence of the tolerance, Thereby increasing the safety of the apparatus.

While the present invention has been particularly shown and described with reference to exemplary embodiments thereof, it is to be understood that the scope of the invention is not limited to the disclosed exemplary embodiments. . It will be apparent to those skilled in the art that other modifications based on the technical idea of the present invention are possible in addition to the embodiments disclosed herein.

10: Tolerance correction unit
20: Camera
30: Can communication line

Claims (7)

A method for correcting a camera camera image based tolerance,
Receiving image information from a camera;
Detecting corresponding feature points P and P 'of two consecutive images in the received image information;
Calculating and estimating ego-motion of the minutiae points P and P ', estimating a positional relationship and a moving direction (d) of the camera at the time of photographing two images based on the estimated ego motion;
Projecting the position of the camera on the virtual image plane, calculating the distance and direction from the projected camera position to the imaginary straight line (g); And
Estimating the tolerance by calculating the direction and height of the camera from the distance and direction from the position of the projected camera to the straight line g;
Wherein the camera image based tolerance correction method comprises:
The method according to claim 1,
Tracking additional feature points based on a moving direction (d) of the camera, and calculating three-dimensional coordinates of the additional feature points tracked;
Projecting additional feature points calculated in three-dimensional coordinates on a virtual image plane perpendicular to the moving direction of the camera; And
Detecting a straight line (g) including the largest number of additional feature points on the virtual image plane;
Further comprising the step of correcting the tolerance of the camera image based on the tolerance.
The method according to claim 1,
Correcting image distortion based on an optical characteristic of a wide-angle lens of a camera;
Further comprising the step of correcting the tolerance of the camera image based on the tolerance.
The method according to claim 1,
Estimating a front direction of the vehicle based on the calculated moving direction (d) of the camera;
Further comprising the step of correcting the tolerance of the camera image based on the tolerance.
The method according to claim 1,
Receiving information on a moving distance of the vehicle from the vehicle tolerance correcting unit via the CAN communication line;
Calculating an error ratio between information on the received movement distance and a movement distance d of the camera calculated from the image information; And
Estimating the tolerance by calculating the direction and height of the camera by applying the calculated error ratio to the distance and direction from the projected camera position to the straight line g and the calculated error ratio;
Further comprising the step of correcting the tolerance of the camera image based on the tolerance.
The method according to claim 1,
Generating and outputting a warning message upon determining a tolerance occurrence;
Further comprising the step of correcting the tolerance of the camera image based on the tolerance.
The method according to claim 1,
When judging the occurrence of the tolerance,
Automatically correcting the tolerance by reflecting the estimated tolerance ratio on the position of the camera on the virtual image plane in the image information acquired from the camera;
Further comprising the step of correcting the tolerance of the camera image based on the tolerance.

Images