WO2021033812A1 - Heterogeneous stereo camera system and camera calibration method - Google Patents

Abstract

Disclosed are a heterogeneous stereo camera system and a camera calibration method. The heterogeneous stereo camera system includes: an edge detection unit which uses a pre-designated lane detection technique to detect a wide angle lane from a wide angle image captured by a wide angle camera, and detect a narrow angle lane from a narrow angle image captured by a narrow angle camera; a camera calibration unit which converts the wide angle image and the narrow angle image into top view images, and updates respective conversion rules for the wide angle camera and the narrow angle camera so that the wide angle lane and the narrow angle lane each meet pre-defined infinite vanishing point conditions; and a matching unit which uses the conversion rules updated in order to meet the infinite vanishing point conditions to further update the conversion rules for at least one of the wide angle camera and the narrow angle camera so that the widths of the wide angle lane and the narrow angle lane, which have been converted into the top view images, become the same.

Description

Heterogeneous stereo camera system and camera calibration method

The present invention relates to a heterogeneous stereo camera system and a camera calibration method.

In general, the driver's visibility inside the vehicle is mainly directed to the front, and the driver's left and right sides and rear view are substantially obscured by the vehicle body and are very limited.

In order to solve this problem, vision assistance means such as side mirrors have been used, and in recent years, technologies for photographing the outside of the vehicle with a camera means and providing it to the driver are also variously applied to the vehicle.

1 is a diagram illustrating camera devices provided in a vehicle.

As illustrated in FIG. 1, an Around View Monitoring (AVM) system is provided that displays images in all directions around the vehicle by mounting a plurality of cameras 111, 113, 115, and 117 on the vehicle. .

The AVM system combines images around the vehicle captured by cameras 111, 113, 115, and 117 provided at each location of the vehicle, and is an AVM in the form of a Top View image, as if the driver is looking at the vehicle from the sky. Provide video. The AVM system has the advantage of eliminating blind spots and allowing the driver to easily check obstacles around the vehicle.

In addition, in recent years, a front narrow angle camera 120 is additionally provided in order to enable the driver to effectively recognize the condition of a long-distance road in front of the vehicle.

The cameras for the AVM system described above are implemented to shoot a relatively short distance area in a wide angle, but the narrow angle camera is implemented to take a relatively long distance area in a narrow angle.

In this way, the wide-angle camera 111 and the narrow-angle camera 120 for an AVM system provided to photograph the same direction (for example, the front) among a plurality of cameras provided in the vehicle may be used as a stereo camera as needed. .

For this, correction processing for each camera is required. Conventionally, a method of capturing a predetermined correction pattern and correcting it from a three-dimensional coordinate to the position of a camera has been used. However, such a camera correction method uses the position of each point in each pattern image, and there is a problem that a lot of time is required during the correction operation.

The above-described background technology is technical information possessed by the inventors for derivation of the present invention or acquired during the derivation process of the present invention, and is not necessarily known to be publicly known prior to filing the present invention.

The present invention is a heterogeneous stereo camera system that can quickly perform a stereo camera correction process by acquiring and matching 3D information of each camera using lane images taken from each of a wide-angle camera and a narrow-angle camera arranged to photograph the same direction. And a camera calibration method.

Objects other than the present invention will be easily understood through the following description.

According to an aspect of the present invention, a wide-angle lane is detected from a wide-angle image image captured by a wide-angle camera using a predetermined lane detection technique. An edge detection unit for detecting a narrow-angle lane in a narrow-angle image image captured by a narrow-angle camera; A camera beam that converts the wide-angle video image and the narrow-angle video image into a top-view image and updates conversion rules for the wide-angle camera and the narrow-angle camera so that each of the wide-angle lane and the narrow-angle lane satisfies a predefined vanishing point infinite condition. government; And a conversion rule for at least one of the wide-angle camera and the narrow-angle camera so that the width of the wide-angle lane and the narrow-angle lane converted into a top-view image by using the updated conversion rule to satisfy the infinite vanishing point condition is further updated. A heterogeneous stereo camera system including a matching unit is provided.

The camera correction unit is a rotation matrix (R) according to an angle obtained by accumulating the rotation angle θ of the camera calculated immediately before until the rotation angle θ of the camera calculated using the following equations satisfies the infinite vanishing point condition. The process of recalculating the rotation angle θ of the camera may be repeated using. Here, the above equations are,

Can be,

Wherein x _i is the coordinates of the vanishing point in the captured video image, x _v is the coordinates of the vanishing point in the top view image, K is an internal parameter of the camera, and R is a rotation matrix according to the rotation angle in each axis direction of the camera Is a rotation matrix calculated by the product of _{R x} , R _y and R _{z, and T is a translation matrix,}

Is

It is the direction component value calculated as, and v _u is [0 1 0] ^T.

Other aspects, features, and advantages other than those described above will become apparent from the following drawings, claims, and detailed description of the invention.

According to an embodiment of the present invention, a stereo camera correction process is quickly performed by acquiring and matching 3D information of each camera using lane images taken from each of the wide-angle camera and the narrow-angle camera arranged to photograph the same direction. There is.

1 is a diagram illustrating camera devices provided in a vehicle.

2 is a block diagram of a heterogeneous stereo camera system according to an embodiment of the present invention.

3 is a view for explaining a correction process of each camera according to an embodiment of the present invention.

4 is a diagram illustrating a matching process for configuring a stereo camera according to an embodiment of the present invention.

In the present invention, various modifications may be made and various embodiments may be provided, and specific embodiments will be illustrated in the drawings and described in detail in the detailed description. However, this is not intended to limit the present invention to a specific embodiment, it is to be understood to include all changes, equivalents, or substitutes included in the spirit and scope of the present invention.

The terms used in the present specification are only used to describe specific embodiments, and are not intended to limit the present invention. Singular expressions include plural expressions unless the context clearly indicates otherwise. In this specification, terms such as "comprise" or "have" are intended to designate the presence of features, numbers, steps, actions, components, parts, or a combination thereof described in the specification, but one or more other features. It is to be understood that the presence or addition of elements or numbers, steps, actions, components, parts, or combinations thereof, does not preclude in advance.

In addition, terms such as "... unit", "... unit", "... module", and "... group" described in the specification mean a unit that processes at least one function or operation, which is hardware or software, or hardware and It can be implemented as a combination of software.

2 is a block diagram of a heterogeneous stereo camera system according to an embodiment of the present invention, FIG. 3 is a view for explaining a correction process of each camera according to an embodiment of the present invention, and FIG. 4 is A diagram for describing a matching process for configuring a stereo camera according to an embodiment of

Referring to FIG. 2, the heterogeneous stereo camera system 200 may include a wide-angle camera 210, a narrow-angle camera 220, a matching unit 230, and a conversion rule storage unit 240.

The wide-angle camera 210 is a camera for an AVM system, for example, cameras (111, 113 in FIG. Among 115 and 117, it may be a camera arranged to photograph the same direction as the narrow angle camera 220.

The wide-angle camera 210 may use, for example, a wide-angle lens (or fisheye lens) having an angle of view of around 180 degrees so that a wide area close to the vehicle may be used as a photographing area.

The narrow-angle camera 220 may be a camera in which a narrow-angle lens having an angle of view of 30 to 50 degrees is used, for example, so that an area relatively far from the vehicle can be taken as a photographing area compared to the wide-angle camera 210.

Each of the wide-angle camera 210 and the narrow-angle camera 220 generates a captured image image corresponding to a photographing angle in a mounted state and provides it to the matching unit 230. Although not shown, an input unit and a storage unit for providing image data captured by each of the wide-angle camera 210 and the narrow-angle camera 220 to the matching unit 230 may be further included.

The matching unit 230 may include an edge detection unit 232, a camera correction unit 234 and a matching unit 236. As will be described later, the edge detection unit 232 and the camera correction unit 234 perform predetermined correction processing on the wide-angle image image captured by the wide-angle camera 210 and the narrow-angle image image captured by the narrow-angle camera 220, respectively. Then, the matching unit 236 matches the processed wide-angle and narrow-angle image images, and processes the wide-angle camera 210 and the narrow-angle camera 220 to function as heterogeneous stereo cameras.

The edge detection unit 232 detects a lane using a predetermined lane detection technique in each of the wide-angle and narrow-angle image images. An example of detecting a lane 310 using a lane detection technique in a wide-angle image and a narrow-angle image is illustrated in FIG. 3A.

A lane detection technique for recognizing the existence of the lane 310 in an image image and detecting an outline of the recognized lane 310 may be, for example, a sobel edge detection technique. In addition to the Sobel edge detection technique, there may be various techniques for recognizing the lane and detecting the outline, and this is obvious to a person skilled in the art (a person having ordinary knowledge in the technical field to which the invention belongs), so a description thereof will be omitted.

The camera correction unit 234 corrects each of the wide-angle camera 210 and the narrow-angle camera 220 by using the lane detection result by the edge detector 232 for each of the wide-angle and narrow-angle image images. The correction for each of the wide-angle camera 210 and the narrow-angle camera 220 may be based on finding a location where each camera is installed and a current installation angle. Here, the position of each camera may be fixed, but the angle of the camera may be changed due to vibration of the vehicle, so that correction processing such as updating the conversion rule of each camera may be performed by finding the rotation angle of each axis.

The camera correction unit 234 converts the lane detection results (refer to (a) of FIG. 3) of the wide-angle image image and the narrow-angle image image into a top-view image, respectively, using conversion rules previously stored in the conversion rule storage unit 240 ( 3(b) and 4(a) and (b)), the installation angles of the wide-angle camera 210 and the narrow-angle camera 220 satisfying the predetermined vanishing point infinite condition are calculated, respectively, and The conversion rules of the camera 210 and the narrow angle camera 220 are updated.

Here, in the infinite vanishing point condition, the two lanes in the top view image are not only parallel to each other, but also the outer and inner boundary lines of each of the left and right lanes are parallel to each other, so the vanishing point (Fig. 3(a)) P) is a condition that must be an infinite point (see Fig. 3(b)).

The camera correction unit 234 calculates the rotation angle of the wide-angle camera 210 and the narrow-angle camera 220 by referring to the corresponding video image among the wide-angle and narrow-angle video images, and each corresponding conversion rule satisfies the infinite vanishing point condition. Equations 1 to 3 below may be used in order to update.

Hereinafter, the process of updating the corresponding camera conversion rule to satisfy the infinite vanishing point condition using Equations 1 to 3 is the same in the wide-angle camera 210 and the narrow-angle camera 220, so that the conversion rule for the camera is collectively referred to below. It will be described together as a calibration process.

Here, x _i denotes the coordinates of the vanishing point in the captured image image, and x _v denotes the coordinates of the vanishing point in the real world (top view image) in a three-dimensional space. K is an intrinsic parameter determined by the camera itself, such as the focal length of the camera, and R is a rotation matrix and rotation matrices (R _x , R _y , which are rotation angles in each axis direction of the camera) It _{is calculated as the product of R z} ), and T is a translation matrix that is the distance from the origin. Here, x _i , K and T can be said to be known values.

The camera correction unit 234 is a rotation matrix R corresponding _{to the rotation angle θ 0} in each axial direction at a reference point of view (eg, a time when the camera is first mounted or a previous rotation angle calculation time, etc.) And Equations 1 and 2 can be used to calculate _{the coordinates (x v} ) of the vanishing point in the real world in 3D space.

Subsequently, the camera correction unit 234 can calculate the rotation angle of the camera by Equation 3 below using the _{coordinates (x v} ) of the vanishing point in the real world in the three-dimensional space calculated using Equations 1 and 2. have.

Here, θ is the calculated rotation angle of the camera,

Is

It is calculated as the direction component value, and v _u is [0 1 0] ^T.

The camera correction unit 234 determines whether or not the calculated rotation angle θ of the camera is equal to or less than a predetermined threshold. This is the direction component value

This is because as the dot product of and v _u approaches 0, it can be determined that the infinite vanishing point condition is satisfied.

If the calculated rotation angle θ of the camera satisfies a predetermined threshold value (ie, the coordinates of the vanishing point are infinite), the camera correction unit 234 determines that the conversion rule for the current camera is appropriate.

However, if the calculated rotation angle θ of the camera is equal to or greater than a predetermined threshold (that is, the coordinates of the vanishing point are not infinity), the camera correction unit 234 determines the rotation angle θ ₀ and the above-described _{Calculate the rotation matrix R as the rotation angle (i.e., θ 1} =θ ₀ +θ) that is the sum of the rotation angle θ calculated in the process, and recalculate using the calculated rotation matrix and Equations 1 and 3 It is determined whether or not the rotation angle θ ₂ of the camera is satisfied with a predetermined threshold.

The above-described process is repeatedly performed until the calculated rotation angle θ of the camera satisfies a predetermined threshold. That is, the process of calculating the camera rotation angle θ using equations 1 and 3 and a rotation matrix calculated by summing all the camera rotation angles calculated in each step and the camera rotation angle at the above-described reference point of view is prescribed in advance. Repeat until the infinite vanishing point condition is satisfied.

This is because the correct rotation angle of the camera that satisfies the infinite vanishing point condition may not be calculated at once due to an error, so this is performed repeatedly to converge to an appropriate value.

Thereafter, when the camera rotation angle that satisfies the infinite vanishing point condition is calculated, the camera correction unit 234 updates the conversion rule for the corresponding camera to a conversion rule corresponding to the finally calculated camera rotation angle, and the conversion rule storage unit ( 240).

The wide-angle video image captured by the wide-angle camera 210 according to each conversion rule updated by the above-described process is converted into a first top-view image (refer to (a) of FIG. 4), and photographed by the narrow-angle camera 220 The narrow angle image image may be converted into a second top view image (refer to FIG. 4B). For reference, reference numeral 310a denotes a lane photographed by the wide-angle camera 210 (hereinafter, referred to as a wide-angle lane), and 310b denotes a lane photographed by the narrow-angle camera 220 (hereinafter, referred to as a narrow-angle lane). .

As illustrated in FIGS. 4A and 4B, respectively, the first top-view image is generated by the wide-angle camera 210 and has a characteristic specialized for a short-range image, and the second top-view image is a narrow-angle camera 220 It is produced by and has features specialized for distant images.

The wide-angle lane 310a of the first top-view image may be shown to have a relatively wide width compared to the narrow-angle lane 310b of the second top-view image. Therefore, when the first top-view image and the second top-view image overlap, As illustrated in (c) of FIG. 4, the widths of the lane areas are displayed in a manner that does not match each other.

Therefore, in order to allow the wide-angle camera 210 and the narrow-angle camera 220 to function as a stereo camera, the matching unit 236 processes the wide-angle lane 310a and the narrow-angle lane 310b to be displayed with the same lane width. It can be (see Fig. 4(d)).

That is, the matching unit 236 overlaps the first top-view image and the second top-view image, and updates the conversion rule of the wide-angle camera 210 or the narrow-angle camera 220 so that the lane images displayed in each top-view image match each other. . To this end, the following Equations 4 to 9 may be used.

Here, x denotes one point of the narrow-angle lane, x'denotes one point of the wide-angle lane, and x and x'have the same relationship as in Equation 5. Each of x and x'may be points on a straight line shown to intersect perpendicularly to the lane. K is an internal parameter of the camera itself,

Is the difference between the camera rotation angle of the wide-angle camera 210 and the narrow-angle camera 220 (

Represents a rotation function corresponding to ).

Equation 5 can be simplified as in Equation 6 below, and y and y'in Equation 6 can be expressed as Equation 7, and if Equation 6 is expressed as a matrix using Equation 7, Equation 8 It can be expressed as, Equation 8 can be simplified as Equation 9.

The matching unit 236 is the rotation angle of any one of the wide-angle camera 210 and the narrow-angle camera 220 calculated to meet the infinite vanishing point condition by the camera correction unit 234 (for example, when the wide-angle camera is referenced, θ_wide angle) and summing the difference in rotation angle to the other rotation angle (for example, θ_narrow angle+

) And applied, the difference between each camera rotation angle using Equations 4 to 9 described above until the widths of the wide-angle lane 310a and the narrow-angle lane 310b become the same (

) Is repeatedly calculated. In the process of being repeatedly performed, all differences between the rotation angles calculated up to the previous step are summed (for example, θ_narrow angle + △

Applied) and applied.

In other words, the difference between x and x'is the angle parameter (

), and the difference between each camera rotation angle (

) Represents a modified angle from the existing installation angle, and the rotation function (R matrix) is changed by repeated processing until the widths of the wide-angle lane 310a and the narrow-angle lane 310b become the same, resulting in a corresponding The camera's calibration result is changed.

As described above, after the conversion rules are updated so that the lanes of the wide-angle and narrow-angle video images respectively photographed by the wide-angle camera 210 and the narrow-angle camera 220 satisfy the infinite vanishing point condition, they correspond to the wide-angle video image. When any one of the conversion rules is updated so that the width of the wide-angle lane 310a and the narrow-angle lane 310b corresponding to the narrow-angle image image are updated, the wide-angle camera 210 and the narrow-angle camera 220 are composed of heterogeneous stereo cameras. Can be used.

As described in detail with reference to the related drawings so far, the heterogeneous stereo camera system and the camera correction method according to the present embodiment use the lane images taken by the wide-angle cameras and narrow-angle cameras respectively arranged to photograph the same direction. There is an advantage of being able to quickly perform a stereo camera correction process by acquiring and matching 3D information.

Although the above has been described with reference to embodiments of the present invention, those of ordinary skill in the art can variously modify the present invention within the scope not departing from the spirit and scope of the present invention described in the following claims. And it will be appreciated that it can be changed.

Claims (2)

1. A wide-angle lane is detected from a wide-angle video image captured by a wide-angle camera using a predetermined lane detection technique. An edge detection unit for detecting a narrow-angle lane in a narrow-angle video image captured by a narrow-angle camera;

   A camera beam that converts the wide-angle video image and the narrow-angle video image into a top-view image and updates conversion rules for the wide-angle camera and the narrow-angle camera so that each of the wide-angle lane and the narrow-angle lane satisfies a predefined vanishing point infinite condition. government; And

   Further updating a conversion rule for at least one of the wide-angle camera and the narrow-angle camera so that the width of the wide-angle lane and the narrow-angle lane converted into a top-view image using an updated conversion rule to satisfy the infinite vanishing point condition Heterogeneous stereo camera system including a matching unit.
2. The method of claim 1,

   The camera correction unit is a rotation matrix (R) according to an angle obtained by accumulating the rotation angle θ of the camera calculated immediately before until the rotation angle θ of the camera calculated using the following equations satisfies the infinite vanishing point condition. Repeat the process of calculating the rotation angle (θ) of the camera again using,

   The above equations are:

   

   

   Is,

   Wherein x _i is the coordinates of the vanishing point in the captured video image, x _v is the coordinates of the vanishing point in the top view image, K is an internal parameter of the camera, and R is a rotation matrix according to the rotation angle in each axis direction of the camera Is a rotation matrix calculated by the product of _{R x} , R _y and R _{z, and T is a translation matrix,}

   

   Is

   

   A heterogeneous stereo camera system, characterized in that it is a direction component value calculated as, and v _u is [0 1 0] ^T.

Images