KR102232276B1 - Apparatus and method for generating AVM image in trailer truck - Google Patents

Abstract

The present invention relates to an apparatus for providing an around view image for a trailer truck, which is a method for providing an AVM image of an AVM image providing apparatus for a trailer truck having front cameras and rear cameras distributed on each of a tractor and a trailer of the trailer truck, Generating a front image and a rear image repeatedly using the front cameras and the rear cameras while varying the tractor rotation angle; After deriving the front coordinate system from the first or second front image, the rear reference coordinate system from the first rear image, and the rear comparison coordinate system from the second rear image, respectively, the rotation center between the front image and the rear image from the geometric correlation between the coordinate systems Deriving an axis; If a front image and a rear image are newly acquired as the trailer truck is operated, overlaying the new front image and the rear image based on a rotation center axis between the front image and the rear image; And obtaining and outputting an AVM image by rotating any one of a front image and a rear image around the rotation center axis whenever the tractor rotation angle is changed, and stitching the image.

Description

BACKGROUND OF THE INVENTION 1. Apparatus and method for generating AVM image in trailer truck}

The present invention relates to an apparatus and method for providing an around view image for a trailer truck to generate and provide an around view image (AVM) that does not contain error information in a trailer truck in which a tractor and a trailer are connected separately. About.

Due to the recent development of the automobile industry, the spread of automobiles has been commercialized to the extent that it is the era of one automobile for one household, and various advanced electronic technologies are applied to automobiles in order to improve the safety of tractors and promote the convenience of the driver.

Among these advanced electronic technologies, there is an Around View Monitoring (AVM) for trailer trucks that allows the driver to conveniently visually check the surrounding environment of the vehicle by photographing and displaying the surrounding environment of the vehicle.

The tractor surrounding image display system captures the surrounding environment through cameras installed at the front, rear, left and right sides of the vehicle, and generates an around view image by correcting the overlapped area to look natural based on the captured image. This is provided to a display device provided in a vehicle.

Accordingly, the driver can accurately recognize the surrounding situation of the vehicle through the displayed surrounding environment, and can conveniently park without looking at the side or rearview mirror.

However, when the AVM is applied to a trailer truck in which the tractor (V1) and the trailer (V2) are connected separately, as shown in FIG. 1, when the trailer truck rotates, the tractor (V1) and the trailer (V2) of the trailer truck are separated. As a result, a camera viewing angle combined area between the camera disposed on the tractor V1 and the camera disposed on the trailer V2 is generated.

Then, the same image information is duplicated in the camera field of view, and when multiple camera images are combined into one AVM image in the future, an additional problem occurs in that the actual trailer part is visible or the rear image appears strange. .

In order to solve the above problems, an around view image for trailer trucks is provided that enables generation and provision of AVM images for trailer trucks that prevent separate error information from being included even if the tractor and trailer of the trailer truck are rotated separately. It is intended to provide an apparatus and method.

The object of the present invention is not limited to the above-mentioned object, and other objects not mentioned will be clearly understood by those of ordinary skill in the art from the following description.

As a means for solving the above problem, an AVM image providing method of an AVM image providing apparatus for a trailer truck having front cameras and rear cameras distributedly disposed on each of a tractor and a trailer of a trailer truck according to an embodiment of the present invention , Repeatedly generating a front image and a rear image using the front cameras and the rear cameras while varying the tractor rotation angle; After deriving the front coordinate system from the first or second front image, the rear reference coordinate system from the first rear image, and the rear comparison coordinate system from the second rear image, respectively, the rotation center between the front image and the rear image from the geometric correlation between the coordinate systems Deriving an axis; If a front image and a rear image are newly acquired as the trailer truck is operated, overlaying the new front image and the rear image based on a rotation center axis between the front image and the rear image; And obtaining and outputting an AVM image by rotating any one of a front image and a rear image around the rotation center axis whenever the tractor rotation angle is changed, and stitching the image.

”와 “

”로 정의되며, 상기 F는 전방 좌표계이고, 상기 B는 후방 기준 좌표계이고, 상기 B'는 후방 비교 좌표계이고, 상기 R 는 후방 기준 좌표계에 대응되는 회전축 좌표계이고, 상기 R'은 후방 비교 좌표계에 대응되는 회전축 좌표계이고, 상기 τ_i ^j 는 {i} 좌표계 기준으로 {j} 좌표계의 자세를 기술한 동차변환행렬(homogeneous transformation matrix, HTM)인 것을 특징으로 한다. The geometric correlation between the coordinate systems is “

"Wow "

”, wherein F is a front coordinate system, B is a rear reference coordinate system, B'is a rear comparison coordinate system, R is a rotation axis coordinate system corresponding to a rear reference coordinate system, and R'is a rear comparison coordinate system. It is a corresponding rotation axis coordinate system, and τ _i ^j is a homogeneous transformation matrix (HTM) describing a posture of the {j} coordinate system based on the {i} coordinate system.

벡터와 후방 좌표계와 회전 좌표계간의 위치 관계를 나타내는 기준

벡터를 도출하는 단계; 및 상기 기준

벡터와 상기 기준

벡터 중 어느 하나를 전방 영상과 후방 영상간 회전 중심축으로 설정하는 단계를 포함하는 것을 특징으로 하며, 상기 기준

벡터는 “

”으로 정의되고, 상기 기준

벡터는 “

”으로 정의되며, 상기 R_i ^j 는 {i} 좌표계 기준으로 {j} 좌표계의 방향을 기술한 회전행렬(rotation matrix)이고, 상기 t_i ^ij 는 {i} 기준

벡터, T는 전치 행렬(transpose matrix) 연산자인 것을 특징으로 한다. The step of deriving the rotation center axis between the front image and the rear image is a reference indicating the positional relationship between the front coordinate system and the rotation coordinate system from the geometric correlation between the coordinate systems.

A criterion representing the positional relationship between the vector and the rear coordinate system and the rotational coordinate system.

Deriving a vector; And the above criteria

Vector and above criteria

And setting any one of the vectors as a central axis of rotation between the front image and the rear image, wherein the reference

The vector is “

Defined as ”, and the criteria above

The vector is “

”, wherein R _i ^j is a rotation matrix describing the direction of the {j} coordinate system based on the {i} coordinate system, and the t _i ^ij is based on the {i}

The vector, T, is characterized in that it is a transpose matrix operator.

The acquiring and outputting the AVM image may further include acquiring the tractor rotation angle through one of an image processing method and a sensor sensing method.

As a means for solving the above problem, according to another embodiment of the present invention, front cameras and rear cameras distributedly disposed on each of a tractor and a trailer of a trailer truck; A front image generator for synthesizing images captured by the front cameras to generate a front image; A rear image generator for synthesizing images captured by the rear cameras to generate a rear image; A rotation center axis calculator configured to calculate a rotation center axis between a front image and a rear image by analyzing a positional relationship change pattern after receiving a plurality of front and rear images obtained in a state in which the rotation angle of the tractor is different; And an AVM image generator for generating and providing an AVM image by overlaying the rear image on the front image, rotating any one of the front image and the rear image around a rotation center axis by the tractor rotation angle, and then performing image stitching. It provides an around view image providing apparatus for a trailer truck including.

In the present invention, even if a camera viewing angle combined area occurs as the tractor and trailer of the trailer truck are rotated separately, the actual trailer part is visible in the AVM image or the rear image is strange by accurately grasping the location of the camera viewing angle combined area and then editing the image. Prevent the occurrence of problems that come out beforehand. That is, it is possible to obtain and provide an AVM image that does not contain error information.

1 is a view showing an example of an around view image for a trailer truck according to the prior art.
2 is a diagram illustrating an apparatus for providing an around view image for a trailer truck according to an embodiment of the present invention.
3 is a view for explaining a method of deriving a rotation center axis between a front image and a rear image according to an embodiment of the present invention.
4 is a view for explaining a method for providing an around view image for a trailer truck according to an embodiment of the present invention.

Objects and effects of the present invention, and technical configurations for achieving them will become apparent with reference to the embodiments described below in detail together with the accompanying drawings. In describing the present invention, if it is determined that a detailed description of a known function or configuration may unnecessarily obscure the subject matter of the present invention, a detailed description thereof will be omitted.

In addition, terms to be described later are terms defined in consideration of functions in the present invention, which may vary according to the intention or custom of users or operators.

However, the present invention is not limited to the embodiments disclosed below, and may be implemented in various different forms. The present embodiments are provided only to make the disclosure of the present invention complete, and to fully inform the scope of the invention to those skilled in the art to which the present invention pertains, and the present invention is defined by the scope of the claims. It just becomes. Therefore, the definition should be made based on the contents throughout the present specification.

2 is a diagram illustrating an apparatus for providing an around view image for a trailer truck according to an embodiment of the present invention.

Referring to FIG. 2, the apparatus 200 for providing an around view image for a trailer truck includes a plurality of front cameras 111 to 113, a plurality of rear cameras 121 to 123, a front image generator 130, and a rear image generator. 140, a rotation center axis calculation unit 150, and an AVM image generation unit 160, and the like.

The plurality of front cameras 111 to 113 are distributedly disposed on the tractor V1 of the trailer truck to acquire and provide a plurality of images photographing each of the front of the vehicle, the left in front of the vehicle, and the right in front of the vehicle.

The plurality of rear cameras 121 to 123 are distributedly arranged on the trailer V2 of the trailer truck to obtain and provide a plurality of images photographing the rear of the vehicle, the left of the rear of the vehicle, and the right of the rear of the vehicle.

The front image generator 130 acquires and stores calibration information (inside and outside of the camera parameters) of each of the front cameras 111 to 113 in advance, and based on this, a plurality of images acquired through the front cameras 111 to 113 are converted into a vehicle. By synthesizing in an around view method, a single front image is generated and output.

The rear image generation unit 140 acquires and stores calibration information (inside and outside the camera parameters) of each of the rear cameras 121 to 123 in advance, and based on this, a plurality of images acquired through the rear cameras 121 to 123 are converted into a vehicle. By synthesizing in an around view method, a single rear image is generated and output.

The rotation center axis calculating unit 150 receives a plurality of front and rear images obtained in a state in which the tractor rotation angle is different, and then analyzes the pattern of change in the positional relationship of these images to determine the rotation center axis between the front and rear images. Calculate.

After overlaying the rear image on the front image, the AVM image generator 160 rotates either the front image or the rear image by the tractor rotation angle around the rotation center axis. Then, by removing redundant images through image stitching, the AVM image optimized for the trailer truck is finally generated and provided.

In this case, the tractor rotation angle can be obtained through an image processing method or based on a sensor, and a detailed description thereof will be provided below.

Hereinafter, a method of providing an around view image for a trailer truck according to an embodiment of the present invention will be described in more detail with reference to FIGS. 3 and 4.

3 is a view for explaining a method of deriving a rotation center axis between a front image and a rear image according to an embodiment of the present invention.

First, as shown in (a) of FIG. 3, the tractor V1 is maximally rotated in a first direction (eg, right), and then a front image and a rear image are first acquired.

And, as shown in (b) of FIG. 3, after the tractor V1 is rotated in a second direction opposite to the first direction (for example, left), the front image and the rear image are secondarily acquired. . In this case, the tractor rotation angle may be acquired based on a sensor or may be manually input by a user.

And, as shown in (c) of FIG. 3, the front coordinate system {F} from one of the first front image and the second front image, the rear reference coordinate system {B} from the first rear image, and the rear comparison coordinate system from the second rear image. Each of {B'} is derived.

And the rotation axis coordinate system corresponding to the rear reference coordinate system {B} (that is, the rotation axis coordinate system fixed to the first rear image) as {R}, and the rotation axis coordinate system corresponding to the rear comparison coordinate system {B'} (that is, fixed to the second rear image). After the rotation, the rotation axis coordinate system) is set to {R'}. In this case, it is assumed that the z-axis of all coordinate systems faces the ground direction.

벡터와, 수학식 5로 정의되는 후방 좌표계와 회전 좌표계간의 위치 관계를 나타내는 기준

벡터를 최종 도출하도록 한다. Then, the geometric correlation between these coordinate systems is expressed by Equations 1 and 2, and in the present invention, a reference representing the positional relationship between the forward coordinate system and the rotational coordinate system defined by Equation 4 from Equations 1 and 2

A reference indicating the positional relationship between the vector and the rear coordinate system and the rotation coordinate system defined by Equation 5

The vector is finally derived.

[Equation 1]

[Equation 2]

In this case, τ _i ^j is a homogeneous transformation matrix (HTM) describing the posture of the {j} coordinate system based on the {i} coordinate system.

[Equation 3]

벡터, 즉 {i} 좌표계의 원점에서 {j} 좌표계 원점까지의 벡터를 기술한 위치 벡터(translation vector)이다. At this time, R _i ^j is a rotation matrix describing the direction of the {j} coordinate system based on the {i} coordinate system, and t _i ^ij is the {i} reference

It is a vector, that is, a translation vector describing a vector from the origin of the {i} coordinate system to the origin of the {j} coordinate system.

[Equation 4]

[Equation 5]

In more detail, the rotation center axis of Equations 4 and 5 is derived through the following process.

First, Equations 1 and 2 may be converted into Equations 6 and 7.

[Equation 6]

[Equation 7]

Meanwhile, since the geometric relationship between {B'} and {R'} and the geometric relationship between {B} and {R} are the same, Equation 8 can be derived from the above equations.

[Equation 8]

벡터가 도출된다. And if Equation 8 is summarized as in Equation 9, finally, the reference corresponding to the rotational central axis in terms of the front image of Equation 4 described above

The vector is derived.

[Equation 9]

In this case, T denotes a transpose matrix operator.

벡터 또한 도출된다.In addition, by arranging Equations 6 and 8 as in Equation 10, the reference corresponding to the rotation center axis in terms of the rear image defined by Equation 5

Vectors are also derived.

[Equation 10]

벡터와 기준

벡터를 중 어느 하나를 전방 영상과 후방 영상간의 회전 중심축으로 결정하도록 한다. Then, the standard

Vectors and standards

One of the vectors is determined as the rotation center axis between the front image and the rear image.

벡터를 전방 영상과 후방 영상간의 회전 중심축으로 결정한 경우에는, 전방 영상을 트랙터 회전 각도에 따라 회전시키고, 기준

벡터를 전방 영상과 후방 영상간의 회전 중심축으로 결정한 경우에는, 후방 영상을 트랙터 회전 각도에 따라 회전시킴으로써, 트럭용 AVM을 최종 생성할 수 있도록 한다. If, standard

When the vector is determined as the rotation center axis between the front image and the rear image, the front image is rotated according to the tractor rotation angle, and the reference

When the vector is determined as the rotation center axis between the front image and the rear image, the rear image is rotated according to the tractor rotation angle, so that the truck AVM can be finally generated.

4 is a diagram illustrating a method of generating an around view image for a trailer truck according to an embodiment of the present invention.

벡터와 기준

벡터로부터 {F} 좌표계, {B} 좌표계, {R} 좌표계간의 위치 관계를 파악하고, 이를 고려하여 전방 영상 위에 후방 영상(또는 후방 영상 위에 전방 영상)을 오버레이시킨다. First, as shown in Fig. 4 (a), the reference

Vectors and standards

The positional relationship between the {F} coordinate system, the {B} coordinate system, and the {R} coordinate system is determined from the vector, and the rear image (or the front image on the rear image) is overlaid on the front image in consideration of this.

벡터로부터 {F} 좌표계와 {R} 좌표계간의 위치 관계, 기준

벡터로부터 {B} 좌표계와 {R} 좌표계간의 위치 관계를 파악한다. 그리고 상기 위치 관계를 유지하되, 기준

벡터에 대응되는 {R} 좌표계의 원점과 기준

벡터에 대응되는 {R} 좌표계의 원점이 일치되도록, 전방 영상 위에 후방 영상(또는 후방 영상 위에 전방 영상)을 오버레이시킨다. More specifically, the criteria

The positional relationship between the {F} coordinate system and the {R} coordinate system from the vector, the reference

Find the positional relationship between the {B} coordinate system and the {R} coordinate system from the vector. And maintaining the above positional relationship, but the standard

Origin and reference of the {R} coordinate system corresponding to the vector

The rear image (or the front image on the rear image) is overlaid on the front image so that the origin of the {R} coordinate system corresponding to the vector is matched.

And whenever the tractor rotation angle is changed, as shown in FIG. 4(b), one of the front image and the rear image is rotated around the rotation center axis by the changed tractor rotation angle.

In this case, the tractor rotation angle is a relative rotation angle of the tractor with respect to the trailer, and can be obtained by image processing or based on a sensor.

More specifically, in the image processing method, when the trailer truck is rotated left and right, as the tractor and the trailer are rotated separately, the camera viewing angles of the front and rear cameras overlap, and as a result, the same image information is overlapped in the front and rear images. Considering that there is a feature to be photographed, a feature point of each of the front image and the rear image is extracted, and then the tractor rotation angle is inversely extracted through the positional relationship thereof.

On the other hand, the rotation angle sensor is installed at any one of the rear portion of the tractor V1 and the front portion of the trailer V2, and senses and reports the distance between the rear portion of the vehicle V1 and the front portion of the trailer V2. And outputs the sensing distance value by converting the relative rotation angle of the tractor V1 with respect to the trailer V2.

That is, according to the present invention, after searching for a camera viewing angle overlap region through a tractor rotation angle, two image information belonging to the camera viewing angle overlap region is overlaid on the same position.

Then, as shown in (c) of FIG. 4, the front image and the rear image are combined into one image, but the final AVM image is generated by performing image stitching in which one of the two image information of the overlay area is removed. And output.

As a result, there is no problem that the actual trailer part is visible in the final AVM image or the same image information is displayed redundantly.

The above description is merely illustrative of the technical idea of the present invention, and those of ordinary skill in the art to which the present invention pertains will be able to make various modifications and variations without departing from the essential characteristics of the present invention. Accordingly, the embodiments disclosed in the present invention are not intended to limit the technical idea of the present invention, but to explain the technical idea, and the scope of the technical idea of the present invention is not limited by these embodiments. The scope of protection of the present invention should be interpreted by the following claims, and all technical ideas within the scope equivalent thereto should be construed as being included in the scope of the present invention.

Claims (5)

In the AVM image providing method of an AVM image providing apparatus for a trailer truck having front cameras and rear cameras distributedly arranged on a tractor and a trailer of a trailer truck, respectively,
Generating a front image and a rear image repeatedly using the front cameras and the rear cameras while varying the tractor rotation angle;
After deriving the front coordinate system from the first or second front image, the rear reference coordinate system from the first rear image, and the rear comparison coordinate system from the second rear image, respectively, the rotation center between the front image and the rear image from the geometrical correlation between the coordinate systems Deriving an axis;
If a front image and a rear image are newly acquired as the trailer truck is operated, overlaying the new front image and the rear image based on a rotation center axis between the front image and the rear image; And
Whenever the tractor rotation angle changes, rotating any one of a front image and a rear image around the rotation center axis and then stitching the image to obtain and output an AVM image,
The geometric correlation between the coordinate systems is
“

"Wow "

”, wherein F is a front coordinate system, B is a rear reference coordinate system, B'is a rear comparison coordinate system, R is a rotation axis coordinate system corresponding to a rear reference coordinate system, and R'is a rear comparison coordinate system. A corresponding rotation axis coordinate system, and τ _i ^j is a homogeneous transformation matrix (HTM) describing the posture of the {j} coordinate system based on the {i} coordinate system,
The step of deriving a rotation center axis between the front image and the rear image
Criteria representing the positional relationship between the forward coordinate system and the rotational coordinate system from the geometric correlation between the coordinate systems

A criterion representing the positional relationship between the vector and the rear coordinate system and the rotational coordinate system.

Deriving a vector; And
Above criteria

Vector and above criteria

Including the step of setting any one of the vectors as a rotation center axis between the front image and the rear image,
Above criteria

The vector is “

Defined as ”, and the criteria above

The vector is “

”, wherein R _i ^j is a rotation matrix describing the direction of the {j} coordinate system based on the {i} coordinate system, and the t _i ^ij is based on the {i}

A method for providing an around view image for a trailer truck, characterized in that the vector and T are a transpose matrix operator. delete delete The method of claim 1, wherein the obtaining and outputting the AVM image comprises:
The method of providing an around view image for a trailer truck, further comprising the step of obtaining the tractor rotation angle through one of an image processing method and a sensor sensing method. Front cameras and rear cameras distributed in each of the tractor and trailer of the trailer truck;
A front image generator for synthesizing images captured by the front cameras to generate a front image;
A rear image generator for synthesizing images captured by the rear cameras to generate a rear image;
A rotation center axis calculator configured to calculate a rotation center axis between a front image and a rear image by analyzing a positional relationship change pattern after receiving a plurality of front and rear images obtained in a state in which the rotation angle of the tractor is different; And
After overlaying the rear image on the front image, the AVM image generator generates and provides an AVM image by rotating any one of the front image and the rear image around the rotation center axis by the tractor rotation angle, and then performing image stitching. And
The geometric correlation between the coordinate systems is
“

"Wow "

”, wherein F is a front coordinate system, B is a rear reference coordinate system, B'is a rear comparison coordinate system, R is a rotation axis coordinate system corresponding to a rear reference coordinate system, and R'is a rear comparison coordinate system. A corresponding rotation axis coordinate system, and τ _i ^j is a homogeneous transformation matrix (HTM) describing the posture of the {j} coordinate system based on the {i} coordinate system,
The rotation center axis calculation unit
Criteria representing the positional relationship between the forward coordinate system and the rotational coordinate system from the geometric correlation between the coordinate systems

A criterion representing the positional relationship between the vector and the rear coordinate system and the rotational coordinate system.

After deriving the vector, the above criteria

Vector and above criteria

One of the vectors is set as the central axis of rotation between the front image and the rear image,
Above criteria

The vector is “

Defined as ”, and the criteria above

The vector is “

”, wherein R _i ^j is a rotation matrix describing the direction of the {j} coordinate system based on the {i} coordinate system, and the t _i ^ij is based on the {i}

Vector, T is an around-view image providing apparatus for a trailer truck, characterized in that the transpose matrix (transpose matrix) operator.

Images