KR102583959B1 - method of correcting distortion in 3D SVM video by use of adjustment of video composition portion of boundary line corresponding to 3D viewpoint - Google Patents

Abstract

The present invention generally relates to 3D surround view monitoring (3D SVM) technology for vehicles. In particular, the present invention relates to a technology for correcting distortion of 3D SVM images by adaptively adjusting the borderline composite area ratio between multi-channel camera images for generating 3D SVM images according to the virtual camera position at the 3D view point. According to the present invention, obstacles around a vehicle can be expressed without distortion in a 3D SVM image, which has the advantage of improving the vehicle driver's obstacle recognition ability. In addition, according to the present invention, there is an advantage in preventing traffic accidents because the vehicle driver can intuitively recognize the situation around the vehicle.

Description

3D SVM image distortion correction method by controlling the boundary composition area according to the 3D view point {method of correcting distortion in 3D SVM video by use of adjustment of video composition portion of boundary line corresponding to 3D viewpoint}

The present invention generally relates to 3D surround view monitoring (3D SVM) technology for vehicles.

In particular, the present invention relates to a technology for correcting distortion of 3D SVM images by adaptively adjusting the borderline composite area ratio between multi-channel camera images for generating 3D SVM images according to the virtual camera position at the 3D view point.

Recently, Surround View Monitoring (SVM) systems have been introduced to automobiles. Surround view monitoring (SVM) is also called around view monitoring (AVM). Cameras are installed on the front, left, and right sides of the vehicle, and the images from these cameras are combined to create a top view or bird view. It is a technology that provides a bird’s eye view. Drivers can understand the situation around the vehicle as if looking down from above, making driving and parking more convenient.

[Figure 1] is a diagram showing the concept of automotive surround view monitoring (SVM) technology. As shown in [Figure 1], cameras (11 to 14) are mounted on the front and rear of the vehicle and on both side mirrors, and the unit camera images (15 to 18) provided from these multi-channel camera devices (11 to 14) are displayed. After applying image improvement and distortion correction (calibration) to create a flat image, a surround view image (19) in the form of a bird's eye view (top view) is obtained through stitching (image registration and synthesis) processing. This surround view image 19 is provided to the driver through a monitor inside the vehicle.

[Figure 2] is an example diagram showing a 2D and 3D surround view monitoring (SVM) screen. [Figure 2] (a) is a 2D SVM image, where the left side is a rear camera image and the right side is a bird's eye view (top view) image. In the 2D SVM image, two adjacent camera images are shown semi-transparently overlapping in the composite portion of the boundary between the front and rear camera images (15, 16) and the bilateral camera images (17, 18). The border line appears in a diagonal shape, and areas of about 5 to 10 degrees overlap. In 2D SVM images, distortion due to image overlap is not noticeable. [Figure 2] (b) is a 3D SVM image. 3D SVM provides a composite image from a stereoscopic (3D) perspective, allowing the driver to more accurately understand the situation around the vehicle.

[Figure 3] is a diagram showing the concept of 3D SVM technology. The 3D SVM system generates a three-dimensional image by projecting multi-channel camera images (15 to 18) onto a regular 3D projection surface in the form of a cylinder or bowl and places a 3D vehicle model at the center. [FIG. 3] (a) and (b) are examples of using a bowl-shaped 3D projection surface, and [FIG. 3] (c) shows a cylinder-shaped 3D projection surface.

[Figure 4] is an example diagram showing problems with general 3D SVM images. When observing the rear side area of the vehicle in [Figure 4], it can be seen that image distortion appears prominently along the diagonal line. In the 3D SVM image, like the 2D SVM image, the front and rear camera images (15, 16) and the bilateral camera images (17, 18) are shown overlapping translucently along the border. However, unlike 2D SVM images, 3D SVM images display both the floor and sides in three-dimensional (3D) space, so the image overlap area near the boundary line widens in proportion to the distance from the camera. For this reason, when obstacles around the vehicle exist in the boundary composite area, significant image distortion occurs in the process of overlapping the front and rear camera images (15, 16) and both side camera images (17, 18).

As such, in the conventional 3D SVM image, large image distortion occurred in the border composite portion between the front and rear camera images (15, 16) and both side camera images (17, 18), making it difficult for the driver to recognize obstacles around the vehicle.

Republic of Korea Patent Publication No. 10-2018-0001869 “Image improvement device and method for AVM system” Republic of Korea Patent Publication No. 10-2017-0111504 “AVM system around image consistency evaluation method and device”

The purpose of the present invention is to generally provide 3D surround view monitoring (3D SVM) technology for vehicles.

In particular, the purpose of the present invention is to provide a technology for correcting distortion of 3D SVM images by adaptively adjusting the ratio of borderline composite areas between multi-channel camera images to generate 3D SVM images according to the virtual camera position at the 3D view point. It is done.

The problem to be solved by the present invention is not limited to this matter, and other problems to be solved can be understood from the description in this specification.

In order to achieve the above objective, the present invention proposes a method for correcting distortion of 3D SVM images by adaptively adjusting the boundary composite area between multi-channel camera images according to the virtual camera position at the 3D view point of the vehicle 3D SVM system. .

The 3D SVM image distortion correction method through border composite area control according to the 3D view point according to the present invention is a first method of acquiring multi-channel camera images (15 to 18) from multi-channel camera devices (11 to 14) installed in a vehicle. step; A second step of acquiring the virtual camera position of the 3D view point for the 3D SVM image; A third step of setting the area ratio of overlapping multi-channel camera images (15 to 18) corresponding to the virtual camera position for each border composite area (21 to 24) of the 3D SVM image; A fourth step of projecting multi-channel camera images 15 to 18 on the 3D projection surface corresponding to the area ratio set above for each border composite area.

In the present invention, the third step is to preset the area ratio of the overlapping side camera images 17 and 18 for each border composite area 21 to 24 when the virtual camera position is the front or rear of the vehicle. setting the area ratio of the overlapping front and rear camera images 15 and 16 to a maximum value and setting the area ratio of the overlapping front and rear camera images 15 and 16 to a preset minimum value; When the virtual camera position is on the side of the vehicle, the area ratio of the overlapping front and rear camera images 15 and 16 for each border composite area 21 to 24 is set to a preset maximum value, and the overlapping side area is set to a preset maximum value. Setting the area ratio of the camera images 17 and 18 to a preset minimum value; When the virtual camera position is located in the middle rather than the front, rear, or side of the vehicle, the area ratio of the front and rear camera images (15, 16) and the side camera images (17, 18) that overlap each other by interpolation according to the virtual camera position. It may be configured to include; setting a .

In the present invention, the third step is to configure multi-channel video in proportion to the degree to which the virtual camera position and the mounting position of the multi-channel camera devices 11 to 14 are close to orthogonal for each border composite area 21 to 24 of the 3D SVM image. It may be configured to include: setting the area ratio of the camera images 15 to 18.

Meanwhile, the computer program according to the present invention is stored in a non-volatile storage medium in order to execute the 3D SVM image distortion correction method through controlling the border composite area according to the 3D view viewpoint on the computer as described above.

According to the present invention, obstacles around a vehicle can be expressed without distortion in a 3D SVM image, which has the advantage of improving the vehicle driver's obstacle recognition ability.

In addition, according to the present invention, there is an advantage in preventing traffic accidents because the vehicle driver can intuitively recognize the situation around the vehicle.

[Figure 1] is a diagram showing the concept of automotive surround view monitoring (SVM) technology.
[Figure 2] is an example diagram showing typical 2D and 3D SVM images.
[Figure 3] is a diagram showing the concept of 3D SVM technology.
[Figure 4] is an example diagram showing problems with general 3D SVM images.
[Figure 5] is an example of observing image distortion from a lateral perspective in a typical 3D SVM image.
[Figure 6] is an example of observing image distortion from a front view in a typical 3D SVM image.
[Figure 7] is a flowchart showing the 3D SVM image distortion correction process according to the present invention.
[Figure 8] is a conceptual diagram of setting the area ratio of a multi-channel camera image in the present invention.

Hereinafter, the present invention will be described in detail with reference to the drawings.

Comparing [Figure 2] and [Figure 4], the image distortion in the area where the front and rear camera images (15, 16) and the bilateral camera images (17, 18) overlap is less in the 2D SVM image, whereas in the 3D SVM image, It was confirmed that it appeared prominently. This is because, unlike 2D SVM images, 3D SVM images display both the floor and sides in three-dimensional (3D) space, so the image overlap area near the boundary line widens in proportion to the distance from the camera. Because of this, in 3D SVM images, image distortion in the overlapping area increases as the distance from the camera increases.

However, image distortion in 3D SVM images has the characteristic that the degree of distortion appears differently depending on the camera image at each viewpoint. The present invention utilizes these characteristics to reduce overall distortion of 3D SVM images. In other words, the overall distortion of the 3D SVM image is reduced by adaptively determining the boundary composite area so that image distortion appears less for each viewpoint.

[Figure 5] is an example diagram observing image distortion from a lateral perspective in a general 3D SVM image.

Referring to [Figure 5] (a), the left camera image from a lateral view shows a lot of distortion in both end areas. On the other hand, referring to [Figure 5] (b), less distortion occurs in both end areas in the front and rear camera images from a lateral view. In the lateral view of the 3D view, using front and rear camera images rather than lateral camera images causes less distortion in the image of the composite area.

[Figure 6] is an example diagram observing image distortion from a front view in a general 3D SVM image.

Referring to (a) of Figure 6, the rear camera image from the front view has a lot of distortion in both end areas, and the front camera image also has a lot of distortion in both end areas. On the other hand, referring to [Figure 6] (b), less distortion occurs in both end areas in the side camera image from the front view. In the front view of the 3D view, using side camera images rather than front and rear camera images causes less distortion in the image of the composite area.

From the results of [FIG. 5] and [FIG. 6] above, it can be seen that the phenomenon of image distortion occurring in multi-channel camera images (15 to 18) changes depending on the 3D view point in the 3D SVM image. Specifically, it can be seen that the closer the 3D view point (virtual camera position) and the vehicle camera mounting position are to orthogonal, the less image distortion occurs in the border composite area of the 3D SVM image.

[Figure 7] is a flowchart showing the 3D SVM image distortion correction process through border composite area control according to the 3D view viewpoint according to the present invention.

The 3D SVM image distortion correction technology according to the present invention adaptively adjusts the borderline composite area ratio between multi-channel camera images according to the virtual camera position at the 3D view point during the process of generating a 3D SVM image. The 3D SVM image generated through this process shows less image distortion in areas where unit camera images are overlapped compared to the prior art.

Step (S110): First, multi-channel camera images, that is, multiple unit camera images (15 to 18), are captured from a plurality of cameras (11 to 14) installed in various directions of the vehicle, such as the front bumper, rear bumper, and left and right side mirrors. ) to obtain.

Step (S120): Next, the virtual camera position of the 3D view point for the 3D SVM image is acquired. As illustrated in [Figure 2] (b) and [Figure 4] to [Figure 6], in the 3D SVM image, the 3D view point is set in the three-dimensional space outside the vehicle and the 3D view point is set as if a virtual camera exists at that location. Create a video. In step S120, the virtual camera position value in 3D space is retrieved corresponding to the 3D view point.

Step (S130): Next, for each border composite area of the 3D SVM image, the area ratio of the overlapping multi-channel camera images (15 to 18) is set in response to the virtual camera position.

[Figure 8] is a conceptual diagram for setting the area ratio of multi-channel camera images (15 to 18) in the present invention. Although the present invention relates to 3D SVM images, it is described in terms of 2D SVM images for convenience of explanation. The same concept as [Figure 8] is applied to 3D SVM images, only expanded to three-dimensional space.

Referring to (a) of FIG. 8, boundary composite areas 21 to 24 exist in areas where multi-channel camera images 15 to 18 overlap. In each border composite area (21 to 24), there are overlapping multi-channel camera images (15 to 18). For example, the border composite area 21 on the front left of the vehicle is displayed by overlapping two unit camera images 15 and 17, and the border composite area 23 on the rear right of the vehicle is displayed with two unit camera images 16 and 17, respectively. 18) are displayed overlapping.

In step S130, the area ratio of the overlapping multi-channel camera images 15 to 18 for each of these border composite areas 21 to 24 is set in response to the virtual camera position (i.e., 3D view point). For example, in the case of the border composite area 23 on the rear right side of the vehicle, the ratio of the area where the two unit camera images 16 and 18 are displayed is set in accordance with the virtual camera position.

At this time, as described above with reference to [FIG. 5] and [FIG. 6], the area ratio of the multi-channel camera images (15 to 18) is adjusted in a direction that causes less image distortion at the current virtual camera position (3D view point). Set it.

For example, when the virtual camera is located in the front or rear of the vehicle, the area ratio of the side camera images 17 and 18 is set to the maximum value when forming the border composite area 21 to 24. As a result, the area ratio of the front and rear camera images 15 and 16 is set to the minimum value. At this time, it is assumed that the maximum and minimum values are set in advance. Taking the border composite area 23 on the rear right side of the vehicle as an example, when the virtual camera is located in the front or rear, the area ratio of the right camera image 18 is maximized as shown in [Figure 8] (d), and the rear camera image ( The area ratio of 16) becomes minimum.

Additionally, when the virtual camera is located on the side of the vehicle, the area ratio of the front and rear camera images 15 and 16 is set to the maximum value when forming the border composite area 21 to 24. As a result, the area ratio of the side camera images 17 and 18 will be set to the minimum value. Taking the boundary composite area 23 on the rear right side of the vehicle as an example, when the virtual camera is located on the side, the area ratio of the rear camera image 16 is maximized, as shown in [Figure 8] (c), and the right camera image 18 is The area ratio of becomes minimum.

In addition, when the virtual camera is located at an intermediate point rather than the front, rear, or side of the vehicle, interpolation is applied according to the position of the virtual camera to set the area ratio of the overlapping multi-channel camera images (15 to 18). In other words, the closer the virtual camera location is to the front or rear of the vehicle, the higher the area ratio of the side camera images 17 and 18 is. Conversely, the closer the virtual camera location is to the side of the vehicle, the higher the area ratio of the front and rear camera images 15 and 16 are. The goal is to increase the area ratio. [FIG. 8] (b) shows a case where the position of the virtual camera is diagonally toward the right rear of the vehicle, so the area ratio of the right camera image 18 and the rear camera image 16 is the default value. Here, as the position of the virtual camera moves to the side of the vehicle, the area ratio approaches [Figure 8] (c), and conversely, as the position of the virtual camera moves to the rear of the vehicle, the area ratio approaches [Figure 8] (d). Getting closer.

Meanwhile, from the results of [Figures 5] and [Figure 6], it was confirmed that the closer the 3D view point (virtual camera position) and the vehicle camera mounting position are to each other, the less image distortion occurs in the border composite area of the 3D SVM image. . Accordingly, for each border composite area (21 to 24), the area of the multi-channel camera images (15 to 18) is proportional to the degree to which the virtual camera position and the mounting position of the multi-channel camera devices (11 to 14) are close to orthogonal. It may also be configured to set a ratio.

Steps (S140, S150): Create a 3D composite image by projecting multi-channel camera images (15 to 18) on the 3D projection surface corresponding to the area ratio set above for each border composite area, and use 3D SVM with the 3D vehicle model. Output the video.

Meanwhile, the present invention can be implemented in the form of computer-readable code on a computer-readable non-volatile recording medium. These non-volatile recording media include various types of storage devices, such as hard disks, SSDs, CD-ROMs, NAS, magnetic tapes, web disks, and cloud disks. Additionally, the present invention can be implemented in a form in which code is distributed and stored and executed in a plurality of storage devices connected through a network. Additionally, the present invention may be implemented in the form of a computer program stored on a medium in order to execute a specific procedure in combination with hardware.

Claims (4)

A method for correcting distortion of 3D SVM images by adaptively adjusting the border composite area between multi-channel camera images according to the virtual camera position at the 3D view point of the vehicle 3D SVM system,
A first step of acquiring multi-channel camera images (15 to 18) from multi-channel camera devices (11 to 14) installed in the vehicle;
A second step of acquiring the virtual camera position of the 3D view point for the 3D SVM image;
A third step of setting the area ratio of the overlapping multi-channel camera images (15 to 18) corresponding to the virtual camera position for each border composite area (21 to 24) of the 3D SVM image;
A fourth step of projecting the multi-channel camera images (15 to 18) on a 3D projection surface corresponding to the set area ratio for each borderline composite area;
It is composed including,
The third step is,
When the virtual camera position is the front or rear of the vehicle, the area ratio of the overlapping side camera images 17 and 18 for each border composite area 21 to 24 is set to a preset maximum value, and Setting the area ratio of the overlapping front and rear camera images 15 and 16 to a preset minimum value;
When the virtual camera position is on the side of the vehicle, the area ratio of the front and rear camera images 15 and 16 overlapping with each other for each border composite area 21 to 24 is set to a preset maximum value, and the overlapped areas are set to a preset maximum value. setting the area ratio of the side camera images 17 and 18 to a preset minimum value;
When the virtual camera position is located at an intermediate point rather than the front, rear, or side of the vehicle, the front and rear camera images 15, 16 and the side camera images 17, 18 overlap each other by interpolation according to the virtual camera position. ) setting the area ratio of;
A 3D SVM image distortion correction method through controlling the border composite area according to the 3D view viewpoint, including.
delete A method for correcting distortion of 3D SVM images by adaptively adjusting the border composite area between multi-channel camera images according to the virtual camera position at the 3D view point of the vehicle 3D SVM system,
A first step of acquiring multi-channel camera images (15 to 18) from multi-channel camera devices (11 to 14) installed in the vehicle;
A second step of acquiring the virtual camera position of the 3D view point for the 3D SVM image;
A third step of setting the area ratio of the overlapping multi-channel camera images (15 to 18) corresponding to the virtual camera position for each border composite area (21 to 24) of the 3D SVM image;
A fourth step of projecting the multi-channel camera images (15 to 18) on a 3D projection surface corresponding to the set area ratio for each borderline composite area;
It is composed including,
The third step is,
For each border composite area (21 to 24) of the 3D SVM image, the virtual camera position and the mounting position of the multi-channel camera devices (11 to 14) are close to orthogonal to the multi-channel camera image (15 to 14). 18) Setting the area ratio;
A 3D SVM image distortion correction method through controlling the border composite area according to the 3D view viewpoint, including.
A computer program stored in a storage medium for executing the 3D SVM image distortion correction method through control of the border composite area according to the 3D view viewpoint according to claim 1 or 3 on the computer.

Images