KR102469815B1 - 3D AVM system by use of Deep Learning - Google Patents

Abstract

The present invention relates to a technology capable of configuring a distortion-free 3D AVM screen by estimating a separation distance for a camera image using a deep learning neural network model and constructing a 3D map. The 3D around view monitoring system based on deep learning according to the present invention comprises: a plurality of cameras (11 to 14) installed in the vehicle in different directions to generate a plurality of camera images (15 to 18); a deep learning neural network model unit (100) which generates a plurality of depth maps (105 to 108) having separation distance information for each unit image of the plurality of camera images (15 to 18) with a deep learning neural network model which receives a plurality of camera images (15 to 18) and estimates and outputs separation distance information for each unit image for a series of image frames constituting the corresponding camera images (15 to 18); and a 3D AVM image synthesis unit (200) for generating a 3D AVM image (209) by applying a projection according to separation distance information to the plurality of camera images (15 to 18) with reference to the plurality of depth maps (105 to 108) and synthesizing the images. According to the present invention, there is an advantage in that the convenience of driving a vehicle can be increased by removing distortion from a 3D AVM screen.

Description

3D around view monitoring system based on deep learning {3D AVM system by use of Deep Learning}

The present invention relates generally to 3D around view monitoring (3D AVM) technology for vehicles.

In particular, the present invention relates to a technology capable of configuring a distortion-free 3D AVM screen by estimating a separation distance for a camera image using a deep learning neural network model and constructing a 3D map.

Recently, an around view monitoring (AVM) system has been introduced into automobiles. Surround View Monitoring (AVM) is also called Surround View Monitoring (SVM). Cameras are installed on the front, rear, left, and right sides of the vehicle to take pictures of the front, rear, left, and right sides of the vehicle, and then the camera images are synthesized to provide a top view. It is a technique to The driver can grasp the situation around the vehicle as if looking down from above, making driving and parking more convenient.

1 is a diagram showing the concept of AVM image processing. As shown in [Fig. 1], cameras 11 to 14 are mounted on the front and rear of the vehicle and on both side mirrors, and the camera images 15 to 18 provided from these cameras 11 to 14 are used to improve and improve images. After making a flat image by applying distortion correction, a top-view around-view image 19 is obtained through stitching (image matching and compositing) processing. This around-view image 19 is provided to the driver through a monitor.

As such, while AVM provides a top view, 3D AVM provides a screen in a stereoscopic (3D) perspective. Because it converts the viewpoint into 3D, the driver can see the surroundings of the vehicle in more detail than conventional AVMs. [Fig. 2] is a diagram conceptually showing a 3D AVM. In the 3D AVM, images obtained by projecting the camera images 15 to 18 onto a cylinder or bowl-shaped 3D projection surface are provided to the driver. [Fig. 2] shows an example using the bowl-shaped 3D projection surface.

The disadvantage of 3D AVM is that stereoscopic objects (objects with height) appear distorted. Since the camera images 15 to 18 are projected onto a 3D projection surface having a preset shape (eg, cylinder, bowl), the shape of the object is distorted. In addition, distortion occurs in the shape of an object in an area where the camera images 15 to 18 overlap. 3 is a diagram illustrating an example of a 3D AVM screen. Referring to FIG. 3 , it can be confirmed that flat objects such as lanes and parking lines have little distortion, whereas three-dimensional objects appear different from the actual shape. In addition, it can be confirmed that the 3D AVM screen is distorted even in the area where the camera images 15 to 18 overlap.

Republic of Korea Patent Publication No. 10-2018-0001869 "Image enhancement device and method of AVM system" (published on January 5, 2018) Republic of Korea Patent Publication No. 10-2017-0111504 "AVM system around image consistency evaluation method and apparatus" (published on October 12, 2017)

An object of the present invention is generally to provide a 3D around view monitoring (3D AVM) technology for vehicles.

In particular, an object of the present invention is to provide a technology capable of constructing a distortion-free 3D AVM screen by estimating a separation distance for a camera image using a deep learning neural network model and constructing a 3D map.

On the other hand, the problem of the present invention is not limited to these matters, and other problems can be understood from the description of this specification.

In order to achieve the above object, a deep learning-based 3D around view monitoring system according to the present invention includes a plurality of cameras 11 to 14 installed in a vehicle in different directions to generate a plurality of camera images 15 to 18; It is equipped with a deep learning neural network model that receives multiple camera images (15 to 18) and estimates and outputs separation distance information for each unit image for a series of image frames constituting the corresponding camera images (15 to 18). a deep learning neural network model unit 100 generating a plurality of depth maps 105 to 108 having distance information for each unit image of the images 15 to 18; A 3D AVM image synthesis unit for generating a 3D AVM image 209 by applying a projection according to separation distance information to a plurality of camera images 15 to 18 with reference to a plurality of depth maps 105 to 108 and synthesizing the images ( 200);

The deep learning-based 3D around view monitoring system according to the present invention estimates separation distance information through bilinear interpolation of corresponding depth maps 105 to 108 for the occlusion area of a plurality of camera images 15 to 18. Occlusal processing unit 220; may be configured to further include. At this time, when the 3D AVM image synthesis unit 200 identifies the occlusal area during the image synthesis process for the plurality of camera images 15 to 18, the occlusion processing unit 220 receives separation distance information about the occlusal area and performs image synthesis. configured to process.

At this time, the 3D AVM image synthesis unit 200 reflects the separation distance information for each unit image stored in the depth maps 105 to 108 corresponding to the corresponding image frames of the camera images 15 to 18 to set a 3D projection surface of irregular shape And may be configured to project the camera images (15 to 18) on the 3D projection surface of the atypical shape.

In addition, the deep learning-based 3D around view monitoring system according to the present invention uses internal parameters and external parameters stored in advance for the plurality of cameras 11 to 14 to transmit images from the plurality of cameras 15 to 18 to the camera lens. It may be configured to further include; a camera correction processing unit 210 that removes distortion caused by camera mounting and distortion due to the camera mounting position and angle.

In addition, the deep learning-based 3D around view monitoring system according to the present invention includes a vehicle camera 310 installed in a vehicle and generating a training camera image 311 according to a first time interval; a distance measurement sensor 320 installed in the vehicle to face the same direction as the vehicle camera 310 and generating distance measurement sensor data according to a second time interval; The training camera image 311 and the distance measurement sensor data are provided, and the distance measurement sensor data is interpolated to be synchronized with the training camera image 311 at the time of information acquisition to obtain training separation distance information 321 for training. Further comprising can be configured.

In this case, the distance measuring sensor 320 may include a LiDAR sensor.

On the other hand, the computer program according to the present invention is stored in a non-volatile storage medium to operate the 3D around view monitoring system based on deep learning as described above in the computer.

According to the present invention, there is an advantage in that the convenience of driving a vehicle can be increased by removing distortion from a 3D AVM screen.

[Fig. 1] is a diagram showing the concept of AVM image processing.
[Fig. 2] is a diagram conceptually illustrating a 3D AVM.
[Fig. 3] is a diagram showing an example of a 3D AVM screen.
[Figure 4] is a block diagram showing the overall configuration of a 3D AVM system according to the present invention.
[Figure 5] is a block diagram showing the configuration for learning the deep learning neural network model in the present invention.
[Fig. 6] is an exemplary view of obtaining camera images and LIDAR sensor outputs for constructing deep learning learning data in the present invention.
[Fig. 7] is a diagram illustrating the concept of occlusion in a stereoscopic image.
[Fig. 8] is a diagram showing an example in which a blank area is displayed in a stereoscopic image due to occlusion.

The present invention relates to 3D around view monitoring (3D AVM) technology. In general, the disadvantage of 3D AVM is that three-dimensional objects appear distorted. In the present invention, a 3D AVM image is obtained by estimating the distance between unit images (eg, pixels, pixel blocks, etc.) Composition eliminates this distortion. At this time, it is preferable to distinguish the boundary of the object from the background because it can improve the accuracy of estimating the distance to the object.

In the present invention, a separation distance from a corresponding camera is estimated for unit images in a camera image. In this case, an area serving as a unit for estimating the separation distance in the camera image is referred to as a 'unit image' in this specification. This unit image may be set in advance, and may be an individual object unit, a pixel unit, or a unit of a pixel block (eg, 8x8 pixels) of a certain size. In addition, in the present invention, a deep learning neural network model is used to estimate the separation distance. The deep learning neural network model can estimate the separation distance from a single video image without analyzing a series of image frames, so it is possible to estimate the distance even when the vehicle is stopped.

The cause of distortion in the 3D AVM image in the prior art is that the camera image is uniformly projected onto a 3D projection surface (eg, a bowl or a cylinder) of a preset shape. On the other hand, in the present invention, after estimating the separation distance from the corresponding camera for each unit image in the camera image, mapping the estimated separation distance to the unit image to construct a 3D map, 3D AVM images without distortion can be obtained by applying projection to camera images for each unit image accordingly.

Additionally, according to the present invention, a distortion-free screen can be configured in an area where the cameras 11 to 14 overlap.

Hereinafter, the present invention will be described in detail with reference to the drawings.

[Figure 4] is a block diagram showing the overall configuration of the 3D AVM system according to the present invention. Referring to FIG. 4, camera images 15 to 18 are obtained from a plurality of cameras 11 to 14 installed in a vehicle in different directions, and these camera images 15 to 18 are deep learning neural network model units 100 ) is transmitted to The deep learning neural network model unit 100 generates depth maps 105 to 108 from the camera images 15 to 18 using the deep learning neural network model. The depth maps 105 to 108 have separation distance information for each unit image with respect to a series of image frames constituting the camera images 15 to 18. At this time, the depth maps 105 to 108 may have separation distance information for all or part of the image frame, or may have separation distance information for all or part of the unit image.

The 3D AVM image synthesis unit 200 synthesizes (stitches) the camera images 15 to 18 in multiple directions of the vehicle using the separation distance information and camera correction information of the depth maps 105 to 108 to generate a 3D AVM image. .

At this time, the 3D projection plane does not have a preset shape (eg, cylinder, bowl), but configures the 3D projection plane based on the separation distance information of the depth maps 105 to 108 . That is, when the camera images 15 to 18 are projected, the 3D projection plane is set by reflecting the separation distance information for each unit image stored in the depth maps 105 to 108 corresponding to the corresponding image frame of the camera images 15 to 18 do. Accordingly, this 3D projection surface does not have a standard shape such as a cylinder or a bowl, but has an irregular shape according to the separation distance information for each unit image. Then, the camera images 15 to 18 are projected onto the atypical 3D projection surface.

The concept of using camera calibration information to synthesize the camera images 15 to 18 will be described. In general, after installing front and rear and left and right cameras 11 to 14 in a vehicle to implement the AVM function, a camera calibration process is performed. In this camera calibration process, the AVM system obtains intrinsic camera parameters and extrinsic camera parameters for each of the cameras 11 to 14 installed in the vehicle, which are collectively referred to as camera calibration information. The internal parameter is information representing distortion characteristics due to the camera lens, and the external parameter is information representing distortion characteristics due to the position and angle at which the corresponding camera is mounted on the vehicle.

Since distortions according to the internal and external parameters of the corresponding cameras 11 to 14 are inserted into the camera images 15 to 18, the camera correction processing unit 210 of the 3D AVM image synthesis unit 200 converts the camera images 15 to 14 into In the synthesizing process of 18), distortion according to internal and external parameters, that is, distortion caused by the camera lens and distortion due to the camera mounting position and angle, is removed. In general, the process of removing the distortion of the camera images 15 to 18 using camera correction information is the same as [Equation 1], but since this is a technique already known in the field of image synthesis, a detailed description is omitted herein.

와

는 각각 카메라 내부 파라미터와 카메라 외부 파라미터를 나타낸다. At this time, two matrices

Wow

denote camera internal parameters and camera external parameters, respectively.

[Figure 5] is a block diagram showing the configuration for learning the deep learning neural network model in the present invention.

In the present invention, separation distance information is estimated for each camera image 15 to 18 using a deep learning neural network model. In order to utilize this, the deep learning learning unit 110 needs to train the deep learning neural network model of the deep learning neural network model unit 100 in advance. In deep learning, the training process refers to the process of learning a neural network model through a training dataset. In the present invention, multiple combinations of the camera image 311 for training and the separation distance information 321 for training are used as a learning dataset. The deep learning neural network model trained by this training dataset can estimate the separation distance for each unit image for a series of image frames given a camera image.

At this time, it is very inefficient if a person manually collects the training camera image 311 and the separation distance information 321 for training in order to configure the learning dataset of the deep learning neural network model. Accordingly, the present invention proposes to utilize the vehicle camera 310 and the distance measuring sensor 320. That is, the camera 310 and the distance measuring sensor 320 are installed in the vehicle so as to be directed in the same direction, and the vehicle travels to several points.

The vehicle camera 310 generates a training camera image 311 and the distance measurement sensor 320 generates distance measurement sensor data. The deep learning learning unit 110 synchronizes with the information acquisition time of the training camera image 311 to obtain training separation distance information 321 from the distance measurement sensor data, and obtains the training camera image 311 and the training separation distance A training data set is formed by combining the distance information 321 . Through this process, it is possible to effectively obtain a large amount of training dataset for learning a deep learning neural network model.

As a preferred embodiment of the distance measuring sensor 320, a LiDAR sensor is suitable. LiDAR sensor is an abbreviation of 'Light Detection And Ranging' or 'Laser Imaging, Detection and Ranging'. It is a technology that obtains 3D spatial information by measuring the time of a laser beam reflected from an object using a high-output pulse laser. Through synchronization between the vehicle camera 310 and the lidar sensor 320, the separation distance can be estimated for each pixel of the training camera image 311, and using this, the training camera image 311 and the training distance information A combination of (321) can be obtained.

At this time, the vehicle camera 310 and the LIDAR sensor 320 are installed to face the same direction, and mutual synchronization must be performed even at the time of information acquisition. However, in general, information acquisition time points of the vehicle camera 310 and the lidar sensor 320 are different. The time interval at which the vehicle camera 310 generates the training camera image 311 is 33 msec in the case of 30 frames per second. On the other hand, the time interval at which the lidar sensor 320 generates the separation distance information 321 for training is much longer than 33 msec.

[Fig. 6] is an exemplary view of obtaining camera images and LIDAR sensor outputs for constructing deep learning learning data in the present invention. In order to construct a training dataset for a deep learning neural network model, separation distance information needs to be synchronized with the information acquisition time of the training camera image 311 . That is, in [FIG. 6], separation distance information for information acquisition time points (t1 to t5) of the training camera image 311 is required.

In the first embodiment, the LIDAR sensor output close to the information acquisition time is matched with the separation distance information. In [Fig. 6], for the viewpoints t2 and t3, the lidar sensor output at the viewpoint t6 is set as the separation distance information, and for the viewpoints t4 and t5, the lidar sensor output at the viewpoint t7 is set as the separation distance to set information.

In the second embodiment, separation distance information is obtained by applying interpolation (interpolation) based on the time axis to the lidar sensor output. 6, the information generation period of the lidar sensor 320 is longer than that of the vehicle camera 310, and linear interpolation such as [Equation 2] is applied to the lidar sensor output, It is to acquire (estimate) separation distance information for information acquisition time points (t2 to t5) of the training camera image 311. Since the linear interpolation technique is a widely known technique, a detailed description thereof is omitted in this specification.

At this time, x is the time value and y is the separation distance value. (x0, y0) and (x1, y1) are time values and lidar sensor output values of left and right points where lidar sensor output values exist.

As described above, the deep learning learning unit 110 uses the training camera image 311 obtained from the vehicle camera 310 and the lidar sensor 320 and the distance information 321 for training to set a training dataset of a certain size or more. is configured, and deep learning is performed on the deep learning neural network model of the deep learning neural network model unit 100 using this training dataset.

When the deep learning neural network model unit 100 inputs the camera images 15 to 18 to the deep learning neural network model that has been learned in this way, the unit image (e.g., The separation distance can be estimated for each pixel, pixel block, etc.). This separation distance is also called depth. Accordingly, in this specification, a set of separation distance information generated by the deep learning neural network model unit 100 for the camera images 15 to 18 is called a depth map 105 to 108.

The 3D AVM image synthesis unit 200 synthesizes the camera images 15 to 18 according to the depth maps 105 to 108 to generate a 3D AVM image 209 .

At this time, the occlusion processing unit 220 of the 3D AVM image synthesis unit 200 performs processing to remove errors due to occlusion when synthesizing the camera images 15 to 18 in multiple directions. As in the prior art, when the camera images 15 to 18 are uniformly projected onto a cylinder or bowl shape, there is no empty space in the synthesized result (synthesized image). On the other hand, as in the present invention, separation distance information exists for each unit image, and when the camera images 15 to 18 are synthesized by reflecting the separation distance information, there is a problem in that occlusion occurs. If such occlusion is neglected, the quality of the 3D AVM image 209 deteriorates, so the occlusion processing unit 220 of the 3D AVM image synthesis unit 200 removes the error caused by the occlusion.

Referring to [Fig. 7] and [Fig. 8], the principle of creating a void in a 3D image due to occlusion will be described. [Fig. 7] is a diagram illustrating the concept of occlusion in a stereoscopic image, and [Fig. 8] is a diagram illustrating an example in which a blank area is displayed in a stereoscopic image due to occlusion.

A part of a 3D object may be visible in one camera but not visible in the other camera. The area marked 'occlusion' in [Fig. 7] is a part visible from the left camera but not visible from the right camera. 3D matching cannot be made in this area because it is not possible to obtain information on the separation distance on either side, and an empty area appears when the 3D AVM image 209 is constructed.

[Fig. 8] (a) is an image of an ordinary cafe with desks and stools, and [Fig. 8] (b) shows the result of 3D synthesis after photographing this point with the left and right cameras. Referring to [Fig. 8] (b), the area that is not visible to either camera because it is covered by an object is processed as black in the 3D image because separation distance information cannot be obtained from the corresponding camera image. In [FIG. 8] (b), red color indicates a near area, blue color indicates a far area, and black indicates an area without separation distance information.

In the 3D AVM system, an occlusion area may occur in an area where the cameras 11 to 14 overlap. If the 3D AVM image 209 is constructed with the error in the occlusal region as it is, the quality will be evaluated low. Perform processing to remove .

The 3D AVM image synthesis unit 200 identifies the occlusal region during image synthesis of the camera images 15 to 18 . Identification of the occlusion region belongs to a known technology in the field of image synthesis, so it is not described in detail in this specification. Regarding the occlusal region, separation distance information does not exist in any one of the depth maps 105 to 108. In this case, the occlusion processing unit 220 of the 3D AVM image synthesis unit 200 estimates the separation distance information for the occlusion region by bilinear interpolation. Bilinear interpolation is a method of generating a value at a specific point using surrounding values. have. Since the bilinear interpolation technique is a widely known technique, a detailed description thereof is omitted in this specification.

At this time, (x, y) is a point where separation distance information does not exist in the depth maps 105 to 108 because it is occluded by an object in the specific camera images 15 to 18, that is, to obtain the separation distance value through bilinear interpolation. is the coordinate value of the point to be At this time, referring to [FIG. 7], it is preferable to apply the coordinate values converted to the global coordinate system instead of the coordinate system of the camera images 15 to 18. A rectangle is defined as a point where the separation distance value exists in the depth map (105 ~ 108) near the point where the separation distance value is to be obtained, and its vertices are Q11 (x1, y1), Q12 (x1, y2), Q21 (x2, y1), Q22 (x2, y2). At this time, f(Q11), f(Q12), f(Q21), and f(Q22) are separation distance values of these points Q11, Q12, Q21, and Q22.

The occlusion processor 220 may estimate separation distance information for the occlusion region through bilinear interpolation, and accordingly, the 3D AVM image synthesis unit 200 partially removes errors caused by occlusion and produces high-quality 3D AVM images ( 209) can be created. Distortion in the region where the cameras 11 to 14 overlap can be effectively removed by such processing of the occlusion region.

On the other hand, the present invention can be implemented in the form of computer readable codes on a computer readable non-volatile recording medium. As such non-volatile recording media, there are various types of storage devices, such as hard disks, SSDs, CD-ROMs, NAS, magnetic tapes, web disks, and cloud disks. and can be implemented in a form that is executed. In addition, the present invention may be implemented in the form of a computer program stored in a medium in order to execute a specific procedure in combination with hardware.

11 to 14: Camera
15 ~ 18: Camera video
100: deep learning neural network model unit
105 to 108: depth map
110: deep learning learning unit
200: 3D AVM video synthesis unit
209: 3D AVM video
210: camera calibration processing unit
220: occlusal processing unit
310: vehicle camera
311: camera video for training
320: distance measurement sensor
321: distance information for training

Claims (6)

A plurality of cameras (11 to 14) for generating a plurality of camera images (15 to 18) in front, rear, left and right directions of the vehicle;
For a series of image frames, a deep learning neural network model for estimating separation distance information for each preset unit image is provided, and the deep learning neural network is provided for the plurality of camera images 15 to 18 in the front, rear, left, and right directions of the vehicle. a deep learning neural network model unit 100 that generates a plurality of depth maps 105 to 108 having separation distance information for each unit image for a series of image frames by the model;
The depth map ( 105 to 108), an occlusal processing unit 220 for estimating distance information for the occlusal region by defining a quadrangle having a distance value and applying bilinear interpolation to the distance value of each vertex of the quadrangle;
For the plurality of camera images 15 to 18 in the front, rear, left, and right directions of the vehicle, separation distance information for each unit image stored in the depth maps 105 to 108 corresponding to the corresponding image frame is reflected to obtain an atypical image for the front, rear, left, and right directions of the vehicle. Setting a 3D projection plane of the shape, projecting a plurality of camera images 15 to 18 in the front, rear, left and right directions of the vehicle onto the 3D projection plane of the atypical shape and stitching the images to generate a 3D AVM image 209, and the image stitching process In the area where the plurality of camera images 15 to 18 for the front, rear, left, and right directions of the vehicle are mutually overlapped, if an occlusal area in which separation distance information does not exist is identified in either depth map, the occlusal processing unit 220 a 3D AVM image synthesis unit 200 configured to process the image stitching while removing an error caused by the occlusion by receiving separation distance information for the identified occlusion area and supplementing the separation distance information to the depth map;
A deep learning-based 3D around-view monitoring system comprising a.
delete delete The method of claim 1,
A camera that removes distortion due to camera lenses and distortion due to camera mounting positions and angles from the plurality of camera images 15 to 18 using internal parameters and external parameters stored in advance for the plurality of cameras 11 to 14 correction processing unit 210;
Deep learning-based 3D around view monitoring system, characterized in that configured to further include.
The method of claim 1,
A vehicle camera 310 installed in a vehicle to generate a training camera image 311 according to a first time interval;
a distance measurement sensor 320 installed in the vehicle to face the same direction as the vehicle camera 310 and generating distance measurement sensor data according to a second time interval;
Distance measurement information 321 for training by applying interpolation to the distance measurement sensor data so that the training camera image 311 and the distance measurement sensor data are provided and synchronized with the information acquisition time of the training camera image 311 A deep learning learning unit ( 110);
Deep learning-based 3D around view monitoring system, characterized in that configured to further include.
The method of claim 5,
The distance measurement sensor 320 is a deep learning-based 3D around view monitoring system, characterized in that configured to include a LiDAR sensor.

Images