KR20240025323A - Device and Method for Creating a Rear Panoramic View - Google Patents

Abstract

According to the present invention, a plurality of cameras are attached to a vehicle and include a plurality of side mirror cameras that capture first and second side mirror images of the surroundings of the vehicle and a rear camera that captures a rear image; Obtain posture estimation data from a plurality of images including the first side mirror image, the second side mirror image, and the rear image, and estimate the posture of the plurality of cameras based on the posture estimation data to determine the posture of the plurality of cameras. An image analysis unit that generates a LUT for the image; and an image synthesis unit that generates a rear panoramic view image by applying the LUT to the plurality of images, and the image analysis unit is a rear panoramic view generating device that generates the LUT larger than a preset reference point.

Description

Device and Method for Creating a Rear Panoramic View}

The present invention relates to an apparatus and method for generating a rear panoramic view.

In general, rear cameras are installed at the rear of a vehicle to reduce the vehicle design and air resistance coefficient. However, the installed camera has a limited field of view, resulting in blind spots that cannot be seen in the video.

Meanwhile, the supply of side mirror cameras is increasing due to increased demand for vehicle convenience functions and driver assistance functions. Side mirror cameras provide drivers with a wide angle of view without distortion. In addition, it is attracting attention from drivers and producers as it can significantly reduce the air resistance coefficient.

Despite these convenience and auxiliary functions, blind spots at the rear of the vehicle still exist. The rear camera and side mirror camera cannot detect obstacles in the direction of the vehicle's rear bumper and tail lights, the location of other vehicles, or the vehicle's expected path, so it relies on the driver's senses and experience. However, novice drivers lack sense and experience. Therefore, the possibility of collision with obstacles is very high.

In addition, the positions of each camera and output device are different, which can cause the driver's gaze to be dispersed and become a factor that interferes with driving concentration. In addition, it is insufficient to express space when reversing because only a fixed screen is displayed without considering driving information.

Republic of Korea Patent Publication No. 10-2021-0097078 Republic of Korea Patent Publication No. 10-2020-0000953

The present invention is intended to solve the above-mentioned problems. The purpose of the present invention is to improve the insufficient angle of view of the vehicle rear camera and the spatial expression of the image, and to improve the convenience and safety of driving by concentrating the driver's gaze. It provides a rear panoramic view image.

The present invention is a rear panoramic view generating device, which is attached to a vehicle and includes a plurality of side mirror cameras that capture first and second side mirror images of the surroundings of the vehicle and a rear camera that captures a rear image. multiple cameras; Obtain posture estimation data from a plurality of images including the first side mirror image, the second side mirror image, and the rear image, and estimate the posture of the plurality of cameras based on the posture estimation data to determine the posture of the plurality of cameras. An image analysis unit that generates a LUT for the image; and an image synthesis unit that generates a rear panoramic view image by applying the LUT to the plurality of images, and the image analysis unit generates the LUT larger than a preset reference point.

In one embodiment of the present invention, the image analysis unit may derive posture estimation data based on distortion parameters predefined for the plurality of images and image patterns defined for the plurality of images.

In one embodiment of the present invention, the image synthesis unit corrects at least one of illuminance and noise of an overlapping area in the rear panoramic view image.

In one embodiment of the present invention, the rear panoramic view image includes a first image defined from a straight-line viewing perspective of the vehicle, a second image defined from a right viewing perspective of the vehicle, and a left viewing perspective of the vehicle. Includes third video.

In one embodiment of the present invention, it further includes an image output unit that selectively outputs the rear panoramic view image based on vehicle information, wherein the image output unit includes speed, yaw, and steering of the vehicle included in the vehicle information. a communication module that receives at least one of information and gear information; and an image selection unit that outputs one of the first to third images based on the vehicle information.

In one embodiment of the present invention, the preset reference point is defined for each of the first to third images.

It includes a fully integrated LUT from the plurality of images.

In one embodiment of the present invention, a method performed by a rear panoramic view image generating device, wherein a first side view of the surroundings of the vehicle is obtained from a plurality of cameras including a plurality of side mirror cameras and a rear camera attached to the vehicle. A plurality of side mirror cameras for capturing a mirror image and a second side mirror image and capturing a rear image; Obtain posture estimation data from a plurality of images including the first side mirror image, the second side mirror image, and the rear image, and estimate the posture of the plurality of cameras based on the posture estimation data to determine the plurality of cameras. Generating a LUT for an image; and applying the LUT to the plurality of images to generate a rear panoramic view image, wherein the image analysis step generates the LUT larger than a preset reference point.

In one embodiment of the present invention, the image analysis step derives posture estimation data based on distortion parameters predefined for the plurality of images and image patterns defined for the plurality of images.

In one embodiment of the present invention, the image synthesizing step further includes correcting at least one of illuminance and noise of an overlapping area in the rear panoramic view image.

In one embodiment of the present invention, the rear panoramic view image includes a first image defined from a straight view perspective of the vehicle, a second image on the right side of the vehicle, and a second image on the left side of the vehicle.

In one embodiment of the present invention, it further includes an image output step of selectively outputting the rear panoramic view image based on vehicle information, wherein the image output step includes the speed and yaw of the vehicle included in the vehicle information. , a communication method for receiving vehicle information including at least one of steering information and gear information; and an image selection step of outputting one of the first to third images based on the vehicle information.

In one embodiment of the present invention, the preset reference point is defined for each of the first to third images.

According to the present invention, a rear panoramic view generation method and a device supporting the same can contribute to the convenience and safety of vehicle operation by correcting and combining multiple camera images to reduce blind spots of a vehicle and provide concentrated information.

In addition, meaningful spatial expressions can be created and provided based on vehicle operation information.

Additionally, because it uses an existing camera, costs can be reduced as there is no need to install a separate camera.

The effects according to the present invention are not limited to the contents exemplified above, and further various effects are included in the present specification.

1 is a block diagram of an apparatus and method for generating a rear panoramic view according to an embodiment of the present invention.
Figure 2 illustrates a rear panoramic view image according to an embodiment of the present invention.
Figure 3 is a block diagram of an apparatus and method for generating a rear panoramic view according to another embodiment of the present invention.
Figure 4a illustrates a rear panoramic view image that changes depending on the steering angle difference according to an embodiment of the present invention.
FIG. 4B is a diagram illustrating a rear panoramic view image that changes depending on the difference in steering angle when driving a vehicle according to an embodiment of the present invention.
Figure 5 is a flow chart for generating a rear panoramic view image according to an embodiment of the present invention.
Figure 6 is a flowchart of an image analysis unit according to an embodiment of the present invention.
Figure 7 is a flowchart of a camera pose estimation method according to an embodiment of the present invention.
Figure 8 is a flowchart of an image output unit according to an embodiment of the present invention.

Hereinafter, embodiments of the present invention will be described clearly and in detail so that a person skilled in the art can easily practice the present invention.

1 is a block diagram of a rear panoramic view generating device according to an embodiment of the present application.

Referring to FIG. 1, the rear panoramic view generating device includes a plurality of cameras 100 and an image generating unit 200.

The plurality of cameras 100 include a first rear camera 130, a 2-1 left side mirror camera 110, and a 2-2 right side mirror camera 120.

The plurality of cameras 100 capture left and right side mirror images and rear images of the vehicle. The plurality of cameras 100 generally use a wide-angle lens with a large angle of view, and a fisheye lens, which is an ultra-wide-angle lens with a view angle of 180° or more, may be used, but is not limited to this and various other lenses. can be used.

The image generator 200 may analyze and synthesize images received from the plurality of cameras 100 to create an adaptive rear panoramic view image.

The image generator 200 includes an image analysis unit 210 and an image synthesis unit 220 for the above-described operations.

The image analysis unit 210 receives left and right side mirror images and rear images from the plurality of cameras 100, and the image synthesis unit 220 generates a look up table (LUT) for image synthesis.

To this end, the image analysis unit 210 derives the posture estimation data based on distortion parameters predefined for a plurality of images received from a plurality of cameras 100 and image patterns defined for the plurality of images. You can.

In one embodiment, the image analysis unit 210 may use a camera calibration method to analyze left, right, and rear images including tolerance correction patterns. Camera calibration is a method used to obtain the positions where 3D points are projected onto an image or, conversely, to restore 3D spatial coordinates from image coordinates, and operates to estimate the pose of the camera.

Camera calibration methods include calibration patterns, geometric clues, and deep learning based methods. Hereinafter, the present invention will explain the calibration pattern as an example for convenience, but will not be limited thereto. Additionally, camera calibration may use algorithms such as OpenCV or SolvePnP, but is not limited to these.

The calibration pattern is a checkerboard-based method that uses the intersection of the black and white pattern. Checkerboard is a method used to obtain 3D information through images or videos. Checkerboards can be produced in various sizes and shapes, taking into account the camera's shooting distance and angle of view.

The image analysis unit 210 may derive camera internal parameters and camera external parameters, which are internal factors of the camera, in order to estimate the camera posture using camera calibration. Camera internal parameters include focal length, principal point, and skew coefficient. Camera external parameters are elements that describe the transformation relationship between the camera coordinate system and the world coordinate system.

In order to calculate internal parameters, the image analysis unit 210 may first calculate lens distortion coordinates defined by Equation 1 below.

[Equation 1]

= (1+ + + ) , = +

Here, a point in space is a point on the image through the lens ( , ), and the image analysis unit 210 measures the distance to the center. , coordinates reflecting the distortion of the lens using the radial distortion coefficient k ( , ) can be obtained.

Next, the image analysis unit 210 may calculate internal parameters based on Equation 2 below.

[Equation 2]

=

Here, the elements of the internal parameters are the focal length ( , ), Pub( , ), asymmetry coefficient ( ), and the image analysis unit 210 is the coordinates in which the distortion is reflected ( , ) and the actual image pixel coordinates ( , ), you can obtain internal parameters.

The image analysis unit 210 may obtain external parameters and obtain distortion parameters by matching the 3D world coordinates and 2D image coordinates of a camera image sample for which the derived internal parameters are known in advance.

The image analysis unit 210 acquires a reference image or video captured through a lens with a predefined pattern in order to estimate the camera posture. Afterwards, the image analysis unit corrects the distortion based on the external parameters derived based on the above-described operation of the pattern detected in the image or video that is different from the reference image or video and derives an expected pattern that is the same as the pattern of the reference image or video. Estimate the camera pose until it calculates the same information (size, length, shape, etc.). If the pattern information is not the same, the image analysis unit may perform repetitions to minimize the information deviation between the expected pattern and the pattern of the existing image or video.

Once the camera pose is estimated, the image analysis unit 210 can generate an LUT of the image by mapping the pixel positions of the image in three-dimensional space.

Specifically, the image analysis unit 210 receives camera pose estimation data obtained by minimizing the difference between the expected pattern and the pattern information of the image and performs three-dimensional spatial mapping.

In order to perform 3D spatial mapping, the image analysis unit 210 sets a random circular cylinder mesh in 3D space, and uses the estimated camera posture to form a straight line connecting the information of each pixel of the image to the center of the camera. Calculate . Afterwards, the image analysis unit 210 may calculate the intersection point between the projected straight line and the set mesh, and match the location of the intersection point with the coordinates of the image just before the projection to generate an LUT. In addition, although the present invention has been described based on a cylindrical mesh as an embodiment, it is not limited thereto.

The reason for generating the LUT in the image analysis unit 210 is that it represents a set of pre-calculated results for a given operation, and the result for the given operation can be quickly calculated.

According to an embodiment of the present invention, since the viewpoint of the rear panoramic view image changes depending on the steering angle and must have real-time and continuity, preset reference points are defined for each of the first to third images. In other words, by generating an LUT of an integrated image larger than the reference point rather than a separate LUT for each reference point of a plurality of images, the amount of data calculation can be reduced and an improved image can be generated.

The image synthesis unit 220 synthesizes images based on the integrated LUT generated by the image analysis unit 210 and generates a real-time rear panoramic view image with a reduced data calculation amount.

The image synthesis unit 220 according to an embodiment may correct at least one of the illuminance and noise of the overlapping area in the rear panoramic view image. In other words, the image synthesis unit 220 includes an image synthesis unit 220 that can perform correction for overlapping areas in a plurality of images used for rear panoramic view image synthesis. For example, in order to correct the synthesized image, the image synthesis unit 220 adjusts the intensity of pixels with different brightness values by adding or subtracting the same value from the intensity of all pixels to make the brightness value of the image the same. You can. Additionally, the image synthesis unit 220 may generate a rear panoramic view image by improving the opacity of a portion of the overlapping area of the image.

The rear panoramic view image generated according to the embodiment of FIG. 1 has a wider angle of view than the existing rear camera image, and the overlapping area of the image can be processed and corrected to generate a natural image. In addition, since it is based on the LUT of integrated images rather than the LUT of individual images, the amount of data calculation is improved, allowing images to be provided in real time.

Figure 2 shows a rear panoramic view image generated by the image synthesis unit according to an embodiment of the present invention.

As shown in Figure 2, the present invention synthesizes the right side mirror image 501, the rear camera image 502, and the left side mirror image 503, processes and corrects the overlapping area, and replaces the existing rear camera image 502 with the existing rear camera image 502. ) can generate a rear panoramic view image 500 that can provide visual information of a wider posterior area.

Figure 3 is a block diagram of a rear panoramic view generating device according to another embodiment of the present invention. Hereinafter, detailed description of parts that overlap with the parts described above will be omitted.

Referring to FIG. 3, a rear panoramic view generating apparatus according to another embodiment may further include an image output unit 300 in addition to a plurality of cameras 100 and an image generator 200.

The plurality of cameras 100 capture left and right side mirror images and rear images, receive the images from the image generator 200, estimate the pose of the camera and generate a LUT, and synthesize the images based on the LUT to create a rear view mirror image. A panoramic view image 500 is generated.

The image output unit 300 can selectively output a rear panoramic view image 500 based on vehicle information, and includes a communication module 310 and an image selection unit 320 for this purpose.

The communication module 310 is, for example, a CAN BUS (Controller Area Network Bus), which allows devices to communicate with each other in a vehicle without a host computer. Vehicle information transmitted and received through the communication module 310 may include, for example, at least one of the vehicle's speed, yaw, steering information, and gear information.

The image selection unit 320 may select an output image point of any one of the first to third images based on vehicle speed, yaw, steering information, and gear information. For example, the viewpoint of the image can be changed to the same direction as the direction of the vehicle obtained based on information about the vehicle's steering angle.

FIGS. 4A and 4B are for explaining the operation of an image selection unit that selectively outputs images based on vehicle information according to an embodiment of the present invention.

Referring to FIGS. 4A and 4B, the image output unit 300 displays a rear panoramic view image 500 in the straight or backward direction according to the difference in the steering angle of the vehicle based on the data on the vehicle's posture and direction, and the steering angle to the right. A composite image 511 with a small steering angle and a composite image 512 with a large steering angle to the right can be output. Here, it may be referred to as a first image defined from the vehicle's straight viewing perspective, a second image defined from the vehicle's right viewing perspective, and a third image defined from the vehicle's left viewing perspective. Accordingly, the rear and side blind spots of the vehicle can be reduced as much as possible, and the angle of view can be changed to match the direction of the vehicle when entering a lane depending on the direction of the vehicle, providing the user with information such as the location and expected path of nearby vehicles.

Figure 5 is a flowchart for performing a method for generating a rear panoramic view image.

A detailed description of the operation method of the rear panoramic view image generating device according to various embodiments of the present invention described above will be described below with reference to FIG. 5.

As shown in FIG. 5, the rear panoramic view generating device collects images using a plurality of cameras 100 (S100). The image generator 200 performs vehicle information analysis (S200), such as the vehicle's direction of travel and gear information, based on the collected images. The image analysis unit 210 analyzes the image based on vehicle information and performs image analysis and LUT generation (S300) to estimate the camera pose and generate a comprehensive LUT of the image. The image synthesis unit 220 synthesizes the images from which the LUT is generated (S400), processes and corrects the overlapping area (S500), and outputs the corrected rear panoramic view image (S600).

Figure 6 is a flowchart for performing an image analysis and LUT generation method.

As shown in FIG. 6, the image analysis unit 210 acquires left and right side mirror camera images and rear camera images including tolerance correction patterns (S310) and estimates each camera posture (S320). The image analysis unit 210 generates a LUT for an adaptive panoramic view image (S340) by performing 3D spatial mapping (S330) based on the image from which the camera pose is estimated.

Figure 7 is a flowchart for performing a camera pose estimation method.

As shown in FIG. 7, the image analysis unit 210 collects a plurality of images from the plurality of cameras 100 (S321) and detects a pattern in a standard form within the images (S322). The image analysis unit 210 corrects distortion of the detected pattern using distortion parameters derived using camera calibration (S323) and estimates the expected pattern (S324). The image analysis unit 210 determines whether the information (size, length, shape, etc.) of the expected pattern and the already input actual pattern is the same (S325). If they are not identical, the repeatedly detected pattern is subjected to distortion correction (S323) and expected pattern estimation (S324), and optimization is performed until the information of the expected pattern and the actual pattern are identical. If the information is the same after optimization, the camera posture is estimated (S326) and the process ends.

Figure 8 is a flowchart for performing video output.

As shown in FIG. 8, the image output unit 300 receives vehicle information through a communication module (S601) and analyzes the vehicle steering angle (S602). The image output unit 300 selects an image in the same direction as the vehicle direction based on the vehicle steering angle (S603) and outputs the image (S604).

The above-described details are specific embodiments for carrying out the present invention. In addition to the above-described embodiments, the present invention will also include embodiments that can be simply changed or easily changed in design. In addition, the present invention will also include technologies that can be easily modified and implemented using the embodiments. Therefore, the scope of the present invention should not be limited to the above-described embodiments, but should be determined by the claims and equivalents of the present invention as well as the claims described later.

100: plural cameras 320: video selection unit
110: Left side mirror camera 500: Panoramic view video
120: Right side mirror camera 501: Right side mirror video
130: rear camera 502: rear image
200: Image generator 503: Left side mirror image
210: Image analysis unit 510: Composite image according to steering angle
220: Image synthesis unit 511: Forward or backward synthesis image
300: Video output unit 512: Composite image with a small steering angle to the right
310: Communication module 513: Composite image with a large steering angle to the right

Claims (12)

A plurality of cameras attached to the vehicle including a plurality of side mirror cameras that capture first and second side mirror images of the surroundings of the vehicle and a rear camera that captures a rear image;
Obtain posture estimation data from a plurality of images including the first side mirror image, the second side mirror image, and the rear image, and estimate the posture of the plurality of cameras based on the posture estimation data to determine the posture of the plurality of cameras. An image analysis unit that generates a LUT for the image; and
An image synthesis unit that generates a rear panoramic view image by applying the LUT to the plurality of images,
The image analysis unit is a rear panoramic view generating device that generates the LUT larger than a preset reference point. According to paragraph 1,
The image analysis unit derives the posture estimation data based on distortion parameters predefined for the plurality of images and image patterns defined for the plurality of images. According to paragraph 1,
The image synthesis unit corrects at least one of illuminance and noise of an overlapping area in the rear panoramic view image. According to paragraph 1,
The rear panoramic view image includes a first image defined from a straight viewing perspective of the vehicle, a second image defined from a right viewing perspective of the vehicle, and a third image defined from a left viewing perspective of the vehicle. View creation device. According to paragraph 4,
It further includes an image output unit that selectively outputs the rear panoramic view image based on vehicle information,
The video output unit,
a communication module that receives at least one of vehicle speed, yaw, steering information, and gear information included in the vehicle information; and
A rear panoramic view generating device further comprising an image selection unit that outputs one of the first to third images based on the vehicle information. According to paragraph 4,
The preset reference point is defined for each of the first to third images. A rear panoramic view generation method performed by a rear panoramic view image generating device, comprising:
A plurality of side mirror cameras that capture a first side mirror image and a second side mirror image of the surroundings of the vehicle from a plurality of cameras including a plurality of side mirror cameras and a rear camera attached to the vehicle and a rear image. step;
Obtain posture estimation data from a plurality of images including the first side mirror image, the second side mirror image, and the rear image, and estimate the posture of the plurality of cameras based on the posture estimation data to determine the plurality of cameras. Generating a LUT for an image; and
Generating a rear panoramic view image by applying the LUT to the plurality of images,
A method of generating a rear panoramic view in which the step of generating the LUT generates the LUT larger than a preset reference point. In clause 7,
In the step of generating the LUT, the posture estimation data is derived based on a distortion parameter predefined for the plurality of images and an image pattern defined for the plurality of images. In clause 7,
The step of generating the image further includes correcting at least one of illuminance and noise of an overlapping area in the rear panoramic view image. In clause 7,
The rear panoramic view image includes a first image defined from a straight viewing perspective of the vehicle, a second image defined from a right viewing perspective of the vehicle, and a third image defined from a left viewing perspective of the vehicle. How to create a view. According to clause 10,
Further comprising selectively outputting the rear panoramic view image based on vehicle information,
The step of selectively outputting the rear panoramic view image is:
Receiving at least one of speed, yaw, steering information, and gear information of the vehicle included in the vehicle information; and
A rear panoramic view generation method further comprising an image selection step of outputting one of the first to third images based on the vehicle information. According to clause 10,
The preset reference point is defined for each of the first to third images.

Images