KR102478074B1 - Around view monitoring system and method for compensating camera tolerance there of - Google Patents

Abstract

An around-view monitoring system according to the present invention includes an image capture unit including two or more of front, rear, left, and right cameras installed around a vehicle, a memory storing a program for correcting a tolerance of the camera, and a memory stored in the memory. A processor for executing a stored program, wherein the processor detects one or more correction points from each image captured by each camera included in the image capture unit as the program is executed, and corrects a preset reference coordinate system. Based on the result of projecting the point to the position of the correction point of each image, rotation and position information for each camera corresponding to the reference coordinate system is estimated, and each camera is based on the estimated rotation and position information of each camera. After calculating an inter-camera transformation matrix, a look-up table for generating a top view image is generated based on the transformation matrix, wherein the processor satisfies the closed-loop condition when the inter-camera transformation matrix satisfies, The look-up table is generated based on the transformation matrix.

Description

AROUND VIEW MONITORING SYSTEM AND METHOD FOR COMPENSATING CAMERA TOLERANCE THERE OF}

The present invention relates to an around view monitoring system and a tolerance correction method thereof.

The vehicle's Around View Monitoring System (AVMS) is a system that receives images from four cameras (front, rear, left, and right) installed in the vehicle and displays the surroundings of the vehicle in the form of a Bird's Eye View.

In this way, the around-view monitoring system using images captured by multiple cameras needs to correct tolerances when assembling the cameras. To this end, each manufacturer of the vehicle releases the vehicle after correcting the tolerance to satisfy the consistency of the around-view image of the vehicle equipped with the around-view monitoring system.

However, due to various environmental factors such as vehicle vibration during operation, folding of side mirrors, and opening and closing of vehicle doors as the vehicle is continuously used after the vehicle is shipped, the tolerance corrected at the time of shipment is changed to improve image consistency. This lowering will cause problems.

Accordingly, when driving or parking, the driver has to use the distorted around-view image, causing inconvenience.

In order to solve this problem, it is necessary to correct the changed tolerance, but the inconvenience of visiting and repairing a service center or business office where tolerance correction is possible exists in order to correct the current tolerance.

Accordingly, there is a need for an around-view monitoring system capable of providing a matched around-view image by correcting the tolerance even when the tolerance is changed.

In this regard, Korean Patent Laid-Open Publication No. 10-2015-0144090 (title of the invention: around-view monitoring system and operation method of the around-view monitoring system) discloses that after accurately moving a vehicle to a designated position in a tolerance correction environment, each camera Disclosed is a method of generating a top view image by calculating a 3×3 transformation matrix using a correction point detected from an image.

However, in the case of the prior art, there is a limit in that the vehicle must be accurately moved to a designated position, and tolerance correction is possible only in a stopped state after moving the vehicle.

An embodiment of the present invention calculates a conversion matrix between a plurality of cameras installed in a vehicle, and performs tolerance correction when each calculated conversion matrix satisfies a closed loop condition, thereby correcting the tolerance even when the vehicle is moving. A view monitoring system and a tolerance correction method thereof are provided.

However, the technical problem to be achieved by the present embodiment is not limited to the technical problem as described above, and other technical problems may exist.

As a technical means for achieving the above-described technical problem, an around view monitoring system according to a first aspect of the present invention includes an image capture unit including two or more of front, rear, left and right cameras installed around a vehicle; A memory storing a program for correcting the tolerance of the camera and a processor executing the program stored in the memory. At this time, as the processor executes the program, the processor detects one or more correction points from each image captured by each camera included in the image capture unit, and sets the correction point of the preset reference coordinate system to the correction point of each image. Based on the result of projecting to the position of , rotation and position information for each camera corresponding to the reference coordinate system is estimated, and a transformation matrix between each camera is calculated based on the estimated rotation and position information of each camera. Then, A look-up table for generating a top view image is generated based on the transformation matrix, but when the transformation matrix between cameras satisfies a closed loop condition, the look-up table is generated based on the transformation matrix generate

Also, a tolerance correction method in an around view monitoring system according to a second aspect of the present invention includes detecting one or more correction points from each image captured by a plurality of cameras; projecting a correction point on a preset reference coordinate system to a position of the detected correction point on the coordinate system of the camera; estimating rotation and position information for each camera corresponding to the reference coordinate system based on a projection result of the correction point; Calculating a transformation matrix between each camera based on the estimated rotation and position information of each camera; Determining whether the transformation matrix between the cameras satisfies a closed-loop condition, and if the closed-loop condition is satisfied, calculating a look-up table for generating a top view image based on the transformation matrix It includes steps to

According to any one of the above-described problem solving means of the present invention, the tolerance can be corrected not only in a stationary state of the vehicle but also in a moving state and an arbitrary position.

Accordingly, it is possible to shorten the tolerance correction time, thereby minimizing the operator's intervention, thereby increasing the efficiency of the vehicle production process, and providing more accurate images of the surroundings of the vehicle.

1 is a diagram for explaining a tolerance correction method in an around view monitoring system according to the prior art.
2 is a block diagram of an around view monitoring system according to an embodiment of the present invention.
3 is a diagram for explaining a process of projecting a correction point on a preset reference coordinate system to a position of a detected correction point on a camera coordinate system.
4 is an exemplary view of generating a top view image from an image captured by a camera.
5 is a diagram for explaining a closed loop condition.
6 is a flowchart of a tolerance correction method according to an embodiment of the present invention.

Hereinafter, embodiments of the present invention will be described in detail so that those skilled in the art can easily practice the present invention with reference to the accompanying drawings. However, the present invention may be embodied in many different forms and is not limited to the embodiments described herein. And in order to clearly explain the present invention in the drawings, parts irrelevant to the description are omitted.

In the entire specification, when a part "includes" a certain component, it means that it may further include other components without excluding other components unless otherwise stated.

1 is a diagram for explaining a tolerance correction method in an around view monitoring system according to the prior art.

A tolerance correction method in an around-view monitoring system according to the related art may recognize correction point coordinates in a correction plate pl included in an image using an image captured by a camera. That is, it is possible to recognize correction point coordinates corresponding to grid edges in the horizontal direction in the correction plate pl formed of a grid pattern. In addition, a top-view image may be generated by calculating a 3×3 transformation matrix (homography) from the recognized correction point coordinates.

To this end, in the prior art, the vehicle must be stopped at a preset vehicle position, a plurality of compensating plates (pl) must be installed at the vehicle position, and a camera takes an image including a plurality of compensating plates (pl) to correct the tolerance. do.

As such, in the case of the prior art, there is a problem that the vehicle must be accurately moved to a designated position, and there is a limit that tolerance correction is possible only in a stopped situation after moving, and it takes a lot of time to correct the camera tolerance.

In order to solve this problem, the around view monitoring system 100 and its tolerance correction method according to an embodiment of the present invention calculate a conversion matrix between a plurality of cameras installed in a vehicle, and each calculated conversion matrix is a closed loop condition. By performing the tolerance correction when is satisfied, the tolerance can be corrected even in a moving situation of the vehicle or an arbitrary position other than a designated position.

Hereinafter, the around view monitoring system 100 according to an embodiment of the present invention will be described with reference to FIGS. 2 to 5 .

2 is a block diagram of an around view monitoring system 100 according to an embodiment of the present invention. 3 is a diagram for explaining a process of projecting a correction point on a preset reference coordinate system to a position of a detected correction point on the coordinate system of the cameras 111 to 114. 4 is an exemplary diagram for generating a top view image from images captured by cameras 111 to 114. Referring to FIG. 5 is a diagram for explaining a closed loop condition.

The around view monitoring system 100 according to an embodiment of the present invention includes an image capturing unit 110 , a memory 120 and a processor 130 .

The image capturing unit 110 includes two or more cameras among front, rear, left and right cameras 111 to 114 installed around the vehicle. That is, in one embodiment of the present invention, at least two or more cameras are installed in a vehicle, and preferably, front, rear, left and right cameras 111 to 114 are installed around the vehicle, respectively, so that the front of the vehicle, Take pictures of the rear, left and right sides.

Specifically, the front camera 111 is installed at the center of the vehicle's bone net, the left and right cameras 113 and 114 are installed to be positioned at the edge or bottom of both side mirrors of the vehicle, respectively, and the rear camera 112 is It can be installed in the upper center of the rear bumper. At this time, the heights of the front and rear cameras 111 and 112 are preferably installed the same, and the heights of the left and right cameras 113 and 114 are preferably installed the same.

Here, the installation position of each of the cameras 111 to 114 is not necessarily limited thereto, and may be changed depending on the type and design of the vehicle.

Each of the cameras 111 to 114 of the image capture unit 110 may be provided with a lens having a large angle of view, such as a wide-angle lens or a fisheye lens.

The image capture unit 110 may capture a 3D object as a 2D image through a lens and provide the captured image to the processor 130 . At this time, the processor 130 may receive an image captured by the image capturing unit 110 through a communication module (not shown).

Such a communication module may include both a wired communication module and a wireless communication module. The wired communication module may be implemented as a power line communication device, a telephone line communication device, cable home (MoCA), Ethernet, IEEE1294, an integrated wired home network, and an RS-485 control device. In addition, the wireless communication module may be implemented with wireless LAN (WLAN), Bluetooth, HDR WPAN, UWB, ZigBee, Impulse Radio, 60 GHz WPAN, Binary-CDMA, wireless USB technology, wireless HDMI technology, and the like. More preferably, the communication module may transmit and receive data through CAN (Controller Area Network) communication.

A program for correcting the tolerance of the cameras 111 to 114 is stored in the memory 120 . At this time, the memory 120 collectively refers to a non-volatile storage device and a volatile storage device that continuously maintain stored information even when power is not supplied.

For example, the memory 120 may include a compact flash (CF) card, a secure digital (SD) card, a memory stick, a solid-state drive (SSD), and a micro SD card. NAND flash memory such as cards, magnetic computer storage devices such as hard disk drives (HDD), and optical disc drives such as CD-ROMs and DVD-ROMs. can

The processor 130 executes the program stored in the memory 120. As the program is executed, the processor 130 first detects one or more correction points from each image captured by each of the cameras 111 to 114 included in the image capturing unit 110 .

At this time, the processor 130 may preferably detect four correction points from each image taken by each of the cameras 111 to 114, and detect the correction points not only when the vehicle is stopped but also during movement or at an arbitrary position. can do.

After detecting the correction point, the processor 130, as shown in FIG. 3, converts the correction point P0 of the preset reference coordinate system O to the detected correction point on the coordinate system C of the cameras 111 to 114. By projecting to the position Pc, rotation and position information for each camera (111 to 114) corresponding to the reference coordinate system is estimated.

That is, since the correction point of the correction plate is set in advance, the processor 130 can estimate the rotation and position information for each camera 111 to 114 by projecting the location of the correction point to the detected correction point. At this time, rotation and position information of the cameras 111 to 114 may be estimated based on nonlinear optimization.

At this time, the processor 130 may estimate the rotation information of the corresponding camera when one or more of the two or more cameras performs one or more of yawing, pitching, and rolling operations. there is.

Referring to FIG. 2 again, when the rotation and position information of each camera 111 to 114 is estimated in this way, the processor 130, based on the estimated rotation and position information of each camera 111 to 114, A transformation matrix for converting the images captured at (111 to 114) into a direction looking at the ground surface from above the vehicle, that is, a top view image is calculated.

At this time, when the image capturing unit 110 includes four cameras, that is, front, rear, left, and right cameras 111 to 114 centered on the vehicle, the processor 130 performs a 4×4 conversion between the four cameras. A transformation matrix can be calculated for each matrix (Homogeneous Transform).

By calculating such a transformation matrix, it is possible to know which position the position of each camera 111 to 114 corresponds to on the top-view image coordinate system.

In addition, it is possible to calculate which positions the four correction points for each of the cameras 111 to 114 correspond to on the top-view image coordinate system. That is, in the case of the example of the front camera 111 shown in FIG. 4, the four correction points P1, P2, P3, P4 in the image taken by the front camera 111 using the transformation matrix correspond to Positions (P1', P2', P3', P4') on the top-view image coordinate system may be calculated.

Accordingly, in an embodiment of the present invention, a top-view image may be generated by calculating a 3×3 transformation matrix for each camera, similarly to the above-described tolerance correction method according to the prior art.

Based on the conversion matrix obtained in this way, the processor 130 generates a look up table (LUT) for generating a top view image. That is, since it takes a lot of resources and time to convert an image captured by a camera into a top-view image using a transformation matrix, the image captured by each camera is topped up in advance based on the distortion correction algorithm and transformation matrix of the camera. A top view image may be generated by generating a lookup table for conversion into a view point image and processing each image through the lookup table. Such a lookup table may be generated by applying various algorithms such as a distortion correction algorithm, an affine transformation algorithm, and a viewpoint transformation algorithm.

Meanwhile, the processor 130 may determine whether a transformation matrix between cameras satisfies a loop closing condition before generating a lookup table.

Referring to FIG. 5 , when front, rear, left, and right cameras 111 to 114 are installed in a vehicle, the processor 130 calculates a 4×4 homogeneous transform between each camera as shown in Equation 1. can be calculated

[Equation 1]

In addition, it may be determined whether the calculated inter-camera transformation matrix satisfies the closed loop condition.

Here, the closed loop condition may satisfy the condition of Equation 2.

[Equation 2]

는 카메라 i(1≤i≤4)에서 카메라 j(1≤j≤4)로의 변환 행렬을 의미한다.In this case, I is an identity matrix,

denotes a transformation matrix from camera i (1≤i≤4) to camera j (1≤j≤4).

), 우측 카메라(114)를 기준으로 후방 카메라(112)로의 변환 행렬(

), 후방 카메라(112)를 기준으로 좌측 카메라(113)로의 변환 행렬(

), 좌측 카메라(113)를 기준으로 전방 카메라(111)로의 변환 행렬(

)의 곱이 단위 행렬을 만족하는지 여부를 판단한다. That is, as shown in FIG. 5, the processor 130 converts the front camera 111 to the right camera 114 into a transformation matrix (

), a transformation matrix from the right camera 114 to the rear camera 112 (

), a transformation matrix from the rear camera 112 to the left camera 113 (

), a transformation matrix from the left camera 113 to the front camera 111 (

) determines whether the product satisfies the identity matrix.

As a result of the determination, when the product of each transformation matrix satisfies the identity matrix, the processor 130 considers the estimated rotation and position information for each camera 111 to 114 to be accurate, and generates a lookup table based on the transformation matrix there is.

At this time, each direction of the transformation matrix in FIG. 5 is designated as a specific direction for convenience of explanation, and can be set in various ways according to the opposite direction or the number of cameras.

Meanwhile, when the transformation matrix between the cameras 111 to 114 does not satisfy the closed loop condition, the processor 130 may generate notification information corresponding to the warning message and end tolerance correction.

Also, the processor 130 may end tolerance correction even when the closed loop condition is satisfied.

That is, the processor 130 may generate notification information corresponding to a warning message and terminate tolerance correction when the difference between the calculated conversion matrix between the cameras 111 to 114 and the preset design reference value is greater than or equal to a threshold value. .

In addition, the processor 130 may generate notification information corresponding to a warning message and terminate tolerance correction when the difference between the predicted coordinate value of the correction point corresponding to the top-view image and the preset design reference value is greater than or equal to a threshold value. there is.

Tolerance correction end conditions performed when the closed loop condition is satisfied may be performed independently or in combination.

For reference, the components shown in FIG. 2 according to an embodiment of the present invention may be implemented in the form of software or hardware such as a Field Programmable Gate Array (FPGA) or Application Specific Integrated Circuit (ASIC), and perform predetermined roles. can do.

However, 'components' are not meant to be limited to software or hardware, and each component may be configured to be in an addressable storage medium or configured to reproduce one or more processors.

Thus, as an example, a component includes components such as software components, object-oriented software components, class components, and task components, processes, functions, properties, procedures, sub routines, segments of program code, drivers, firmware, microcode, circuits, data, databases, data structures, tables, arrays and variables.

Components and the functionality provided within them may be combined into fewer components or further separated into additional components.

Hereinafter, a tolerance correction method in the around view monitoring system 100 according to an embodiment of the present invention will be described with reference to FIG. 6 .

6 is a flowchart of a tolerance correction method according to an embodiment of the present invention.

In the tolerance correction method in the around-view monitoring system 100 according to an embodiment of the present invention, first, when the tolerance correction mode starts (S110), one or more images are obtained from each image captured by the plurality of cameras 111 to 114. A correction point is detected (S120).

At this time, in one embodiment of the present invention, tolerance correction is possible when two or more cameras among the four cameras 111 to 114 on the front, rear, left, and right sides of the vehicle are installed. Recently, all four cameras are installed in a vehicle, and an embodiment of the present invention also preferably detects a correction point from each image taken from each of the four cameras 111 to 114, and four Correction points can be detected.

Next, the correction point on the preset reference coordinate system is projected onto the position of the detected correction point on the coordinate system of the cameras 111 to 114, and each camera 111 to 114 corresponding to the reference coordinate system based on the projection result of the correction point Estimating star rotation and position information (S130).

Next, after calculating the transformation matrix between each camera 111 to 114 based on the estimated rotation and position information of each camera 111 to 114 (S140), the transformation matrix between each camera 111 to 114 is closed. It is determined whether the loop condition is satisfied (S150).

As a result of the determination, when the closed loop condition is satisfied, a look-up table for generating a top-view image is calculated based on the transformation matrix (S180).

In contrast, when the closed loop condition is not satisfied, the around view monitoring system 100 may generate notification information corresponding to the warning message and end tolerance correction (S150 and S190).

In addition, even if the closed loop condition is satisfied, when the difference between the calculated transformation matrix between the cameras 111 to 114 and the preset design reference value is greater than or equal to a threshold value (S160), or the predicted top view image When the difference between the coordinate value of the correction point and the preset design reference value is greater than or equal to the threshold value (S170), the around view monitoring system 100 may generate notification information corresponding to the warning message and end tolerance correction (S190). .

Meanwhile, in the above description, steps S110 to S190 may be further divided into additional steps or combined into fewer steps, depending on an embodiment of the present invention. Also, some steps may be omitted if necessary, and the order of steps may be changed. In addition, even if other omitted contents, the contents already described in FIGS. 2 to 5 may also be applied to the tolerance correction method of FIG. 6 .

According to any one of the embodiments of the present invention, the tolerance can be corrected not only in a stationary state of the vehicle but also in a moving state and at an arbitrary position.

Accordingly, it is possible to shorten the tolerance correction time, thereby minimizing the operator's intervention, thereby increasing the efficiency of the vehicle production process, and providing more accurate images of the surroundings of the vehicle.

Meanwhile, the tolerance correction method in the around view monitoring system 100 according to an embodiment of the present invention is implemented in the form of a computer program stored in a medium executed by a computer or a recording medium including instructions executable by a computer. It can be. Computer readable media can be any available media that can be accessed by a computer and includes both volatile and nonvolatile media, removable and non-removable media. Also, computer readable media may include both computer storage media and communication media. Computer storage media includes both volatile and nonvolatile, removable and non-removable media implemented in any method or technology for storage of information such as computer readable instructions, data structures, program modules or other data. Communication media typically includes computer readable instructions, data structures, program modules, or other data in a modulated data signal such as a carrier wave, or other transport mechanism, and includes any information delivery media.

Although the methods and systems of the present invention have been described with reference to specific embodiments, some or all of their components or operations may be implemented using a computer system having a general-purpose hardware architecture.

The above description of the present invention is for illustrative purposes, and those skilled in the art can understand that it can be easily modified into other specific forms without changing the technical spirit or essential features of the present invention. will be. Therefore, the embodiments described above should be understood as illustrative in all respects and not limiting. For example, each component described as a single type may be implemented in a distributed manner, and similarly, components described as distributed may be implemented in a combined form.

The scope of the present invention is indicated by the following claims rather than the detailed description above, and all changes or modifications derived from the meaning and scope of the claims and equivalent concepts should be construed as being included in the scope of the present invention. do.

100: around view monitoring system
110: video recording unit
111: front camera 112: rear camera
113: left camera 114: right camera
120: memory
130: processor

Claims (14)

In the around view monitoring system,
An image capture unit including two or more cameras among front, rear, left and right cameras installed around the vehicle;
A memory storing a program for correcting the tolerance of the camera; and
Including a processor that executes the program stored in the memory,
As the program is executed, the processor detects one or more correction points from each image captured by each camera included in the image capture unit, and sets the correction point of a preset reference coordinate system to the location of the correction point of each image. Based on the result projected by , rotation and position information for each camera corresponding to the reference coordinate system is estimated, a transformation matrix between each camera is calculated based on the estimated rotation and position information of each camera, and the transformation Create a look-up table for generating a top view image based on the matrix,
Wherein the processor generates the look up table (LUT, Look Up Table) based on the transformation matrix when the transformation matrix between cameras satisfies a closed loop condition. According to claim 1,
Wherein the processor detects four correction points from each image captured by each camera while the vehicle is moving. According to claim 1,
The image capture unit includes front, rear, left and right cameras installed around the vehicle,
Wherein the processor generates the look-up table based on the transformation matrix when the transformation matrix between the four cameras satisfies a closed loop condition. According to claim 3,
The closed loop condition is

is to satisfy
Where I is an identity matrix,

Means a transformation matrix from camera i (1≤i≤4) to camera j (1≤j≤4). According to claim 1,
Wherein the processor generates notification information corresponding to a warning message and terminates tolerance correction when the transformation matrix between the cameras does not satisfy a closed loop condition. According to claim 1 or 5,
Wherein the processor generates notification information corresponding to a warning message and terminates tolerance correction when a difference between the calculated transformation matrix between each camera and a preset design reference value is greater than or equal to a threshold value. According to claim 1 or 5,
Wherein the processor generates notification information corresponding to a warning message and terminates tolerance correction when a difference between a coordinate value of a correction point predicted corresponding to the top-view image and a preset design reference value is equal to or greater than a threshold value. monitoring system. According to claim 1,
Wherein the processor estimates a rotation for each camera corresponding to a case in which one or more of the two or more cameras performs one or more of yawing, pitching, and rolling motions. In the tolerance correction method in the around view monitoring system,
Detecting one or more correction points from each image taken by a plurality of cameras installed in the vehicle;
projecting a correction point on a preset reference coordinate system to a position of the detected correction point on the coordinate system of the camera;
estimating rotation and position information for each camera corresponding to the reference coordinate system based on a projection result of the correction point;
Calculating a transformation matrix between each camera based on the estimated rotation and position information of each camera;
Determining whether the transformation matrix between the cameras satisfies a closed loop condition; and
and calculating a look up table (LUT) for generating a top view image based on the transformation matrix when the closed loop condition is satisfied. According to claim 9,
When the transformation matrix between the cameras does not satisfy the closed loop condition,
and generating notification information corresponding to the warning message and terminating the tolerance correction. According to claim 9 or 10,
When the difference between the calculated conversion matrix between each camera and the preset design reference value is greater than or equal to a threshold value,
and generating notification information corresponding to the warning message and terminating the tolerance correction. According to claim 9 or 10,
When the difference between the coordinate value of the correction point predicted corresponding to the top-view image and the preset design reference value is greater than or equal to a threshold value,
and generating notification information corresponding to the warning message and terminating the tolerance correction. delete According to claim 12,
The closed loop condition is

is to satisfy
Where I is an identity matrix,

denotes a transformation matrix from camera i (1≤i≤4) to camera j (1≤j≤4).

Images