KR20200135606A - Method and apparatus for providing camera calibration for vehicles - Google Patents

Abstract

The present invention relates to a camera calibration device for a vehicle and to a method thereof. The camera calibration device for a vehicle comprises: a camera module which acquires an image of a lane from a plurality of cameras mounted on the vehicle; an input/output module receiving the image acquired from the camera module or outputting a corrected image; a lane detection module detecting the lane from the image received from the input/output module and extracting characteristic points of the lane; and a camera correction module correcting an image by estimating a new external parameter using an equation of the lane and a lane width using initial camera information and external parameter information from the image received from the input/output module.

Description

Vehicle camera calibration device and its method {METHOD AND APPARATUS FOR PROVIDING CAMERA CALIBRATION FOR VEHICLES}

The present invention relates to a vehicle camera calibration apparatus and method thereof, and more particularly, to automatically calibrate external parameters of a camera even while driving by using images of lane photographs on the road received from a plurality of cameras mounted on the vehicle. It relates to a vehicle camera calibration apparatus and method thereof.

In recent years, cameras for various purposes such as black boxes or rear cameras for parking are used in automobiles. Recently, the development of a top view system that can completely remove the blind spots of the front, rear, left, and right of the car has been developed since it is possible to provide the driver with a screen like looking down the car from the sky by installing multiple cameras on the car It is becoming active.

However, when a plurality of cameras are mounted in a vehicle, a plurality of image information transmitted from each camera must be integrated, and for such image information integration, integration of a coordinate system is most important. In other words, each captured image acquired from a plurality of cameras is input with various information, and the real-time spatial information is converted to the common coordinate system after changing the real-time spatial information to the common coordinate system of the vehicle among the information of each captured image. It becomes possible to display a suitable top view image to the driver by matching a plurality of captured images based on the information.

However, it is practically impossible to perfectly match a plurality of camera coordinate axes with a common coordinate system of a vehicle due to an error that occurs when the camera is actually mounted on a vehicle. This is because camera mounting errors can be precisely calibrated in automobile manufacturing plants, but after the car is shipped, the mounting position of the camera changes due to physical forces such as shock and torsion during operation, resulting in camera mounting errors. to be. Therefore, a number of cameras mounted on the camera must be calibrated periodically or at each impact.

As described above, in order to match the images input from multiple cameras after the car is shipped, a camera calibration that corrects the mounting error of the camera must be performed as a prerequisite. For such camera calibration, the installed height or angle of the installed camera The same information is required. Conventionally, in order to obtain information for camera calibration, a method of obtaining information using a pattern image photographed after installing a specific reference pattern such as a checker board on the ground has been used.

The conventional method of acquiring camera information using a specific reference pattern has the advantage of obtaining precise camera information because the relative position of a specific marker such as a pattern can be accurately known in advance. After securing a large enough space to be installed on the device, there is a problem in that it is cumbersome to perform the operation.

In particular, in the case of general automobiles, camera mounting errors occur periodically or at each impact due to the actual operation of the vehicle. Therefore, it is too much for the operator to acquire camera information and perform camera calibration by using a specific reference pattern. There is a problem that is cumbersome and requires a lot of time and cost. Therefore, it is necessary to automatically calibrate the camera even while the vehicle is running.

As related prior art, there is Korean Patent Publication No. 10-2008-0028531 (published date: April 01, 2008).

The present invention receives an image of the driving lane of the vehicle from a plurality of cameras mounted on the vehicle while the vehicle is driving, detects the lane of the lane from the photographed image, and uses the detected lane to detect the lane of the vehicle. An object of the present invention is to provide a camera calibration apparatus for a vehicle and a method thereof that enables a camera calibration in which a physical installation error occurs due to a vehicle operation by automatically correcting an external parameter to be performed quickly and easily even while the vehicle is driving.

The objects of the embodiments of the present invention are not limited to the above-mentioned objects, and other objects not mentioned will be clearly understood by those of ordinary skill in the technical field to which the present invention pertains from the following description. .

According to an aspect, a camera calibration method performed by an apparatus for calibrating a plurality of cameras includes generating a plurality of images using a plurality of cameras photographing around a vehicle, detecting lanes within the plurality of images. Steps, generating a top view image using a plurality of images based on initial camera information and initial external parameters, calculating equations for lanes in the top view image, and based on the equation And calibrating the plurality of cameras by calculating external parameters for the plurality of cameras.

The detecting of a lane within the plurality of images may include generating a first image by accumulating frames generated by the first cameras of the plurality of cameras, detecting an edge in the first image, and the It may include determining a lane of the first image based on an edge.

The step of detecting a lane within the plurality of images further includes at least one of removing noise in the first image and converting the first image into a black and white image, and an edge in the first image The detecting of may include detecting an edge in the first image from which noise is removed or the first image converted into a black and white image.

The initial external parameter includes initial posture information and initial position information of the camera, and the initial posture information includes pitch (R _x ), yaw (R _y ), and roll (R _z ) of the camera, and the initial posture information The location information may include a location movement (T _z ) of the camera.

The step of calibrating the plurality of cameras by calculating external parameters for the plurality of cameras based on the equation may include a pitch (R _x ) and a yaw (R) for each of the front and rear cameras among the plurality of cameras. _y), and the roll (R _z) a step of calculating a pitch (for the front camera and the rear camera, each R _x), I (R _y), and a roll (the position of the rear view camera on the basis of the R _z) mobile phase, and the pitch (R _x), I (R _y) of the left camera and a right camera, each of the plurality of cameras based on position movement (T _z) of the rear-view camera to calculate the (T _z), It may include calculating the roll (R _z ), and the positional movement (T _z ).

The step of calibrating the plurality of cameras by calculating external parameters for the plurality of cameras based on the equation further comprises adjusting a roll (R _z ) of each of the plurality of cameras, the roll adjusting the (R _z) is, for calculating a first difference between the average of calculating a first difference between the calculated and the initial rolls (R _z) for each of the plurality of cameras, the calculated first difference Determining whether the first difference average exceeds a preset first average threshold, and when the first difference average does not exceed the first average threshold, the plurality of It may include the step of adjusting the roll (R _z ) of each of the cameras.

The adjusting of the roll may include: when the first difference average exceeds the first average threshold, the plurality of first differences are performed based on first differences of the cameras other than the camera representing the largest difference among the first differences. It may further include the step of adjusting the roll (R _z ) of each of the cameras.

Adjusting said plurality of cameras, each of the rolls (R _z) based on the first difference in the average on the basis of the average of said first difference for adjusting the front camera and the roll (R _z) of the rear camera It may include steps.

Adjusting the roll (R _z ) of the front camera and the rear camera based on the first difference average may include subtracting the first difference average from the previous roll (R _z ) of the front camera. It may include the step of adjusting the roll (R _z ).

Adjusting the roll R _z of each of the plurality of cameras based on the first difference average may include a roll of the left camera so that a point of a front left lane and a point of the left lane located on the same lane match. It may include the step of adjusting (R _z ).

According to another aspect, an apparatus for calibrating a plurality of cameras includes a plurality of cameras for generating a peripheral image of a vehicle, a memory in which a program for calibrating the plurality of cameras is recorded, and a processor for executing the program, , The program includes: generating a plurality of images using the plurality of cameras, detecting lanes within the plurality of images, and using a plurality of images based on initial camera information and initial external parameters top view) generating an image, calculating an equation for lanes in the top view image, and calibrating the plurality of cameras by calculating external parameters for the plurality of cameras based on the equation. Can be done.

The detecting of a lane within the plurality of images may include generating a first image by accumulating frames generated by the first cameras of the plurality of cameras, detecting an edge in the first image, and the It may include determining a lane of the first image based on an edge.

The step of detecting a lane within the plurality of images further includes at least one of removing noise in the first image and converting the first image into a black and white image, and an edge in the first image The detecting of may include detecting an edge in the first image from which noise is removed or the first image converted into a black and white image.

The initial external parameter includes initial posture information and initial position information of the camera, and the initial posture information includes pitch (R _x ), yaw (R _y ), and roll (R _z ) of the camera, and the initial posture information The location information may include a location movement (T _z ) of the camera.

The step of calibrating the plurality of cameras by calculating external parameters for the plurality of cameras based on the equation may include a pitch (R _x ) and a yaw (R) for each of the front and rear cameras among the plurality of cameras. _y), and the roll (R _z) a step of calculating a pitch (for the front camera and the rear camera, each R _x), I (R _y), and a roll (the position of the rear view camera on the basis of the R _z) mobile phase, and the pitch (R _x), I (R _y) of the left camera and a right camera, each of the plurality of cameras based on position movement (T _z) of the rear-view camera to calculate the (T _z), It may include calculating the roll (R _z ), and the positional movement (T _z ).

The step of calibrating the plurality of cameras by calculating external parameters for the plurality of cameras based on the equation further comprises adjusting a roll (R _z ) of each of the plurality of cameras, the roll adjusting the (R _z) is, for calculating a first difference between the average of calculating a first difference between the calculated and the initial rolls (R _z) for each of the plurality of cameras, the calculated first difference Determining whether the first difference average exceeds a preset first average threshold, and if the first difference average does not exceed the first average threshold, based on the first difference average, the plurality of It may include adjusting the roll (R _z ) of each of the cameras.

The adjusting of the roll may include: when the first difference average exceeds the first average threshold, the plurality of first differences are performed based on first differences of the cameras other than the camera representing the largest difference among the first differences. It may further include the step of adjusting the roll (R _z ) of each of the cameras.

Adjusting said plurality of cameras, each of the rolls (R _z) based on the first difference in the average on the basis of the average of said first difference for adjusting the front camera and the roll (R _z) of the rear camera It may include steps.

Adjusting the roll (R _z ) of the front camera and the rear camera based on the first difference average may include subtracting the first difference average from the previous roll (R _z ) of the front camera. It may include the step of adjusting the roll (R _z ).

Adjusting the roll R _z of each of the plurality of cameras based on the first difference average may include a roll of the left camera so that a point of a front left lane and a point of the left lane located on the same lane match. It may include the step of adjusting (R _z ).

According to another aspect, a top-view image generation method performed by an apparatus for generating a top-view image includes generating a plurality of images using a plurality of cameras, the plurality of cameras based on the plurality of images Calibrating the plurality of cameras by calculating external parameters for the cameras-the external parameters include the pitch (R _x ), yaw (R _y ), and roll (R _z ) of the camera -, the calibrated plurality And generating a plurality of images using the cameras of, and generating a top view image based on the plurality of images.

As described above, the present invention receives an image of a driving lane of a vehicle from a plurality of cameras mounted on a vehicle, detects a lane from the captured image, and uses the detected lane to be used to detect the outside of the camera mounted on the vehicle. Since camera calibration to correct parameters is performed, there is an effect that camera calibration can be performed quickly and easily without stopping the vehicle.

In addition, since the present invention performs camera calibration while the vehicle is running, there is no need to perform secondary operations such as securing a space after stopping the vehicle and installing a correction pattern for correction of the conventional camera mounted on the vehicle. There is an effect that can reduce the time or cost required for calibration.

1 is a schematic diagram of a vehicle camera calibration device according to an example.
2 is a block diagram of an electronic device according to an exemplary embodiment.
3 is a flowchart of a camera calibration method according to an exemplary embodiment.
4 is a flowchart of a method for detecting a lane according to an example.
5 is a flowchart of a method of estimating an external parameter according to an example.
6 is an image generated when the pitches Rx of the front and rear cameras are different according to an example.
7 is an image generated when the yaw Ry of the front and rear cameras is wrong according to an example.
8 is an image generated when rolls Rz of a front camera and a rear camera are wrong according to an example.
9 is an image generated when the position movement (Tz) of the rear camera is wrong according to an example.
10 is an image generated when the pitches Rx of the left and right cameras are different according to an example.
11 is an image generated when the yaw Ry of the left and right cameras is wrong according to an example.
12 is an image generated when the rolls Rz of the left and right cameras are wrong according to an example.
13 is an image when the position movement Tz of the left and right cameras is wrong according to an example.
14 to 16 are flowcharts of a method of adjusting a roll Rz of a camera according to an example.
17 is an image generated by an unadjusted roll Rz when a vehicle and a lane are not parallel according to an example.
18 is an image generated when a roll Rz of a front camera and a rear camera is adjusted according to an example.
19 is an image generated when a roll Rz of a front camera, a rear camera, a left camera, and a right camera is adjusted according to an example.

Hereinafter, embodiments will be described in detail with reference to the accompanying drawings. However, the scope of the patent application is not limited or limited by these embodiments. The same reference numerals in each drawing indicate the same members.

Various changes may be made to the embodiments described below. The embodiments described below are not intended to be limited to the embodiments, and it should be understood that all changes, equivalents, and substitutes for them are included.

The terms used in the examples are used only to describe specific embodiments, and are not intended to limit the embodiments. Singular expressions include plural expressions unless the context clearly indicates otherwise. In the present specification, terms such as "comprise" or "have" are intended to designate the presence of features, numbers, steps, actions, components, parts, or combinations thereof described in the specification, but one or more other features. It is to be understood that the presence or addition of elements or numbers, steps, actions, components, parts, or combinations thereof, does not preclude in advance.

Unless otherwise defined, all terms used herein, including technical or scientific terms, have the same meaning as commonly understood by one of ordinary skill in the art to which the embodiment belongs. Terms as defined in a commonly used dictionary should be interpreted as having a meaning consistent with the meaning in the context of the related technology, and should not be interpreted as an ideal or excessively formal meaning unless explicitly defined in this application. Does not.

In addition, in the description with reference to the accompanying drawings, the same reference numerals are assigned to the same components regardless of the reference numerals, and redundant descriptions thereof will be omitted. In describing the embodiments, when it is determined that a detailed description of related known technologies may unnecessarily obscure the subject matter of the embodiments, the detailed description thereof will be omitted.

Hereinafter, exemplary embodiments of the present invention will be described in detail with reference to the accompanying drawings.

First, in order to actually use the vehicle camera calibration apparatus and method according to an embodiment of the present invention, at least some lanes must exist in images captured by a camera mounted on a vehicle, and the road is flat and a straight lane To be possible. If the road is not flat, correction is performed, but since the accuracy of the corrected external parameter is degraded, it is preferable to perform correction on the flat ground as much as possible to prevent this point.

In addition, the algorithm described below is described as an embodiment of the present invention, so it is natural that the present invention is not limited to that one described algorithm, and any other algorithm that performs the function can be used. will be.

1 is a schematic diagram of a vehicle camera calibration device according to an example.

According to an aspect, the vehicle camera calibration apparatus 100 includes a camera module 110 for obtaining images captured by a plurality of cameras mounted on a vehicle taking a lane (or lane), and an image obtained from the camera module 110. The input/output module 120 that receives and outputs the corrected image, the lane detection module 130 that detects a lane from the image received from the input/output module 120 and detects characteristic points of the lane, and the input/output module 120 A camera correction module 140 for correcting an image by estimating a new external parameter after calculating a lane equation and a lane width using initial camera information and external parameter information in the image may be included.

According to one aspect, the camera module 110 may include a plurality of cameras located at any position of the vehicle. Each camera creates an image by photographing the scene from the installed location. A lane or a portion of a lane may be included in the generated image. For example, cameras are installed in the front, rear, left and right sides of the vehicle to visualize all the surroundings of the vehicle. The number of cameras mounted on the vehicle may vary according to embodiments, and is not limited to the described embodiments.

According to an aspect, the input/output module 120 may include an image input/output unit 121 and a storage unit 122.

The image input/output unit 121 receives the image generated by the camera of the camera module 110, transmits it to the lane detection module 130, receives the corrected image from the camera correction module 140, and receives the corrected image. Can be output to an external device such as a monitor. The image input/output unit 121 may pre-process the image using an image filter or the like, if necessary.

The storage unit 122 may store an image generated by the camera and may store a correction image received from the camera correction module 140.

According to an aspect, the lane detection module 130 may include an image processing unit 131, a frame accumulating unit 132, an edge detection unit 133, and a lane determination unit 134.

The image processing unit 131 may remove noise in an image received from the image input/output unit 121 and may process the image so that a boundary portion of the lane is more clearly displayed. For example, the image processing unit 131 may apply morphology calculation and median filtering to an image in order to remove noise.

The image processing unit 131 may convert a color image into a black and white image. That is, the image processing unit 131 may use the received color image as it is, or may convert the color image into a black and white image and use it.

When the lane in the processed image is a dotted line, the frame accumulator 132 accumulates the image as a continuous frame. Images may be accumulated so that the dotted lane appears as a solid (or straight) lane. For example, four images may be accumulated and are not limited to the described embodiment. By comparing all pixel values of the four images and storing the largest pixel value in a new image, as a result, a dotted line in the image looks like a solid line.

When the lane in the image is a straight line (or a solid line), the images may not be accumulated.

The edge detection unit 133 detects an edge in a lower portion of an image (or frame) and detects a feature point. In general, in the accumulated image (or frame) of images generated based on a front or rear camera, a lane exists in the lower part of the image, and the upper part is in the sky (above the horizon). The lower part can be defined as a region of interest (ROI).

The lane determination unit 134 detects or determines a lane based on the feature points detected by the edge detection unit 133.

The camera correction module 140 corrects external parameters of cameras of the vehicle based on the detected lane. For example, external parameters of each of the four cameras may be corrected.

According to an aspect, the camera correction module 140 may include an image conversion unit 141, an equation calculation unit 142, a parameter estimation unit 143, and a correction unit 144.

The image conversion unit 141 extracts a feature point of a lane in an image based on initial camera information and external parameter information, and then converts it into a top view image.

The equation calculator 142 sets an equation of a lane by line fitting the feature points in the converted top view image. Here, a least mean square (LMS) algorithm may be used for line fitting. The suboptimal equation can be an equation of a straight line or a multidimensional equation other than a straight line.

The parameter estimating unit 143 estimates or calculates a new external parameter based on the set lane equation.

The correction unit 144 corrects the image based on the estimated new external parameter.

Hereinafter, a method of calibrating a vehicle camera will be described in detail with reference to FIGS. 2 to 19.

2 is a block diagram of an electronic device according to an exemplary embodiment.

The electronic device 200 includes a communication unit 210, a processor 220, and a memory 230. According to an embodiment, the electronic device 200 may further include a camera 240.

According to one aspect, the electronic device 200 corresponds to the vehicle camera calibration device 100. For example, the communication unit 210 may correspond to the image input/output unit 121, and the processor 220 includes an image processing unit 131, a frame accumulating unit 132, an edge detection unit 133, and a lane determining unit 134. , The image conversion unit 141, the equation calculation unit 142, the parameter estimating unit 143, and the correction unit 144, respectively, and the memory 230 may correspond to the storage unit 122.

The communication unit 210 is connected to the processor 220 and the memory 230 to transmit and receive data. The communication unit 210 may be connected to other external devices to transmit and receive data. Hereinafter, the expression "transmitting/receiving A" may refer to transmitting/receiving "information or data representing A".

The communication unit 210 may be implemented as a circuit network in the electronic device 200. For example, the communication unit 210 may include an internal bus and an external bus. As another example, the communication unit 210 may be an element connecting the electronic device 300 and an external device. The communication unit 210 may be an interface. The communication unit 210 may receive data from an external device and transmit the data to the processor 220 and the memory 230.

The processor 220 processes data received by the communication unit 210 and data stored in the memory 230. The “processor” may be a data processing device implemented in hardware having a circuit having a physical structure for executing desired operations. For example, desired operations may include code or instructions included in a program. For example, a data processing device implemented in hardware is a microprocessor, a central processing unit, a processor core, a multi-core processor, and a multiprocessor. , Application-Specific Integrated Circuit (ASIC), and Field Programmable Gate Array (FPGA).

The processor 220 executes computer-readable code (eg, software) stored in a memory (eg, memory 230) and instructions induced by the processor 220.

The memory 230 stores data received by the communication unit 210 and data processed by the processor 220. For example, the memory 230 may store a program (or application or software). The stored program may be a set of syntaxes that are coded to calibrate the camera by adjusting external parameters of the camera and executed by the processor 220.

According to an aspect, the memory 230 may include one or more volatile memories, nonvolatile memories and random access memories (RAM), flash memories, hard disk drives, and optical disk drives.

The memory 230 stores an instruction set (eg, software) for operating the electronic device 200. An instruction set for operating the electronic device 200 is executed by the processor 220.

The camera 240 generates an image by photographing a scene.

The communication unit 210, the processor 220, the memory 230, and the camera 240 will be described in detail with reference to FIGS. 3 to 19 below.

3 is a flowchart of a camera calibration method according to an exemplary embodiment.

The steps 310 to 360 below are performed by the electronic device 200 described above with reference to FIG. 2.

In step 310, the electronic device 200 captures an image using the camera 240. For example, the camera 240 may include a wide-angle lens having a larger viewing angle than a normal lens, or a fisheye lens having an ultra-wide angle lens having a viewing angle of more than 180°. When an image is generated through a lens having a large viewing angle, all around the vehicle may be photographed.

When the camera 240 includes four cameras respectively photographing the front, rear, left and right sides of the vehicle, and calibration between the cameras is performed based on the lanes within the images captured by the four cameras, the camera Pose information of each of them may be estimated. For example, the posture information may include pitch (Rx), yaw (Ry), and roll (Rz), which are external parameters indicating the degree to which the camera is rotated at a specific position. As another example, the posture information includes position movement (Tz), which is an external parameter indicating a movement degree of a specific position of the rear camera and the left and right cameras. For example, when the camera 240 includes four cameras respectively photographing front, rear, left and right, four channels for each camera may be defined.

When the straight lane appearing in the straight lane is converted to the viewpoint of the top view, the lane appears perpendicular to the bottom of the top view image, and these lanes exist in all four cameras. When the external parameters of each of the four cameras are correct, one lane in the top view image appears, and when the external parameters are incorrect, several lanes appear.

Using this point, the exact external parameter of each of the cameras can be estimated. In the following, a method of estimating an accurate external parameter using a captured image will be described.

In step 320, the electronic device 200 detects a lane in the image based on the image. A method of detecting a lane will be described in detail with reference to FIG. 4. According to one aspect, step 320 may include steps 410 to 440 of FIG. 4.

In step 410, the electronic device 200 may perform pre-processing on the image as necessary. For example, noise in an image may be removed, or a color image may be converted into a black and white image.

According to an aspect, the electronic device 200 may perform a morphology operation to smooth a boundary of an object in an image in order to control noise. The morphology operation is a type of image filter used in a preprocessing process such as image separation, that is, noise removal and feature point extraction prior to full-scale image processing, and includes an erosion operation and a dilation operation. For example, morphology opening is performed after morphology closing.

Foliage closing can remove discontinuous data in an image by smoothing the boundary of an object appearing in the image. Noise can be removed while the size of an object is maintained by morphology opening.

Median filtering can be performed to remove noise. The electronic device 200 may remove noise by changing the size of an image after median filtering is performed. In the case of the lane shown in the image generated by the left camera or the right camera, there may be a split part, but since the split part may be detected as an edge, it is possible to prevent the lane recognition rate from dropping by removing noise from the split part of the lane. I can.

In step 420, when the lane in the frame generated by the first camera is a dotted line, the electronic device 200 generates a first image by accumulating frames generated by the first camera. If the lane within the frame is not a solid line but a dotted line, the accumulated image can be created by comparing the pixel values of all the same coordinates of the accumulated frames to make the lane appear as a solid lane and storing the largest pixel value in a new image. have. If the lane within the frame is a straight line, the frames may not be accumulated.

In step 430, the electronic device 200 may detect an edge in the image by processing the image. For example, since the boundary between the color of the road and the color of the lane is clear, the edge is made strong through image processing, and then Canny Edge, which can best find the outline and remove all the edges related to the gray matter in the original image. The edge can be detected using an algorithm. The Canny Edge algorithm can detect the location of edge points with a low error rate and relatively accurately. The Canny Edge algorithm has been described as an example, but is not limited to the described embodiment.

The electronic device 200 may determine the detected edge and the center coordinate of the edge as a feature point. Since the edge may appear not only by the lane of the image but also by the noise of the image, the edge and the center coordinates of the edge may be determined as feature points in order to reduce the detected edge. When the edge and the center coordinates of the edge are determined as feature points, the lanes of the image appear as one continuous long line, and appear short or do not appear in noise.

In operation 440, the electronic device 200 may determine a lane within the image based on the feature point. For example, a contour line having the longest length among the lines indicated by the determined feature points may be determined as the lane.

The electronic device 200 may detect two contours in images generated by the front camera and the rear cameras, and may detect one contour in images generated by the left and right cameras. This is because a left lane and a right lane exist in images generated by the front and rear cameras, and a left lane and a right lane exist in images generated by the left and right cameras. The detected contour is determined as the lane.

Referring back to FIG. 3, in step 330, the electronic device 200 converts each of the captured images into a top view image based on initial camera information and initial external parameters. That is, a top view image may be generated by changing the point in time at which the image is captured to a bird view point of the virtual viewpoint.

For example, the memory 230 may store initial camera information and initial external parameters. In addition, external parameters of the new camera estimated through step 350 to be described later may be stored in the memory 230.

For example, the initial camera information may include information on internal parameters such as a focal length, a main point, and a distortion coefficient corresponding to a unique characteristic of each camera. For example, the initial external parameters may include initial posture information and initial position information of each camera.

If the external parameters are correct, the lanes in the top view image exist on the same straight line and appear vertical. However, when the position of the camera 240 is changed due to a physical force, a lane in the top view image generated based on an initial external parameter does not exist in a straight line. A top view image may be generated so that the changed external parameter of the camera may be estimated through additional processing.

In step 340, the electronic device 200 calculates an equation for a lane in the top-view image and a lane width. The calculated equation and lane width can be used to estimate new external parameters in a later step.

The electronic device 200 may set an equation of a lane by line fitting the feature points of the top view image. For line fitting, a least mean square (LMS) algorithm may be used and is not limited to the described embodiment. The calculated lane equation may be a straight line, or a multidimensional equation other than a straight line. When the equation of the lane is the equation of a straight line, it can be expressed by [Equation 1] below.

[Equation 1]

In the formula 1], a is the slope (G _l, G _r) represents the gradient (G _l), and the gradient (G _r) of the right lane to the left lane, b is the y intercept of the lane, (x, y ) Represents the coordinate point of the lane. The x and y values of [Equation 1] are values obtained from the top-view image, and the goal of calculating the equation is to determine a and b from the above values. A and b may be determined by applying line fitting based on the x and y values.

A suboptimal equation can be calculated for each of the top view images. For example, the equation of the lane for the top view image for the front image is calculated, the equation for the lane for the top view image for the rear image is calculated, the equation for the lane for the top view image for the left image is calculated, and the right A lane equation for the top-view image for the image may be calculated.

In step 350, the electronic device 200 estimates a new external parameter using the calculated lane equation and lane width. A method of estimating an external parameter will be described with reference to FIG. 5. According to one aspect, step 350 may include steps 510 to 540 of FIG. 5.

Through step 510, the pitch (R _x ), yaw (R _y ), and roll (R _z ) of each of the front and rear cameras are calculated, respectively. According to one aspect, step 510 may include steps 511, 512, 513, and 514.

In steps 511 and 513, the electronic device 200 may determine whether the pitch (R _x ), yaw (R _y ), and roll (R _z ) of each of the front and rear cameras are correct. The pitch R _x is described with reference to FIG. 6, the yaw R _y is described with reference to FIG. 7, and the roll R _z is described with reference to FIG. 8.

6 shows a case where the pitch (R _x) of the front camera of the vehicle pitch (R _x) and the rear camera is twisted. Left figure of FIG. 6 is a front camera and a pitch (R _x) of the rear camera is smaller than the value of the actual reference ground truth values, also the right side illustration of a 6 value of the pitch (R _x) of the front camera and the rear camera, the actual reference This is the case that is greater than the ground truth value.

Here, when the slope of the lane is calculated based on the side lane, it is indicated that the result values are displayed as + and-respectively. In other words, when the pitch (R _x ) of the front and rear cameras is incorrect on the top view image, the slope of the lane appears as + and -, and when the pitch (R _x ) of the front and rear cameras is correct, It is expressed as the reciprocal of the slope (1/G _l ) = the reciprocal of the slope of the right lane (1/G _r ).

7 shows a case I (R _y) of I (R _y) and a rear camera of the front camera of the vehicle is twisted. The left figure of FIG. 7 is a case where the yaw (R _y ) of the front camera and the yaw (R _y ) of the rear camera are larger than the ground truth value, which is an actual reference value, and the right figure of FIG. 7 shows the yaw (R _y ) of the front camera and the rear camera. _{This is the case where y} ) is less than the actual ground truth value. Here, it can be seen that the width of the lane appearing above the vehicle and the width of the lane appearing below are different from each other.

8 shows a case where the front camera rolls (R _z) of (R _z), and a rear camera of the vehicle is twisted. Left figure of Figure 8 is the roll of the front camera (R _z), and the rear camera roll (R _z) of the actual reference value of a is greater than ground truth values, and the right figure of Figure 8 is a front camera and a rear camera roll (Rz of ) Is less than the actual ground truth value. Here, if the roll of the front camera (R _z ) and the roll of the rear camera (R _z ) are wrong, the sign and size of (1/G _l ) and (1/G _r ) appear the same, and if it is correct, (1/ G _l ) + (1/Gr ₎ appears close to zero.

In steps 512 and 514, the electronic device 200 is the pitch (R _x), I (R _y), roll (R _z), of the front camera pitch (R _x), if this is not correct for the front camera, The yaw (R _y ) and the roll (R _z ) are recalculated.

The new pitch (R _x ) of the front camera and the rear camera may be calculated based on [Equation 2] below.

[Equation 2]

In [Equation 2], New R _x is the pitch of the new front or rear camera, α is an arbitrary constant value that can be specified by the user, G _l is the slope of the left lane, and G _r is the slope of the right lane. . That is, until 1/G _l = 1/G _r is satisfied, the new pitch (New R _x ) may be modified.

The new yaw (R _y ) of the front camera and the rear camera may be calculated based on [Equation 3] below.

[Equation 4]

In [Equation 4], New R _y is the yaw of the new front or rear camera, Pre R _y is the yaw of the current front or rear camera, α is an arbitrary constant value that can be specified by the user, and W _l is Is the width of the left lane, and W _r is the width of the right lane.

Until New R _y and Pre R _y become the same (ie, W _l = W _r ), a new yaw (New R _y ) may be modified.

A new roll (R _z ) of the front camera and the rear camera may be calculated based on [Equation 5] below.

[Equation 5]

In [Equation 5], New R _z is the roll of the new front or rear camera, Pre R _y is the roll of the current front or rear camera, α is an arbitrary constant value that can be specified by the user, and G _l is It is the slope of the left lane, and G _r is the slope of the right lane.

Until New R _z and Pre R _z become the same (i.e., 1/G _l + 1/G _r = 0), a new roll (New R _z ) can be modified.

When the pitch (R _x ), yaw (R _y ), and roll (R _z ) of the front and rear cameras are recalculated, the electronic device 200 calculates the calculated pitch (R _x ), yaw (R _y ), and roll. Each image captured by the front and rear cameras based on (R _z ) is converted back to a top view image (i.e., re-performed in step 330), and an equation of a lane and a lane based on the converted top view image Recalculate the width. After this, steps 511 to 514 may be performed again.

Through step 520, the position movement (T _z ) of the rear camera is calculated. According to one aspect, step 520 may include steps 521 and 522.

In step 521, the electronic device 200 may determine whether the position movement (T _z ) of the rear camera is correct. A method of determining whether the position movement T _z of the rear camera is correct will be described in detail with reference to FIG. 9.

9 shows a case where the position movement (T _z ) of the rear camera of the vehicle is wrong. The left figure of FIG. 9 is a case where the rear camera position movement (T _z ) is greater than the actual reference value, the ground truth value, and the right figure of FIG. 9 shows the rear camera position movement (T _z ) than the actual reference value, the ground truth value. It is a small case. Here, when the position movement (Tz) of the rear camera is wrong, the width between the lanes of the vehicle's front lane and the lanes of the rear lane appear different, and in the case of being correct, they appear the same.

In step 522, when the position movement T _{z of the} rear camera is not accurate, the electronic device 200 recalculates the position movement T _z of the rear camera.

The new positional movement (T _z ) of the rear camera may be calculated based on [Equation 6] below.

[Equation 6]

In [Equation 6], New T _z is the position movement of the new rear camera, Pre T _z is the current position movement of the rear camera, α is an arbitrary constant value that can be specified by the user, and C _f is the forward position. It is the width between the left and right lanes, and C _r is the width between the rear left and right lanes.

Until New T _z and Pre T _z become the same (ie, C _f = C _r ), the new position movement (New T _z ) may be corrected.

Through step 530, the position movement (T _z ) of the left and right cameras is calculated. According to one aspect, step 520 may include steps 521 and 522.

In step 521, the electronic device 200 may determine whether the pitch (R _x ), yaw (R _y ), roll (R _z ), and position movement (T _z ) of the rear camera are correct. The pitch (R _x ) is described with reference to FIG. 10, the yaw (R _y ) is described with reference to FIG. 11, the roll (R _z ) is described with reference to FIG. 12, and the positional movement (T _z ) Will be described with reference to FIG. 12.

10 shows a case where the pitches (R _x ) of the left and right cameras of the vehicle are wrong. The left picture of FIG. 10 is a case where the pitch (R _x ) of the left and right cameras is greater than the ground truth value, which is the actual reference value, and the right picture of FIG. 10 shows the pitch (R _x ) of the left and right cameras is the actual reference value. It is less than the ground truth value. Here, when the pitch (R _x ) of the left and right cameras is wrong, the lanes located in the front and rear of the vehicle and the lanes located on the left and right of the vehicle do not exist on one straight line, but appear to deviate. It exists on the straight line of

11 shows a case where the yaw (R _y ) of the left and right cameras of the vehicle is wrong. The left figure of FIG. 11 is a case where the yaw (R _y ) of the left and right cameras is greater than the ground truth value, which is the actual reference value, and the right figure of FIG. 11 shows the yaw (R _y ) of the left and right cameras is the actual reference value. It is less than the ground truth value. Here, if the yaw (R _y ) of the left and right cameras is correct, (X ₂₂ -X ₁₁ ) and (X _66- X ₅₅ ) are the same, or (X ₄₄ -X ₃₃ ) and (X ₈₈ -X ₇₇ ) Becomes the same.

12 shows a case in which the rolls R _z of the left and right cameras of the vehicle are misaligned. Left figure is a roll of the left camera and right camera of Fig. 12 (R _z) of the actual reference value of ground is greater than the truth values, and the right illustration in Figure 12 is the value of the actual reference roll (R _z) of the left camera and right camera It is less than the ground truth value. Here, if the rolls (R _z ) of the left and right cameras are wrong, 1/G _l and 1/G _r have a large value, and if the roll (R _z ) is correct, 1/G _l and 1/G _r are It will appear close to '0'.

13 shows a case where the position movement (T _z ) of the left and right cameras of the vehicle is wrong. The left figure of FIG. 13 is a case where the position movement (T _z ) of the left and right cameras is greater than the ground truth value, which is an actual reference value, and the right figure of FIG. 12 shows the position movement (T _z ) of the left and right cameras. It is less than the reference value of ground truth. Here, when the positional movement (T _z ) of the left and right cameras is correct, the width of the front and rear lanes of the vehicle and the width of the left and right lanes of the vehicle are the same.

In steps 534, if the pitch (R _x ), yaw (R _y ), roll (R _z ), and position movement (T _z ) of the left and right cameras are not correct, the electronic device 200 The pitch (R _x ), yaw (R _y ), roll (R _z ), and position movement (T _z ) of the camera and the right camera are recalculated.

The pitch (R _x ) of the left camera may be calculated based on [Equation 7] below.

[Equation 7]

In [Equation 7], LeftCameraNew R _x is the pitch of the new left camera, LeftCameraPre R _x is the pitch of the current left camera, α is an arbitrary constant value that can be specified by the user, and x ₁ is the left lane of the front lane Is the left x-coordinate of, x ₂ is the right x-coordinate of the left lane of the front lane, x ₅ is the left x-coordinate of the left lane of the rear lane, and x ₆ is the right x-coordinate of the left lane of the rear lane. Until LeftCameraNew R _x and LeftCameraPre R _x are equal (ie x ₂ + x ₆ = x ₁ + x ₅ ), the pitch of the new left camera is calculated.

The pitch (R _x ) of the right camera may be calculated based on [Equation 8] below.

[Equation 8]

In [Equation 8], RightCameraNew R _x is the pitch of the new right camera, RightCameraPre R _x is the pitch of the current right camera, α is an arbitrary constant value that can be specified by the user, and x ₃ is the right lane of the front lane Is the left x-coordinate of, x ₄ is the right x-coordinate of the right lane of the front lane, x ₇ is the left x-coordinate of the right lane of the rear lane, and x ₈ is the right x-coordinate of the right lane of the rear lane. Until RightCameraNew R _x and RightCameraPre R _x are equal (ie x ₃ + x ₇ = x ₄ + x ₈ ), the pitch of the new right camera is calculated.

The yaw (R _y ) of the left camera may be calculated based on [Equation 9] below.

[Equation 9]

In [Equation 9], LeftCameraNew R _y is the yaw of the new left camera, LeftCameraPre R _y is the yaw of the current left camera, α is an arbitrary constant value that can be specified by the user, and x ₁₁ is the left lane of the front lane. X-coordinate of the left side, x ₂₂ is the right x-coordinate of the left lane of the front lane, x ₅₅ is the left x-coordinate of the left lane of the rear lane, and x ₆₆ is the right x-coordinate of the left lane of the rear lane. Until LeftCameraNew R _y and LeftCameraPre R _y are equal (i.e. x ₂₂ -x ₁₁ = x ₆₆ -x ₅₅ ), the yoga of the new left camera is calculated.

The yaw (R _y ) of the right camera may be calculated based on [Equation 10] below.

[Equation 10]

In [Equation 10], RightCameraNew R _y is the yaw of the new right camera, RightCameraPre R _y is the yaw of the current right camera, α is an arbitrary constant value that can be specified by the user, and x ₃₃ is the right lane of the front lane. Is the left x-coordinate of, x ₄₄ is the right x-coordinate of the right lane of the front lane, x ₇₇ is the left x-coordinate of the right lane of the rear lane, and x ₈₈ is the right x-coordinate of the right lane of the rear lane. Until RightCameraNew R _y and RightCameraPre R _y are equal (i.e. x ₄₄ -x ₃₃ = x ₈₈ -x ₇₇ ), the pitch of the new right camera is calculated.

The roll (R _z ) of the left camera may be calculated based on [Equation 11] below.

[Equation 11]

In [Equation 11], LeftCameraNew R _z is the roll of the new left camera, LeftCameraPre R _z is the roll of the current left camera, α is an arbitrary constant value that can be specified by the user, and G _l is the slope of the left lane to be. Until LeftCameraNew R _z and LeftCameraPre R _z are equal (i.e. 1/G _l = 0), the roll of the new left camera is calculated.

The roll (R _z ) of the right camera may be calculated based on [Equation 12] below.

[Equation 12]

In [Equation 12], RightCameraNew R _z is the roll of the new right camera, RightCameraNew R _z is the roll of the current right camera, α is an arbitrary constant value that can be specified by the user, and G _r is the slope of the right lane to be. Until RightCameraNew R _z and RightCameraNew R _z are equal (i.e. 1/G _r = 0), the roll of the new right camera is calculated.

The positional movement (T _z ) of the left camera may be calculated based on Equation 13 below.

[Equation 13]

In [Equation 13], LeftCameraNew T _z is the movement of the new left camera, LeftCameraPre T _z is the movement of the current left camera, α is an arbitrary constant value that can be specified by the user, and x ₁₁₁ is the forward lane The left x-coordinate of the left lane, x ₂₂₂ is the right x-coordinate of the left lane, x ₅₅₅ is the left x-coordinate of the left lane, x ₆₆₆ is the right x-coordinate of the left lane, and x ₉₉₉ is the left side of the left lane. The x-coordinate on the left is the x-coordinate, and x ₁₀ is the x-coordinate on the right of the left lane by the rear lane. Until LeftCameraNew T _z and LeftCameraPre T _z become the same (ie (x ₂₂₂ -x ₁₁₁ )/2= x ₆₆₆ -x ₅₅₅ ), the position movement of the new left camera is calculated.

The positional movement (T _z ) of the right camera may be calculated based on [Equation 14] below.

[Equation 14]

In [Equation 14], RightCameraNew T _z is the position movement of the new right camera, RightCameraPre T _z is the current position movement of the right camera, α is an arbitrary constant value that can be specified by the user, and x ₃₃₃ is the forward lane The left x-coordinate of the right lane, x ₄₄₄ is the right x-coordinate of the right lane, x ₇₇₇ is the left x-coordinate of the right lane, x ₈₈₈ is the right x-coordinate of the right lane, and x ₁₁ is the right lane of the rear lane. Is the left x-coordinate of and x ₁₂ is the right-hand x-coordinate of the right lane of the rear lane. Until RightCameraNew T _z and RightCameraPre T _z are equal (i.e. ((x ₄₄₄ -x ₃₃₃ )+(x ₁₂ -x ₁₁ ))/2 = x ₈₈₈ -x ₇₇₇ ), the position of the new right camera is calculated. do.

In step 540, the electronic device 200 adjusts at least one roll R _z of the camera 240. A method of adjusting at least one roll R _z of the camera 240 will be described with reference to FIGS. 14 to 19.

According to one aspect, the difference between the initial roll R _z and the roll R _z calculated through the steps 510 to 530 is calculated for each channel, and if all the calculated differences are almost zero or similar, step 540 ) May not be performed. For example, if all of the calculated differences are almost zero, the calculated R _z may be relatively accurate, and the vehicle and the lane may be parallel. As another example, when the calculated difference is similar, the calculated R _z may be relatively accurate, and the vehicle and the lane may not be parallel. The difference may vary from channel to channel.

That is, when R _z calculated in a state in which the vehicle and the lane are not parallel, step 540 may be performed. Referring to FIG. 17, although the vehicle and the lane are not actually parallel, the image 1700 generated using R _z calculated through steps 510 to 530 includes front lanes 1710 and right lane 1720. ), the left lane 1730 and the rear lanes 1740 may appear parallel to the vehicle. R _z must be recalculated so that all lanes 1710 to 1740 are connected.

Referring to FIG. 14, step 540 may include the following steps 1410 to 1440.

In step 1410, the electronic device 200 for each of the channels (eg, four channels) of the camera 240, the initial R _z and R calculated through the steps 510 to 530 Compute the first difference between _z and. The initial R _z may be a value set as an initial value. The information acquired by each camera can be defined as a channel.

In step 1420, the electronic device 200 calculates a first difference average of the calculated first differences.

In step 1430, the electronic device 200 determines whether the first difference average exceeds a preset first average threshold.

When the first difference average exceeds the first average threshold, since there is an error in the initial R _z of some of the channels of the camera 240, the calculated R _z of the corresponding channel is to adjust R _z Cannot be used. For example, when the first difference calculated for the channel of the front camera is larger than the first difference of other channels, the calculated R _z of the channel of the front camera may be excluded.

If the first difference average does not exceed the first average threshold, the calculated R _z of all channels of the camera 240 may be used to adjust R _z .

In step 1440, the electronic device 200 adjusts R _z of each of the channels based on the first difference average. R _z of the front camera and the rear camera can be adjusted using [Equation 15] below.

[Equation 15]

는 전체의 채널들에 대한 상기의 제1 차이 평균이고, NewR_z는 조정된 R_z이다. [수학식 15]가 반복적으로 수행되는 경우, PreR_z는 이전에 조정된 R_z이 될 수 있다.In [Equation 15], PreR _z is R _z of the front or rear camera calculated through steps 510 to 530,

Is the difference between the first average for the whole channel, NewR _z is the adjusted R _z. When [Equation 15] is repeatedly performed, PreR _z may be previously adjusted R _z .

After adjusting R _z of the front and rear cameras, an exemplary image generated is as shown in FIG. 18. Front lanes and rear lanes in the image after R _z of the front and rear cameras are adjusted may not be parallel to the vehicle.

When R _z of the left and right cameras are not adjusted, the first point 1801 and the third point 1803 are not matched to the same point, and the second point 1802 and the fifth point 1805 are the same. The points are not matched, the fourth point 1804 and the seventh point 1807 are not matched as the same point, and the sixth point 1806 and the eighth point 1808 are not matched as the same point. R _z of the left camera and the right camera may be adjusted so that the above points are matched, respectively.

For example, R _z of the left and right cameras may be adjusted so that points in the image are matched. For example, R _z of the left camera and the right camera may be adjusted so that the point of the front left lane and the point of the left lane located on the same lane match.

R _z of the left camera can be adjusted using the following [Equation 16].

[Equation 16]

,

는 각각 제1 포인트(1801)의 x 좌표 및 y 좌표이고,

,

는 각각 제7 포인트(1807)의 x 좌표 및 y 좌표이다.

,

는 각각 제3 포인트(1803)의 x 좌표 및 y 좌표이고,

,

는 제4 포인트(1804)의 x 좌표 및 y 좌표이다.In [Equation 16], LeftCameraPreR _z may be R _z of the left camera calculated through steps 510 to 530, and LeftCameraNewR _z may be R _z of the adjusted left camera. When [Equation 16] is repeatedly performed, LeftCameraPreRz may be previously adjusted R _z .

,

Is the x coordinate and y coordinate of the first point 1801, respectively,

,

Is the x coordinate and y coordinate of the seventh point 1807, respectively.

,

Is the x coordinate and y coordinate of the third point 1803, respectively,

,

Is the x coordinate and y coordinate of the fourth point 1804.

이 0이 되는 경우), 좌측 카메라의 R_z 조정이 완료될 수 있다.If LeftCameraPreR _z and LeftCameraNewR _z become equal (i.e.

Is 0), the left camera's R _z adjustment can be completed.

R _z of the right camera can be adjusted using [Equation 17] below.

[Equation 17]

,

는 각각 제2 포인트(1802)의 x 좌표 및 y 좌표이고,

,

는 각각 제8 포인트(1808)의 x 좌표 및 y 좌표이다.

,

는 각각 제5 포인트(1805)의 x 좌표 및 y 좌표이고,

,

는 각각 제6 포인트(1806)의 x 좌표 및 y 좌표이다.In [Equation 17], RightCameraPreR _z may be R _z of the right camera calculated through steps 510 to 530, and RightCameraNewR _z may be R _z of the adjusted right camera. When [Equation 17] is repeatedly performed, RightCameraPreR _z may be previously adjusted R _z .

,

Is the x coordinate and y coordinate of the second point 1802, respectively,

,

Is the x coordinate and y coordinate of the eighth point 1808, respectively.

,

Is the x coordinate and y coordinate of the fifth point 1805, respectively,

,

Is the x coordinate and y coordinate of the sixth point 1806, respectively.

이 0이 되는 경우), 우측 카메라의 R_z 조정이 완료될 수 있다.If RightCameraPreR _z and RightCameraNewR _z become equal (i.e.

Is 0), the R _z adjustment of the right camera can be completed.

와 우측 차선의 기울기인

는 변하지 않는 값이기 때문에 그대로 두면 되고, 각 좌측 차선의 포인트 및 우측 차선의 포인트를 이용한 좌측 차선의 기울기인

와 우측 차선의 기울기인

는 좌측 카메라의 R_z와 우측 카메라의 R_z가 조정될 때마다 변화할 수 있다.The slope of the left lane using the points of the front lane and the rear lane in the equation

And the slope of the right lane

Is an unchanging value, so you can leave it as it is, and the slope of the left lane using the points of each left lane and

And the slope of the right lane

It may change each time the R _z of R _z and right camera of the left camera to be adjusted.

19 illustrates an image 1900 that is finally generated when R _z of each of the four cameras (or channels) is appropriately adjusted. When R _z is properly adjusted, the slopes of the front lanes, the left lane, the right lane and the rear lanes may appear the same.

Referring back to FIG. 14, when it is determined in step 1430 that the first difference average exceeds the first average threshold, step 1500 described with reference to FIG. 15 is additionally performed.

According to an aspect, step 1500 of FIG. 15 may include steps 1510 to 1530.

In step 1510, the electronic device 200 calculates a second difference average of the first differences calculated for channels (eg, three channels) excluding one channel from all channels. For example, a second difference average for three channels excluding the channel of the front camera may be calculated.

In step 1520, the electronic device 200 determines whether the calculated second difference average exceeds a preset second average threshold. The second average threshold may be the same as or different from the first average threshold.

When the second difference average exceeds the second average threshold, since there is an error in the initial R _z of some of the three channels of the camera 240, the calculated R _z of the corresponding channel adjusts R _z It cannot be used to do. For example, when the first difference calculated for the channel of the left camera is larger than the first difference of other channels, the calculated R _z of the channel of the left camera may be additionally excluded. In this case, step 1600 described with reference to FIG. 16 may be additionally performed.

If the second difference average does not exceed the second average threshold, the second difference average may be used to adjust R _z .

가 제2 차이 평균으로 대체될 수 있다.In step 1530, the electronic device 200 adjusts R _z of each of the four channels based on the second difference average. A description of a method of adjusting R _z of each of the four channels may be replaced with a description of step 1440. Of [Equation 15]

Can be replaced by the second difference mean.

Referring back to FIG. 15, when it is determined in step 1520 that the second difference average exceeds the second average threshold, step 1600 described with reference to FIG. 16 is additionally performed.

According to an aspect, step 1600 of FIG. 16 may include steps 1610 to 1630.

In step 1610, the electronic device 200 calculates a third difference average of the first differences calculated for channels (for example, two channels) excluding two channels from the entire channels. do. For example, a third difference average for two channels excluding the front channel and the left channel may be calculated.

In step 1620, the electronic device 200 determines whether the calculated third difference average exceeds a preset third average threshold. The third average threshold may be the same as or different from the first average threshold and the second average threshold. 3 if the average exceeds the threshold, the third threshold average, is determined to be no longer able to adjust the R _z R _z may be the adjustment end.

가 제3 차이 평균으로 대체될 수 있다.In step 1630, the electronic device 200 adjusts R _z of each of the four channels based on the third difference average. A description of a method of adjusting R _z of each of the four channels may be replaced with a description of operation 1440. Of [Equation 15]

Can be replaced by the third difference mean.

Referring back to FIG. 5, in step 550, the electronic device 200 stores the external parameter finally calculated through steps 510 to 540.

Referring back to FIG. 3, in step 360, the electronic device 200 corrects the image based on the calculated external parameter. The corrected image may be a final (or target) top view image.

Since the above-described steps 310 to 360 may be performed while the vehicle is driving, external parameters may be automatically calibrated.

As described above, the present invention receives images of a driving lane of a vehicle from a plurality of cameras mounted on a vehicle, detects a lane from the captured image, and uses the detected lane to detect the exterior of the camera mounted on the vehicle. Since camera calibration to correct parameters is performed, there is an effect that camera calibration can be performed quickly and easily without stopping the vehicle.

The method according to the embodiment may be implemented in the form of program instructions that can be executed through various computer means and recorded in a computer-readable medium. The computer-readable medium may include program instructions, data files, data structures, and the like alone or in combination. The program instructions recorded on the medium may be specially designed and configured for the embodiment, or may be known and usable to those skilled in computer software. Examples of computer-readable recording media include magnetic media such as hard disks, floppy disks, and magnetic tapes, optical media such as CD-ROMs and DVDs, and magnetic media such as floptical disks. -A hardware device specially configured to store and execute program instructions such as magneto-optical media, and ROM, RAM, flash memory, and the like. Examples of the program instructions include not only machine language codes such as those produced by a compiler, but also high-level language codes that can be executed by a computer using an interpreter or the like. The hardware device described above may be configured to operate as one or more software modules to perform the operation of the embodiment, and vice versa.

The software may include a computer program, code, instructions, or a combination of one or more of these, configuring the processing unit to behave as desired or processed independently or collectively. You can command the device. Software and/or data may be interpreted by a processing device or to provide instructions or data to a processing device, of any type of machine, component, physical device, virtual equipment, computer storage medium or device. , Or may be permanently or temporarily embodyed in a transmitted signal wave. The software may be distributed over networked computer systems and stored or executed in a distributed manner. Software and data may be stored on one or more computer-readable recording media.

As described above, although the embodiments have been described by the limited drawings, a person of ordinary skill in the art can apply various technical modifications and variations based on the above. For example, the described techniques are performed in a different order from the described method, and/or components such as a system, structure, device, circuit, etc. described are combined or combined in a form different from the described method, or other components Alternatively, even if substituted or substituted by an equivalent, an appropriate result can be achieved.

Therefore, other implementations, other embodiments and claims and equivalents fall within the scope of the following claims.

110: camera module 120: input/output module
121: image input/output unit 122: storage unit
130: lane detection module 131: image processing unit
132: frame accumulation unit 133: edge detection unit
134: lane determination unit 140: camera correction module
141: image conversion unit 142: equation calculation unit
143: parameter estimation unit 144: correction unit

Claims (22)

A camera calibration method performed by a device for calibrating a plurality of cameras,
Generating a plurality of images using a plurality of cameras photographing around the vehicle;
Detecting lanes within the plurality of images;
Generating a top view image using a plurality of images based on initial camera information and initial external parameters;
Calculating an equation for lanes in the top-view image; And
Calibrating the plurality of cameras by calculating external parameters for the plurality of cameras based on the equation
Containing,
How to calibrate the camera.
The method of claim 1,
The step of detecting a lane within the plurality of images,
Generating a first image by accumulating frames generated by the first cameras of the plurality of cameras;
Detecting an edge in the first image; And
Determining a lane of the first image based on the edge
Containing,
How to calibrate the camera.
The method of claim 2,
The step of detecting a lane within the plurality of images,
Removing noise in the first image; And
Converting the first image into a black and white image
Further comprising at least one of,
The step of detecting an edge in the first image,
Detecting an edge in the first image from which noise has been removed or the first image converted into a black and white image
Containing,
How to calibrate the camera.
The method of claim 1.
The initial external parameter includes initial posture information and initial position information of the camera,
The initial posture information includes pitch (R _x ), yaw (R _y ), and roll (R _z ) of the camera,
The initial position information includes a position movement (T _z ) of the camera,
How to calibrate the camera.
The method of claim 4,
Calibrating the plurality of cameras by calculating external parameters for the plurality of cameras based on the equation,
Calculating a pitch (R _x ), a yaw (R _y ), and a roll (R _z ) for each of the front and rear cameras among the plurality of cameras;
Calculating a positional movement (T _z ) of the rear camera based on a pitch (R _x ), a yaw (R _y ), and a roll (R _z ) for each of the front camera and the rear camera; And
Based on the position movement (T _z ) of the rear camera, the pitch (R _x ), yaw (R _y ), roll (R _z ), and position movement (T) for each of the left and right cameras among the plurality of cameras Steps to calculate _z )
Containing,
How to calibrate the camera.
The method of claim 5.
Calibrating the plurality of cameras by calculating external parameters for the plurality of cameras based on the equation,
Adjusting the roll (R _z ) of each of the plurality of cameras
Including more,
Adjusting the roll (R _z ),
Calculating a first difference between an initial roll and a calculated roll (R _z ) for each of the plurality of cameras;
Calculating a first difference average of the calculated first differences;
Determining whether the first difference average exceeds a preset first average threshold;
When the first difference average does not exceed the first average threshold, adjusting the roll R _z of each of the plurality of cameras based on the first difference average
Containing,
How to calibrate the camera.
The method of claim 6,
The step of adjusting the roll,
When the first difference average exceeds the first average threshold value, the roll (R) of each of the plurality of cameras is based on first differences of cameras other than the camera representing the largest difference among the first differences. _z ) adjusting step
Further comprising,
How to calibrate the camera.
The method of claim 6,
Adjusting the roll (R _z ) of each of the plurality of cameras based on the first difference average,
Adjusting the roll (R _z ) of the front camera and the rear camera based on the first difference average
Containing,
How to calibrate the camera.
The method of claim 8.
Adjusting the roll (R _z ) of the front camera and the rear camera based on the first difference average,
Adjusting the roll (R _z ) of the front camera by subtracting the average of the first difference from the previous roll (R _z ) of the front camera
Containing,
How to calibrate the camera.
The method of claim 6,
Adjusting the roll (R _z ) of each of the plurality of cameras based on the first difference average,
Adjusting the roll (R _z ) of the left camera so that the point of the front left lane and the point of the left lane located on the same lane are matched
Containing,
How to calibrate the camera.
A computer-readable recording medium containing a program for performing the method of any one of claims 1 to 10.
A device for calibrating a plurality of cameras,
A plurality of cameras for generating an image around the vehicle;
A memory in which a program for calibrating the plurality of cameras is recorded; And
Processor that executes the above program
Including,
The above program,
Generating a plurality of images using the plurality of cameras;
Detecting lanes within the plurality of images;
Generating a top view image using a plurality of images based on initial camera information and initial external parameters;
Calculating an equation for lanes in the top-view image; And
Calibrating the plurality of cameras by calculating external parameters for the plurality of cameras based on the equation
To do,
Camera calibration device.
The method of claim 12,
The step of detecting a lane within the plurality of images,
Generating a first image by accumulating frames generated by the first cameras of the plurality of cameras;
Detecting an edge in the first image; And
Determining a lane of the first image based on the edge
Containing,
Camera calibration device.
The method of claim 13,
The step of detecting a lane within the plurality of images,
Removing noise in the first image; And
Converting the first image into a black and white image
Further comprising at least one of,
The step of detecting an edge in the first image,
Detecting an edge in the first image from which noise has been removed or the first image converted to a black and white image
Containing,
Camera calibration device.
The method of claim 12.
The initial external parameter includes initial posture information and initial position information of the camera,
The initial posture information includes pitch (R _x ), yaw (R _y ), and roll (R _z ) of the camera,
The initial position information includes a position movement (T _z ) of the camera,
Camera calibration device.
The method of claim 15,
Calibrating the plurality of cameras by calculating external parameters for the plurality of cameras based on the equation,
Calculating a pitch (R _x ), a yaw (R _y ), and a roll (R _z ) for each of the front and rear cameras among the plurality of cameras;
Calculating a positional movement (T _z ) of the rear camera based on a pitch (R _x ), a yaw (R _y ), and a roll (R _z ) for each of the front camera and the rear camera; And
Pitch (R _x ), yaw (R _y ), roll (R _z ), and position movement (T) for each of the left and right cameras among the plurality of cameras based on the position movement (T _z ) of the rear camera. Steps to calculate _z )
Containing,
Camera calibration device.
The method of claim 16.
Calibrating the plurality of cameras by calculating external parameters for the plurality of cameras based on the equation,
Adjusting the roll (R _z ) of each of the plurality of cameras
Including more,
Adjusting the roll (R _z ),
Calculating a first difference between an initial roll and a calculated roll (R _z ) for each of the plurality of cameras;
Calculating a first difference average of the calculated first differences;
Determining whether the first difference average exceeds a preset first average threshold;
When the first difference average does not exceed the first average threshold, adjusting the roll R _z of each of the plurality of cameras based on the first difference average
Containing,
Camera calibration device.
The method of claim 17,
The step of adjusting the roll,
When the first difference average exceeds the first average threshold, the roll (R) of each of the plurality of cameras is based on first differences between the cameras other than the camera representing the largest difference among the first differences. _z ) adjusting step
Further comprising,
Camera calibration device.
The method of claim 17,
Adjusting the roll (R _z ) of each of the plurality of cameras based on the first difference average,
Adjusting the roll (R _z ) of the front camera and the rear camera based on the first difference average
Containing,
Camera calibration device.
The method of claim 19.
Adjusting the roll (R _z ) of the front camera and the rear camera based on the first difference average,
Adjusting the roll (R _z ) of the front camera by subtracting the average of the first difference from the previous roll (R _z ) of the front camera
Containing,
Camera calibration device.
The method of claim 17,
Adjusting the roll (R _z ) of each of the plurality of cameras based on the first difference average,
Adjusting the roll (R _z ) of the left camera so that the point of the front left lane and the point of the left lane located on the same lane are matched
Containing,
Camera calibration device.
The top-view image generation method performed by the device for generating the top-view image,
Generating a plurality of images using a plurality of cameras;
Calibrating the plurality of cameras by calculating external parameters for the plurality of cameras based on the plurality of images-The external parameters are the pitch (R _x ), yaw (R _y ), and roll ( Including R _z ) -;
Generating a plurality of images using the calibrated plurality of cameras; And
Generating a top view image based on the plurality of images
Containing,
How to create a top view image.

Images