JP5240517B2 - Car camera calibration system - Google Patents

Description

  The present invention relates to an in-vehicle camera calibration apparatus for calibrating an in-vehicle camera.

  In recent years, an increasing number of vehicles are equipped with a camera so that a driver of the vehicle can visually recognize a scene such as a side or rear of the vehicle via a monitor inside the vehicle. Furthermore, an apparatus that supports driving such as parking by performing image processing using a photographed image of the camera has been developed. In particular, for a camera that captures a captured image serving as a base for creating information used for vehicle positioning, calibration such as optical axis adjustment is required with high accuracy. For example, Japanese Unexamined Patent Application Publication No. 2008-131250 (Patent Document 1) takes a marker (calibration index) having a black and white checkered pattern arranged at two locations in the field of view of a camera with an in-vehicle camera, and performs image processing. A technique for detecting the center point (calibration point) of the marker through the camera and calibrating the in-vehicle camera is shown.

  Further, the following techniques are known in the technical field of image scanners that accurately detect the position of a target previously written on a document for document alignment. Japanese Patent Laid-Open No. 9-153139 (Patent Document 2) discloses a video signal for converting a captured image signal of a target composed of a concentric circle pattern of red, blue and green into a color difference signal and detecting the position of the target using the color difference signal. A processing unit is shown.

JP 2008-131250 A (23rd to 40th paragraphs, FIG. 1 etc.) JP-A-9-153139 (17th-30th paragraphs, FIG. 1 etc.)

  In the technique of Patent Document 1, it is necessary to accurately detect the center point of the black and white checkered pattern. However, depending on the illumination light and sunlight conditions, the edge detection accuracy of the black and white region becomes unstable, and the center point of the marker The detection accuracy may be reduced. In the technique of Patent Document 2, since the color difference signal is used for target detection, it is accurate that the target pattern color information (color difference component) is sufficiently held in the captured image signal of the target. Is a prerequisite for accurate position detection. Since the image scanner can ignore the influence of disturbance light, it can be considered that an accurate color difference component is held in the captured image signal of the target. On the other hand, in target shooting using an in-vehicle camera, the target is shot under the influence of ambient light, and therefore it is difficult to detect the target position accurately by simply using the color difference signal. In addition, there is a possibility of erroneous recognition even if an object similar to the target pattern is placed in the camera field of view. Strict management of the workplace to prevent this can be cumbersome and can reduce productivity.

  Against this background, there is a need for a technology that enables accurate detection and position calculation of calibration points from captured image data obtained by photographing a calibration index consisting of a combination pattern of multiple colors under the influence of ambient light. ing. However, it takes a lot of calculation time to extract a calibration point from image data of a calibration index having a pattern such as a checkered pattern through image processing such as edge detection. When a colored calibration index is used in order to suppress the above-described problem caused by using a black and white calibration index, the captured image data also becomes color image data, which increases the amount of data and the calculation time. Therefore, there is a need for a technique that can efficiently and accurately detect a calibration point for calibration processing of an in-vehicle camera.

  The present application was devised in view of the above-mentioned problems, and calibration points can be efficiently and accurately obtained from captured image data obtained by capturing a calibration index composed of a plurality of color combination patterns under the influence of ambient light. It is an object of the present invention to provide an in-vehicle camera calibration apparatus capable of detecting the position and calculating the position.

In order to solve the above problems, the characteristic configuration of the calibration device for an in-vehicle camera according to the present invention is:
An image receiving unit that receives captured images taken by a vehicle-mounted camera that is arranged in at least two different locations and includes a calibration index within the imaging range with the intersection of the boundary lines of the grid pattern of the first color and the second color as calibration points When,
A first region setting unit that sets a first region including a first calibration point that is one of the calibration points on the captured image;
A first boundary line detection unit that detects a first boundary line that is one of the boundary lines of the lattice pattern of the calibration index in the first region;
A first intersection boundary detection unit that detects a first intersection boundary that intersects the first boundary in the first region;
A first calibration point detector that detects the coordinates of the intersection as the first calibration point based on the coordinates of the detected first boundary line and the first intersection boundary line on the captured image;
Based on the detected coordinate value of the first calibration point on the photographed image, a second region including a second calibration point different from the first calibration point is represented on the photographed image by the first region. A second region setting unit that sets the narrow range,
A second boundary line detection unit that detects a second boundary line that is one of the boundary lines of the lattice pattern of the calibration index in the second region;
A second intersection boundary detection unit for detecting a second intersection boundary intersecting the second boundary in the second region;
A second calibration point detector that detects the coordinates of the intersection as the second calibration point based on the coordinates of the detected second boundary line and the second intersection boundary line on the captured image;
The coordinates in the reference coordinate system indicating the arrangement positions of the first calibration point and the second calibration point in space, the first calibration point detected on the captured image, and the second calibration point in the camera coordinate system And a calibration calculation unit that calibrates the in-vehicle camera based on the coordinates.

  According to this configuration, the second region is set based on the first calibration point detected previously. Since the first calibration point is a point that has already been detected, the coordinate value is accurate, and the possibility that the calibration index is included in the second region set based on this is increased. Therefore, the second region can be a narrower region, that is, a region constituted by a smaller number of pixels than the first region. The second boundary line and the second intersecting boundary line are detected in the second region, but the number of pixels to be subjected to image processing at the time of detection is smaller than that in the first region. Therefore, calculation time can be suppressed. In addition, since the second region is set based on the first calibration point, the second calibration point is detected stably. As a result, it is possible to provide an in-vehicle camera calibration device capable of efficiently and accurately detecting a calibration point and calculating a position.

Another characteristic configuration of the on-vehicle camera calibration device according to the present invention is as follows.
An image receiving unit that receives captured images taken by a vehicle-mounted camera that is arranged in at least two different locations and includes a calibration index within the imaging range with the intersection of the boundary lines of the grid pattern of the first color and the second color as calibration points When,
A first region setting unit that sets a first region including a first calibration point that is one of the calibration points on the captured image;
A first boundary line detection unit that detects a first boundary line that is one of the grid pattern boundary lines of the calibration index in the first region;
Based on the detected coordinates of the first boundary line on the photographed image, a second area including a second calibration point different from the first calibration point on the photographed image is more than the first area. A second region setting unit for setting as a narrow range;
A first intersection boundary detection unit that detects a first intersection boundary that intersects the first boundary in the first region;
A second boundary line detection unit for detecting a second boundary line that is one of the grid pattern boundary lines of the calibration index in the second region;
A second intersection boundary detection unit for detecting a second intersection boundary intersecting the second boundary in the second region;
A first calibration point detector that detects the coordinates of the intersection as the first calibration point based on the coordinates of the detected first boundary line and the first intersection boundary line on the captured image;
A second calibration point detector that detects the coordinates of the intersection as the second calibration point based on the coordinates of the detected second boundary line and the second intersection boundary line on the captured image;
The coordinates in the reference coordinate system indicating the arrangement positions of the first calibration point and the second calibration point in space, the first calibration point detected on the captured image, and the second calibration point in the camera coordinate system And a calibration calculation unit that calibrates the in-vehicle camera based on the coordinates.

  According to this configuration, the second region is set on the basis of the first boundary line of one calibration index detected earlier. Since the first boundary line is a straight line including the first calibration point, the coordinate value of the first calibration point is included between the coordinate values at both ends thereof. That is, since the first calibration point is detected with a certain degree of accuracy, there is a high possibility that the calibration index is included in the second region set based on this coordinate. Therefore, the second region can be a narrower region, that is, a region constituted by a smaller number of pixels than the first region. The second boundary line and the second intersecting boundary line are detected in the second region, but the number of pixels to be subjected to image processing at the time of detection is smaller than that in the first region. Therefore, calculation time can be suppressed. In addition, since the second region is set based on the coordinates near the first calibration point, the second calibration point can be detected stably. As a result, it is possible to provide an in-vehicle camera calibration device capable of efficiently and accurately detecting a calibration point and calculating a position.

In addition, the in-vehicle camera calibration device according to the present invention includes the above-described components,
A first narrowed area setting unit configured to set a first narrowed area narrower than the first area in the first area based on the detected coordinate value of the first boundary line on the captured image;
A second narrowed area setting unit configured to set a second narrowed area narrower than the second area in the second area based on the detected coordinate value of the second boundary line on the captured image; Prepared,
The first intersecting boundary line detection unit detects the first intersecting boundary line in the first narrowing region,
The second cross-boundary detection portion, it is preferable to detect the second intersection boundary line in the second narrowing region.

  The first and second narrowed areas are areas that are greatly narrowed with respect to the first and second areas. Image processing such as an edge detection algorithm is used to detect the grid pattern boundary of the calibration index. However, if the number of pixels subject to image processing is small, the calculation load is reduced and the calculation time is reduced accordingly. Shortened. In addition, since the area to be subjected to image processing becomes narrow, even if there are straight lines similar to these near the first and second intersecting boundary lines, they are outside the range of the first and second narrowing areas. If it exists, it will not be detected. Accordingly, noise resistance is improved and stable image processing is possible.

  Hereinafter, embodiments of the present invention will be described with reference to the drawings. FIG. 1 is an explanatory diagram illustrating an example of an arrangement relationship between the calibration marker 2 (calibration index) and the vehicle 1. The camera 1 (vehicle camera) is installed above the license plate behind the vehicle 100 at a position offset laterally from the longitudinal axis of the vehicle body with the camera optical axis facing downward (for example, 30 ° downward from the horizontal). . Of course, the camera 1 may be arranged in the center without being offset. In the illustrated example, the longitudinal axis of the vehicle body and the camera optical axis are not parallel. The camera 1 is, for example, a wide-angle camera with a horizontal viewing angle of 110 to 120 °, and can capture an area up to about 8 m behind. The camera 1 is calibrated to absorb mounting errors when mounted on the vehicle 100 at a vehicle production factory or the like. Further, in a repair shop or the like, calibration is performed to correct the displacement of the camera 1 due to the accumulation of vibration caused by running or impact. Hereinafter, camera calibration in an automobile production factory will be described as an example.

  As shown in FIG. 1, the calibration of the camera 1 is performed in a state where the vehicle 100 is stopped at a predetermined position. For example, if the vehicle 100 is moved back or forward and the wheels are stopped by a tire groove or a tire stop provided at a predetermined position, the vehicle 100 can be stopped at an accurate position. In the example shown in FIG. 1, two calibration indices 2 (2a, 2b) are arranged on the floor surface. As is apparent from the figure, the interval between the two calibration indices 2a and 2b is narrower than the tread width of the vehicle 1, and the marker 2 is difficult to be stepped on by the wheels of the vehicle 1. Contrary to this arrangement, an arrangement may be adopted in which the interval between the two markers 2 a and 2 b is wider than the tread width of the vehicle 100 so that the marker 2 is not easily stepped on by the wheels of the vehicle 1.

In FIG. 1, the vehicle 100 is stopped so that the floor surface at the center of the rear end is the origin O _{W of the} world coordinate system (reference coordinate system, X _W , Y _W , Z _W ). Here, in the camera coordinate system (X _C , Y _C , Z _C ) centered on the optical center O _C of the camera 1, each axis is not parallel to the world coordinate system. World coordinate system, a camera coordinate system are both right-handed coordinate system, in the figure, X _W axis in a direction perpendicular to the paper surface, and is not shown X _C-axis of the substantially vertical direction. Coordinate conversion between the world coordinate system and the camera coordinate system can be performed using a well-known calculation method as described in, for example, Japanese Patent Application Laid-Open No. 2008-131250 (Patent Document 1).

  The markers 2 are arranged in at least two places within the visual field range of the camera 1. The marker 2 is arranged so that its coordinates are known in the world coordinate system. FIG. 2 is an explanatory diagram showing an example of the calibration marker 2. In this example, the marker 2 has a blue and red checkered pattern (an example of a lattice pattern) as shown in FIG. The intersection of the boundary lines of the pattern, that is, the center point Q is a calibration point and serves as a calibration reference for the camera 1. That is, the marker 2 is arranged so that the coordinates of the calibration point Q are known in the world coordinate system. In addition, although the example of a total of four rectangles of two blue rectangles and two red rectangles was shown here, there may be a total of four or more, and the number and shape are shown here. It is not limited to.

  The marker 2 is directly or indirectly disposed on the floor surface exposed to ambient light such as light from a fluorescent lamp or sunlight incident from a window glass. Indirect arrangement is a case where the marker 2 is arranged on a pedestal or the like. In any case, image recognition becomes difficult depending on the environment light reflection state. Therefore, in the present invention, by adopting a coating method in which a matte satin-like paint is applied twice to the iron plate, the surface of the marker 2 is made to be in a matte state, and a photographed image that is not easily affected by reflection of ambient light or the like can be obtained. I have to.

In the example shown in FIG. 1, the two markers 2 are arranged on the floor symmetrically with respect to the main axis of the vehicle (the Z _W axis of the world coordinate system) (D1 = D2, W1 = W2). However, it is not always necessary to be left-right symmetric. If the coordinate values are known within the field of view of the camera 1, the arrangement is free. That is, the arrangement of the markers 2 can be arbitrarily set depending on the area that can be secured in the factory and the relationship with other equipment. There is no problem if the coordinate value of the calibration point Q is known in the world coordinate system.

  The dimension of the marker 2 is appropriately determined so that the calibration point Q can be detected with high accuracy in accordance with the resolution of the camera 1, the performance of the image processing function for processing a photographed image by the camera 1, the arrangement position of the marker, and the like. As an example, when D1 and D2 are about 1 to 2 m and W1 and W2 are about 0.5 m, as shown in FIG. 2, markers 2 each having a square of 10 to 15 cm square and a whole of 20 to 30 cm square are provided. Used. The marker 2 may be installed not on the floor but on the wall or on the screen. That is, the marker 2 can be placed on any surface as long as it is within the field of view of the camera 1.

  FIG. 3 is a block diagram schematically showing the configuration of the in-vehicle camera calibration device of the present invention. As shown in the figure, the calibration device includes an image receiving unit 3, an image output unit 4, a system control unit 5, an input I / F unit 6, an output I / F unit 7, an image processing unit 10, have. Each of these units shows a sharing as a function, and is not necessarily provided independently. Of course, each function may be realized by cooperation of hardware such as a microcomputer and software such as a program executed on the hardware.

  The input I / F unit 6 receives an input signal such as an instruction to start calibration of the camera 1 from a system higher than the calibration device of the in-vehicle camera, and transmits it to the system control unit 5. Here, the host system is an assembly adjustment system of the vehicle 100, a back monitor system of the vehicle 100, a navigation system, a multimedia system, or the like. The system control unit 5 controls the entire calibration apparatus of the camera 1 and controls the image processing unit 10 and the like based on an instruction to start calibration. The output control unit 7 receives an image processing result or the like via the system control unit 5 and outputs an electrical signal to a host system, a notification unit of the calibration apparatus, or the like.

  The image receiving unit 3 receives an image captured by the camera 1 which is a digital video camera via a known image I / F such as a buffer 3a, a synchronization separation unit 3b, a clock generation unit 3c, an A / D converter 3c, and the like, and a frame memory Store in 3e. An image data control unit (not shown) provided in the image processing unit 10 controls the image receiving unit 3. For example, it controls storage of captured images in the frame memory 3e and reading of captured images from the frame memory 3e. The image processing unit 10 has a pre-processing unit (not shown), and if necessary, level adjustment such as correction of image distortion caused by lens characteristics of the camera 1, density correction, contrast correction, shading correction, etc. Execute.

  In addition, the image processing unit 10 includes a first region setting unit 11, a first boundary line detection unit 12, a first narrowed region setting unit 13, a first intersection boundary line detection unit 14, a first calibration point detection unit 15, a second Each functional unit includes an area setting unit 21, a second boundary line detection unit 22, a second narrowed area setting unit 23, a second intersection boundary line detection unit 24, a second calibration point detection unit 25, a calibration calculation unit 31, and the like. Configured. Details of each functional unit will be described later.

  FIG. 4 is an explanatory diagram illustrating the principle of setting a capture area. As shown in the figure, when the optical axis of the camera 1 is directed in the direction as designed, the marker 2 placed at a predetermined position on the reference coordinates is taken on the photographed image (hv coordinate). The capture region R1 is set based on the position of the calibration point Q. In the present embodiment, as described above, the two markers 2 are used. Here, the capture region R1 is first set with the calibration point Q of the first marker 2a being one of them as a reference. The capture region R1 corresponds to the first region of the present invention.

  The shift in the rotation direction of the camera 1 may occur in three directions of roll, tilt, and pan, with the respective axes of the three-dimensional orthogonal coordinates as rotation axes. Therefore, when the optical axis of the camera 1 is directed as designed, even if a deviation of ± 3 degrees occurs, the checkered borders H1 and V1 including the calibration point Q are always present. The capture region R1 is set so as to be included. The first area setting 11 sets the capture area R1 so that the calibration point Q (first calibration point) of the first marker 2a is included on the captured image IM.

  When the capture region R1 is set, the first boundary line detection unit 12 detects the first boundary line V1 that is one of the boundary lines of the checkered cross in the capture region R1. The first boundary line V1 is a boundary line that extends in the direction intersecting the direction in which the two markers 2 are arranged, in this example, the v-axis direction of the captured image IM. However, the first boundary line does not prevent the boundary line h1 extending along the direction in which the two markers 2 are arranged, in this example, along the h-axis direction of the captured image IM. In the present embodiment, as shown in FIG. 1, since the two markers 2 are arranged in the vehicle width direction, the boundary line extending in the direction along the central axis in the traveling direction of the vehicle 100 is the first boundary. Detected as line V1.

  The first boundary line detection unit 12 detects the first boundary line V1 in the capture region R1. FIG. 5 is an explanatory diagram showing the principle of detecting the first boundary line V1 in the capture region R1. The first boundary line detection unit 12 detects the first boundary line V1 using a known edge detection algorithm and straight line recognition algorithm described in, for example, Japanese Patent Application Laid-Open No. 2008-131250 (Patent Document 1). be able to. Here, the first boundary line V1 is a straight line extending in the vertical direction in FIG. Therefore, the first boundary line detection unit 12 only needs to detect the edge in the direction of the broken line arrow in FIG. 5 and the direction in the direction of 45 degrees with respect to this arrow without trying to detect the edge in all directions. As a result, the calculation time for detecting the first boundary line V1 is greatly shortened.

  In the present embodiment, as described above, the marker 2 is a colored indicator. Therefore, when the algorithm described below is further applied, the tolerance to the influence of ambient light is enhanced. In order to implement this algorithm, the first boundary line detection unit 12 includes a color difference conversion unit (not shown), a threshold setting unit, a color area determination unit, and a boundary detection unit. These functional units are also included in a first intersection boundary detection unit 14, a second boundary detection unit 22, and a second intersection boundary detection unit 24 described later.

  The color difference conversion unit calculates the RGB pixel value of the captured image IM developed as the RGB image data in the frame memory 3e to obtain the blue component value, the red component value, and the luminance value, and generates the color difference image data. For this conversion, for example, the following expression used in the conversion from the RGB color system to the YCbCr color system can be used.

Y = 0.299 x R + 0.587 x G + 0.114 x B
Cb = 0.564 × (B−Y)
Cr = 0.713 × (R−Y)
Here, Y: luminance value, Cb: blue component value, Cr: red component value, R: R pixel value, G: G pixel value, B: B pixel value. As can be understood from this equation, the first color component value and the second color component value, for example, the blue component value: Cb and the red component value: Cr change according to the luminance (brightness) of the photographed image. It is.

  The threshold setting unit determines whether the target pixel (target region) is red or blue based on the luminance value (Y) included in the color difference image data generated by the color difference conversion unit. A specific color detection threshold is set as a determination condition. The blue component value and the red component value in the YCbCr color system are affected by the luminance value. The brighter the captured image IM, the larger the component value, and the darker the image, the smaller the component value. FIG. 6 shows how the blue component value: Cb and the red component value: Cr change when the color difference image including the target 2 is scanned in the horizontal direction. FIG. 6A is based on the captured image IM acquired in a bright environment, the luminance value: Y is relatively high, and the levels of the blue component value: Cb and the red component value: Cr are also relative. Is getting higher. FIG. 6B is based on the captured image IM acquired in a dark environment, and its luminance value: Y is relatively low, and its luminance value: Y is relatively low. . Therefore, in order to accurately detect the blue region or red region of the target 2 from the color difference image data, when the luminance value is high as shown in FIG. 6A, thresholds (THb, THr as detection conditions) ) Level should be relatively high. On the contrary, when the luminance value is low as shown in FIG. 6B, the threshold level as the detection condition may be relatively lowered.

  As one of the preferred methods for dynamically changing the threshold level according to the brightness (luminance level) of the image as described above, as schematically shown in FIG. The average value of the color component values to be determined: VA_avg, and the pixels whose color component values to be determined belong to the higher level among all the pixels in the captured image (for example, the color component value of 95% or more of the maximum color component value The pixel of the photographed image is a specific color region (here, from the average value VH_avg of the upper level corresponding to the color component corresponding red region of the color component value of the color component value) and the difference value ΔV_avg (= VH_avg−VA_avg). Then, a specific color threshold value TH (THb, THr) for determining that it belongs to either blue or red can be obtained by the following equation.

TH = VA_avg + K × ΔV_avg
Here, K is a constant value and is a value obtained experimentally in advance.

  The threshold value TH calculated in this way is derived by a function including the luminance component in the captured image as a parameter of Cb: blue component value or Cr: red component value, and therefore the luminance level in the captured image is high. And increases when the luminance level is low. In order to simplify the calculation processing for obtaining this threshold value, it is preferable to obtain a common threshold value by obtaining one of the blue threshold value: THb or the red threshold value: THr and using this as the other threshold value.

  Various methods other than those described above can be considered as a method of dynamically changing the threshold level according to the brightness (luminance level) of the photographed image. The upper concept can be explained as follows. it can. For example, the threshold value THb for determining the blue region is Y: statistical (average value, median value, etc.) numerical value: f (Y) and Cb: blue component value. It can be obtained by the following function: Fb with a parameter of statistical numerical value: f (Cb). The threshold value THr for determining the red area is Y: statistical value of luminance value of the captured image or luminance value of color difference information: f (Y) and Cb: statistical value of red component value: f (Cr ) With the following function: Fr.

THb = Fb (f (Y), f (Cb))
THr = Fr (f (Y), f (Cr))

  Here, the two threshold values of the threshold value for determining the blue region: THb and the threshold value for determining the red region: THr are obtained, but it is also possible to use one threshold value in combination.

  The color region determination unit sequentially scans the color difference image including the target 2 using the threshold value set dynamically by the threshold setting unit, that is, for each captured image, and determines a blue region and a red region. At this time, as a determination condition for determining the blue region, it is preferable to set the AND condition that the blue component value exceeds the threshold and the red component value falls below the threshold. Similarly, as a determination condition for determining the red region, an AND condition may be that the red component value exceeds the threshold and the blue component value falls below the threshold. That is, in the color difference signal, since the blue component value and the red component value are in a counter electrode relationship, more accurate blue region determination can be performed by using the difference. However, in the present invention, when determining the blue region and the red region, it is limited to the AND condition that one color component value exceeds the threshold value and the other color component value falls below the threshold value. I don't mean. As a determination condition for determining the blue region, only that the blue component value exceeds the threshold value may be employed, or as a determination condition for determining the red region, only that the red component value exceeds the threshold value is employed. You may do it. An important point in this method is to change the determination condition of the specific color region using luminance information.

  When the first boundary line V1 is detected in this way, the first narrowed area setting unit 13 then narrows down the capture area R1 based on the coordinate values on the captured image IM of the first boundary line V1. A narrowing area S1 narrower than the capture area R1 is set. This narrowing area S1 corresponds to the first narrowing area of the present invention. FIG. 8 is an explanatory diagram showing the principle of setting the narrowed-down area S1 from the capture area R1.

  As shown in FIG. 5, the first boundary line V1 is a line segment connecting the start point V1s and the end point V1e. Note that the names of the start point and end point are for convenience, and either may be the start point or end point. The calibration point Q of the first marker 2a should be included somewhere on the first boundary line V1. Since the first intersecting boundary line H1 intersects somewhere on the first boundary line V1, the narrowed-down area S1 is set within the range of the v coordinate between the start point V1s and the end point V1e in the first area R1. Further, the length of the line segment of the first intersecting boundary line H1 also falls within a certain number of pixels depending on the specification of the marker 2 and the simulation result. Therefore, a range obtained by adding the marginal pixel numbers M1 and M2 to the ranges of the predetermined pixel numbers A and B outward from the h coordinates of the start point V1s and the end point V1e, respectively, is set as the narrowed down region S1. The predetermined pixel numbers A and B may be the same pixel number, and the margin pixel numbers M1 and M2 may be the same pixel number.

  The first intersecting boundary line detection unit 14 detects the first intersecting boundary line H1 in the narrowed down area S1. Similar to the first boundary line detection unit 12, the first intersection boundary line detection unit 14 detects the first intersection boundary line H1 using a known edge detection algorithm and straight line recognition algorithm. In addition, since the marker 2 is a colored index, the above-described algorithm involving the color difference conversion process is also applied.

  Here, the first intersecting boundary line H1 is a straight line extending in the horizontal direction in FIG. Therefore, it is sufficient for the first intersection boundary detection unit 14 to detect an edge in the illustrated arrow direction and a 45-degree direction with respect to this arrow without trying to detect edges in all directions. As a result, the calculation time for detecting the first intersection boundary line H1 is greatly shortened. Further, since the narrowed area S1 is an area that is greatly narrowed down relative to the capture area R1, the number of pixels scanned using the edge detection algorithm is greatly reduced, and the calculation time is further shortened. Further, even if a straight line N similar to the first intersecting boundary line H1 exists in the capture region R1, it is not detected if it is outside the first narrowing region S1. Therefore, noise tolerance is also improved by narrowing the scanning area.

  The first calibration point detector 15 detects the intersection as the first calibration point Q1 based on the detected first boundary line V1 and the first intersection boundary line H1. FIG. 9 is an explanatory diagram showing the principle of detecting the calibration point Q. Symbols H1s and H1e indicate the start point and the end point of the detected first intersection boundary line H1, respectively. If the two straight lines V1 and H1 are detected, the first calibration point Q1 can be easily calculated by a simple geometric calculation.

  The second region setting unit 21 is a functional unit that sets a capture region R2 (second region) for detecting the calibration point Q of the second marker 2b based on the coordinate value of the first calibration point Q1. FIG. 10 is an explanatory diagram showing the principle of setting a capture area corresponding to another marker. Here, the maximum left-right width (h-direction width) is set with reference to the first calibration point Q1 detected earlier. Specifically, a range in which extra pixels M3 and M4 are added to the outside of the range sandwiched between the coordinates separated by the pixel number C and the coordinates separated by the pixel number D with respect to the h coordinate of the calibration point Q1. Is set as an area in the h direction. The pixel numbers C, D, M3, and M4 are determined according to the specifications of the marker 2 and the simulation results. The marginal number of pixels M3 and M4 may be the same number of pixels. In addition, in the vertical direction (v direction), a region is set in consideration of a case where it is shifted ± 3 degrees in the h direction. Regarding the v direction, the number of pixels is approximately the same as the number of pixels in the v direction of the first region R1. Since the second region R2 is set based on the detected first calibration point Q1, the region is more narrowed than the first region R1 particularly in the width in the h direction.

  The second boundary line detection unit 22 is a functional unit that detects the second boundary line V2 in the capture region R2. Similar to the first boundary line detection unit 12, the second boundary line detection unit 22 detects the second boundary line V2 by using a known edge detection algorithm, straight line recognition algorithm, and algorithm with the color difference conversion process described above. To do. As described above, since the second region R2 is set based on the detected first calibration point Q1, it is a narrower region narrowed down than the first region R1. Accordingly, the calculation time for detecting the second boundary line V2 is shortened compared to the calculation time for detecting the first boundary line V1.

  Next, the second narrowed area setting unit 23 narrows down the capture area R2 based on the coordinate value on the captured image IM of the second boundary line V2, and sets a narrowed area S2 that is narrower than the capture area R2. This narrowing area S2 corresponds to the second narrowing area of the present invention. FIG. 11 is an explanatory diagram showing the relationship between the capture region R2 and the narrowed-down region S2. Similar to the above-described narrowing down to the first narrowing area S1, the capture area R2 is narrowed down to the second narrowing area S2 as shown in FIG.

  The second intersection boundary line detection unit 24 detects the second intersection boundary line H2 in the narrowed-down area S2. Similar to the first intersection boundary detection unit 14, the second intersection boundary detection unit 24 uses a known edge detection algorithm, a straight line recognition algorithm, and an algorithm with the above-described color difference conversion processing to generate the second intersection boundary detection unit 24. Line H2 is detected. Since the narrowed area S2 is an area that is greatly narrowed with respect to the capture area R2, the number of pixels scanned using the edge detection algorithm is also greatly reduced, and the calculation time is shortened. Further, even if a straight line similar to the second intersecting boundary line H2 exists in the capture region R2, it is not detected if it is outside the second narrowing region S2. Therefore, noise tolerance is also improved by narrowing the scanning area.

  The second calibration point detector 25 detects the intersection as the second calibration point Q2 based on the detected second boundary line V2 and the first intersection boundary line H2. Similar to the first calibration point Q1, as shown in FIG. 11, if two straight lines are detected, the second calibration point Q2 can be easily calculated by simple geometric calculation.

  The calibration calculation unit 31 has coordinates in the world coordinate system (reference coordinate system) indicating the arrangement positions of the calibration points Q (Q1 and Q2) in space, and calibration points Q (Q1 and Q2) detected on the captured image IM. The camera 1 is calibrated by calculating the rotation amount of the camera 1 based on the coordinates in the camera coordinate system. For example, a rotation vector is obtained based on the obtained rotation amount, and the captured image IM is corrected using the rotation vector or its inverse vector. The corrected image is the same image as when the camera 1 is installed as designed. The corrected image is output via the image output unit 4 and displayed on, for example, a display of a navigation system mounted on the vehicle 100. The calculation method of the rotation amount and the rotation vector can be performed using a known method as described in JP 2008-131250 A and the like, and thus detailed description thereof is omitted here.

  The in-vehicle camera calibration device of the present invention is used in a vehicle production process or repair process. A vehicle produced through such a production process and a vehicle maintained through a repair process have, for example, a back camera that photographs the back of the vehicle, and displays the photographed image on a display such as a navigation system. It has a multimedia system with system functions. Here, the multimedia system is a system having multiple functions such as an audio visual function and a navigation function. This multimedia system can include, for example, a parking assistance system that performs image processing using a captured image of a back camera and executes parking assistance. In the back monitor system and the parking support system, it is very important that the in-vehicle camera is calibrated with high accuracy. And nowadays, when the number of vehicles equipped with in-vehicle cameras is increasing, it is very important to calibrate the in-vehicle cameras efficiently in order to increase the productivity of the vehicles. ADVANTAGE OF THE INVENTION According to this invention, it becomes possible to provide the calibration apparatus of the vehicle-mounted camera which can perform a detection and position calculation of a calibration point efficiently and correctly.

[Another embodiment 1]
In the above embodiment, an example in which the second region R2 is set based on the first calibration point Q1 has been described. However, even before the first calibration point Q1 is detected, the second region R2 can be set based on the first boundary line V1, and may be configured as such. In this case, the first boundary line V1 may be a boundary line in the direction intersecting the direction in which the two markers 2 are arranged, that is, the direction along the v direction, among the checkered pattern boundary lines of the marker 2. preferable. This is because the deflection width in the direction along the v direction becomes small, and the second region R2 is narrowed down satisfactorily. Hereinafter, description will be given with reference to FIG. 12 showing the principle of setting the second region R2 based on the first boundary line V1. Note that the second region R2 can also be set based on the boundary line in the direction along the direction in which the two markers 2 are arranged, that is, in the direction along the h direction. As described above, the start points V1s and H1s and the end points V1e and H1e are substantially specified for the boundary lines V1 and H1. Therefore, the second region R2 can be set based on the coordinate values of the start point and the end point even in the boundary line in the direction along the h direction.

  The second area setting unit 21 related to the second marker 2b has a maximum horizontal width (h-direction width) with reference to the first boundary line V1 of the first marker 2a, which is a direction orthogonal to the direction in which the two markers 2 are arranged. ) Is set. The maximum left-right width (h-direction width) is set with reference to the start point V1s and end point V1e of the first boundary line V1. Specifically, the number of extra pixels on the outside of the range sandwiched between the coordinates that are separated by the number of pixels C from the h coordinate of the start point V1s and the coordinates that are separated by the number of pixels D from the h coordinate of the start point V1e. A range including M3 and M4 is set as an area in the h direction. The pixel numbers C, D, M3, and M4 are determined according to the specifications of the marker 2 and the simulation results. The marginal number of pixels M3 and M4 may be the same number of pixels.

  Further, in the vertical direction (v direction), outside the range sandwiched between the coordinates E apart from the v coordinate of the start point V1s and the coordinates F away from the v coordinate of the start point V1e. A range including the extra pixel numbers M5 and M6 is set as the v-direction region. The number of pixels E, F, M5, and M6 is determined according to the specification of the marker 2 and the simulation result. The marginal number of pixels M5 and M6 may be the same number of pixels.

[Another embodiment 2]
In the above-described embodiment, the first narrowed area setting unit 13 and the second narrowed area setting unit 23 are provided, and after further narrowing down the capture area (the first area R1 and the second area R2), the first intersection The example which detected the boundary line H1 and the 2nd crossing boundary line H2 was shown. However, it goes without saying that the first intersection boundary line H1 and the second intersection boundary line H2 may be detected without further narrowing down the capture regions R1 and R2.

  As described above, according to the present invention, calibration point detection and position calculation can be performed efficiently and accurately from captured image data obtained by photographing a calibration index composed of a combination pattern of a plurality of colors under the influence of ambient light. It is possible to provide an in-vehicle camera calibration device capable of performing

Explanatory drawing which shows an example of arrangement | positioning relationship between the marker for calibration and a vehicle Explanatory drawing showing an example of a marker for calibration The block diagram which shows typically the structure of the calibration apparatus of the vehicle-mounted camera of this invention Explanatory diagram showing the capture area setting principle Explanatory drawing which shows the principle which detects a 1st boundary line within a capture area | region Explanatory drawing which shows the mode of a fluctuation | variation of blue component value: Cb and red component value: Cr when the color difference image containing a marker is scanned to the horizontal direction. The graph which shows the investigation result with respect to the picked-up image in YCbCr color system Explanatory drawing showing the principle of setting the narrowed down area from the capture area Explanatory diagram showing the principle of detecting calibration points Explanatory drawing showing the principle of setting the capture area corresponding to another marker It is explanatory drawing which shows the relationship between a capture area | region and a narrowing-down area | region. Explanatory drawing which shows the principle which sets a 2nd area | region based on a 1st boundary line

Explanation of symbols

1: Camera (on-vehicle camera)
2: Marker (calibration index)
3: Image receiving unit 11: First region setting unit 12: First boundary line detection unit 13: First narrowing region setting unit 14: First intersection boundary line detection unit 15: First calibration point detection unit 21: Second region Setting unit 22: second boundary line detection unit 23: second narrowing area setting unit 24: second intersection boundary line detection unit 25: second calibration point detection unit 31: calibration calculation unit H1: first intersection boundary line H2: first Two intersecting boundary lines IM: Captured image Q, Q1, Q2: Calibration point Q1: First calibration point Q2: Second calibration point R1: First region R2: Second region S1: First narrowing region S2: Second narrowing region V1: First boundary line V2: Second boundary line

Claims (3)

An image receiving unit that receives captured images taken by a vehicle-mounted camera that is arranged in at least two different locations and includes a calibration index within the imaging range with the intersection of the boundary lines of the grid pattern of the first color and the second color as calibration points When,
A first region setting unit that sets a first region including a first calibration point that is one of the calibration points on the captured image;
A first boundary line detection unit that detects a first boundary line that is one of the boundary lines of the lattice pattern of the calibration index in the first region;
A first intersection boundary detection unit that detects a first intersection boundary that intersects the first boundary in the first region;
A first calibration point detector that detects the coordinates of the intersection as the first calibration point based on the coordinates of the detected first boundary line and the first intersection boundary line on the captured image;
Based on the detected coordinate value of the first calibration point on the photographed image, a second region including a second calibration point different from the first calibration point is represented on the photographed image by the first region. A second region setting unit that sets the narrow range,
A second boundary line detection unit that detects a second boundary line that is one of the boundary lines of the lattice pattern of the calibration index in the second region;
A second intersection boundary detection unit for detecting a second intersection boundary intersecting the second boundary in the second region;
A second calibration point detector that detects the coordinates of the intersection as the second calibration point based on the coordinates of the detected second boundary line and the second intersection boundary line on the captured image;
The coordinates in the reference coordinate system indicating the arrangement positions of the first calibration point and the second calibration point in space, the first calibration point detected on the captured image, and the second calibration point in the camera coordinate system An in-vehicle camera calibration apparatus comprising: a calibration calculation unit that calibrates the in-vehicle camera based on coordinates. An image receiving unit that receives captured images taken by a vehicle-mounted camera that is arranged in at least two different locations and includes a calibration index within the imaging range with the intersection of the boundary lines of the grid pattern of the first color and the second color as calibration points When,
A first region setting unit that sets a first region including a first calibration point that is one of the calibration points on the captured image;
A first boundary line detection unit that detects a first boundary line that is one of the grid pattern boundary lines of the calibration index in the first region;
Based on the detected coordinates of the first boundary line on the photographed image, a second area including a second calibration point different from the first calibration point on the photographed image is more than the first area. A second region setting unit for setting as a narrow range;
A first intersection boundary detection unit that detects a first intersection boundary that intersects the first boundary in the first region;
A second boundary line detection unit for detecting a second boundary line that is one of the grid pattern boundary lines of the calibration index in the second region;
A second intersection boundary detection unit for detecting a second intersection boundary intersecting the second boundary in the second region;
A first calibration point detector that detects the coordinates of the intersection as the first calibration point based on the coordinates of the detected first boundary line and the first intersection boundary line on the captured image;
A second calibration point detector that detects the coordinates of the intersection as the second calibration point based on the coordinates of the detected second boundary line and the second intersection boundary line on the captured image;
The coordinates in the reference coordinate system indicating the arrangement positions of the first calibration point and the second calibration point in space, the first calibration point detected on the captured image, and the second calibration point in the camera coordinate system An in-vehicle camera calibration apparatus comprising: a calibration calculation unit that calibrates the in-vehicle camera based on coordinates. A first narrowed area setting unit configured to set a first narrowed area narrower than the first area in the first area based on the detected coordinate value of the first boundary line on the captured image;
A second narrowed area setting unit configured to set a second narrowed area narrower than the second area in the second area based on the detected coordinate value of the second boundary line on the captured image; Prepared,
The first intersecting boundary line detection unit detects the first intersecting boundary line in the first narrowing region,
The second cross-boundary detection unit detects the second intersection boundary line in the second narrowing region, the in-vehicle camera calibration apparatus according to claim 1 or 2.

Images