JP2023018438A - In-vehicle camera and calibration method for in-vehicle camera - Google Patents

Abstract

To provide a technology capable of highly accurately calibrating a wide-angle in-vehicle camera using a small chart.SOLUTION: An in-vehicle camera includes an imaging unit that images an object, an estimation unit that calculates the displacement of the pixels of the entire surface on the basis of the image captured by the imaging unit, an area selection unit that selects a plurality of predetermined areas determined in advance according to the shape and inclination of the windshield in the image captured by the imaging unit, and a calibration unit that corrects pixel misalignment on the entire surface on the basis of information of the estimation unit. The estimation unit estimates the pixel displacement of the entire image using images detected from the plurality of predetermined areas in the image captured by the imaging unit.SELECTED DRAWING: Figure 1

Description

The present invention relates to a vehicle-mounted camera and a vehicle-mounted camera calibration method.

Japanese Patent Laying-Open No. 2011-49733 (Patent Document 1) and Japanese Patent Laying-Open No. 2013-109416 (Patent Document 2) describe calibration techniques for a camera system such as an in-vehicle camera.

In Patent Document 1, "In the present invention, the distortion of an image captured by a camera is approximated by a function model that differs for each region. By approximating the lens distortion to a function model, correction can be performed with high accuracy." It is possible to improve the visibility and improve the accuracy of image recognition.”

In Patent Document 2, "In this embodiment, the distortion coefficient is calculated after weighting the field angle region. FIG. It is an example of a diagram for explanation.By making the weighting of the narrow angle of view larger than that of the wide angle of view, the distortion coefficient that can accurately correct the distortion of the image of the narrow angle of view can be obtained.For this reason, in the case of the stereo camera, High-precision distance measurement is possible for a subject in a narrow angle of view.Fig. 1(c) is an example of a diagram illustrating a distortion coefficient when weighting a WIDE region (wide angle of view) is increased. By weighting the wide angle of view more than the narrow angle of view, it is possible to obtain a distortion coefficient that can accurately correct the distortion of the wide angle of view image. Highly accurate distance measurement is possible."

JP 2011-49733 A JP 2013-109416 A

Distance measurement with conventional in-vehicle cameras is mainly a function that supports collision avoidance against obstacles in front of the vehicle. On the other hand, there are still many accidents at intersections, and in order to further reduce accidents, future in-vehicle cameras will need a wider recognition angle of view. However, in order to widen the recognition angle of view, a wide-angle lens is used, and the amount of distortion (pixel shift) also increases, so the demand for accuracy in calibration increases.

On the other hand, as the recognition angle of view widens, the range to be calibrated also increases, and the size of a calibration image (hereinafter referred to as a chart) for measuring pixel deviation also increases. In particular, in the case of on-board cameras, the pixel deviation becomes even greater due to the refraction of the windshield, and furthermore, variations in the installation of the on-board camera or variations in the shape of the windshield cannot be ignored, so calibration must be performed for each vehicle. the need increases. In this case, if the size of the chart is too large, it will be difficult to install the chart in the factory or dealer, making calibration difficult. In an on-vehicle camera, for example, when the left and right (horizontal direction) angle of view (also referred to as the viewing angle FOV) is 40 degrees, a chart of 3 m (horizontal width)×3 m (vertical width) may be used. Also, when the left and right angle of view is 120 degrees, a chart of 14 m (horizontal width)×3 m (vertical width) may be required.

SUMMARY OF THE INVENTION An object of the present invention is to provide a technique capable of highly accurately calibrating a vehicle-mounted camera with a wide angle using a small chart.

The above and other objects and novel features of the present invention will become apparent from the description of the specification and the accompanying drawings.

A brief description of representative inventions among the inventions disclosed in the present application is as follows.

An in-vehicle camera according to an embodiment includes an imaging unit that captures an image of an object, an estimating unit that calculates the pixel deviation of the entire surface based on the image captured by the imaging unit, and a front camera in the image captured by the imaging unit. An area selection unit that selects a plurality of predetermined areas determined in advance according to the shape and inclination of the glass, and a calibration unit that corrects pixel deviations on the entire surface based on information from the estimation unit. The estimating unit estimates the pixel displacement of the entire image using images detected from a plurality of predetermined regions in the image captured by the imaging unit.

According to the technique of the present invention, it is possible to calibrate an in-vehicle camera with high accuracy even with a small chart.

Problems, configurations, and effects other than those described above will be clarified by the following description of the mode for carrying out the invention.

FIG. 1 is a block diagram showing a configuration example of an in-vehicle camera in an embodiment; FIG. 2 is a flowchart showing a procedure of calibrating the vehicle-mounted camera in the embodiment. FIG. 3 is an explanatory diagram showing an example of how an in-vehicle camera captures a subject; FIG. 4 is an explanatory diagram showing an example of an image of an object captured by an in-vehicle camera; FIG. 5 is an explanatory diagram showing an example of measuring the pixel deviation of the entire screen using a large chart in the comparative example. FIG. 6 is an explanatory diagram showing an example of measuring only a specific region using a small chart; 7A is a diagram showing Formula 1. FIG. 7B is a diagram showing Equations 2, 3, and 4; FIG. FIG. 8 is a diagram showing an example of variations in mounting of an in-vehicle camera and an example of variations in the shape of a windshield; FIG. 9 is a diagram showing an example in which relevant variability parameters are indistinguishable. FIG. 10 is a diagram for explaining the sensitivity of the variation parameter for the inclination angle of the windshield (30 degrees to 35 degrees). FIG. 11 is a diagram for explaining the sensitivity of the variation parameter when the tilt angle of the windshield is 35 degrees or more: 40 degrees to 70 degrees.

BEST MODE FOR CARRYING OUT THE INVENTION Hereinafter, embodiments of the present invention will be described with reference to the drawings. The embodiments and examples are examples for explaining the present invention, and are appropriately omitted and simplified for clarity of explanation. The present invention can also be implemented in various other forms. Unless otherwise specified, each component may be singular or plural.

The position, size, shape, range, etc. of each component shown in the drawings may not represent the actual position, size, shape, range, etc., in order to facilitate understanding of the invention. As such, the present invention is not necessarily limited to the locations, sizes, shapes, extents, etc., disclosed in the drawings.

When there are a plurality of components having the same or similar functions, they may be described with the same reference numerals and different suffixes. Further, when there is no need to distinguish between these constituent elements, the subscripts may be omitted in the description.

The contents of this embodiment will be described with reference to FIGS. 1 to 6. FIG.

FIG. 3 is an explanatory diagram showing an example of how an in-vehicle camera captures a subject. 31 is a vehicle having an in-vehicle camera 30 . 32, 33 and 34 are subjects (there are three subjects in this example). Detection images DPA and DPB showing these three subjects 32-34 are shown in FIG. In the detection images DPA and DPB, the horizontal direction is H, and the vertical direction is V. As shown in FIG. The detection image DPA in FIG. 4 shows the positions of the images of the subjects 32A, 33A, and 34A without the windshield and the positions of the images of the subjects 32B, 33B, and 34B with the windshield. The positions of the images (32A, 33A, 34A) without the windshield are ideal positions where there is no refraction of the windshield, that is, there is no pixel shift. On the other hand, the positions of the images (32B, 33B, and 34B) with the windshield are assumed to be the positions of the images projected without variations in the mounting of the vehicle-mounted camera and variations in the shape and thickness of the windshield. The difference between the positions of the images (32A, 33A, 34A) and the positions of the images (32A, 33A, 34A) is called pixel shift (PS1). If all cars of the same model are equipped with on-board cameras, and if there are no variations in the shape and thickness of the windshield, all cars of the same model can be calibrated by measuring only once for one model.

However, as shown in the detection image DPB in FIG. 4, in actual cases, there are variations in the mounting of the vehicle-mounted camera and variations in the shape and thickness of the windshield, so the position of the image refracted by the windshield differs from the design value. . The detected image DPB includes the positions of the images of the subjects 32A, 33A, and 34A when there is no windshield, and the images (32C, 33C , 34C) are shown. The pixel shift, which is the difference between the positions of the images (32A, 33A, 34A) and the positions of the images (32C, 33C, 34C), is pixel shift PS2. The amount of pixel shift PS2 (length of arrow) is greater than the amount of pixel shift PS1 (length of arrow).

A pixel shift (PS2-PS1) occurs between the positions of the images (32B, 33B, 34B) of the detection image DPA and the positions of the images (32C, 33C, 34C) of the detection image DPB. For example, comparing the horizontal pixel shift component of the pixel shift PS1 of the image 32B with respect to the image 32A and the horizontal pixel shift component of the pixel shift PS2 of the image 32C from the image 32A, the horizontal pixel shift PSH (= PS2 (horizontal) - PS1 (horizontal)). In the case of a narrow-angle camera, the effect of such pixel deviation is small, so it can be ignored in some cases. The resulting pixel deviation cannot be ignored. Therefore, calibration is required for each vehicle.

FIG. 5 is an explanatory diagram showing an example of measuring the pixel displacement of the entire screen using a large chart in the comparative example. In the calibration method of the comparative example shown in FIG. 5, a large chart (calibration image) 51 that can cover the entire wide-angle region, which is the widened viewing angle of the vehicle-mounted camera 52, is prepared. For example, when the left and right angles of view of the in-vehicle camera 52 are 120 degrees, the chart 51 of 14 m (horizontal width)×3 m (vertical width) is used.

A large chart 51 is photographed through a windshield 53 by a wide-angle camera of an in-vehicle camera 52.例文帳に追加As shown in FIG. 5, by measuring with a large chart 51, the pixel shift (δε) of the entire screen of the on-vehicle camera 52 can be obtained. In the pixel deviation (δε) of the entire screen, the horizontal axis indicates the horizontal angle of view (FOVH), and the vertical axis indicates the vertical angle of view (FOVV). The direction of the arrow indicates the direction of pixel displacement, and the length of the arrow indicates the amount of pixel displacement. By canceling the pixel displacement (Δε) of the entire screen, image distortion such as the pixel displacement (Δε) of the entire screen can be eliminated, and all pixels in the entire screen can be calibrated. Pixel displacement (δε) in the entire screen is caused by distortion due to variations in mounting of the in-vehicle camera, distortion due to variations in windshield shape, inclination, and plate thickness, and distortion due to variations in the shape of the wide-angle lens of the in-vehicle camera. do.

In this case, if the size of the large chart 51 is too large, it will be difficult to install the large chart 51 in the factory or dealer. Therefore, there is a concern that full screen calibration for each vehicle will not be possible.

A calibration method according to the embodiment will be described with reference to FIG. FIG. 6 is an explanatory diagram showing an example of a calibration method for measuring only a specific area using a small chart.

Instead of using the large chart 51 described in FIG. 5 to measure the pixel deviation (δε) of the entire image screen, in FIG. is measured. In this example, three small charts 61 are used to measure pixel shifts (δεp1, δεp2, δεp3) in three specific regions (64, 65, 66). The size of each of the three specified areas (64, 65, 66) is less than 1/2 of the full screen image. When the plurality of predetermined areas is n areas, at least one area of each predetermined area is 1/n or less of the entire screen image. The size of the small chart 61 can be, for example, 1 m to 2 m or less×1 m to 2 m or less. The in-vehicle camera 62 is a wide-angle camera of the in-vehicle camera 62 through the windshield 63, and by photographing a plurality of small charts 61, pixel shifts ( δεp1, δεp2, δεp3) are measured.

The in-vehicle camera 62 uses the pixel shifts (δεp1, δεp2, δεp3) of the specific regions 64, 65, 66 to calculate the variation parameter δφ (eg, δφ1, δφ2, δφ3, . to calculate Then, the vehicle-mounted camera 62 finally calculates an estimated pixel deviation (δε′) for estimating the pixel deviation (δε) of the entire screen from the variation parameter (δφ). As a result, even with a plurality of small charts 61, it is possible to calibrate the pixel deviation (δε) of the entire screen of the wide-angle vehicle-mounted camera 62 . Variation parameters δφ (eg, δφ1, δφ2, δφ3, . can be done.

Details of an in-vehicle camera that can implement the calibration method of FIG. 6 and the calibration method will be described with reference to FIGS. 1 and 2. FIG. FIG. 1 shows a block diagram showing a configuration example of an in-vehicle camera in an embodiment. FIG. 2 is a flow chart showing a procedure of calibrating the vehicle-mounted camera in the embodiment.

As shown in FIG. 1, the vehicle-mounted camera 62 includes an imaging unit 10, an area selection unit 11, an external input unit 12, a parameter calculation unit 13, and a full-screen pixel shift estimation unit 14 for estimating pixel shifts on the entire screen. , and a calibration unit 15 . Here, the full screen means the entire horizontal angle of view and vertical angle of view that can be acquired by the camera built into the imaging unit 10 .

The imaging unit 10 is an imaging circuit that images an object, and is connected to the area selection unit 11 . The imaging unit 10 is, for example, an imaging device capable of acquiring images, such as a visible camera or an infrared camera having an optical lens and an imaging device, and is composed of a single camera (monocular camera) or multiple cameras (stereo camera). The imaging unit 10 is not limited to these cameras, and various imaging circuits may be applied. The imaging unit 10 acquires an image of one or a plurality of small charts 61 and outputs data of the acquired images to the area selection unit 11, as described with reference to FIG. In this image, each pixel indicates a luminance value.

The region selection section 11 is connected to the imaging section 10 , the external input section 12 and the parameter calculation section 13 . The region selection unit 11 selects a plurality of predetermined regions in the image captured by the imaging unit 10, which are predetermined according to the shape and inclination of the windshield. The region selection unit 11 receives from the external input unit 12 design data such as design values of the windshield 63 and region selection data for selecting a predetermined region such as one or a plurality of specific regions, and selects one or a plurality of corresponding regions. (or a predetermined (desired) specific area, predetermined area) is selected. The small chart 61 illustrated in FIG. 6 is arranged so as to correspond to each of the selected one or more specific regions. The area selection unit 11 calculates the pixel deviations (δεp1, δεp2, δεp3) of each specific region from the image of the small chart 61 of one or more selected specific regions. The region selection unit 11 outputs the calculated pixel deviations (δεp1, δεp2, δεp3) of the specific regions to the parameter calculation unit 13 .

The external input unit 12 is connected to the area selection unit 11, the parameter calculation unit 13, and the full pixel deviation estimation unit 14. FIG. The external input unit 12 transmits the region selection data to the region selection unit 11 and passes the US matrix to the parameter calculation unit 13 and the full-surface pixel deviation estimation unit 14 . Here, the US matrix is a matrix representing a mathematical relationship (mathematical model) between the orthogonalized variation parameters δφ (δφ1, δφ2, δφ3, . . . ) and pixel shifts (δεp1, δεp2, δεp3).

The parameter calculation unit 13 is connected to the region selection unit 11 and the full pixel deviation estimation unit 14 . The parameter calculator 13 receives the pixel shifts (δεp1, δεp2, δεp3) of the specific region from the region selector 11, and calculates variation parameters δφ (δφ1, δφ2, δφ3, . . . ). The parameter calculator 13 outputs the calculated variation parameters δφ (δφ1, δφ2, δφ3, .

The full-surface pixel deviation estimation unit 14 is connected to the parameter calculation unit 13 and the calibration unit 15 . The overall pixel shift estimator 14 estimates the pixel shift of the entire image using images detected from a plurality of predetermined regions in the image captured by the imaging unit 10 . The plurality of predetermined areas can include three or more predetermined areas. The full-screen pixel displacement estimating unit 14 receives the values of the variation parameters δφ (δφ1, δφ2, δφ3, . Calculate The full-screen pixel deviation estimation unit 14 outputs the estimated pixel deviation (δε′) of the entire screen to the calibration unit 15 .

The calibration unit 15 is connected to the full pixel deviation estimation unit 14 . The calibration unit 15 corrects the displacement of the pixels on the entire surface based on the information from the overall pixel displacement estimation unit 14 . The calibration unit 15 receives the pixel displacement (δε′) of the entire screen estimated from the full-screen pixel displacement estimating unit 14, cancels the pixel displacement (δε) of the entire screen, and performs calibration of the pixel positions of the entire screen. conduct.

After the calibration is completed, the full-screen image acquired by the imaging unit 10 is calibrated by the calibration unit 15 and then used as image data for distance measurement. As a result, it is possible to provide an in-vehicle camera having a wide recognition angle of view for the purpose of reducing accidents at intersections.

The in-vehicle camera in FIG. 1 can be summarized as follows.

That is, the in-vehicle camera 62 includes an imaging unit 10 that captures an image of an object, an entire pixel deviation estimating unit 14 that calculates deviations of pixels on the entire surface based on the image captured by the imaging unit 10, and an image captured by the imaging unit 10. Based on the information of the area selection unit 11 that selects a plurality of predetermined areas (specific areas 64, 65, 66) determined in advance according to the shape and inclination of the windshield 63 in the image, and the entire surface pixel shift estimation unit 14 and a calibration unit 14 that corrects the deviation of the pixels. The overall pixel displacement estimator 14 estimates the overall pixel displacement using images detected from a plurality of predetermined regions (specific regions 64 , 65 , 66 ) in the image captured by the imaging unit 10 .

Further, the calibration method for the vehicle-mounted camera 62 described with reference to FIG. 1 can be summarized as follows. In other words, the calibration method for the in-vehicle camera 62 includes an image capturing process of capturing an image of an object by the image capturing unit 10 and an estimation of calculating the pixel deviation of the entire surface based on the image captured in the image capturing process by the pixel deviation estimating unit 14 of the entire surface. a step of selecting a plurality of predetermined regions (specific regions 64, 65, 66) predetermined according to the shape and inclination of the windshield in the image captured in the photographing step by the region selection unit 11; and a calibration step of correcting the displacement of the pixels of the entire surface by the calibration unit 15 based on the information of the estimation step. In the estimating step, pixel deviations of the entire surface are estimated using images detected from a plurality of predetermined regions in the image photographed in the photographing step.

In the configuration example of the vehicle-mounted camera 62 shown in FIG. 1, each section (10 to 15) can be configured by a hardware circuit. In this case, the vehicle-mounted camera 62 can be rephrased as, for example, the image pickup circuit 10, the area selection circuit 11, the external input circuit 12, the parameter calculation circuit 13, the overall pixel deviation estimation circuit 14, and the calibration circuit 15.

In addition, in the configuration example of the vehicle-mounted camera 62 shown in FIG. 1, each section (11 to 15) other than the photographing section 10 can be configured by a software program. In this case, the vehicle-mounted camera 62 is provided with a data processing device that executes the operations of the respective units (11 to 15). The data processing device includes, for example, a program memory device storing a software program describing the operation of each unit (11 to 15), and a central processing unit (CPU) for executing the software program stored in this program memory device. and a temporary storage memory device for temporarily holding data in the middle of computation by the central processing unit (CPU). The program memory device can be composed of a rewritable non-volatile storage device such as a flash memory or a non-volatile storage device such as a read only memory (ROM). The temporary storage memory device may comprise a volatile memory device such as static random access memory (SRAM). The central processing unit (CPU) may be multiple central processing units (CPUs).

Next, with reference to FIG. 2, a flowchart showing a procedure for calibrating the vehicle-mounted camera will be described. 7A is a diagram showing Equation 1. FIG. FIG. 7B is a diagram showing Equations 2, 3 and 4;

Prior to all operations, it is necessary to construct a mathematical model showing the mathematical relationship between the full-screen pixel displacement (δε) and the variation parameters δφ (δφ1, δφ2, δφ3, . . . ). The mathematical model is represented by (equation 1) in FIG. 7A. δε is the pixel shift. The Q matrix is a matrix showing the mathematical relationship between the pixel deviation (δε) and the variation parameter (δφ). δφ is a variation parameter. m is the number of variations. For example, the variations shown in FIG. 8 indicate X shift, Y shift, and Z shift as variations in the mounting position of the vehicle-mounted camera 62, and pitch rotation, yaw rotation, and roll rotation as variations in the camera posture of the vehicle-mounted camera 62. , horizontal curvature, vertical curvature, and thickness as variations in the windshield 63 . That is, m=9 if the nine types shown in FIG. 8 are assumed as variations. n is the number of pixels in the image. For example, if there are 100 pixels in the horizontal direction and 50 pixels in the vertical direction, n=100*50=5000 pixels. Therefore, the Q matrix is a matrix showing the mathematical relationship between the pixel deviation of each pixel point and each variation parameter.

The calculation of the Q matrix is based on inputting each variation into an optical simulation such as ray tracing and calculating the pixel shift. Since all practical variations are small, the Q matrix can be calculated approximately by dividing each pixel by the variations. A mathematical relational Q-matrix is obtained by trying different variations of the variability.

If the variability is independent of each other, the Q matrix can be used directly to calculate the correct variability parameter from the pixel shift. However, the actual variability parameters are related to each other and not completely independent. Therefore, the parameters may not be distinguished, or different parameters may be calculated. For example, as shown in FIG. 9, the variation in X shift (horizontal shift in the X direction) has an X direction component similar to the variation in yaw rotation. have a similar region of similarity SMR. If the variation parameter (δφ) is calculated from the partial pixel shift (δεp) of this similarity region SMR, the X shift and yaw rotation cannot be distinguished, so the estimation of the pixel shift (δε′) of the entire screen will be erroneous. Become. Therefore, it is better to make the parameters independent or orthogonal. Here, let the orthogonalized variation parameters be δψ(δψ1, δψ2, δψ3, . . . ). If the variation parameter (.delta..phi.) is the actual variation parameter, the orthogonalized variation parameter (.delta..psi.) can be said to be the orthogonal variation parameter. The orthogonalized variation parameters δψ (δψ1, δψ2, δψ3, . . . ) are mathematically orthogonal to each other.

For example, the orthogonalization of mathematical schemes is better using US matrices that can transform parameters independently. Principal component analysis may be performed at the same time as orthogonalization, leaving only parameters with large changes and omitting parameters with small changes. Here, a method of orthogonalization and principal component analysis called Singular Value Decomposition (SVD) can be taken as an example, by which a US matrix can be generated. (Equation 2) in FIG. 7B shows an equation using the US matrix and the orthogonalized variation parameter (δψ). Note that the calculation method is not limited to singular value decomposition, and various calculation methods may be employed. (Equation 3) in FIG. 7B shows a relational expression between the orthogonal variation parameter (δψ) and the actual variation parameter (δφ). Here, V is a transformation matrix that transforms the real variation parameter (δφ) into the orthogonal variation parameter (δψ). (Formula 4) in FIG. 7B shows a formula obtained by substituting (Formula 3) into (Formula 2).

Next, with reference to FIG. 2, a flowchart of detailed calibration of the vehicle-mounted camera will be described.

1) Step 201:
Power is supplied to the in-vehicle camera 62 to start calibration.

2) Step 202:
Step 202 is a chart screen acquisition step. As shown in FIG. 6, an in-vehicle camera 62 is placed behind a windshield 63, and images of a plurality of small charts 61 are captured by the imaging unit 10 through the windshield 63 from the inside of the vehicle. A photographed image of a small chart 61 is acquired. The chart 61 has patterns such as checkers and circles. The imaging unit 10 outputs the acquired captured image to the area selection unit 11 .

3) Step 203:
Step 203 is a step of inputting design values of the windshield 63 . The external input unit 12 transmits the design values of the windshield 63 input from the external input unit 12 to the region selection unit 11 .

4) Step 204:
Step 204 is a step of calculating the pixel displacement of each specific area. Based on the design values of the windshield 63, a pre-calculated mathematical model can be used to determine where each variation is sensitive on the image. A location with good sensitivity is a location where pixel deviation is large when each variation changes. Select an image region that is sensitive to each variation. The chart 61 is arranged (moved and left) in the selected image area (for example, corresponding to the specific areas 64, 65, and 66 in FIG. 6). Also, the chart 61 of each selected image area is photographed and the pixel shift is calculated from the image. The region selection unit 11 calculates the pixel deviation δε _P (δεp1, δεp2, δεp3) of each specified region from the image of the chart 61 of the plurality of selected specified regions. The region selection unit 11 outputs the calculated pixel shifts δε _P (δεp1, δεp2, δεp3) of the specific regions to the parameter calculation unit 13 .

When three image areas (specific areas 64, 65, and 66) are selected and three charts 61 are used, the three charts 61 are arranged so as to correspond to the selected image areas, and the imaging unit 10 It is preferable to photograph the entire screen and calculate the pixel deviation of each specific area by measuring the image data of the entire screen once. This has the effect of being able to calculate pixel deviations in a plurality of specific regions in a short time.

When three image areas (specific areas 64, 65, 66) are selected and one chart 61 is used, the chart 61 is arranged so as to correspond to one image area (specific area 64), The entire screen is photographed by the imaging unit 10, then the chart 61 is arranged so as to correspond to the next image area (specific area 65), the entire screen is photographed by the imaging unit 10, and the chart 61 is further arranged as follows. The images are arranged so as to correspond to the image area (specific area 66), and the image pickup unit 10 photographs the entire screen. It is preferable to calculate the pixel deviation of each specific area by measuring the image data of the entire screen three times. Since only one chart is used, it is possible to calculate pixel shifts in a plurality of specific regions at low cost and in a small area.

5) Step 205:
Step 205 is a step of calculating the variation parameter δφ. The parameter calculation unit 13 calculates the values of each variation parameter _δφ (δφ1, δφ2, δφ3, . calculate. A least squares method or the like can be used as a calculation method. Note that the calculation method is not limited to the least squares method, and various calculation methods can be applied. The parameter calculator 13 receives the pixel shift δε _P (δεp1, δεp2, δεp3) of the specific region from the region selector 11, and calculates variation parameters δφ (δφ1, δφ2, δφ3, . . . ). The parameter calculator 13 outputs the calculated variation parameters δφ (δφ1, δφ2, δφ3, .

At this time, instead of the variation parameters δφ (δφ1, δφ2, δφ3, . Matrix and V-matrix calculations can also be performed (see FIG. 7B).

6) Step 206:
Step 206 is a step of estimating the pixel deviation of the entire screen. Using a pre-stored Q-matrix or an optical simulation such as ray tracing, the pixel deviation (δε) of the entire screen is estimated from the calculated variation parameter δφ (δφ1, δφ2, δφ3, . . . ). The full-screen pixel displacement estimating unit 14 receives the values of the variation parameters δφ (δφ1, δφ2, δφ3, . Calculate The full-screen pixel deviation estimation unit 14 outputs the estimated pixel deviation (δε′) of the entire screen to the calibration unit 15 .

7) Step 207:
Step 207 is a calibration step. Based on the estimated pixel deviation (δε′) of the entire screen, the pixel deviation of the entire screen in the captured image is canceled (or corrected or corrected) at each pixel point, and calibration is performed. The calibration unit 15 receives the pixel displacement (δε′) of the entire screen estimated from the full-screen pixel displacement estimating unit 14, cancels the pixel displacement (δε) of the entire screen, and performs calibration of the pixel positions of the entire screen. conduct.

8) Step 208:
At step 208, the calibration flow of the vehicle-mounted camera 62 ends.

After the above steps 201-208 are performed, the image captured by the vehicle-mounted camera 62 has been calibrated for the positions of the pixels of the full screen, and there is no pixel shift (or the pixel shift has been corrected or corrected). The image will be used for distance measurement (also called ranging) at intersections. As a result, the in-vehicle camera 62 can accurately measure the distance to an obstacle at an intersection, and can reliably provide a function of supporting collision avoidance at the intersection.

Next, with reference to FIGS. 10 and 11, the tilt angle of the windshield as the design value of the windshield and the sensitivity of the variation parameter will be described. FIG. 10 is a diagram for explaining the sensitivity of the variation parameter for the inclination angle of the windshield (30 degrees to 35 degrees). FIG. 11 is a diagram for explaining the sensitivity of the variation parameter when the tilt angle of the windshield is 35° or more, eg, 40° to 70°.

FIG. 10 shows, when the design value of the tilt angle of the windshield 63 is 30 degrees to 35 degrees, for example, three specified points considering the sensitive portions of the three variation parameters (δφ1, δφ2, δφ3). The position (camera angle of view) of the area (64, 65, 66) is shown. FIG. 11 shows, when the design value of the tilt angle of the windshield 63 is 40 degrees to 70 degrees, for example, three specific points considering the sensitive portions of the three variation parameters (δφ1, δφ2, δφ3). The positions (angles of view of the camera) of areas (64, 65, 66) are shown. Here, in FIGS. 10 and 11, the variation parameters (actual variation parameters: δφ1, δφ2, δφ3) may be orthogonal variation parameters (δφ1, δφ2, δφ3). In FIGS. 10 and 11, legends (0-300) for the sensitivities of the variation parameters (δφ1, δφ2, δφ3) are shown on the right side of each diagram of the sensitivities of the variation parameters (δφ1, δφ2, δφ3).

As shown in FIG. 10, the variation parameter .delta..phi.1 has high sensitivity in the first upper region of the angle of view of the in-vehicle camera 62, which is 10 degrees to 20 degrees vertically and -15 degrees to -5 degrees horizontally. Therefore, the first specific region 64 is placed in the first top region in the image. That is, the first specific region 64 is set to a region of the angle of view of the image of the in-vehicle camera 62 (length (vertical direction): 10 degrees to 20 degrees, horizontal (horizontal direction): -15 degrees to -5 degrees). . Variation parameter δφ2 has high sensitivity in the second upper region of the angle of view of vehicle-mounted camera 62, which is 10 to 20 degrees vertically and 30 to 40 degrees horizontally. Therefore, the second specific area 65 is arranged in a second upper area different from the first upper area in the image. In other words, the second specific area 65 is set to the area of the angle of view of the image of the vehicle-mounted camera 62 (vertical: 10 to 20 degrees, horizontal: 30 to 40 degrees). The variation parameter δφ3 has high sensitivity in the range of −20° to −10° vertically and −35° to −25° horizontally in the angle of view of the vehicle-mounted camera 62 . Therefore, the third specific region 66 is placed in the first bottom region in the image. That is, the third specific area 66 is set to the area of the angle of view of the image of the in-vehicle camera 62 (vertical: -20 degrees to -10 degrees, horizontal: -35 degrees to -25 degrees).

Three charts 61 are arranged so as to correspond to the three specific areas (64, 65, 66) thus determined (see FIG. 6). The pixel displacements in these specific regions (64, 65, 66) can be measured with a chart 61 of size 1m x 1m.

On the other hand, as shown in FIG. 11, the variation parameter .delta..phi.1 has high sensitivity in the upper region of the angle of view of the in-vehicle camera 62, which is 10 degrees to 20 degrees vertically and -15 degrees to -5 degrees horizontally (in this case, in FIG. 10 is the same angle of view as the variation parameter .delta..phi.1 of . Therefore, the first specific area 64 is placed in the upper area within the image. That is, the first specific area 64 is set to the area of the angle of view of the image of the in-vehicle camera 62 (vertical: 10 degrees to 20 degrees, horizontal: -15 degrees to -5 degrees). The variation parameter δφ2 has a high sensitivity in the central region of the angle of view of the in-vehicle camera 62, which is -5 degrees to 5 degrees vertically and 30 degrees to 40 degrees horizontally. Therefore, the second specific area 65 is arranged in the central area within the image. In other words, the second specific area 65 is set to the area of the angle of view of the image of the in-vehicle camera 62 (-5 degrees to 5 degrees vertically, 30 degrees to 40 degrees horizontally). The variation parameter δφ3 has high sensitivity in the lower region of the view angle of the vehicle-mounted camera 62 from -20 degrees to -10 degrees in the vertical direction and from -10 degrees to 0 degrees in the horizontal direction. Therefore, the third specific area 66 is located in the central area within the image. That is, the third specific area 66 is set to the area of the angle of view of the image of the in-vehicle camera 62 (vertical -20 to -10 degrees, horizontal -10 to 0 degrees).

Three charts 61 are arranged so as to correspond to the three specific areas (64, 65, 66) thus determined (see FIG. 6). These specific areas (64, 65, 66) are measurable with a chart 61 of size 1 m x 1 m. The size of the specific areas (64, 65, 66) is smaller than the chart 61, which is the calibration image. That is, the area of the chart 61 is made larger than the size of the specific regions (64, 65, 66) on the image so that the specific regions (64, 65, 66) are included inside the chart 61. FIG.

In this way, the small chart 61 is used to measure only pixel deviations in specific areas (64, 65, 66). Then, each variation parameter is calculated from the displacement amount of the pixel displacement of the specific regions (64, 65, 66). Then, the displacement amount of the pixel displacement of the entire screen is calculated from the calculated variation parameter. This makes it possible to cancel the pixel displacement of the entire screen at each pixel point and perform the calibration of the pixel positions of the entire screen. Since the selection of the specific regions (64, 65, 66) selects the sensitive part of the variation parameter, the pixel displacement in the specific regions (64, 65, 66) can be measured with a chart 61 of size 1m x 1m. be.

Although the invention made by the present inventor has been specifically described above based on the embodiment, the invention is not limited to the above embodiment, and it goes without saying that various modifications are possible.

10: Imaging Unit 11: Area Selecting Unit 12: External Input Unit 13: Parameter Calculating Unit 14: Overall Pixel Misalignment Estimating Unit 15: Calibration Unit 61: Small Chart 62: Vehicle-mounted Camera 63: Windshield

Claims (19)

an imaging unit that images an object;
an estimating unit that calculates the displacement of the pixels of the entire surface based on the image captured by the imaging unit;
an area selection unit that selects a plurality of predetermined areas that are predetermined according to the shape and inclination of the windshield in the image captured by the imaging unit;
a calibration unit that corrects displacement of pixels on the entire surface based on information from the estimation unit;
with
The in-vehicle camera, wherein the estimating unit estimates the pixel displacement of the entire surface using images detected from the plurality of predetermined regions in the image captured by the imaging unit. The vehicle-mounted camera according to claim 1,
The vehicle-mounted camera, wherein the plurality of predetermined areas are three or more predetermined areas. The vehicle-mounted camera according to claim 1,
An in-vehicle camera, wherein the size of at least one predetermined area among the plurality of predetermined areas is 1/2 or less of the size of the entire image. The vehicle-mounted camera according to claim 1,
When the plurality of predetermined regions are n predetermined regions, the size of at least one of the plurality of predetermined regions is 1/n or less of the entire image. in-vehicle camera. The vehicle-mounted camera according to claim 1,
When calculating the pixel deviation, a calibration image is arranged so as to appear in the image captured by the imaging unit and to correspond to the plurality of predetermined regions, and the imaging unit captures the entire surface. ,
An in-vehicle camera, wherein the size of the calibration image is larger than the size of the corresponding predetermined area. An in-vehicle camera according to claim 5,
An in-vehicle camera, wherein two or more calibration images are used. An in-vehicle camera according to claim 6,
The vehicle-mounted camera, wherein the calibration image has a size of 2 m×2 m or less. An in-vehicle camera according to claim 6,
When the plurality of predetermined areas are n predetermined areas and the calibration image is n calibration images,
When calculating the pixel shift, the n calibration images are arranged so as to correspond to the n predetermined regions, respectively, and the entire surface is photographed once by the imaging unit and measured once. An in-vehicle camera characterized by calculating a pixel shift in. An in-vehicle camera according to claim 6,
When the plurality of predetermined regions are n predetermined regions and the calibration image is one calibration image,
When calculating the pixel shift, the arrangement of the one calibration image is changed so that the entire surface on which the one calibration image is arranged can be photographed for all of the n predetermined regions. A vehicle-mounted camera characterized in that the image of the entire surface is taken n times and the pixel shift is calculated by measuring the n times. The vehicle-mounted camera according to claim 1,
The estimating unit uses images detected from the plurality of predetermined regions to detect pixel misalignment in each predetermined region, and estimates pixel misalignment of the entire surface based on the amount of each pixel misalignment. in-vehicle camera. An in-vehicle camera according to claim 10,
An in-vehicle camera, comprising: a parameter calculator that calculates a variation parameter based on the amount of pixel shift in each region. The vehicle-mounted camera according to claim 11,
An in-vehicle camera, wherein the variation parameters are substantially orthogonal to each other. An in-vehicle camera according to claim 2,
At least a first upper region in the image, a second upper region different from the first upper region, and a lower An in-vehicle camera, characterized in that the image of the area is used to estimate the pixel deviation of the entire surface. 14. The vehicle-mounted camera according to claim 13,
When the inclination angle of the windshield is 30° to 35°,
The first upper region is arranged in a region where the angle of view of the image is 10° to 20° in the vertical direction and -15° to -5° in the horizontal direction,
The second upper region is arranged in a region where the angle of view of the image is 10° to 20° in the vertical direction and 30° to 40° in the horizontal direction,
The vehicle-mounted camera, wherein the lower region is arranged in a region where the angle of view of the image is -20° to -10° in the vertical direction and -35° to -25° in the horizontal direction. An in-vehicle camera according to claim 2,
When the inclination angle of the windshield of the vehicle in which the vehicle-mounted camera is mounted is 35° or more,
An in-vehicle camera characterized by estimating a pixel shift of the entire surface by using at least images of an upper area, a middle area, and a lower area in the image. 16. The vehicle-mounted camera according to claim 15,
When the inclination angle of the windshield is 35° or more,
The upper region is arranged in a region where the angle of view of the image is 10° to 20° in the vertical direction and -15° to -5° in the horizontal direction,
The middle region is arranged in a region where the angle of view of the image is -5° to 5° in the vertical direction and 30° to 40° in the horizontal direction,
The vehicle-mounted camera, wherein the lower region is arranged in a region where the angle of view of the image is -20° to -10° in the vertical direction and -10° to 0° in the horizontal direction. The vehicle-mounted camera of claim 1,
The in-vehicle camera, wherein the estimation unit, the area selection unit, and the calibration unit are configured by a software program or a hardware circuit. a photographing step of photographing an object;
an estimating step of estimating the displacement of the pixels of the entire surface based on the image captured in the photographing step;
a selection step of selecting a plurality of predetermined regions determined in advance according to the shape and inclination of the windshield in the image captured in the photographing step;
a calibration step of correcting the pixel deviation of the entire surface based on the information of the estimation step;
A method of calibrating an in-vehicle camera, wherein in the estimation step, pixel deviations of the entire surface are estimated using images detected from the plurality of predetermined regions in the photographed image. placing an on-board camera behind a windshield, capturing images of a plurality of small charts from the inside of a vehicle through the windshield with the on-vehicle camera, and obtaining captured images of the plurality of small charts; and,
inputting design values for the windshield;
a step of selecting an image area with good variation sensitivity based on the design value of the windshield, and calculating a pixel shift of each selected image area from an image captured by arranging the chart in the selected image area;
calculating the value of each variation parameter from the pixel displacement of each selected image region;
calculating an estimated pixel shift for estimating the pixel shift of the entire screen from the value of the variation parameter;
A method of calibrating an in-vehicle camera, comprising: performing calibration by canceling pixel shifts in the entire screen within a captured image at each pixel point based on the estimated pixel shifts in the entire screen.

Images