JP7405710B2 - Processing equipment and in-vehicle camera equipment - Google Patents

Description

The present invention relates to a processing device and a vehicle-mounted camera device that calculate data for correcting one or more camera images into distortion-free images.

The prior art disclosed in Patent Document 1 is a method for correcting a first camera that calibrates an image taken by the first camera through a windshield (transparent body) after installing a stereo camera having a first camera and a second camera in a vehicle. Calculate parameters. Next, the image shift caused by the windshield is removed by calculating a correction parameter for the second camera that calibrates the parallax shift between the subject of the first camera and the second camera.

The conventional technology disclosed in Patent Document 2 photographs an object with a stereo camera with and without a windshield (transparent body) interposed, calculates the coordinates of the object on the image, and calculates the difference between these coordinates. First, the correction parameters that make up the image are calculated. Based on this correction parameter, the image shift caused by the windshield is removed.

JP2019-132855A Japanese Patent Application Publication No. 2015-169583

If the vehicle to which the camera is installed is misaligned with respect to the chart, or the camera is misaligned with respect to the vehicle, the correction parameters configured as image misalignment due to the windshield may include the image due to misalignment of the vehicle and camera. This will include deviations. As a result, when an object is photographed at a distance different from the distance of the object photographed for calibration, a vertical shift occurs between the objects on the images of the first and second cameras, resulting in pattern matching. During processing, the images of the object do not match properly, making it impossible to accurately detect parallax. Furthermore, a horizontal shift occurs between the objects on the images of the first camera and the second camera, resulting in a parallax error. As described above, due to the parallax error, the distance to the object cannot be accurately measured.

An object of the present invention is to detect misalignment of the camera relative to the marker due to misalignment of the vehicle or misalignment of the camera when a monocular camera or a stereo camera is attached to a vehicle to photograph markers for calibration. It is an object of the present invention to provide a processing device that calibrates image shift caused by glass and accurately measures distance so as not to be affected by image shift caused by factors.

The processing device according to the present invention acquires the installation deviation of the in-vehicle camera from images taken of a plurality of markers at different distances from the in-vehicle camera and modeled vehicle window data, and uses the installation deviation, A processing device that changes correction data for correcting an image of the in-vehicle camera, wherein the installation deviation is a positional deviation with respect to a position where the in-vehicle camera should be placed with respect to a chart where the marker is present. .

According to the present invention, based on the distance and positional shift on the image of markers having different distances, the magnification ratio and offset shift of the vehicle glass model that does not depend on the distance, and the position and orientation between the marker and the camera that depend on the distance are determined. Misalignment can be detected. Then, by changing the image correction data to calibrate the image shift due to the windshield, assuming that there is a misalignment in the vehicle installation and the camera installation, it is possible to correct the misalignment in the vehicle installation and the camera in just one shot. Image deviations due to misalignment can be removed and distances can be measured accurately.

1 is a diagram showing an embodiment of a processing device and a vehicle-mounted camera device of the present invention. FIG. 3 is a diagram showing an example of a marker arrangement area. 1 is a diagram showing an embodiment of a processing device and a vehicle-mounted camera device of the present invention. It is a figure showing operation of a processing device and an in-vehicle camera device. FIG. 7 is a diagram showing the positions of markers on a corrected image. FIG. 3 is a diagram illustrating an example of the arrangement of markers, stereo cameras, and vehicles viewed from above. It is a figure showing an example of arrangement of a marker, a stereo camera, and a glass seen from the side. FIG. 3 is a diagram showing the operation of a stereo camera. FIG. 7 is a diagram illustrating an example of calculating parallax on corrected images of two imaging system units. 4 is a diagram showing a modification of the processing device and vehicle-mounted camera device shown in FIG. 3. FIG. 4 is a diagram showing a modification of the processing device and vehicle-mounted camera device shown in FIG. 3. FIG. It is a figure showing operation of a camera. FIG. 3 is a diagram showing an example of an image in which a distance measurement target object is photographed.

(Embodiment 1)
FIG. 1 is a diagram showing the configuration of an image proofing device according to an embodiment of the present invention. FIG. 2 is a diagram showing an example of a marker placement area. FIG. 3 is a diagram showing the configuration of a stereo camera of an image proofing device according to an embodiment of the present invention.

The processing device of this embodiment is, for example, an image proofing device. The image correction device has image correction data for correcting distortion and deviation of images taken without passing through the windshield 112, and the image correction device has image correction data for correcting distortion and deviation of images taken without passing through the windshield 112. It is installed in a shifted position and orientation (direction 123). When the stereo camera 104 is attached to the vehicle 111 at a position and attitude (direction 124) that is deviated from the predetermined position and attitude (direction 122), the influence of the installation deviation of the vehicle 111 and the installation deviation of the stereo camera 104 is considered. The image correction data is changed so as to correct the image shift caused by the windshield 112.

FIGS. 1 and 3 show the configuration of an embodiment of the image proofing apparatus of the present invention. Here, the corrected image storage unit 316, the parallax image storage unit 317, and the distance detection unit 330, which are displayed in gray, are not included in the image calibration device and operate in the case of detecting the distance of a three-dimensional object. These operations will be described later.

The image calibration device includes a plurality of markers, a vehicle installation deviation detection section 105, and a stereo camera 104.

As shown in FIG. 2, the plurality of markers have a circular shape. A plurality of markers 201 are arranged in front of the vehicle, and a plurality of markers 202, a plurality of markers 203, a plurality of markers 204, and a plurality of markers 205 are arranged in front of the upper left, in front of the upper right, and lower left of the plurality of markers 201, respectively. It is placed in front of and in front of the bottom right. Here, marker 201, marker 202, marker 203, marker 204, and marker 205 are drawn on chart 206, chart 207, chart 208, chart 209, and chart 210, respectively. Further, the marker 101 represents the marker 201, the marker 102 represents the marker 202 or 204, and the marker 203 represents the marker 203 or the marker 205. The markers 201 to 205 may be arranged other than that shown in FIG. 2 as long as they are arranged at different distances from the camera.

The vehicle installation deviation detection unit 105, which is composed of LiDAR (Light Detection and Ranging), a calculation unit, etc., has surface shape data (design values) of the vehicle and the position of the vehicle (design position) when installed as designed. Then, the surface position (measured value) of the vehicle 111 is measured, and the design value of the surface shape data of the vehicle 111 is compared with the measured value of the surface position of the vehicle 111 to determine the positional deviation and attitude of the vehicle 111 from the designed position. Detect deviation.

The stereo camera 104 is attached to the vehicle 111 and photographs the markers 201 to 205 through the windshield 112 of the vehicle 111. Further, the stereo camera 104 includes an imaging system section 300a, an imaging system section 300b, and a calculation section 310, and is connected to the vehicle position shift detection section 105.

An imaging system section 300a such as a camera includes an optical element section 301a and an image sensor section 302a. The optical element section 301a such as a lens refracts light and forms an image on the image sensor section 302a. An image sensor section 302a such as an image sensor receives an image of light refracted by the optical element section 301a, and generates an image according to the intensity of the light.

An imaging system section 300b such as a camera includes an optical element section 301b and an image sensor section 302b. Further, the design values of the focal lengths of the imaging system section 300a and the imaging system section 300b are the same. The directions of the optical axes of the imaging system section 300a and the imaging system section 300b are generally the same. The optical element section 301b such as a lens refracts light and forms an image on the image sensor section 302b. An image sensor section 302b such as an image sensor receives an image of the light refracted by the optical element section 301b, and generates an image according to the intensity of the light.

The image captured by the imaging system unit 300a is referred to as a reference image because it is a reference image when creating a parallax image. Further, the image captured by the imaging system unit 300b is an image in which an area matching the area extracted from the reference image is searched for when creating a parallax image, and is referred to as a reference image.

The calculation unit 310, such as a CPU (central processing unit) and memory, includes a captured image storage unit 311, an image correction information storage unit 312, a calibration information storage unit 313, a synchronization signal generation unit 314, and a reference image capture unit. 315a, a reference image capture unit 315b, an image calibration unit 320, a corrected image storage unit 316, a parallax image storage unit 317, a shift correction storage unit 318, and a distance detection unit 330.

A captured image storage unit 311 such as a memory stores images output from the imaging system unit 300a and the imaging system unit 300b.

An image correction information storage unit 312 such as a memory stores a corrected image (a post-distorted image) corresponding to each pixel on the captured image (image without distortion) for the image of the imaging system unit 300a and the image of the imaging system unit 300b. The two-dimensional coordinates (image correction data) on the image) are stored. The image correction information storage unit 312 stores the image correction data of the imaging system unit 300a and the imaging system unit 300b obtained in advance without the windshield 112, and also stores the image correction data of the imaging system unit 300a and the imaging system unit 300b obtained in advance without the windshield 112. Image correction data for correcting an image without distortion is stored over the 112. Here, the image without distortion is an image obtained by perspective projection using a pinhole model. This image correction data is used for processing to correct lens distortion and optical axis deviation of the image after the image is captured, and represents the distortion of the image at the time of image capture.

A calibration information storage unit 313 such as a memory stores the focal lengths of the imaging system unit 300a and the imaging system unit 300b, the pixel pitch, the optical axis position on the captured image, the optical axis position on the corrected image, and the vehicle 111 so that there is no deviation. Three-dimensional positions of markers 201 to 205 with respect to stereo camera 104 when installed, design values of image deformation model data (glass deformation data) by windshield 112 in imaging system section 300a and imaging system section 300b, and corrected image at that time Each of the above marker images, horizontal and vertical enlargement ratios and offset deviations of the model of image deformation by the windshield 112, positional deviations and posture deviations of vehicle installation, and positional deviations and posture deviations of camera installation are stored.

Here, the horizontal and vertical magnification ratio and offset shift amount of the image deformation caused by the windshield 112, the positional shift and posture shift of the vehicle installation, and the positional shift and posture shift of the camera installation are calculated by the image shift factor detection unit 322. has been done. The initial values of the horizontal and vertical magnification ratios for image deformation by the windshield 112 are 1, and the other initial values are zero. The glass deformation data is the two-dimensional coordinates of the position on the image after passing through the windshield 112 corresponding to each pixel on the image before passing through the windshield 112.

The synchronization signal generator 314 generates and transmits a synchronization signal.

The reference image capture unit 315a sends a synchronization signal and exposure time information to the image sensor unit 302a in accordance with the synchronization signal from the synchronization signal generator 314, and then acquires an image generated by the image sensor unit 302a. The image is stored in the captured image storage unit 311.

The reference image capture unit 315b sends a synchronization signal and exposure time information to the image sensor unit 302b in accordance with the synchronization signal from the synchronization signal generator 314, and then acquires an image generated by the image sensor unit 302b. The image is stored in the captured image storage unit 311.

The image calibration section 320 includes a marker position detection section 321 , an image shift factor detection section 322 , and an image correction data change section 323 .

The marker position detection unit 321 corrects an image taken through the windshield 112 based on image correction data stored in advance in the image correction information storage unit 312 and design values of image deformation data due to the windshield 112. Correction data is calculated, and using the image correction data, images captured by the imaging system section 300a and the imaging system section 300b are corrected. Next, the marker position detection unit 321 performs pattern matching processing to match the marker images stored in the calibration information storage unit 313 with the marker images on those corrected images, and the marker positions on the corrected images. Detect (detection position).

The image shift factor detection unit 322 stores the focal lengths and pixel pitches of the imaging system unit 300a and the imaging system unit 300b stored in the calibration information storage unit 313, the optical axis position on the captured image, the optical axis position on the corrected image, Based on the three-dimensional position of the marker with respect to the stereo camera 104 when the vehicle 111 is installed so that there is no misalignment, and the design values of the glass deformation data of the imaging system section 300a and the imaging system section 300b, the vehicle 111 is set so that there is no misalignment. The marker position (design position) on the corrected image is calculated when the image shift due to the windshield 112 is as designed.

Furthermore, based on the detected positions and design positions of the markers 201 to 205 on the corrected image, the image deviation factor detection unit 322 determines that the image deviation caused by the windshield 112 is caused by the distance between the stereo camera 104 and the markers 201 to 205. By using the fact that the camera installation deviation depends on the distance between the stereo camera 104 and the markers 201 to 205, the horizontal and vertical magnification ratio and offset deviation of the image deformation by the windshield 112, the camera Calculate the installation deviation. Here, the camera installation deviation is a positional deviation and a posture deviation (deviation from a designed position) between the markers 201 to 205 and the stereo camera 104. Next, the camera installation deviation is calculated based on the vehicle installation deviation detected by the vehicle installation deviation detection unit 105 and the camera installation deviation calculated by the image deviation factor detection unit 322. Here, the camera installation deviation is a positional deviation and posture deviation (deviation from a designed position) of the stereo camera 104 with respect to the vehicle 111 when the stereo camera 104 is attached to the vehicle 111.

The image correction data changing unit 323 stores the focal lengths and pixel pitches of the imaging system unit 300a and the imaging system unit 300b stored in the calibration information storage unit 313, the optical axis position on the captured image, the optical axis position on the corrected image, The three-dimensional positions of the markers 201 to 205 with respect to the stereo camera 104 when installed on the vehicle 111 without deviation are read.

In addition, the image correction data changing unit 323 uses the camera installation deviation, camera installation deviation, horizontal and vertical enlargement ratios of the model of the windshield 112, and offset deviation detected by the image deviation factor detection unit 322 to install the camera. Camera installation deviation data representing image deviation due to deviation, camera installation direction deviation data representing image deviation due to deviation in camera installation direction, horizontal and vertical magnification ratio of the model of windshield 112, glass deviation data when there is offset deviation. Calculate.

In addition, the image correction data changing unit 323 uses two-dimensional linear interpolation to perform conversion using camera installation direction deviation data, conversion using glass deviation data, conversion using glass deformation data, conversion using camera installation deviation data, and conversion using two-dimensional linear interpolation. Conversion using image correction data (representing image distortion at the time of imaging) is performed sequentially for each pixel. As a result, the image correction data changing unit 323 removes pixel deviation due to camera installation deviation, and calculates image correction data for the standard image and the reference image through the windshield 112 when facing the direction of travel of the vehicle 111. , these image correction data are stored in the image correction information storage section 312.

A corrected image storage unit 316 such as a memory stores corrected images obtained by correcting the captured images by the image correction unit 331 using the image correction data of the imaging system unit 300a and the imaging system unit 300b. A parallax image storage unit 317 such as a memory stores parallax images. The misalignment correction information storage unit 318 such as a memory stores time-series data of the positions of areas with both vertical edges and horizontal edges in the standard image and the reference image, design values of vanishing point positions, vanishing point position data (detected values), Stores horizontal and vertical image shifts due to camera installation misalignment.

The distance detection section 330 includes an image correction section 331, a parallax calculation section 332, a recognition section 333, and a camera mounting misalignment detection section 334.

The image correction unit 331 uses the image correction data of the standard image and reference image read from the image correction information storage unit 312 to convert the standard image and reference image at the time of shooting read from the captured image storage unit 311 into images without distortion. Convert to These corrected standard images and reference images are stored in the corrected image storage unit 316.

The parallax calculation unit 332 reads the corrected standard image and reference image from the corrected image storage unit 316, and performs correction corresponding to an area of a predetermined size (hereinafter referred to as a reference area) extracted from the corrected standard image. Search for an area at the same height on the later reference image. The parallax calculation unit 332 calculates the difference between the position of an area on the reference image that matches the reference area (hereinafter referred to as a reference area) and the position of the reference area, that is, the parallax. A parallax image is calculated by calculating parallax for each reference area.

The recognition unit 333 reads the parallax image from the parallax image storage unit 317, and performs imaging based on the parallax, the focal length (baseline length) of the imaging system unit 300a and the imaging system unit 300b, the focal length, and the size of one pixel. The distance from the stereo camera 104 to the object on the image is calculated in the optical axis direction of the system section 300a and the imaging system section 300b. The recognition unit 333 calculates a distance image by calculating the distance for each area.

The recognition unit 333 then reads the corrected reference image from the corrected image storage unit 316, and uses the corrected reference image and distance image to identify the object in the reference image and the object on the reference image. The position is recognized, and the three-dimensional relative position and relative velocity of the object with respect to the stereo camera 104 are calculated. Here, the three-dimensional relative position coordinate system with respect to the stereo camera 104 has an origin at the center of the entrance pupil of the imaging system section 300a, an x coordinate in the right direction with respect to the imaging system section 100a, a y coordinate in the downward direction, and an optical axis direction. Take the z coordinate. Furthermore, the recognition unit 333 calculates the time until the collision based on the relative position and relative speed of the stereo camera 104 and the object, and determines whether the collision will occur within a predetermined time.

The camera mounting misalignment detection unit 334 creates horizontal and vertical edge images for the previous and current corrected reference images read from the corrected image storage unit 316, and creates vertical edges (vertical edges) and horizontal edges. Detect areas that include both edges (horizontal edges). Next, the camera mounting misalignment detection unit 334 performs pattern matching processing, searches for the area on the reference image after the previous correction that corresponds to the area on the reference image after the current correction, and searches for vertical edges and horizontal edges. Obtain time-series data of the position of the region that includes both edges.

Further, the camera mounting misalignment detection unit 334 obtains a plurality of approximate curves of the time-series data of the position of the area, and detects the position (detected value) of the vanishing point on the reference image based on the intersection points of these curves. The camera mounting deviation detection unit 334 stores the difference between the detected value and the design value of the position of the vanishing point on the reference image as the camera mounting deviation in the deviation correction information storage unit 318. Finally, the camera mounting misalignment detection section 334 corrects the image correction data of the reference image read from the image correction information storage section 312 by the camera mounting misalignment, and stores it in the image correction information storage section 312. Similar processing is performed for the reference image.

The operation procedure of one example of the stereo camera 104 of this embodiment shown in FIGS. 1 to 3 will be explained using FIG. 4.

Step 401: The synchronization signal generation section 314 generates a synchronization signal and sends it to the reference image capture section 315a and the reference image capture section 315b.

Immediately after receiving the synchronization signal from the synchronization signal generation section 314, the reference image capture section 315a sends the synchronization signal and exposure time information to the image sensor section 302a. Immediately after receiving the synchronization signal and exposure time information from the reference image capture unit 315a, the image sensor unit 302a receives an image of the light refracted by the optical element unit 301a for the exposure time, and adjusts the intensity of the light. A corresponding image is generated and sent to the reference image capture unit 315a. The reference image capture unit 315a receives an image from the image sensor unit 302a and stores the image in the captured image storage unit 311.

Immediately after receiving the synchronization signal from the synchronization signal generation section 314, the reference image capture section 315b sends the synchronization signal and exposure time information to the image sensor section 302b. Immediately after receiving the synchronization signal and exposure time information from the reference image capture unit 315b, the image sensor unit 302b receives an image of the light refracted by the optical element unit 301b for the exposure time, and adjusts the intensity of the light. A corresponding image is generated and sent to the reference image capture unit 315b. The reference image capture unit 315b receives an image from the image sensor unit 302b and stores the image in the captured image storage unit 311.

Step 402: The marker position detection unit 321 receives the standard image and reference image at the time of imaging from the captured image storage unit 311, and the image correction data of the standard image and reference image stored in advance from the image correction information storage unit 312. , the glass deformation data of the standard image and the reference image stored in advance are read from the calibration information storage unit 313, respectively.

Image correction tables (F12x, F12y) for the reference image when the windshield 112 is not passed through are shown in equations (1) and (2). Here, (X1, Y1) is the position on the photographed image (after passing through the optical element), and (X2, Y2) is the position on the corrected image (before passing through the optical element) when not passing through the windshield 112. be.

X1=F12x(X2, Y2)...(1)
Y1=F12y(X2, Y2)...(2)

Using equations (3) and (4), the windshield corresponding to the position (X3, Y3) on the corrected reference image before passing through the windshield 112 is determined based on the glass deformation data (Gx, Gy). The position (X2, Y2) on the corrected reference image after passing through 112 is calculated.

X2=Gx(X3, Y3)...(3)
Y2=Gy(X3, Y3)...(4)

The position (X2, Y2) on the corrected reference image after passing through the windshield 112 is a real number. Therefore, using equations (1) and (2) for the four positions where the values near the position (X2, Y2) are integers, after the correction after passing through the windshield 112 (the optical element The position on the reference image after photographing (after passing through the optical element) corresponding to the position on the reference image before passing through the optical element is determined, and two-dimensional linear interpolation is performed. Thereby, the position on the reference image after photographing corresponding to the position (X3, Y3) on the corrected reference image through the windshield 112 is calculated. This process is performed for each pixel, and image correction data (Equations (5) and (6)) for converting the photographed image into a corrected image of the view through the windshield 112 is calculated.

X1=F13x(X3, Y3)...(5)
Y1=F13y(X3, Y3)...(6)

Using this image correction data, two-dimensional linear interpolation is performed on the luminance values of four pixels around the position (X1, Y1) of the reference image after image capture, and the pixel (X3, Y3) of the reference image after image correction is ) is calculated. The above procedure is performed for each pixel of the corrected reference image to calculate the brightness value of the corrected reference image when the windshield 112 is as designed.

The marker position detection unit 321 detects the focal lengths and pixel pitches of the imaging system unit 300a and the imaging system unit 300b stored in the calibration information storage unit 313, the optical axis position on the captured image, the optical axis position on the corrected image, and the vehicle The three-dimensional positions of the markers 201 to 205 with respect to the stereo camera 104 when the markers 111 are installed without any deviation are read. Based on the values of these parameters, the marker position detection unit 321 further determines the marker position ( design position).

Next, the marker position detection unit 321 reads each marker image stored in the calibration information storage unit 313, and calculates the difference between the luminance value of the reference image after image correction and the luminance value of the marker image in the vicinity of the designed marker position. The sum of absolute values (SAD, Sum of Absolute Difference) is calculated. The marker position detection unit 321 finds the position where the SAD is the smallest near the designed positions of the markers 201 to 205.

The marker position detection unit 321 performs isometric straight line fitting based on the SAD of the pixels on the left and right of the position where the SAD is the smallest, and calculates the sub-pixel at the position where the reference image after image correction and the marker image most match. . The marker position detection unit 321 calculates the horizontal positions of the markers 201 to 205 by adding this sub-pixel to the horizontal position where the SAD is the smallest.

Next, the marker position detection unit 321 performs isometric straight line fitting based on the SAD of the pixels above and below the position where the SAD is the smallest, and performs the subpixel at the position where the reference image after image correction and the marker image most match. Calculate. By adding this subpixel to the vertical position where the SAD is the smallest, the vertical positions of the markers 201 to 205 are calculated. Similarly, similar processing is performed for the other markers 201 to 205 to calculate the positions of the markers 201 to 205 on the corrected reference image. The positions of the markers 201 to 205 detected through the above processing will be referred to as detected positions. Further, the detected positions of the markers 201 to 205 correspond to the markers 502 and 504 on the corrected image 501 in FIG.

The above image correction and marker position detection processing is also performed on the reference image.

Step 403: The image shift factor detection unit 322 extracts from the calibration information storage unit 313 the focal lengths of the imaging system unit 300a and the imaging system unit 300b, the pixel pitch, the optical axis position on the captured image, the optical axis position on the corrected image, and the vehicle. The three-dimensional positions of the markers 201 to 205 with respect to the stereo camera 104 and the design values of the glass deformation data of the imaging system section 300a and the imaging system section 300b are read when the markers 201 to 205 are installed so as not to be misaligned. Based on this information, the image shift factor detection unit 322 detects the markers 201 to 205 on the corrected image when the vehicle 111 is installed so that there is no shift, and the image shift due to the windshield 112 is as designed. Calculate the position (design position). The designed positions of the markers 201 to 205 correspond to the markers 503 and 505 on the corrected image 501 in FIG.

The image shift factor detection unit 322 determines the front position as described below based on the design positions of the markers 201 to 205 on the corrected image calculated as described above and the detected positions of the markers 201 to 205 detected in step 402. The magnification ratio of the glass 112, camera installation deviation, etc. are estimated. Here, based on the positions of the markers 201 to 205 having different distances from the stereo camera 104, it is confirmed that the image shift due to the windshield 112 does not depend on the distance between the stereo camera 104 and the markers 201 to 205, and that the camera installation shift is 104 and the markers 201 to 205.

Step 403: The image shift factor detection unit 322 determines the magnification ratio of the windshield 112 and the camera installation shift based on the design positions and detection positions of the markers 201 to 205 on the corrected image obtained in step 402. We will estimate the following. Here, based on the positions of the markers 201 to 205 having different distances from the stereo camera 104, it is confirmed that the image shift due to the windshield 112 does not depend on the distance between the stereo camera 104 and the markers 201 to 205, and that the camera installation shift is 104 and the markers 201 to 205.

First, regarding the reference image, the image shift factor detection unit 322 calculates the average interval between horizontally and vertically adjacent markers 201 regarding the detected position and design position of the marker 201 on the chart 206 in the corrected image. Similarly, the average interval between the detected positions and designed positions of the markers 202 to 205 on the charts 207 to 210 is calculated. Therefore, we take advantage of the fact that the image shift caused by the windshield 112 does not depend on the distance between the stereo camera 104 and the markers 201 to 205, and that the camera installation shift depends on the distance between the stereo camera 104 and the markers 201 to 205. do. Thereby, using equations (7) to (10), the magnification ratios Mgh and Mgv in the horizontal and vertical directions by the windshield 112 and the distance dLc between the stereo camera 104 and the marker 201 are calculated. Here, the far marker is the marker 201, and the closer markers are the markers 202 to 205.

Mgh = Mifh Minh (Lf - Ln) / (Minh Lf - Mifh Ln) ・・・(7)
Mgv = Mifv Minv (Lf - Ln) / (Minv Lf - Mifv Ln) ... (8)
Mifh = Wifh / Wdfh ... (9)
Mifv = Wifv / Wdfv ... (10)
Minh = Winh / Wdnh ... (11)
Minv = Winv / Wdnv ... (12)
dLc = (dLch + dLcv) / 2 ... (13)
dLch = (Mifh - Minh) Lf Ln / (Minh Lf - Mifh Ln) ... (14)
dLcv = (Mifv - Minv) Lf Ln / (Minv Lf - Mifv Ln) ... (15)
- Design value of distance between far marker 201 and stereo camera 104: Lf
- Design value of distance between nearby markers 202 to 205 and stereo camera 104: Ln
・Distance deviation between markers 201 to 205 and stereo camera 104: dLc
・Horizontal magnification rate by windshield 112: Mgh
・Vertical magnification rate by windshield 112: Mgv
・Average horizontal spacing of design positions of distant markers 201 on the corrected image: Wdfh
・Average horizontal distance between design positions of nearby markers 202 to 205 on the corrected image: Wdnh
- Average vertical spacing of design positions of far markers 201 on the corrected image: Wdfv
・Average vertical distance between design positions of nearby markers 202 to 205 on the corrected image: Wdnv
・Average horizontal distance between detection positions of distant markers 201 on the corrected image: Wifh
・Average horizontal distance between detection positions of nearby markers 202 to 205 on the corrected image: Winh
- Average interval in the detection direction of the design position of the far marker 201 on the corrected image: Wifv
・Average vertical distance between detection positions of nearby markers 202 to 205 on the corrected image: Winv
・Horizontal magnification rate of the distance between distant markers 201 on the corrected image: Mifh
・Horizontal magnification rate of the interval between nearby markers 202 to 205 on the corrected image: Minh
・Vertical magnification rate of the distance between distant markers 201 on the corrected image: Mifv
・Vertical magnification rate of the interval between nearby markers 202 to 205 on the corrected image: Minv

The marker 201 on the chart 206 located at a position close to the marker 202 is taken as a far marker, and the marker 202 is taken as a near marker. Using equations (13) to (15), the marker 202 at the upper left of the image and the stereo camera 104 are Calculate the distance deviation dLclu between. Similar calculations are performed to calculate distance deviations dLcru (upper right on the image), dLcld (lower left on the image), and dLclu (lower right on the image) between the markers 203 to 205 and the stereo camera 104. As shown in FIG. 6, both the distance deviation between the left and right markers with respect to horizontal lines 602 and 603 perpendicular to the optical axis direction 601 of the stereo camera 104 are calculated, and the yaw angle deviation between the markers and the stereo camera 104 is calculated. It can be calculated. Therefore, the yaw angle deviation dψc between the marker and the stereo camera 104 is calculated using equations (16) to (18).

dψc = atan((dLcl - dLcr) / Llr) ...(16)
dLcl = (dLcul + dLcdl) / 2 ... (17)
dLcr = (dLcur + dLcdr) / 2 ... (18)
・Average distance deviation dLcl of left markers 202 and 204
・Average dLcr of distance deviation of right side markers 203 and 205
・Average Llr of the design value of the distance between the left and right markers

As shown in FIG. 7, the pitch angle deviation between the marker and the stereo camera 104 can be calculated by both the distance deviation between the upper and lower markers with respect to the vertical line 704 and the line 705 perpendicular to the optical axis direction 703 of the stereo camera 104. . Therefore, the pitch angle deviation dφc between the marker and the stereo camera 104 is calculated using equations (19) to (21).

dφc = atan((dLcd - dLcu) / Lud) ...(19)
dLcu = (dLcul + dLcur) / 2...(20)
dLcd = (dLcdl + dLcdr) / 2 ... (21)
・Average distance deviation between upper markers 202 and 203 and stereo camera: dLcu
・Average distance deviation between lower markers 204 and 205 and stereo camera: dLcd
- Average design value of distance between upper and lower markers: Lud

The roll angle deviation between the marker and the stereo camera 104 corresponds to the rotational deviation between the detected position and the designed position of the marker about the optical axis position (or center position) on the corrected image. Therefore, the roll angle deviation dθc is calculated using equation (22) on the corrected image.

dθc = - atan[Σ{(Vi - Vo)(Ud - Uo) - (Ui - Uo)(Vd - vo)} / Σ{(Ui - Uo)(Ud - Uo) + (Vi - Vo)(Vd - Vo)}] ...(22)
・Each detection position of markers 201 to 205: (Ui, Vi)
・Each design position of markers 201 to 205: (Ud, Vd)
・Optical axis position (center position) on corrected image: (Uo, Vo)

The horizontal and vertical positional deviation between the marker and the stereo camera 104 depends on the distance between the stereo camera 104 and the marker, but the offset deviation due to the windshield 112 depends on the distance between the stereo camera 104 and the marker. Not dependent. Utilizing this fact, using equations (23) to (26), the horizontal and vertical offset deviations dUg and dVg due to the windshield 112 and the horizontal and vertical offsets between the marker and the stereo camera 104 are calculated. Calculate positional deviations dXc and dYc. Here, the far marker corresponds to the marker 201, and the closer markers correspond to the markers 202 to 205.

dXc = (dUin - dUif) c Lf Ln / {f (Lf - Ln)} ...(23)
dYc = (dVin - dVif) c Lf Ln / {f (Lf - Ln)} ... (24)
dUg = (dUin Ln - dUif Lf) / (Lf - Ln) ... (25)
dVg = (dVin Ln - dVif Lf) / (Lf - Ln) ... (26)
・Lens focal length: f
・Pixel pitch of image sensor: c
・Average difference between horizontal detection position and design position of distant marker on corrected image: dUif
・Average difference between horizontal detection position and design position of nearby marker on corrected image: dUin
・Average difference between vertical detection position and design position of distant marker on corrected image: dVif
・Average difference between vertical detection position and design position of distant marker on corrected image: dVin

The above processing performed on the reference image is also performed on the reference image.

Using the standard image and the reference image, the average of the calculated horizontal and vertical positional deviations, distance deviations, pitch angle deviations, yaw angle deviations, and roll angle deviations between the stereo camera 104 and the marker is calculated. These are assumed to be installation deviations of the stereo camera 104. Further, through the above calculation, the horizontal and vertical magnification ratios and offset deviations of the standard image and the reference image due to the windshield 112 are determined.

Step 404: The vehicle installation deviation detection unit 105 irradiates the vehicle 111 with a laser beam, and detects the distance to a plurality of locations on the surface of the vehicle 111 and the light receiving angle based on the reception time of the reflected light. Based on these distances and light receiving angles, the detection positions are converted to multiple locations on the surface of the vehicle 111.

The vehicle installation deviation detection unit 105 has surface shape data of the vehicle 111 and the position (design position) and attitude (design attitude) of the vehicle 111 when installed as designed, and based on these data, the vehicle The design positions of multiple locations on the surface of 111 are calculated.

The design position closest to the detected position at each location on the surface of the vehicle 111 is searched for, and the designed position is set as the corresponding point of the detected position. Based on the detected position (Xvm, Yvm, Zvm) and design position (Xvd, Yvd, Zvd) of each location on the surface of the vehicle 111, use equations (27) to (32) to calculate the positional deviation of the vehicle 111. (dXv, dYv, dLv) and posture deviation (dφv, dψv, dθv).

dXv = Σ(Xvd - Xvm) / Nm...(27)
dYv = Σ(Yvd - Yvm) / Nm...(28)
dLv = Σ(Zvd - Zvm) / Nm...(29)
dφv = - atan[Σ{(Zvm - Zo)(Yvd - Yvo) - (Yvm - Yvo)(Zvd - Zvo)} / Σ{(Yvm - Yvo)(Yvd - Yvo) + (Zvm - Zvo)(Zvd - Zvo)}] ...(30)
dψv = - atan[Σ{(Xvm - Xo)(Zvd - Zvo) - (Zvm - Zvo)(Xvd - Xvo)} / Σ{(Zvm - Zvo)(Zvd - Zvo) + (Xvm - Xvo)(Xvd - Xvo)}] ...(31)
dθv = - atan[Σ{(Yvm - Yo)(Xvd - Xvo) - (Xvm - Xvo)(Yvd - Yvo)} / Σ{(Xvm - Xvo)(Xvd - Xvo) + (Yvm - Yvo)(Yvd -Yvo)}] ...(32)
・Number of measurement points Nm
・Rotation center of vehicle 111 (Xvo, Yvo, Zvo)

The positional deviation and posture deviation of the vehicle 111 calculated as described above are added to the designed position and designed posture of the vehicle 111, and based on the surface shape data of the vehicle 111, a plurality of positions on the surface of the vehicle 111 are again calculated. Calculate the design position. Then, the design position closest to the detection position of each location on the surface of the vehicle 111 is searched, and the design position is set as the corresponding point of the detection position, and using equations (27) to (32), the vehicle The process of calculating the positional deviation and posture deviation in step 111 is repeated. If the value of the deviation calculated using equations (27) to (32) is smaller than a predetermined threshold, the process is terminated and the value obtained by adding those deviations up to that point is calculated as the positional deviation and posture deviation of the vehicle installation. do.

Step 405: Using equations (33) to (38), the camera installation deviation detection unit 334 calculates the camera installation position by subtracting the vehicle installation position deviation and attitude deviation from the camera installation position deviation and attitude deviation. Calculate the deviations (dXa, dYa, dLa) and posture deviations (dφa, dψa, dθa).

dXa = dXc - dXv...(33)
dYa = dYc - dYv...(34)
dZa = dZc - dZv...(35)
dφa = dφc - dφv ... (36)
dψa = dψc - dψv...(37)
dθa = dθc - dθv ... (38)

Step 406: The image correction data changing unit 323 uses the focal lengths of the imaging system unit 300a and the imaging system unit 300b, the pixel pitch, the optical axis position on the captured image, and the light on the corrected image stored in the calibration information storage unit 313. The axis position and the three-dimensional position of the marker relative to the stereo camera 104 when installed so that there is no deviation on the vehicle 111 are read. Based on the values of these parameters, the image correction data changing unit 323 determines the marker position (design position).

The image correction data changing unit 323 calculates the marker position on the corrected image when the windshield 112 is not passed, based on the installation positional deviation and posture deviation of the stereo camera 104 calculated by the image deviation factor detection unit 322. . The image correction data changing unit 323 creates camera installation deviation data representing image deviation due to camera installation deviation based on changes in marker positions on these two corrected images.

Next, the image correction data changing unit 323 changes from the state where there is a positional deviation and posture deviation of the stereo camera 104 installation calculated by the image deviation factor detection unit 322, there is no deviation only in the mounting direction of the camera with respect to the traveling direction of the vehicle 111. Calculate the marker position on the corrected image when (removed). Then, the image correction data changing unit 323 adjusts the camera installation based on the marker position on the corrected image when there is a deviation in the installation of the stereo camera and when the deviation in the installation direction of the camera with respect to the traveling direction of the vehicle 111 is eliminated. Calculate direction deviation data.

The image correction data changing unit 323 calculates the magnification ratio in the horizontal and vertical directions of the model of the windshield 112 calculated by the image deviation factor detection unit 322, and the glass deviation data when there is an offset deviation. The image correction data changing unit 323 uses two-dimensional linear interpolation as shown in step 402 to perform conversion based on camera installation direction deviation data, conversion based on glass deviation data, conversion based on glass deformation data, conversion based on camera installation deviation data, and conversion based on camera installation deviation data. Conversion using image correction data (representing distortion caused by the optical element) when not passing through the glass 112 is performed sequentially for each pixel.

Thereby, the image correction data changing unit 323 removes pixel deviation due to camera installation deviation, and calculates image correction data for the standard image and the reference image as seen through the windshield 112 when facing in the traveling direction of the vehicle 111. The image correction data changing unit 323 stores these image correction data in the image correction information storage unit 312.

The stereo camera 104 of this embodiment shown in FIG. 3 performs the operation shown in FIG. 8 while the vehicle 111 is running, and uses the created image correction data to detect the distance of a three-dimensional object. Detect camera installation misalignment and change image correction data.

Step 801: The synchronization signal generation section 314 generates a synchronization signal and sends it to the reference image capture section 315a and the reference image capture section 315b.

Immediately after receiving the synchronization signal from the synchronization signal generation section 314, the reference image capture section 315a sends the synchronization signal and exposure time information to the image sensor section 302a. Immediately after receiving the synchronization signal and exposure time information from the reference image capture unit 315a, the image sensor unit 302a receives an image of the light refracted by the optical element unit 301a for the exposure time, and adjusts the intensity of the light. A corresponding image is generated and sent to the reference image capture unit 315a. The reference image capture unit 315a receives an image from the image sensor unit 302a and stores the image in the captured image storage unit 311.

Immediately after receiving the synchronization signal from the synchronization signal generation section 314, the reference image capture section 315b sends the synchronization signal and exposure time information to the image sensor section 302b. Immediately after receiving the synchronization signal and exposure time information from the reference image capture unit 315b, the image sensor unit 302b receives an image of the light refracted by the optical element unit 301b for the exposure time, and adjusts the intensity of the light. A corresponding image is generated and sent to the reference image capture unit 315b. The reference image capture unit 315b receives an image from the image sensor unit 302b and stores the image in the captured image storage unit 311.

Step 802: The image correction unit 331 reads the captured standard image and reference image from the captured image storage unit 311. The image correction unit 331 reads image correction data of the standard image and the reference image from the image correction information storage unit 312.

Using equations (39) and (40), calculate the coordinates (Fx (X4, Y4), Fy (X4, Y4)) on the image after image capture corresponding to each pixel on the corrected reference image. Next, the position (X1, Y1) of the reference image after imaging corresponding to the pixel (X4, Y4) of the reference image after correction is calculated.

X1=Fx(X4, Y4)...(39)
Y1=Fy(X4, Y4)...(40)

By performing two-dimensional linear interpolation on the luminance values of four pixels surrounding the position (X1, Y1) of the reference image, the luminance value of the pixel (X4, Y4) of the reference image after correction is calculated. The above procedure is performed for each pixel of the corrected reference image to calculate the brightness value of the corrected reference image. The above procedure is also performed for the reference image to calculate the brightness value of the corrected image of the reference image. The corrected standard image and reference image are stored in the corrected image storage unit 316.

Step 803: The parallax calculation unit 332 reads the corrected images of the standard image and the reference image from the corrected image storage unit 316. As shown in FIG. 9, the parallax calculation unit 332 extracts a region 903 (hereinafter referred to as a reference region) of a predetermined size from the corrected reference image 901. The parallax calculation unit 332 searches for an image of an area in the corrected reference image 902 in which the same object as the reference area 903 is captured using the following pattern matching.

The parallax calculation unit 332 extracts an area 904 (reference area) of a predetermined size of the reference image 902 that is at the same height as the reference image 903, and calculates the difference between the brightness value of the standard area 903 and the brightness value of the reference area 904. The sum of absolute values (SAD, Sum of Absolute Difference) is calculated. The parallax calculation unit 332 calculates the SAD for each reference area 904 on the reference image 902 that is at the same height as the reference area 903, and searches for the reference area 905 with the smallest SAD value. The parallax calculation unit 332 performs isometric straight line fitting using the SAD of the reference area 905 and the SAD of the reference areas adjacent to the left and right of one pixel from the reference area 905, and selects a reference area on the reference image that most matches the reference area 903. 905 sub-pixels are calculated. The parallax calculation unit 332 calculates the parallax of the reference area 903 on the corrected reference image 901 by adding subpixels to the difference in position between the reference area 903 and the reference area 905.

The parallax calculation unit 332 performs such processing on all regions on the reference image 901 after image correction, and calculates the parallax of the entire reference image 901. The parallax calculation unit 332 stores the parallax image calculated in this manner in the parallax image storage unit 317.

Step 804: The recognition unit 333 reads the parallax image from the parallax image storage unit 317.

The recognition unit 333 calculates the distance L in the optical axis direction from the stereo camera 104 in the region on the parallax image using equation (41). Here, f is the design value of the focal length of the imaging system section 300a and the imaging system section 300b, B is the distance (baseline length) between the principal points of the imaging system section 300a and the imaging system section 300b, d is the parallax, and c is the imaging system section. This is the pixel pitch of the element section 302a and the image sensor section 302b.

L=f×B/(d×c) (41)

The recognition unit 333 performs this process on all regions of the parallax image, calculates the distance in the optical axis direction from the stereo camera 104 in the entire parallax image, and creates a distance image.

The recognition unit 333 uses equations (42) to (44) to calculate the three-dimensional position in the area on the distance image. Here, the three-dimensional coordinate system has an origin at a point on the road surface vertically dropped from the principal point of the optical element section 301a of the imaging system section 300a of the stereo camera 104, an X coordinate in the right direction, and a Y coordinate in the upward direction. Take the Z coordinate in the direction of travel. Moreover, (U, V) is each position on the distance image, (U0, V0) is the optical axis position on the reference image, and H is the mounting height of the stereo camera 104 from the road surface.

X=L×c×(U-U0)/f...(42)
Y=H+L×c×(V-V0)/f...(43)
Z=L...(44)

The recognition unit 333 performs this process on all regions of the parallax image and calculates a three-dimensional position for the entire parallax image.

The recognition unit 333 detects adjacent areas on the distance image where the distance ratio is within the processing threshold, the area of the area is greater than or equal to the threshold, and the height of these areas from the road surface (Y coordinate ) is greater than or equal to the threshold, it is estimated that the object is a three-dimensional object.

Step 805: The camera mounting misalignment detection unit 334 reads the corrected reference images created in the previous and current processing from the corrected image storage unit 316. The camera mounting misalignment detection unit 334 creates edge images in the horizontal direction and the vertical direction for the reference images after the previous and current corrections. The camera mounting misalignment detection unit 334 detects a region where both a vertical edge and a horizontal edge exist within a region of a predetermined size on these edge images. The camera mounting misalignment detection unit 334 uses pattern matching processing to search for an area on the previously corrected reference image that matches this area.

When the camera mounting misalignment detection unit 334 detects a matching edge on the previous and current corrected images, it detects the previous and current edge positions and stores the detected previous edge position and the current edge position in the misalignment correction information storage unit 318. Add the current edge position to the matching edge positions and store it. If there is no matching edge on the previous and current corrected images, the camera mounting deviation detection unit 334 newly stores the current edge position in the deviation correction information storage unit 318. In this way, the camera mounting misalignment detection unit 334 stores the time-series positions of the edges.

When the number of time-series data at a certain edge position is equal to or greater than a predetermined threshold, the camera mounting misalignment detection unit 334 determines an approximate straight line connecting the time-series data of edge coincidences. The camera mounting misalignment detection unit 334 performs this process for each region including both vertical edges and horizontal edges. The camera mounting misalignment detection unit 334 determines the positions of intersections on the image for each combination of these approximate straight lines, and averages the positions of these intersections.

If the average value of the positions of the intersection points is within a threshold value from the designed vanishing point position, the camera installation deviation detection unit 334 determines that the vehicle is traveling straight, and sets the average value of the positions of the intersections as the vanishing point position. , the number of vanishing point position data is increased by one and stored in the shift correction information storage unit 318. If the average value of the positions of the intersection points is a distance greater than or equal to the threshold value from the designed vanishing point position, the camera installation deviation detection unit 334 determines that the vehicle is turning, and the vanishing point position data is not detected correctly. , and discard it.

When the number of vanishing point position data stored in the displacement correction information storage section 318 is greater than or equal to a predetermined threshold, the camera mounting displacement detection section 334 detects the vanishing point position data stored in the displacement correction information storage section 318. The detected value of the vanishing point position is calculated on average, and the difference between the detected value of the vanishing point position and the design value is taken as the image shift in the horizontal and vertical directions due to camera mounting misalignment.

The camera mounting misalignment detection unit 334 performs the above processing on the reference image. The average of image shifts in the horizontal and vertical directions due to camera mounting misalignment in the standard image and the reference image is calculated and stored in the shift correction information storage unit 318.

Step 806: The camera mounting misalignment detection section 334 reads the image correction data of the reference image from the image correction information storage section 312 and the image shift in the horizontal and vertical directions due to the camera mounting misalignment from the misalignment correction information storage section 318. Using two-dimensional linear interpolation as shown in step 402, the camera mounting misalignment detection unit 334 performs conversion of the reference image using image correction data, and conversion for correcting image shifts in the horizontal and vertical directions due to camera mounting misalignment, to each pixel. Perform each step sequentially. Thereby, the camera mounting misalignment detection unit 334 calculates image correction data that removes the camera mounting misalignment.

The camera mounting misalignment detection unit 334 performs the above processing on the reference image. The camera mounting misalignment detection unit 334 stores the image correction data of the standard image and reference image calculated in this way in the image correction information storage unit 312.

When the stereo camera 104 is attached to the vehicle 111, if there is a misalignment in the vehicle installation or the stereo camera installation, it cannot be distinguished from image shift due to the windshield 112. Then, the image deviation due to the vehicle installation deviation or the stereo camera installation deviation is included in the image deviation due to the windshield 112, and an error occurs in the changed image correction data. Then, image shifts occur in the horizontal and vertical directions between the corrected reference image and the reference image of the object located at a different distance from the marker. Then, due to this vertical image shift, the regions of the standard image and the reference image do not match in the pattern matching process, making it impossible to correctly calculate the parallax. Further, even if the vertical shift is corrected, a parallax error occurs due to the horizontal image shift.

According to this embodiment, based on the distance and positional deviation on the image of markers at different distances, the magnification ratio and offset deviation of the glass model of the vehicle 111 that does not depend on the distance, the position between the marker and the camera at the distance, and the Detect posture deviation. Then, assuming that there is a vehicle installation deviation and a camera installation deviation in the camera position, data for correcting the image is changed so as to correct the image deviation due to the windshield 112. With this, it is possible to eliminate image deviations due to vehicle installation deviations and camera installation deviations, and to accurately measure distances, with just one photographing.

According to the present embodiment shown in FIG. 3, while the vehicle 111 is traveling, in step 805 and step 806 of the operation procedure shown in FIG. ) is detected. Then, based on the difference in the designed positions of the vanishing points, the camera mounting deviation detection unit 334 changes the image correction data so as to correct the horizontal and vertical image deviations due to the camera mounting deviation. This makes it possible to correct correction deviations due to misalignment or deformation of the stereo camera 104 that occurs over time.

Note that the processing device and vehicle-mounted camera device of the present invention are not limited to the above-described embodiments, and can be modified and applied in various ways. Modifications of the embodiments of the processing device and vehicle-mounted camera device of the present invention will be described below.

(Modification 1-1) (Embodiment in which image calibration processing is performed by the calculation unit)
FIGS. 1 and 10 show the configuration of an embodiment of a processing device and a vehicle-mounted camera device of the present invention. This modification example removes the calibration information storage unit 313 and the image calibration unit 320 from the stereo camera 104 in the embodiment of the processing device and in-vehicle camera device shown in FIGS. 1 and 3, and newly adds the calculation unit 106 ( 1000) is provided with a calibration information storage unit 313 and an image calibration unit 320.

The calculation unit 106 (1000) is connected to the stereo camera 104 and the vehicle installation deviation detection unit 105. The operations of each part are similar to those of the embodiment of the processing device and the vehicle-mounted camera device shown in FIGS. 1 and 3. In this way, by removing the calibration information storage unit 313 and the image calibration unit 320 from the stereo camera 104, the executable file that performs image calibration processing can be removed from the stereo camera 104, and the amount of memory used within the stereo camera 104 can be reduced. . Furthermore, the newly added calculation unit 106 (1000) is installed exclusively for the process of attaching to the vehicle 111, and the calculation processing capacity of the calculation unit 106 (1000) is faster than that of the stereo camera 104. Therefore, the time required for image calibration processing can be shortened, and the number of devices that can calibrate images per hour can be increased.

(Modification 1-2) (Detection of camera installation deviation only)
In this modification, the vehicle installation deviation detection unit 105 is deleted from the embodiment of the processing device and vehicle-mounted camera device of the present invention shown in FIGS. 1 and 3. Further, step 404 and step 405 in the operation procedure shown in FIG. 4 are not performed. Further, in step 406, conversion is performed using camera installation direction deviation data, glass deviation data, glass deformation data, camera installation deviation data, and image correction data when the windshield 112 is not passed through (distortion due to optical elements). Instead of sequentially performing conversion based on glass displacement data, glass deformation data, camera installation deviation data, and image correction data when not passing through the windshield 112 (representing distortion caused by optical elements) for each pixel, ) is performed sequentially for each pixel.

Thereby, even without providing the vehicle installation deviation detection unit 105, correction errors due to installation deviation of the stereo camera 104 due to vehicle installation deviation can be removed from the image correction data. In this modification, an error occurs in the vanishing point position on the image due to the misalignment of the stereo camera, but in steps 805 and 806 of the operation procedure shown in FIG. 8, the misalignment of the camera is detected and the resulting image shift is corrected. By calculating the image correction data, it is possible to correct image deviation due to misalignment of the stereo camera.

(Modification 1-3) (Embodiment that detects the position of the marker center)
In this modification, in the embodiments of the processing device and in-vehicle camera device of the present invention shown in FIGS. 1 and 3, the calibration information storage unit 313 does not store each marker image on the corrected image. In addition, in step 402, instead of performing pattern matching processing using the marker image and detecting the feature point position, pixels having a predetermined brightness value or less are set to 1 in the standard image and reference image after image correction, and Binarization processing is performed to set other pixels to 0, and the position of the marker is detected by calculating the position of the center of gravity of each marker on the binarized image. Thereby, the work of creating and storing marker images in advance can be reduced.

(Modification 1-4) (Image calibration device for monocular camera)
In this modification, the stereo camera 104 shown in FIG. 1 is changed to a monocular camera 104 in the embodiment of the processing device and in-vehicle camera device of the present invention shown in FIGS. 1 and 3, and the stereo camera 104 shown in FIG. The system is equipped with a monocular camera shown in FIG. 11, performs the operation procedure shown in FIG. 12 instead of the operation procedure during running shown in FIG. 8, and in the operation procedures shown in FIGS. 4 and 12, each part is Instead of processing images in the imaging system section 300a and imaging system section 300b, processing on images in the imaging system section 1000 is performed.

FIGS. 1 and 11 show the configuration of an embodiment of the image proofing apparatus of the present invention. Here, the corrected image storage section 316 and the distance detection section 330, which are displayed in gray, are not included in this embodiment, and operate in the case of detecting the distance of a three-dimensional object. These operations will be described later.

The monocular camera is attached to the vehicle 111 and photographs the markers 201 to 205 through the windshield 112 of the vehicle 111. Further, the monocular camera includes an imaging system section 1000 and a calculation section 210, and is connected to the vehicle installation deviation detection section 105.

An imaging system section 1000 such as a camera includes an optical element section 1001 and an image sensor section 1002. An optical element section 1001 such as a lens refracts light and forms an image on an image sensor section 1002. An image sensor unit 1002 such as an image sensor receives an image of light refracted by the optical element unit 1001, and generates an image according to the intensity of the light.

The calculation unit 310, such as a CPU (central processing unit) and memory, includes a captured image storage unit 311, an image correction information storage unit 312, a calibration information storage unit 313, a corrected image storage unit 316, and a deviation correction storage unit 318. , an image capture section 1015, an image proofreading section 320, and a distance detection section 1030.

The distance detection section 1030 includes an image correction section 331 , a recognition section 1032 , a three-dimensional object distance detection section 1033 , and a camera attachment displacement detection section 334 .

A captured image storage section 311, an image correction information storage section 312, a calibration information storage section 313, a corrected image storage section 316, and a deviation correction storage section 318, such as a memory, store information in place of the information of the imaging system section 300a and the imaging system section 300b. Information about the imaging system section 1000 is stored.

The image capture unit 1015 captures the image generated by the image sensor unit 1002 and stores the image in the captured image storage unit 311 .

The marker position detection section 321, the image shift factor detection section 322, the image correction data change section 323, the image correction section 331, and the camera mounting misalignment detection section 334 perform processing related to the images of the imaging system section 300a and the imaging system section 300b. , performs similar processing on the image of the imaging system unit 1000.

The recognition unit 1032 calculates the feature amount of the vehicle 111 in each region in the image corrected by the image correction unit 331, and detects a region that matches the feature of the vehicle 111. Furthermore, the recognition unit 1032 calculates the feature amount of the pedestrian in each region in the corrected image, and detects a region that matches the feature of the pedestrian.

The three-dimensional object distance detection unit 1033 calculates the distance between the vehicle 111 and the pedestrian based on the positions of the lower part of the area of the vehicle 111 and the area of the pedestrian, which are in contact with the road surface.

This modified example shown in FIGS. 1 and 11 operates according to the operating procedure shown in FIG. 4. Here, in step 401, the following processing is performed, and the marker position detection section 321, image shift factor detection section 322, and image correction data change section 323 perform processing regarding the images of the imaging system section 300a and the imaging system section 300b. Instead, by performing processing on the image of the imaging system unit 1000, the camera installation deviation, vehicle installation deviation, camera installation deviation, and horizontal and vertical magnification ratios of the windshield 112 are detected.

Step 401: The image capture unit 1015 sends the image capture signal and exposure time information to the image sensor unit 1002. Immediately after receiving the imaging signal and exposure time information from the image capture unit 1015, the image sensor unit 1002 receives an image of the light refracted by the optical element unit 1001 for the exposure time, and then adjusts the image according to the intensity of the light. The generated image is sent to the image capture unit 1015. The image capture unit 1015 receives an image from the image sensor unit 1002 and stores the image in the captured image storage unit 311.

The embodiment of the monocular camera of this modification shown in FIGS. 1 and 11 performs the operation procedure shown in FIG. 12 instead of the operation procedure shown in FIG. Detect the deviation and change the image correction data. Here, in step 801, the same process as step 401 (described above) of modification 1-4, which does not use the synchronization signal generator, is performed. Furthermore, in step 802 and steps 805 to 806, similar processing is performed on the image of the imaging system section 1000 instead of processing on the image of the imaging system section 300a and the imaging system section 300b, so explanations of those steps are omitted. The explanation of only steps 1103 and 1104 will be shown below.

Step 1103: The recognition unit 1032 reads the corrected image stored in the corrected image storage unit 316. The feature amount of the vehicle 111 is calculated in each region in the corrected image 1201, and a region 1202 that matches the feature of the vehicle 111 is detected as shown in FIG. Furthermore, the recognition unit 1032 calculates the characteristic amount of the pedestrian in each region in the corrected image 1201, and detects the region 1203 that matches the characteristic of the pedestrian.

Step 1104: The three-dimensional object distance detection unit 1034 determines the distance between the vehicle 111 and the pedestrian based on the position of the lower part of the area 1202 of the vehicle 111 and the area of the pedestrian 1203 as the position in contact with the road surface. Calculate.

When installing a monocular camera on the vehicle 111, if there is a misalignment in the vehicle installation or monocular camera installation, it cannot be distinguished from an image shift caused by the windshield 112. An error occurs in the changed image correction data that is included in the deviation, and horizontal and vertical image deviation occurs in the corrected image of the target at a different distance from the marker. Due to the shift, an error occurs in those distances.

In Modified Example 1-4, based on the distance and positional shift on the image of markers at different distances, the magnification rate and offset shift of the model of the windshield 112 of the vehicle 111 that does not depend on distance, and the distance between the marker and the camera that exist at different distances are calculated. Detect position and orientation deviations. Then, by changing the image correction data to calibrate the image shift caused by the windshield 112, assuming that there is a vehicle installation misalignment and a camera installation misalignment in the camera position, the vehicle installation misalignment and camera installation misalignment can be corrected with just one shooting. Image deviations caused by camera installation misalignment can be removed, and distances can be measured accurately.

Note that the monocular camera of Modification 1-4 is not limited to the embodiment described above, and can be modified and applied in various ways, as shown in Modifications 1-1 to 1-3. You can get the same effect.

Although the embodiments of the processing device and the in-vehicle camera device according to the present disclosure have been described above in detail using the drawings, the specific configuration is not limited to this embodiment and is within the scope of the gist of the present disclosure. Even if there are design changes, etc., they are included in the present disclosure.

DESCRIPTION OF SYMBOLS 101... Marker, 102... Marker, 103... Marker, 104... Stereo camera, 105... Vehicle installation deviation detection part, 106... Calculation part, 111... Vehicle, 112... Windshield, 121... Predetermined vehicle position, 122... Predetermined Attitude (direction) of the vehicle, 123... Attitude (direction) of the vehicle, 124... Attitude (direction) of the stereo camera, 201... Marker, 202... Marker, 203... Marker, 204... Marker, 205... Marker, 206... Chart , 207...chart, 208...chart, 209...chart, 210...chart, 300b...imaging system section, 301a...optical element section, 301b...optical element section, 302a...imaging element section, 302b...imaging element section, 310...operation 311... Captured image storage section, 312... Image correction information storage section, 313... Calibration information storage section, 314... Synchronization signal generation section, 315a... Reference image capture section, 315b... Reference image capture section, 316... Correction Image storage section, 317... Parallax image storage section, 318... Misalignment correction information storage section, 320... Image proofing section, 321... Marker position detection section, 322... Image misalignment factor detection section, 323... Image correction data changing section, 330... Distance detection unit, 331... Image correction unit, 332... Parallax calculation unit, 333... Recognition unit, 334... Camera installation deviation detection unit, 501... Corrected image, 502... Marker detection position, 503... Marker design position, 504... Marker detection position, marker design position, 601...Optical axis direction, 602...Horizontal line perpendicular to the optical axis direction, 603...Horizontal line perpendicular to the optical axis direction, 701...Marker, 702...Marker, 703...Optical axis direction, 704...Vertical line perpendicular to the optical axis direction, 705...Vertical line perpendicular to the optical axis direction, 901...Reference image after correction, 902...Reference image after correction, 903... Reference area, 904...Reference area, 905...Reference area that most matches the reference area, 1000...Calculation section, 1000...Imaging system section, 1001...Optical element section, 1002...Imaging element section, Image acquisition section...1015, 1030... Distance detection unit, 1032... Recognition unit, 1033... Three-dimensional object distance detection unit, 1201... Corrected image, 1202... Vehicle, 1203... Pedestrian

Claims (6)

Obtain the installation deviation of the in-vehicle camera from images taken of multiple markers at different distances from the in-vehicle camera and modeled vehicle window data, and use the installation deviation to correct the image of the in-vehicle camera. A processing device for changing correction data for
The processing device, wherein the installation deviation is a positional deviation with respect to a position where the in-vehicle camera should be placed with respect to a chart where the marker is present. The markers are arranged at different distances from the in-vehicle camera in the vertical direction,
The processing device obtains the installation deviation based on the interval and distance deviation of the markers,
The processing device according to claim 1, wherein the installation deviation is a pitch angle deviation of the vehicle. The markers are arranged at different distances from the in-vehicle camera in the left-right direction,
The processing device obtains the installation deviation based on the interval and distance deviation of the markers,
The processing device according to claim 1, wherein the installation deviation is a yaw angle deviation of the vehicle. 2. The processing device according to claim 1, wherein the processing device obtains the horizontal and vertical magnification factors of the window based on the horizontal and vertical spacing deviations of the markers on the image. comprising a vehicle installation deviation detection unit that detects installation deviation of the vehicle;
Based on the installation deviation of the vehicle and the installation deviation of the in-vehicle camera, calculate the installation deviation of the in-vehicle camera with respect to the vehicle, and use the installation deviation of the in-vehicle camera and the installation deviation of the in-vehicle camera. The processing device according to claim 1, wherein the processing device changes correction data for correcting the image of the vehicle-mounted camera. An in-vehicle camera device comprising the processing device according to claim 1,
having a first camera and a second camera;
The processing device includes:
acquiring a first image of the marker taken by the first camera and a second image of the marker taken by the second camera;
Based on the marker interval deviation and positional deviation obtained from the first image and the second image, the magnification deviation, offset deviation, and common Obtain the vehicle installation deviation of
An in-vehicle camera device that changes correction data for the first camera and the second camera based on the magnification shift, the offset shift, and the common vehicle installation shift.

Images