JP2013195783A - Position shift detection device, vehicle, and position shift detection method - Google Patents

Abstract

PROBLEM TO BE SOLVED: To provide a detection method and a detection device that can detect whether an on-board camera shifts in fitting position relative to a vehicle due to secular change or a vibration shock.SOLUTION: A camera unit 10 includes a feature point extraction section 20 which extracts feature point coordinates included in an image photographed through a transmission body and associated with a featured pattern, and a position shift determination section 24 which determines whether the camera unit shifts in position relative to the transmission body on the basis of the relation between reference feature point information extracted by the feature point extraction section 20 from the image photographed in the absence of a position shift and comparative feature point information extracted from an image photographed in a state to be determined.

Description

  The present invention relates to an on-vehicle camera positional deviation detection device, a vehicle, and a positional deviation detection method.

In recent years, driving assistance systems and parking assistance systems for automobiles have attracted attention, and monocular cameras and stereo cameras are used for this.
As an example of such a driving support system, there is a risk that the front of the vehicle is monitored by a mounted camera (hereinafter referred to as an in-vehicle camera), the distance from a human, another vehicle, an obstacle, etc. is detected and collided with them. There are known ASV (Advanced Safety Vehicle) devices that notify the driver, operate the vehicle brake to stop it, or decelerate it.
In such a driving support system, when the vehicle-mounted camera is attached to the vehicle body, a high level of accuracy is required with respect to the attachment position.
This is because if the mounting position of the in-vehicle camera is deviated, the imaging direction of the in-vehicle camera is deviated from that, and driving assistance based on accurate distance measurement may not be possible.

FIG. 19 is a diagram illustrating the positional relationship between the vehicle and the in-vehicle camera and the imaging direction.
FIG. 19A shows a case where the direction 101 in which the vehicle 100 travels straightly matches the optical axis direction 102 of the in-vehicle camera.
In the case of FIG. 19A, when the human M exists in the center with respect to the traveling direction 101 of the vehicle 100, it can be determined that the vehicle M will collide if it moves forward.
On the other hand, FIG. 19B shows a state in which the optical axis direction 102 of the camera does not coincide with the direction 101 in which the vehicle 100 travels straight and has rotated. In this case, even if the human M is in the center with respect to the traveling direction of the vehicle, the human M appears on the right on the camera image, and thus driving assistance cannot be performed accurately.
Even if it is attached in the state shown in FIG. 19A at the time of shipment, it is necessary to detect and correct the deviation when the attachment position is shifted over time as shown in FIG. 19B.

When the in-vehicle camera is a stereo camera, the shift in the imaging direction directly affects the calculated distance to the object.
Furthermore, in a stereo camera as a distance measuring device, even if the entire camera is not misaligned with respect to the vehicle, even if the relative positional relationship between the cameras constituting the stereo camera deviates, it is recognized. Accuracy will be reduced.
Of course, a shift in the mounting position of the camera may cause a reduction in recognition accuracy itself.
As an example of a decrease in recognition accuracy, there may be a case where the search area for each target is limited in order to improve the recognition processing speed. In such a case, if the camera mounting position is shifted. There are cases where areas in the captured image such as roadways, sidewalks, signals, signs, guardrails, etc. are shifted, and the target object cannot be searched.
As described above, the mounting position of the in-vehicle camera with respect to the vehicle is very important, and it is necessary to accurately adjust at the time of manufacture / shipment or subsequent maintenance.

However, during daily use, it is difficult for the user to determine whether the mounting position of the in-vehicle camera is shifted due to aging or impact and adjustment is necessary.
Therefore, as a method for determining whether or not the adjustment of the in-vehicle camera is necessary, for example, Patent Document 1 discloses the position of the in-vehicle camera attached to the vehicle body from the captured image of the inspection chart installed at a predetermined position. A method for checking whether adjustment is necessary is disclosed.
In addition, in the case of a stereo camera, Patent Document 2 discloses a process for calculating a corresponding positional shift between images of a stereo camera for determining whether or not it is necessary to adjust a relative positional shift between cameras constituting the stereo camera. Disclosed is a method of detecting and correcting whether or not the relative positional deviation has occurred due to the variance value of the luminance change amount of the candidate area.

However, in the method disclosed in Patent Document 1, it is necessary to detect misalignment using an inspection chart installed at a predetermined position, and the use of the vehicle is stopped and the misalignment is detected at a predetermined location. Not only does this impair the convenience of the user of the vehicle, but also the problem that the camera cannot be detected when the camera mounting position shifts due to time or vibration shock is shifted during traveling.
Further, in the method disclosed in Patent Document 2, it is possible to detect a relative position shift between cameras constituting a stereo camera from an image of a running landscape or the like. There is a problem that the mounting position shift of the entire vehicle-mounted camera device cannot be detected.
In addition to these methods, a method of detecting a camera mounting position shift with a characteristic marker installed on the vehicle within the imaging range is also conceivable, but it is possible to secure a sufficient distance from the marker to the camera. Since this is not possible, there is a problem in that the marker is not in focus and the amount of mounting displacement cannot be detected with high accuracy.
In the present invention, in view of the above-described problems, when a positional deviation (including a relative positional deviation between the cameras of the stereo camera) occurs due to time or vibration shock, or when a positional deviation occurs. It is an object of the present invention to provide a detection method and a detection apparatus that can detect the amount of deviation while using a vehicle without using a special inspection chart or the like.

  In order to solve the above-described problem, the invention according to claim 1 is a camera unit that acquires an image through a transmissive object having a distortion that reduces the straightness of incident light and forms an image on an image sensor. Based on the amount of change in the image formation position of the image formed on the image sensor surface, a displacement of the camera unit relative to the transmission object is detected.

  Since it is configured as described above, according to the present invention, without the inspection chart or marker installed at a predetermined position, the relative position shift between the cameras due to aging or vibration impact and the position of the camera mounting position with respect to the vehicle can be determined. A shift can be detected.

The figure which shows the structure of the camera apparatus which concerns on 1st Embodiment. 5 is a flowchart for explaining misregistration detection processing according to the first embodiment. The figure explaining the position shift detection process concerning 1st Embodiment in detail. The figure explaining in detail the method of calculating the candidate value of a relative position shift in a position shift amount calculation part. The figure which shows the histogram of the candidate data of the deviation | shift state from a positional deviation amount calculation part. The figure which shows the area suitable for distinguishing a deviation | shift state. The figure which shows the amount of position shift obtained by weighting the histogram shown in FIG. 6 is a flowchart for describing processing for acquiring an initial value of distortion data in the first embodiment. The figure which showed the arrangement | positioning relationship of a camera, a test | inspection chart, and a distortion transmission object at the time of acquiring the initial value of distortion data. The figure explaining the acquisition method of initial data. The figure which showed the state of the pixel with and without a distortion | transmittance transmission object, when the test | inspection chart which makes a feature point a linear form is set ahead, and it image | photographed with the pinhole camera. The figure which shows the structure of the stereo camera apparatus which concerns on 2nd Embodiment. 9 is a flowchart for describing processing for acquiring an initial value of distortion data in the second embodiment. The figure which showed the arrangement | positioning relationship between a stereo camera unit, an inspection chart, and a distortion transmission object at the time of acquiring the initial data of distortion data in 2nd Embodiment. The figure which showed the arrangement | positioning relationship between a stereo camera unit, an inspection chart, and a distortion transmission object at the time of acquiring the initial data of distortion data in 2nd Embodiment. The figure which shows the displacement of the stereo camera with respect to a vehicle. The flowchart explaining the position shift determination process by the position shift determination part in 2nd Embodiment. The schematic diagram which shows the vehicle which can apply the distance measuring device of this embodiment. The figure explaining the positional relationship and imaging direction of a vehicle and a vehicle-mounted camera.

Embodiments of the present invention will be described below in detail with reference to the drawings.
This embodiment uses the distortion of a strain transmitting object (such as a front or rear window glass) provided in a vehicle for the purpose of preventing rain and wind or preventing obstacles flying while the driver or passenger is traveling. This detects the displacement of the mounting position of the in-vehicle camera.
Specifically, by measuring the distortion of a strain-transmitting object at the time of shipment when the on-vehicle camera is mounted and adjusted accurately on the vehicle, the distortion data is compared with the distortion data obtained from the image captured during travel. Then, the positional deviation is detected, and the positional deviation amount is calculated.
Specifically, the strain-transmitting object provided in the vehicle does not cause a positional shift with respect to the vehicle over time, and the strain-transmitting object is not a parallel plate and is usually formed in a curved surface shape having different curvatures in the vertical direction and the horizontal direction. In addition, a position shift position of the in-vehicle camera with respect to the vehicle is detected by utilizing the existence of distortion due to manufacturing variation.
Further, the present invention is not limited to the displacement of the camera mounting position with respect to the “vehicle”. When an image is acquired by the camera, a change in the imaging position of the subject on the camera image plane occurs between the subject and the camera. This can be applied when there is an object with distortion that occurs.

[First Embodiment]
First, as a first embodiment, an attachment position deviation detection method and apparatus when a monocular camera is attached to a vehicle body as an in-vehicle camera will be described.
A pinhole camera is taken as an example of a monocular camera, and a camera with a windshield in front is used as a distorted transmissive object that reduces the straightness of incident light.
When a linear object is imaged by a pinhole camera, the image is usually captured as a straight line.
However, if there is a distorted transparent object such as a windshield between the camera and the linear object, the straight line is not captured even if it is captured.
For example, the image formation position on the camera and the sensor changes by about 0.003 mm between the case where a general windshield is present and the case where there is no general windshield between the camera and the linear object. It has been found that the amount of change varies with the windshield and camera position.
The change in the image formation position of 0.003 mm corresponds to 0.5 pixel in the case of a sensor having a pixel pitch of 6 μm. This change in the imaging position causes the distance between the chart 19.5 m ahead and a distance different by 1.3 m or more in the case of a stereo camera having a base length of 16 cm and a focal length of 8 mm.
In the present embodiment, the displacement of the mounting position of the in-vehicle camera is detected by using such a change in the imaging position caused by the distortion of the windshield.
Since the positional deviation of the in-vehicle camera is further amplified due to the distortion of the windshield, the positional deviation can be easily detected.

FIG. 1 is a diagram illustrating a configuration of a camera device according to the first embodiment.
The camera device shown in FIG. 1 includes a camera unit 10 including a lens and an image sensor (sensor), and a control unit 11. The control unit 11 receives a sensor image and matches the use of each camera. An image processing unit 12 that performs image processing, a recognition unit 13 that performs recognition for each purpose such as parking assistance, distance measurement, and white line detection, and a positional deviation detection unit 14 that performs positional deviation detection of the camera unit 10. A display / alarm control unit 15 that receives a recognition result by the unit 13 and a detection result by the positional deviation detection unit 14 and notifies the driver of the input result on a display (not shown) or by sound.
Various modes are conceivable, such as displaying the recognition result by the recognition unit 13 and the detection result by the misalignment detection unit 14 on a head-up display.

In addition, the recognition result and the detection result may be used as information when performing control such as brake control and door lock, instead of displaying on a display or alarming.
The positional deviation detection unit 14 receives a sensor output image from the camera unit 10 and outputs a feature point in the image as coordinates on the sensor, and a feature point extracted by the feature point extraction unit 20 The position shift amount calculation unit 22 that calculates the relative position shift amount candidate value between the windshield and the camera unit 10 as input, and the relative position shift amount candidate value obtained by the position shift amount calculation unit 22 as input. A position shift determination unit 24 that determines whether or not the mounting position of the unit 10 is shifted from the windshield and outputs the information to the display / alarm control unit 15.
Further, the misregistration detection unit 14 shown in FIG. 1 includes memories (ROM) 21 and 23 that store parameters and data used for the processing of the feature point extraction unit 20, the misregistration amount calculation unit 22, and the misregistration determination unit 24, respectively. However, these memories may be combined with a single memory shared by the processing units.
Further, part or all of the processing by the control unit 11 may be executed by hardware such as ASIC, or may be executed by software processing by a CPU (Central Processing Unit) included in the control unit 11.

Next, each processing unit in the misregistration detection unit 14 will be described.
FIG. 2 is a flowchart for explaining misalignment detection processing according to the first embodiment.
FIG. 3 is a diagram for explaining in detail the misregistration detection process according to the first embodiment.
The misregistration detection process according to the first embodiment will be described with reference to FIGS.
First, in step S51 of FIG. 2, the feature point extraction unit 20 acquires a plurality of frames of landscape images A including a linear object B as shown in FIG.
Next, in step S52, the landscape image (input image) A is binarized to obtain an image A ′ as shown in FIG.
Next, in step S53, the coordinates (pi, qi) at the edge of the linear object (subject) B ′ and the background are extracted as feature point coordinates (comparison feature point information) from the image of FIG. To do.
Then, the feature point extraction unit 20 outputs the extracted feature points to the positional deviation amount calculation unit 22 as coordinates on the sensor (imaging device).
The input image A may be taken while the vehicle is running or may be taken while the vehicle is stopped.
In the case of using an image taken while traveling, a straight line portion of a subject having linearity (for example, a signal pole or a crosswalk pattern, a white line, a vehicle edge, a building edge, or the like) is detected from surrounding scenery. Feature points can be detected by combining edge extraction and Hough transform.

In addition, dictionary data for recognizing a subject on which feature points are to be extracted may be stored in advance in the memory 21 or the like.
In addition, as a situation where an image obtained during a stop is used as an input image, for example, a periodic inspection such as an automobile inspection can be considered.
In this case, it is conceivable to arrange a chart in which feature points are arranged in a straight line within the imaging field angle range of the camera, acquire an image, and detect these feature points.
Whether to detect misalignment during traveling, whether to check for misalignment in periodic inspections, etc., decides what feature points should be extracted according to the purpose of use, and stores them in the memory 21 Decide what information you want.
In addition, since it is very unlikely that the linear object B is captured in only one frame, when the feature point of the landscape image (the edge of the linear object B) can be extracted in only one frame, What is necessary is just not to perform the output of the feature point coordinate from the feature point extraction unit 20 to the positional deviation amount calculation unit 22 described later.

Next, in step S54, the positional deviation amount calculation unit 22 receives the feature point coordinates extracted by the feature point extraction unit 20, and outputs a candidate value for the relative positional deviation amount between the windshield and the camera.
For this purpose, reference feature point coordinate data (xi, yi) is stored in the memory 23 as initial shipment data (described later).
The reference feature point data is coordinates on the inspection chart including a positional deviation caused only by the distortion of the windshield when a linear inspection chart is photographed with no positional deviation in the camera unit.
When a straight line in the sensor vertical direction is used as the feature point coordinates, the positional deviation amount calculation unit 22 calculates Σ for each feature point coordinate (pi, qi) and each reference feature point coordinate (xi, yi) described above. (Xi-pi) is calculated, and the one with the smallest difference in the x-coordinate for each frame (FIG. 3C) is set as a candidate value for the amount of relative displacement between the windshield and the camera calculated from the image ( FIG. 3 (d)).
In step S55, if there is a relative positional deviation amount candidate exceeding the threshold value from among these, it is determined that a positional deviation has occurred between the windshield and the camera (step S56). If not, it is determined that there is no displacement (step S57).
In addition, when a change in distortion amount due to temperature is obtained, temperature information is output together with the above-described deviation amount.

FIG. 4 is a diagram for explaining in detail a method for calculating a candidate value for relative displacement in the displacement amount calculation unit.
In the landscape image shown in FIG. 4A, a pixel area of (p _max −p _min ) × (q _max −q _min ) that includes all the point groups of the feature point coordinates Σ (xi−pi) is set ( FIG. 4 (b)).
As the initial data, the range of the pixel area described above is cut out from the plurality of shift state data stored in the memory 23 (shift state 1 and shift state 2 in FIG. 4C).
Then, within the cut out range, Σ (xi−pi) is performed on a plurality of shift state data at the y coordinate of the landscape image shown in FIG. 4A, and a minimum one is obtained from a certain landscape image. It is set as the shifted state.
Not only the displacement state but also the area information used for the determination is output to the subsequent position displacement determination unit 24. In the position deviation determination unit 24, the relative position deviation amount candidate value obtained by the position deviation amount calculation unit 22 is input, and determination information indicating that the mounting position of the camera unit 10 is deviated from the windshield is displayed / alarm controlled. To the unit 15.
The positional deviation determination unit 24 creates a histogram from the input relative positional deviation amount candidate values with the positional deviation amounts of the windshield and the camera unit 10 as the horizontal axis.
Here, since feature point extraction errors and the like are included, as described with reference to FIG. 3D, when a threshold that can be determined is set and the threshold is exceeded, it is determined that the positional deviation between the windshield and the camera has occurred. To do.

FIG. 5 is a diagram showing a histogram of deviation state candidate data obtained by the processing of FIG.
In FIG. 5, the horizontal axis indicates a shift state (shift amount), and the vertical axis indicates the frequency.
This histogram is not necessarily data like a normal distribution.
As can be seen from the comparison between the shift state 1 and the shift state 2 in FIG. 4C, there are a portion where the straight line distortion caused by the windshield is similar and a portion where the windshield is not similar depending on the image area. Because.
In FIG. 4C, the similar part is the screen center area, and the dissimilar part is the screen both end area.
When the linear object extracted from the landscape image shown in FIG. 4A concentrates on a specific area of the screen, even if the displacement state is completely different, the two states will appear as displacement state candidates. .

Therefore, as shown in FIG. 5, a threshold value is provided for the frequency of the histogram, and weighting is performed on a screen area exceeding a certain frequency.
When the frequency of a certain deviation state exceeds the weighted position, attention is paid to the frequency and the near future state.
In FIG. 5, the shift states 1, 2, and 3 are the shift amounts to be noticed.
The initial data of the shift state is compared, and an area suitable for distinguishing the shift states 1, 2, and 3 is searched (areas indicated by frames 40 and 41 in FIG. 6).
It is also possible to obtain the difference between the x-coordinates of each shift state with the same y-coordinate, and use the largest area as the feature area.
The feature areas of all the deviation states may be calculated in advance, but the area position changes depending on whether the deviation states are compared with each other, and there are many states that are initially stored. An example of searching for an area after narrowing down is given above.
If there is a feature point of a landscape image in the feature area, a histogram is created with a weight more than other areas.
In the histogram shown in FIG. 7, a difference is generated even in the shift states 1, 2, and 3 by weighting.
In the histogram of FIG. 7, if the deviation amount exceeds a threshold that can be determined and the difference from other deviation amount candidates is equal to or larger than a reference value, the deviation amount can be determined at that time.

FIG. 8 is a flowchart for describing processing for acquiring an initial value (reference feature coordinate data) of distortion data in the first embodiment.
(1) Obtaining an image of an inspection chart (step S101)
FIG. 9 is a diagram showing an arrangement relationship between the camera, the inspection chart, and the strain transmitting object when initial data (initial value of strain data) is acquired. FIG. 10 is a diagram illustrating a method for acquiring initial data.
In addition, the camera unit 10 is a pinhole camera without distortion, and is in a state where adjustment of attachment to the vehicle is finished.
In the present embodiment, a deviation from the ideal initial mounting position is detected.
Although FIG. 9 shows an example in which the strain transmission object 50 is installed in front of the camera unit 10, there may be a distortion transmission object other than the front.
For example, it suffices if there is a distorted transmission object that does not change its position with respect to the vehicle over time on the optical path until a light beam from an object to be imaged (landscape or human) enters the camera optical system.
As the inspection chart 55, it is necessary to prepare a linear (linear) pattern.
When the pattern of the inspection chart 55 has linearity, it is not necessary to install the inspection chart 55 and the vehicle 100 or the positions of the inspection chart 55 and the camera unit 10 with high accuracy.
A white and black straight edge image (pattern) as shown in FIG. 10A is displayed on the display for displaying the inspection chart, and the edge position of the image is moved by one pixel at a time. It is desirable to obtain distortion image data of the straight line data. The inspection chart acquires as much data as possible by rotating not only the vertical straight line but also horizontally.
Moreover, the temperature of the measurement environment may be changed to obtain an image of the inspection chart at each temperature.

(2) Binarize the image (step S102)
A straight subject (inspection chart) is distorted and imaged by the strain transmitting object 50 as shown in FIG. In order to obtain the distortion coordinates, the sensor image is binarized (FIG. 10C).
(3) Extract (reference) feature point data (xi, yi) from the binarized image (step S103)
Feature point data (reference feature points) is acquired from the binarized image as in the example of FIG.
In FIG. 10, a straight line in the vertical direction is taken as an example, but the displacement can be detected with higher accuracy by obtaining as many types of inclination as possible in the approximate straight line.
In a region where a linear feature point is obtained during traveling, it is preferable that many types of inclination of the approximate straight line can be obtained in the same sensor region.
Since the inspection chart 55 does not particularly define the position with respect to the camera unit 10, there is no ideal coordinate on which the feature point is imaged, but it is not necessary to have the ideal coordinate because it is desired to detect a deviation from the initial position. .

FIG. 11 is a diagram showing the state of a pixel when a distortion transmission object is present and when there is a distortion transmission object when an inspection chart with a feature point having a linear shape is set in front and photographed by a pinhole camera. It is.
The positions of the camera unit 10 and the inspection chart 55 are not moved.
In the state without the strain transmission object 50, the linear object is imaged linearly even on the sensor coordinates, but when there is a distortion transmission object, the image is not linear.
The strain transmission object 50 is usually formed in a curved surface shape having different curvatures in the vertical direction and the horizontal direction, and there is a manufacturing variation, and the angles of the light rays entering and exiting the distortion transmission object are slightly different. .
Therefore, the angle of the light ray incident on the camera varies depending on the presence or absence of the strain transmitting object 50, and the image forming position on the camera sensor changes.

Since the imaging position change amount changes depending on which region of the windshield the light beam passes, if the relative position between the vehicle and the strain transmitting object 50 does not shift with time, the change in the imaging position change amount It can be determined whether or not the relative positional relationship between the strain transmitting object 50 and the camera unit 10 has shifted, and finally the change in the relative positional relationship between the vehicle and the camera can be determined.
(4) The relative positional relationship between the distorted transmission object and the camera and the value of the reference feature point data (xi, yi) are stored (step S104).
The feature point data (xi, yi) obtained in step S103 and the relative positional relationship between the strain transmitting object 50 and the camera unit 10 at that time are stored in a memory (memory 21 in FIG. 1) or the like.
As data to be stored as a relative positional relationship between the strain transmitting object 50 and the camera unit 10, the initial state after adjusting the position by installing the camera on the vehicle body is set to zero, and the adjusted installation position of the camera relative to the vehicle (absolute Value) need not be calculated accurately.

Depending on the present embodiment, the imaging position shift on the camera image plane caused by distortion of a distortion transmission object (front or rear window glass, etc.) installed in front of the camera is used. Can be realized.
That is, if it can be detected that the camera mounting position has shifted with respect to the vehicle during driving, for example, in a camera that performs brake control or steering wheel control based on the recognition result, the reliability of the recognition result may be reduced by the driver due to the mounting position shift. It can warn that it has fallen, and can release brake control and steering wheel control.
In addition, if the amount of camera mounting position deviation relative to the vehicle can be calculated, the camera mounting position can be corrected by a mounting mechanism that can finely adjust the camera mounting position (or replacement of mounting parts), and the mounting position can be adjusted again. The camera user can be warned of the determination that the camera must be repaired because it has changed from the initial shipping state.

[Second Embodiment]
The first embodiment described above relates to a monocular camera, but the second embodiment described below is a mounting position shift amount of a stereo camera for measuring a distance to a subject and a camera. The present invention relates to an apparatus for calculating a shift amount of a relative positional relationship between each other.
As the strain transmission object, a windshield can be used as in the first embodiment.
Here, the stereo camera device is a device in which a plurality of cameras are arranged at positions apart from the base line length, and a distance to the subject is calculated by detecting a difference in image formation position (parallax) between the cameras.
When the stereo camera device is attached to the vehicle, it is necessary to detect an attachment position shift of the entire stereo camera device and a relative position shift between the cameras over time.
If the mounting position shift in the entire stereo camera occurs, the distance to the subject will change, and when used for brake control, steering wheel control, etc., the accuracy of the calculated distance due to the mounting position shift will cause a serious accident. It becomes.
In addition, if a relative position shift occurs between the cameras that make up the stereo camera device (particularly the relative position shift of the camera that changes the parallax), it will be a major factor in the deterioration of distance accuracy, so early detection of the shift is important. It is.
When ranging with high accuracy using a stereo camera, it is necessary to perform correlation calculation after correcting distortion caused by lens and manufacturing variations.

FIG. 12 is a diagram illustrating a configuration of a stereo camera device according to the second embodiment.
The stereo camera device shown in FIG. 12 includes a stereo camera unit 60 including left and right camera units 60L and 60R each including a lens and an image sensor (sensor), and a control unit 60a.
The control unit 60a receives a positional deviation detection unit 61, sensor images of both camera units, a pre-processing unit 62 that performs distortion correction processing of each camera, a correlation calculation unit 64 described later, parking assistance, distance measurement, A recognition unit 66 that performs recognition in accordance with each purpose such as white line detection, a recognition result by the recognition unit 66 and a detection result by the misalignment detection unit 61 are input, and these input results are displayed or sounded on a display (not shown). And a display / alarm control unit 75 for informing the driver.

  The misregistration detection unit 61 receives a sensor output image from the camera units 60L and 60R included in the stereo camera unit 60, and outputs a feature point extraction unit 69 that outputs a feature point in the image as coordinates on the sensor. A positional deviation amount calculation unit 71 for calculating a relative positional deviation amount candidate value between the windshield and each of the camera units 60L and 60R by using the feature point coordinates extracted by the feature point extraction unit 69, and a positional deviation amount calculation unit 71 Is used as an input to determine whether or not the mounting position of the stereo camera unit 60 (camera units 60L, 60R) has shifted with respect to the windshield, and display the information / alarm control unit 15 is provided.

・・・（式１）
式１において、δｘ、δｙはあるレンズのある点のｘ方向、ｙ方向の歪み量であり、ｘ，ｙは理想正規化座標、ｆ１，ｆ２…,ｆｋ、ｇ１，ｇ２…，ｇｋはある温度における歪み係数である。
これらの歪み係数を求めるためにｘ，ｙ，δｘ，δｙの値を既知データとし、画像全体でこの歪み式を満たすような歪み係数の最適化を行うなどが考えられる。これらの歪み係数もしくはＬＵＴデータはメモリ６３に予め記憶されている事が望ましい。 As a distortion correction processing method performed by the pre-processing unit 62, an LUT (lookup table) having conversion destination information before and after correction may be used, or a distortion polynomial may be used.
Since a large storage capacity is required in the LUT, it is preferable that the distortion is expressed by a polynomial expression. As the distortion correction formula, for example, a high-order polynomial shown in (Formula 1) can be considered.

... (Formula 1)
In Equation 1, δx and δy are the amounts of distortion in the x and y directions at a certain point of a lens, x and y are ideal normalized coordinates, f1, f2... Fk, g1, g2. Is the distortion coefficient.
In order to obtain these distortion coefficients, the values of x, y, δx, δy are known data, and optimization of the distortion coefficients so as to satisfy the distortion equation in the entire image can be considered. These distortion coefficients or LUT data are preferably stored in the memory 63 in advance.

Further, the distortion recognized as the distortion of the lens constituting the camera unit or the distortion of the windshield may be included.
As a method for obtaining the distortion polynomial including the distortion of the windshield, for example, adjustment is performed after the camera is mounted on the vehicle, and then the inspection chart is installed at the specified installation position, and the actual coordinates of the feature points in the inspection chart And a method for obtaining a distortion polynomial from the relationship of ideal coordinates.

In this embodiment, it is assumed that distortion correction including distortion of the windshield is performed after the camera mounting adjustment.
In addition to the distortion coefficient, the memory 63 stores information such as the focal length of the camera used for the distortion correction process and the pixel pitch.
The image data whose distortion has been corrected by the preprocessing unit 62 is output to the feature point extraction unit 69 and the correlation calculation unit 64 in the subsequent stage.
Since the feature point extraction unit 69 calculates the relative positional deviation amount between the cameras that perform the processing described in the first embodiment and the mounting position deviation of the entire camera, the feature point extraction unit 69 uses the image data of the two cameras. Input to 69.

When the stereo camera unit 60 has a mechanism in which the two camera units 60L and 60R are displaced as a unit, and only the mounting position shift in the entire camera is detected without detecting the relative position shift between the two cameras. Only the image data of one of the cameras needs to be input to the feature point extraction unit 69.
Since the process of detecting the mounting position shift of the entire stereo camera unit 60 is the same as the method described in the first embodiment, the case of detecting the relative position shift amount between cameras will be described.
The positional deviation amount calculation unit 71 has the same configuration as the positional deviation amount calculation unit 22 of the first embodiment (FIG. 1).
The memory 72 stores initial data acquired when the stereo camera unit 60 is attached and adjusted.

Here, an initial data acquisition method according to the second embodiment will be described.
FIG. 13 is a flowchart for describing processing for acquiring an initial value of distortion data in the second embodiment.
(1) An image of an inspection chart is acquired (step S201).
FIG. 14 is a diagram illustrating an arrangement relationship among the stereo camera unit 60, the inspection chart 55, and the distortion transmission object 50 when acquiring initial distortion data in the second embodiment.
The stereo camera unit 60 is a camera in which distortion including distortion of the windshield (distorted transmission object) 50 is corrected.
As the inspection chart 55, a linear (linear) pattern is prepared. However, since the distortion correction including the distortion of the windshield 50 is performed, the mounting position is not shifted. The straight line of the inspection chart 55 is imaged as a straight line.

Here, an image is acquired by the two camera units 60L and 60R constituting the stereo camera unit 60.
Further, for example, as shown in FIG. 15, a light emitter 56 may be used instead of the inspection chart 55.
When the illuminant 56 is used, the incident angle with respect to the camera is changed, and the angle is formed on the image plane of the pinhole camera so that a straight line is formed when there is no distorted transmission object. In addition, the light emitted from the light emitter needs to be unexpanded light.
(2) Binarize the image (step S202)
This is the same as in the first embodiment.
(3) Feature point data (xi, yi) is extracted from the binarized image (step S203).
This is also the same as in the first embodiment.

(4) Store the relative positional relationship between the strain transmission object and each camera unit and the value of the feature point data (xi, yi) (step S204).
This is basically the same as in the first embodiment.
When the initial position deviation does not occur, the feature point data is linear, but when the relative position between a stereo camera unit 60 (described later) and the distortion transmission object 50 added to calculate the deviation amount is changed. The optical path of the light beam from the subject changes.
(5) The relative positional relationship between the distortion transmission object and the stereo camera unit is changed (step S205).
This step is necessary not only for determining whether the installation position of the stereo camera unit 60 has deviated from the initial state, but also for determining the amount of deviation.
The position of the stereo camera unit 60 may be changed in accordance with a range where it is estimated that the mounting position can be shifted to the maximum with time, or when there is a correction mechanism.
When the relative position between the stereo camera unit 60 and the distorted transmission object (the windshield 50) is changed, the region where the light rays from the subject (chart 55) pass through the distorted transmission object changes, so that the feature point data (xi, The value of yi) changes.

When changing the relative position, the strain transmitting object 50 may be moved or the stereo camera unit 60 may be moved. However, assuming that the strain transmitting object is attached to the vehicle body, the translation is performed on the camera mounting portion in the vehicle. If a mechanism is provided or a rotation mechanism for rotating each camera unit in the stereo camera unit is provided, it is considered that handling is easy.
Alternatively, some types of attachment parts may be prepared in advance in the camera attachment unit, and data when the parts are changed may be acquired.
Changes in the relative positional relationship include x, y, z axis translation and x, y, z axis rotation when the camera position after camera attachment adjustment is zero, and the relative relationship between the plurality of camera units (60L, 60R). The position of the entire stereo camera unit with respect to the distortion transmission object is changed without changing the positional relationship (FIG. 16A), or the data acquisition method, or the relative position of each camera unit and the position of the entire stereo camera unit are both changed. (FIG. 16B) A data acquisition method is conceivable.
The misregistration determination unit 73 includes feature point data (xi, yi) of the camera 60R and the camera 60L input from the misregistration amount calculation unit 71, and the camera unit 60R and the camera unit 60L stored in the memory 74. Compare with the initial data, search for similar data and detect the amount of deviation.

In the following, the misregistration determination based on this misalignment amount will be described.
FIG. 17 is a flowchart for explaining misregistration determination processing by the misregistration determination unit in the second embodiment.
In step S300, the displacement determination unit determines whether the feature point data of the camera units 60L and 60R matches the initial value data (step S300).
Next, when both coincide (Yes in step S300), it is determined that the stereo camera unit as a whole is not displaced (step S301).
If the two do not match (No in step S300), it is checked whether similar data exists for both camera units (step S302).
If similar data exists for both (Yes in step S302) and the relative positional deviation amount candidates of the respective camera units match (Yes in step S303), it is determined that the entire stereo camera unit is displaced. (Step S304).

If they do not match (No in step S303), it is determined that the camera units are displaced from each other (step S305).
If it is determined in step 302 that there is no similar data for both camera units (No in step S302), and it is determined that there is similar data for one camera unit (Yes in step S306), one of the data that is not similar to each other is determined. It is determined that only the camera unit is displaced (step S307).
Finally, if it is determined in step S306 that there is no similar data in both camera units, it is determined that the imaging position displacement is due to other factors (step S308).

The above determination process will be described in detail.
(1) When the camera unit 60R and the camera unit 60L are not displaced at all with respect to the windshield 50, the data of the camera unit 60R and the camera unit 60L output from the displacement amount calculation unit 71 is stored in the memory 74. Is substantially the same as the initial shipping data stored in, and the difference in the x coordinate is substantially zero. In this case, it is determined that the two camera units are not displaced with respect to the windshield 50, and it is determined that distance measurement may be continued as it is.
(2) When the entire stereo camera unit is displaced with respect to the windshield Data similar to the data of the camera unit 60R and the camera unit 60L output from the displacement amount calculation unit 71 exists in the memory 74, and When the relative position shift amount candidate values of the camera units 60L and 60R and the windshield 50 match, the position shift determination unit 73 determines that the entire stereo camera unit 60 has shifted from the windshield 50. .
Since the attitude of the camera unit 60 with respect to the windshield (with respect to the vehicle) is known, if the stereo camera unit has a positional deviation within the allowable correction range and a correction mechanism is provided, the position relative to the windshield is moved by the amount of deviation. Alternatively, if the camera posture is greatly deviated, it is determined that the camera is out of the allowable correction range, and an error message indicating that the position adjustment of the stereo camera unit is necessary is output to the display or the head-up display.

(3) When a part of the camera units constituting the stereo camera unit is misaligned from the position at the time of shipment Data of the camera unit that is not misaligned and output from the misalignment amount calculation unit 71 Is substantially the same as the initial shipping data stored in the memory 74, but when there is no data similar to the data of the camera unit that has caused the position shift, the position shift determination unit 73 sets the second position to the windshield 50. One of the cameras is determined not to be displaced from the time of shipment, and one camera is determined to be displaced.
In other words, among the camera unit data output from the positional deviation amount calculation unit 71, the positional deviation from the shipping position has occurred for the camera unit corresponding to the data that substantially matches the initial factory data. For the camera unit corresponding to the data that does not match the initial shipping position (there is no corresponding data), it is determined that a positional deviation from the shipping position has occurred.
In this case, if a mechanism for correcting the relative position of the camera is provided, correction is performed, and if not provided, the display / alarm control unit 75 notifies the driver that adjustment is necessary.

(4) When at least one of the two cameras has an imaging position change that does not have a relative positional relationship from the time of shipment, the memory for all the data of the two camera units output from the positional deviation amount calculation unit 71 If there is no similar data in the data stored in 74, this is not a positional deviation within the expected positional deviation range, so the positional deviation determination unit 73 causes a change in the imaging position of the camera due to other factors. It is determined that
As the imaging position change that is not in the relative positional relationship, for example, a cause such as a change in lens distortion called an internal parameter with time can be considered.
In this case, the correlation calculation unit 64 performs correlation calculation of the image output from the preprocessing unit 62.
The memory 65 preferably stores the base line length, the number of pixels, and the search width information of each stereo camera unit 60L, 60R, and performs stereo matching according to the base line length, the number of pixels, the search width, etc. of the stereo camera, Perform correlation calculation. As the matching process, various known processes can be used.

For example, the SAD (Sum of Absolute Difference) method can be mentioned.
The SAD method calculates an integrated value of absolute values of differences in luminance values of pixels corresponding to each other between a template set in the reference image and a block in the comparison image to be correlated with the template. Thus, an evaluation value is obtained, and a block in the comparison image relating to the smallest evaluation value is used as a matching region.
Further, correction may be performed by image processing based on the positional deviation amount input from the positional deviation amount calculation unit 71.

The recognizing unit 66 receives the parallax image calculated and generated by the correlation calculating unit 64 as input, and recognizes the distance measuring object.
When recognizing an object to be measured, the features of the object to be measured may be stored in the memory as dictionary data in advance, and what kind of object is present with a group having the same parallax as one block Whether or not there is an object may be recognized. In the latter case, the higher the distance measurement accuracy, the higher the accuracy of recognition.
The display / alarm control unit 75 receives the recognition result from the recognition unit 66 and the determination result from the displacement determination unit 73.
Various methods are conceivable, such as displaying the determination result on a display (not shown), notifying the driver by sound, or displaying the determination result on a head-up display.
It is also conceivable that the display or alarm is not output as a recognition result by the device but is used as information when performing control such as brake control or door lock.

<Other embodiments>
FIG. 18 is a schematic diagram showing a vehicle to which the distance measuring device of the present embodiment can be applied.
The image processing system of the present embodiment performs processing such as calculating the distance to an imaging unit 81 for acquiring an image ahead of the vehicle and another vehicle existing in front of the vehicle 80 based on the acquired image. An image analysis unit 82 is included. The imaging unit 81 is installed at a position of a seat mirror or the like so that an image in front of the vehicle 80 traveling can be captured. An image in front of the vehicle imaged by the imaging unit 81 is converted into an image signal and input to the image analysis unit 82. The image analysis unit 82 analyzes the image signal output from the imaging unit 81.
As the imaging unit 81, the imaging units 10 and 60 of the above embodiment can be applied.
In addition, the image processing unit 12 and the recognition unit 13 of the above embodiment can be applied as a part of the functions of the image analysis unit 82.
In addition, the determination result by the position shift determination units 24 and 73 can be displayed on the head-up display 89.
The vehicle travel control unit 88 can also control the steering wheel and the brake based on the distance calculated by the image analysis unit 82.

  DESCRIPTION OF SYMBOLS 10 Camera unit 11 Control part 12 Image processing part 13 Recognition part 14 Detection part 15 Display / alarm control part 20 Feature point extraction part 22 Misalignment amount calculation part 24 Misalignment judgment part 26 Display / Alarm control unit, 26 quantity calculation unit, 31 parallax calculation unit, 33 CPU, 50 windshield, 55 inspection chart, 56 light emitter, 60 stereo camera unit, 60L, 60R camera unit, 60a control unit, 61 misregistration detection unit 62 Pre-processing unit, 64 correlation calculation unit, 66 recognition unit, 69 feature point extraction unit, 71 misregistration amount calculation unit, 73 misregistration determination unit 73, 75 display / alarm control unit

Japanese Patent No. 3479006 Japanese Patent No. 4285618

Claims (8)

  In a camera unit that acquires an image through a transmissive object having a distortion that reduces the straightness of incident light and forms an image on the image sensor, the amount of change in the image formation position of the image formed on the image sensor surface Based on this, a displacement detection device that detects a displacement of the camera unit relative to the transmission object. In the position shift detection device according to claim 1,
A feature point extraction unit that extracts feature point coordinates relating to a characteristic pattern included in an image captured by the camera unit through the transparent object;
Reference feature point information extracted by the feature point extraction unit from an image photographed in the absence of positional deviation, comparison feature point information extracted from an image photographed in a state to be determined,
Based on the relationship, and a displacement determination unit that determines the presence or absence of displacement of the camera unit with respect to the transmission object;
A misalignment detection apparatus comprising: In the position shift detection device according to claim 2,
A positional deviation amount calculation unit that compares the reference feature point information and the comparative feature point information to calculate a candidate value of a relative positional deviation amount between the camera unit and the transmission object;
The misregistration detection apparatus, wherein the misregistration determination unit determines presence / absence of misalignment of the camera unit from the frequency distribution of the candidate values. In the position shift detection device according to claim 2,
The camera unit is a compound eye camera unit having a plurality of camera units,
The feature point extraction unit extracts the comparison feature point information and the reference feature point information for each image acquired by each camera unit,
The positional deviation amount calculation unit is configured to calculate a relative positional deviation amount between the camera units or the entire compound-eye camera unit based on the similarity between the comparison feature point information and the reference feature point information regarding each camera unit. Calculating a candidate value of the amount of displacement with respect to the transparent object;
The position shift determination unit determines whether or not there is a position shift between the camera units or a position shift of the entire compound-eye camera unit based on the candidate value based on the candidate value. In the position shift detection device according to any one of claims 2 to 4,
The predetermined pattern is a linear pattern, and an image including the linear pattern is acquired by photographing the illuminant so that the illuminant emitting non-spreading light is imaged as a straight line. An apparatus for detecting misalignment. The misregistration detection apparatus according to claim 3,
The reference feature point information is weighted according to a region in the image.   A vehicle comprising: a camera unit; and a camera unit misalignment detection device according to any one of claims 1 to 6. A feature point extracting step of extracting feature point coordinates relating to a characteristic pattern included in an image captured by the camera unit through the transparent object;
Based on the relationship between the reference feature point information extracted by the feature point extraction step from the image photographed in the absence of misalignment and the comparison feature point information extracted from the image photographed in the determination target state A displacement step for determining the presence or absence of displacement of the camera unit with respect to the transmission object;
A misregistration detection method comprising:

Images