JP2018125706A - Imaging apparatus - Google Patents

Abstract

PROBLEM TO BE SOLVED: To determine whether or not it is required to correct absolute attitude of multiple imaging parts in an imaging apparatus.SOLUTION: An imaging apparatus (1) includes multiple imaging parts (11a, 11b), detectors (23a, 23b) for detecting a heavenly body from a set of images captured by the multiple imaging parts, respectively, a calculation part (25) for calculating the relative posture angle among multiple captured parts based on the set of images, with the heavenly body detected by the detectors as an infinity point, a first determination part (25) for determining necessity of correction of the multiple imaging parts, by comparing the calculated relative posture angle with a first threshold, and a second determination part (25) for comparing the calculated relative posture angle with a second threshold, when a determination is made that correction of the multiple imaging parts is necessary by the first determination part, and determining whether correction of the multiple imaging parts is possible by changing the parameters related to the multiple imaging parts, and whether physical correction of the multiple imaging parts is necessary.SELECTED DRAWING: Figure 1

Description

  The present invention relates to a technical field of an imaging apparatus having a plurality of cameras.

  As this type of apparatus, for example, an area where an object at infinity is projected from one set of images taken by each of a plurality of cameras is extracted as an infinity area, and the corresponding set of images is used. A pair of points is detected as a corresponding point, a corresponding point included in the infinity region is detected as a corresponding point at infinity, and a corresponding point not included in the infinity region is detected as a general corresponding point, and an infinite corresponding point is used. Identify the relative posture between the multiple cameras, identify the relative position between the multiple cameras using the general corresponding point and the identified relative posture, and use the relative posture and the relative position to determine the self-position between the cameras. An apparatus for performing calibration has been proposed (see Patent Document 1).

  A stereo that detects smear from two image data obtained by imaging the sun with the first and second cameras, and calculates camera parameters so that the smear positions of the two image data substantially coincide. A camera correction method has been proposed (see Patent Document 2). In addition, the position of a known object that is an object related to pattern information is detected from an image represented by an input image signal of multiple viewpoints, and a significant parallax cannot be detected between the multiple viewpoints from each image represented by the input image signal of multiple viewpoints. Based on the relationship information indicating the position of the existing object and the correspondence between the viewpoints of the multi-lens infinity region. An apparatus for calculating a positional parameter indicating a positional relationship among a plurality of viewpoints has been proposed (see Patent Document 3).

International Publication No. 2013/145025 JP 2006-279239 A JP 2013-257244 A

  When an imaging apparatus having a plurality of cameras is mounted on a moving body such as a vehicle, for example, the relative posture between the plurality of cameras is caused by, for example, time lapse, vibration or impact due to movement of the moving body, and the like. There can be relatively large changes. When the relative posture changes relatively large, the absolute posture of each camera may also change relatively large. In this case, self-calibration as described in Patent Document 1 cannot be handled.

  The present invention has been made in view of the above problems, and an object of the present invention is to provide an imaging apparatus that can determine whether or not calibration of the absolute posture of the camera is necessary.

  In order to solve the above problems, an imaging apparatus of the present invention is detected by a plurality of imaging units, a detection unit that detects a celestial body from a set of images captured by each of the plurality of imaging units, and the detection unit. A calculation unit that calculates a relative attitude angle between the plurality of imaging units based on the set of images with the celestial body as an infinite point, and compares the calculated relative attitude angle with a first threshold value. A first determination unit that determines the necessity of calibration of the plurality of imaging units; and when the first determination unit determines that calibration of the plurality of imaging units is necessary, the calculated relative attitude angle; It is possible to calibrate the plurality of imaging units by changing a parameter relating to the plurality of imaging units by comparing with a second threshold value that is greater than the first threshold value, or to perform physical calibration of the plurality of imaging units. A second determination unit that determines whether it is necessary.

  In the imaging apparatus, an infinite point is set by detecting a celestial body from a set of captured images. For example, celestial bodies such as the sun and the moon are guaranteed to be infinite. For this reason, for example, map information required when setting an infinite point using buildings, such as a building, becomes unnecessary. In addition, for example, celestial bodies such as the sun and the moon have a relatively strong pattern (that is, for example, a relatively large difference in brightness). Can be improved. In particular, the appearance time, position, and shape (for example, fullness, orientation, size, etc.) of the sun and the moon can be obtained relatively easily.

  In the imaging apparatus, in particular, the second determination unit compares the relative attitude angle and the second threshold value, and can calibrate a plurality of imaging units by changing parameters related to the plurality of imaging units (that is, self-calibration). Whether or not physical calibration of a plurality of imaging units (that is, calibration of absolute posture) is necessary is determined. Therefore, according to the imaging apparatus, it is possible to determine whether or not calibration of the absolute posture of the imaging unit is necessary.

  The effect | action and other gain of this invention are clarified from the form for implementing demonstrated below.

It is a block diagram which shows the structure of the imaging device which concerns on embodiment. It is a flowchart which shows the calibration execution determination process which concerns on embodiment. It is an example of the figure for demonstrating storage of a relative position. It is an example of the figure for demonstrating the calculation method of a relative attitude | position.

  An embodiment according to an imaging apparatus of the present invention will be described with reference to FIGS. 1 to 3. In the following embodiments, “moon” is given as an example of the “celestial body” according to the present invention.

(Constitution)
The configuration of the imaging apparatus according to the embodiment will be described with reference to FIG. FIG. 1 is a block diagram illustrating a configuration of an imaging apparatus according to the embodiment.

  In FIG. 1, the imaging device 1 includes a stereo camera 10, an automatic calibration device 20, and a distance estimation unit 30. The imaging device 1 is mounted on a vehicle (not shown).

  The stereo camera 10 includes image sensors 11a and 11b and image generation units 12a and 12b. The image generation unit 12a generates an image from an analog electric signal output from the image sensor 11a. Similarly, the image generation unit 12b generates an image from an analog electrical signal output from the image sensor 11b.

  The distance estimation unit 30 estimates the distance to an object shown in the image (that is, captured by the stereo camera 10) based on the images generated by the image generation units 11a and 11b. Since the existing technique can be applied to the distance estimation method, a detailed description thereof will be omitted.

  Here, in order to accurately estimate the distance to the object (in other words, the position of the object), it is necessary to accurately identify the internal parameters and the external parameters related to the stereo camera 10. Identifying internal and external parameters is called “calibration” or “calibration”. Here, the internal parameters include, for example, a focal length, an image center position, lens distortion, and the like. The external parameter includes a relative attitude between a camera (not shown) having the image sensor 11a and a camera (not shown) having the image sensor 11b.

  During travel of a vehicle on which the imaging device 1 is mounted, an external parameter related to the stereo camera 10 may change due to a physical impact or vibration on the imaging device 1. The automatic calibration device 20 is configured to be able to recalibrate external parameters related to the stereo camera 10 using an image captured by the stereo camera 10 while the vehicle is traveling. In particular, the automatic calibration device 20 recalibrates the external parameters related to the stereo camera 10 using the moon shown in the captured image.

  In order to recalibrate the external parameters related to the stereo camera 10, the automatic calibration device 20 includes a vehicle orientation estimation unit 21 as a processing block that is logically realized therein or as a processing circuit that is physically realized. The celestial body appearance position estimation unit 22, the celestial body detection units 23 a and 23 b, the celestial body relative position storage unit 24, and the relative posture calculation unit 25 are configured.

  The vehicle azimuth estimation unit 21 estimates the azimuth of the vehicle on which the imaging device 1 is mounted, based on, for example, an output from a gyro sensor or an azimuth sensor and a signal from, for example, a GPS (Global Positioning System) satellite. In the present embodiment, it is assumed that the direction of the vehicle is the same as the direction in which the stereo camera 10 is facing. In addition, since the existing technique can be applied to the direction estimation method, a detailed description thereof is omitted.

  The celestial body appearance position estimation unit 22 acquires the current date and the current location of the vehicle on which the imaging device 1 is mounted, for example, based on a signal from a GPS satellite. The celestial body appearance position estimation unit 22 estimates the appearance position of the moon (for example, altitude, direction, size, fullness, etc.) based on the acquired current date and time and the current location of the vehicle. The celestial body appearance position estimation unit 22 generates a template image of the moon based on the estimated appearance position. In addition, since the existing technique is applicable to the estimation method of the appearance position of a moon, the description about the detail is abbreviate | omitted.

  The astronomical detectors 23a and 23b detect the moon from a set of images captured by the stereo camera 10 (that is, images simultaneously captured by the image sensors 11a and 11b). Specifically, the celestial body detection unit 23a detects the moon from the image generated by the image generation unit 12a by template matching using the template image of the moon generated by the celestial body appearance position estimation unit 22. Similarly, the celestial body detection unit 23b detects the moon from the image generated by the image generation unit 12b by template matching using the template image of the moon generated by the celestial body appearance position estimation unit 22.

  The astronomical relative position storage unit 24 uses a reference image as a reference image and a reference image as a reference image from a set of images captured by the stereo camera 10 when the moon is detected by the astronomical detectors 23a and 23b. The difference between the position of the moon and the position of the moon in the referenced image is stored as a month relative position (the month relative position will be described in detail in the “calibration execution determination process”).

  The relative posture calculation unit 25 calculates a relative posture as an external parameter related to the stereo camera 10 based on the moon relative position stored by the astronomical relative position storage unit 24 (refer to “Calibration execution determination process” for the relative posture calculation method). ”).

(Calibration execution judgment process)
The calibration execution determination process performed in the automatic calibration apparatus 20 configured as described above will be described with reference to the flowchart of FIG.

  In FIG. 2, first, the vehicle orientation estimating unit 21 estimates the orientation of the vehicle on which the imaging device 1 is mounted (step S101). Subsequently, the celestial body appearance position estimation unit 22 estimates the appearance position of the moon based on the current date and time and the current location of the vehicle (step S102).

  Next, the celestial body appearance position estimation unit 22 determines whether or not a moon appearance condition that is a condition for the moon to appear in the image captured by the stereo camera 10 is satisfied based on the moon appearance position and the vehicle orientation ( Step S103). Here, in the moon appearance condition, (i) the difference between the azimuth angle of the vehicle and a half value of the horizontal angle of view of the camera having the image sensor 11a or 11b is larger than the azimuth angle at which the moon appears. (Ii) The sum of the azimuth angle of the vehicle and the half value of the horizontal angle of view of the camera having the image sensor 11a or 11b is smaller than the azimuth angle at which the moon appears, (iii) the image sensor 11a or 11b The value obtained by inverting the sign of the half of the vertical angle of view of the camera having the larger than the appearance height of the moon, and (iv) 1/2 of the vertical view angle of the camera having the image sensor 11a or 11b The condition that the value is smaller than the appearance height of the month is included.

  If it is determined in step S103 that the moon appearance condition is not satisfied (that is, if at least one of the above conditions (i) to (iv) is not satisfied) (step S103: No), step The process of S101 is performed. On the other hand, when it is determined in step S103 that the moon appearance condition is satisfied (that is, when all of the above conditions (i) to (iv) are satisfied) (step S103: Yes), the astronomical object appearance position estimation unit 22 generates a template image of the moon based on the estimated appearance position. Subsequently, the astronomical detectors 23a and 23b detect the moon from a set of images captured by the stereo camera 10 using the template image of the moon (step S104).

The astronomical detectors 23a and 23b detect the month based on, for example, a cost value calculated by the following cost function. Here, “Cost _template ” and “I _template ” are “cost value using template image” and “template luminance value”, respectively. “I”, “j”, “a”, “b”, “N” and “M” are “image horizontal position”, “image vertical position”, “template image horizontal position”, “ “Vertical position of template image”, “Horizontal width of template image”, and “Vertical width of template image”.

One set of images captured by the stereo camera 10 is referred to as a reference image and the other as a referenced image. The position in the reference image of the moon detected in the process of step S104 is (i ₀ , j ₀ ).

  After the process of step S104, the astronomical relative position storage unit 24 sets a square image region including the moon in the reference image detected by the astronomical detector 23a or 23b as a template image of a new month. The astronomical object relative position storage unit 24 uses the template image of the new moon to scan a position that matches the template image of the new moon in the referenced image.

At this time, the astronomical relative position storage unit 24 scans the matching position based on the cost value calculated by the following cost function. Here, “Cost _relative ”, “I _reference ”, and “I _referenced ” are “cost value using reference image (that is, template image of new month)”, “reference image luminance”, and “referenced”, respectively. "Image brightness". “I”, “j”, “a”, “b”, “N”, and “M” are “horizontal position of the referenced image”, “vertical position of the referenced image”, and “new month”, respectively. “Horizontal position of template image”, “Vertical position of template image of new month”, “Horizontal width of template image of new month”, and “Vertical width of template image of new month”.

Let (i ₁ , j ₁ ) be the position where the cost value is minimized, that is, the position that matches the new month template image. Next, the astronomical object relative position storage unit 24 calculates a cost value (that is, (i ₁ -1, j ₁ ) calculated by shifting the template image of the new moon by 1 pixel to the left and right from (i ₁ , j ₁ ). ) And (i ₁ + 1, j ₁ ) cost values) are used to calculate the lateral position i _sub with subpixel accuracy from the following equation (3). Similarly, the astronomical object relative position storage unit 24 calculates a cost value (that is, (i ₁ , j ₁ −1)) by shifting the template image of the new moon by 1 pixel from the top and bottom of (i ₁ , j ₁ ). ) And (i ₁ , j ₁ +1) are used to calculate the vertical position j _sub with sub-pixel accuracy from the following equation (4).

Next, the astronomical relative position storage unit 24 stores (i ₀ , j ₀ , i ₁ + i _sub , j ₁ + j _sub ) as a set of data (that is, a month relative position) (step S105). Here, the astronomical object relative position storage unit 24 divides the image into four regions (i) to (iv) in advance, for example, as shown in FIG.
The astronomical relative position storage unit 24 _calculates the relative month position based on the value of “i ₀ ” of (i ₀ , j ₀ , i ₁ + i _sub , j ₁ + j _sub ) as the month relative position. ) To (iv), the month relative position is stored in association with each other (step S105).

Next, the relative posture calculation unit 25 determines whether or not the number of month relative positions stored by the astronomical relative position storage unit 24 is greater than a threshold value _Tnum (step S106). When the number of month relative positions associated with each of the areas (i) to (iv) is greater than the threshold value T _{num, the} relative posture calculation unit 25 determines that “the number of month relative positions is greater than the threshold value T _num ”. judge. In other words, if the number of month relative positions associated with a certain area among the areas (i) to (iv) is equal to or less than the threshold value T _num , the relative posture calculating unit 25 is associated with other areas. the number even if greater than the threshold value T _num of determines "number of months relative position is not greater than the threshold value T _num".

If it is determined in step S106 that the number of month relative positions is not greater than the threshold value _Tnum (step S106: No), the process of step S101 is performed. On the other hand, if it is determined in step S106 that the number of monthly relative positions is greater than the threshold value _Tnum (step S106: Yes), the relative attitude calculation unit 25 calculates the relative attitude as an external parameter related to the stereo camera 10. Calculate (step S107).

Specifically, the relative posture calculation unit 25 calculates a relative posture by setting an area where the moon of the reference image is captured as an infinite region (that is, a set of infinity points). The relative attitude calculation unit 25 obtains a plurality of (i ₀ , j ₁ + j _sub −j ₀ ) based on each of all the moon relative positions stored in the astronomical relative position storage unit 24. When the plurality of (i ₀ , j _sub + j ₁ −j ₀ ) are plotted on coordinates with the horizontal axis “i _o ” and the vertical axis “j ₁ + j _sub −j ₀ ”, for example, as shown in FIG. become.

The relative attitude calculation unit 25 obtains an approximate straight line “y = ax + b” using a least square method based on a plurality of (i ₀ , j _sub + j ₁ −j ₀ ). Here, “a” and “b” are expressed by the following equations. In the following equation, “N”, “x”, and “y” are respectively “the number of data used in the calculation”, “horizontal axis data value (ie, i ₀ )”, and “vertical axis data value (ie, j). _sub + j ₁ −j ₀ ) ”.

The relative posture calculation unit 25 obtains the vertical deviation amount Δj and the rotational deviation amount Δθ from the above-described formulas “a” and “b”.

The relative posture calculation unit 25 further obtains the lateral displacement amount Δi by calculating the average of the lateral position change in consideration of the rotational displacement amount Δθ, that is, the following equation. In the following formula, “H” is “vertical width of the reference image”.

Next, the relative posture calculation unit 25 obtains a pitch angle (Pitch), a roll angle (Roll), and a yaw angle (Yaw) as a relative posture by the following formula. In the following formula, “f” is a focal length.

Next, the relative posture calculation unit 25 determines whether or not the calculated relative posture satisfies the first condition (step S108). Here, the first condition includes (i) the pitch angle is less than the threshold value T _{p_small} , (ii) the roll angle is less than the threshold value T _{r_small} , and (iii) the yaw angle is less than the threshold value T _{y_small} , This condition is included. Note that “threshold value T _{p_small} ”, “threshold value T _{r_small} ”, and “threshold value T _{y_small} ” are examples of the “first threshold value” according to the present invention.

  When it is determined in step S108 that the relative posture satisfies the first condition (that is, when all of the above conditions (i) to (iii) are satisfied) (step S108: Yes), the automatic calibration device 20 ends the process without calibrating the stereo camera 10 (step S109).

In the determination of step S108, when it is determined that the relative posture does not satisfy the first condition (that is, when at least one of the conditions (i) to (iii) is not satisfied) (step S108: No), The relative posture calculation unit 25 determines whether or not the relative posture satisfies the second condition (step S110). Here, the second condition includes a condition that (i) the pitch angle is larger than the threshold value T _{p_large} , (ii) the roll angle is larger than the threshold value T _{r_large} , and (iii) the yaw angle is larger than the threshold value T _{y_large.} included.

However, “threshold value T _{p_large} > threshold value T _{p_small} ”, “threshold value T _{r_large} > threshold value T _{r_small} ” and “threshold value T _{p_large} > threshold value T _{y_small} ”. The “threshold value T _{p_large} ”, “threshold value T _{r_large} ”, and “threshold value T _{y_large} ” are examples of the “second threshold value” according to the present invention.

  When it is determined in step S110 that the relative posture satisfies the second condition (that is, when all of the above conditions (i) to (iii) are satisfied) (step S110: Yes), the relative posture calculator In 25, the dealer determines that the stereo camera 10 should be physically calibrated (step S111). In this case, the auto-calibration device 20 can physically send the stereo camera 10 to the user of the vehicle on which the imaging device 1 is mounted, for example, by sound or image, or by turning on or blinking a predetermined indicator. Notify that calibration is required.

  On the other hand, when it is determined in step S110 that the relative posture does not satisfy the second condition (that is, when at least one of the above conditions (i) to (iii) is not satisfied) (step S110: No) ) The automatic calibration device 20 calibrates the stereo camera 10 (step S112). It should be noted that since the existing technology can be applied to the calibration of the stereo camera 10, a detailed description thereof will be omitted.

(Technical effect)
When the relative posture between the camera having the image pickup device 11a and the camera having the image pickup device 11b changes relatively large, the absolute postures of the two cameras may also change relatively large. In this case, it is difficult to sufficiently improve the accuracy of the distance estimated by the distance estimation unit 30 only by calibrating the stereo camera 10 by the automatic calibration device 20.

  In the imaging apparatus 1, whether or not the relative posture satisfies the second condition, that is, is it possible to cope with the calibration of the stereo camera 10 by the automatic calibration apparatus 20, or is the physical calibration by the dealer of the stereo camera 10 necessary? , Is determined. That is, the imaging apparatus 1 can determine whether the stereo camera 10 needs physical (that is, absolute posture) calibration.

  The “imaging elements 11a and 11b” according to the embodiment are examples of the “imaging unit” according to the invention. The “celestial body detection units 23a and 23b” according to the embodiment is an example of the “detection unit” according to the present invention. The “relative attitude calculation unit 25” according to the embodiment is an example of the “calculation unit”, “first determination unit”, and “second determination unit” according to the present invention.

(Modification)
In addition to adding an impact sensor to the imaging apparatus 1, it is determined whether or not the impact detected by the impact sensor is greater than a predetermined value before the process of step S101 of the calibration execution determination process. In some cases, the dealer may determine that the stereo camera 10 should be physically calibrated. If the impact is not more than the predetermined value, the process of step S101 is performed.

  Further, the detection target of the calibration execution determination process may be the sun instead of or in addition to the moon.

  The present invention is not limited to the above-described embodiment, and can be appropriately changed without departing from the gist or concept of the invention that can be read from the claims and the entire specification. Moreover, it is included in the technical scope of the present invention.

  DESCRIPTION OF SYMBOLS 1 ... Imaging device, 10 ... Stereo camera, 11a, 11b ... Imaging device, 12a, 12b ... Image generation part, 20 ... Automatic calibration apparatus, 21 ... Vehicle direction estimation part, 22 ... Astronomical object appearance position estimation part, 23a, 23b ... Astronomy detection unit, 24 ... Astronomy relative position storage unit, 25 ... Relative posture calculation unit, 30 ... Distance estimation unit

Claims (1)

A plurality of imaging units;
A detection unit for detecting a celestial body from a set of images captured by each of the plurality of imaging units;
A calculation unit that calculates a relative posture angle between the plurality of imaging units based on the set of images, with the celestial body detected by the detection unit as a point at infinity;
A first determination unit that compares the calculated relative attitude angle with a first threshold value to determine the necessity of calibration of the plurality of imaging units;
When the first determination unit determines that calibration of the plurality of imaging units is necessary, the calculated relative attitude angle is compared with a second threshold value that is greater than the first threshold value, and the plurality of imaging units is compared. A second determination unit that determines whether the plurality of imaging units can be calibrated by changing a parameter according to the above, or whether physical calibration of the plurality of imaging units is necessary;
An imaging apparatus comprising: