JP2017040549A - Image processing device and error determination method - Google Patents

Abstract

PROBLEM TO BE SOLVED: To provide an image processing device which can detect the occurrence of an error in a value of a camera parameter regardless of whether or not an object in the known size is reflected on a photographed image.SOLUTION: An image processing device 100 includes: a deviation range calculation part 104 which calculates deviation ranges in the longitudinal direction and short side direction of two calibration images obtained by calibrating a left image and a right image on the basis of a camera parameter and a reference imaging distance; a deviation amount calculation part 105 which calculates a deviation amount in each direction of the two calibration images; and a determination part 106 which determines whether or not an error occurs in the camera parameter by comparing the deviation range with the deviation amount.SELECTED DRAWING: Figure 1

Description

  The present invention relates to an image processing apparatus that performs image processing for enabling highly accurate three-dimensional measurement, and a method for determining an error of a parameter referred to for three-dimensional measurement by such an image processing apparatus. About.

  An example of a system that measures the three-dimensional position of a subject is a three-dimensional measurement system that uses a stereo camera system that includes two cameras. Such a measurement system calculates parallax by detecting corresponding points between the two images taken through the two cameras, and captures the two cameras using the principle of triangulation. The three-dimensional position of the subject is calculated. In the case of a stereo camera system in which the two cameras modeled as a pinhole camera are arranged in parallel, the three-dimensional position of the subject can be calculated by the following equation (Equation 1).

  Here, X, Y, and Z are the X coordinate values of the position of the subject in the world coordinate system with the position where the reference camera (one of the two cameras) is set as the origin, Y coordinate values and Z coordinate values are shown. Further, x and y are projection coordinates on the image sensor of the reference camera, f is a focal length, B is an inter-camera baseline length, and D is an inter-camera parallax.

  However, in reality, the values of X, Y, and Z are not very accurate because of errors due to the effects of “installation error on the camera fixture, displacement of the mounting position of the image sensor inside the camera, lens distortion, etc.” Not a value. Therefore, it is common to perform three-dimensional measurement by correcting each parameter on the right side of the above equation in consideration of these influences and then performing an operation according to the above equation using each parameter after correction.

  In consideration of the error as described above, a relational expression between an arbitrary point (Xw, Yw, Zw) in the world coordinate system and an image coordinate (u, v) on which the arbitrary point is projected is, for example, It can be modeled as an equation (Equation 2).

Here, r _ij (i = 1, 2, 3, j = 1, 2, 3) is a rotation parameter indicating the rotation transformation between the world coordinate system and the camera coordinate system, and t _x , t _y , and t _z are This is a translation parameter between the world coordinate system and the camera coordinate system. These parameters are called external parameters.

K _i (i = 1, 2, 3) and p _i (i = 1, 2) are parameters relating to camera lens distortion, fx and fy are focal lengths of the two cameras, cx, cy is the principal point position of the two cameras, and these parameters are called internal parameters. Here, the principal point refers to a position moved from the focal point of the lens toward the lens by a focal length.

  Although these parameters are generally unknown, nonlinear solutions, etc. based on a plurality of correspondences between known world coordinate system points (Xw, Yw, Zw) and projected points (u, v) on the image Can be used to estimate these parameters. As an example of this estimation method, there is a method disclosed in Non-Patent Document 1. In this way, the process of estimating camera parameters (lens distortion parameters, internal parameters, external parameters) is called calibration. An image can be handled as a pinhole camera model by reducing an error caused by an influence such as an inherent shift related to the optical path of the camera by the lens distortion parameter and the internal parameter.

  In stereo cameras, the external parameters are parameters indicating the translational / rotational relationship between the common world coordinate system and each camera coordinate system. Therefore, based on the estimated external parameters, the external parameters are Translation / rotation parameters to the camera coordinate system can be calculated. Since the translational / rotational relationship between the cameras is known, it is possible to calibrate the image so that the optical axis direction is parallel.

  As described above, the two images are calibrated and parallelized, and the parallax between the two images is calculated, so that the three-dimensional measurement based on Equation 1 can be performed. Further, since the three-dimensional measurement by the stereo camera as described above is performed based on the calibrated / parallelized image, the measurement accuracy of the three-dimensional measurement changes depending on the estimation accuracy of the camera parameter estimated by the calibration.

  However, when considering the actual operation of the stereo camera 3D measurement system, the camera parameters at the time of calibration are determined by the effects of aging and vibration of the camera fixture. There is a possibility that the value has changed from the parameter value (or the camera parameter value (after correction) used in the measurement system).

  Based on the camera parameter value at the time of calibration (or the camera parameter value (after correction) used in the measurement system) at a certain point after being affected by the above effects, If it is performed, an erroneous three-dimensional measurement result is calculated. For this reason, it is necessary to detect a camera parameter value that is no longer an appropriate value (an error has occurred in the camera parameter value) and correct the camera parameter value to a correct value.

  Patent Document 1 discloses a technique for reducing such an error in camera parameter values.

  First, the calibration device described in Patent Document 1 recognizes a planar object whose size is known in advance, such as a license plate, from a captured image (one camera image) of a stereo camera. The feature point coordinates of the object are detected.

  Subsequently, the coordinate position of the projection point at which the feature point coordinate of one camera image is projected onto the other camera image is compared with the coordinate position of the feature point corresponding to the feature point coordinate in the other camera image. Thus, a new camera parameter is estimated.

  Finally, the camera parameters are corrected by comparing the new camera parameters with the original camera parameters.

JP 2014-74632 A (published April 24, 2014)

Z. Zhang, “A Flexible New Technology for Camera Calibration”, [online], November 5, 2009, [May 27, 2015 search], Internet <http://research.microsoft.com/en-us/um /people/zhang/Papers/TR98-71.pdf>

  As described above, the accuracy of camera parameters is important in three-dimensional measurement, and errors due to the effects of vibration and other factors are present in the values of current camera parameters (or current camera parameters corrected thereafter) obtained during calibration. If it occurs, accurate three-dimensional measurement cannot be performed. Therefore, if an error occurs in the camera parameter value, it is necessary to quickly reduce the error.

  However, the calibration device described in Patent Document 1 cannot reduce the error of the camera parameter value when an object having a known size does not appear in the captured image of the stereo camera.

Therefore, for example, when measuring the three-dimensional position of the “person” that is the object of unknown size using the measurement system including the calibration device and the stereo camera described in Patent Document 1, the following two conditions overlap. The three-dimensional position of the “person” cannot be measured accurately.
(Condition 1) An error occurred in the camera parameter value of the stereo camera before measuring the three-dimensional position of “person”.
(Condition 2) Other objects (specifically, objects with known sizes (for example, license plates)) are not included in the shooting range of the stereo camera together with “person”.

  The present invention has been made in view of the above problems, and its main purpose is to generate an error in the value of the camera parameter regardless of whether or not an object having a known size is reflected in the captured image. It is to realize an image processing apparatus and an error determination method capable of detecting the above.

  In order to solve the above problems, an image processing apparatus according to an aspect of the present invention includes two image acquisition units that acquire at least two viewpoint images, and two camera parameters including a translation parameter and a rotation parameter. An image calibration unit that generates two calibration images from the viewpoint image, and a determination unit that determines whether or not an error has occurred in the camera parameter, wherein the image calibration unit has a longitudinal length according to the translation parameter. The translation error in at least one of the direction and the short direction is not calibrated, and the two calibration images are generated in which the rotation deviation according to the rotation parameter is calibrated. Based on this, a deviation range in the at least one direction in the two calibration images is calculated, and the one calibration image in the two calibration images is calculated. By calculating a deviation amount in the at least one direction of a corresponding point corresponding to the reference point in the other calibration image with respect to an arbitrary reference point, and comparing the deviation range and the deviation amount, It is determined whether an error has occurred.

  In order to solve the above problems, an error determination method according to an aspect of the present invention is an error determination method by an image processing apparatus, which includes an image acquisition step of acquiring at least two viewpoint images, a translation parameter, and a rotation parameter. An image calibration step for generating two calibration images from the two viewpoint images, and a determination step for determining whether or not an error has occurred in the camera parameters, In the step, the two calibration images in which the translational deviation in at least one of the longitudinal direction and the lateral direction according to the translation parameter is not calibrated, and the rotational deviation according to the rotational parameter is calibrated. Generating and determining, based on the camera parameters, in the longitudinal direction and the short side in the two calibration images. Calculating a deviation range in at least one direction of the direction, and an arbitrary reference point in one of the two calibration images, and a corresponding point corresponding to the reference point in the other calibration image, The method includes a step of calculating a shift amount in at least one direction and a step of determining whether or not an error has occurred in the camera parameter by comparing the shift range with the shift amount.

  The image processing apparatus and the error determination method according to one aspect of the present invention can detect that an error has occurred in the value of the camera parameter regardless of whether or not an object having a known size is reflected in the captured image. There is an effect.

It is a figure which shows the structural example of the image processing apparatus which concerns on 1st Embodiment. It is a figure which shows the processing flow of the image processing apparatus which concerns on 1st Embodiment. It is a figure which shows the concept of the production | generation image which a rotation calibration part produces | generates. It is a figure which shows the concept of the shift | offset | difference range based on a camera parameter. It is a figure which shows the example of a different structure of the image processing apparatus which concerns on 1st Embodiment. It is a figure showing an example of 1 composition of an image pick-up device concerning a 1st embodiment. It is a figure which shows the structural example of the image processing apparatus which concerns on 2nd Embodiment. It is a figure which shows the processing flow of the image processing apparatus which concerns on 2nd Embodiment. It is a figure which shows an example of the image shown to an information presentation part. It is a figure which shows the structural example of the three-dimensional measurement process part which concerns on 3rd Embodiment. It is a figure which shows the processing flow of the three-dimensional measurement process part which concerns on 3rd Embodiment. It is a figure which shows the structural example of the vehicle-mounted system which concerns on 4th Embodiment. It is a figure which shows the processing flow of the vehicle-mounted system which concerns on 4th Embodiment. It is a figure which shows the structural example of the three-dimensional image display system which concerns on 5th Embodiment.

  Hereinafter, each embodiment of the present invention will be described with reference to the drawings.

(First embodiment)
A basic configuration of an image processing apparatus according to an embodiment of the present invention will be described. FIG. 1 is a diagram illustrating a configuration example of an image processing apparatus according to the present embodiment.

(Configuration of the image processing apparatus 100)
As illustrated in FIG. 1, the image processing apparatus 100 includes an image acquisition unit 101, a parameter acquisition unit 102, a rotation calibration unit 103, a deviation range calculation unit 104, a deviation amount calculation unit 105, and a determination unit 106. Yes.

  The image acquisition unit 101 acquires at least two viewpoint images. The at least two viewpoint images may be, for example, two viewpoint images photographed substantially simultaneously by two cameras that constitute a stereo camera system.

  The parameter acquisition unit 102 acquires camera parameters and the like (camera parameters and parameters related to the imaging distance).

  The rotation calibration unit 103 corrects the rotation of the at least two images based on camera parameters.

  The shift range calculation unit 104 calculates the translation shift range of the two images based on the camera parameters.

The shift amount calculation unit 105 calculates a shift amount calculation unit 105 based on the corresponding points between the at least two images.
The determination unit 106 compares the shift range and the shift amount to perform camera parameter shift determination (determination as to whether an error has occurred in the camera parameter).

  The translational deviation range indicates “the range of the translational deviation amount in which three-dimensional measurement based on the camera parameters can be performed with a certain degree of accuracy”.

  The processing of each part of the image processing apparatus 100 is implemented by, for example, an FPGA (Field Programmable Gate Array), a DSP (Digital Signal Processor), a GPU (Graphics Processing Unit), an ASIC (Application Specific Integrated Circuit), or the like. Also good. Alternatively, the image processing apparatus 100 may perform the above-described processing by a processor such as a CPU (Central Processing Unit) executing a program.

(Operation of the image processing apparatus 100)
Next, the operation of the image processing apparatus 100 will be described with reference to FIG. FIG. 2 is a diagram illustrating a processing flow of the image processing apparatus 100.

  In step S101, the image acquisition unit 101 acquires multi-viewpoint image data including a plurality of viewpoint images photographed from a plurality (two or more) different viewpoints.

Examples of the multi-view image data acquired by the image acquisition unit 101 include the following multi-view image data.
A single imaging device composed of an imaging element such as a CMOS (Complementary Metal Oxide Semiconductor) or CCD (Charge Coupled Device) and an optical component such as a lens is translated while being mounted on a translation table. Multi-viewpoint image data including a plurality of viewpoint images obtained by photographing a target object a plurality of times. Multi-viewpoint image data including a plurality of viewpoint images captured by a multi-view imaging system configured to include a plurality of the imaging devices. Multi-viewpoint image data including a plurality of viewpoint images photographed by a multi-view imaging system capable of photographing a multi-viewpoint image by attaching a lens system configured to have a plurality of optical paths to one image sensor side-by-side And multiple viewpoints such as frame packing Multi-viewpoint image data stored in an image storage format Hereinafter, multi-viewpoint image data acquired by the image acquisition unit 101 is viewpoint image data obtained by a stereo camera system including two cameras arranged side by side. The operation of the image processing apparatus 100 will be described with an example.

  In the following, the left camera of the two cameras is referred to as the reference camera, the right camera of the two cameras is referred to as the reference camera, and the viewpoint image data captured by the left camera is referred to as the reference image data and the right camera. The data of the viewpoint image captured by the above is called reference image data. The operation of the image processing apparatus 100 will be described while referring to the left-right direction in which the two cameras are lined up as the longitudinal direction and the up-down direction of the two cameras as the short direction.

  The description will be made with the left-right direction defined as the X-axis direction, the up-down direction defined as the Y-axis direction, and the front-back direction defined as the Z-axis direction. Here, in this description, it is assumed that the origin and the coordinate axis direction of the world coordinate system coincide with the origin of the coordinate and the coordinate axis system of the reference camera.

  Regarding the operation of the image processing apparatus 100 when the multi-viewpoint image data acquired by the image acquisition unit 101 is viewpoint image data obtained by a stereo camera system including two cameras arranged one above the other, A person skilled in the art will understand by replacing “X-axis direction” with “Y-axis direction” and “Y-axis direction” with “X-axis direction” in the following description.

  When the multi-viewpoint image data acquired by the image acquisition unit 101 is multi-viewpoint image data including three or more viewpoint images, the image processing apparatus 10 selects one viewpoint image among the three or more viewpoint images. Can be performed as reference image data, and the remaining viewpoint image data among the three or more viewpoint images can be handled as reference image data, so that operations similar to those described below can be performed. is there.

  The reference image data and reference image data acquired by the image acquisition unit 101 may be data of a calibrated image (calibration image) captured by a multi-view image capturing system in which lens distortion parameters and internal parameters are calibrated in advance. Alternatively, these parameters may be data of a pre-calibration image captured by a multi-viewpoint image capturing system that has not been calibrated.

  In the latter case, the image acquisition unit 101 or an image data processing unit (not shown) receives the lens distortion parameter and the internal parameter from the parameter acquisition unit 102, and calibrates the reference image data and the reference image data based on each parameter. Then, each calibrated image data may be transmitted to the rotation proofing unit 103.

  For example, when the image data processing unit performs image processing using an image in which the lens distortion parameter and the internal parameter are calibrated, the image data processing unit is reduced by reading the calibrated image data such as the former. The image processing can be performed with a processing load.

  In step S102, the parameter acquisition unit 102 reads camera parameters and parameters related to the imaging distance.

  The camera parameters are, for example, camera parameters (lens distortion parameters, internal parameters, external parameters) calculated by camera calibration, camera parameters used in the measurement system (after correction), and the like.

  The parameters relating to the imaging distance are, for example, a focal position, a depth of field, a provisional distance (tentatively calculated distance) of the image center, an estimated distance from the camera to the subject, a distance after performing the camera calibration, and the like.

  Here, the temporary distance at the center of the image is the distance of the subject (the distance in the Z direction) at the image center pixel, which is calculated by the three-dimensional measurement process described in [Background Art]. The image center pixel may be one pixel at the center of the image. Alternatively, the temporary distance may be the distance of the subject in the center area of the image. In other words, the provisional distance may be an average distance of measurement results regarding a predetermined area (for example, an image center area of 5 pixels vertically by 5 pixels horizontally) centered on the same position as the center of the image.

  The “distance for which the camera calibration has been performed” may be, for example, a distance from the camera to the reference board when the calibration reference board is installed in the camera calibration, or the camera focus. It may be a distance.

  Note that the multi-viewpoint image data and the camera parameters may be included in a file in a specific format. For example, the camera parameter may be included in the image file as header information of the image file of the multi-viewpoint image. The image processing apparatus 100 can be configured such that a decoding unit (not shown) decodes a file of this format, and the image acquisition unit 101 and the parameter acquisition unit 102 receive the respective data.

  In step S103, the rotation calibration unit 103 receives camera parameters from the parameter acquisition unit 102, and calibrates the image data received from the image acquisition unit 101 based on the camera parameters.

Details of the camera parameters received by the rotation calibration unit 103 will be described. Camera parameters can be classified into lens distortion parameters, internal parameters, and external parameters. Furthermore, the external parameters can be classified into rotation parameters and translation parameters. The rotation calibration unit 103 receives external parameters, particularly rotation parameters, among the camera parameters. The rotation parameters R, R _Y , R _L , and R _R are expressed by the following formula (Formula 3).

R corresponds to a rotation transformation matrix from coordinates (reference camera coordinates) in the camera coordinate system of the reference camera to coordinates (reference camera coordinates) in the camera coordinate system of the reference camera. R _Y is a rotation parameter for making the XY plane of the camera coordinate system of the reference camera parallel to the XY plane of the camera coordinate system of the reference camera, and the XY plane of the camera coordinate system of the reference camera is the Y-axis direction. Indicates that α radians are rotated. R _L and R _R are rotation parameters for the standard image data and rotation parameters for the reference image data, respectively. Expression indicating the relationship between coordinate data before rotation calibration (coordinate data indicating a position on the image before rotation calibration) and coordinate data after rotation calibration (coordinate data indicating the same position on the image after rotation calibration) ( Equation 4) is shown below.

(X _L , Y _L , Z _L ) is the reference camera coordinates before calibration, (X _L ′, Y _L ′, Z _L ′) is the reference camera coordinates after calibration, (X _R , Y _R , Z _R) ) Indicates reference camera coordinates before calibration, and (X _R ′, Y _R ′, Z _R ′) indicate reference camera coordinates after calibration. The rotational calibration unit 103 can obtain a rotationally calibrated image by executing an image conversion process using the above formula (Formula 4).

  When the rotational deviation in the Y-axis direction can be ignored, α = 0 in Equation 3, so that the reference camera coordinate values are (XL ′, YL ′, ZL ′) = (XL, YL, ZL), and the matrix Calculation can be omitted.

  The concept of the rotationally calibrated image will be described with reference to FIG.

  FIG. 3A is an example showing the relationship between the standard image and the reference image before calibration. FIG. 3B is an example showing a relationship between a standard image after a general parallelization process that has been performed conventionally and a reference image. FIG. 3C is an example showing the relationship between the reference image generated by the rotation calibration unit 103 and the reference image.

  In a measurement system using a general stereo camera, in order to reduce the amount of calculation in block matching processing between images, as shown in FIG. 3B, parallel processing is performed on the reference image and the reference image. . After the parallelization process, the rotational deviation and translational deviation between the standard image and the reference image are calibrated. That is, when the stereo camera system includes two cameras arranged on the left and right, the two images after the parallelization process are two cameras in a state where there is no rotational deviation in the Y-axis direction and in the X-axis direction. This corresponds to two images taken by two cameras that are separated by the length of the baseline (only distance B). Thereby, the search direction in the block matching process for parallax calculation can be limited to the X-axis direction.

  On the other hand, in the present embodiment, the rotation calibration unit 103 calibrates only the rotation parameters as shown in FIG. 3C, so that the two images generated by the rotation calibration unit 103 are rotated in the Y-axis direction. This corresponds to two images taken by two cameras that are not displaced and separated by Bx in the X-axis direction and by By in the Y-axis direction. That is, the rotation calibration unit 103 generates two calibration images in which the rotational deviation according to the rotation parameter is calibrated and the translational deviation of the deviation amount (Bx, By) according to the translation parameter is not calibrated. Here, Bx represents the translational displacement amount (long baseline length) in the longitudinal direction of the two cameras, and By represents the translational displacement amount (short baseline length) in the lateral direction of the two cameras. Bx and By are calculated by the following formula (Formula 5) based on the translation parameters (tx, ty, tz).

  In step S <b> 104, the deviation range calculation unit 104 receives the camera parameter and the parameter related to the imaging distance from the parameter acquisition unit 102, and calculates the translational deviation range in the two images with reference to these parameters.

  The shift ranges Sx1_near, Sx1_far, Sy1_near, Sy1_far are calculated by the following formula (Formula 6).

Here, f is a focal length calculated as an internal parameter, L _near is a near point distance of the reference shooting range, L _far is a far point distance of the reference shooting range, and Bx and By are translational deviation amounts calculated by Expression 5. is there. The relationship of each parameter in Formula 6 is as shown in FIG. The distance in the X-axis direction from the coordinate center 107 of the reference camera to the coordinate center 108 of the reference camera is Bx, and the distance in the Y-axis direction is By. The reference shooting distances L _near and L _far are distances in the Z-axis direction from the reference camera coordinates to the reference shooting distance near point 109 and the reference shooting distance far point 110, respectively, and are set from the parameters relating to the imaging distance. As described above, the deviation ranges Sx1_near, Sx1_far, Sy1_near, and Sy1_far are calculated by using the parameter similarity. The shift range is broken down into a shift range in the X-axis direction and a shift range in the Y-axis direction. The range from Sx1_near to Sx1_far is the shift range in the X-axis direction, and the range from Sy1_near to Sy1_far is the shift range in the Y-axis direction.

As described above, L _near and L _far are parameters related to the imaging distance, and include, for example, the depth of field, the provisional distance of the image center (provisionally calculated distance), the estimated distance from the camera to the subject, and the camera calibration. Each parameter of the station distance may be composed of two parameters, a parameter related to the near point and a parameter related to the far point. The shift range calculation unit 104 may read two parameters forming the pair for all the above pairs as parameters relating to the imaging distance, or two parameters forming the pair for any of the above pairs. May be read.

In the above description, a configuration has been described in which two parameters, a parameter relating to the near point and a parameter relating to the far point, are read. For example, the deviation range calculation unit 104 assumes that the reference shooting distance L = L _near = L _far. In this case, only one parameter may be read from the parameter group related to the imaging distance. Further, the reference photographing distance L = Sx can be set by using a deviation amount Sx2 calculated by a deviation amount calculation unit 105 described later. When only one parameter is read in this way, the deviation range calculation unit 104 may obtain the deviation distance instead of the deviation range by using the following equation (Equation 7) instead of Equation 6.

  Here, Sx1 and Sy1 are deviation distances.

  In step S105, the deviation amount calculation unit 105 receives the standard image data and the reference image data subjected to the rotation calibration process, detects the amount of positional deviation in the longitudinal direction and the short side direction, and determines the detected deviation amount. 106.

  The deviation amount calculation unit 105 detects the deviation amounts (Sx2, Sy2) in the X-axis direction and the Y-axis direction of “corresponding points corresponding to the reference point” on the reference image with respect to an arbitrary reference point on the reference image. . Here, Sx2 indicates a shift amount in the X-axis direction on the image, and Sy2 indicates a shift amount in the Y-axis direction on the image. That is, these shift amounts are detected as a vertical parallax amount and a horizontal parallax amount. The shift amount is calculated by a method using block matching, for example, in the same manner as the parallax calculation means in a general stereo image. In block matching, a template composed of a reference point and its surrounding pixels in the reference image is generated, and a matching position is searched for on the other reference image, thereby calculating a deviation with respect to each direction. For the evaluation of matching, for example, SAD (Sum of Absolute Difference), SSD (Sum of Squared Difference), ZNCC (Zero-means Normalized Cross-Correlation), or the like may be used. In block matching, a shift amount between two images is calculated. Therefore, based on the above-described formula 2, the amount of deviation between the two images (the amount of deviation in the image coordinate system) is converted into the amount of deviation in the camera coordinate system.

  Here, the reference point will be described. At least one or more arbitrary reference points are selected. For example, one or more points whose edge strength is greater than a predetermined threshold within a range belonging to the center region of the reference image and belonging to a predetermined region on the reference image are selected as the reference points. The edge strength may be calculated using, for example, an output value of a Sobel filter, a Prewit filter, a DoG (Difference of Gaussian), or the like.

  As another method for specifying an arbitrary reference point, for example, when the image processing apparatus 100 including a stereo camera system is used in a fixed manner, a background image (including a background image and an image of a subject) captured in advance is used. In a specific region (region where the difference in pixel values between the two images is not 0) determined from the reference image (an image including a part of the background image and the subject image) It is also possible to set one or more points whose edge strength is greater than a predetermined threshold. This method is useful for determining a reference point on an image whose background is a plane that cannot detect an edge, or a reference point on an image that is entirely covered with an image of a subject. Further, when such an image processing apparatus 100 is used as a measurement apparatus, a point to be measured can be used as a reference point. Alternatively, the image processing apparatus 100 detects a specific marker pattern (checkered pattern or arbitrary color) in the image when the operation of detecting the camera parameter error is performed according to the user operation, and the marker pattern exists. An arbitrary point in the area to be used may be used as the reference point, or an arbitrary point on the image designated by the user may be used as the reference point.

  When a plurality of arbitrary reference points are selected, the deviation amount calculation unit 105 calculates, for each of the plurality of reference points, the deviation amount of the “corresponding point corresponding to the reference point” with respect to the reference point. May be. Then, the deviation amount calculation unit 105 may determine a representative value of the deviation amount based on the calculated plural deviation amounts. For example, the deviation amount calculation unit 105 may use a statistical value such as an average value or a median value of the plurality of deviation amounts as a representative value of the deviation amount, and among the plurality of deviation amounts, The deviation amount of the “corresponding point corresponding to the reference point” with respect to the reference point having the highest edge strength may be determined as a representative value of the deviation amount.

Here, the image processing apparatus 100 sets L _near and L _far so that the positions advanced by L _near and L _far in the Z-axis direction from the camera are included in the range of the depth of field of the image pickup apparatus. It is desirable to do. This is because if the reference point is detected based on the edge strength, the probability of detecting a measurement point within the range of the reference shooting distance increases. For example, when the edge strengths at all the reference points do not exceed a predetermined threshold, it is determined that the reference points cannot be detected within the range of the reference shooting distance, and the subsequent determination process may not be performed. .

Further, when the image processing apparatus 100 is used in a measurement system using a stereo camera system, the values of L _near and L _far are values indicating the distance from the camera to the near point of the measurement range assumed by the measurement system, The image processing apparatus 100 can also be configured to use a value indicating the distance from the camera to the far point of the measurement range and use any measurement point as a reference point.

When the shooting range is limited (for example, when the distance from the camera is only within the range of 1 to 5 m), the L _near value is set to 1 m and the L _far value is set to 5 m. (Ie, a value indicating the distance from the camera to the near point of the shooting range and a value indicating the distance from the camera to the far point of the shooting range are assigned to L _near and L _far , respectively) An arbitrary upper point can be set as the reference point.

Furthermore, when the image processing apparatus 100 is used in a line inspection system such as a factory, since the range in which a moving subject such as a part is reflected in an image can be known, the set values of L _near and L _far can be limited. For example, if the inspection system is installed 400 mm above the line and the size of the part passing through the line is 10 mm, L _{near is set} to 390 mm, L _far is set to 400 mm, and the point on the part is used as the reference point. It can also be configured to set.

Thus, by setting the values of L _near and L _far according to the size of the object, the calculated deviation range becomes a range based on the actual size of the object. Therefore, it can be said that such a reference point setting method is a preferable setting method capable of reducing the error of the camera parameter.

  As described above, in step S103 and step S104, the shift range estimated by the camera parameter and the shift amount related to the two captured images after the rotation calibration are calculated.

  In step S <b> 106, the determination unit 106 receives the shift range from the shift range calculation unit 104, receives the shift amount from the shift amount calculation unit 105, and performs shift determination. Specifically, the determination unit 106 performs deviation determination based on the determination formulas shown in the following formulas (Formula 8 and Formula 9).

  Here, Sx1_near, Sx1_far, Sy1_near, and Sy1_far are values indicating the deviation ranges in the X-axis direction and the Y-axis direction calculated in step S104, and Sx2 and Sy2 are X detected in step S105. It is a value indicating the amount of deviation in the axial direction and the Y-axis direction, and THx_far, THx_near, THy_far, and THy_near are arbitrary threshold values.

  That is, based on the deviation amounts (Sx1_near, Sx1_far, Sy1_near, Sy1_far) of the reference photographing distance range and arbitrary threshold values (THx_far, THx_near, THy_far, THy_near), the deviation amounts Sx2 and Sy2 based on the image are determined. The arbitrary threshold values (THx_far, THx_near, THy_far, THy_near) are set so that THx_far = THx_near, THy_far = THy_near even if THx_far and THx_near are set differently and THy_far and THy_near are set differently. You may do it.

  In the case where both of the above formulas 8 and 9 hold, the determination unit 106 determines that “the camera parameter has no error”. On the other hand, if not, the determination unit 106 determines that “an error has occurred in the camera parameters” and outputs a value indicating the determination result. For example, the value indicating the determination result that “the camera parameter has no error” is the flag value 0, and the value indicating the determination result that the “camera parameter is shifted” is the flag value 1. Also good. For example, the flag value may be output as binary data for processing by another processing unit.

  Alternatively, when the determination unit 106 determines that “an error has not occurred in the camera parameter”, the level of the output voltage of the electronic circuit (determination unit 106) becomes High, and “an error has occurred in the camera parameter”. May be configured so that the level of the output voltage of the electronic circuit (determination unit 106) becomes Low.

  Alternatively, in step S106, when the determination unit 106 receives the shift distance calculated by the formula 7 from the shift range calculation unit 104, the determination unit 106 is based on the determination formulas shown in the following formulas (Formula 10 and Formula 11). The deviation determination may be performed.

  Here, Sx1 and Sy1 are values indicating deviation distances in the X-axis direction and the Y-axis direction calculated in step S104, respectively, and Sx2 and Sy2 are X-axis direction and Y-axis direction detected in step S105, respectively. It is a value indicating the amount of deviation. Moreover, THx_far, THx_near, THy_far, and THy_near are arbitrary threshold values. That is, based on the shift distance (Sx1, Sy1) calculated by Expression 7 and an arbitrary threshold (THx_far, THx_near, THy_far, THy_near), the shift amounts Sx2, Sy2 based on the image are determined.

  When both Expression 10 and Expression 11 hold, the determination unit 106 determines that “the camera parameter has no error”. On the other hand, if not, the determination unit 106 determines that “an error has occurred in the camera parameters” and outputs a value indicating the determination result. For example, the value indicating the determination result that “the camera parameter has no error” is the flag value 0, and the value indicating the determination result that the “camera parameter is shifted” is the flag value 1. Also good.

  As shown in FIG. 1, the deviation range calculation unit 104, the deviation amount calculation unit 105, and the determination unit 106 may be provided as separate processing units in the image processing apparatus 100, but are shown in FIG. As described above, the above three processing units can be configured as one deviation determination unit 111. The shift determination unit 111 calculates a shift range by the same processing as the shift range calculation unit 104, performs block matching for the shift range in the same manner as the shift amount calculation unit 105, and calculates the maximum similarity calculated by block matching. Is determined not to exceed a predetermined value, it is determined that the parameter is deviated.

  Through the above processing, the image processing apparatus 100 can output the deviation calculation results of the deviation range calculation unit 104 and the deviation amount calculation unit 105 and the flag value of the determination unit 106 to the outside.

  As shown in FIG. 6, the image processing apparatus 100 can be configured as a part of the image capturing apparatus 112.

  As shown in FIG. 6, the image capturing device 112 includes the image processing device 100, an image capturing unit 113 that captures at least two images, a data storage unit 114 that stores camera parameters and image processing results, and the image. An information presentation unit 115 that presents processing results and the like to the user, and a data bus 116 through which data exchanged between the units passes.

  The imaging unit 113 is an imaging device that captures a plurality of field images showing a state seen from a plurality (two or more) of different viewpoints. The imaging unit 113 may be an imaging device including an imaging element such as a complementary metal oxide semiconductor (CMOS) or a charge coupled device (CCD) and an optical component such as a lens.

  In this case, the imaging unit 113 may photograph the target object a plurality of times while translating in a state of being on the translation table.

  Alternatively, the imaging unit 113 may be a multi-eye imaging system configured to include a plurality of the imaging devices.

  Alternatively, the imaging unit 113 may be a multi-view imaging system that can capture a multi-viewpoint image by attaching a lens system configured to have a plurality of optical paths with respect to one imaging device.

  The data holding unit 114 is configured by a storage device such as an HDD (Hard Disk Drive) and an SSD (Solid State Drive).

  The data holding unit 114 is read by the image processing apparatus 100, such as camera parameters, parameters related to the imaging distance, data indicating the results of image processing by the image processing apparatus 100 (flag values, etc.), images taken by the imaging unit 113, and the like. Retain data.

  The information presentation unit 115 is a light emitting unit such as an LED (Light Emitting Diode), for example, and is a light emitting unit that performs light emission control according to the determination result by the determination unit 106. For example, the information presentation unit 115 can notify the user of the determination result by emitting no light when the determination unit 106 outputs the flag value 0 and emitting light when the determination unit 106 outputs the flag value 1. . Or the information presentation part 115 can also display the warning message by a text or an image according to the said flag value on display parts, such as LCD (Liquid Crystal Display) and OLED (Organic Light Emitting Diode), for example. In addition, the information presentation unit 115 can output a warning sound when the flag value is 1 (when an error occurs in the camera parameter), for example, a speaker.

  The data bus 116 is a bus for a computer such as PCI (Peripheral Component Interconnect) and SCSI (Small Computer System Interface). Or it is an electron beam for connecting each unit based on Ethernet (trademark) specification.

  The image capturing apparatus 112 configured as described above can detect that an error has occurred in the camera parameter and can present the detection result to the user. Note that the image capturing apparatus 112 does not necessarily need to perform the shift determination process for all shootings, and may perform the shift determination process only when designated by the user.

  In addition, the error detection method of the present embodiment has an effect that an error can be detected at the time of photographing an arbitrary object as well as a known object, and a reference point to be selected can be minimized.

  That is, in the conventional method of using between known objects (eg, Patent Document 1), in order to estimate a new camera parameter, there are 8 unknown variables in the camera parameter estimation, so between two camera images. It is necessary to detect at least four or more common feature points. On the other hand, in the method of the present embodiment, it is possible to detect an error in the camera parameter only by determining at least one reference point. Furthermore, since it is indispensable to perform object recognition processing on two images in order to detect a known object, the amount of calculation has increased, but in this embodiment, one image is detected. Therefore, it is only necessary to perform object recognition processing, and the amount of calculation can be reduced.

(Additional notes)
The rotation calibration unit 103 calibrates the rotational deviation according to the rotation parameter, calibrates the translational deviation of the deviation amount in one of the longitudinal direction and the lateral direction according to the translation parameter, and the deviation in the other direction. Two calibration images in which the quantity (Bx or By) is not calibrated may be generated.

  When two calibration images in which the displacement amount (Bx) in the longitudinal direction is not calibrated are generated, the displacement range calculation unit 104 calculates the displacement ranges Sx1_near and Sx1_far using Equation 6, and the determination unit 106 uses the equation The shift determination may be performed according to the determination formula of 8.

  Alternatively, when two proof images are generated in which the lateral shift amount (By) is not calibrated, the shift range calculation unit 104 calculates the shift ranges Sy1_near and Sy1_far using Equation 6, and the determination unit 106 May perform the deviation determination according to the determination formula of Formula 9.

(Second Embodiment)
In the second embodiment, an example in which the image processing apparatus 100 is applied to three-dimensional measurement processing by a stereo camera system will be described. In the description of the present embodiment, detailed description of each part common to the above-described embodiments is omitted.

  FIG. 7 is a configuration example in which the image processing apparatus 100 according to this embodiment includes a three-dimensional measurement processing unit 200.

  The three-dimensional measurement processing unit 200 calculates the parallax based on the camera parameter correction unit 201 that corrects the camera parameters, the image parallelization unit 202 that parallelizes the acquired two field images, and the two parallel field images. A parallax calculation unit 203 that performs the calculation, and a three-dimensional coordinate calculation unit 204 that calculates the three-dimensional coordinates of the measurement point from the calculated parallax and the camera parameters.

  The camera parameter correction unit 201, the image parallelization unit 202, the parallax calculation unit 203, and the three-dimensional coordinate calculation unit 204 may be implemented by electronic circuits such as an FPGA, a DSP, a GPU, and an ASIC, for example. Alternatively, the three-dimensional measurement processing unit 200 may perform the processes of the above-described units when a processor such as a CPU (Central Processing Unit) executes a program.

(Operation of the three-dimensional measurement processing unit 200)
Next, the operation of the three-dimensional measurement processing unit 200 will be described with reference to FIG. FIG. 8 is a diagram illustrating a processing flow of the three-dimensional measurement processing unit 200.

  In step S201, the camera parameter correction unit 201 receives the flag value information from the determination unit 106 and performs a determination process. When the flag value is 0, that is, when there is no error in the camera parameter (S201: Yes), the process proceeds to step S202. If the flag value is 1, that is, if an error has occurred in the camera parameter (S201: No), the process proceeds to step S203.

  In step S202, the camera parameter correction unit 201 determines that there is no error in the camera parameters acquired by the parameter acquisition unit 102. The camera parameters acquired by the parameter acquisition unit 102 are read as they are from a data holding unit (not shown in FIG. 7), and the camera parameters are sent to the image parallelization unit 202.

  In step S <b> 203, the camera parameter correction unit 201 determines that an error has occurred in the camera parameter, performs a process of correcting the camera parameter, and sends the corrected camera parameter to the image parallelization unit 202. The camera parameter correction unit 201 calculates a correction amount in the X-axis direction and a correction amount in the Y-axis direction based on the following formulas (Formula 12 and Formula 13).

  Here, CorX and CorY are a camera parameter correction amount in the X-axis direction and a camera parameter correction amount in the Y-axis direction, respectively. Since the other variables (variables used in the above-described equations 8 and 9) have already been described, the description of the other variables is omitted here.

  The camera parameter correction unit 201 reads the camera parameters before correction from the data holding unit 114 and corrects the parameters. The camera parameter correction unit 201 corrects the translation parameter as can be seen from the following equation (Equation 14), for example.

  Here, (tx ′, ty ′, tz ′) is a translation parameter after correction, and (tx, ty, tx) is a translation parameter before correction.

  The camera parameter correction unit 201 corrects the corrected camera parameter and the correction amount (CorX, which is the correction amount from the X component value tx of the translation parameter before correction, and the Y component value ty of the translation parameter before correction). (CorY) is stored in the data holding unit 114.

  For example, the camera parameter correction unit 201 may overwrite the camera parameter before correction with the camera parameter after correction, or may store the camera parameter after correction separately from the camera parameter before correction. . When the camera parameter correction unit 201 overwrites and saves the camera parameter, the camera parameter correction unit 201 can check the cumulative shift amount from the initial camera parameter by storing the cumulative amount of the correction amount (CorX, CorY).

  The camera parameter correction unit 201 may store a value indicating the number of corrections in the data holding unit 114. That is, the camera parameter correction unit 201 may hold a value indicating the number of times the camera parameter is corrected by incrementing a value indicating the number of corrections every time the camera parameter is corrected.

  When the camera calibration is performed again (that is, when the camera parameter is updated to the latest parameter), the camera parameter correction unit 201 has a value indicating the accumulated deviation amount and a value indicating the number of corrections. May be reset to zero. A value indicating the cumulative deviation amount or a value indicating the number of corrections may be presented to the user via the information presenting unit 115. Accordingly, the user can determine the necessity of recalibration and the reliability of the three-dimensional measurement result calculated based on the corrected camera parameter through the presented information.

  In step S204, the image collimating unit 202 receives the standard image data and the reference image data from the image acquisition unit 101, receives the camera parameters from the camera parameter correction unit 201, and calculates 2 from the camera parameters, the standard image, and the reference image. A parallel image is generated. The two parallel images are two images composed of a standard image and a reference image that are parallel to each other as shown in FIG.

  In step S205, the parallax calculation unit 203 sets measurement points for performing three-dimensional measurement. The parallax calculation unit 203, for example, in the base image, all points within a range (specific range) made up of all pixels for which corresponding points exist in the reference image, and a plurality of edges detected as edges within the specific range. A point or a point indicating a marker appearing as a subject in the reference image in the reference image may be automatically set as a measurement point.

  Alternatively, the three-dimensional measurement processing unit 200 may include a user designation unit (not shown), and the user designation unit may set an arbitrary point designated by the user as a measurement point.

  In step S206, the parallax calculation unit 203 calculates the parallax of the selected measurement point. The parallax is calculated by, for example, block matching.

  In step S <b> 207, the three-dimensional coordinate calculation unit 204 calculates the three-dimensional coordinates of each measurement point based on Equation 1 described above. The three-dimensional coordinates calculated here are three-dimensional coordinates based on the camera coordinate system. However, the three-dimensional coordinate calculation unit 204 converts a three-dimensional coordinate based on the camera coordinate system into a three-dimensional measurement value of another coordinate system by using a coordinate conversion matrix from the camera coordinate system to another coordinate system. be able to.

  In addition, the three-dimensional coordinate calculation unit 204 may hold the measured three-dimensional measurement value in the data holding unit 114 in order to compare with other measurement results. This is because the user can obtain the distance between the two measurement points and the volume of the target object, or can compare the current measurement result regarding the target object with the past measurement result regarding the target object.

  Through the above processing, the three-dimensional measurement processing unit 200 can obtain the three-dimensional coordinates of each measurement point. The three-dimensional measurement processing unit 200 displays the calculated three-dimensional coordinates and camera parameter correction information on the information presenting unit 115 as shown in FIG. 9, for example. In this case, the information presentation unit 115 is a display device such as an LCD or an OLED.

  The image 205 displayed on the information presentation unit 115 includes a captured image area 206, a measurement result display area 207, and a camera parameter warning area 208. 9 shows an example in which the three-dimensional measurement processing unit 200 displays the captured image area 206, the measurement result display area 207, and the camera parameter warning area 208 at different positions. The measurement processing unit 200 may display the measurement result display area 207 and the camera parameter warning area 208 superimposed on the captured image area 206.

  The three-dimensional measurement processing unit 200 displays an image showing an arbitrary measurement target 209 in the captured image area 206.

  The three-dimensional measurement processing unit 200 displays three-dimensional position information indicating the position that is the measurement point in the area of the arbitrary measurement target 209 in the measurement result display area 207. For example, the three-dimensional measurement processing unit 200 displays information indicating the distance from the camera to an arbitrary measurement target 209 in the measurement result display area 207.

  The three-dimensional measurement processing unit 200 displays information such as the number of corrections and the accumulated deviation amount stored in the data holding unit 114 in step S203 in the camera parameter warning area 208.

  The three-dimensional measurement processing unit 200 may display a figure (such as a warning mark) according to the number of corrections and the accumulated deviation amount, instead of a numerical value indicating the number of corrections and the accumulated deviation amount. For example, the three-dimensional measurement processing unit 200 may blink (highlight) the graphic when the number of corrections and / or the accumulated deviation amount is equal to or greater than a predetermined threshold. Thereby, the user is alerted.

  The camera parameter warning area 208 may not be displayed when the correction count is zero.

  With the above configuration, the three-dimensional measurement processing unit 200 displays the three-dimensional measurement result measured using the corrected camera parameter (the error is reduced) when an error occurs in the camera parameter, and the camera Information indicating that the parameter has been corrected can be presented to the user.

(Third embodiment)
In some cases, the image processing apparatus 100 according to the first and second embodiments performs a camera parameter deviation determination process even though no error has occurred in the camera parameter. There is still room to reduce the amount of processing.

  The image processing apparatus 100 according to the third embodiment automatically determines the timing for executing the camera parameter deviation determination process, and performs the deviation determination process at such timing. In the description of the present embodiment, detailed description of each part common to the above-described embodiments is omitted.

  FIG. 10 is an example of a three-dimensional measurement processing unit 300 included in the image processing apparatus according to the present embodiment. Although not shown in FIG. 10, the image processing apparatus according to the present embodiment includes each unit included in the image processing apparatus according to the second embodiment.

  Similar to the three-dimensional measurement processing unit 200 according to the second embodiment, the three-dimensional measurement processing unit 300 includes a camera parameter correction unit 201, an image parallelization unit 202, a parallax calculation unit 203, and a three-dimensional coordinate calculation unit. 204, and an abnormal value determination unit 301 is newly provided.

(Operation of the three-dimensional measurement processing unit 300)
Next, the operation of the three-dimensional measurement processing unit 300 will be described with reference to FIG. FIG. 11 is a diagram illustrating a processing flow of the three-dimensional measurement processing unit 300.

  In step S301, the three-dimensional measurement processing unit 300 calculates the three-dimensional coordinates by executing the same steps (S201 to S207) as in the second embodiment, and uses the calculated three-dimensional coordinate values as the abnormal value determination unit. 301.

  In step S <b> 302, the abnormal value determination unit 301 refers to the value indicating the measurement result received from the three-dimensional coordinate calculation unit 204 and the past measurement result recorded in the data holding unit 114 and performs an abnormal value determination. . Abnormal value determination means determination of whether or not a value (three-dimensional coordinate value of a measurement point) indicating a measurement result received from the three-dimensional coordinate calculation unit 204 is an abnormal value.

  Alternatively, when photographing a subject whose shape is known in advance, correct data such as CAD (Computer Aided Design) data of the subject is received, and the measurement result received from the three-dimensional coordinate calculation unit 204 is compared with the correct data. As described above, the abnormal value determination may be performed.

  For example, when the image processing apparatus according to the present embodiment is used for a line inspection in a factory, and the size of a part flowing through the line is known, the abnormal value determination unit 301 determines the known value. A difference value between the size and the size indicated by the measurement result is calculated, and when the difference value is equal to or greater than a threshold value, the abnormal value determination flag may be turned ON. Alternatively, the abnormal value determination unit 301 calculates a difference value between the known installation inclination of the part and the installation inclination indicated by the measurement result of the part, and turns on the abnormal value determination flag when the difference value is equal to or greater than a threshold value. You may comprise as follows.

  On the other hand, the abnormal value determination unit 301 may set the abnormal value flag to OFF when the value indicating the measurement result is not an abnormal value (when the difference value does not exceed the threshold value).

  In step S303, the abnormal value determination unit 301 determines whether or not the abnormal value determination flag is ON. If the abnormal value determination unit 301 determines that the abnormal value determination flag is ON (S303: Yes), the process proceeds to step S304. If the abnormal value determination unit 301 determines that the abnormal value determination flag is OFF (S303: No), the process proceeds to S306. When the abnormal value determination unit 301 determines that the abnormal value determination is ON, the abnormal value determination unit 301 stores “1” in the data holding unit 114 as a value indicating the number of abnormal value determinations.

  In step S304, the abnormal value determination unit 301 repeats the determination abnormal value determination until it determines that the abnormal value determination flag is OFF or the number of determinations that the abnormal value determination flag is ON exceeds a predetermined number. Make a decision. Each time the abnormal value determination unit 301 determines that the abnormal value determination flag is ON, the abnormal value determination unit 301 increments a value indicating the number of abnormal value determinations recorded in the data holding unit 114.

  When the number of determinations exceeds the predetermined number (S304: Yes), the abnormal value determination unit 301 determines that the camera parameter error is not large enough to be corrected, and informs the information presentation unit 115. An error message is output (step S307). If it is determined that the abnormal value determination flag is OFF before the determination number exceeds the predetermined number (S304: No), the process proceeds to step S305.

  In step S305, the image processing apparatus 100 performs camera parameter deviation determination processing by executing the same steps (S102 to S105) as in the first embodiment.

  As described above, the abnormal value determination unit 301 performs the deviation determination process only when an abnormal value is detected as the value of the measurement result.

  In step S306, the measurement result is displayed on the information presentation unit 115 as in the second embodiment.

  The image processing apparatus according to the present embodiment described above performs determination of camera parameter error in response to a measurement result. Therefore, it can be said that the image processing apparatus according to the present embodiment has a reduced processing amount when no error occurs in the camera parameters, as compared with the image processing apparatuses according to the first and second embodiments.

(Fourth embodiment)
In the fourth embodiment, an example in which the image processing apparatus 100 is applied to an in-vehicle system will be described. In the description of the present embodiment, detailed description of each part common to the above-described embodiments is omitted.

  As shown in FIG. 12, the in-vehicle system 401 according to the present embodiment includes a stereo camera 402 for photographing the front or rear of the vehicle 400, a control unit 403 for performing various control processes, and a braking unit for braking the vehicle. 404 and a multimedia unit 405 capable of presenting various types of information. Although not shown in FIG. 12, the in-vehicle system 401 according to the present embodiment includes each unit included in the image processing apparatus according to the second embodiment.

  The in-vehicle system 401 according to the present embodiment performs a brake control based on a distance between an object (another vehicle in front or behind the vehicle 400, a person, a structure, or the like) captured by the stereo camera 402 and the vehicle 400. (Anti-collision system). The in-vehicle system 401 is configured to present information related to camera parameter errors to the user.

  The stereo camera 402 is configured to execute processing of each unit of the image processing apparatus 100 according to the first embodiment. That is, the stereo camera 402 is an embedded device in which a program for executing the processing of each unit is incorporated.

  The control unit 403 includes various processing devices such as an FPGA and a CPU, and performs overall system control of the in-vehicle system 401.

  The braking unit 404 is, for example, a hydraulic brake device, and performs a braking operation so as to decrease the vehicle speed in response to an instruction from the control unit 403.

  The multimedia unit 405 is, for example, a car navigation system, and includes a display device such as an LCD or OLED and a voice presentation device such as a speaker system.

(Operation of in-vehicle system 401)
Next, the operation of the in-vehicle system 401 will be described with reference to FIG. FIG. 13 is a diagram showing a flow of processing of the in-vehicle system 401.

  In step S <b> 401, the stereo camera 402 calculates information related to the determination of the deviation between the three-dimensional coordinate value of the measurement point and the camera parameter by the same process as the image processing apparatus 100 according to the second embodiment. In the present embodiment, for example, a plurality of edge points (points corresponding to the boundary between the measurement object and the background) within the specific range described in the second embodiment in the reference image captured by the stereo camera 402. ) Is used as a measurement point. In addition, the stereo camera 402 has the three-dimensional coordinate values of some measurement points among the calculated three-dimensional coordinate values of each measurement point on the edge (for example, three-dimensional values determined as outliers by the Smirnov-Grubbs test or the like) The three-dimensional coordinate values of the remaining measurement points excluding the coordinate values are stored in the data holding unit 114. In addition, the stereo camera 402 stores, for each of the remaining measurement points, distance data indicating the distance to the measurement point in the data holding unit 114 as data of the measurement point.

  In step S402, the control unit 403 determines whether or not the flag value based on the camera parameter held in the data holding unit 114 is zero. If the control unit 403 determines that the flag value is 1 (when an error occurs in the camera parameter, S402: Yes), the control unit 403 proceeds to step S403, and determines that the flag value is 0 (camera parameter). If not, the process proceeds to step S404 in S402: No).

  In step S403, the control unit 403 refers to the distance data indicating the distance to the measurement point closest to the vehicle 400 among the data of the measurement points calculated in step S401, and the distance and the predetermined distance. Comparison with threshold A is performed.

  When the distance to the closest measurement point is smaller than the predetermined distance threshold A (S403: Yes), the control unit 403 proceeds to step S406. If not (S403: No), the control unit 403 proceeds to step S405.

  In step S404, the control unit 403 refers to the distance data indicating the distance to the measurement point closest to the vehicle 400 among the data of the measurement points calculated in step S401. Comparison with the distance threshold B is performed.

  When the distance to the closest measurement point is smaller than the predetermined distance threshold B (S404: Yes), the control unit 403 proceeds to step S406. If not (S404: No), the control unit 403 proceeds to step S405.

  In step S405, the braking unit 404 determines not to perform brake control.

  In step S <b> 406, the braking unit 404 performs brake control according to the brake instruction from the control unit 403.

  Here, the distance threshold A and the distance threshold B in step S403 and step S404 are set so as to satisfy the relationship A ≧ B. If an error occurs in the camera parameter, the measured value of the distance to the measurement point may be larger than the actual distance value, so refer to this when the error occurs in the camera parameter as described above. The distance threshold A is set to a value larger than the distance threshold B that is referred to otherwise.

  The distance threshold A and the distance threshold B may be set to the same value. In this case, when it is detected that an error has occurred in the camera parameter, the control unit 403 applies stronger braking than when no error has been detected in the camera parameter. The braking unit 404 may be instructed.

  In addition, when an error occurs in the camera parameter, the control unit 403 may present a warning indicating that an error has occurred in the camera parameter to the display device of the multimedia unit 405. Further, the control unit 403 may warn the user by voice via the speaker system of the multimedia unit 405. Or the control part 403 can also be comprised so that a warning may be shown to a user by blinking LED attached to the housing of the stereo camera 401. FIG.

  The control unit 403 may present these warnings at all times, but may present these warnings only when, for example, energization of the in-vehicle system 401 is started immediately after the engine is started. Alternatively, when vehicle 400 is an automatic transmission vehicle, a warning may be presented when the shift lever is changed from a position indicating “parking” to another position.

  In the above description, the stereo camera 402 executes various processes of the image processing apparatus 100. However, the control unit 403 may execute various processes of the image processing apparatus 100.

(Fifth embodiment)
In the fifth embodiment, an example of a three-dimensional image display system will be described. In the description of the present embodiment, detailed description of each part common to the above-described embodiments is omitted.

  A configuration example of the image processing apparatus 500 according to the present embodiment is shown in FIG. Similar to the image processing apparatus 100 in the first embodiment, the image processing apparatus 500 includes an image acquisition unit 101, a parameter acquisition unit 102, a rotation calibration unit 103, a deviation range calculation unit 104, and a deviation amount calculation unit 105. And a determination unit 106, and further includes an image parallelization unit 501 and a display image generation unit 502. The image processing apparatus 500 is connected to an external display device (display unit 503) via the data bus 116.

The display unit 503 is a display device exemplified in the following (a) to (c).
(A) glasses-type three-dimensional display device capable of displaying as a three-dimensional image by anaglyph, polarized glasses, liquid crystal shutter glasses, etc. (b) autostereoscopic three-dimensional display devices such as parallax barrier method and lenticular method (c) left and right A head-mounted display device that can present different images to the eyes The image collimating unit 501 receives standard image data and reference image data from the image acquisition unit 101 and also receives camera parameters from the parameter acquisition unit 102. The image collimating unit 501 generates two parallelized images from the camera parameters, the standard image, and the reference image. The two parallel images are two images composed of a standard image and a reference image that are parallel to each other as shown in FIG.

  The display image generation unit 502 receives two parallel images from the image parallelization unit 501, receives a flag value from the determination unit 106, and uses the two parallelized images and the flag value to display on the display unit 503. Generate a three-dimensional image for display.

  When the flag value received from the determination unit 106 is 0, that is, when it is determined that “the camera parameter has no error”, the display image generation unit 502 performs parallelization with the parallelized reference image data data. The pair with the reference image data thus converted is converted into data in a three-dimensional image format for display on the display unit 503, and the converted data is sent to the display unit 503.

  On the other hand, when the flag value received from the determination unit 106 is 1, that is, when it is determined that “an error has occurred in the camera parameter”, the display image generation unit 502 performs parallel reference image data or parallel Any one of the converted reference image data is copied. Then, the display image generation unit 502 converts a data set of two images (standard image or reference image) having the same content into a three-dimensional image format for display on the display unit 503, and displays the converted data. To part 503. For this reason, the image displayed on the display unit 503 is substantially a two-dimensional image.

  With the above configuration, the 3D image display system according to the present embodiment performs 3D display when there is no error in the camera parameter, and performs 2D display when there is an error in the camera parameter. be able to.

<About the first to fifth embodiments>
In each of the above-described embodiments, the configuration and the like illustrated in the accompanying drawings are merely examples, and the present invention is not limited to these. The configuration of each embodiment can be appropriately changed within a range in which the effect of the present invention is exhibited. In addition, the configuration of each embodiment can be changed as appropriate without departing from the scope of the object of the present invention.

[Example of software implementation]
Control blocks of the image processing apparatus 100 (particularly, the image acquisition unit 101, parameter acquisition unit 102, rotation calibration unit 103, deviation range calculation unit 104, deviation amount calculation unit 105, determination unit 106, information presentation unit 115, camera parameter correction unit 201) , The parallax calculation unit 203 and the three-dimensional coordinate calculation unit 204) may be realized by a logic circuit (hardware) formed in an integrated circuit (IC chip) or the like, or using a CPU (Central Processing Unit). It may be realized by software.

  In the latter case, the image processing apparatus 100 includes a CPU that executes instructions of a program that is software that implements each function, and a ROM (Read Only Memory) in which the program and various data are recorded so as to be readable by a computer (or CPU). Alternatively, a storage device (these are referred to as “recording media”), a RAM (Random Access Memory) that expands the program, and the like are provided. And the objective of this invention is achieved when a computer (or CPU) reads the said program from the said recording medium and runs it. As the recording medium, a “non-temporary tangible medium” such as a tape, a disk, a card, a semiconductor memory, a programmable logic circuit, or the like can be used. The program may be supplied to the computer via an arbitrary transmission medium (such as a communication network or a broadcast wave) that can transmit the program. The present invention can also be realized in the form of a data signal embedded in a carrier wave in which the program is embodied by electronic transmission.

[Summary]
An image processing apparatus (an image processing apparatus 100, an image capturing apparatus 112) according to aspect 1 of the present invention includes an image acquisition unit that acquires at least two viewpoint images (a reference image and a reference image), a translation parameter, and a rotation parameter. An image calibration unit (rotation calibration unit 103) that generates two calibration images from the two viewpoint images based on the included camera parameters, and a determination unit (determination unit) that determines whether an error has occurred in the camera parameters 106), and the image calibration unit is not calibrated for translational deviation in at least one of the longitudinal direction and the lateral direction according to the translation parameter, and the rotational deviation according to the rotational parameter is calibrated. The two calibration images are generated, and the determination unit determines whether the at least one of the two calibration images is based on the camera parameter. A deviation range in the direction is calculated, and a deviation amount in the at least one direction of a corresponding point corresponding to the reference point in the other calibration image with respect to an arbitrary reference point in one calibration image of the two calibration images Is calculated, and the deviation range and the deviation amount are compared to determine whether an error has occurred in the camera parameter.

  According to the above configuration, the image processing apparatus can detect that an error has occurred in the value of the camera parameter regardless of whether or not an object having a known size is reflected in the captured image (viewpoint image). , Has the effect.

  The image processing apparatus (image capturing apparatus 112) according to aspect 2 of the present invention may include an information presentation unit (information presentation unit 115) that presents information indicating a determination result by the determination unit in the aspect 1. .

  According to said structure, the said image processing apparatus has the further effect that a user can recognize whether the error has arisen in the value of the camera parameter.

  In the image processing apparatus (image processing apparatus 100) according to aspect 3 of the present invention, in the aspect 1 or 2, when the determination unit determines that an error has occurred in the camera parameter, the shift range and the shift A parameter update processing unit (camera parameter correction unit 201) that calculates a correction amount of the translation parameter based on the amount, and the parameter update processing unit performs correction of the correction amount on the translation parameter. Also good.

  According to the above configuration, the image processing apparatus has an additional effect that the error of the camera parameter can be automatically reduced when an error occurs in the camera parameter.

  An image processing apparatus (image processing apparatus 100) according to an aspect 4 of the present invention, in any one of the aspects 1 to 3, has a parallax calculation unit (parallax calculation unit 203) that calculates the parallax between the two calibration images. And a three-dimensional coordinate value calculation unit (three-dimensional coordinate calculation unit 204) that calculates a three-dimensional coordinate value of a measurement point based on a calculation result by the parallax calculation unit, and the determination unit includes the three-dimensional coordinate value It may be determined whether the coordinate value is an abnormal value, and when it is determined that the three-dimensional coordinate value is an abnormal value, it may be determined whether an error has occurred in the camera parameter.

  According to the above configuration, the image processing apparatus determines whether or not an error has occurred in the camera parameter when the image processing apparatus determines that the three-dimensional coordinate value is an abnormal value. is there.

  That is, the image processing apparatus determines whether an error has occurred in the camera parameter when there is a high possibility that an error has occurred in the camera parameter.

  Therefore, it can be said that the image processing apparatus has a further advantage of not wastefully executing a process for determining whether an error has occurred in the camera parameter (a process requiring a load).

  An error determination method according to aspect 5 of the present invention is an error determination method by an image processing device, based on an image acquisition step of acquiring at least two viewpoint images, and camera parameters including a translation parameter and a rotation parameter. An image calibration step for generating two calibration images from the two viewpoint images, and a determination step for determining whether or not an error has occurred in the camera parameter, and in the image calibration step, according to the translation parameter Further, the translational deviation in at least one of the longitudinal direction and the lateral direction is not calibrated, and the two calibration images in which the rotational deviation according to the rotational parameter is calibrated are generated. Based on the camera parameters, in the two calibration images, at least one of the longitudinal direction and the lateral direction And a deviation amount in at least one direction of a corresponding point corresponding to the reference point in the other calibration image with respect to an arbitrary reference point in one of the two calibration images. And a step of determining whether or not an error has occurred in the camera parameter by comparing the shift range and the shift amount.

  According to said structure, the error determination method which concerns on the said aspect 5 has an effect similar to the image processing apparatus which concerns on the said aspect 1. FIG.

DESCRIPTION OF SYMBOLS 100 Image processing apparatus 101 Image acquisition part 103 Rotation calibration part (image calibration part)
106 Determination Unit 112 Image Imaging Device (Image Processing Device)
115 Information Presentation Unit 200 Three-dimensional Measurement Processing Unit 201 Camera Parameter Correction Unit (Parameter Update Processing Unit)
203 parallax calculation unit 204 three-dimensional coordinate calculation unit (three-dimensional coordinate value calculation unit)

Claims (5)

An image acquisition unit for acquiring at least two viewpoint images;
An image calibration unit that generates two calibration images from the two viewpoint images based on camera parameters including a translation parameter and a rotation parameter;
A determination unit that determines whether an error has occurred in the camera parameter,
The image calibration unit is not calibrated for translational deviation in at least one of the longitudinal direction and the lateral direction according to the translation parameter, and the two calibrations are calibrated for rotational deviation according to the rotational parameter. Generate an image,
The determination unit calculates a shift range in the at least one direction in the two calibration images based on the camera parameter, and with respect to an arbitrary reference point in one calibration image of the two calibration images, Whether or not there is an error in the camera parameter by calculating a deviation amount of the corresponding point corresponding to the reference point in the other calibration image in the at least one direction and comparing the deviation range with the deviation amount. An image processing apparatus characterized by determining whether or not.   The image processing apparatus according to claim 1, further comprising an information presentation unit that presents information indicating a determination result by the determination unit. A parameter update processing unit that calculates a correction amount of the translation parameter based on the shift range and the shift amount when the determination unit determines that an error has occurred in the camera parameter;
The image processing apparatus according to claim 1, wherein the parameter update processing unit is configured to perform correction of the correction amount on the translation parameter. A parallax calculator that calculates parallax between the two calibration images;
A three-dimensional coordinate value calculation unit that calculates a three-dimensional coordinate value of a measurement point based on a calculation result by the parallax calculation unit;
The determination unit determines whether or not the three-dimensional coordinate value is an abnormal value, and when determining that the three-dimensional coordinate value is an abnormal value, whether or not an error has occurred in the camera parameter. The image processing apparatus according to claim 1, wherein the determination is performed. An error determination method by an image processing apparatus,
An image acquisition step of acquiring at least two viewpoint images;
An image calibration step for generating two calibration images from the two viewpoint images based on camera parameters including a translation parameter and a rotation parameter;
Determining whether or not an error has occurred in the camera parameter,
In the image calibration step, the translational deviation in at least one of the longitudinal direction and the lateral direction according to the translation parameter is not calibrated, and the rotational deviation according to the rotational parameter is calibrated. Produce a calibration image,
The determining step calculates a deviation range in at least one of a longitudinal direction and a short direction in the two calibration images based on the camera parameters, and calibration of one of the two calibration images. Calculating a deviation amount in the at least one direction of a corresponding point corresponding to the reference point in the other calibration image with respect to an arbitrary reference point in the image, and comparing the deviation range and the deviation amount, And determining whether an error has occurred in the camera parameter.

Images