JP2013178656A - Image processing device, image processing method, and image processing program - Google Patents

Abstract

PROBLEM TO BE SOLVED: To provide a technique capable of highly accurately detecting correspondence between a plurality of images.SOLUTION: An image processing device includes: a specific shape extraction unit 111 that extracts a specific shape in each of two left/right photographic images on the basis of feature points extracted from each of the two photographic images; and a correspondence identification unit 115 that identifies correspondence between the two photographic images on the basis of the extracted specific shapes. The specific shape includes at least one of a straight line and a curve, the straight line is identified by end points at both ends thereof, and the curve is identified by end points at both ends thereof and separation points separating an interval between the end points at both ends.

Description

  The present invention relates to an image processing apparatus, an image processing method, and an image processing program having a function of obtaining a correspondence relationship between a plurality of images.

  A technique for obtaining three-dimensional data of a photographing object using the principle of photogrammetry is known (for example, see Patent Document 1). In this technique, a technique for identifying corresponding portions in two photographic images obtained by photographing an object from two different viewpoints is important. That is, it is important to specify which part in the first photographic image corresponds to which part in the second photographic image. Hereinafter, this technique is collectively referred to as stereo matching.

  Template matching is known as a technique for automatically performing stereo matching. Template matching is a method in which the coordinate data of images in two coordinate systems are compared with each other, and the correspondence between the two images is obtained by the correlation between the two. In template matching, the template image is moved within a larger search range in the input image, and the position of the template image that maximizes the cross-correlation function (that is, the maximum degree of correlation) is obtained. This is a technology performed by software processing. In addition, there is a technique in which an operator manually designates several feature points and obtains a correspondence relationship between feature points of two images by template matching using the feature points as a reference. In this method, it is possible to reduce unnecessary processing by manually inputting initial values, and it is possible to increase the accuracy of final matching if the matching accuracy of manually designated feature points is high.

JP 2005-308553 A

  Stereo matching that is performed only by calculation processing without inputting an initial value has a problem that matching accuracy is low and calculation time is long. On the other hand, the method of manually designating corresponding points and then performing stereo matching by calculation is heavy on the operator and is easily influenced by the skill of the operator. In addition, the method in which the operator designates the initial value has a problem that the accuracy of the final matching is remarkably lowered when there is an error in the operator's work.

  In such a background, an object of the present invention is to provide a technique capable of detecting a correspondence relationship between a plurality of images with high accuracy.

  According to the first aspect of the present invention, the specific shape viewed from the first viewpoint and the second viewpoint are based on the feature points of the measurement object viewed from the first viewpoint and the second viewpoint. A specific shape extracting unit that extracts a specific shape, a correspondence relationship specifying unit that specifies a correspondence relationship between the specific shape viewed from the first viewpoint and the specific shape viewed from the second viewpoint, The specific shape includes at least one of a straight line and a curve, and the straight line is specified by endpoints at both ends thereof, and the curve is defined by a dividing point that divides between the endpoints at both ends and the endpoints at both ends. An image processing apparatus characterized by being specified.

  According to the first aspect of the present invention, a specific shape is converted into data as a straight line, a curve, or a combination of a straight line and a curve, a straight line is specified by the end points, and the curve is defined by the end points and the end points of both ends. Since it is specified by the dividing points that divide the space, highly accurate matching can be performed with a small amount of data.

  For example, in the case where a straight line is handled as a set of a large number of feature points distributed linearly, a case will be considered in which the correspondence between the straight line in the left photograph image and the straight line in the right photograph image is examined. In this case, since a straight line is handled as a plurality of feature points distributed in a linear shape, the amount of data is large and the amount of calculation is large compared to the case of the present invention. In addition, in the method of specifying a straight line by a plurality of feature points distributed in a linear shape, a large number of the same points can be obtained even if there is a deviation in the extending direction of the straight line. May be identified. On the other hand, in the method of the present invention, since the straight line is grasped by the end points at both ends, the amount of data handled can be reduced. In addition, since the correspondence is specified by matching the end points, the possibility that the correspondence is specified in a state where there is a deviation in the extending direction is low.

  In the first aspect of the present invention, the feature point includes a case where the feature point is a feature point extracted from a photographic image and a case where the feature point is a feature point obtained from 3D point cloud position data obtained from a laser scanner. It is. Also, a photograph is taken from the first viewpoint, and a feature point is extracted from the first viewpoint by image analysis processing and used as one feature point data. It is also possible to acquire position data and handle the feature point data obtained there as data of the other feature point.

  The invention according to claim 2 is characterized in that, in the invention according to claim 1, the feature point of the measurement object viewed from the first viewpoint and the second viewpoint is extracted from each of the two photographic images. A point extraction unit is provided.

  The invention according to claim 3 is the invention according to claim 1 or 2, wherein the feature point constituting the specific shape viewed from the first viewpoint and the specific point viewed from the second viewpoint. Based on the parallax estimated by the parallax estimated by the parallax estimator and the parallax estimated by the parallax estimator, the search range for specifying the correspondence is set based on the parallax estimated by the feature points constituting the shape And a search range setting unit.

  According to the third aspect of the present invention, the deviation between the screen when the measurement object is viewed from the first viewpoint and the screen when the measurement object is viewed from the second viewpoint is estimated as parallax. Based on this, a range for searching for correspondence is determined. According to this method, the search range is narrowed, so that unnecessary calculations are reduced and the calculation efficiency can be increased.

  According to a fourth aspect of the present invention, in the third aspect of the present invention, the parallax estimation unit is calculated based on a plurality of calculated parallax values obtained from a spread area including the focused feature point. It is characterized by that. According to the invention described in claim 4, since the parallax is estimated based on the data obtained from the area having a certain extent without being based on the pinpoint data, the error of the estimated parallax value is suppressed. Can do. As a method for estimating the parallax, a nearest neighbor method or a linear interpolation method can be cited. Examples of the linear interpolation method include a method of calculating a simple average using the parallax of pass points of n nearby points, and a method of calculating a weighted average according to a distance described later. Further, as a method of estimating the parallax, there are a method of using a triangular parallax including a target feature point after TIN generation, a method of estimating a parallax in a Voronoi region, and the like.

  According to a fifth aspect of the present invention, in the fourth aspect of the invention, the disparity estimation unit includes a plurality of feature points from around a specific feature point that forms the specific shape viewed from the first viewpoint. As a first group pass point, and a second group pass point corresponding to the first group pass point is extracted from the characteristic points of the measurement object viewed from the second viewpoint. Calculating a parallax Dn for each corresponding pass point of the first group of pass points and the second group of pass points, and using a weight corresponding to the distance from the specific feature point And calculating the parallax estimate D based on the result.

  According to the fifth aspect of the present invention, even if the calculated value of each Dn is inaccurate and includes an error, the error is averaged, and further, due to the difference in distance from the feature point of interest. The increase in error is suppressed, and the estimated value D of parallax can be estimated with high accuracy. For example, in the case of FIG. 9 as an example, D1 to D3 include an error that has shifted in a direction larger than the true parallax value and an error that has shifted in a smaller direction. Therefore, by calculating the average value, an error that is shifted in the vertical direction from the true value is canceled, and an estimated value of parallax closer to the true value is obtained. Further, since the influence of an error factor other than the error factor relating to F becomes stronger when moving away from the feature point F of interest, if the average is simply taken, the accuracy of the value of D decreases due to the influence. Therefore, by calculating a weighted average in a state where a weight according to the distance from F is applied, the influence of the error factor generated according to the distance from F can be suppressed and the estimation accuracy of D can be improved.

  The invention according to claim 6 is the invention according to any one of claims 1 to 5, wherein a lattice setting unit that sets a lattice in a first photographed image photographed from the first viewpoint; A divided shape forming unit that forms a divided shape based on feature points that constitute the specific shape specified by the correspondence specifying unit in the lattice, and the divided shape formed by the divided shape forming unit is interpolated in the lattice. The grid in the first image is deformed in the second captured image captured from the second viewpoint based on the segmented shape interpolation unit and the inclination of the segmented shape interpolated in the grid with respect to the grid. A deformed grid setting unit for setting the deformed grid, and a corresponding grid check for detecting a correspondence between the grid points of the grid in the first captured image and the grid points of the deformed grid in the second captured image And a projecting portion.

  According to the invention described in claim 6, in the matching process between the set grid points in the left and right photographic images, the effect of the distortion of the grid shape caused by the parallax is taken into account, and the grid set in the other photographic image Is transformed. For this reason, the processing accuracy of matching between lattice points is increased.

  The invention according to claim 7 is the invention according to claim 6, wherein the lattice setting means sets the first lattice of the first lattice interval in the processing of the first layer, and the second layer In the processing, a second lattice having a lattice interval narrower than the first lattice interval is set, and the modified lattice setting unit deforms the lattice with respect to the first lattice interval and the second lattice interval. And the corresponding grid point detection unit detects the corresponding grid points with respect to the first grid interval and the second grid interval.

  According to the seventh aspect of the present invention, since the lattices whose intervals are gradually reduced over a plurality of hierarchies are set, matching of finer portions is performed in stages. For this reason, processing efficiency can be improved. For example, when matching up to a minute portion at a stroke, the burden imposed on the arithmetic unit is large, and there is a high possibility that an error will occur or an error will increase in the middle. This is advantageous because the process is performed automatically.

  The invention according to claim 8 is characterized in that, in the process according to claim 7, in the process of the specific hierarchy, in addition to the corresponding points specified in the previous hierarchy, the feature points constituting the specific shape are An initial value adding unit that is added as an initial value is further provided. According to the eighth aspect of the present invention, in the matching process in each layer, the feature point for which the matching between the approximate shapes is specified is used as the initial value. And the phenomenon that matching is greatly shifted is suppressed.

  The invention according to claim 9 is the invention according to claim 1, wherein at least one of the feature points of the measurement object viewed from the first viewpoint and the second viewpoint is measured by a laser scanner. It is point cloud position data.

  According to a tenth aspect of the present invention, the specific shape viewed from the first viewpoint and the second viewpoint are based on the feature points of the measurement object viewed from the first viewpoint and the second viewpoint. A specific shape extraction step for extracting a specific shape; a correspondence relationship specifying step for specifying a correspondence relationship between the specific shape viewed from the first viewpoint and the specific shape viewed from the second viewpoint; The specific shape includes at least one of a straight line and a curve, and the straight line is specified by endpoints at both ends thereof, and the curve is defined by a dividing point that divides between the endpoints at both ends and the endpoints at both ends. An image processing method characterized by being specified.

  The invention according to claim 11 is a program that is read and executed by a computer, and the first viewpoint is based on feature points of the measurement object viewed from the first viewpoint and the second viewpoint. A specific shape extracting step for extracting a specific shape viewed from the second viewpoint and a specific shape viewed from the second viewpoint; a specific shape viewed from the first viewpoint; and a specific shape viewed from the second viewpoint A correspondence specifying step for specifying a correspondence relationship with a shape, wherein the specific shape includes at least one of a straight line and a curve, the straight line is specified by end points at both ends thereof, and the curve is An image processing program characterized by being specified by an end point at both ends and a dividing point for dividing between the end points at both ends.

  According to a twelfth aspect of the present invention, an image viewed from the first viewpoint and an image viewed from the second viewpoint based on the feature points of the measurement object viewed from the first viewpoint and the second viewpoint. A correspondence relationship identifying unit that identifies a correspondence relationship with the grid, a lattice setting unit that sets a lattice in the first captured image captured from the first viewpoint, and the lattice viewed from the first viewpoint. A divided shape forming unit that forms a divided shape based on a feature point of the first measurement object; a divided shape inserting unit that interpolates the divided shape formed by the divided shape forming unit into the lattice; and A deformed grid setting unit that sets a deformed grid obtained by deforming the grid in the first image in the second captured image captured from the second viewpoint based on the inclination of the divided shape with respect to the grid. And lattice points of the lattice in the first captured image and the second An image processing apparatus, characterized in that it comprises a corresponding lattice point detecting unit for detecting a correspondence between the grid points of the deformed grid in the shadow image.

  In the invention described in claim 12, the method for obtaining the initial value of the matching point is not limited. That is, the corresponding point specifying unit may obtain the initial value of the matching point by the method described in claim 1 or may obtain the initial value of the matching point by another method. As another method, there is an OCM method (Orientation Code Matching).

  According to a thirteenth aspect of the present invention, in the invention according to the twelfth aspect, the lattice setting means sets the first lattice having the first lattice interval in the processing of the first layer, and the second layer. In the processing, a second lattice having a lattice interval narrower than the first lattice interval is set, and the modified lattice setting unit deforms the lattice with respect to the first lattice interval and the second lattice interval. And the corresponding grid point detection unit detects the corresponding grid points with respect to the first grid interval and the second grid interval. In the invention described in claim 14, in the invention described in claim 13, in the processing of the specific hierarchy, in addition to the corresponding point specified in the previous hierarchy, the correspondence is specified by the correspondence specifying part. An initial value adding unit for adding a feature point as an initial value is further provided.

  ADVANTAGE OF THE INVENTION According to this invention, the technique which can perform the detection of the correspondence between several images with high precision is provided.

It is a block diagram of an embodiment. It is a flowchart which shows an example of the procedure of the process in embodiment. It is explanatory drawing explaining a back intersection method. It is explanatory drawing explaining the forward meeting method. It is a block diagram of the matching process part in embodiment. It is a principle figure explaining the principle of the process which extracts a specific shape. It is a principle figure explaining the principle of the process which extracts a specific shape. It is a principle figure explaining the principle which estimates parallax FIG. 5 is a principle diagram showing a state in which left and right photographic images are overlaid on the principle of performing parallax estimation. It is a principle figure which shows the state which performs the matching between the feature points which comprise the template shape on either side. It is a principle figure which shows the principle of template matching. It is a flowchart which shows an example of the procedure of a matching process. It is an image figure which shows the content of the matching process in steps. It is an image figure which shows the content of the matching process in steps. It is an image figure which shows the content of the matching process in steps. It is an image figure which shows the relationship between the grating | lattice and the triangle which comprises TIN. It is a block diagram of an embodiment.

1. First embodiment (overall configuration)
In the following, there will be described an apparatus having a function of preparing two left and right cameras, taking a left photographic image and a right photographic image at the same time, and specifying a correspondence relationship between the left and right photographic images. In this example, a case will be described in which the positions and orientations of the two cameras are adjusted so that a deviation corrected image is obtained in advance. Of course, a method of correcting a photographed image into a displacement corrected image is also possible. Alternatively, a method may be used in which a left camera image and a right camera image are obtained by performing shooting at different positions using a single camera.

  FIG. 1 shows a block diagram of a system according to an embodiment. FIG. 1 shows an image processing apparatus 100. The image processing apparatus 100 is connected to a left camera 200 and a right camera 300 for taking a stereoscopic photograph. The left camera 200 and the right camera 300 are digital still cameras, for example. The left camera 200 and the right camera 300 are arranged with a predetermined interval (distance between left and right) in order to take a pair of left and right photographic images constituting a stereo image, and shoot a subject to be photographed in duplicate. It is a positional relationship. This is the same as the setting for taking a normal stereoscopic photograph. The left and right cameras may be configured such that the cameras are individually arranged as shown, or may be configured using dedicated cameras for stereo photography. It is assumed that the left camera 200 and the right camera 300 are calibrated in advance, and the internal orientation elements and the vertical parallax between the cameras are known.

  The image processing apparatus 100 is configured using a general-purpose computer having a CPU, a RAM, a ROM, a hard disk device, and various interfaces. The image processing apparatus 100 includes the following functional units configured as software. Note that the functional units shown below are not limited to those configured as software, and at least a part thereof may be configured with dedicated hardware. In addition, the image processing apparatus 100 can be configured not by a general-purpose computer but by dedicated hardware including the illustrated function unit.

  The image processing apparatus 100 includes a photographic image acquisition unit 101, an external orientation element acquisition unit 102, a feature point extraction unit 103, a matching processing unit 104, and a feature point coordinate calculation unit 105. The photographic image acquisition unit 101 captures image data of left and right photographic images taken by the left camera 200 and the right camera 300 into the image processing apparatus 100. The external orientation element acquisition unit 102 acquires the external orientation elements of the left camera 200 and the right camera 300, that is, the position and orientation data of the left camera 200 and the right camera.

  The feature point extraction unit 103 extracts feature points of the shooting target object from each of the left and right photographic images by software processing. A feature point is a point that can be distinguished from other parts as a feature of an image in a photographic image. For example, a feature point is a portion that can be recognized as an edge portion or a point of a captured object. For extracting feature points, differential filters such as Sobel, Laplacian, Prewitt, and Roberts are used.

  The matching processing unit 104 detects corresponding feature portions in the paired left and right photographic images by a method to be described later, and specifies the corresponding relationship. The matching processing unit 104 has the configuration shown in FIG. Details of the matching processing unit 104 shown in FIG. 5 will be described later.

  The feature point coordinate calculation unit 105 calculates the three-dimensional coordinates of the feature points extracted from the pair photo images using the forward intersection method. The forward intersection method is a method of observing a direction from two or more known points to an unknown point and determining the coordinates of the position of the unknown point as an intersection of these direction lines. By calculating the three-dimensional coordinates of the feature points of the photographing object, the photographing object can be converted into three-dimensional data.

(Example of processing)
Hereinafter, an example of processing performed in the image processing apparatus 100 will be described. FIG. 2 is a flowchart illustrating an example of a processing procedure performed in the image processing apparatus 100. A program for executing the flowchart of FIG. 2 is stored in an appropriate memory area included in the image processing apparatus 100 and executed by the CPU included in the image processing apparatus 100. Note that a form in which a program for executing the flowchart of FIG. 2 is stored in an appropriate external storage medium and provided to the image processing apparatus 100 from the storage medium is also possible. This also applies to a program that executes a flowchart described later.

  In the process shown in FIG. 2, first, an external orientation element is acquired (step S201). If an external orientation element has already been acquired, step S201 is not performed, and the processing from step S202 onward is executed. In step S201, the position and orientation values of the left camera 200 and the right camera 300 are acquired. This process is performed in the external orientation element acquisition unit 102. In this process, the external orientation elements of the left camera 200 and the right camera 300 are calculated from an image obtained by capturing a reference point whose position is clearly known.

  An example of the process performed in step S201 will be described. First, a reference point is set in the real space to be measured, and the three-dimensional coordinates of the reference point in the real space are measured using a total station or GPS. Alternatively, a reference point whose three-dimensional coordinates in the real space are known is prepared. Next, the reference point is photographed by the left camera 200 and the right camera 300, and an external orientation element of the camera is calculated by a backward intersection method based on the three-dimensional coordinates of the reference point and the image coordinates in each photographic image. Alternatively, a scale in which two points with known positional relationships are determined may be prepared, and the external orientation elements of the cameras 200 and 300 may be calculated based on the captured image of this scale. The external orientation element obtained in this case is in the local coordinate system.

Hereinafter, an example of a procedure for calculating the external orientation elements of the left camera 200 and the right camera 300 will be described. FIG. 3 is an explanatory diagram for explaining the backward intersection method. The backward intersection method is a method of observing directions from the unknown point O toward three or more known points P ₁ , P ₂ , P ₃ and obtaining the position of the unknown point O as an intersection of these direction lines.

For example, focus on the left camera 200. Here, it is assumed that the position of the left camera 200 is O (X ₀ , Y ₀ , Z ₀ ), and reference points whose positions are known are taken from P ₁ , P ₂ , and P ₃ . First, the camera coordinate system is assumed to be x, y, z, the photographic coordinate system x, y, and the reference coordinate system X, Y, Z which is the coordinate system of the measurement object. It is assumed that photographing is performed in a direction in which the camera is sequentially rotated counterclockwise by ω, φ, and κ counterclockwise with respect to the positive direction of each coordinate axis. Then, the four image coordinates (only three points are shown in FIG. 3) and the corresponding three-dimensional coordinates of the reference point are substituted into the quadratic projective transformation equation shown in the following equation 1, and an observation equation is established to set the parameter b1. -B8 is obtained.

Then, using the parameters b1 to b8 of Equation 1, external orientation elements (X ₀ , Y ₀ , Z ₀ , ω, φ, κ) are obtained from Equation 2 below. Here, (X ₀ , Y ₀ , Z ₀ ) are coordinates indicating the position of the left camera 20, and (ω, φ, κ) is the posture (orientation) of the left camera 200 at that position.

  As described above, the external orientation element of the left camera 200 using the reference point whose position is clearly known is calculated. Further, the external orientation element of the right camera 300 is calculated by a similar method. The above is a detailed example of the process performed in step 201 of FIG.

  When the external orientation element is acquired, a left photographic image is acquired from the left camera 200, and a right photographic image is acquired from the right camera 300 (step S202). This process is performed in the photographic image acquisition unit 101. Here, the left and right photographic images are acquired as displacement correction images. The displacement corrected image is an image in which left and right photographic images are aligned on an epipolar line (on the Z axis line of the model coordinate system (on the axis line in the depth direction of the object to be photographed)) and has an inclination with respect to the model coordinate system. There are no photographic images. For example, left and right deviation correction images are arranged side by side, and the left deviation correction image is selectively visually recognized by the left eye using deflection glasses or the like, and the right deviation correction image is selectively visually recognized by the right eye. Can be seen.

  Next, feature points in each of the acquired left and right photographic images are extracted (step S203). This process is performed in the feature point extraction unit 103. Filters such as Moravec, Laplacian, and Sobel are used for feature point detection. Details of the method of extracting feature points will be described later.

  After the feature points are extracted, matching processing is performed based on the feature points to obtain the correspondence between the left and right photographic images (step S204). Details of the processing in step S204 will be described later. After performing the matching process, the three-dimensional coordinates of the corresponding feature points between the left and right photographic images are calculated (step S205).

Hereinafter, an example of a detailed procedure of the process performed in step S205 will be described. Here, an example will be described in which the forward intersection method is used to calculate the three-dimensional coordinates of the feature points whose correspondence is specified in the left and right photographic images. FIG. 4 is an explanatory diagram for explaining the forward meeting method. The forward intersection method is a method of observing a direction from _two known points (O ₁ , O ₂ ) toward the unknown point P and obtaining the position of the unknown point P as an intersection of these direction lines. In FIG. 4, the coordinate system of the object space is O-XYZ. Here, the coordinates (X ₀₁ , Y ₀₁ , Z ₀₁ ) of the projection center O ₁ of the camera on the right image and the tilts (ω ₁ , φ ₁ , κ ₁ ) of the camera coordinate axis, and the projection center O ₂ of the camera on the left image. The coordinates (X ₀₂ , Y ₀₂ , Z ₀₂ ) and the tilt (posture) (ω ₂ , φ ₂ , κ ₂ ) of the camera coordinate axes have already been acquired and known in step S201. Also, the internal orientation elements (focal length, principal point position, lens distortion coefficient) are known.

Here, the feature point p ₁ (x ₁ , y ₁ ) on the right photographic image and the corresponding feature point p ₂ (x ₂ , y ₂ ) on the left photographic image are obtained from the left and right photographic images. It is done. Therefore, the three-dimensional coordinates (X, Y, Z) of the feature point P, which is an unknown point in the target space, are determined as the intersection of the light beam O ₁ p ₁ and the light beam O ₂ p ₂ . However, in reality, there is an error, and the two light beams do not accurately intersect, so the intersection position is obtained by the least square method. Specifically, two collinear conditional expressions (Formula 3) are established. A known external orientation element, internal orientation element, and image coordinates of corresponding points are substituted for this.

  Further, the approximate value (X ′, Y ′, Z ′) of the unknown point P plus the correction amount (X ′ + ΔX, Y ′ + ΔY, Z ′ + ΔZ) is substituted into this collinear conditional expression. Then, Taylor expansion is performed around the approximate value to linearize, and the correction amount is obtained by the least square method. The approximate value is corrected by the obtained correction amount, and the same operation is repeated to obtain a converged solution. By this operation, the three-dimensional coordinates P (X, Y, Z) of the feature points are obtained. This calculation is performed for all feature points, and the three-dimensional coordinates of a plurality of feature points whose correspondences are specified in the left photograph image and the right photograph image are calculated. The above is a detailed example of the contents of the process performed in step S205. In this way, the three-dimensional coordinate data of the photographed object is obtained using the left photograph image photographed by the left camera 200 and the right photograph image photographed by the right camera 300.

(Extraction of feature points)
Hereinafter, a detailed example of the process of step S203 performed in the feature point extraction unit 103 will be described. Extraction of feature points is performed using a feature amount called OCR (Orientation Code Richness). First, a parameter called direction code θ (x, y) is introduced. The direction code θ (x, y) is a parameter for evaluating the change in the lightness gradient in the surrounding direction from the straight point. The direction code θ (x, y) is defined as follows. First, the brightness at the pixel (x, y) of the target image is I (x, y), the horizontal gradient is ▽ I _x = ∂I / ∂x, and the vertical gradient is ▽ I _y = ∂I / ∂. y. Here, the parameter indicating the gradient direction in the pixel of interest is the direction code θ (x, y). The direction code θ (x, y) is calculated by θ (x, y) = tan ⁻¹ (▽ I _y / ▽ I _x ). A Sobel filter is used to calculate the gradient direction. With the direction code θ (x, y), the direction of the lightness gradient at the point of interest can be evaluated.

The direction code θ (x, y) is quantized with an appropriate quantization width Δθ = 2π / N. A quantized version of this direction code θ (x, y) is denoted by c _xy ( _Equation 4).

For example, if N = 16, the direction is quantized at an angular interval of 22.5 degrees. Incidentally, ▽ the threshold is the sum of the absolute value of the absolute value and ▽ I _y of I _x, assign N in the direction of low contrast as invalid sign.

Then, a local region I _xy having a size of M × M pixels centered on the pixel (x, y) of interest is considered, and the appearance frequency of each direction code included in the region is expressed as h _xy (i) (i = 0,1,2, ...). Then, the relative frequency of h _xy (i), that is, the ratio at which each direction code appears in the local region I _xy is calculated. This relative frequency P _xy is calculated by the following _equation 5.

Here, h _xy (N) is a value corresponding to low contrast brightness, and is excluded from the relative frequency P _xy . Next, the entropy E _{xy of the} direction code is calculated by _Equation 6 in order to extract a region where the distribution of changes in the gradient direction is more random as a feature point.

The nominal maximum value Emax of entropy calculated by Equation 7 is Emax = log ₂ N when each direction code follows a uniform distribution P _xy (i) = 1 / N. Here, a threshold α _e (0 <α _e <1) is set, a group having an entropy equal to or greater than α _e Emax is considered as a feature region, and richness R _xy that is a parameter indicating the feature region is expressed by Equation 7.

The richness R _xy is calculated for the entire image to create an OCR image. Then, the OCR image is gridded with a predetermined width, and a point where the richness R _xy in each grid shows the maximum value is extracted as a feature point. As a result of the above processing, a region having a more gradual distribution of gradient direction changes, in other words, a region having a larger distribution of gradient changes, is extracted as a feature point.

(Details of matching processing section)
Hereinafter, an example of a detailed configuration of the matching processing unit 104 in FIG. 1 will be described. FIG. 5 is a block diagram showing details of the matching processing unit 104. As shown in FIG. 5, the matching processing unit 104 includes a specific shape extraction unit 111, a template shape setting unit 112, a parallax estimation unit 113, a search range setting unit 114, and a correspondence relationship specifying unit 115.

  The specific shape extraction unit 111 extracts a specific shape from a plurality of feature points extracted from the photographic image. This process is performed for each of the left and right photographic images. Specific shapes include straight lines, curved lines, polygonal shapes, circles, ellipses, combinations thereof, and the like. Hereinafter, details of processing performed by the specific shape extraction unit 111 will be described. 6 and 7 are principle diagrams for explaining processing for extracting a specific shape. FIG. 6 shows a case where a straight line is extracted as the specific shape, and FIG. 7 shows a case where a curve is extracted as the specific shape.

  For example, consider a case where the linear shape shown in FIG. 6A can be extracted from a set of feature points extracted from a photographic image. In this case, the specific shape extraction unit 111 first removes a line having a large number of end points with respect to the length using a threshold value, and leaves only a straight line having a certain length. That is, a shape with many bends is excluded, and a straight line having a certain length is extracted. Next, end point data at both ends of the remaining straight line is acquired (FIG. 6B). In this case, as shown in FIG. 6B, the shape of a straight line is specified as data by two end points. Such processing is performed in order to reduce the amount of data to be handled and to improve the processing efficiency.

  For example, when a curve is extracted from a collection of feature points as shown in FIG. 7A, the specific shape extraction unit 111 first acquires the end points of the curve (FIG. 7B). Next, a straight line connecting the two end points is set, a perpendicular of the straight line is set from each feature point constituting the curve excluding the end points, and the length thereof is calculated (FIG. 7C). Then, a dividing point D is set in the portion of the curve R where the distance d between the curve R and the straight line L is maximum (FIG. 7D), and a finer method is performed between the dividing point D and the end point by the same method. A division point is set (FIG. 7E). This is repeated until d falls below a predetermined threshold. In this way, a shape obtained by approximating the curve with a plurality of straight lines is obtained as shown in FIG. Here, each straight line is acquired as data in the same manner as in the case of a straight line connecting two end points shown in FIG. That is, the curve R is specified as data by the end points and the division points. Although the description is omitted in FIG. 7, in the case of a curve, similarly to the case of the straight line in FIG. 6, in order to limit the amount of data to be handled, a short-length curve is eliminated by a threshold. .

  Then, the outline of the specific shape that forms the object to be photographed can be obtained by grasping the straight line by specifying the end point shown in FIG. 6B and the method of grasping the curve shown in FIG. 7F by the end point and the dividing point. It is converted into data. The above process is performed by the specific shape extraction unit 111.

  The template shape setting unit 112 selects one or a plurality of specific shapes extracted by the specific shape extraction unit 111 and sets them as a template shape. The template shape is set for each of the left and right photographic images. FIG. 13 shows an example of the template shape. Although it is not clear in FIG. 13, the shape of FIG. 13 is a shape obtained by simplifying the actual shape and roughly grasping the shape, and the shape is composed of a combination of straight lines, a combination of straight lines and curves, or a combination of curves. Yes. And the straight line and / or curve used as the element which comprises the said template shape are specified by the method using the end point and division | segmentation point shown to FIG. 6 (B) and FIG. 7 (F).

  The parallax estimation unit 113 estimates the parallax between the left and right photographic images. Hereinafter, an example of processing performed by the parallax estimation unit 113 will be described. FIG. 8 is a principle diagram for explaining the principle of performing parallax estimation. FIG. 8 shows the feature points F of the left photographic image and the feature points F ′ of the right photographic image corresponding to the endpoints of FIG. 6, the endpoints and the dividing points of FIG. 7. Note that, at this stage, the feature point F and the feature point F ′ are in a state where no correspondence is specified.

  Here, the parallax estimation unit 113 performs the following processing. First, pass points P1 to P3 that can be used as feature points are extracted from the vicinity of the feature point F in the left photograph image. The pass points P1 to P3 are used as auxiliary feature points for calculating the parallax between the left and right photographic images related to the feature point F. The pass points P1 to P3 are extracted from positions where the distance from the feature point F is several to several thousand times the resolution of the image. This process is performed by the function of the feature point extraction unit 103 in FIG. The number of pass points is not limited to three, but may be two or more. Increasing the number of pass points improves the accuracy of the calculation, but increases the processing load. Therefore, the number of pass points is selected in view of the hardware calculation speed and the allowable calculation time. Here, the extraction of the pass points P1 to P3 is performed focusing on the left photograph image, but the path points may be extracted focusing on the right photograph image.

Next, detection of the pass point of the right photographic image corresponding to the pass points P1 to P3 of the left photographic image is performed. This processing is performed using LSM (least square) matching or template matching. Hereinafter, LSM (least square) matching will be described. First, let t ₁ (i, j) be a template for pattern comparison in the LSM method, f (x, y) be a modified matching window, and approximate the deformation of the matching window by the affine transformation of Eq. .

  Here, the density difference r (i, j) in the pixel to be compared is given by the following equation (9).

r (i, j) represents the difference in shading between the template and the matching window after geometric transformation. Then, parameters t _{0 to} t ₅ are obtained so as to minimize the square sum of the density differences shown in Equation ₁₀ .

Here, t ₁ , t ₂ , t ₄ , and t ₅ represent the deformation of the matching window, and t ₀ and t ₃ are the coordinates of the detection position to be obtained.

Next, an example of template matching will be described. Template matching is a method in which the coordinate data of images in two coordinate systems are compared with each other, and the correspondence between the two images is obtained by the correlation between the two. In template matching, a correspondence relationship between feature points of an image viewed from two viewpoints is obtained. FIG. 11 is an explanatory diagram for explaining the principle of template matching. In this method, as shown in the drawing, a template image of N ₁ × N ₁ pixel is moved on a search range (M ₁ −N ₁ +1) ² in an input image of M ₁ × M ₁ pixel larger than that, The upper left position of the template image is calculated so that the cross-correlation function C (a, b) expressed by the following formula 11 is maximized (that is, the degree of correlation is maximized).

  The above processing is performed while changing the magnification of one image and rotating it. And the area | region where both images correspond is calculated | required on the conditions from which the correlation became the maximum, Furthermore, the corresponding point is detected by extracting the feature point in this area | region.

  By the method described above with an example as described above, the correspondence between the pass points in the left and right photographic images, that is, the correspondence between P1 and P1 ', P2 and P2', and P3 and P3 'is obtained. At this stage, the correspondence between the pass points in the left and right photographic images includes a certain error.

Next, the parallax between the corresponding pass points in the left and right photographic images, that is, the difference Dn (D ₁ , D ₂ , D ₃ ) between the photographic coordinate positions of the corresponding pass points when the two photographic images are overlapped. calculate. FIG. 9 is a principle diagram showing a state in which left and right photographic images are superimposed.

  Here, the position of the pass point P1 in the left photographic image and the position of the pass point P1 'in the right photographic image can be obtained as coordinates in the photographic image. Therefore, with the correspondence relationship between P1 and P1 'specified, the position difference D1 in the photographic image between P1 and P1' can be calculated as the parallax. The parallax D2 between P2 and P2 'and the parallax D3 between P3 and P3' are calculated by the same method. Then, the weighted average D of parallax is calculated by performing weighting according to the distance Ln from the focused feature point F using the following formula 12.

  Equation 12 has the following meaning. First, as described above, the parallax between the pass points between the left and right photographic images (for example, P1 and P1 ') includes a certain error. Further, as the value is farther from F, the tendency to be different from the true parallax related to F (that is, the actual parallax between F and F ′) increases. Therefore, the contribution of Dn is weighted according to the distance from the pass point F, and the weighted average D of Dn is calculated. This D is an estimated value of the parallax between the feature points F and F ′.

  According to this method, even if the calculated value of each Dn is inaccurate and an error is included in the calculated value, the error is averaged, and an increase in error due to a difference in distance from F can be suppressed. It is possible to calculate an estimated value of parallax related to high F and F ′. That is, in the case of FIG. 9, D1 to D3 include an error that is blurred in a direction larger than the true parallax value and an error that is blurred in a small direction. Therefore, by calculating the average value, an error that is shifted in the vertical direction from the true value is canceled, and an estimated value of parallax closer to the true value is obtained. Further, since the influence of an error factor other than the error factor relating to F becomes stronger when moving away from F, if the average is simply taken, the accuracy of the value of D decreases due to the influence. Therefore, by calculating the weighted average in a state where the weight (Ls−Ln) / Ls corresponding to the distance from F is multiplied, the influence of the error factor generated according to the distance from F is suppressed, and D is estimated. Accuracy can be increased. Note that the value of D is likely to be different for each feature point, so it is calculated for each feature point of interest.

  The above is the function of the parallax estimation unit 113. The above parallax estimation is performed for each of a plurality of template shapes set by the template shape setting unit 112. Further, it is performed on at least one of end points and division points (see FIGS. 6 and 7) for specifying each template shape. In the above description, the calculation of Formula 12 is performed focusing on the left photograph image, but the same processing may be performed focusing on the right photograph image. Also, here, the description has been made on the assumption that the parallax in the vertical direction is known, but when the parallax in the vertical direction is unknown, the estimated value of the parallax including the vertical direction is calculated by the same method. This can be done in the parallax estimation unit 113.

  Next, the search range setting unit 114 will be described. The search range setting unit 114 determines a search range between the left and right photographic images of the template shape set by the template shape setting unit 112 based on the parallax value estimated by the parallax estimation unit 113. Here, the search range is set for each feature point (end point and division point) constituting the template shape in the left and right photographic images. In the case of the present embodiment, the vertical parallax is obtained when the left camera 200 and the right camera 300 are oriented. Further, the parallax between the template shapes corresponding to the left and right photographic images is obtained by the parallax estimation unit 113 by the processing according to Equation 12. D in Expression 12 means that the feature points constituting the left and right photographic images are shifted to the left and right by the degree of D. Therefore, by searching in the range of D on the left and right, the template shape of the right image corresponding to the template shape of the left image can be found. Actually, since D is an estimated value and includes an error, a range of D + ΔD (ΔD = D / 4 to D) is set as a search range. When the parallax estimation unit 113 calculates the vertical parallax in addition to the horizontal direction, the search range is also set in the vertical and horizontal directions.

  The correspondence relationship specifying unit 115 searches for the correspondence relationship between the template shape of the left photograph image and the template shape of the right photograph image in the range set by the search range setting unit 114, and performs a process of specifying it. Specifically, the correspondence between the end points and the dividing points constituting the template shape of the left photographic image and the end points and the dividing points constituting the template shape of the right photographic image is searched in the search range, and the correspondence relationship is searched. To identify. FIG. 10 shows the principle of examining the correspondence (matching) between the feature point F1 in the left photographic image and the feature point F1 'in the right photographic image. The feature points F1 and F1 'are one of the end points and division points described with reference to FIG.

  In the example of FIG. 10, the parallax between the feature points F1 and F1 ′ is calculated according to the principle shown in FIGS. 8 and 9, and the search range (D + ΔD) in the figure is set using the value. Yes. According to this method, the search range is limited and narrowed down, and a search for the correspondence (matching property) of the template shapes for which the features are easier to grasp is performed. For this reason, compared with a method in which a large number of feature points are matched with each other, the matching of corresponding shapes can be obtained with high accuracy and with a small amount of calculation.

  The search range is preferably set for each of a plurality of feature points in terms of accuracy. The shape of the template shape may be a closed shape such as a circle or a polygon, or may be a non-closed shape such as a line or a curve. In practice, not only one template shape but also a plurality of template shapes are extracted from the screen, and a combination thereof or each of them is a processing target. The above is the outline of the processing performed in the correspondence relationship specifying unit 115.

  The matching processing unit 104 includes a lattice setting unit 121, a divided shape forming unit 122, a divided shape interpolation unit 123, a modified lattice setting unit 124, a corresponding lattice point detecting unit 125, and an initial value adding unit 126. The lattice setting unit 121 sets a lattice in the first captured image (in this case, the left photograph image) captured from the first viewpoint. The divided shape forming unit 122 forms a divided shape based on feature points constituting the template shape specified by the correspondence specifying unit 115 in the set lattice. As this divided shape, a triangle constituting TIN can be cited. It is also possible to divide into polygons such as quadrilaterals and pentagons other than triangles.

  The divided shape interpolation unit 123 interpolates the divided shape formed by the divided shape forming unit 122 into the lattice. The deformation grid setting unit 124 is based on the inclination with respect to the grid surface of the divided shape interpolated in the grid, in the second captured image captured from the second viewpoint (if the first photographic image is a left photographic image, A deformed grid obtained by deforming the grid in the first image is set in the right photograph image). The corresponding grid point detection unit 125 identifies the correspondence between the grid points of the grid in the first captured image and the grid points of the deformed grid in the second captured image. The initial value adding unit 126 performs a process of adding a feature point constituting the template shape as an initial value in addition to the corresponding point specified in the previous hierarchy in the process of the specific hierarchy.

  For example, the lattice setting unit 121 performs the process according to Step 5 in FIG. 14 and the process according to Step 9 in FIG. For example, the divided shape forming unit 122 performs the process according to Step 5 in FIG. 14 and the process according to Step 9 in FIG. For example, the divided shape interpolation unit 123 performs the process according to Step 5 in FIG. 14 and the process according to Step 9 in FIG. For example, the modified grid setting unit 124 performs the process according to Step 6 in FIG. 14 and the process according to Step 10 in FIG. For example, the corresponding grid point detection unit 125 performs the process according to Step 7 in FIG. 14 and the process according to Step 11 in FIG.

(Example of corresponding point processing)
A detailed example of the process performed in step S204 of FIG. 2 will be described. In this process, a process for specifying a more detailed correspondence between the left and right photographic images is performed. FIG. 12 is a flowchart showing a detailed example of the process performed in step S204 of FIG. 13 to 15 are image diagrams of left and right photographic images showing the processing state in stages. In this process, feature points whose matching is specified by edge matching using a template shape are used as initial values, and the initial values are interpolated into a lattice to generate lattice points, and the other photographic image is generated from the interpolated lattice points. The corresponding grid (search range) is modified, and stereo matching is performed in the modified search range. Further, this series of processing is made one-level processing, and the lattice matching is sequentially reduced to perform stereo matching while gradually narrowing the search range. Also, each time the hierarchy progresses, feature points identified by edge matching using template shapes are added as initial values, and each time the initial value is used for setting a search range, the matching accuracy is improved. It is increasing. Hereinafter, an example of this process will be described in detail.

  When the processing of FIG. 12 is started, extraction of a specific shape based on feature points in the left and right photographic images is first performed (step S401). This processing is performed in the specific shape extraction unit 111 of FIG. 5 according to the principle described in relation to FIGS.

  Next, a template shape is set based on the specific shape extracted in step S401 (step S402, step 1). This processing is performed in the template shape setting unit 112 in FIG. FIG. 13 shows an example of the template shape.

  Next, the parallax between the left and right photographic images at the feature point of interest in the left photographic image and the right photographic image is estimated (step S403, step 2). Here, as the feature point of interest, an end point and a division point that specify the template shape are selected. This process is performed in accordance with the principle shown in FIGS. 8 and 9 in the parallax estimation unit 113 of FIG. Step 2 in FIG. 13 shows an image at this stage.

  When the parallax is estimated in step S403, the search range based on the estimated parallax is set for each feature point of interest (step S404). This processing is performed according to the principle shown in FIG. 10 in the search range setting unit 114 of FIG. Next, the correspondence between the feature points in the left and right photographic images with the estimated parallax as the search range is searched, and the correspondence between the end points constituting the template shape and the division points is specified (step S405, Step 3). This process is shape matching (edge matching). The identification of the correspondence is performed using, for example, the aforementioned least square matching (LSM). At this time, since the correspondence between feature points along a shape that is easy to grasp, such as a straight line or a curve, is searched in a limited range, highly accurate matching is possible. At this time, the matching between the end points and the dividing points, that is, the points between the end points and the dividing points, that is, the matching of all feature points constituting the template shape is examined using the result of matching the end points and the dividing points that specify the template shape.

  Next, the three-dimensional coordinates of the feature points in the left and right photographic images for which the correspondence relationship is specified in step S405 (step 3) are calculated (steps S406 and step 4). This processing is performed in the feature point coordinate calculation unit 105 using the forward intersection method whose principle is shown in FIG. At this time, the calculation of the three-dimensional coordinates is performed not only on the end points and the division points, but also on the points in between, that is, on all the feature points constituting the template shape.

Next, the process of step S407 (S5 in FIG. 14) is performed. In this process, first, a TIN (Triangular Irregular Network) is created using the feature point (this is the initial value) for which the three-dimensional coordinates are calculated in step S406 (step S4). TIN is an approximate image obtained by capturing an image of an object as a set of triangular patches. Also, (in this example, use the left photographic image) on one of the photographic image in a length of one side to form a square lattice of p ^k. Note that k is the number of hierarchies. The number k of layers is the number of times obtained by dividing the smaller number of pixels of H and V from the number of pixels (H, V) of the left and right deviation corrected images by 2, which is 32 pixels or less, that is, min (H, V). It is set as the minimum value of k that satisfies / 2 ^k ≦ 32. The grid interval ^pk is calculated by p ^k = Pi × (dm / B) where Pi is a preset resolution, dm is the parallax at the center coordinates of the grid, and B is the distance between the left and right cameras. The value of p ^k are capable of interpolation to be described later TIN value is adopted.

Then, the created TIN interpolated in a square lattice of lattice spacing p ^k, create a TIN is decorated in a square lattice. In this process, first, Delaunay triangles are obtained by dividing Delaunay triangles on feature points on the straight lines and curves constituting the template shape (step 1 in FIG. 13). Then, the Z value at the lattice point of the obtained Delaunay triangle is internally provided to obtain a triangle having the lattice point as a vertex. At this time, not only the end points and the division points but also the points between them are used as the feature points constituting the template shape.

FIG. 14 shows one of TIN ⁰ and TIN ^{0 of} Delaunay triangles (triangles constituting TIN) created using the feature points (this is the initial value) for which the three-dimensional coordinates have been calculated in step S406 (step S4). The relationship of TIN ^k obtained by interpolating ⁰ into a square lattice is shown. Here, the three vertices of TIN ⁰ are obtained from the feature points whose three-dimensional coordinates are calculated in step S406 (step S4), and the three vertices of TIN ^k are the vertices of the lattice. As shown, the Z value of TIN ⁰ at each vertex of a square grid that is included in the TIN ^0, each vertex of the TIN ^k is specified. Thus, a triangle TIN ^k interpolated in a square lattice is obtained. Step 5 of FIG. 14 shows the TIN ^k and TIN ⁰ formed in this way as a simplified image. Here, only TIN ^k is shown as a triangle interpolated in the lattice, but the same processing is performed in each part (each square lattice) of the lattice. The above is the description of the processing in step S407 for performing interpolation on the square lattice of TIN.

Next, step S408 is performed. Here, TIN ^k is ^obtained by approximating a part of a three-dimensional shape having a depth with a triangle, and its surface does not always coincide with the surface of the photographic image (it may coincide). Therefore, as shown in FIG. 16A, for example, coordinate values Z ^(K) i, j, Z ^(K) i + 1, j, Z ^(K) i + 1 of the three vertices of TIN ^k in the z direction. , j + 1 are generally not the same, and TIN ^k is inclined with respect to the lattice plane. Here, since the three-dimensional coordinates of each vertex of TIN ^k are calculated in step S406, the difference between the orientation of the surface of TIN ^k and the corresponding lattice surface, in other words, the correspondence of TIN ^k . The slope with respect to the square lattice can be calculated quantitatively.

  Here, since the left photograph image and the right photograph image are taken from different viewpoints, the lattice of the right photograph image corresponding to the square lattice set in the left photograph image is not square but is distorted from square. Become. Taking this distortion into account, searching for the correspondence between the lattice points set in the left photograph image and the lattice points set in the right photograph image using LSM or the like enables more accurate stereo matching. . In step S408, the grid set in the right photograph image is deformed in order to realize the above intention.

The deformation of the lattice in the right photograph image is performed as follows. First, a grid similar to that for the left photographic image is introduced into the right photographic image. At this time, the positional relationship between the lattices (mesh) in the left and right photographic images is specified using the feature points for which the correspondence relationship has been specified in step S405 (step 3). Then, the normal vector of the lattice point of the left photograph image is calculated. The normal vector of the lattice point is calculated as the average of the normal vectors of the triangular planes adjacent to the lattice point. Here, since the three-dimensional coordinate of each vertex of the triangle indicated by TIN ^k is known, the normal vector of the surface can be calculated. When the normal vector of each vertex of the square lattice is obtained, the shape of the corresponding lattice in the right photographic image is deformed so as to correspond to the direction of the normal vector of each vertex. The deformation of the corresponding lattice (the lattice of the right photographic image) is performed so that the shift due to the parallax is eliminated. That is, in this case, when the square lattice in the right photographic image is viewed from the viewpoint of the left photographic image, the square lattice in the right photographic image is distorted, but matches (or approximates) this distorted shape. As described above, the grid of the right photographic image is deformed from the square grid. In this way, the lattice set in the right perspective image is deformed based on the inclination of TIN ^k .

  It is an example of the process performed by the above step S409 (step 6 of FIG. 14). Next, the correspondence between the lattice points set in the right photograph image (the lattice points of the deformed lattice) and the lattice points set in the left photograph image is performed by LSM (steps S409 and S7). This matching process is not limited to LSM, and may be performed by other methods such as template matching.

  Next, it is determined whether or not the process performed at that time is the last hierarchy (step S410). If it is the last hierarchy, the process ends. If not, the process proceeds to step S411. move on. FIG. 15 shows a case where the process proceeds to the second layer. In FIG. 15, the feature points matched in step 7 of FIG. 14 are indicated by ●. The matching points indicated by ● are feature points (matching points) whose correspondence is specified in the left and right photographic images by comparing the lattice points. In the figure, in order to simplify the description, the position of the matching point in the right photograph image does not reflect the state of deformation of the lattice shown in step 7 of FIG.

  In step S411, feature points constituting the template shape specified in step S405 (step 3 in FIG. 12) are reintroduced into the left and right photographic images as initial values. At this time, points other than the end points and the division points are also targeted. For example, in step 8 of FIG. 15, the matching point (●) and the initial value (◯) in the previous hierarchy are shown. As will be apparent from the following description, as the hierarchy progresses, the matching points corresponding to ● increase and the left and right photographic images are more precisely matched. In the matching in each layer, by mixing the initial values, the basic (or rough) matching is maintained by the initial values, and the phenomenon that the error increases as the layers progress can be suppressed.

After step S411, a process of setting the lattice interval to half of the previous time is performed (step S412), and the processes after step S407 are executed again. Step 9 and the subsequent steps in FIG. 15 show an example of processing when the lattice spacing is set to half of the previous time. At this time, in TIN ^{k −1 which} is one layer higher than TIN ^k , the Z values of the three vertices are calculated by the relational expression shown in FIG. In this way, a lattice in which the dimension of each side is scaled down to half (area ratio (1/4)) is set, and the same processing as in the previous layer is performed on this lattice. In this case, as shown in step 10, the search range is deformed for a smaller lattice, and matching between lattice points in the left and right photographic images is examined (step 11). In this way, the matching point is specified more finely.

  As can be understood from the comparison between Step 4 in FIG. 13 and Step 8 in FIG. 15, in the next stage after Step 11, since the matching points are found at the lattice points set in Step 9, the number of matching points ( Density) further increases. In this way, the number of matching points increases as the second and third hierarchies pass, and the correspondence between the left and right photographic images is specified more precisely in stages.

(Superiority)
When examining the correspondence between the left and right photographic images, template shapes are extracted from each photographic image, and by examining the correspondence between the template shapes, high matching accuracy can be obtained and the calculation time can be shortened. In addition, when extracting template shapes, the template shape is regarded as a collection of straight lines and curves, the straight line is captured at the end points of both ends, and the curve is captured at the end points of both ends and the dividing points between them, thereby obtaining high calculation accuracy. It is done. In addition, this method can reduce the amount of data to be handled, so that the calculation time can be shortened. Also, by estimating the parallax between the left and right photographic images and determining the search range based on the estimated parallax, the possibility of inaccurate matching is reduced, and the matching accuracy is improved. In addition, the amount of calculation can be greatly reduced by limiting the search range. In addition, the parallax can be estimated with high accuracy by performing the parallax estimation using a weighted average of calculated values at a plurality of pass points selected from around the feature point of interest.

In addition, a feature point whose matching is specified by edge matching using a template shape is used as an initial value, and the initial value is interpolated into a lattice to generate a lattice point. This interpolated lattice point (Z value of TIN ^k ) Based on the above, the corresponding lattice (search range) of the other photographic image is deformed, and stereo matching between lattice points in the deformed search range is performed, so that more detailed matching using the initial value is performed. Then, the density of matching points can be increased step by step by performing this series of processing as one-layer processing and performing stereo matching while sequentially decreasing the lattice interval and gradually narrowing the search range.

  In addition, as the hierarchy progresses, feature points whose matching is specified by edge matching using template shapes are added as initial values, and this initial value is used for matching, thereby suppressing an increase in errors and high matching accuracy. Can be obtained. In other words, in the matching in each layer, the template shape feature points that specify the correspondence of the rough shape of the image content are used as the reference value, so that the details of the shape of the large frame are used as the basis. Part matching is performed in stages. For this reason, as compared to the case where the initial value due to the template shape is not used in each layer, the matching error gradually increases as the layer progresses, and finally the rough contour correspondence is specified by being shifted. Inconvenience can be avoided.

(Other)
In the embodiment described above, an example in which correspondence between two photographic images constituting a stereo image is specified has been described. However, the target photographic image is not limited to two, and is three or more. Also good. In addition, the present invention can be applied to a technique for capturing a moving image and specifying a correspondence relationship among a plurality of frame images constituting the moving image. In addition, the present invention is not limited to use in stereoscopic photogrammetry technology, the correspondence between a plurality of photographic images taken from different viewpoints is specified, and the camera position is the same, but the orientation is changed and taken. The correspondence relationship between a plurality of photograph images can also be used to identify the correspondence relationship between a plurality of photograph images taken by changing at least one of the position and orientation of one camera.

  In the above description, an example in which the lattice of the right photograph image is modified (correction of the search range) using the left photograph image as a reference has been described. However, the lattice of the left photograph image is represented using the right photograph image as a reference. A process of performing deformation (correction of the search range) is also possible.

  In FIG. 12, the process of step S406 may be changed to “calculate the three-dimensional coordinates of the feature points that are matched at that time”, and the process may return to the previous stage of step S406 after step S412. In this case, in the second and subsequent layers, the calculation of the three-dimensional coordinates of the feature points whose correspondence is specified in step S409 (that is, the matching between the left and right photographic images) is performed, and this is the new TIN. Used for creation. Since the density of matching points increases as the hierarchy progresses, in this case, a finer TIN is created as the hierarchy progresses, and the process in which it is interpolated into a grid that gradually becomes finer is performed for each hierarchy. Done in That is, in each hierarchy, a TIN is created each time based on the feature points that are matched at that time and whose three-dimensional coordinates are calculated. This TIN is created in step 5 of FIG. 14 and FIG. A process for interpolating the lattice is performed by the same method as shown in FIG. In this case, the initial value using the template shape and the matching point using the lattice in each layer are newly formed in the subsequent layer, the TIN formation in the subsequent layer, the deformation of the search range (the deformation of the other lattice), and the subsequent layer. Used for matching. For this reason, the precision of matching performed gradually in steps is further increased.

2. Second Embodiment The present invention can also be used for processing of three-dimensional point cloud position data measured by a three-dimensional laser scanner. A three-dimensional laser scanner irradiates a measurement object with laser light and detects the reflected light to measure data on the distance, azimuth, and elevation angle (or depression angle) from the measurement position to the reflection point of the laser light, The three-dimensional position data of the reflection point is acquired. Here, by setting an infinite number of reflection points on the measurement object, a three-dimensional model in which the measurement object is converted into data using an infinite number of three-dimensional point groups can be obtained. In the three-dimensional point cloud position data, a two-dimensional image and a three-dimensional coordinate are linked. That is, in the three-dimensional point group position data, the data of the two-dimensional image of the measurement object, a plurality of measurement points (point group) associated with the two-dimensional image, and the three-dimensional space of the plurality of measurement points Are associated with each other (three-dimensional coordinates).

  According to the three-dimensional point cloud position data, it is possible to obtain a three-dimensional model that reproduces the outer shape of the measurement object using a set of points. In addition, since the three-dimensional coordinates of each point are known, the relative positional relationship of each point in the three-dimensional space can be grasped, and the image of the three-dimensional model displayed on the screen can be rotated or switched to an image viewed from a different viewpoint. Processing is possible.

  The reflected light of the laser beam from the measurement object reflects the hue and shape of the reflection point. Therefore, each of the point groups in the three-dimensional point group position data obtained from the laser scanner is regarded as pixel data, so that the three-dimensional point group position data can be handled as image pixel information. Feature points can also be extracted from an image based on the three-dimensional point cloud position data. That is, in the 3D point cloud position data, as in the case of a photographic image, point clouds such as edge portions and visually characterized portions are included, and these are the same as in the case of a photographic image. It can be handled as a unique feature point.

  Therefore, the extraction of the specific shape in the present invention can be performed based on the three-dimensional point cloud position data. For example, when acquiring three-dimensional point cloud position data of a measurement object from different positions (viewpoints) in order to eliminate occlusion, the first three-dimensional point cloud position data obtained from the first viewpoint, The present invention can be used to specify the correspondence with the second three-dimensional point cloud position data obtained from a second viewpoint different from the first viewpoint. In this case, extraction of a specific shape, setting of a template shape, estimation of parallax, and a matching method using a lattice are the same as in the case of the first embodiment.

  For example, extraction of a specific shape using 3D point cloud position data is performed as follows. Since the three-dimensional point group position data includes the data of the three-dimensional coordinates of the reflection point, data related to the contour of the measurement object can be obtained from the three-dimensional point group position data. A specific shape can be extracted from the data relating to the contour. Here, the method of specifying the straight line or the curve constituting the specific shape by the end points or the dividing points is the same as the method shown in FIGS.

  FIG. 17 shows a three-dimensional point cloud image processing apparatus 500 that performs the above-described processing based on three-dimensional point cloud position data. A laser scanner 400 is connected to the three-dimensional point cloud image processing apparatus 500. The laser scanner 400 irradiates a measurement object with laser light and detects the reflected light, thereby measuring and measuring data on the distance, azimuth, and elevation angle (or depression angle) from the measurement position to the reflection point of the laser light. The three-dimensional point cloud position data of the object is acquired.

  The 3D point cloud image processing apparatus 500 includes a 3D point cloud position data acquisition unit 501, an edge extraction unit 502, and a feature point extraction unit 503. The three-dimensional point group position data acquisition unit 501 acquires the three-dimensional point group position data output from the laser scanner 400 and takes the three-dimensional point group position data into the three-dimensional point group image processing apparatus 500. The edge extraction unit 502 extracts a contour line constituting the appearance of the measurement object from the 3D point cloud position data acquired by the 3D point cloud position data acquisition unit 501. The feature point extraction unit 503 extracts feature points based on the edges extracted by the edge extraction unit 502.

  Hereinafter, the function of the edge extraction unit 502 will be described. First, based on the three-dimensional point cloud position data acquired from the laser scanner 400, a square region (grid-like region) whose side is about 3 to 7 pixels around the target point is acquired as a local region. Next, the normal vector of each point in the local region is calculated. Then, the variation (local curvature) of the normal vector in the local region described above is calculated. Here, the average (mNVx, mNVy, mNVz) of the intensity values (NVx, NVy, NVz) of the three-axis components of each normal vector is obtained in the local region of interest. Based on this result, standard deviations (StdNVx, StdNVy, StdNVz) are further obtained. Next, the square root of the sum of squares of the standard deviation is calculated as a local curvature (crv).

  Next, a local plane fitting (approximate) to the local region is obtained. In this process, an equation of the local plane is obtained from the three-dimensional coordinates of each point in the local region of interest. The local plane is a plane that is fitted to the local region of interest. Here, the equation of the surface of the local plane to be fitted to the local region is calculated using the least square method. The above process is repeated so that all three-dimensional point cloud position data is targeted while sequentially shifting the local area, and the normal vector, local plane, and local curvature in each local area are obtained.

  Next, based on the normal vector, local plane, and local curvature in each local area obtained above, processing for removing points in the non-surface area is performed. That is, in order to extract a surface (a plane and a curved surface), a portion (non-surface region) that can be determined not to be a surface in advance is removed. The non-surface region is a region that is neither a plane nor a curved surface, but may include a curved surface with a high curvature depending on the following threshold values (1) to (3).

  The non-surface area removal process can be performed using at least one of the following three methods. Here, the determination by the following methods (1) to (3) is performed for all the local regions described above, and the local region determined as the non-surface region by one or more methods is configured as the non-surface region. Extract as a local region. Then, the three-dimensional point cloud position data relating to the points constituting the extracted non-surface area is removed.

(1) A portion having a high local curvature: The above-described local curvature is compared with a preset threshold value, and a local region having a local curvature exceeding the threshold value is determined as a non-surface region. Since the local curvature represents the variation of the normal vector at the point of interest and its peripheral points, the value is small for a surface (a flat surface and a curved surface with a small curvature), and the value is large for a surface other than a surface (non-surface). Therefore, if the local curvature is larger than a predetermined threshold, the local region is determined as a non-surface region.

(2) Fitting accuracy to the local plane: When the distance between each point of the local area and the corresponding local plane is calculated and the average of these distances is greater than a preset threshold, the local area is determined as a non-surface area. judge. That is, when the local area is in a state of being deviated from the plane, the distance between the local plane corresponding to each point of the local area increases as the degree becomes more severe. Using this fact, the degree of non-surface of the local region is determined.

(3) Coplanarity check: Here, the directions of corresponding local planes are compared in adjacent local regions. If the difference in orientation of the local planes exceeds the threshold value, it is determined that the local area to be compared belongs to the non-plane area. Specifically, if the inner product of the normal vector of the two local planes fitted to each of the two target local regions and the vector connecting the center points is 0, both local planes are on the same plane. Is determined to exist. Further, as the inner product increases, it is determined that the degree that the two local planes are not on the same plane is more remarkable, and the degree of non-surface is determined to be higher.

  In the determination by the above methods (1) to (3), a local region determined as a non-surface region by one or more methods is extracted as a local region constituting the non-surface region. Then, the three-dimensional point group position data relating to the points constituting the extracted local region is removed from the three-dimensional point group position data to be calculated. The non-surface area is removed as described above.

  Next, surface labeling is performed. The surface labeling unit performs surface labeling on the 3D point cloud position data from which the 3D point cloud position data of the non-surface area is removed based on the continuity of the normal vectors. Specifically, if the angle difference between the normal vector of a specific point of interest and an adjacent point is less than a predetermined threshold, the same label is attached to those points. By repeating this operation, the same label is attached to a continuous flat surface and a continuous gentle curved surface, and these can be identified as one surface. Further, after the surface labeling, it is determined whether the label (surface) is a plane or a curved surface with a small curvature by using the angle difference between the normal vectors and the standard deviation of the three-axis components of the normal vectors. Identification data for identifying the fact is associated with each label.

  Subsequently, the label (surface) having a small area is removed as noise. This noise removal may be performed simultaneously with the surface labeling process. In this case, while performing surface labeling, the number of points of the same label (the number of points constituting the label) is counted, and a process of canceling the label having the number of points equal to or less than a predetermined value is performed. Next, the same label as the nearest surface (closest surface) is given to the point having no label at this time. As a result, the already labeled surface is expanded.

  That is, an equation of a labeled surface is obtained, and a distance between the surface and a point without a label is obtained. If there are a plurality of labels (surfaces) around a point where there is no label, the label with the shortest distance is selected. If there is still a point with no label, the threshold values for non-surface area removal, noise removal, and label expansion are changed, and related processing is performed again. For example, in non-surface area removal, the number of points to be extracted as non-surface is reduced by increasing the threshold value of local curvature. Alternatively, in label expansion, by increasing the threshold of the distance between a point without a label and the nearest surface, more labels are given to the point without a label.

  Next, even if the labels are different surfaces, the labels are integrated when they are the same surface. In this case, even if the surfaces are not continuous, the same label is attached to the surfaces having the same position or orientation. Specifically, by comparing the position and orientation of the normal vector of each surface, the same non-continuous surface is extracted and unified to the label of any surface.

  Next, a boundary region surface connecting adjacent surfaces is formed. In this example, a local plane is fitted to a non-surface region between adjacent surfaces, and a plurality of these local planes are connected to form a boundary region surface that approximates the non-surface region by a plurality of local planes. This can be regarded as a process of approximating a non-surface region with a polyhedron composed of a plurality of local planes and forming a boundary region surface that is a boundary portion between the surfaces. It is also possible to set a smooth curved surface to be fitted to the boundary region surface having this polyhedral structure and replace the boundary region surface with this curved surface.

  Next, based on the three-dimensional point cloud position data of the adjacent surfaces, a contour line that is a three-dimensional edge of the measurement object is calculated (estimated). In this case, in the formation of the boundary area surface, a process of connecting the local plane from the adjacent surface is performed. At this time, the line of intersection between the adjacent local planes is calculated as the contour line. By calculating the contour line, the contour image of the measurement object becomes clear. As a method for calculating the contour line using the boundary region surface, the contour line is set so that the contour line passes through the center position of the boundary region surface, or the contour line passes through the portion where the curvature of the boundary region surface is minimized. A method of setting the contour line as described above is also possible. In addition, the process which makes the intersection line of an adjacent surface and a surface an outline without forming a boundary area surface is also possible.

  Next, a two-dimensional edge is calculated. In this process, based on the intensity distribution of reflected light from the object, a two-dimensional image corresponding to a segmented (segmented) surface using a known edge extraction operator such as Laplacian, Pleuwit, Sobel, Canny, etc. Edges are extracted from within the region. In other words, since the two-dimensional edge is recognized by the difference in shading in the surface, the difference in shading is extracted from the information on the intensity of the reflected light, and by setting a threshold value in the extraction condition, the border between the shading is used as an edge. Extract. Next, the height (z value) of the three-dimensional coordinates of the points constituting the extracted edge, and the height (z value) of the three-dimensional coordinates of the points constituting the neighboring contour line (three-dimensional edge) If the difference is within a predetermined threshold, the edge is extracted as a two-dimensional edge. That is, it is determined whether or not a point constituting the edge extracted on the two-dimensional image is on the segmented surface, and if it is determined that the point is on the surface, it is set as the two-dimensional edge.

  After calculating the two-dimensional edge, the previously calculated contour line and the two-dimensional edge are integrated. Thereby, extraction of an edge characterizing the appearance of the object is performed. By extracting the edge, a characteristic portion constituting the appearance of the measurement object when the measurement object is visually recognized is extracted, and three-dimensional model data is obtained. The function of the edge extraction unit 502 has been described above.

  The feature point extraction unit 503 extracts a point group constituting an edge portion characterizing the object as a feature point. Alternatively, the feature point extraction unit 503 newly sets a point group that constitutes an edge portion that characterizes the object. Also in this case, as shown in FIGS. 6A and 7A, a straight line and a curve constituting the edge are represented by a large number of feature points. Then, as shown in FIGS. 6B and 7F, the straight line constituting the edge is specified by the end points at both ends, and the curve is specified by the end points and the dividing points at both ends. Thus, based on the three-dimensional point cloud position data measured by the laser scanner 400, an edge characterizing the appearance of the measurement object and the characteristic points constituting the edge are obtained, and the measurement object is handled as an image based on the edge. It will be in a state that can be.

  The three-dimensional point cloud image processing apparatus 500 includes a matching processing unit 104. The matching processing unit 104 has the configuration shown in FIG. 5 and executes the processing shown in FIG. In this case, at least one target image is not a photographic image, but a model image of a measurement object configured by the above-described edges obtained from the three-dimensional point cloud position data. At this time, the matching processing unit 104 extracts a specific shape from the edge data according to the principle shown in FIGS. 6 and 7, and further sets a template shape.

  The present invention can also be applied to a technique for obtaining a correspondence relationship between feature points extracted from a photographic image and three-dimensional point cloud position data obtained from a laser scanner. That is, the present invention can also be used in a technique for taking a photograph from the first viewpoint, performing laser scanning from the second viewpoint, and obtaining a correspondence relationship between the data obtained from both viewpoints. In this case, the correspondence between the template shape extracted from the photographic image obtained from the first viewpoint and the template shape extracted from the model image based on the 3D point cloud position data obtained from the second viewpoint. Is performed, and further, matching processing between lattice points illustrated in FIGS. 13 to 15 is performed.

3. Third Embodiment As a method of first performing outline matching and obtaining an initial value, an OCM method (Orientation Code Matching) can be used instead of edge matching. In the OCM, feature points are extracted using the OCR described above. At this time, by setting the threshold appropriately, a particularly characteristic portion (for example, a clear edge portion or the like) is extracted by software. In this case, the process after step S403 in FIG. 12 is executed using the feature points extracted by OCR as initial values.

  The present invention can be used for a technique for specifying the correspondence between a plurality of images.

Claims (14)

Specification for extracting a specific shape viewed from the first viewpoint and a specific shape viewed from the second viewpoint based on the feature points of the measurement object viewed from the first viewpoint and the second viewpoint A shape extraction unit;
A correspondence specifying unit that specifies the correspondence between the specific shape viewed from the first viewpoint and the specific shape viewed from the second viewpoint;
The specific shape includes at least one of a straight line and a curve;
The straight line is specified by the end points at both ends thereof,
2. The image processing apparatus according to claim 1, wherein the curve is specified by an end point at both ends thereof and a dividing point that divides between the end points at both ends.   The image processing according to claim 1, further comprising a feature point extraction unit that extracts a feature point of the measurement object viewed from the first viewpoint and the second viewpoint from each of two photographic images. apparatus. A parallax estimation unit that estimates a parallax between a feature point that forms the specific shape viewed from the first viewpoint and a feature point that forms the specific shape viewed from the second viewpoint;
The search range setting unit that sets a search range for specifying the correspondence in the correspondence specifying unit based on the parallax estimated by the parallax estimation unit, further comprising: The image processing apparatus described.   The image processing apparatus according to claim 3, wherein the parallax estimation unit is calculated based on a plurality of calculated parallax values obtained from a wide area including the feature point of interest. The parallax estimation unit
Extracting a plurality of feature points as a first group of pass points from around a specific feature point constituting the specific shape viewed from the first viewpoint;
Extracting a second group of pass points corresponding to the first group of pass points from the feature points of the measurement object viewed from the second viewpoint;
Calculating a parallax Dn for each corresponding pass point of the first group of pass points and the second group of pass points;
5. A step of calculating a weighted average of the Dn using a weight according to a distance from the specific feature point, and calculating an estimated value D of the parallax based on the result. An image processing apparatus according to 1. Lattice setting means for setting a lattice in the first captured image captured from the first viewpoint;
A divided shape forming unit that forms a divided shape based on feature points constituting the specific shape specified by the correspondence specifying unit in the lattice;
A split shape interpolation section for interpolating the split shape formed by the split shape forming section into the lattice;
Deformation of setting a deformed grid obtained by deforming the grid in the first image in the second captured image captured from the second viewpoint based on the inclination of the divided shape inserted into the grid with respect to the grid A lattice setting unit;
The correspondence grid point detection part which specifies the correspondence of the lattice point of the lattice in the 1st picked-up image and the lattice point of the deformation lattice in the 2nd picked-up image is provided. The image processing apparatus according to any one of 1 to 5. The lattice setting means sets a first lattice having a first lattice interval in the processing of the first layer, and sets a second lattice narrower than the first lattice interval in the processing of the second layer. Set the grid,
The deformation lattice setting unit performs lattice deformation on the first lattice interval and the second lattice interval,
The image processing apparatus according to claim 6, wherein the corresponding lattice point detection unit detects corresponding lattice points with respect to the first lattice interval and the second lattice interval.   The processing of a specific layer further includes an initial value adding unit that adds, as an initial value, a feature point constituting the specific shape in addition to the corresponding point specified in the previous layer. 8. The image processing apparatus according to 7.   The image according to claim 1, wherein at least one of the feature points of the measurement object viewed from the first viewpoint and the second viewpoint is three-dimensional point cloud position data measured by a laser scanner. Processing equipment. Specification for extracting a specific shape viewed from the first viewpoint and a specific shape viewed from the second viewpoint based on the feature points of the measurement object viewed from the first viewpoint and the second viewpoint A shape extraction step;
A correspondence specifying step for specifying a correspondence between the specific shape viewed from the first viewpoint and the specific shape viewed from the second viewpoint; and
The specific shape includes at least one of a straight line and a curve;
The straight line is specified by the end points at both ends thereof,
2. The image processing method according to claim 1, wherein the curve is specified by an end point at both ends thereof and a dividing point for dividing between the end points at both ends. A program that is read and executed by a computer,
The computer extracts a specific shape viewed from the first viewpoint and a specific shape viewed from the second viewpoint based on the feature points of the measurement object viewed from the first viewpoint and the second viewpoint. A specific shape extraction step,
A correspondence specifying step for specifying the correspondence between the specific shape seen from the first viewpoint and the specific shape seen from the second viewpoint;
The specific shape includes at least one of a straight line and a curve;
The straight line is specified by the end points at both ends thereof,
The program for image processing, wherein the curve is specified by an end point at both ends thereof and a dividing point that divides between the end points at both ends. Correspondence for specifying the correspondence between the image viewed from the first viewpoint and the image viewed from the second viewpoint based on the feature points of the measurement object viewed from the first viewpoint and the second viewpoint A relationship identification part;
A grid setting unit for setting a grid in the first captured image captured from the first viewpoint;
A divided shape forming unit that forms a divided shape based on a feature point of the first measurement object viewed from the first viewpoint on the lattice;
A split shape interpolation section for interpolating the split shape formed by the split shape forming section into the lattice;
Deformation of setting a deformed grid obtained by deforming the grid in the first image in the second captured image captured from the second viewpoint based on the inclination of the divided shape inserted into the grid with respect to the grid A lattice setting unit;
An image processing comprising: a corresponding grid point detection unit that specifies a correspondence relationship between a grid point of the grid in the first captured image and a grid point of the deformed grid in the second captured image. apparatus. The lattice setting means sets a first lattice having a first lattice interval in the processing of the first layer, and sets a second lattice narrower than the first lattice interval in the processing of the second layer. Set the grid,
The deformation lattice setting unit performs lattice deformation on the first lattice interval and the second lattice interval,
The image processing apparatus according to claim 12, wherein the corresponding lattice point detection unit detects corresponding lattice points with respect to the first lattice interval and the second lattice interval.   In the processing of a specific layer, in addition to the corresponding point specified in the previous layer, the method further comprises an initial value adding unit that adds, as an initial value, the feature point whose correspondence is specified by the correspondence specifying unit The image processing apparatus according to claim 13.

Images