JP2003242485A - Processor and processing method for stereo image - Google Patents

Abstract

<P>PROBLEM TO BE SOLVED: To provide a processor for a stereo image, capable of easily using a general purpose camera or a digital camera as a stereo camera. <P>SOLUTION: This processor for a stereo image includes an extract part 4 for extracting a first mark from the calibration image photographing a calibration chart 1 having the first mark and a second mark at least in two directions with stereo cameras 2R, 2L, a rough mark position measuring part 5 for obtaining the rough position of the second mark in the calibration image by projective transformation using the first mark extracted by the extract part, a precise mark position measuring part 6 for obtaining the position of photographing the second mark position in the vicinity of the rough position, and an arithmetic processing part 7 for calculating an external spotting element of the calibration image from the position of the second mark in the calibration chart 1 and the position in the calibration image corresponding to the second mark. <P>COPYRIGHT: (C)2003,JPO

Description

Detailed Description of the Invention

[0001]

BACKGROUND OF THE INVENTION 1. Field of the Invention The present invention relates to an external orientation element (position and inclination of a camera) of a stereo camera when the stereo camera is constructed by using a camera having lens aberration such as a general-purpose camera or a digital camera. The present invention relates to a stereo image processing apparatus and method capable of calculating a value and generating an image for stereo measurement.

[0002]

2. Description of the Related Art Stereo cameras for stereo measurement are fixed so that the cameras do not move relative to each other. Therefore, the baseline length corresponding to the camera interval does not change, and the shooting distances of the cameras also need to be accurately matched between the cameras.Therefore, instead of a zoom lens with a variable focal length, a fixed focal length lens is used. It is used. Moreover, since the measurement accuracy of the stereo camera is determined depending on the resolution of the camera, a high-precision lens with little aberration is used.

[0003]

Therefore, the conventional stereo camera has the following problems. Each camera is fixed so that it does not move relatively, making it difficult to carry. Since the selection range of the base length and the shooting distance is small, the size of the measurement object of the stereo camera is limited. Dedicated cameras that use high-precision lenses are expensive and difficult for the general public to use. With a general-purpose camera, positioning during stereo shooting is ambiguous, and it is only possible to obtain an accuracy of viewing, and it is practically impossible to use it for measurement.

The present invention has solved the above-mentioned problems, and a first object thereof is to provide a stereo image processing apparatus and method in which a general-purpose camera or a digital camera can be easily used as a stereo camera. The second purpose is
It is an object of the present invention to provide a stereo image processing apparatus capable of obtaining an accuracy enough for photo measurement even if stereo photography is performed with a lens having large lens aberration such as a general-purpose camera or a digital camera.

[0005]

The stereo image processing apparatus of the present invention achieves the first object and, as shown in FIGS. 1 and 2, is provided with at least three positions.
A calibration chart 1 having a mark and a second mark that is provided so as to be visually distinguishable from the first mark is taken from at least two calibration images captured by stereo cameras 2R and 2L. The extraction unit 4 that extracts the mark and the first extracted by the extraction unit 4
The approximate mark position measuring unit 5 that obtains the approximate position of the second mark in the calibration image by projective transformation using the mark, and the approximate position of the second mark in the vicinity of the approximate position of the second mark with respect to the calibration image. The precision mark position measuring unit 6 for obtaining the photographed position of the second mark position, the position of the second mark in the calibration chart 1, and the position of the second mark in the calibration image corresponding to this second mark. An arithmetic processing unit 7 for calculating an external orientation element of the calibration image from the position.

In the apparatus constructed as described above, the stereo cameras 2R and 2L are provided with at least three first marks and a second mark which is visually distinguishable from the first marks. A calibration image obtained by capturing at least two sheets of the calibration chart 1 having the above is prepared. Since these two calibration images are the images of the stereo cameras 2R and 2L, respectively, there are at least two shooting directions. The extraction unit 4 extracts the first mark from the calibration image. The approximate mark position measuring unit 5 obtains the approximate position of the second mark in the calibration image by projective transformation using the first mark. The precision mark position measuring unit 6 obtains the imaged position of the second mark position in the vicinity of the approximate position of the second mark. This is because the influence of the lens aberration is different for each second mark position, and thus the position estimated by the rough mark position measuring unit 5 and the actual position on the calibration image are different. The arithmetic processing unit 7 calculates the external orientation element of the calibration image from the position of the second mark in the calibration chart 1 and the position of the second mark in the calibration image corresponding to this second mark. This external orientation element is applied as an external orientation element to a stereo image captured by the stereo cameras 2R and 2L under the same imaging conditions as when the external orientation element was calculated.

Preferably, when the rough mark position measuring unit 2 of the present invention is configured to obtain the rough position of the second mark by projective transformation, the rough position of the second mark can be easily calculated by projective transformation. . The lens aberrations of the stereo cameras 2R and 2L appear as a difference between the position estimated by the rough mark position measuring unit 5 and the actual position on the calibration image.

Preferably, the precision mark position measuring unit 6 of the present invention is configured to determine the second mark position by using at least one of template matching and barycentric position detection. When the second mark has an irregular shape or is unclearly shown in the calibration image, the position of the center of gravity of the second mark is used to stably and accurately obtain the second mark position. .

Preferably, the arithmetic processing unit 7 of the present invention determines the position of the second mark in the calibration chart 1 and the position of the second mark in the calibration image corresponding to the second mark from the position of the second mark. It may be configured to calculate the external orientation element at the two marks.

Preferably, the calibration device of the present invention is further configured to have a display unit 9 for displaying the external orientation element in the second mark as a numerical value. By providing the display unit 9, it is convenient for the operator to easily judge the quality of the calculation result.

Preferably, the arithmetic processing unit 7 of the present invention selects an appropriate second mark from the second marks whose position is calculated by the precision mark position measuring unit when calculating the external orientation element of the lens. And may be configured to choose to do so. For selecting the second mark, for example, a bundle adjustment method may be used. According to the bundle adjustment method, it is possible to obtain the external orientation element of the camera at each photographing position and to distinguish between the appropriate second mark position and the inappropriate second mark position.

[0012] Preferably, the external orientation element of the present invention is configured to include at least one of the position, inclination, and base line length of the stereo camera. It is the position, inclination, or baseline length of the stereo camera that contributes to the external orientation element.

A stereo image processing apparatus according to the present invention comprises a second
As shown in FIG. 1, an arithmetic processing unit 7 for calculating a calibration element for compensating for lens aberrations of the stereo cameras 2R and 2L, and the stereo cameras 2R and 2L are used for shooting. The image processing unit 8 may be configured to correct the image into an image in which the lens aberration is compensated. When the image processing unit 8 is provided, it is possible to remove the influence of lens aberration from the images captured by the stereo cameras 2R and 2L by using the calibration element calculated by the arithmetic processing unit 7, especially as in a stereoscopic photograph. It is suitable for use in a field where a slight image position distortion appears as a large elevation error.

A stereo image processing method according to the present invention is a first method.
As shown in FIG. 7, a calibration mark having at least three or more first marks and a second mark visually distinguishable from the first marks. A first step (S40) of extracting a first mark from at least two calibration images obtained by photographing a chart with a stereo camera.
And a second step (S50) of obtaining a rough position of the second mark in the calibration image by projective transformation using the extracted first mark.
And the second image for the calibration image.
A third step (S60) of obtaining the photographed position of the second mark position near the approximate position of the mark, the position of the second mark in the calibration chart, and the calibration corresponding to the second mark. A fourth step (S90) of calculating the external orientation element of the calibration image from the position of the second mark in the calibration image.

[0015]

DESCRIPTION OF THE PREFERRED EMBODIMENTS The present invention will be described below with reference to the drawings. FIG. 1 is an overall configuration block diagram for explaining the first embodiment of the present invention. In the figure, a chart 1 as a calibration chart has a first mark and a second mark printed on a sheet on a plane. The first mark is used for measuring the rough mark position and associating the images of the calibration photograph set as the calibration image with each other, and at the same time, the photographing angles at which the stereo cameras 2R and 2L photograph the chart 1. Used to determine. The first marks are provided at at least three places on the chart 1, and preferably, they are provided on each quadrant when the chart 1 is divided into four sections.

The second mark is the stereo camera 2R, 2L.
It is used to specify the position of the image data of the chart 1 photographed by, and is also called a target, and preferably, it is arranged on the chart 1 evenly and uniformly. The second marks are preferably provided at 30 or more places on the chart 1, and more preferably at about 100 to 200 places. However, if a large number of second marks are provided, the second mark itself becomes small and becomes difficult to see, and the measurement calculation time of the lens aberration also becomes long. is there. Details of the chart 1 will be described later.

The stereo cameras 2R and 2L are cameras to be calibrated, and here, a single camera moves along a stereo bar 2B forming a base line. In this embodiment, the camera is
Typically, a lens having a large lens aberration can be used as compared with a general-purpose optical camera or a digital camera, which has a larger lens aberration than that of a camera for photogrammetry or photographing. The cameras used for the stereo cameras 2R and 2L may be wide-angle lenses or standard lenses, or may be equipped with a telephoto lens. The image data storage unit 3 is a storage device that stores the image data of the chart 1 captured by the cameras forming the stereo cameras 2R and 2L, and is, for example, a magnetic disk or a CD-ROM.
An electromagnetic storage medium such as The image data stored in the image data storage unit 3 is preferably stereo image data obtained by photographing the chart 1 so that the chart 1 can be viewed stereoscopically by the stereo cameras 2R and 2L. Typically, the left photographing position 2L and the right photographing position are used. The image is taken at a set of shooting positions at the position 2R. Preferably, in the image data storage unit 3, it is preferable that the images of the calibration photograph set be stored in a manner that the photographing angles at which the stereo cameras 2R and 2L photograph the chart 1 can be determined.

The stereo image processing apparatus includes an extracting unit 4, a rough mark position measuring unit 5, a precision mark position measuring unit 6, a calculation processing unit 7, an image processing unit 8, a mark coordinate storage unit 10, and a lens aberration compensation parameter storage unit. 11 and an external orientation element storage unit 12, and an image data storage unit 3 and a display unit 9 as external devices. For the stereo image processing apparatus, for example, a computer equipped with Pentium (registered trademark) or Celeron (registered trademark) manufactured by Intel Corporation as a CPU may be used.

The extraction unit 4 extracts the first mark from the image data stored in the image data storage unit 3 and performs a first mark extraction process for obtaining the image coordinate value of the first mark. First
The mark extracting process is performed by the second step performed by the rough mark position measuring unit 5.
This is performed as a pre-process for calculating the approximate position of the mark and associating it. The image coordinate value of the first mark is stored in the mark coordinate storage unit 10. When the first mark includes a symbol that is common to the second mark, the image coordinate value of the first mark may be set according to the position of the second mark in the first mark. Details of the first mark extraction processing by the extraction unit 4 will be described later.

The general mark position measuring unit 5 performs projective transformation from the image coordinate values of the first mark extracted by the extracting unit 4 to obtain an external orientation element, and the single photograph orientation theorem and the collinear conditional expression are obtained. The approximate position of the second mark is calculated by using the images, and the images of the proof set are associated with each other. Details of the second mark rough position calculation processing by the rough mark position measurement unit 5 will be described later.

The precision mark position measuring unit 6 recognizes the second mark in the image of the photo set for calibration, and precisely calculates the position of the second mark by the gravity center position detection method or the like. When the position of the second mark calculated by the precision mark position measuring unit 6 is significantly different from the positions of the other second marks in the image data of the chart 1, the calculation processing unit 7 causes a discrepancy. It has a function of excluding the position of the generated second mark. Further, the arithmetic processing unit 7 includes a precision mark position measuring unit 6
The second mark suitable for calibration is extracted from the second marks calculated in step 1, the external orientation element and the coordinates of the target point are simultaneously adjusted, and the internal parameters of the camera are calculated.

The internal parameters of the camera calculated by the calculation processing unit 7 may be stored in the lens aberration compensation parameter storage unit 11. The internal parameters of the camera include principal point position, screen distance, and distortion parameter. Although only the distortion parameter is obtained here, spherical aberration, coma, astigmatism, and image plane warp that constitute Seidel's 5 aberrations may also be obtained. The internal parameters obtained by the arithmetic processing unit 7 are displayed graphically on the display unit 9. The external orientation element for the second mark, which is calculated by the arithmetic processing unit 7 and is suitable for calibration, may be stored in the external orientation element storage unit 12. The details of the precision mark position measuring unit 6 and the camera internal parameter calculation processing of the calculation processing unit 7 will be described later.

The image processing section 8 uses the internal parameters obtained by the arithmetic processing section 7 to set the stereo cameras 2R and 2R.
The data images of the images (in particular, the images other than the chart 1) taken by the cameras constituting L are rearranged. Then, the images captured by the cameras forming the stereo cameras 2R and 2L are displayed on the display unit 9 as an image with significantly less distortion in which most of the lens aberrations are removed.
The display unit 9 is an image display device such as a CRT or a liquid crystal display. The mark coordinate storage unit 10 stores the image coordinate value of the first mark, the management number of the second mark, and the image coordinate value thereof. The lens aberration compensation parameter storage unit 11 stores internal parameters of the cameras forming the stereo cameras 2R and 2L calculated by the calculation processing unit 7. The external orientation element storage unit 12 stores the external orientation element for the second mark calculated by the arithmetic processing unit 7 and suitable for the calibration.

Next, the chart 1 as a calibration chart will be described. FIG. 2 is a plan view showing an example of the calibration chart. The chart 1 is in the form of a flat sheet, and has a readily visible first mark and a second mark composed of a large number of dots printed on the front side. Here, a total of five first marks are arranged on the chart 1, and the outline is a rhombus, and the second mark is formed on the central portion.
The same pattern as the mark is drawn. First mark 1a,
1b, 1c, and 1d are provided in each quadrant when the chart 1 is divided into four quadrants. The first mark 1a is the upper left quadrant, the first mark 1b is the upper right quadrant, and the first mark 1c is the lower left quadrant. The 1 mark 1d is located in the lower right quadrant. The first mark 1e is provided at the origin position common to each quadrant. For example, the first marks 1a, 1b, 1c, 1d
Are provided at positions equidistant from the first mark 1e. If the chart 1 is rectangular, the first mark 1
a, 1b and the first mark 1e in the vertical direction, h
The vertical distance between the marks 1c and 1d and the first mark 1e is l. At this time, the first marks 1a, 1b, 1c, 1
The distance d between the d and the first mark 1e satisfies the following relationship. d = (h ² + l ² ) ^1/2 (1)

The first mark and the second mark are printed with desired dimensions or the dimensions are measured in advance. The numerical values of the print positions of the first mark and the second mark are read into the mark coordinate storage unit 10 of the stereo image processing apparatus, and are used by the rough mark position measuring unit 5 for measuring rough position and associating them. The chart 1 may be stored as image data in a storage device of a computer, and may be printed and used at a place for calibration. For the positions of the first mark and the second mark, if the ones stored in advance in the stereo image processing apparatus are used and printing is performed on the sheet at the stored coordinates, the measurement work becomes unnecessary, so that the work is not required. It will be easy. Alternatively, the chart 1 may be precisely measured to measure the coordinate positions of the first mark and the second mark, and the coordinate values may be stored in the mark coordinate storage unit 10 and used.

The first mark is used not only for measuring and associating the approximate mark position, but also as a visual target for determining the shooting direction. Furthermore, by making the central part of the outer shape rhombus of the first mark the same pattern as the second mark,
It is used as a template for precision measurement by the precision mark position measuring unit 6.

FIG. 3 is an explanatory view showing an example of the first mark.
(A) shows a diamond, (B) shows four arrows, and (C) shows a black rectangle. In FIGS. 3 (A) and 3 (B), the first mark is arranged so as to have a diamond shape or four arrows so as to surround a common pattern with the second mark so that the operator can easily recognize it. By making the design easy to see,
The first mark can be easily extracted, and even if one shooting angle is selected from the wide shooting angles as the shooting directions of the cameras forming the stereo cameras 2R and 2L, the first mark can be extracted from the shot image. Do not miss it. Figure 3
In (C), the first mark is a black-painted rectangle, and the design at the center has a color that is the reverse of the second mark. However, even in this case, detection is easy. Further, even when the precision mark position measuring unit 5 measures, the tone of the first mark can be reversed with respect to the pattern of FIG. 3C to be used as the template of the second mark. .

FIG. 4 is an explanatory view showing an example of the second mark.
(A) black circle "●", (B) plus "+", (C) double circle "◎", (D) English letter "X", (E) star "★", ( F) indicates a black-painted square “■”, (G) a black-painted triangle “▲”, and (H) a black-painted rhombus “◆”. Since many second marks are evenly arranged on the chart 1, various designs may be adopted as long as they can facilitate precise position measurement.

Next, a procedure for subjecting the lens aberration to compensation and photographing the chart 1 by the cameras used for the stereo cameras 2R and 2L will be described. 5A and 5B are explanatory views of the camera arrangement when measuring the lens aberration of the standard lens and the wide-angle lens. FIG. 5A shows the chart 1 viewed from the front, and FIG. 5B shows the status viewed from above.
If there are two or more images obtained by photographing the chart 1 from different photographing angles, the calibration can be performed. Preferably, when a plane chart printed on a sheet is used as the chart 1, the measurement values of each calibration element, particularly the focal length, are stable and reliable by photographing from three or more photographing angle directions. Will be high. In the case of FIG. 5, five directions, that is, FIG.
Front (I), lower right (II), upper right (II) based on (A)
I), the lower left (IV), and the upper left (V) are shown. The shooting method will be described below step by step. The numbers (I) to (V) correspond to the camera positions in FIG.

(I): Shoot from the front so that all the first and second marks of the chart 1 are fully filled. By putting the first mark and the second mark as close as possible to the edge of the captured image, the distortion correction up to the peripheral portion of the lens is ensured. (II): Next, for example, the first mark 1d in the lower right quadrant (see FIG. 2).
Change the camera position so that (See Chart 1 in Figure 1) becomes the center of shooting. Then, the cameras forming the stereo cameras 2R and 2L are directed so that the first mark 1e in the center is centered as it is, and the first mark and the second mark are fully filled for shooting. (III): The camera position is changed so that the first mark 1b in the upper right quadrant becomes the center of photography. Then, the camera is directed so that the first mark 1e at the center is centered as it is, and the first mark and the second mark are filled so as to shoot.

(IV): The camera position is changed so that the first mark 1c in the lower left quadrant becomes the center of photography. Then, the camera is directed so that the first mark 1e at the center is centered as it is, and the first mark and the second mark are filled so as to shoot. (V): The camera position is changed so that the first mark 1a in the upper left quadrant becomes the center of photography. Then, the camera is directed so that the first mark 1e at the center is centered as it is, and the first mark and the second mark are filled so as to shoot. By such a procedure, the angles of the cameras forming the stereo cameras 2R and 2L can be secured as the difference between the required shooting angles, so that the focal length can be measured reliably.

Next, the advantages of the photographing procedure using the first mark in the case of using a plane chart printed on a sheet as the chart 1 will be described. Usually, when a mark printed on a flat sheet is photographed, a desired angle cannot be attached to the cameras forming the stereo cameras 2R and 2L, and as a result, the screen distance (focal length) cannot be accurately obtained. That is, if there is no change in the focal length direction (height, depth direction), there is no constraint on the calculated values of the internal parameters of the camera, and it is extremely unreliable even if the internal parameters of the camera are calculated. Become.
Therefore, the targets arranged three-dimensionally are measured to obtain the focal length. However, the measurement is difficult and cannot be automated, and it is not easy to create. The reason is that the three-dimensionally arranged targets may not be reflected in the image due to the shadow of the target, or the positions in the image that are two-dimensionally reflected in the image may be moved back and forth between the targets. Because there is.

First, since the chart 1 in which the first mark and the second mark are printed on the sheet is used, the problem of difficulty in associating the targets, which has occurred in the case of the targets arranged three-dimensionally, is a problem. Is eliminated. Next, when shooting using the first mark, the stereo camera 2R,
It is possible to attach a desired angle to the camera forming the 2L, ensure a change in the height (depth) direction, and accurately calculate the focal length. For example, Chart 1 for camera angles
If it can be tilted by 10 degrees or more, the focal length can be reliably obtained in the stereo image processing apparatus of the present invention.

Here, the distance H between the camera and the chart 1 is
It is determined from the focal length f of the standard lens or the wide-angle lens.
For example, with a standard lens having a focal length of 35 mm, the shooting distance H is about 90 cm. Since the mutual distance d between the first marks provided on the chart 1 is, for example, 20 cm,
When tilting the shooting direction from the front (I) to the lower right (II),
A shooting angle of about 10 degrees is secured. The upper limit of the tilt angle in the shooting direction is determined by the depth of focus and the like. That is, when the tilt angle in the shooting direction is large, the distance between the camera and the first mark differs for each first mark, and the image of the first mark in the image is blurred. Therefore, the upper limit of the tilt angle in the shooting direction is, for example, 30 degrees. The actual shooting procedure is as shown in (I) to (V) above, and if the shooting is performed so that the first mark and the second mark are included in the full screen of the camera, the above conditions are automatically satisfied. The condition of is satisfied.

FIG. 6 is an explanatory view of the camera arrangement when measuring the lens aberration of the telephoto lens, and shows the state in which the chart 1 is viewed from above. In the case of a telephoto lens, the angle of view becomes narrow and the angle cannot be adjusted, so from the front (I) to the lower right (I
When tilting the shooting direction to I) etc., the shooting angle is 10
The degree cannot be secured. That is, in the case of a telephoto lens, the cameras and the chart 1 that form the stereo cameras 2R and 2L
This is because the shooting distance H is 1 m or more and the mutual distance d between the first marks is only about 20 cm. Therefore, the camera position on the right side (I
I) and (III) and left camera positions (IV) and (V) are determined. At this time, the distance between the left and right camera positions is set to the shooting distance H.
The above (II), (III) and (IV), (V) may be performed at a position of about 1/3 of the above. The optical axis of the camera may be aligned with the normal direction of the chart 1, but the chart 1 direction may be oriented.

In the above embodiment, the photographing positions are the front (I), lower right (II), upper right (III), lower left (IV) and upper left (V) directions. In the case of the lowest photographing position, the left and right directions may be two, or three or more directions. Also in the case of two left and right directions, the chart 1 is photographed so that a photographing angle of about 10 degrees is secured.

Next, the flow of overall processing in the stereo image processing apparatus of the present invention will be described. FIG. 7 is a flowchart illustrating a calibration method using the stereo image processing apparatus. First, the chart 1 is photographed by the cameras forming the stereo cameras 2R and 2L that are the targets of lens aberration compensation (S10). The photographing method is the photographing procedure described in FIG. 5B for the standard lens or the wide-angle lens, and the photographing procedure described in FIG. 6 for the telephoto lens. The image data of each shooting direction shot by the camera is temporarily stored in the image data storage unit 3. Next, the stereo image processing apparatus reads the image data stored in the image data storage unit 3 and displays it on the display unit 9 (S20). Then, the operator selects an image for which target association and measurement are to be performed, from the images displayed on the display unit 9 (S30). Then, the extraction unit 4 performs a first mark extraction process on the selected image (S4).
0).

(I): First Mark Extraction Processing In the first mark extraction processing, in order to determine the secondary projective transformation formula between the plane coordinates of the chart 1 set on the surface to be measured and its image coordinates (camera side). , At least three points or more of the first marks on the plane coordinates are measured on the image data. Here, since the second mark is included in the first mark, it is possible to accurately specify the first mark by specifying the position of the included second mark.
Specify the position of the mark. In the first mark extraction processing, the following processing from I- to I- is repeated for the number of points of the first mark. For example, in the chart 1 shown in FIG. 2, the process is performed for each of the first marks 1a, 1b, 1c, 1d on each of the left and right points.

I -... The operator aligns the cursor position of the mouse with the second mark in the first mark to be detected on the entire image displayed on the display unit 9 and clicks the mouse to obtain the approximate position of the first mark. I -... A local image including the second mark is cut out from the enlarged image centering on the image coordinates obtained by I-, and is displayed. At this time, the image including the second mark can be used as a template for the second mark precise position measurement. With respect to the enlarged image indicated by I -... I-, the cursor position of the mouse is aligned with the barycentric position of the second mark and clicked, and the image coordinates are set as the barycentric position of the first mark. It should be noted that I-
The alignment at does not have to be exact. I -... Next, the second stored in the mark coordinate storage unit 10.
In order to correspond with the management number of the mark, the management number of the second mark corresponding to the barycentric position of the first mark measured by I- is input. At this time, the barycentric position of the first mark measured by I- is stored as the reference point coordinate in the input management number of the second mark.

In the first mark extraction process, for example, if the measurement order of the first marks on the chart 1 is predetermined, the extraction unit 4 side automatically assigns numbers without inputting the management numbers of the second marks. It can be processed. Further, in the first mark extraction processing, for example, the selection image displayed on the display unit 9 is divided into two so that the operator can easily work, and the whole image as shown in FIG. Figure 3 (A), (B)
If an enlarged image like this is displayed, position measurement becomes easier.

Next, as another processing procedure of the first mark extraction processing, there is a method of measuring only the whole image as shown in FIG. 2 without using the enlarged image. In this case, the process of I- is performed, and the management number of the second mark corresponding to the barycentric position of the first mark measured in I- is input in I-. In this way, since the enlarged image is not used, I-
, I- can be omitted. However, since the entire image is displayed, the first mark is displayed in a small size, and therefore it may be determined whether the enlarged image is used or not according to the operator's preference.

Next, the case where the extraction unit 4 automatically performs the first mark extraction processing will be described. First, the outer frame portion that does not include the second mark of the first mark is registered as a template. For this registration, for example, the first first mark in the first mark extraction processing described above may be registered as a template image. Then, the remaining first marks can be automatically measured by the template matching process. Also, the position correspondence for the first mark is
Since the position of the first mark is clear from the image, it can be easily performed. For example, in the case of the first mark arrangement in FIG. 2, it is easy to associate the five first marks from the detected coordinates. Note that the template matching process is the same as the target recognition process (S62) in the second mark precise position measurement, which will be described later, and a description thereof will be omitted.

Next, the case where the extraction unit 4 further automatically processes the first mark extraction processing will be described. The template image of the first mark in the first mark extraction processing is registered in the extraction unit 4 in advance. Then, since the first mark is individually extracted by the template matching process using the template image of the first mark, the first image of I-
It is possible to omit all the work of designating the mark. That is, if the first mark is a mark that is clearly different from the second mark, the automatic processing can be performed even if the extraction unit 4 has a virtual template image. However, since it is sufficient to measure at least 3 points for the first mark,
Even manual work is a simple task.

Returning to FIG. 7, the rough mark position measuring unit 5 associates the rough mark position with the second mark (S5).
0). The second mark approximate position measurement and association includes a step (II-1) of obtaining an external orientation element and a step (II-2) of calculating an approximate position of the second mark. (II-1):
In the process outline mark position measuring unit 5 for obtaining the external orientation element, the reference point coordinates corresponding to the image coordinates of the first mark obtained in S40 are substituted into the secondary projective transformation formula shown in Formula (2),
The observation equation is established and the parameters b1 to b8 are obtained. X = (b1 * x + b2 * y + b3) / (b7 * x + b8 * y + 1) Y = (b4 * x + b5 * y + b6) / (b7 * x + b8 * y + 1) (2) where X and Y are reference point coordinates and x and y indicates the image coordinates.

Next, the relationship between the reference point coordinates and the image coordinates will be described. FIG. 9A is an explanatory diagram of the image coordinate system and the target coordinate system in the central projection. In the case of central projection, the projection center point O
The target coordinate system 52 as a reference point coordinate system on which the chart 1 is placed with reference to c and the image coordinate system 50 on which the film or CCD of the camera is placed have a positional relationship as shown in FIG. 9A. The coordinates of an object such as a reference mark in the object coordinate system 52 are (X, Y, Z), and the coordinates of the projection center point Oc are (X0, Y0, Z0). The coordinates in the image coordinate system 50 are (x, y), and the screen distance from the projection center point Oc to the image coordinate system 50 is C. ω, φ, κ are the image coordinate system 5
It represents the tilt at the time of photographing by the camera with respect to the three axes X, Y, and Z that configure the target coordinate system 52 of 0, and is called an external orientation element.

Subsequently, the parameters b1 to b of the formula (2) are used.
8 is used to obtain the next external orientation element from the equation (3). ω = tan ⁻¹ (C · b8) φ = tan ⁻¹ (−C · b7 · cos ω) κ = tan ⁻¹ (−b4 / b1) (when φ = 0) κ = tan ⁻¹ (−b2 / b5) (when φ ≠ 0 and ω = 0) κ = tan ⁻¹ {− (A1 · A3−A2 · A4) / (A1
・ A2-A3 ・ A4)} (when φ ≠ 0 and ω ≠ 0)

Z0 = C · cosω · {(A22 + A32) / (A12 + A42)} 1/2 + Z m X0 = b3− (tan ω · sin κ / cos φ−tan φ · cos κ) × (Zm−Z0) Y0 = b6− (tan ω · cos κ / cos φ-tan φ · sin κ) × (Zm−Z0) (3) where A1 = 1 + tan2φ, A2 = B1 + B2 · t
anφ / sinω, A3 = B4 + B5 · tanφ / si
Let nω and A4 = tanφ / (cosφ · tanω). Zm is the first mark 1a, 1b, 1c, 1d4
The average elevation of the reference points. Here, the first mark 1
Since the reference points of points a, 1b, 1c, and 1d are on the plane coordinates, it can be assumed that they are planes with a constant altitude. C is the focal length, which corresponds to the screen distance described above.

(II-2): Step of calculating approximate position of second mark Next, from the principle of single photograph orientation, with respect to the ground object (X, Y, Z) represented by the object coordinate system 52, Image coordinate system 5
The camera coordinates (xp, yp, zp) in the tilted camera coordinate system represented by 0 are given by Expression (4).

[Equation 1] Here, X0, Y0, and Z0 are as shown in FIG.
The ground coordinates of the projection center point Oc as shown in FIG.
Next, the camera tilt (ω, φ, κ) calculated by equation (3)
Is substituted into the equation (4), the rotation matrix is calculated, and the elements a11 to a33 of the rotation matrix are obtained.

Next, the elements a11 to a33 of the obtained rotation matrix, the camera position (X0, Y0, Z0) obtained by the equation (3), and the reference point coordinates (X, Y, Z) of the target are calculated. Substituting in collinear conditional expression {expression (5)}, image coordinates of target (x,
y) is calculated. Here, the collinear conditional expression is a relational expression that holds when the projection center, the photographic image, and the object on the ground are on a straight line. As a result, the position of the second mark in the case where there is no lens aberration is calculated, so the approximate image coordinates of the target in the image taken by the actual camera with lens aberration are obtained. x = −C · {a11 (X−X0) + a12 (X−X0) + a13 (Z−Z0)} / {a31 (X−X0) + a32 (X−X0) + a33 (Z−Z0)} y = −C * {A21 (X-X0) + a22 (X-X0) + a23 (Z-Z0)} / {a31 (X-X0) + a32 (X-X0) + a33 (Z-Z0)} (5)

By the way, since two solutions are obtained in the calculation of tan ^-1 in the equation (3), the camera inclinations (ω, φ, κ) each have two solutions, and the entire calculation is performed. Then, the first marks 1a, 1b, 1 measured in the first mark extraction process
c, 1d The correct answer ω is obtained by comparing the residuals between the image coordinates of the four points and the corresponding image coordinates of the four points obtained by the equation (5),
Calculate φ and κ. Although the quadratic projection transformation equation is used as the projection transformation equation here, the present invention is not limited to this, and other projection transformation equations such as a cubic projection transformation equation may be used. good.

Further, in the rough mark position measuring section 5, for example, the management number of the second mark added to the reference point coordinate file stored in the mark coordinate storage section 10 is assigned to each first mark.
The second marks are associated with each other by allocating the marks to the targets (second marks).

Returning to FIG. 7, the precision mark position measuring unit 6 measures the precision position of the second mark (S60). Hereinafter, the processing procedure of the precise position measurement of the second mark will be described in detail with reference to FIG. First, the precision mark position measuring unit 6
The target as the second mark is recognized (S62).
For this target recognition, for example, template matching using normalized correlation is used. The details of target recognition will be described below.

(III) Target Recognition FIG. 9B is an explanatory diagram of a template image of a normalized correlation used for target recognition and a target image. First, an arbitrary target is selected from the barycentric positions of the first marks measured in the first mark extraction processing (S40), for example, the first marks 1a, 1b, 1c, and 1d4 points. The template image of the normalized correlation is an image of M × M pixels centered on the barycentric position (image coordinate) of the selected target. Further, with the approximate position (image coordinates) of the target calculated in the second mark approximate position measurement (S50) as the center,
An image of N × N pixels is set as a target image.

Next, the target image is subjected to template matching by the normalized correlation shown in the equation (6) to find the position where the correlation value is maximum. It is considered that the superposition is achieved at the position where the correlation value becomes the maximum value and the target is recognized at the position where the correlation value becomes the maximum value. Convert the center coordinates of the template image here to the image coordinates on the normal size image,
Use as the detection point. A = {M ² × Σ (Xi × Ti) −ΣXi × ΣTi} / [{M ² × ΣXi ² − (ΣXi) ² } × {M ² × ΣTi ² − (ΣTi) ² }] (6) where , A is the correlation value, M is the square size of the template image, Xi is the target image, and Ti is the template image.
Further, although the square sizes N and M of the image are variable, it is preferable to make N and M as small as possible on the assumption that the target can be sufficiently stored in order to speed up the processing time.

Returning to FIG. 8, in order to measure the precise position of the second mark, subpixel edge detection of the second mark is performed (S64). The target image for which the sub-pixel edge detection of the second mark is performed is an image of N × N pixels centered on the detection point recognized as the target in S62. For the grayscale waveform existing in the target image, the Laplacian-Gaussian filter (LOG), which is the second derivative of the Gaussian function shown in Expression (7), is used.
Filtering is performed, and two zero crossing points, that is, edges of the curve of the calculation result are detected by subpixels. here,
The sub-pixel means that the position detection is performed with a finer precision than one pixel. ^{∇ 2 · G (x) =} {(x 2 -2σ 2) / 2πσ 6} · exp (-x 2 / 2σ 2) (7) where, sigma is the Gaussian function parameters - data - a.

Next, the center of gravity of the target is detected (S
66) and return (S68). Here, the intersection of the edge positions in the x and y directions obtained by using equation (7) is taken as the center of gravity of the target. The precise position measurement of the second mark is not limited to the process disclosed in S62 to S66, and any other method of detecting the position of the center of gravity, for example, the moment method or the template matching method may be further improved and used. You may make such a request.

Returning to FIG. 7, the positions of the centers of gravity of all the targets are confirmed, and it is confirmed that there is no apparent error (S7).
0). That is, it is confirmed whether the position of the target recognized is appropriate. The position of the detected target is displayed on the display unit 9 for the convenience of confirmation by the operator. If there is no error, go to S80. If there is an error, the inappropriate target position is corrected (S75).
For example, a target with a low correlation value calculated in S62 or a target whose center-of-gravity detection position is too far from the approximate detection position is displayed in red on the display unit 9 so that the operator can clearly recognize it. To display. Then, with respect to the target with the error, the operator manually re-measures (the position of the center of gravity is designated by the mouse). It should be noted that, even if the target position in which the error has occurred is not forcibly corrected, it can be removed because it is also detected as an abnormal point in the subsequent process step of S90 for obtaining the calibration parameter.

Then, the processing of S30 to S75 is repeated for the image necessary for measuring the lens aberration used in the camera (S80). For example, if the number of images captured by the cameras forming the stereo cameras 2R and 2L is 5, all 5 may be repeated, and if the number of images required for measuring the lens aberration has been reached. It is not necessary to repeatedly process all of the captured images.

After the measurement processing for the images necessary for measuring the lens aberration is completed, the internal parameter calculation processing of the camera of the calculation processing unit 7 is used to obtain the external orientation element and the lens aberration calibration element. Move (S9
0). Here, as the calculation target of the calibration element, for the second mark on the chart 1, all the second marks for which the positions of the centers of gravity have been obtained by associating them with each other by the processing of the rough mark position measuring unit 5 and the precision mark position measuring unit. Do about the mark.

(IV): Camera internal parameter calculation process (bundle adjustment method with self-calibration) As the camera internal parameter calculation process of the calculation processing unit 7, for example, "with self-calibration" used in the field of photogrammetry. Bundle adjustment method "is used. Here, the “bundle adjustment” is based on the collinear condition that the light flux (bundle) connecting the subject, the lens, and the CCD surface must be on the same straight line, and an observation equation is prepared for each light flux of each image. , The method of simultaneously adjusting the position and inclination of the camera (external orientation element) and the coordinate position of the second mark by the least squares method. “With self-calibration” is a method in which a calibration factor, that is, an internal localization (lens aberration, principal point, focal length) of the camera can be obtained. The collinear condition basic equations of the bundle adjustment method with self-calibration (hereinafter simply referred to as “bundle adjustment method”) are the following [Equation 2] and [Equation 3].

[0061]

[Equation 2]

[Equation 3]

[Equation 2] and [Equation 3] are based on the collinear conditional expression (5) of the single photograph orientation described in the first mark extraction processing. That is, the bundle adjustment method is [Equation 2]
And [Equation 3], the least-squares approximation is performed from a plurality of images,
This is a method of calculating various solutions, and it becomes possible to simultaneously obtain the external orientation elements of the camera at each photographing position. That is, it becomes possible to obtain the calibration element of the camera.

Next, as an internal localization correction model (lens aberration), an example in the case of having radial lens distortion is shown in the following [Equation 4].

[Equation 4] The correction model is not limited to this, and a model applicable to the lens used may be selected. These calculations are performed by the iterative solution if there are 6 or more reference points in ground coordinates and image coordinates. Note that the arithmetic processing unit 7 can obtain an accurate calibration element by omitting the second mark on the chart 1 that has a large error when the threshold value is equal to or greater than the threshold value of the iterative solution method. Therefore, in the target position center of gravity position confirmation (S70),
Even if the second mark having a large error is not detected, it is possible to detect and remove the erroneous second mark in S90.

Returning to FIG. 7, the calculation processing result for obtaining the calibration element by the calculation processing unit 7 is determined (S10).
0), if the calculation process does not converge, or if the obtained calibration element is not considered to be appropriate, then a countermeasure is taken in S110. In S110, the image including the erroneous second mark is selected. At the time of completion of the calibration in S90, the arithmetic processing unit 7 has determined which second mark of which image is erroneous. Therefore, the corresponding target detection point in each image is displayed and confirmed.

Then, the operator manually corrects the erroneous second mark (S120). That is, since the barycentric position coordinates of the erroneous second mark are displayed in a displaced manner, the mark displayed as the erroneous second mark is moved to the barycentric position displayed as aptitude, and the correction is performed. Is done. Then, it is judged whether or not the position correction of the erroneous second mark is completed (S130), and if completed, the process returns to the calibration element calculation of S90 to recalculate the calibration element. On the other hand, if there are other correction points, the process returns to S110 and the position correction operation of the erroneous second mark is repeated.

If the calculation processing result for obtaining the calibration element is appropriate, the result is displayed on the display unit 9 (S1).
40). FIG. 10 is an explanatory diagram illustrating an example of the calculation processing result of the calibration element. For example, the display unit 9 displays the focal length, principal point position, and distortion parameter that are calibration elements. It is easy to understand the distortion indicating the lens aberration by graphically displaying the pre-correction curve 102, the post-correction curve 104, and the ideal correction 106.

Further, based on the result of calibration, an image whose distortion has been corrected can be created by the image processing unit 8 and displayed on the display unit 9. By doing so, it is possible to provide an image display device that displays an image captured by a camera with large distortion after distortion correction.

Next, details of the process of calculating the external orientation elements (camera position and tilt) of the stereo cameras 2R and 2L necessary for three-dimensional measurement, which is performed in the arithmetic processing unit 7 or S90, will be described. (V): [Principle of Stereo Image Measurement] FIG. 11 is an explanatory diagram of the principle of stereo image measurement. An arbitrary point p (x, y, z) on an object such as the chart 1 is projected as a point P1 (x1, y1) on the image 1 via the one camera lens 1 of the stereo cameras 2R, 2L, A point P2 (x2, x2,
projected as y2). Images 1 and 2 have been subjected to orientation and deviation correction in advance, and their optical axes z1 and z2 have been corrected.
Are parallel and CCD (Charge
The distance a to the Coupled Device surface is equal, and the CCD is converted so as to be placed at right angles to the optical axis. If the distance between the two optical axes (baseline length) is B, the point P1 (x1, y
The following relationships exist between the coordinates 1) and P2 (x2, y2).

X1 = ax / z --- (8) y1 = y2 = ay / z --- (9) x2-x1 = aB / z --- (10) However, the whole coordinate system (x, y , Z) is the origin of the lens of the camera 1. Z is obtained from the equation (10), and using this, x and y are obtained from the equation (8) and the equation (9). That is, since the corresponding points (x1, x2) of the left and right images are found in the mark coordinate storage unit 10, the distance z from the lens principal point of the camera 1 to an arbitrary point p on the object is found.

Therefore, in the case where one camera is attached to the stereo bar 2B and the stereo cameras 2R and 2L are obtained by photographing from two places (see FIG. 1), the camera shown in FIG.
The chart 1 is photographed according to the flow chart of FIG. Then, the calibration element is calculated by performing measurement calculation processing on the two captured images, but since the bundle adjustment method is used for the calculation, the external orientation element is calculated at the same time. If the stereo images of the object are taken by the stereo cameras 2R and 2L under the same conditions as when the external orientation element is calculated, the three-dimensional measurement of the object can be performed by the stereo image.

That is, if the external orientation elements of the stereo cameras 2R and 2L are obtained, it is possible to perform three-dimensional measurement of an arbitrary object by the stereo camera whose external orientation element is known without the chart 1. Also, if the fixed position does not go wrong,
Conversely, it is also possible to obtain the external orientation elements of the stereo cameras 2R and 2L by measuring the three-dimensional measurement of the object whose shape is known. That is, since the positions and the tilts of the stereo cameras 2R and 2L are obtained when the three-dimensional measurement is performed, the three-dimensional measurement can be performed by the principle of the stereo method by correcting the positions and the tilts.

FIG. 12 is an explanatory view showing a display example of the external orientation element. When the external orientation elements of the stereo cameras 2R and 2L are obtained, the external orientation elements are displayed on the display unit 9.
The external orientation elements displayed on the display unit 9 include a baseline length corresponding to the distance between the stereo cameras 2R and 2L and a projection center point O.
There are coordinates (X0, Y0, Z0) of the model origin corresponding to c, and model rotation angles ω, φ, κ. Here, 1 after the decimal point
Although each numerical value is displayed with 0 digits, it may be displayed with an appropriate number of digits in view of measurement accuracy and the number of effective digits. Further, as the photographing parameters and the ground resolution, the photographing scale, the photographing distance (altitude), the photographing base line length, and the ground resolution (plane XY direction and height Z direction) may be displayed.

Further, in the present embodiment, if one or two cameras are attached to the stereo bar as a stereo camera and the calibration chart is photographed, the external orientation element is calculated by the stereo image processing device. Therefore, the following usage advantages are obtained. The camera does not have to be attached to the stereo bar strictly, but it does not have to move during measurement. Therefore, the camera and the stereo bar may be separated during transportation, and the camera may be attached to the stereo bar at the time of shooting, which makes the handling easier than in a conventional stereo camera. A stereo image of an arbitrary object may be measured with the stereo camera in the assembled state, and at the same time, the calibration chart may also be taken with the stereo camera to capture an image. If a calibration element that compensates for lens aberration is measured in advance, calibration can be performed even for images taken in stereo with an inexpensive lens with large lens aberration, especially in applications where even slight lens aberration causes problems, such as stereoscopic vision. It is possible to create an image that is less affected by lens aberrations by using the optical element, and easily stereoscopically view.

In the above embodiment, the case where one camera is attached to the stereo bar 2B and the stereo cameras 2R and 2L are formed by photographing from two places is shown, but the present invention is not limited to this. However, as shown in FIG. 13, for example, the stereo camera 2R,
2L has two cameras on the stereo bar 2B at a predetermined interval B
The chart 1 or the object may be stereoscopically photographed from a predetermined photographing direction.

[0075]

As described above, according to the stereo image processing apparatus of the present invention, the calibration chart is extracted from the calibration image taken by the stereo camera, and the first mark is extracted from the calibration chart. By performing a projective transformation using the first mark extracted by the extraction unit, a rough mark position measuring unit that obtains a rough position of the second mark in the calibration image, and the calibration image with respect to the calibration image. A precise mark position measuring unit for obtaining a photographed position of the second mark position in the vicinity of the approximate position of the second mark, a position of the second mark in the calibration chart, and the calibration corresponding to the second mark. From the position of the second mark in the calibration image, the external orientation element of the calibration image Since a configuration comprising a processing section for output, it is possible to determine the essential external orientation parameters for stereoscopic viewing using a stereo image.

[Brief description of drawings]

FIG. 1 is an overall configuration block diagram illustrating a first embodiment of the present invention.

FIG. 2 is a plan view showing an example of a calibration chart.

FIG. 3 is an explanatory diagram showing an example of a first mark.

FIG. 4 is an explanatory diagram showing an example of a second mark.

FIG. 5 is an explanatory diagram of a camera arrangement when measuring lens aberrations of a standard lens and a wide-angle lens.

FIG. 6 is an explanatory diagram of a camera arrangement when measuring lens aberration of a telephoto lens.

FIG. 7 is a flowchart illustrating a calibration method using a stereo image processing apparatus.

FIG. 8 is a detailed flowchart illustrating a processing procedure for precise position measurement of a second mark.

9A is an explanatory diagram of an image coordinate system and a target coordinate system in central projection, and FIG. 9B is an explanatory diagram of a template image of a normalized correlation used for target recognition and a target image.

FIG. 10 is an explanatory diagram showing an example of a calculation processing result of a calibration element.

FIG. 11 is a diagram illustrating the principle of stereo image measurement.

FIG. 12 is an explanatory diagram showing a display example of external orientation elements.

FIG. 13 is an explanatory diagram of stereo cameras 2R and 2L in which two cameras are attached to a stereo bar.

[Explanation of symbols]

1 chart (chart for calibration) 2R, 2L stereo camera 3 Image data storage 4 extractor 5 Schematic mark position measurement unit 6 Precision mark position measurement unit 7 Arithmetic processing section 8 Image processing unit 9 Display 10 Mark coordinate storage 11 Lens Aberration Compensation Parameter Storage Unit 12 External orientation element storage

Continued front page    F term (reference) 2F065 AA03 AA17 BB28 EE08 FF05                       JJ03 JJ05 JJ13 JJ26 QQ24                       QQ25 QQ31 QQ45                 5B057 AA20 BA02 DA07 DA17 DB03                       DB09 DC06 DC09                 5C054 AA01 AA05 CA04 CC02 EA01                       FC15 FD02 GA00 GB01 HA05                       HA16

Claims (9)

[Claims] 1. A stereo camera captures at least a calibration chart having a first mark provided at least at three or more locations and a second mark visually distinguishable from the first mark. An extracting unit that extracts the first mark from the two calibration images; and a projective transformation using the first mark extracted by the extracting unit to determine the approximate position of the second mark in the calibration image. A rough mark position measuring unit for obtaining; a precise mark position measuring unit for obtaining a photographed position of the second mark position in the vicinity of the rough position of the second mark with respect to the calibration image; The position of the second mark on the chart and the calibration image corresponding to the second mark A processing unit for calculating an external orientation element of the calibration image from the position of the second mark; 2. The stereo image processing apparatus according to claim 1, wherein the rough mark position measuring unit is configured to obtain a rough position of the second mark by projective transformation. 3. The stereo image according to claim 1, wherein the precision mark position measuring unit is configured to determine the second mark position by using at least one of template matching and barycentric position detection. Processing equipment. 4. The calculation processing unit is configured to detect the position of the second mark in the calibration chart and the second mark position.
The external orientation element in the second mark is calculated from the position of the second mark in the calibration image corresponding to the mark.
Item 3. The stereo image processing device according to item. 5. The display device according to claim 1, further comprising a display portion for displaying an external orientation element on the second mark.
The stereo image processing device according to any one of 1. 6. The arithmetic processing unit, when calculating an external orientation element for the second mark, selects an appropriate second mark from the second marks whose positions have been calculated by the precision mark position measuring unit. The stereo image processing apparatus according to claim 5, wherein the stereo image processing apparatus is configured as described above. 7. The stereo image processing apparatus according to claim 5, wherein the external orientation element includes at least one of a position, an inclination, and a baseline length of the stereo camera. 8. The arithmetic processing unit for calculating a calibration element for compensating lens aberration of the stereo camera; and correcting an image captured by the stereo camera into an image in which the lens aberration is compensated. An image processing unit; The stereo image processing apparatus according to claim 1. 9. A calibration chart having a first mark provided at least at three or more places and a second mark provided so as to be visually distinguishable from the first mark is photographed by a stereo camera, at least. A first step of extracting a first mark from the two calibration images; a first step of obtaining a rough position of the second mark in the calibration image by projective transformation using the extracted first mark; Step 2; the second step for the calibration image
A third step of obtaining a photographed position of the second mark position in the vicinity of the approximate position of the mark; the position of the second mark in the calibration chart, and the calibration image corresponding to the second mark And a fourth step of calculating an external orientation element of the calibration image from the position of the second mark in the stereo image processing method.