JP4280572B2 - Automatic orientation method using special marks - Google Patents

Description

[0001]
BACKGROUND OF THE INVENTION
In the present invention, two digital cameras are fixed to a fixed base line device, continuous shooting is performed, relative orientation is performed to obtain a relative orientation coefficient related to the model coordinate system, and the relative orientation coefficient is updated according to the shake of the digital camera. Related to automatic orientation method.
[0002]
Further, the present invention relates to a method for recognizing a sign when obtaining the relative orientation coefficient with the two digital cameras.
[0003]
[Prior art]
For example, in land reclamation work, earth and sand transported by a belt conveyor or truck is thrown into a soil transport ship from a ship loader traveling on a pier, transported to a target sea area, and discarded.
[0004]
At this time, the transport amount of the soil carrier was calculated based on the soil amount and the number of times. However, some carriers do not carry to the target sea area, but throw back about halfway along the way and return.
[0005]
For this reason, it is necessary to accurately measure the amount of input to the soil transporter when leaving the port and when entering the port.
[0006]
There are the following methods for measuring a large amount of loaded soil loaded on a soil carrier.
[0007]
(1) Measurement by human power
A large number of workers stand on the loaded soil on the ship, measure the level of the loaded soil to obtain a section of the loaded soil, and multiply the measurement interval to obtain the total amount of loaded soil.
[0008]
(2) Measurement by draft
Measure the draft of the soil carrier before and after the soil input, determine the weight of the loaded soil, and divide by the preset apparent specific gravity of the loaded soil to determine the amount of loaded soil.
[0009]
(3) Measurement with an ultrasonic distance meter
An array of ultrasonic receivers is arranged above the sediment loader (ship loader), and the shape of the loaded soil surface of the soil carrier is measured at the same time to obtain a cross-section of the loaded soil. Find the amount.
[0010]
(4) Measurement by optical wave rangefinder
Arrange the array of light wave rangefinders above the ship loader and measure the shape of the soil surface of the soil carrier at the same time, or turn one light wave rangefinder to determine the cross-sectional shape of the load soil, and measure the interval Multiply to find the total load capacity.
[0011]
However, this has problems such as that work safety cannot be ensured, measurement values are not accurate, and the system is expensive.
[0012]
For this reason, for example, Japanese Patent No. 3038447 (Patent Document 1), as shown in FIG. 19, arranges earth transport vessels 3 a and 3 b on both sides of the pier 2 provided with the ship loader 1, and one soil transport vessel 3 a. While the earth and sand are being loaded, the shape of the loaded earth and sand of one of the earth carrying ships is photographed by the digital camera 7 provided on the arm 6a, and the shape of the remaining earth of the other earth carrying ship 3b is taken by the digital camera 7 of the arm 6b provided on the other. The stereo image data is transmitted to the computer, and the cross-section of the load soil of one soil carrier and the remaining soil of the other soil carrier is measured by stereo image measurement with the computer, Is calculated.
[0013]
The above-mentioned survey by the digital camera is performed using stereo photographs taken simultaneously by two digital cameras fixed to the stereo base line device (arms 6a and 6b). Then, relative orientation is performed with a digital camera fixed to the stereo baseline device, and relative orientation coefficients (κ₁, ψ₁, κ₂, ψ₂, ω₂) Is calculated, model coordinates are calculated, and then the interval between the two digital cameras (baseline length) is taken as a scale, and the model coordinates are developed into a real space coordinate system.
[0014]
What is important in this method is that the coefficient (κ₁, ψ₁, κ₂, ψ₂, ω₂) And all the accuracy depends on how accurately this inclination is obtained.
[0015]
The method for obtaining the relative orientation coefficient using the stereo baseline device was to first determine the basic stereo model and perform the orientation manually by a professional engineer.
[0016]
[Patent Document 1]
Japanese Patent No. 3038447
Conventionally, a reference template is applied to a marker in a certain area, a difference is taken, and a marker position is determined when the difference is less than a specified value.
[0017]
[Problems to be solved by the invention]
The above method is optimal as long as the relationship between the left and right digital cameras is maintained after obtaining the relative orientation coefficient of the initial model. In other words, photogrammetry without a reference point is based on the premise that the object (soil carrier) never moves.
[0018]
However, when a digital camera is used for an arm outdoors or the like, the digital camera vibrates due to the influence of temperature change, external pressure (wind), vibration, etc., and the initial relative orientation coefficient changes, making measurement impossible and making it unusable (FIG. 15, See FIG.
[0019]
The specific change will be described later, but roughly speaking, the camera tilt relationship (κ₁, ψ₁, κ₂, ψ₂, ω₂) Will change and the epipolar geometry (see FIGS. 16, 17, and 18) cannot be constructed, resulting in a problem that the condition of the relative orientation coefficient is broken and greatly affects the accuracy. That is, as shown in FIGS. 17 and 18, which axis changes when the relationship between the left and right digital cameras is tilted in the three-axis direction, which affects the three-dimensional measurement value, particularly the Y-axis direction. (Κ₁, κ₂, ω₂) Value of about two pixels on the image, when the angle rotates more than a certain angle, vertical parallax occurs in the Y-axis direction, and the relationship between the displacement corrected images is lost and the geometric relationship (epipolar geometry) can be maintained. The three-dimensional measurement becomes impossible.
[0020]
That is, even if the digital camera is shaken, the sign can be recognized accurately, and the model is coordinated based on the recognition result to construct the model coordinate system so that the epipolar can be determined accurately.
[0021]
In particular, when photogrammetry is performed with a digital camera attached to the ship loader arm as in Document 1, the influence of sea breeze is significant.
[0022]
For this reason, when the object moved, there was a problem that the professional engineer was again seeking the relative orientation coefficient manually.
[0023]
The present invention has been made to solve the above problems, and even if the initial relative orientation coefficient changes due to the influence of temperature change, external pressure (wind), vibration, etc., the sign can be automatically recognized and automatically An object is to obtain a stereo base line automatic orientation method for obtaining a new relative orientation coefficient.
[0024]
[Means for Solving the Problems]
  On the object, continuous shooting is performed with at least two digital cameras fixed to a linear base line device so that a lower camera can be taken at a certain camera distance, and the left and right images are transmitted to the computer. This is an automatic orientation method in which the computer recognizes the sign so as to obtain a relative orientation coefficient and define a model coordinate system to automatically define corresponding points on the left image and the right image.
[0025]
  Around the objectProvided in each of the opposing rectangular special mark areas,As a sign, on a rectangular baseA black cross mark is added so that the remainder is left and right non-targetSpecial mark signs,in frontPhotographing the object in a field of view including the special mark sign with two digital cameras;I do.
[0026]
  And the computer
  The computer
  Before the special mark sign is likely to exist from the stereo image composed of the left image and the right imageSpecial mentionIn particular, the step of obtaining the pixel coordinates of the mark area sequentially,
  The image of the special mark area is moved by applying a reference plate having a pixel value corresponding to the special mark indicator from a specified position, and both residuals are sequentially obtained at the time of the application.TurnProcess,
  Each time the residual is determined, storing the reference plate in i regions separated by vertical and horizontal straight lines;
  Among the i areas that delimit the reference plate, the difference between the black mark area and the extra area of the special mark marker is obtained, and when these differences satisfy a certain condition, Determining a matching special mark sign that matches the reference plate;
  Obtaining pixel coordinates of the matching special mark sign from the stereo image;
  When the matching special mark mark is determined, the pixel coordinates of the matching special mark mark are replaced with the pixel coordinates of the special mark area, and this is used as the recognition coordinates of the mark.DoThis is the gist.
[0027]
  Around the objectProvided in each of the opposing rectangular special mark areas,As a sign, on a rectangular baseA black cross mark is added so that the remainder is left and right non-targetSpecial mark signs are provided,Downward on the object, with a certain camera distance to the linear baseline deviceTheContinuously shoot with at least two digital cameras fixed so that you can shoot, and capture the left and right images to the computerShiAutomatic orientation that defines a model coordinate system based on a relative orientation coefficient obtained in advance based on the special mark indicator recognized by the computer and automatically defines corresponding points on the left image and the right image Is the method.
[0028]
  The computer
  A stereo image consisting of a left image and a right image from the two digital cameras is taken into the interior, and the outline of the special mark indicator is determined by a residual sequential test method by moving a reference plate with respect to the left image and the right image. After recognizing the coordinates, the step of specifying the detailed coordinates with respect to the approximate coordinates in the fine position specifying process for obtaining the residual of the black portion with respect to the remainder of the special mark mark,
  Extracting a special mark sign corresponding to detailed coordinates of the special mark sign on the left image and the right image;
  The left image of the stereo image, the step of reading the detailed coordinates of the special mark indicator specified on the right image, performing relative orientation for each image to obtain each relative orientation coefficient,
  The vertical parallax standard deviation of each of the left image and the right image of the previous left image and the right image, in which the vertical parallax standard deviation of each of the left image and the right image of the previous time is set in advance. Comparing and determining whether the difference is less than the previous time;
  When it is determined that the difference is small, the vertical parallax standard deviations of the respective relative orientation coefficients of the previous left image and right image that have been obtained are determined as the vertical disparity coefficients of the respective left and right images. Step to replace with parallax standard deviation
TheDoAn automatic orientation method using special marks.
[0029]
DETAILED DESCRIPTION OF THE INVENTION
The method for obtaining the relative orientation coefficient by the fixed baseline device is to first manually specify a special mark Ai (Aai, Abi) for automatic recognition (described later) installed in the vicinity of the measurement target, and then use the basic model coordinate system (Xm, Ym, Zm). ). Next, a special mark Ai, which is a marker to be described later, is automatically recognized on the left and right stereo images (Ba, Bb) automatically every time the image is taken by a computer, and a combination of left and right identical points of the recognized special mark Ai is performed.
[0030]
Then, relative orientation is performed using the coordinates of these observed special marks, and when the vertical parallax standard deviation of the obtained relative orientation coefficient is smaller than the vertical parallax standard deviation of the previous model, automatically A system is adopted in which the relative orientation coefficient values are replaced so that measurement accuracy can be maintained without manual orientation.
[0031]
That is, as needed, the computer automatically extracts special marks on the stereo image, combines the marks on the left and right images, and obtains the relative orientation coefficient. When the longitudinal parallax standard deviation of the obtained relative orientation coefficient is smaller than the longitudinal parallax standard deviation value in the previous relative orientation coefficient, the mutual orientation coefficient value is automatically replaced.
[0032]
And this embodiment is
(1) A technology for automatically recognizing special marks placed in the vicinity of the measurement target on the left and right digital images taken in stereo using the template matching method.
[0033]
(2) A technology that combines marks recognized independently on the left and right images.
[0034]
(3) Technology that automatically replaces the initial relative orientation coefficient by comparing the standard deviation of vertical parallax.
[0035]
It is.
[0036]
<Embodiment 1>
A system using the above-described special mark will be described. In the present embodiment, as shown in FIG. 2, a cross-shaped special mark Ai is provided on a linear structure around the measurement object. The special marks Ai are provided, for example, at intervals of 2 m and 4 m. If the distance to the camera is several tens of meters, the white background (may be yellow) is set to about 50 mm, and the black background is set to 100 mm. Further, the margin (white background) of the special mark Ai is not targeted. This is to determine whether the special mark Ai is right or left.
[0037]
FIG. 1 is a schematic configuration diagram of an automatic orientation system using special marks according to the present embodiment. As shown in FIG. 1, the left and right digital cameras 10b and 10a are fixed to the arm 11, and the measurement object is continuously photographed at regular intervals under the control of the photographing control device 12, and this stereo image Ba (Ba1, Ba2,...), Bb (Bb1, Bb2,...) Is obtained in the database 30.
[0038]
The special mark Ai is recognized by the following units using the stereo image Bi of the database 30, and the previous initial relative orientation coefficient is automatically replaced by comparing the standard deviation of the vertical parallax.
[0039]
The analysis device 15 includes an initial relative orientation unit 17, a linear range determination unit 18, a special mark extraction unit 19, a template matching unit 20, a pairing unit 22, a calibration unit 23, and a displacement correction image creation. Part 24.
[0040]
When the initial relative orientation unit 17 first captures the measurement object (with the special mark Ai on the linear structure) at the same time by the two digital cameras 10a and 10b, the vertical parallax erasing method is used. To determine the relative orientation coefficient (κ₁, ψ₁, κ₂, ψ₂, ω₂) And stored in the database 26 in advance. Where ω₁  = Ω₂ It is.
[0041]
When the stereo images Bai and Bbi stored in the database 30 are selected, the linear range determination unit 18 takes out the selected stereo images, and sets the brightness level of the portion of the image excluding the preset mask area. Adjustment, binarization, and Hough transform are performed to determine the linear range Ci. That is, since the special marks Ai are arranged in a straight line, the rough range of the special mark Ai is automatically determined somewhere in the selected stereo image Bai, and the data of the range Ci is stored in the database 25. To do. However, the operator may decide. The special mark range Di can be understood by averaging vertical and horizontal straight lines by Hough transform.
[0042]
Each time the linear range Ci is determined, the special mark extraction unit 19 extracts a square area from the linear range Ci as the special mark range Di. At this time, the coordinates di (dai, dbi: pixel value) of the special mark range Di (Dai, Dbi) are output.
[0043]
The template matching unit 20 performs a residual residual method while moving a preset reference plate Ao (Aoa, Aob: Aob on the right, Aob on the left) pixel by pixel with respect to the special mark range Di in the database 26. To find the residual.
[0044]
When it is determined that there is a high possibility that the special mark Ai is located, a fine position specifying process for obtaining a difference between a white background and a black character of the special mark Ai described later is performed, and the special mark is determined based on the result of the fine position specifying process. The fine position Gi of the special mark Ai (area of Ai in Di) is determined by rewriting the coordinates di of the range Di.
[0045]
Such a process is performed for each special mark range Dai of the linear range Cai of the stereo image Ba and each special mark range Dbi of the linear range Cbi of the stereo image Bb.
[0046]
Each time the fine position Gi (initially input by the operator) of the special mark Ai in the left image and the right image is obtained, the pairing unit 22 reads the right image having the same number as the special mark AGai having the left image and the fine position Gai. The special mark AGbi having the fine position Gbi is compared (AGai, AGbi: collectively AGi), and it is determined whether the Y coordinate value in the camera coordinates is equal to or less than the reference value. The above comparison is based on the left image AGai (κ₁, ψ₁, κ₂, ψ₂, ω₂) Is used to calculate a predicted value developed on the right image, and the difference between the predicted value and each point AGbi at the fine position of the right image and the Y coordinate value is determined to determine whether it is equal to or less than the reference value.
[0047]
If it is determined that the value is equal to or less than the reference value, the candidate special mark Qp for the corresponding point (pair) is recognized (the candidate special mark for the corresponding point in the left image is Qap, and the candidate special mark for the corresponding point in the right image is Qbp).
[0048]
The calibration unit 23 calculates the left and right image corresponding points of five known points (special marks Ai) stored in advance in any one of the candidate special marks Qpi of the corresponding points of the stereo image obtained by the template matching unit 20. In addition, it is determined whether or not these vertical parallax standard deviations are equal to or less than a predetermined value. If the vertical parallax standard deviation is equal to or less than the predetermined value, it is determined that the coordinates of the candidate special mark Qp of the obtained stereo image (left and right images) are correct, The paired coordinates WQi of the previous six special marks obtained and stored in advance by the pairing unit 22 are replaced with the pair coordinates WQi, and the relative orientation coefficients (κ₁, ψ₁, κ₂, ψ₂, ω₂) Is updated based on the coordinates Gi (pair coordinates WQi) to obtain pair coordinates (left and right) in the photographic coordinate system.
[0049]
The deviation corrected image creating unit 24 uses the calibration unit 23 to calculate a relative orientation coefficient (κ) related to the previous model coordinate system.₁, ψ₁, κ₂, ψ₂, ω₂) Is updated based on the coordinates Gi (pair coordinates WQi), an epipolar line is generated between the pair coordinates.
[0050]
The operation of the system configured as described above will be described below. FIG. 3 is a flowchart for explaining the outline of the present embodiment.
[0051]
As shown in FIG. 3, the linear range determination unit 18 and the special mark extraction unit 19 perform linear structure extraction (S301). This linear structure extraction process is to identify the vicinity of the special mark Ai (also referred to as a special mark sign) on the photographic image (left or right).
[0052]
This process halves the vertical and horizontal size of the stereo image stored in the database.
[0053]
Next, contour detection filtering is performed, noise removal, expansion, and the like are performed, and mask images that have been facilitated in advance are superimposed, horizontal and vertical edges are extracted, and binarization processing and noise removal are performed. A linear range in which a special mark Ai exists is obtained by performing a Hough transform to detect a straight line component (vertical, horizontal), removing a Hough straight line larger than a specified angle, and averaging straight lines with close inclinations together. Ci is determined.
[0054]
Then, the end point coordinates (pixels) of the linear range Ci on the stereo image are obtained and stored (see FIG. 4A). Alternatively, when the special marks are not arranged in a straight line, HIS conversion is performed from RGB using the Munsell color system based on the color information, and a stereo photograph image, and yellow portions on the left and right sides are extracted and specified.
[0055]
Next, the template matching unit 20 performs template matching (S303). The pairing unit 22 and the calibration unit 23 perform the displacement correction conversion process (S305), the three-dimensional coordinate measurement (S306), and the distance volume area calculation process (S309).
[0056]
The linear structure extraction process and the template matching process will be described below with reference to the flowcharts of FIGS. In the present embodiment, the shooting interval of the digital cameras 10a and 10b is about 1/250 seconds, and the range consisting of the special mark Ai and the measurement object moves in a fixed direction at a constant speed, or the digital cameras 10a and 10b move in a fixed direction. Is moving at a constant speed.
[0057]
First, the operator selects the stereo images Bai and Bbi on the screen (S501).
[0058]
The linear range determination unit 18 performs a preset mask process on the selected stereo images Bai and Bbi to extract a region where the special mark Ai is shown (S502).
[0059]
Next, the brightness level of the extracted region is adjusted (S503), binarized (504), Hough transform is performed to generate horizontal and vertical segments, and each is averaged (S505). As a result, it is possible to assign the special mark Ai area (the linear range Ci) and the special mark Ai of the stereo images Bai and Bbi. That is, as shown in FIG. 4A, the coordinates of the end points (X and Y pixel coordinates) in the linear range Ci of the stereo image are stored in the linear range file.
[0060]
Next, one of the linear ranges Ci in the stereo images Bai and Bbi is set (S506). In the present embodiment, the linear range Cai of the stereo image Bai is used.
[0061]
The linear range determination unit 18 allocates the data of the linear range cai of the stereo image Bai from the database (S507), extracts the special mark range Dai of the linear range Cai, obtains its coordinates (pixels), Save (S509). In this extraction, since the special mark Ai has a cross shape, the portion with the special mark should be a square line segment by binarization processing and Hough transform. Further, since the special marks Ai are provided at regular intervals, it can be roughly predicted where in the stereo image. When a rectangular shape exists at such a predicted position, the position and data are extracted as the special mark range Di. 4B, for example, in the left stereo image Bai, the end point coordinates Lai (La1, La2) are associated with the linear range Ci of the left stereo image Bai, A special mark range number Dai is associated with the end point coordinate Lai, and the special mark number Dai is associated with the range mai (pixel coordinates of four corners) and the coordinate dai (special mark range file). .
[0062]
Next, it is determined whether or not there is another special mark range Dai in the linear range Cai of the stereo image Bai (S510).
[0063]
If it is determined in step S510 that there is another special mark range Dai, the search range is updated (S511). In the present embodiment, the pixel values corresponding to 2 m and 4 m are updated, and the process returns to step S508 again.
[0064]
If it is determined in step S510 that there is no other, it is determined whether or not all the linear ranges Cbi of the stereo image Bbi have been searched (S512). If not all have been searched, the image is updated to the stereo image Bbi (S513), and the process proceeds to step S507.
[0065]
That is, as shown in FIG. 7, the left and right stereo images Bai and Bbi are selected, and the linear range Ci shown in FIG. 7B is extracted.
[0066]
Then, as shown in FIG. 7C, the special mark extraction unit 19 cuts out the special mark range Di sequentially from the linear range Ci, and stores the coordinate values (see FIG. 4B) in the database. ing.
[0067]
Next, as shown in FIG. 6, the template matching unit 20 extracts the data of the special mark range Di, and takes this into the inside as an input image ri (rai, rbi) (S601).
[0068]
Next, the reference template Aoi is applied to the input image ri (S602).
[0069]
Here, the reference template will be described. The A / D conversion size is required to reflect 10 μm which is the particle size of the color film in practical operation. Therefore, if scanning is performed at 2400 dpi, the size of one pixel is about 10 μm. Even in analog photogrammetry, the 50 μ label is the smallest size that can be observed as a point sequence, and the 50 μ label is subjected to A / D conversion at 10 μ, and 4.7 × 4.7 pixels are used as the label. Therefore, for example, when the A / D conversion is performed on a photograph with a size of 50 μ in 1 / 10,000 photographing, the pixels are 4.7 × 4.7 pixels, and one image is 10.575 cm. ± 10 cm.
[0070]
For this reason, the shape of the image used for template matching is assumed to be data in RAW format with 256 colors, gray scale, width of 180 pixels, and vertical direction of 300 pixels (see FIG. 8B). That is, the pixel value corresponding to the special mark is obtained based on the focal length of the digital camera, the distance from the object, the scale ratio, and the like.
[0071]
The reference plate has a rectangular shape with a cross on a white background, and is divided into nine. That is, (1) white file, (2) black file, (3) white file, (4) black file, (5) black file, (6) The black file, (7) white file, (8) black file, and (9) white file correspond to the reference plate.
[0072]
Then, the template matching unit 20 performs a residual sequential test process (see FIG. 8A) (S603), and determines whether the residual is small (S604).
[0073]
If the residual is large, the reference template Aoi is moved for each pixel, the process returns to step S602, and the residual sequential test method is performed again (S605).
[0074]
If it is determined in step S604 that the residual is small, the input image ri and the area coordinates ki (pixel value) of the input image are extracted and stored (S606). Then, the fine position specifying process shown in FIG. 8C is performed (S607).
[0075]
As shown in FIG. 8 (b), the fine position specifying process is performed for the files (1), (2), (3), (4), (5), (6), (7), (8), and (9) corresponding to the reference template. Find the difference from the data. For example, ((1)-(2), (1)-(5), (1)-(4)) ((3)-(2), (3)-(5), (3)-(6) ▼) (7)-(4), (7)-(5), (7)-(8)) The difference of each area (file data) is obtained, and the reference template's (1) is obtained from each difference. (2) (3) (4) (5) (6) (7) (8) Whether the value is close to (9) or not is determined. In other words, it is determined whether or not each difference satisfies a certain condition (S608), and if it is determined that the certain condition is satisfied, the coordinates di of the special mark range Di obtained previously are determined as a fine position. The coordinate range of the fine position Gi of the special mark obtained by the processing is updated (S609). That is, a rough special mark range is extracted, and a special mark existing in the special mark range is specified from the special mark range by a fine position specifying process. For this reason, even if the digital camera is shaken by vibration, wind, etc., the area coordinates of the special mark Ai in the special mark range Di can be recognized.
[0076]
Next, it is determined whether or not there is another special mark range Di (S610). If there is no other special mark range Di, the process is terminated. If there is another special mark range Di, the number of the next special mark range Di is set. Update (S611).
[0077]
That is, the special mark Ai whose left and right margins are asymmetric is subjected to edge filter processing, binarization processing, and Hough transform processing on the original image, and roughly (if there is a color image, it is roughly captured using it) A range Di that is assumed to be captured) is specified, its region coordinates di (dai, dbi) are obtained and stored as shown in FIG. 9, and the data of the specified range Di is input image ri. As described above, the template Aoi is moved for each pixel, and the residual is obtained by the residual successive test method. When this difference is small, the area coordinates ki of the fine position Gi of the special mark range Di are obtained and stored as shown in FIG. Then, in order to specify the area of the special mark Ai in the range Di, the fine position is specified ((1)-(2), (1)-(5), (1)-(4)) ( (3)-(2), (3)-(5), (3)-(6)) ((7)-(4), (7)-(5), (7)-(8))) When the difference between the areas is obtained and the difference satisfies a certain condition, the area coordinates ki (four corner coordinates: special mark range) of the fine position Gi of the special mark range Di after the residual sequential test method are reliable. If possible, template matching is performed in which di (dai, dbi) is replaced with this ki.
[0078]
That is, as shown in FIG. 10, the left and right stereo images (captured images) are read to create projection / affine transformation images (S1001), and the left and right linear ranges Ci are cut out based on the coordinates of the end points (S1002). ), The search range is specified (S1003).
[0079]
Then, template matching is performed (S1004), and the coordinates (di) of the special mark Ai to be matched are acquired and left and right pairing is performed (S1005). The special mark position AGi having the coordinates (ki: fine position Gi) of the special mark Ai is referred to as a matching point (left number ai: a1, a2,..., Right number bi: b1, b2,...) Based on the template. And save. Further, the matching points may be displayed as symbols in the left image and the right image. For example, ○ mark or □ mark. Then, mosaic coordinates (corresponding points corresponding to the special mark positions) of the next captured stereo image (left and right) are extracted (S1006). These data are stored as shown in FIG.
[0080]
The above-described pairing and calibration will be described below with reference to FIGS.
[0081]
First, the linear range Ci (Cai, Cbi) of the left stereo image and the right stereo image is captured (S1101).
[0082]
Next, a file of coordinates (XL, YL, ZL) and (XR, YR, ZR: pixel values) of the captured image stored in advance when the six known points of the object are first captured is read ( S1102). This reads for the left and right cameras.
[0083]
Next, a pre-stored relative orientation coefficient (κ) corresponding to the coordinate values of the six points of this file is used.₁, ψ₁, κ₂, ψ₂, ω₂) (From the ori file). This is done for the left and right cameras.
[0084]
Next, each matching point ai of the left stereo image and its associated position, range, etc. are read (S1103), and the AGai coordinates (Gai) of this matching point ai are represented in the photographic coordinate system (Xm, Ym, Zm). (See FIG. 13) and save (see FIG. 14B) (S1104).
[0085]
Next, each matching point bi of the right stereo image and related positions and ranges are read (S1105), and the AGbi coordinates (Gbi) of these matching points bi are converted into a photographic coordinate system (Xm, Ym, Zm). And save (FIG. 14B) (S1106). Next, the record number Ra1 of the file storing the matching point data on the left image is set (S1107), and the record number Rbi on the right image corresponding to the record number Rai is set (S1108). Then, it is determined whether or not the special mark AGi exists in the record with the record number Rb1 on the right image (S1109).
[0086]
If it is determined in step S1109 that the special mark AGi does not exist on the right image, the record number Rbi of the left image is updated to the next number, and the process returns to step S1109 to determine the presence or absence of the special mark AGi ( S1110).
[0087]
When it is determined in step S1109 that the special mark AGi is present in the right image record, the Y coordinate of the special mark AGi of the right image is compared with the Y coordinate of the special mark AGi of the left image, and the difference is 50 μm or less. Whether or not (S1111).
[0088]
If it is determined in step S1111 that the difference is not less than 50 μm, the record number of the undate image is updated to the next number (S1112).
[0089]
If it is determined in step S1111 that the difference is 50 μm or less, the special mark AGi of the record Ra1 of the left image and the special mark AGi of the record Rb1 of the right image are determined as a pair (corresponding point), and their coordinates ( Pair coordinates) are acquired and stored as individual special marks Qp corresponding to the numbers (FIG. 14C) (S1113).
[0090]
Next, it is determined whether or not there is another record number Rai on the left image (S1114). If there is another record number, it is updated to the next record number Rai, and the process returns to step S1109 (S1115).
[0091]
By such processing, a pair WQi (WQai, WQbi) of each candidate special mark AGi in the left and right images is obtained as shown in FIG.
[0092]
Next, as shown in FIG. 12, all candidate special marks Qp (left and right) are read from the record of the pair file (S1201). Then, 5 points (5 points in step S1102) prepared in advance are added to one of these and saved in a file (S1201).
[0093]
Then, the relative orientation coefficient (κ₁, ψ₁, κ₂, ψ₂, ω₂) Is filed (S1203). Next, the vertical parallax (κ₁) Is 0.020 mm or less (S1204).
[0094]
In step S1204, the vertical parallax (κ₁) Is determined to be 0.020 mm or less, it is stored as a reliable pair coordinate WQi (P in FIG. 13) (S1205). Then, a deviation corrected image is generated (see FIG. 13) (S1206). Next, the six points in step S1102 are updated to the pair coordinates stored in step S1205 in the calibration process (S1206), and the process returns to step S1101.
[0095]
Accordingly, when the object is photographed at a constant interval with the digital camera, even if the digital camera vibrates, a relative orientation coefficient corresponding to the vibration state is obtained. Therefore, as shown in FIG. Since the condition that the corresponding points exist on the same record can be obtained, the epibol line does not shift.
[0096]
Therefore, when two cameras are optically fixed and a specific measurement target is measured at any time, the relationship between the cameras can be accurately obtained in real time. This is particularly effective when the optical relationship between cameras changes accidentally and systematically.
[0097]
【The invention's effect】
As described above, according to the present invention, the special mark mark with the cross is arranged around the object, and the outline of the special mark mark is displayed from the stereo image from the two digital cameras attached to the fixed baseline length device. Obtain a range, take out this range, and apply a reference plate with a shape corresponding to the special mark based on the distance to the camera, scale, and focal length to this range. Since the area is obtained and the fine position specifying process is performed on the special mark area, it is possible to specify the position of the special mark in the camera coordinate system even if the digital camera is shaken by wind, vibration or the like. .
[0098]
For this reason, when the relative orientation is performed, the relative orientation coefficient corresponding to the camera shake can be obtained, so that the epipolar line of the right and left images in the model coordinate system can be accurately obtained.
[0099]
Also, relative orientation is performed using the coordinates of these special marks, and when the vertical parallax standard deviation of the obtained relative orientation coefficient is smaller than the vertical parallax standard deviation of the model obtained previously, the relative orientation automatically Since the numerical value is replaced, the measurement accuracy can be maintained without manual orientation.
[Brief description of the drawings]
FIG. 1 is a schematic configuration diagram of an automatic orientation system using special marks according to the present embodiment.
FIG. 2 is an explanatory diagram of special marks arranged in a straight line.
FIG. 3 is a flowchart for explaining the schematic operation of the automatic orientation system of the present embodiment.
FIG. 4 is an explanatory diagram illustrating storage of end point coordinates of a linear range and storage of a special mark range.
FIG. 5 is a flowchart illustrating a linear structure extraction process and a template matching process.
FIG. 6 is a flowchart illustrating a linear structure extraction process and a template matching process.
FIG. 7 is an explanatory diagram for explaining selection and extraction of left and right stereo images;
FIG. 8 is an explanatory diagram for explaining template matching and fine position specifying processing;
FIG. 9 is an explanatory diagram illustrating replacement of fine positions.
FIG. 10 is a flowchart illustrating template matching.
FIG. 11 is a flowchart illustrating pairing and calibration processing.
FIG. 12 is a flowchart illustrating pairing and calibration processing.
FIG. 13 is an explanatory diagram for explaining conversion of a relative orientation coefficient and creation of a displacement correction image.
FIG. 14 is an explanatory diagram illustrating a pairing file generation process.
FIG. 15 is an explanatory diagram for explaining generation of epipolar lines;
FIG. 16 is an explanatory diagram of a digital image in which epipolar geometry is configured.
FIG. 17 is an explanatory diagram for explaining the influence of a shift due to an external factor on vertical parallax.
FIG. 18 is an explanatory diagram for explaining the effect of a shift due to an external factor on vertical parallax on a stereo image.
FIG. 19 is an explanatory diagram for explaining Patent Document 1;
[Explanation of symbols]
10a Digital camera
10b Digital camera
11 Arm
15 Analysis device
17 Initial relative orientation part
18 Linear range determination unit
19 Special mark extraction unit
20 Template matching section
22 Pairing part
23 Calibration section
24 Deviation correction image creation section

Claims (8)

On the object, continuous shooting is performed with at least two digital cameras fixed to a linear base line device so that a lower camera can be taken at a certain camera distance, and the left and right images are transmitted to the computer. The automatic orientation method for automatically defining the corresponding points on the left image and the right image by recognizing the marker and defining a model coordinate system by obtaining a relative orientation coefficient,
Said each provided in a rectangular shape of the special mark area facing the periphery of the object, as the label, special marks labeled marked with black cross marks as Amarube a rectangular base is horizontally asymmetric, before Photographing the object in a field of view including the special mark sign with two digital cameras;
The computer
The left image, before the stereo images consisting of the right image are likely the special mark labels exist sequentially pixel coordinates of Kitoku Koto mark region, a step of determining,
Wherein the image of the special mark area, is moved against the reference plate of the pixel values corresponding to the special mark labels from defined locations, both residual sequentially during the hit, and determined Mel step,
Each time the residual is determined, storing the reference plate in i regions separated by vertical and horizontal straight lines;
Among the i areas that delimit the reference plate, the difference between the black mark area and the extra area of the special mark marker is obtained, and when these differences satisfy a certain condition, Determining a matching special mark sign that matches the reference plate;
Obtaining pixel coordinates of the matching special mark sign from the stereo image;
When it is determined that the matching special mark labeling, by special mark, characterized in that the matching interchanged special mark signs pixel coordinates to pixel coordinates of the special mark region, performing the steps of this and the label recognition coordinates Automatic orientation method. The stereo image is displayed on the screen divided into a left image and a right image,
2. The method according to claim 1, wherein the recognized special mark mark is displayed with a predetermined symbol on the left image and the right image. The special mark labels, wherein the left and right of the object are arranged in a straight line in the special mark region, and the cross mark special mark labeled the left and right of the special mark region is asymmetrically to each other,
3. The automatic orientation method using a special mark according to claim 1, wherein the special mark area is detected by extracting a horizontal and vertical line segment from the stereo image. The reference plate is
The image of the special mark region, the special mark labeled Amarube has nine regions separated Kurobe, and said have assigned a gray scale value to form a special mark labels in each of the regions 4. An automatic orientation method using a special mark according to claim 1, 2, or 3. The reference plate is
5. The special mark according to claim 1, wherein the special reference mark is a left reference plate and a right reference plate corresponding to a shape of an extra part and a black part of the special mark markers in the left and right special mark regions. Automatic orientation method. Each provided rectangular special mark area facing the periphery of the object, as the label is provided with a special mark labels marked with black cross marks as Amarube a rectangular base is horizontally asymmetric, subject the Butsujo, continuously photographed at a fixed least two digital cameras to allow shoot downwards with a certain camera distance in a straight line baseline apparatus and captures the left image, right image to the computer, An automatic orientation method for defining a model coordinate system based on a relative orientation coefficient obtained in advance based on the special mark indicator recognized by the computer and automatically defining corresponding points on the left image and the right image Because
The computer
A stereo image consisting of a left image and a right image from the two digital cameras is taken into the interior, and the outline of the special mark indicator is determined by a residual sequential test method by moving a reference plate with respect to the left image and the right image. After recognizing the coordinates, the step of specifying the detailed coordinates with respect to the approximate coordinates in the fine position specifying process for obtaining the residual of the black portion with respect to the remainder of the special mark mark,
Extracting a special mark sign corresponding to detailed coordinates of the special mark sign on the left image and the right image;
The left image of the stereo image, the step of reading the detailed coordinates of the special mark indicator specified on the right image, performing relative orientation for each image to obtain each relative orientation coefficient,
The vertical parallax standard deviation of each of the left image and the right image of the previous left image and the right image, in which the vertical parallax standard deviation of each of the left image and the right image of the previous time is set in advance. Comparing and determining whether the difference is less than the previous time;
When it is determined that the difference is small, the vertical parallax standard deviations of the respective relative orientation coefficients of the previous left image and right image that have been obtained are determined as the vertical disparity coefficients of the respective left and right images. An automatic orientation method using a special mark, characterized by performing a step of replacing with a parallax standard deviation. Performing the step of extracting as the corresponding points when the special mark indicator of the right image corresponding to the special mark indicator of the left image, and converts the photographic coordinates, the difference between both of the Y-coordinate meets the reference value The automatic orientation method using a special mark according to claim 6 . The computer
Steps for sequentially obtaining pixel coordinates of a rectangular special mark region where the possibility of the presence of the special mark sign from a stereo image consisting of the left image and the right image, and
Wherein the image of the special mark area, is moved against the reference plate of the pixel values corresponding to the special mark labels from defined locations, both residual sequentially during the hit, and determined Mel step,
Each time the residual is determined, storing the reference plate in i regions separated by vertical and horizontal straight lines;
Among the i areas that delimit the reference plate, the difference between the black mark area and the extra area of the special mark marker is obtained, and when these differences satisfy a certain condition, Determining a matching special mark sign that matches the reference plate;
Obtaining pixel coordinates of the matching special mark sign from the stereo image;
When it is determined that the matching special mark label, and characterized by performing the step of said matching interchanged special mark signs pixel coordinates to pixel coordinates of the special mark area, special mark label recognition coordinates This was the recognized An automatic orientation method using a special mark according to claim 6 or 7 .

Images