JP2723174B2 - Registration correction method between heterogeneous sensor images - Google Patents

Description

Description: BACKGROUND OF THE INVENTION 1. Field of the Invention The present invention relates to an image registration (registration) correction process for automation by a computer, and more particularly to a sensor for a satellite image processing system. The present invention relates to a heterogeneous sensor image registration correction method suitable for automatic registration of images obtained by a plurality of sensors having different sensitivity characteristics (hereinafter, referred to as heterogeneous sensor images). 2. Description of the Related Art The prior art of image registration processing is disclosed in "Image Recognition Theory" by Makoto Nagao (Corona Co., Ltd., Showa 5).
"Development of Image Registration Processing User Support System" by Yoichi Seto, Hideji Nishijima and Shuu Tezuka (Remote Sensing Society, pp. 27-37, V
ol. 10, No. 4, 1991). The outline of the processing is as follows. (1) Match the characteristics (spatial resolution and average luminance level) of a sensor image (template image) as a reference for normalizing an image with the characteristics of another sensor image (search image). (2) Calculation of registration error The amount of positional error between the sensor images to be compared is measured. Measurement methods include a method of automatically performing image matching using spectral characteristics and shape characteristics as feature amounts and a correlation coefficient as similarity, and a method of an operator indicating corresponding points on a heterogeneous sensor image. is there. (3) Interpolation process The positional relationship between pixels in two images (the template image and the search image) is represented by a polynomial using the error amount obtained in the registration error calculation process, and the pixel position is rearranged. (Resampling process). Pixels other than the integer coordinates are subjected to interpolation using a cubic convolution method or the like. As described above, in the prior art, a method of comparing sensor images of the same type or a method of instructing a corresponding point by an operator in the case of different types of sensor images. [0008] The conventional method is an automatic process when applied to registration between similar sensor images having similar spectral characteristics, and operates a corresponding point between different sensor images having different spectral characteristics. Registration processing when the user gives an instruction. As for the corresponding point, a position with a good matching accuracy could be registered in advance. Therefore, when the conventional method is applied to registration processing between different types of sensor images having a large number of images to be processed and having different spectral characteristics, the following problems occur. (1) Since the spectral characteristics of the same ground point, that is, the luminance level distribution is different between the two images, a correct result is obtained even if an image matching process using a correlation value (feature amount) obtained from the luminance level is performed. Can not be obtained. For example, there is almost no luminance level correlation between the visible image and the thermal infrared image, and the image matching process cannot be performed correctly. (2) When the operator performs the image matching process by manually instructing the image displayed on the display, the processing data is large and the operator cannot designate the corresponding area within a realistic time. . For example, assume that when processing images of 1000 scenes per day, it is necessary to process 100 corresponding regions for one scene. Assuming that the processing time for each area is 2 minutes, it takes about 3000 hours to perform all the processing. If this is processed in eight hours, for example, about 400 operators are required. An object of the present invention is to improve the accuracy of registration processing between different types of sensor images and to automate the registration processing of a large amount of image data. [0014] In order to solve the above-mentioned problems, the following processing is performed. (1) In order to enable matching processing of different types of sensor images having different spectral characteristics, a corresponding area necessary for selecting an optimal feature amount for image comparison and detecting a position error between two images. A plurality of (calculation regions) are set, and the corresponding regions for which the position error has been correctly calculated are narrowed down by a multi-stage and threshold determination made of a rough matching process and a fine matching process. (2) After narrowing down the corresponding areas as candidates, a candidate selection process is provided for the operator to specify the corresponding area. (3) If the corresponding area cannot be determined with high accuracy, substitute data is used using the most similar position error. (4) A position error is calculated using the enlarged image. Alternatively, the position error is obtained with high accuracy by interpolating the calculated feature amount. (1) A plurality of corresponding regions are provided in a heterogeneous sensor image, and a corresponding region having a similar feature amount is selected by threshold determination to improve measurement accuracy. Also, the processing time is reduced by multi-stage processing and narrowing down the candidates so that the measurement accuracy is improved in later stages. (2) By narrowing down candidates, the number of candidate corresponding areas is reduced, so that an operator can input an instruction. (3) In the case of a satellite sensor, since the sensor pointing error depends on time, if image matching processing such as a cloud area cannot be determined with high accuracy, it is most likely that the target image to be processed is photographed at the most appropriate time. By using close data as an approximate value, a processing impossible state is prevented. (4) By enlarging an image or interpolating a feature amount, a position error can be accurately obtained in real number coordinates. Embodiments of the present invention will be described below in detail with reference to the drawings. FIG. 2 is a schematic diagram of a satellite terrestrial information system to which the present invention is applied. A desired area on the ground surface is observed by an observation satellite 10 for the purpose of resource exploration and environmental monitoring, and the observed image data is processed by a terrestrial system 20. . The observation satellite 10 is equipped with two different types of sensors having different spectral characteristics. As shown in FIG. 3, the sensor 1 (30) is a sensor having sensitivity in the visible and near-infrared wavelength ranges, has a spatial resolution of 10 m, and has a sensor 2 (30).
(40) is a sensor having sensitivity in the thermal infrared wavelength range, and has a spatial resolution of 40 m. Shoot 60km square at a time. The satellite orbits the earth about 15 times a day and makes 100
Observe 0 scenes. Each sensor has a pointing mechanism that changes the observation angle in a direction orthogonal to the satellite traveling direction (orbit direction). Although there are devices that control and measure the pointing angle, they cannot be controlled and measured accurately, and an error occurs in the set angle even when pointing in the same direction (the dotted line in FIG. 3 indicates the field of view when a pointing error occurs). Shown). For example, when a pointing error occurs at 125 microradians, and when the ground surface is observed from 800 Km above the ground, a position error of 100 m occurs on the ground surface. Therefore, when two sensor images 92 and 94 are superimposed as shown in FIG.
0 results. This error 50 is compared to the ground system 2 in FIG.
It must be removed at zero. The contents of the image registration correction processing in the ground system 20 will be described below with reference to the drawings.
The surface data observed by the observation satellite 10 is received by the receiving antenna 12 on the ground and input to the ground system 20. Each of the input sensor images is subjected to a sensor-specific system correction 60 and a correction image 70 by performing various processes, and is distributed to the user 80. The details of the processing will be described below using the sensor 1 image as an example (the same applies to the sensor 2 image). (1) System Correction Process 60 Specific to Sensor The system correction process 60 specific to the sensor 1 is for correcting geometric distortion and radiation intensity distortion. This processing is, for example, by Yamagata, remote sensing image processing by earth observation satellites,
67-72 (Hitachi Review, Vol. 64, No. 6, 198)
(June 2 years). The processing result is a format-converted observed image 90 and a correction coefficient for geometric distortion and radiation intensity distortion (hereinafter, only the geometric distortion correction coefficient related to the present invention is simply referred to as a distortion correction coefficient) 100. . These are stored in a recording medium. (2) Registration process 110 When the corrected sensor 1 image 92 and sensor 2 image 94 are superimposed as shown in FIG. 4 using the distortion correction coefficient 100, the system correction cannot be considered. A pointing error of the sensor remains, and an overlay error 50 occurs on the image. The registration process 110 corrects this error, and a method for realizing this is the present invention. In this processing, a corrected distortion correction coefficient 130 for correcting a pointing error is calculated using each observed image 90 and the distortion correction coefficient 100. (3) Resampling process 140 This is a process for calculating the corrected image 70 from the distortion correction coefficient 100 such as the radiant intensity distortion, the corrected distortion correction coefficient 130, and the observed image 90. The above-mentioned document (Hitachi Review, Vol. 64) )
Therefore, the details will be omitted. The details of the registration process 110, which is the center of the present invention, will be described with reference to FIG. (1) Pre-processing 150 Image for obtaining registration error (corresponding band)
For f ′ and g ′ (160), the distortion correction coefficient φ1,
Processed using φ2 (155) and corrected images (distortion-corrected sensor 1 image and sensor 2 image) f-, g- (17
0) is created. Here, f- and g- indicate that "-" is added immediately above f and g, respectively. (2) Image normalization processing 180 The distortion-corrected sensor 1 image f- and sensor 2 image g-
Image characteristics are matched. Specifically, the spatial resolution and the average luminance level are adjusted. The obtained image is f,
g (170). As for the spatial resolution, one pixel of the image of the sensor 2 (spatial resolution of 40 meters) is resampled four times vertically and horizontally so as to correspond to 10 meters of the image of one sensor. Also,
A histogram of the brightness level is obtained, and the histogram is adjusted so that the average and the variance of the histogram match. Specifically, Takagi and Shimoda supervised, Image Processing Handbook (published by The University of Tokyo Press, January 1991)
Month) pages 463 to 465 and 425 to 429. (3) Registration error calculation processing 1
90 The registration error calculation processing 190 is constituted by "FIG. 1". (A) Rough Matching Processing 200 As shown in FIG. 6, a template area f202 and a corresponding search area g (20) are located at random positions in an image.
4) is set, and matching is performed with an accuracy of an integer pixel. The features to be used are determined in advance in consideration of sensor characteristics and the like. In comparing the same type of sensor image and similar sensor images, the similarity may be obtained by the area correlation γ represented by “Equation 1” using the luminance level value. ## EQU1 ## Here, f ~ and g ~ are "~" immediately above f and g, respectively.
Means the average of f and g. In the case of registration processing between different types of sensor images, since there is not always a positive correlation between the luminance levels,
After obtaining the shape features and the like, the similarity is obtained. As the shape feature, DP matching, Fourier descriptor, texture feature, edge feature, and the like may be used. here,
The processing using the edge feature will be described. In the edge extraction, after the image is blurred, the filter processing of "Equation 2" and "Equation 3" which take the Laplacian filter into consideration is performed, and then the zero crossing method (supervised by Takagi and Shimoda,
Edges are extracted as line information from pages 550 to 582 of the image processing handbook (published by The University of Tokyo Press, January 1991). For example, to extract edges from the image g, ## EQU3 ## Here, σ: standard deviation, ＠: convolution. The frequency characteristics of the edges to be extracted can be selected by changing the standard deviation σ as a parameter. After binarizing the image g * obtained by "Equation 2", the correlation value can be obtained by "Equation 1".
Here, a correlation value is obtained by “Equation 1” as a grayscale image without binarization. In addition, it is also possible to obtain a correlation value with respect to an image obtained by applying a low frequency pass filter (for example, a Gaussian filter) to the image obtained by “Equation 2”, “Equation 3” and the zero crossing method. It is valid. The coordinates having the largest value of the obtained correlation values are obtained from the image, and the position error (registration error) is calculated. That is, the coordinates (sm,
tm) as a matching position, and the amount of deviation (σx, σy) from the position (s0, t0) of the reference image (for example, the image of the sensor 1) is determined from “Equation 4” and “Equation 5” as a registration error. (Equation 4) (Equation 5) The above processing is performed for each template area. The obtained correlation value γ (i), the error amounts σx (i) and σy (i), and the number i of the matching area (region) are transferred to the next processing. In this embodiment, i is 10 points. (B) Candidate area selection processing 210 Using the magnitude of the position error and the direction in which the error (deviation) occurs, the abnormal value is removed by threshold value judgment. Others are set as candidate areas (areas). In order to explain the flow of the candidate area selection process based on the threshold determination shown in FIG. 7, the error distribution shown in FIG. 8 is assumed. FIGS. 9 and 10 show the error distribution and error amount in each candidate area in FIG. For example, the area where i = 1 is the correlation coefficient γ
(1) = 0.9, error amount σx (1) = 4 pixels, σy
(1) = 0 pixels. First, in block 710, areas having a correlation coefficient γ of 0.5 or less for all areas are removed from candidates (γ determination). The area to be removed is i = 3,6. With respect to the remaining area, the angle r between the position error magnitude r and the x-axis is calculated in block 720. In the area where i = 1, for example, r = 2 and θ = 0. Next, in block 730, the error magnitude and direction average r−, θ− (indicating that “−” is added immediately above r and θ, respectively) and deviations δr, δθ of the candidate area are calculated. The calculation 230 is performed. r = 3.2, δr = 1.2, θ
= 20.25 and δθ = 46.5. Next, at block 740, within θ ± δθ, and at block 750, r ±
Exclude areas not within δr. In the former θ determination, i = 7 and 9 are removed from the candidates, and in the latter r determination, i = 2 is removed from the candidates. Eventually, at block 760, i = 1, 4,
5, 8, and 10 are registered as candidate areas. (C) Precise Matching Process 220 In order to calculate the matching position with high accuracy, the candidate areas (i = 1, 4, 5,
8, 10) are enlarged and interpolated by a method with little deterioration in image quality.
For example, using a cubic convolution kernel, the image is magnified 10 times. The same processing as the rough matching processing is performed on the enlarged area, and the position error between the template image and the search image is obtained. (D) Error Calculation Process 230 The same abnormal value determination process as that shown in FIG. 7 is performed, and the average value of the error of the obtained data of the candidate area is obtained. This is the final registration error. FIG. 10 shows the error amount calculated by the precision matching process (a value enlarged by 10 times). (B) above
The abnormal value is removed by the threshold determination processing similar to the above. "Figure 1
In "0", i = 1 and 5 are removed. Using the remaining candidate areas (i = 4, 8, 10), a registration error corresponding to the resolution of the original image is calculated in consideration of the 10-times magnification. That is, 3.1 pixels are obtained in the x direction and 0.2 pixels are obtained in the y direction (rounded off to the second decimal place). Hereinafter, description will be made returning to FIG. (4) Error Retrieval Process 270 In the registration error calculation process 190, if the processing target scene is nighttime, sea area, cloud area, or the like, the image matching processing is not performed well. In other words, when the deviation is extremely large, or when the registration error is not accurately determined, such as when the value becomes significantly larger than the control / measurement error known in advance, the correction of the geometric distortion correction coefficient In processing 280, the error amount obtained from the image closest to the shooting time of the processing target image is stored in the error file 2
Search from 75 and use. (3) Geometric distortion correction coefficient correction processing 28
0 Using the error amount (Δx, Δy) obtained in the registration error calculation processing 190, the geometric distortion correction coefficient φ1 (a
0, a1, a2, a3), φ2 (b0, b1, b2, b
Modify 3) (φ represents a set of coefficients). As shown in FIG. 11, the geometric distortion correction coefficients φ ₁ and φ ₂ (214) indicate the geometric relationship between the corrected image 208 and the uncorrected image 206. This coefficient is set for each block 212. In general, a bilinear equation shown in “Equation 6” and “Equation 7” is used as the relational expression. [Mathematical formula-see original document] [Mathematical formula-see original document] From the error amounts (Δx, Δy), the modified geometric distortion correction coefficients φ1 ′ and φ2 ′ (290) are obtained as shown in “Equation 8” and “Equation 9”, and are stored in the recording medium. [Mathematical formula-see original document] [Mathematical formula-see original document] Where: [Mathematical formula-see original document] [Mathematical formula-see original document] ## EQU13 ## ## EQU14 ## (Equation 15) ## EQU16 ## [Mathematical formula-see original document] If the error is a bias error, the error may be considered as a bias in the distortion correction coefficient as described above. However, if the error is a higher-order error, the error amount is reduced to the sensor-specific system correction 60 in FIG. It is necessary to take measures by feeding back and calculating the distortion correction coefficient again. As an alternative to this embodiment, instead of the precision matching process 220 in FIG. 1, a candidate selection process 243 in which the operator 244 selects a final candidate area from the candidate areas as shown in FIG. May be provided. If the candidate area can be reduced to about 1/10 by the rough matching processing 200, the above processing becomes possible. Further, the candidate selection process 243 by the operator 244 may be performed only on an image for which the rough matching process 200 does not work. The precision matching process 220 calculates the correlation value after enlarging the image at a desired magnification, but calculates the correlation value by “Equation 1” without enlarging the image and then calculates the correlation value as shown in FIG. The real number coordinate value 265 at which the correlation value becomes maximum may be calculated by interpolating the values. The invention described above is applicable to a satellite terrestrial information system, and is particularly effective in automating the registration process of a large number of images photographed by different types of sensors. According to the present invention, the following effects can be obtained. (1) The number of candidate areas can be reduced by making the number of stages of the matching process coarse and fine, and the load of the error calculation process can be reduced. Further, the operator can intervene to specify a candidate area. (2) By adding a candidate area selection process based on threshold value determination as a function, it is possible to efficiently select a corresponding area having similar error characteristics. (3) In the precise matching process, the position of the corresponding area area can be obtained with high accuracy by enlarging the image and obtaining a correlation value, or by interpolating the correlation value. (4) Matching accuracy can be improved by selecting a feature amount and a similarity based on a combination of images. (5) In the error calculation process using the threshold,
By removing an abnormal value by using an average value or a deviation from the average value of the position error of the corresponding area, the corresponding area error can be obtained with high accuracy. (6) The registration error amount is temporarily stored in the error amount file, and in the case of data obtained by photographing a cloud, the sea, nighttime, or the like, a value closest to the time from the stored error amounts is used. By searching and using the corrected geometric distortion correction coefficient calculation processing, an image that cannot be processed can be eliminated.

BRIEF DESCRIPTION OF THE DRAWINGS FIG. 1 is a calculation flow of a registration error by multi-stage threshold value determination. FIG. 2 is a schematic diagram of a satellite ground information system to which the present invention is applied. FIG. 3 is a schematic diagram illustrating a relationship between a pointing function of a sensor and an error. FIG. 4 is an explanatory diagram of an overlay error between two sensor images. FIG. 5 is a registration processing flow. FIG. 6 is a schematic diagram of an image matching process between a template image and a search image. FIG. 7 is a flowchart of a candidate area calculation process based on threshold determination. FIG. 8 is an explanatory diagram of a candidate area calculation process based on threshold determination. FIG. 9 is a table showing characteristics of an error caused by a rough matching process and a result of a determination process. FIG. 10 is a table showing characteristics of an error caused by a precision matching process. FIG. 11 is an explanatory diagram of a geometric distortion correction coefficient. FIG. 12 is a registration error calculation processing flow in consideration of an operator candidate selection processing; FIG. 13 is an explanatory diagram of an alternative method for calculating an error. [Description of Signs] 200: Coarse matching process, 210: Candidate area calculation process by threshold value, 212: Fine matching process, 213 ...
Error calculation processing using a threshold.

   ────────────────────────────────────────────────── ─── Continuation of front page    (56) References JP-A-4-76788 (JP, A)                 JP-A-63-118889 (JP, A)                 JP-A-4-307682 (JP, A)

Claims (1)

(57) [Claims] [Claim 1] (a) Each image feature for a different sensor image
Normalization for matching between genders is performed. (B) The corresponding area between the normalized heterogeneous sensor images
Into individual sets, for between corresponding regions set (c), set in the image
The feature amount of the corresponding point between the template image and the search image
Calculate and compare the similarity of the feature amount, and respond to the high similarity
The coordinates of a point are determined by integer positions.
Calculating the similarity between the corresponding regions and the registration error, and (d) calculating the similarity and the registration error.
(E) Precise matching processing between the selected candidate corresponding areas
Of similarity between candidate corresponding areas and registration scheme
Calculating the difference, and (f) calculating the calculated similarity and the registration error.
Based narrow down the Hou complement corresponding region, Registration cane between Hou complement corresponding regions narrowed (g)
Registration between heterogeneous sensor images based on the registration error
Calculating a down error, to registration errors calculated finally (h)
To correct registration between different types of sensor images.
A method for correcting registration between different types of sensor images. 2. The method according to claim 1, wherein the candidate area selection process comprises the step of:
The direction of the position error from the coordinates of the corresponding point
Calculate the deviation of the error from the average value, and
2. A method according to claim 1, wherein similar corresponding points are selected.
Registration correction method. [Course 3] The precision matching process is based on the rough match
Real number that maximizes the feature value of the corresponding point obtained by
3. The heterogeneous sensor image according to claim 2, wherein the coordinates are obtained by an interpolation process.
Registration correction method. 4. The method according to claim 1, wherein the characteristic amount is a luminance level value, a spec.
Toll characteristics, edge features, texture features, Fourier description
At least one child is used, and the similarity is calculated using the difference value
2. The register according to claim 1, wherein a correlation value is used.
Stress Shon correction method. 5. The calculation of the registration error of (g).
The processing is based on the corresponding points determined by the precision matching processing described above.
Using the average value of the position error or deviation from the average value,
Calculate the error amount using only corresponding points from which outliers have been removed
A method for correcting registration between different types of sensor images according to claim 1.
Law. 6. The above rough and fine matting treatment.
After applying a low-pass filter to the image,
2. The method according to claim 1, wherein a shape feature calculated by performing an extraction process is used.
Registration correction method between different types of sensor images. 7. An image extracted by the edge extraction processing.
Edge as binary data or shading data.
7. Registration between different types of sensor images according to claim 6.
Correction method. 8. The edge extraction processing according to claim 1 , wherein
Applying a low-pass filter to the processed data
8. The registration rate between different types of sensor images according to claim 7, wherein
Correction method. 9. The precision matching process according to claim 1 , wherein
And the search image is enlarged and the correlation value at integer coordinates
And the coordinate position that takes the maximum inter-phase value is the real number before enlargement.
4. The register according to claim 3, wherein the register is obtained by converting the coordinate value into a coordinate value.
Installation correction method . 10. The registration error calculation of (g).
The output process calculates the registration error amount as the error amount.
Save in a file, save in the case of clouds, sea, night
Search for the closest value in time from the error amount specified and correct it
2. The different type according to claim 1, which is used in a geometric distortion correction coefficient calculation process.
Registration correction method between sensor images. 11. The method according to claim 1, wherein the processing of (c) and (d) is replaced with the processing of
The corresponding area set in (b) is set as a candidate corresponding area.
2. A register between different types of sensor images according to claim 1, wherein the process of (e) is performed.
Torsion correction method. 12. The processes (c) to (f) are repeated a plurality of times.
2. A registration supplement between images of different types of sensors according to claim 1.
Right way. 13. An operation in place of the processes (e) and (f).
3. The heterogeneous method according to claim 2 , wherein a candidate area narrowing process is performed by a user.
Registration method between sensors.