JPH08161474A - Method for correcting registration between images by different kinds of sensors - Google Patents

Abstract

PURPOSE: To attain high speed, highly accurate and automatic registration processing for lots of image data obtained from plural sensors whose sensitivity characteristics differ from each other. CONSTITUTION: In the image registration processing consisting of normalizing processing, registration error calculation processing and re-sampling processing of an image, the registration error calculation processing is executed by a rough matching processing 200, an object area calculation processing 210 based on a threshold level, a precise matching processing 220 and an error calculation processing 230 based on a threshold level. Since multi processing stages as the rough and the precise processing are adopted for the registration error calculation processing, it is possible to decrease number of object areas and the load of the error calculation processing is relieved and the processing speed is improved. Furthermore, the inter-image registration processing for lots of image data picked up by different kind of sensors is automated.

Description

Detailed Description of the Invention

[0001]

BACKGROUND OF THE INVENTION 1. Field of the Invention The present invention relates to an image registration (registration) correction process for the purpose of automation by a computer, and in particular, in a satellite image processing system, it is obtained by a plurality of sensors having different sensitivity characteristics. The present invention relates to a heterogeneous sensor image registration correction processing method suitable for automatic registration processing of an image (hereinafter referred to as a heterogeneous sensor image).

[0002]

2. Description of the Related Art The prior art of image registration processing is described in "Image Recognition" by Shin Nagao (Corona Corp., Showa 5).
(Published in Feb. 8) and Yoichi Seto, Eiji Nishijima, and Shouyu Tezuka "Development of Image Registration Processing User Support System" (Remote Sensing Society, pp.27-37, V.
ol. 10, No. 4, 1991).

The outline of the processing is as follows.

(1) Normalization of image The characteristics (spatial resolution and average luminance level) of the sensor image (template image) serving as a reference are matched with the characteristics of another sensor image (search image).

(2) Registration error calculation The amount of position error between sensor images to be compared is measured. As the measurement method, there are a method of automatically performing image matching using a spectral characteristic or a shape feature as a feature amount and a correlation coefficient as a degree of similarity, and a method in which an operator designates corresponding points on different sensor images. is there.

(3) Interpolation process Using the error amount obtained in the registration error calculation process, the positional relationship between pixels in the two images (the template image and the search image) is expressed by a polynomial, and the pixel position rearrangement process is performed. (Resampling process) is performed. Pixels other than integer coordinates are interpolated using the cubic convolution method or the like.

As described above, the prior art has been a method of comparing and processing sensor images of the same type, or a method of instructing corresponding points by an operator in the case of different types of sensor images.

[0008]

According to the conventional method, an operator indicates an automatic process when applied to registration between similar-type sensor images having similar spectral characteristics, and an operator indicates a corresponding point between different-type sensor images having different spectral characteristics. It was premised on the registration process when doing. In addition, as the corresponding points, positions with good matching accuracy could be registered in advance. Therefore, when the conventional method is applied to the registration process between the different types of sensor images having a large number of images to be processed and different spectral characteristics, the following problems occur.

(1) Since the spectral characteristics of the same ground point, that is, the brightness level distributions of the two images are different, a correct result is obtained even if the image matching process using the correlation value (feature amount) obtained from the brightness level is performed. Can't get

For example, there is almost no brightness level correlation between a visible image and a thermal infrared image, and image matching processing cannot be performed correctly.

(2) When the image matching process is performed by the operator manually instructing the image displayed on the display, the amount of processing data is large, and the operator cannot instruct the corresponding area within a realistic time. .

For example, when processing images of 1000 scenes per day, assume that it is necessary to process 100 corresponding regions for one scene. If the processing time for each area is 2 minutes, it takes about 3000 hours to process all the areas. If this is processed in, for example, 8 hours, about 400 operators are required.

An object of the present invention is to improve the accuracy of registration processing between images of different types of sensors and to automate the registration processing of a large amount of image data.

[0014]

[Means for Solving the Problems] In order to solve the above problems, the following processing is performed.

(1) Corresponding regions required to select a feature amount optimum for image comparison and detect a positional error between two images in order to enable matching processing of different sensor images having different spectral characteristics. A plurality of (calculation areas) are set, and the corresponding area in which the position error has been calculated correctly is narrowed down by a multi-stage and coarse threshold judgment including rough matching processing and fine matching processing.

(2) After narrowing down the corresponding areas that are candidates, an operator selects a candidate area to specify a corresponding area.

(3) If the corresponding area cannot be obtained accurately, the most similar position error is used as alternative data.

(4) The position error is calculated using the enlarged image. Alternatively, the calculated feature amount is interpolated to obtain the position error with high accuracy.

[0019]

[Action]

(1) The measurement accuracy is improved by providing a plurality of corresponding areas in different sensor images and selecting corresponding areas having similar feature amounts by threshold determination. Further, the processing time is shortened by performing the processing in multiple stages and narrowing down the candidates so that the measurement accuracy becomes higher in the subsequent stages.

(2) Since the number of candidate corresponding areas is reduced by narrowing down the candidates, the operator can input an instruction.

(3) In the case of the satellite sensor, the sensor pointing error depends on time. Therefore, if the image matching process cannot be accurately performed, such as in the cloud area, the sensor pointing error is most likely to occur when the target image to be processed is taken. By using close data as an approximate value, the unprocessable state is prevented.

(4) By enlarging the image or interpolating the feature amount, the position error can be accurately obtained in the real number coordinates.

[0023]

Embodiments of the present invention will now be described in detail with reference to the drawings. FIG. 2 is a schematic diagram of a satellite ground information system to which the present invention is applied, in which a desired area on the ground surface is observed by an observation satellite 10 for the purpose of resource exploration and environmental monitoring, and the observed image data is processed by the ground system 20. .

The observation satellite 10 is equipped with two different types of sensors having different spectral characteristics. As shown in FIG. 3, the sensor 1 (30) is a sensor having sensitivity in the visible and near-infrared wavelength regions, the spatial resolution is 10 m, and the sensor 2
Reference numeral (40) is a sensor having sensitivity in the thermal infrared wavelength region, and the spatial resolution is 40 m. Shoot 60km square at a time. The satellite orbits the earth about 15 times a day,
Observe 0 scenes. Each sensor has a pointing mechanism that changes the observation angle in a direction orthogonal to the satellite traveling direction (orbit direction). There is also a device that controls and measures the pointing angle, but it cannot be controlled and measured accurately, and an error occurs in the set angle even when pointing in the same direction (the dotted line in "Fig. 3" indicates the field of view when a pointing error occurs). Shown). For example, when the pointing error is 125 microradians, and when the ground surface is observed from 800 Km above the ground, a position error of 100 m occurs on the ground surface.

Therefore, when the two sensor images 92 and 94 are superposed on each other as shown in FIG.
0 occurs. This error 50 is the ground system 2 of "Fig. 2".
It should be removed with 0.

The contents of the image registration correction processing in the ground system 20 will be described below with reference to the drawings.
The data on the ground surface observed by the observation satellite 10 is received by the receiving antenna 12 on the ground and input to the ground system 20. The input sensor images are distributed to the user 80 after the system correction 60 specific to the sensor and various processings are performed to obtain a corrected image 70.

The details of the processing will be described below using the image of the sensor 1 as an example (the same applies to the image of the sensor 2).

(1) System correction processing 60 peculiar to the sensor The system correction processing 60 peculiar to the sensor 1 is geometric distortion or radiant intensity distortion correction. This processing is, for example, by Yamagata, remote sensing image processing by an earth observation satellite,
Pages 67-72 (Hitachi critique, Vol. 64, No. 6, 198)
The technique described in June, 2) may be used.

The processing result is a format-converted observation image 90 and a geometric distortion or radiation intensity distortion correction coefficient (hereinafter, only the geometric distortion correction coefficient relevant to the present invention is simply referred to as a distortion correction coefficient) 100. . These are stored in the recording medium.

(2) Registration processing 110 When the corrected sensor 1 image 92 and sensor 2 image 94 are superposed as shown in FIG. 4 by using the distortion correction coefficient 100, it cannot be considered in the system correction. The pointing error of the sensor remains and an overlay error 50 occurs on the image. The registration process 110 corrects this error, and a method of realizing this is the present invention.

In this processing, the corrected distortion correction coefficient 130 for correcting the pointing error is calculated using each observed image 90 and the distortion correction coefficient 100.

(3) Resampling process 140 This is a process of calculating the corrected image 70 from the distortion correction coefficient 100 such as the radiant intensity distortion, the corrected distortion correction coefficient 130, and the observed image 90, and is described in the above-mentioned document (Hitachi Review, Volume 64). )
The details will be omitted because it may be performed by the technique described in (1).

The details of the registration process 110, which is the core of the present invention, will be described below with reference to FIG.

(1) Preprocessing 150 Image for finding registration error (corresponding band)
For f ′ and g ′ (160), distortion correction coefficient φ1,
φ2 (155) is used for processing and corrected images (distortion corrected sensor 1 image and sensor 2 image) f−, g− (17
0) is created.

Here, f- and g- mean that "-" is added just above f and g, respectively.

(2) Image Normalization Processing 180 Distortion-corrected sensor 1 image f- and sensor 2 image g-
Match the image characteristics of. Specifically, the spatial resolution and the average brightness level are matched. The obtained image is f,
g (170).

As for the spatial resolution, one pixel of the image of the sensor 2 (spatial resolution of 40 meters) is resampled by 4 times in the vertical and horizontal directions so that it corresponds to 10 meters of the image of the sensor 1. Also,
The histogram of the brightness level is obtained, and the histograms are matched so that the average and the variance of the histogram match.

Specifically, Takagi and Shimoda supervise, Image Processing Handbook (published by The University of Tokyo Press, 1991 1
Month) This can be realized by the method of pages 463 to 465 and 425 to 429.

(3) Registration error calculation processing 1
90 The registration error calculation processing 190 is configured by "FIG. 1".

(A) Coarse matching process 200 As shown in FIG. 6, the template region f202 and the corresponding search region g (20) are randomly located in the image.
4) is set, and matching is performed with the accuracy of integer pixels. The feature amount to be used is determined in advance in consideration of sensor characteristics and the like.

In the case of comparing sensor images of the same type or similar sensor images, the degree of similarity may be obtained by the area correlation γ that can be represented by "Equation 1" using the brightness level value.

[0042]

[Equation 1] Here, f ~ and g ~ are "~" just above f and g, respectively.
Is added and means the average of f and g.

In the case of registration processing between different types of sensor images, the luminance levels do not always have a positive correlation.
After obtaining the shape feature and the like, the degree of similarity is obtained. As the shape feature amount, DP matching, Fourier descriptor, texture feature, edge feature, or the like may be used. here,
The processing using the edge feature will be described.

In the edge extraction, after blurring the image, the filter processing of "Equation 2" and "Equation 3" in which the Laplacian filter is taken into consideration, and the zero-crossing method (supervised by Takagi and Shimoda,
Edges are extracted as line information from pages 550 to 582 of the image processing handbook (published by The University of Tokyo Press, January 1991). For example, for image g, edge extraction

[0045]

[Equation 2]

[0046]

(Equation 3) Here, σ: standard deviation, @: convolution.

The frequency characteristic of the edge to be extracted can be selected by changing the standard deviation σ as a parameter.

It is also possible to binarize the image g * obtained by "Equation 2" and then obtain the correlation value by "Equation 1".
Here, the correlation value is obtained by "Equation 1" as a grayscale image without binarization.

Further, it is also possible to obtain a correlation value for an image obtained by applying a low-frequency pass filter (for example, a Gaussian filter) to the images obtained by "Equation 2" and "Equation 3" and the zero-crossing method. It is valid.

The coordinate having the largest correlation value thus obtained is obtained from the image, and the position error (registration error) is calculated. That is, the coordinates (sm,
tm) as a matching position, the deviation amount (σx, σy) from the position (s0, t0) of the reference image (for example, the image of the sensor 1) is obtained as a registration error by “Equation 4” and “Equation 5”.

[0051]

[Equation 4]

[0052]

(Equation 5) The above processing is performed for each template area. The calculated correlation value γ (i), the error amount σx (i), σy (i) and the matching area number i are passed to the next process. In this embodiment, i is 10 points.

(B) Candidate area selection processing 210 Using the magnitude of the position error and the direction in which the error (deviation) occurs, the abnormal value is removed by the threshold value judgment. The other areas are candidate areas.

In order to explain the candidate area selection processing flow by the threshold value judgment shown in FIG. 7, the error distribution of FIG. 8 is assumed. The error distribution and the error amount in each candidate area of "FIG. 8" are shown in "FIG. 9" and "FIG. 10."

For example, in the area of i = 1, the correlation coefficient γ
(1) = 0.9, error amount σx (1) = 4 pixels, σy
(1) = 0 pixels.

First, in block 710, areas having a correlation coefficient γ of 0.5 or less for all areas are removed from the candidates (γ judgment). The areas to be removed are i = 3 and 6. For the remaining area, in block 720, the angle r between the position error magnitude r and the x axis is calculated. In the area of i = 1, for example, r = 2 and θ = 0.

Next, at block 730, the error magnitudes and directional averages r- and θ- (representing "-" just above r and θ) of the candidate areas and the deviations δr, δθ and Calculate 230. r = 3.2, δr = 1.2, θ
= 20.25 and δθ = 46.5. Next, in block 740, within θ ± δθ, and in block 750, r ±
Excludes areas that are not within δr. In the former θ judgment, i = 7 and 9, and in the latter r judgment, i = 2 is removed from the candidates. Eventually, at block 760, i = 1, 4,
5, 8, and 10 are registered as candidate areas.

(C) Precision Matching Processing 220 In order to calculate the matching position with high accuracy, the candidate areas (i = 1, 4, 5,
8 and 10) are enlarged and interpolated by a method with little deterioration in image quality.
For example, a cubic convolution kernel is used to enlarge 10 times.

With respect to the enlarged area, the same processing as the rough matching processing is performed to obtain the position error between the template image and the search image.

(D) Error calculation process 230 The same abnormal value determination process as in FIG. 7 is performed to obtain the average value of the errors of the obtained candidate area data. This is the final registration error.

FIG. 10 shows the error amount calculated by the precision matching process (value magnified 10 times). Above (b)
The abnormal value is removed by the threshold determination processing similar to. "Figure 1
At "0", i = 1, 5 is removed. Using the remaining candidate areas (i = 4, 8, 10), the registration error corresponding to the resolution of the original image is calculated in consideration of the 10 times enlargement. That is, 3.1 pixels in the x direction and 0.2 pixels in the y direction are obtained (rounded to the second decimal place).

Hereinafter, the description will be returned to FIG.

(4) Error Retrieval Processing 270 In the registration error calculation processing 190, if the processing target scene is at night, a sea area, a cloud area, or the like, the image matching processing does not work well. In other words, if the registration error cannot be obtained accurately, such as when the deviation is extremely large or when it is significantly larger than the known control / measurement error, the next geometric distortion correction coefficient correction In the process 280, the error amount obtained from the image closest to the image capturing time of the process target image is used as the error file 2
Search from 75 and use.

(3) Geometrical distortion correction coefficient correction processing 28
0 Using the error amount (Δx, Δy) obtained in the registration error calculation process 190, the geometric distortion correction coefficient φ1 (a
0, a1, a2, a3), φ2 (b0, b1, b2, b
3) is modified (φ represents a set of coefficients).

As shown in FIG. 11, the geometric distortion correction coefficients φ ₁ and φ ₂ (214) indicate the geometric relationship between the corrected image 208 and the uncorrected image 206. Further, this coefficient is set for each block 212. As the relational expression, the bilinear expression shown in "Equation 6" and "Equation 7" is generally used.

[0066]

(Equation 6)

[0067]

(Equation 7) From the error amounts (Δx, Δy), the corrected geometric distortion correction coefficients φ1 ′ and φ2 ′ (290) are calculated as in “Equation 8” and “Equation 9”, and these are stored in the recording medium.

[0068]

(Equation 8)

[0069]

[Equation 9] here,

[0070]

[Equation 10]

[0071]

[Equation 11]

[0072]

(Equation 12)

[0073]

(Equation 13)

[0074]

[Equation 14]

[0075]

(Equation 15)

[0076]

[Equation 16]

[0077]

[Equation 17] When the error is a bias error, the error may be bias-considered in the distortion correction coefficient as described above. However, in the case of a higher-order error, the error amount up to the sensor-specific system correction 60 in FIG. It is necessary to deal with it by feeding back and calculating the distortion correction coefficient again.

As an alternative to the present embodiment, a candidate selection process 243 in which the operator 244 selects a final candidate area from the candidate areas as shown in FIG. 12 instead of the precise matching process 220 in FIG. May be provided. If the candidate area can be reduced to about 1/10 by the rough matching process 200, the above process becomes possible. Further, the operator 244 may perform the candidate selection processing 243 only on the images for which the rough matching processing 200 is not successful.

The precision matching processing 220 calculates the correlation value after enlarging the image by a desired magnification. However, after calculating the correlation value by "Equation 1" without enlarging, the correlation value is calculated as shown in FIG. The value may be interpolated to calculate the real number coordinate value 265 that maximizes the correlation value.

The invention described above can be applied to a satellite terrestrial information system, and is particularly effective for automating the registration processing of a large number of images photographed by different types of sensors.

[0081]

The present invention has the following effects.

(1) The number of candidate areas can be reduced by making the matching process coarse and precise, and the load of the error calculation process can be reduced. Further, it becomes possible for the operator to intervene and designate a candidate area.

(2) By adding the candidate area selection process based on the threshold value judgment as a function, it becomes possible to efficiently select corresponding areas having similar error characteristics.

(3) In the precise matching process, the position of the corresponding area region can be accurately obtained by enlarging the image to obtain the correlation value or by interpolating the correlation value.

(4) The matching accuracy can be improved by selecting the feature amount and the similarity by combining the images.

(5) In the error calculation process based on the threshold,
The corresponding area error can be obtained with high accuracy by removing the abnormal value using the average value of the position error of the corresponding area or the deviation from the average value.

(6) The registration error amount is temporarily stored in the error amount file, and in the case of data obtained by photographing clouds, sea, night, etc., the value closest in time is selected from the stored error amounts. Images that cannot be processed can be eliminated by searching and using them in the correction geometric distortion correction coefficient calculation processing.

[Brief description of drawings]

FIG. 1 is a flow chart of calculating a registration error by multi-step threshold value determination.

FIG. 2 is a schematic diagram of a satellite ground information system to which the present invention is applied.

FIG. 3 is a schematic diagram illustrating a relationship between a pointing function of a sensor and an error.

FIG. 4 is an explanatory diagram of a superposition error of two sensor images.

FIG. 5 is a registration processing flow.

FIG. 6 is a schematic diagram of image matching processing between a template image and a search image.

FIG. 7 is a candidate area calculation processing flow based on threshold value determination.

FIG. 8 is an explanatory diagram of a candidate area calculation process based on threshold determination.

FIG. 9 is a diagram showing an error characteristic and a determination processing result by the rough matching processing.

FIG. 10 is a chart showing characteristics of an error due to the precision matching processing.

FIG. 11 is an explanatory diagram of a geometric distortion correction coefficient.

FIG. 12 is a registration error calculation processing flow in consideration of operator candidate selection processing.

FIG. 13 is an explanatory diagram of an alternative method for calculating an error.

[Explanation of symbols]

200 ... Coarse matching process, 210 ... Candidate area calculation process by threshold value, 212 ... Fine matching process, 213 ...
Error calculation processing based on a threshold.

─────────────────────────────────────────────────── ─── Continuation of the front page (51) Int.Cl. ⁶ Identification code Office reference number FI technical display location 9061-5H G06F 15/70 320

Claims (15)

[Claims] 1. (A) The different sensor images are normalized to match the image characteristics, (B) The similarity between the normalized images and the registration error are calculated, and (C) Perform registration correction between the images based on the calculated similarity and registration error,
Registration correction method between different sensor images. 2. (a) The different sensor images are normalized so as to match the respective image characteristics, (b) a plurality of corresponding regions between the normalized different sensor images are set, and (c) Calculate the similarity between the corresponding areas and the registration error by the rough matching process between the set corresponding areas, and (d) select the candidate corresponding area from the corresponding area based on the calculated similarity and the registration error. (e) The similarity and registration error between the candidate corresponding areas are calculated by the precision matching process between the selected candidate corresponding areas, and (f) the candidate corresponding area is calculated based on the calculated similarity and registration error. (G) The registration error between different sensor images is calculated based on the registration error between the narrowed candidate corresponding regions, and (h) the final calculated registration error. Based on Diafiltration error perform registration correction between different sensor image, heterologous sensor inter-image registration correction method. 3. The rough matching process calculates feature quantities of corresponding points between a template image set in an image and a search image, compares the similarities of the feature quantities, and calculates the corresponding points of high similarity. The method for correcting registration between different types of sensor images according to claim 2, wherein the coordinates are obtained at integer positions. 4. The candidate area selection process calculates the direction, magnitude, and deviation of the position error from the mean value of the position error from the corresponding point coordinates obtained by the rough matching process, and the corresponding points having similar error characteristics. 4. The registration correction method between different sensor images according to claim 3, wherein 5. The registration correction method between different types of sensor images according to claim 4, wherein in the fine matching process, real number coordinates that maximize the feature amount of the corresponding points obtained by the rough matching process are obtained by interpolation processing. 6. The heterogeneity according to claim 3, wherein at least one of a luminance level value, a spectral characteristic, an edge characteristic, a texture characteristic, and a Fourier descriptor is used as the feature quantity, and a difference value or a correlation value is used as the similarity degree. Registration correction method between sensor images. 7. The registration error calculation process of (g) above uses an average value of the position errors of corresponding points obtained by the precise matching process or a deviation from the average value,
3. The registration correction method between different types of sensor images according to claim 2, wherein the error amount is calculated using only corresponding points from which abnormal values are removed. 8. The rough matching process and the fine matching process use shape features calculated by applying an edge extraction process after applying a low-frequency pass filter to an image.
Method for correcting registration between different types of sensor images. 9. The registration correction method between different sensor images according to claim 8, wherein the edge extracted by the edge extraction processing is used as a shape feature of binary data or grayscale data. 10. The registration correction method between different sensor images according to claim 9, wherein the edge extraction processing calculates a grayscale data by applying a low frequency pass filter to the data subjected to the edge extraction processing. 11. The precise matching process enlarges a template and a search image to obtain a correlation value in integer coordinates, and converts a coordinate position having the maximum correlation value into a real number coordinate value before enlargement. 5. Registration correction method between different types of sensor images. 12. In the registration error calculation process of the above (g), the calculated registration error amount is saved in an error amount file, and in the case of clouds, sea, or night, the time is calculated from the saved error amount. 3. The method of correcting registration between different types of sensor images according to claim 2, wherein a value closest in terms of image is searched for and used for the correction geometric distortion correction coefficient calculation processing. 13. The resist between different sensor images according to claim 2, wherein the process of (e) is performed by using the corresponding region set in (b) as a candidate corresponding region instead of the processes of (c) and (d). Correction method. 14. A registration correction method between different types of sensor images according to claim 2, wherein the processes (c) to (f) are repeated a plurality of times. 15. The registration correction method between different types of sensor images according to claim 2, wherein a candidate region narrowing process is performed by an operator instead of the processes (e) and (f).