JP2004046777A - Image processor, image processing method used therefor, and its program - Google Patents

Abstract

<P>PROBLEM TO BE SOLVED: To provide an image processor which is hardly affected by noise included in DEM data, and obtains a natural orthorectified image with few strange feeling due to a deficiency area. <P>SOLUTION: A stereo processing means 11 performs a stereo matching processing with respect to a plurality of kinds of image data, so as to extract DEM data b. A DEM smoothing means 12 applies a smoothing filter to the DEM data b obtained by the stereo processing means 11, so as to generate smoothing DEM data c. An orthorectifying means 13 orthogonally converts the image through the use of the smoothing DEM data c which is smoothed by the DEM smoothing means 12, thereby generating an ortho-image d. <P>COPYRIGHT: (C)2004,JPO

Description

[0001]
TECHNICAL FIELD OF THE INVENTION
The present invention relates to an image processing apparatus, an image processing method used therefor, and a program therefor, and more particularly to an image processing method used for generating three-dimensional data from a stereo image and performing orthographic transformation.
[0002]
[Prior art]
2. Description of the Related Art Conventionally, in digital photogrammetry and map making, etc., three-dimensional data [hereinafter referred to as DEM (Digital Elevation Map) data based on a stereo image obtained from a camera installed on the ground or an image taken from a satellite or an aircraft. Is generated by stereo matching, an orthographic transformation of the image is performed using the matching to generate an ortho image, and the method is applied to drawing and the like.
[0003]
Here, in the case of a normal camera, an ortho image refers to an object to be photographed at an angle for central projection as shown in FIG. 11A, and a predetermined surface as shown in FIG. 11B. Is an image obtained by orthogonally projecting a plurality of images, and is indispensable for connecting a plurality of images to grasp a wide range of objects and create a drawing or the like.
[0004]
As an example of applying the above method, there is a technique using the above method when performing digital image measurement on the ground (for example, see Patent Documents 1 and 2). In addition, as an example of a system in which the above technology is applied to aerial photographs and satellite images, there is a technology of performing stereo matching on aerial photographs and satellite images and performing orthographic conversion (for example, see Patent Document 3).
[0005]
FIG. 12 is a block diagram showing a configuration example of a conventional image processing apparatus. As shown in FIG. 12, a conventional image processing apparatus 5 performs a stereo matching process on a plurality of images to extract DEM data, and an orthographic unit 11 performs orthographic conversion of the images using the DEM data. Means 13.
[0006]
In the image processing apparatus 5 having the above-described configuration, an operation as shown in FIG. 13 is performed. That is, image data obtained from a plurality of photographs is provided to the stereo processing means 11, and DEM data is automatically extracted by performing stereo matching as in the conventional system (step S41 in FIG. 13).
[0007]
The obtained DEM data is supplied to the orthorification means 13 together with the image data, and an ortho image is generated by moving the pixels in the image based on the DEM information and the orientation information of the photographed angle (FIG. 13). Step S42).
[0008]
[Patent Document 1]
JP-A-2000-182059 (pages 3 to 7, FIGS. 1 and 2)
[Patent Document 2]
JP-A-11-351865 (pages 4, 5; FIG. 1)
[Patent Document 3]
JP-A-2000-348160 (pages 2, 3; FIG. 1)
[0009]
[Problems to be solved by the invention]
However, in the above-described conventional ortho image generation method, DEM data obtained by stereo matching includes an error caused by mismatching of corresponding points, and pixels having erroneous DEM data during orthographic transformation are used. Is moved to the wrong position, and there is a problem that the image quality of the ortho image generated by the above method is low.
[0010]
In addition, in the conventional ortho image generation method, when an imaging target in an image includes many discontinuous shapes, such as an aerial image of an urban area, pixels are not imaged in a hidden area such as a road hidden by a building or the like. Therefore, it is not possible to move the corresponding pixel in the ortho image, and as shown in FIG. 14, there is a missing area in which the pixel is missing in the concealed area, and the pixel is lost in the ortho image generated by the above method. There is a problem that an image becomes unnatural because it includes many regions.
[0011]
Since the conventional ortho image generation method has the above-described problems, there is a demand for a processing method that can be used for drawing and CG (Computer Graphics) generation and that can obtain a more natural ortho image.
[0012]
Therefore, an object of the present invention is to solve the above-mentioned problems, to be less susceptible to noise included in DEM data, and to obtain a more natural orthoimage with less discomfort caused by a defective area. Another object of the present invention is to provide an image processing method used therefor, a program image processing apparatus therefor, an automatic orthographic image shaping method used therefor, and a program therefor.
[0013]
[Means for Solving the Problems]
An image processing device according to the present invention is an image processing device that performs stereo matching processing on a plurality of images to generate three-dimensional data, and a stereo processing unit that extracts the three-dimensional data by the stereo matching processing. A DEM smoothing unit that applies a smoothing filter to the three-dimensional data extracted by the stereo processing unit to remove noise from the three-dimensional data and smoothes the noise; Orthographic means for performing orthographic transformation of the image using the dimensional data.
[0014]
Another image processing apparatus according to the present invention is an image processing apparatus that performs stereo matching processing on a plurality of images to generate three-dimensional data, and a stereo processing unit that extracts the three-dimensional data by the stereo matching processing Orthographic means for orthographically transforming the image using the three-dimensional data, projective transformation means for projectively transforming the image in accordance with an orientation angle with respect to an orthographic projection plane, and orthorectification means obtained by the orthographic means. An image synthesizing means for synthesizing the image and the projective transformed image obtained by the projective transforming means to generate an interpolated ortho image in which defective pixels are interpolated.
[0015]
An image processing method according to the present invention is an image processing method for generating three-dimensional data by performing a stereo matching process on a plurality of images, and extracting the three-dimensional data by the stereo matching process, Applying a smoothing filter to the obtained three-dimensional data to remove noise from the three-dimensional data and perform smoothing; and performing orthographic transformation on the image using the smoothed three-dimensional data. Have.
[0016]
Another image processing method according to the present invention is an image processing method for performing three-dimensional data by performing a stereo matching process on a plurality of images, and extracting the three-dimensional data by the stereo matching process. Orthographically transforming the image using the three-dimensional data, projectingly transforming the image in accordance with an orientation angle with respect to an orthographic projection plane, and orthographic image obtained by orthographically transforming the image and the projection Generating an interpolated ortho image in which defective pixels have been interpolated by combining the projective transformed image obtained by the transformation.
[0017]
A program for an image processing method according to the present invention is a program for an image processing method for performing stereo matching processing on a plurality of images to generate three-dimensional data. Extracting, applying a smoothing filter to the extracted three-dimensional data to remove noise from the three-dimensional data, and smoothing the three-dimensional data, and transforming the image using the smoothed three-dimensional data. And an orthographic conversion process.
[0018]
A program of another image processing method according to the present invention is a program of an image processing method for performing stereo matching processing on a plurality of images to generate three-dimensional data. A process of extracting data, a process of orthographically transforming the image using the three-dimensional data, a process of projectively transforming the image according to an orientation angle with respect to an orthographic projection plane, and a process of orthogonally transforming the image. And a process of generating an interpolated ortho image in which defective pixels are interpolated by combining the obtained ortho image and the projection conversion image obtained by the projection conversion.
[0019]
That is, the image processing apparatus of the present invention performs stereo matching processing on a plurality of images to extract three-dimensional data [hereinafter, referred to as DEM (Digital Elevation Map) data], and smoothes the DEM data. DEM smoothing means for applying noise filters to remove noise, orthographic means for orthogonally transforming an image using DEM data, and projective transformation means for projecting an image according to an orientation angle with respect to an orthographic plane. Image synthesizing means for synthesizing the projective transformed image and the ortho image to generate an interpolated ortho image.
[0020]
In the image processing apparatus of the present invention, by adopting the above configuration, stereo matching processing is automatically performed on a plurality of images, and height information at each point in the image based on the obtained disparity information, that is, DEM After extracting the data and removing noise in the DEM data by applying a smoothing filter, the image is orthorectified using the DEM data obtained by smoothing the image to generate an orthorectified image. And a projection-transformed image obtained by projecting and transforming a plane parallel to the image to generate an interpolated ortho image in which missing pixels are interpolated.
[0021]
Thus, in the image processing apparatus of the present invention, the noise in the DEM data is reduced and used for the ortho-rectification, and the defective area of the pixel due to the concealment is interpolated with the projective transformed image, thereby deteriorating the image quality due to the noise and the defective area Thus, it is possible to generate a more natural orthoimage with less discomfort due to.
[0022]
BEST MODE FOR CARRYING OUT THE INVENTION
Next, embodiments of the present invention will be described with reference to the drawings. FIG. 1 is a block diagram showing the configuration of the image processing apparatus according to the first embodiment of the present invention. In FIG. 1, an image processing apparatus 1 performs stereo matching processing on a plurality of images (aviation image data a) to extract three-dimensional data [hereinafter, referred to as DEM (Digital Elevation Map) data] b. A DEM smoothing means 12 for applying a smoothing filter (not shown) to the extracted DEM data b to remove noise, and direct injection of an image (aerial image data a) using the smoothed DEM data c It is composed of an orthographic unit 13 for conversion and a recording medium 14 for storing a program for realizing each unit of the image processing apparatus 1.
[0023]
Each unit of the image processing apparatus 1 operates as follows. That is, the following processing is performed by the computer mainly configuring the image processing apparatus 1 executing the program of the recording medium 14.
[0024]
First, the stereo processing means 11 performs a stereo matching process on a plurality of image data (aerial image data a) to extract DEM data b. The DEM smoothing means 12 applies a smoothing filter to the DEM data b obtained by the stereo processing means 11 to generate smoothed DEM data c.
[0025]
The orthographic unit 13 performs an orthographic transformation of an image (aerial image data a) using the smoothed DEM data c smoothed by the DEM smoothing unit 12 to generate an orthoimage d. Here, as described above, in the case of a normal camera, an orthoimage refers to an object to be photographed at an inclination for central projection as shown in FIG. 11A, as shown in FIG. Is an image obtained by orthogonally projecting a predetermined surface, and is indispensable for connecting a plurality of images to grasp a wide range of objects and create a drawing or the like.
[0026]
FIG. 2 is a flowchart showing the operation of the image processing apparatus 1 according to the first embodiment of the present invention. The operation of the image processing apparatus 1 according to the first embodiment of the present invention will be described with reference to FIGS. Note that the processing illustrated in FIG. 2 is realized by a computer mainly configuring the image processing apparatus 1 executing a program on the recording medium 14.
[0027]
First, the stereo processing means 11 automatically performs stereo matching processing on a plurality of images (aerial image data a) to extract DEM data (step S1 in FIG. 2). The stereo matching processing used here is not particularly limited, and can be realized by, for example, a method disclosed in Japanese Patent Application Laid-Open No. 3-167678.
[0028]
Next, the DEM smoothing means 12 applies a smoothing filter to the DEM data b obtained by the stereo processing means 11 to generate smoothed DEM data c (step S2 in FIG. 2). The smoothing filter used here is not particularly limited as long as it has an effect of reducing noise. For example, a median filter, a multi-valued morphological filter, or the like may be used. It is also possible to apply a combination of a plurality of these filters.
[0029]
The orthographic means 13 uses the smoothed DEM data c smoothed by the DEM smoothing means 12 to orthorectify an image (aerial image data a) with respect to a certain reference plane to create an orthoimage d. (Step S3 in FIG. 2). The orthographic projection plane at this time is, for example, a plane parallel to the ground surface when using a satellite image or an aerial photograph. However, an orthographic transformation can be performed by defining an arbitrary plane depending on the application.
[0030]
As described above, in the present embodiment, by using a configuration in which the smoothed DEM data is used for the ortho-rectification processing, it is possible to generate an ortho image d in which the influence of noise included in the original DEM data b is reduced. .
[0031]
FIG. 3 is a block diagram showing the configuration of the image processing device according to the second embodiment of the present invention. In FIG. 3, the second embodiment of the present invention is the same as the first embodiment of the present invention shown in FIG. 1 except that the DEM smoothing means 12 is omitted and the projection conversion means 21 and the image synthesizing means 22 are added. The same components are denoted by the same reference numerals. The operation of the same components is the same as in the first embodiment of the present invention.
[0032]
That is, the image processing apparatus 2 performs stereo matching processing on a plurality of images (aviation image data a) to extract DEM data b, and uses the extracted DEM data b to generate an image (aviation image data). An orthographic unit 13 for orthogonally transforming the image data a), a projection transforming unit 21 for projecting an image (aerial image data a) according to a rotation angle with respect to an orthographic projection plane, and a projective transformed image (projected transformed aerial image e) ) And the ortho image d, an image synthesizing means 22 for generating an interpolated ortho image f in which the defective pixels are interpolated, and a recording medium 14 for storing a program for realizing each means of the image processing apparatus 2. It is composed of
[0033]
Each unit of the image processing apparatus 2 operates roughly as follows. That is, the following processing is performed by the computer mainly configuring the image processing apparatus 2 executing the program on the recording medium 14.
[0034]
As described above, the stereo processing unit 11 performs stereo matching processing on a plurality of image data (aerial image data a) to extract DEM data b. The orthorification means 13 uses the DEM data a obtained by the stereo processing means 11 to perform orthographic conversion of an image (aerial image data a).
[0035]
The projection conversion unit 21 performs projection conversion of the image (aerial image data a) according to the rotation angle to the orthographic projection plane obtained by the orientation calculation. The image synthesizing unit 22 superimposes and interpolates the ortho-image d obtained by the above-mentioned ortho-formation unit 13 and the image (projection-transformed aerial image e) converted by the above-mentioned projection conversion unit 21, and eliminates the missing pixels. An interpolated ortho image f is generated.
[0036]
FIG. 4 is a flowchart showing the operation of the image processing apparatus 2 according to the second embodiment of the present invention. The operation of the image processing apparatus 2 according to the second embodiment of the present invention will be described with reference to FIGS. Note that the processing illustrated in FIG. 4 is realized by a computer mainly configuring the image processing apparatus 2 executing a program on the recording medium 14.
[0037]
First, the stereo processing means 11 automatically performs stereo matching processing on a plurality of images (aerial image data a) and extracts DEM data b (step S11 in FIG. 4). The stereo matching processing used here is not particularly limited, and can be realized by, for example, a method disclosed in Japanese Patent Application Laid-Open No. 3-167678.
[0038]
Next, using the DEM data b obtained by the stereo processing means 11, the orthorification means 13 performs an orthographic transformation of an image (aerial image data a) on a certain reference plane to create an ortho image d. (Step S12 in FIG. 4). The orthographic projection plane at this time is, for example, a plane parallel to the ground surface when using a satellite image or an aerial photograph. However, an orthographic transformation can be performed by defining an arbitrary plane depending on the application.
[0039]
The projecting transformation unit 21 performs an image transformation (projection transformed aviation) on any one of the plurality of images (aerial image data a) used in the stereo processing unit 11 according to the rotation angle to the orthographic projection plane. An image e) is generated (step S13 in FIG. 4). This is an operation of approximating an image with a plane and re-projecting the image on an orthographic projection plane while keeping the viewpoint fixed. The image used at this time may be any image among the images (aerial image data a) used by the stereo processing means 11, but an image having a small rotation angle with respect to the orthographic projection plane is preferable.
[0040]
Finally, the image synthesizing unit 22 superimposes and interpolates the orthographic image d obtained by the orthographic unit 13 and the projection transformed aerial image e obtained by the projective transforming unit 21 to generate an interpolated ortho image f (FIG. 4 steps S14). Specifically, a process of interpolating the pixels in the missing area where no value is obtained in the orthographic image d using the pixels of the projection-transformed aerial image e at the corresponding positions.
[0041]
As described above, in this embodiment, the image (aerial image data a) is projected and transformed according to the rotation angle with respect to the orthographic projection plane, and the pixels in the corresponding area of the image (projected transformed aerial image e) are transformed into the ortho image d. , The missing area of the ortho image d can be interpolated. As a result, when applying the ortho image d to CG or the like, it is possible to reduce the discomfort caused by the missing area. Can be.
[0042]
FIG. 5 is a block diagram showing the configuration of the image processing device according to the third embodiment of the present invention. In FIG. 5, the third embodiment of the present invention has the same configuration as that of the second embodiment of the present invention shown in FIG. 3 except that a DEM smoothing means 12 is added. The reference numerals are attached. The operation of the same component is the same as that of the second embodiment of the present invention.
[0043]
That is, the image processing device 3 performs a stereo matching process on a plurality of images (aerial image data a), extracts the DEM data b, and applies a smoothing filter to the DEM data b to reduce noise. DEM smoothing means 12 for removing, orthographic means 13 for orthogonally transforming the image (aerial image data a) using the smoothed DEM data c, and orientation angle of the image (aerial image data a) with respect to the orthographic projection plane A projecting transformation unit 21 for projective transforming according to the above, an image combining unit 22 for combining the projectively transformed aerial image e and the ortho image d to generate an interpolated ortho image f in which defective pixels are interpolated, And a recording medium 14 for storing a program for realizing each means.
[0044]
Each unit of the image processing apparatus 3 operates roughly as follows. That is, the following processing is performed by the computer mainly configuring the image processing apparatus 3 executing the program on the recording medium 14.
[0045]
The stereo processing unit 11 performs a stereo matching process on a plurality of image data (aerial image data a) to extract DEM data b. The DEM smoothing means 12 applies a smoothing filter to the DEM data b obtained by the stereo processing means 11 to generate smoothed DEM data c.
[0046]
An orthographic unit 13 performs orthographic transformation of an image (aerial image data a) using the smoothed DEM data c smoothed by the DEM smoothing unit 12 to create an ortho image d.
[0047]
The projective transformation means 21 performs projective transformation of the image (aerial image data a) in accordance with the rotation angle to the specified orthographic projection plane. The image synthesizing unit 22 superimposes and interpolates the ortho-image d obtained by the above-mentioned ortho-formation unit 13 and the image (projection-transformed aerial image e) converted by the above-mentioned projection conversion unit 21, and eliminates the missing pixels. An interpolated ortho image f is generated.
[0048]
FIG. 6 is a flowchart showing the operation of the image processing apparatus 3 according to the third embodiment of the present invention. The operation of the image processing apparatus 3 according to the third embodiment of the present invention will be described with reference to FIGS. Note that the processing illustrated in FIG. 6 is realized by a computer mainly configuring the image processing apparatus 3 executing a program of the recording medium 14.
[0049]
First, the stereo processing means 11 automatically performs a stereo matching process on a plurality of images (aerial image data a) and extracts DEM data b (step S21 in FIG. 6). The stereo matching process used here is not particularly limited, and can be realized by, for example, a method disclosed in Japanese Patent Application Laid-Open No. 3-167678.
[0050]
Next, the DEM smoothing means 12 creates a smoothed DEM data c by applying a smoothing filter to the DEM data b obtained by the stereo processing means 11 (step S22 in FIG. 6). The smoothing filter used here is not particularly limited as long as it has a noise reducing effect. For example, a median filter, a multi-valued morphological filter, or the like may be used. It is also possible to apply a combination of a plurality of these filters.
[0051]
Using the smoothed DEM data c smoothed by the above-mentioned DEM smoothing means 12, the orthographic means 13 orthogonally transforms an image (aerial image data a) to a certain reference plane to create an orthoimage d. (Step S23 in FIG. 6). The orthographic projection plane at this time is, for example, a plane parallel to the ground surface when using a satellite image or an aerial photograph. However, an orthographic transformation can be performed by defining an arbitrary plane depending on the application.
[0052]
Subsequently, the projection conversion unit 21 performs image projection conversion on any one of the plurality of images (aerial image data a) used by the stereo processing unit 11 according to the rotation angle to the orthographic projection plane. (Projected transformed aerial image e) is generated (step S24 in FIG. 6). This is an operation of approximating an image with a plane and re-projecting the image on an orthographic projection plane while keeping the viewpoint fixed. The image used at this time may be any of the images (aerial image data a) used by the stereo processing means 11, but is preferably an image having a small rotation angle with respect to the orthographic projection plane.
[0053]
Lastly, the image synthesizing means 22 superimposes and interpolates the orthoimage d obtained by the orthographic means 13 and the projection-transformed aerial image e obtained by the projective transformation means 21. The generated interpolated ortho image f is generated (step S25 in FIG. 6). Specifically, a process of interpolating the pixels in the missing area where no value is obtained in the orthographic image d using the pixels of the projection-transformed aerial image e at the corresponding positions.
[0054]
As described above, in the present embodiment, since the smoothed DEM data c is used for the ortho-rectification processing, an ortho image d in which the influence of noise included in the original DEM data b is reduced is generated. Can be.
[0055]
Further, in this embodiment, the image (aerial image data a) is projectively transformed in accordance with the rotation angle with respect to the orthographic projection plane, and the pixels of the corresponding region of the image (projected transformed aerial image e) are replaced with the defective region of the ortho image d. , The missing area of the ortho image d can be interpolated. As a result, when applying the ortho image d to CG or the like, the sense of discomfort caused by the missing area can be reduced.
[0056]
FIG. 7 is a block diagram illustrating a configuration of an image processing apparatus according to a fourth embodiment of the present invention. In FIG. 7, the fourth embodiment of the present invention has the same configuration as the third embodiment of the present invention shown in FIG. 5 except that a stereo pair setting means 42 is added. The reference numerals are attached. The operation of the same component is the same as that of the third embodiment of the present invention.
[0057]
That is, the image processing apparatus 4 includes a stereo pair setting unit 42 that selects an image to be paired from a plurality of images (aerial image data a), an ortho image generating unit 41 that generates an ortho image d, and an image (aerial image data). A projection transformation means 21 for projectively transforming the data a) in accordance with an orientation angle with respect to an orthographic projection plane, a projection transformation aerial image e and an ortho image d are synthesized to generate an interpolated ortho image f in which missing pixels are interpolated. It comprises an image synthesizing unit 22 and a recording medium 14 for storing a program for realizing each unit of the image processing apparatus 2.
[0058]
Here, the ortho image generating means 41 performs a stereo matching process on a plurality of images (aerial image data a) to extract DEM data b, and a smoothing filter for the extracted DEM data b. And an ortho-rectification unit 13 for orthogonally transforming the image (aerial image data a) using the smoothed DEM data c.
[0059]
Each unit of the image processing apparatus 4 operates roughly as follows. That is, the following processing is performed by the computer mainly configuring the image processing apparatus 4 executing the program of the recording medium 14.
[0060]
The stereo pair setting means 42 determines a plurality of stereo image pairs obtained by selecting two of the three or more image data groups from three or more image data obtained by photographing the same point, and sets the stereo image pairs as processing units. The stereo processing unit 11 performs a stereo matching process on each stereo image pair provided by the stereo pair setting unit 42 to extract DEM data b.
[0061]
The DEM smoothing means 12 applies a smoothing filter to the DEM data b obtained by the stereo processing means 11 to smooth the data. The orthorification means 13 uses the smoothed DEM data c smoothed by the DEM smoothing means 12 to perform orthographic transformation on an image (aerial image data a).
[0062]
The projective transformation means 21 performs projective transformation of the image (aerial image data a) in accordance with the rotation angle to the specified orthographic projection plane. The image synthesizing unit 22 superimposes and interpolates each of the ortho images d obtained by the above-described ortho-converting unit 13 and the image (projection-transformed aerial image e) converted by the above-mentioned projection conversion unit 21 to obtain the missing pixels. Generates an interpolated ortho image f.
[0063]
FIG. 8 is a flowchart showing the operation of the image processing apparatus 4 according to the fourth embodiment of the present invention. The operation of the image processing apparatus 4 according to the fourth embodiment of the present invention will be described with reference to FIGS. Note that the processing illustrated in FIG. 8 is realized by a computer mainly configuring the image processing apparatus 4 executing a program of the recording medium 14.
[0064]
First, the stereo pair setting means 42 determines a plurality of stereo image pairs obtained by selecting two of the three or more image data groups from three or more image data obtained by photographing the same point, and sets them as a processing unit (step S31 in FIG. 8). For example, when four images are obtained, one set of two of these images is set, and another stereo image pair is set of the remaining two images. There are a total of six combinations that can be selected, but it is not always necessary to set all of them as stereo image pairs, and a plurality of stereo image pairs may be determined so that all images are used to reduce the concealment area.
[0065]
Next, the stereo processing unit 11 performs a stereo matching process on each stereo image pair set by the stereo pair setting unit 42, and extracts DEM data b (step S32 in FIG. 8). The stereo matching process used here is not particularly limited, and can be realized by, for example, a method disclosed in Japanese Patent Application Laid-Open No. 3-167678.
[0066]
Subsequently, the DEM smoothing unit 12 applies a smoothing filter to each DEM data b obtained by the stereo processing unit 11 to perform a smoothing process (step S33 in FIG. 8). The smoothing filter used here is not particularly limited as long as it has a noise reducing effect. For example, a median filter, a multi-valued morphological filter, or the like may be used. It is also possible to apply a combination of a plurality of these filters.
[0067]
The orthorification means 13 uses the respective DEM data (smoothed DEM data c) smoothed by the DEM smoothing means 12 to perform orthographic conversion of an image (aerial image data a) with respect to a certain reference plane, An ortho image d is created (step S34 in FIG. 8). The orthographic projection plane at this time is, for example, a plane parallel to the ground surface when using a satellite image or an aerial photograph. However, an orthographic transformation can be performed by defining an arbitrary plane depending on the application.
[0068]
The projecting transformation means 21 is an image (projected transformed aerial image) obtained by projecting any one of the plurality of images (aerial image data a) used in the stereo processing means 11 according to the rotation angle to the orthographic projection plane. e) is generated (step S35 in FIG. 8). This is an operation of approximating an image (aerial image data a) by a plane and re-projecting the image on an orthographic projection plane while keeping the viewpoint fixed. The image used at that time may be any image among the images used by the stereo processing means 11, but an image having a small rotation angle with respect to the orthographic projection plane is more preferable.
[0069]
Finally, the image synthesizing unit 22 superimposes and interpolates each of the ortho images d obtained by the above-mentioned ortho-converting unit 13 and the projection-transformed aerial image e obtained by the above-mentioned projection conversion unit 21, and eliminates the missing pixels. An interpolated ortho image f is generated (step S36 in FIG. 8). Specifically, pixels in a missing area where no value is obtained in the ortho image d are interpolated using corresponding pixels in other ortho images. With respect to the pixels in the missing area remaining in this process, a process of interpolating using the corresponding pixels in the projected transformed image is performed.
[0070]
In the above-described fourth embodiment of the present invention, the processing for a plurality of stereo pairs is repeatedly performed by the stereo processing means 11, the DEM smoothing means 12, and the ortho-formation means 13. It is also possible to provide a plurality of 41 and perform processing for a plurality of stereo pairs in parallel.
[0071]
As described above, in this embodiment, since the smoothed DEM data c is used for the ortho-rectification processing, it is possible to generate an ortho image in which the influence of noise included in the original DEM data is reduced.
[0072]
In the present embodiment, ortho-images are generated from a plurality of stereo pairs having different line-of-sight directions by mutually complementing each other, and in addition, the image is projected and transformed in accordance with the rotation angle with respect to the orthographic projection plane, and the corresponding region of the image is transformed. Since the pixels are substituted for the defective area of the ortho image, the missing area of the ortho image can be interpolated. As a result, when applying the ortho image to CG or the like, the discomfort caused by the missing area is reduced. It becomes possible.
[0073]
Note that the present invention is not limited to each of the first to fourth embodiments of the present invention described above, and can be implemented under various conditions. For example, the target image is not limited to an aerial photograph or a satellite image, and can be applied to an image obtained by performing stereo shooting on the ground or indoors. Images may be taken using two cameras, or taken twice using a monocular camera, and image data may be taken directly from a digital camera or scanned from an analog photograph. It can be implemented even if there is.
[0074]
Subsequently, the operation of the embodiment of the present invention will be described using a specific example. In this description, a case will be described in which an ortho image parallel to the ground surface is generated from a stereo aerial photograph according to the above-described third embodiment of the present invention.
[0075]
Stereo aerial photographs are treated by regarding overlapping portions of images continuously taken from an aircraft as stereo images, and are first converted into digital data using a scanner. Subsequently, an orientation operation is performed to determine camera parameters at the time of photographing. Usually, even in aerial photographs taken in a direction close to vertical, it is difficult to grasp the camera parameters at the time of shooting due to the influence of aircraft attitude change and so on. A coordinate value in the image is read, and parameters such as a camera rotation angle between the left and right images are located using the coordinate pair of the corresponding point. The orientation method used here is not particularly limited, and for example, a value calculated by plotting software may be used.
[0076]
After the camera parameters have been located, the left and right images are re-projected on a common parallel plane as necessary, as shown in FIG. This is an operation called “parallelization”, and the same object in the left and right images after the parallelization is performed on the same scanning line.
[0077]
In addition, the stereo processing means 11 automatically performs stereo matching processing on the left and right images that have been parallelized, and generates a dense parallax image by obtaining correspondence between the left and right images. The stereo matching process used here is not particularly limited, and can be realized by, for example, a method disclosed in Japanese Patent Application Laid-Open No. 3-167678.
[0078]
When the parallax image is obtained, the position in the three-dimensional coordinate system corresponding to each pixel is calculated in combination with the oriented camera parameters, and DEM data is extracted. The obtained DEM data contains a lot of spike-like isolated points having discontinuous three-dimensional coordinates as noise as a result of the influence of pixels that are incorrectly associated during the stereo matching processing.
[0079]
In order to eliminate the influence of these isolated points, the DEM smoothing means 12 performs a smoothing process on the DEM data obtained by the stereo processing means 11 by applying a smoothing filter. The smoothing filter used here is not particularly limited as long as it has a noise reducing effect. For example, a median filter that replaces the median of the pixel value in a small area around the target pixel as a representative value is used. For example, the above-described isolated noise can be efficiently removed while maintaining the edge depending on the actual shape.
[0080]
In addition, a multi-valued morphological filter or the like configured by combining a maximum value filter and a minimum value filter for a small area can be expected to have the same effect as described above. It is also possible to apply a combination of a plurality of such filters.
[0081]
Subsequently, using the smoothed DEM data smoothed by the DEM smoothing means 12 and the noise-removed DEM data and the camera parameters specified, the orthographic means 13 corrects the original aerial image with respect to the ground surface. Projective transformation is performed to create an orthoimage.
[0082]
Specifically, first, the three-dimensional coordinates of all the points in the DEM data are projected onto the ground surface, and the plane coordinates in the ground coordinate system (for example, a plane rectangular coordinate system) and the coordinates in the original image are used. Ask for a response. Next, according to the obtained correspondence, the pixels of the original aerial image are mapped while interpolating into the image corresponding to the ground surface.
[0083]
Further, the projective transformation means 21 performs an operation of reprojecting the aerial image data onto a plane parallel to the ground surface by projective transformation. As described above, since the optical axis of the camera at the time of aerial photography is not completely vertical to the ground surface, but is taken obliquely on the ground surface, it is difficult to superimpose it on an ortho image. For this reason, an operation of applying a projective transformation to an aerial image and re-projecting onto a projection plane parallel to the ground without changing the camera center position is applied. By this operation, an image corresponding to a complete vertical photograph can be generated.
[0084]
A specific method of the projective transformation is, for example, a method disclosed in Japanese Patent Application Laid-Open No. 8-567 using a rotation angle of a camera coordinate system with respect to the ground surface and a focal length among camera parameters at the time of shooting. It can be carried out.
[0085]
Finally, the image synthesizing unit 22 performs a process of synthesizing the ortho image generated by the ortho unit 13 and the projection transformed image generated by the project transformation unit 21 to generate an interpolated ortho image. Specifically, after superimposing the orthographic image and the projective transformed image, with respect to the pixels in the missing area in the orthographic image for which no value is obtained due to the effect of concealment, the pixels of the projective transformed image at the corresponding positions are quoted. , An interpolated ortho image is generated by performing an interpolation process.
[0086]
At this time, the image from which the pixels are quoted is an image in which the principal point is close to the target position, that is, the one that faces the target point as much as possible among the left and right projective transformed images. By doing so, it is possible to reduce the number of cases in which the wall surface of the building, which falls down due to the plane approximation, is used as the concealed area, and improve the image quality.
[0087]
In addition, by blurring the boundary portion or the quoted portion at the time of quote, the influence of the generated edge can be suppressed, and an interpolated ortho image with less discomfort can be generated.
[0088]
Next, a specific example in which an ortho image parallel to the ground surface is generated from four stereo aerial photographs according to the example of the fourth embodiment will be described.
[0089]
For example, as shown in FIG. 10, four stereo aerial photographs are taken with a large overlap between two adjacent photographing courses # 1 and # 2, and image pairs photographing the same point are respectively taken. Can be obtained by selecting one set at a time from the courses. However, the photographing method is not limited to this, and four images # 1 to # 4 obtained by individually photographing may be used. The obtained aerial photograph group is first converted into digital data using a scanner.
[0090]
Next, the stereo pair setting means 42 sets a stereo pair from the four stereo aerial photographs converted into digital data. By referring to metadata or the like added to the images, two images in the same shooting course are respectively combined to obtain two stereo pairs. However, the pair setting method is not limited to this, and if all four images # 1 to # 4 are used, image pairs may be set between different courses. Further, up to six stereo pairs may be generated by using each of the images # 1 to # 4 a plurality of times.
[0091]
For each set stereo pair, similarly to the third embodiment of the present invention, orientation work for determining camera parameters at the time of shooting, parallelization, stereo matching processing, extraction of DEM data, and smoothing of DEM data are performed. Is
[0092]
Subsequently, using the smoothed DEM data from which noise has been removed by the DEM smoothing means 12 and the camera parameters which have been standardized, the orthorification means 13 generates an original aerial image for each aerial photograph. An orthographic transformation is performed on the ground surface to create an orthorectified image.
[0093]
Specifically, first, when three-dimensional coordinates at all points in the DEM data are projected onto the ground surface, plane coordinates in a ground coordinate system (for example, a plane rectangular coordinate system) and coordinates in the original image are used. Ask for a response. Next, according to the obtained correspondence, the pixels of the original aerial image are mapped while interpolating into the image corresponding to the ground surface.
[0094]
Further, the projective transformation means 21 performs an operation of projecting the aerial image data of each stereo pair onto a plane parallel to the ground surface by projective transformation. As described above, since the optical axis of the camera at the time of aerial photography is not completely vertical to the ground surface, but is taken obliquely on the ground surface, it is difficult to superimpose it on an ortho image. For this reason, a projective transformation is applied to the aerial image, and an operation of reprojecting onto a projection plane parallel to the ground without changing the camera center position is applied. By this operation, an image corresponding to a complete vertical photograph can be generated.
[0095]
A specific method of the projective transformation is, for example, a method described in Japanese Patent Application Laid-Open No. 8-567 using a rotation angle and a focal length of a camera coordinate system with respect to the ground surface among camera parameters at the time of shooting. Can do this.
[0096]
Finally, the image synthesizing unit 22 synthesizes each ortho image generated by the above-mentioned ortho conversion unit 13 and each projection-converted image generated by the projection conversion unit 21 to generate an interpolation ortho-image. Do.
[0097]
Specifically, after superposing each ortho image and each projective transformed image, first, a plurality of ortho images are synthesized by mutually interpolating pixels. That is, the value is used for a point where a pixel is obtained only for a certain ortho image, and when a value is obtained for a plurality of ortho images, the principal point is located at a position closer to the target position among the target images. The image is synthesized by using pixel values in an image to be captured, that is, an image of the pixel which is photographed while facing the pixel as much as possible.
[0098]
Further, with respect to the pixels in the missing area in the ortho image where the value is not obtained due to the influence of the concealment, the interpolation ortho image is generated by referring to the pixels of the projection conversion image at the corresponding position and performing the interpolation process. At this time, an image from which a pixel is quoted is an image in which a principal point is located close to a target position, that is, an image facing as close as possible to a target point, among a plurality of projective transformed images. As a result, it is possible to reduce the number of cases in which the wall surface of the building that is reflected and falls down for the plane approximation is used as the concealed area, and to improve the image quality. In addition, by blurring the boundary portion or the quoted portion at the time of quote, the influence of the generated edge can be suppressed, and an interpolated ortho image with less discomfort can be generated.
[0099]
As described above, according to the present invention, noise included in the original DEM data can be reduced by using the smoothed DEM data for the ortho-rectification process, and the image quality of the ortho image is reduced due to the noise. Therefore, a natural ortho image with less noise can be generated.
[0100]
Also, the present invention interpolates a missing area of an ortho image by projective transforming an image according to a rotation angle with respect to an orthographic projection plane, and substituting pixels of a corresponding area of the image in a defective area of the ortho image. Therefore, it is possible to generate an orthoimage with less discomfort due to the missing area.
[0101]
【The invention's effect】
As described above, according to the image processing apparatus of the present invention, in an image processing apparatus that performs stereo matching processing on a plurality of images to generate three-dimensional data, three-dimensional data is extracted by stereo matching processing, and the extracted three-dimensional data is extracted. The three-dimensional data is included in the three-dimensional data by applying a smoothing filter to the three-dimensional data to remove noise from the three-dimensional data and performing smoothing, and using the smoothed three-dimensional data to perform orthogonal transformation on an image. The effect is obtained that a more natural ortho image can be obtained, which is less susceptible to noise and has less discomfort due to the defective area.
[0102]
According to another image processing apparatus of the present invention, in an image processing apparatus that performs stereo matching processing on a plurality of images to generate three-dimensional data, the three-dimensional data is extracted by stereo matching processing. The image is orthographically transformed using, the image is projected and transformed according to the orientation angle with respect to the orthographic projection plane, and the orthographic image obtained by the orthogonal transformation of the image and the projective transformed image obtained by the projective transformation are combined. By generating an interpolated ortho image in which defective pixels have been interpolated, it is possible to obtain a more natural ortho image that is less affected by noise included in the three-dimensional data and has less discomfort due to the defective area. The effect is obtained.
[Brief description of the drawings]
FIG. 1 is a block diagram illustrating a configuration of an image processing apparatus according to a first embodiment of the present invention.
FIG. 2 is a flowchart illustrating an operation of the image processing apparatus according to the first embodiment of the present invention.
FIG. 3 is a block diagram illustrating a configuration of an image processing apparatus according to a second embodiment of the present invention.
FIG. 4 is a flowchart illustrating an operation of the image processing apparatus according to the second embodiment of the present invention.
FIG. 5 is a block diagram illustrating a configuration of an image processing apparatus according to a third embodiment of the present invention.
FIG. 6 is a flowchart illustrating an operation of the image processing apparatus according to the third embodiment of the present invention.
FIG. 7 is a block diagram illustrating a configuration of an image processing apparatus according to a fourth embodiment of the present invention.
FIG. 8 is a flowchart illustrating an operation of the image processing apparatus according to the fourth embodiment of the present invention.
FIG. 9 is a schematic diagram showing a stereo shooting geometry of the present invention.
FIG. 10 is a schematic diagram showing an example of aerial photography from four directions according to the embodiment of the present invention.
FIGS. 11A and 11B are schematic diagrams showing a difference between a center projected image and an orthographic image.
FIG. 12 is a block diagram illustrating a configuration of a conventional image processing apparatus.
FIG. 13 is a flowchart illustrating an operation of the image processing apparatus according to the related art.
FIG. 14 is a schematic diagram illustrating occurrence of a missing area in an ortho image.
[Explanation of symbols]
1,2,3,4 image processing device
11 Stereo processing means
12 DEM smoothing means
13 Ortho-formation means
14 Recording medium
21 Projection conversion means
22 Image synthesis means
41 Ortho image generating means
42 Stereo pair setting means

Claims (13)

An image processing apparatus that performs stereo matching processing on a plurality of images to generate three-dimensional data, wherein the stereo processing means extracts the three-dimensional data by the stereo matching processing, and the three-dimensional data extracted by the stereo processing means A DEM smoothing means for applying a smoothing filter to the three-dimensional data to remove noise from the three-dimensional data and smooth the image; and using the three-dimensional data smoothed by the DEM smoothing means to transform the image using the three-dimensional data. An image processing apparatus comprising: an orthographic unit for performing orthographic conversion. Projection conversion means for projecting the image in accordance with the orientation angle with respect to the orthographic projection plane, and defective pixels obtained by synthesizing the orthographic image obtained by the orthographic means and the projection conversion image obtained by the projection conversion means 2. An image processing apparatus according to claim 1, further comprising: image synthesizing means for generating an interpolated ortho image. A stereo pair setting unit that selects an image to be paired from the plurality of images, wherein the stereo processing unit, the ortho-rectification unit, and the projection conversion unit are configured for the image pair selected by the stereo pair setting unit. The image processing apparatus according to claim 2, wherein each processing is performed by performing the respective processing. An image processing apparatus that performs stereo matching processing on a plurality of images to generate three-dimensional data, the stereo processing means extracting the three-dimensional data by the stereo matching processing, and Orthographic means for orthographically transforming an image, projective transforming means for projectively transforming the image in accordance with an orientation angle with respect to an orthographic projection plane, and an orthographic image obtained by the orthographic means and obtained by the projective transforming means. An image processing apparatus, comprising: an image synthesizing unit configured to generate an interpolated ortho image in which a defective pixel is interpolated by synthesizing a projection-transformed image. The image synthesizing unit interpolates pixels for which values are not obtained in a plurality of ortho images with each other, and has an imaging main axis closest to the point in an image having values for pixels whose values are obtained in duplicate. By selecting and synthesizing an ortho-image, for a pixel for which a value is not obtained in any of a plurality of generated ortho-images, the pixel is interpolated by using a corresponding pixel from a projective transformation image having an imaging main axis closest to the point. The image processing apparatus according to claim 4, wherein an interpolated ortho image is synthesized. An image processing method for performing stereo matching processing on a plurality of images in an image processing apparatus to generate three-dimensional data, wherein the image processing apparatus extracts the three-dimensional data by the stereo matching processing. Applying a smoothing filter to the extracted three-dimensional data to remove noise from the three-dimensional data and perform smoothing; and performing orthographic transformation on the image using the smoothed three-dimensional data. An image processing method comprising: The image processing device, projecting the image according to an orientation angle with respect to an orthographic projection plane, and synthesizing an orthoimage obtained by the orthographic transformation of the image and a projective transformed image obtained by the projective transformation. Generating an interpolated ortho image in which the defective pixels have been interpolated. The image processing apparatus includes a step of selecting an image to be paired from the plurality of images, and extracting the three-dimensional data, orthographic transformation of the image, and projective transformation for the selected image pair. 8. The image processing method according to claim 7, wherein An image processing method for performing stereo matching processing on a plurality of images in an image processing apparatus to generate three-dimensional data, wherein the image processing apparatus extracts the three-dimensional data by the stereo matching processing. Orthographically transforming the image using the three-dimensional data, projecting the image according to an orientation angle with respect to an orthographic plane, and orthorectifying the image and orthographically transforming the image. Generating an interpolated ortho image in which defective pixels are interpolated by synthesizing a projective transformed image obtained by the transformation. The step of generating the interpolated ortho image interpolates the pixels for which the values are not obtained in the plurality of ortho images with each other, and is the pixel closest to the point in the image having the value for the pixel whose value is obtained repeatedly. Select and combine ortho-images with the main shooting axis and interpolate for pixels for which no value is obtained in any of the multiple generated ortho-images using the corresponding pixels from the projected transformed image with the main shooting axis closest to the point 10. The image processing method according to claim 9, wherein the interpolated ortho-image is synthesized by performing the processing. A program of an image processing method for performing stereo matching processing on a plurality of images to generate three-dimensional data, comprising: a computer for extracting the three-dimensional data by the stereo matching processing; A process for applying a smoothing filter to the three-dimensional data to remove noise from the three-dimensional data and perform smoothing, and a process for performing orthographic transformation on the image using the smoothed three-dimensional data. Program. A program of an image processing method for generating a three-dimensional data by performing a stereo matching process on a plurality of images, comprising: a computer for extracting the three-dimensional data by the stereo matching process; A process of orthogonally transforming the image using the process, a process of projective transforming the image in accordance with an orientation angle with respect to an orthographic plane, and an orthographic image obtained by orthographic transformation of the image and the projective transform. A process for generating an interpolated ortho image in which defective pixels are interpolated by synthesizing the projection-transformed image. In the computer, when performing the process of generating the interpolated ortho image, an image having a value for a pixel for which a value is obtained by overlapping with a pixel for which a value is not obtained in a plurality of ortho images with each other. The orthographic image having the imaging main axis closest to the point is selected and synthesized, and the projection conversion image having the imaging main axis closest to the point is obtained for pixels for which a value is not obtained in any of the plurality of generated ortho images. 13. The program according to claim 12, wherein the interpolated ortho image is synthesized by interpolating using a corresponding pixel from.

Images