JP2006033294A - Image processing apparatus - Google Patents

Abstract

<P>PROBLEM TO BE SOLVED: To generate a stereoscopic vision image which made vertical parallax alleviate more. <P>SOLUTION: An inverse projection unit 31 determines ground coordinates corresponding to a point on a screen specified by a measuring mark, this ground coordinates are inversely projected on directly under the visual image of a reference image, and directly viewed image coordinates are obtained. An image matching unit 33 extracts a forward visual image coordinates corresponding to the directly under the visual image coordinates from a correlation coordinates information database 70 to determine the coordinates on the forward visual image corresponding to the directly under-viewed image coordinates. A stereo measuring unit 35 obtains the ground coordinates by using the front graphic intersection based on the position/attitude data of the directly under the visual image coordinates, the forward visual image coordinates and the flight locus data treated by the image matching unit 33. <P>COPYRIGHT: (C)2006,JPO&NCIPI

Description

  The present invention relates to an image processing apparatus that generates a stereoscopic image.

Non-Patent Document 1 and Non-Patent Document 2 below disclose a TLS system equipped with a three-line scanner (TLS), which is an optical sensor for aerial surveying. The three-line scanner is composed of three line sensor cameras arranged in parallel, and images in three directions of front view, direct view, and rear view are simultaneously acquired by each line sensor camera. Such a three-line scanner is mounted perpendicular to the traveling direction of a flying object such as a helicopter. Thereby, when the flying object is flying, the ground surface is scanned by each line sensor camera, and a triple stereo image of the ground object can be acquired.
Koichi Tsuno, “Three Line Scanner (TLS) and its Applications, Photogrammetry and Remote Sensing”, Photographic Society of Japan, VOL.41, NO.4, 2002, PP.37-44 Andreas Eckardt, et al, "Performance of the imaging system in the LH Systems ADS40 airborne digital sensor", International Archives of Photogrammetry and Remote Sensing, Vol.XXXIII, Part B1, Amsterdam 2000

  In the TLS system equipped with the above-described three-line scanner, flight trajectory data including the position and orientation data of the line sensor camera is acquired by GPS (Global Positioning System) and IMU (Inertial Measurement Unit). However, an error occurs in the flight trajectory data due to various factors. Therefore, when a stereoscopic image is generated based on each of the forward-view image, the direct-view image, and the rear-view image captured from different directions, a vertical parallax that is a vertical parallax occurs. Factors that cause errors in flight trajectory data include, for example, temperature drift in GPS and IMU, differences in line sensor data acquisition interval and GPS data acquisition interval, ground reference point measurement errors, There are those due to modeling errors in flight trajectory data, variations in flight altitude, and changes in topography with respect to the flight trajectory.

  Here, with reference to FIG. 12 and FIG. 13, a mechanism in which the vertical parallax Y is generated when a certain ground reference point P is photographed will be described. PT shown in FIG. 12 is a line indicating an actual flight trajectory, and UT is a flight trajectory line represented by flight trajectory data. Na shown in FIGS. 12 and 13 is a direct view image, and Ba is a back view image. NaP is a corresponding point of the point P on the direct-view image Na determined based on the flight trajectory data, and BaP is a corresponding point of the point P on the rear-view image Ba determined based on the flight trajectory data. CaP is a corresponding point of the point P determined on the direct-view image Na and the forward-view image Fa when there is no error in the flight trajectory data. That is, when the corresponding point of the point P is determined based on the flight trajectory data having an error, the corresponding point NaP on the direct-view image Na and the corresponding point BaP on the rear-view image Ba as shown in FIG. A vertical parallax Y occurs between them.

  Therefore, an object of the present invention is to provide an image processing apparatus capable of generating a stereoscopic image with reduced vertical parallax in order to solve the above-described problems.

  An image processing apparatus of the present invention is an image processing apparatus that generates a stereoscopic image based on a plurality of two-dimensional images composed of a one-dimensional image group captured by a plurality of line sensors having different shooting directions. The apparatus includes a vertical parallax correction unit that corrects vertical parallax, which is vertical parallax generated between two-dimensional images, and generates a stereoscopic image based on the image after the vertical parallax is corrected by the vertical parallax correction unit. And

  According to the present invention, the vertical parallax correction unit corrects the vertical parallax generated between the plurality of two-dimensional images. Therefore, the stereoscopic image is generated based on the plurality of two-dimensional images in which the vertical parallax is generated due to various factors. Even in this case, it is possible to generate a stereoscopic image in which the vertical parallax is further reduced.

  The image processing apparatus of the present invention further includes display means for displaying a stereoscopic image on the screen, and index display means for displaying an index on the stereoscopic image displayed by the display means. It is preferable to correct the vertical parallax each time an arbitrary position on the stereoscopic image is specified by the index. In this way, every time an arbitrary position on the stereoscopic image is specified by the index, the stereoscopic image after the vertical parallax correction is displayed with reference to the position specified by the index can be displayed. It becomes possible.

  In the image processing apparatus of the present invention, the vertical parallax correction unit is configured to determine a first coordinate that determines any one of a plurality of two-dimensional images as a reference image and coordinates on the reference image as first coordinates. And second coordinate determining means for determining a second coordinate corresponding to the first coordinate determined by the first coordinate determining means from a two-dimensional image other than the reference image among the plurality of two-dimensional images. It is preferable to include a third coordinate determining unit that determines coordinates on the stereoscopic image based on the first coordinates and the second coordinates.

  In this way, the coordinate on the stereoscopic image can be determined based on the first coordinate on the reference image and the second coordinate on the other image corresponding to the first coordinate. it can. In other words, by determining the second coordinate corresponding to the first coordinate, the vertical parallax generated between the first coordinate and the second coordinate is reduced, and as a result, the solid whose vertical parallax is further reduced. A visual image can be generated.

  The image processing apparatus of the present invention further includes storage means for storing correlation coordinate information between a plurality of two-dimensional images, and the second coordinate determination means uses the first coordinate from the correlation coordinate information stored in the storage means. It is preferable to determine the second coordinates by extracting the second coordinates corresponding to. In this way, since the second coordinate corresponding to the first coordinate can be extracted based on the correlation coordinate information stored in advance, an arbitrary position on the stereoscopic image is specified by the index. It is possible to reduce the processing time from when the stereoscopic image is generated to when the stereoscopic image is generated.

  In the image processing apparatus of the present invention, the vertical parallax correction unit includes a low-resolution image generation unit that generates a low-resolution image corresponding to each two-dimensional image by reducing the resolution of the plurality of two-dimensional images, and each low-resolution image. A corresponding region extracting unit that extracts a corresponding region corresponding to the predetermined region from a two-dimensional image corresponding to each low-resolution image based on a predetermined region corresponding to each other, and a high-level corresponding to each low-resolution image A correlation coordinate determination unit that determines correlation coordinates between corresponding regions of a two-dimensional image that is a resolution image, and a stereoscopic image coordinate determination unit that determines coordinates on the stereoscopic image based on the correlation coordinates are configured. It is preferable.

  In this way, a corresponding region on the two-dimensional image corresponding to each predetermined region is extracted based on a predetermined region corresponding to each low resolution image with reduced resolution and reduced influence of vertical parallax. Therefore, the vertical parallax between the two-dimensional image regions after extraction can be reduced compared to the case where the corresponding regions between the two-dimensional images are directly extracted from the two-dimensional image that is a high-resolution image. Can do. In addition, since the correlation coordinates between the corresponding areas are determined based on the corresponding areas after the reduction of the vertical parallax, the coordinates on the stereoscopic image are determined based on the correlation coordinates. A stereoscopic image with reduced vertical parallax can be generated. Furthermore, since the vertical parallax can be corrected by determining the correlation coordinates between the corresponding areas of each two-dimensional image, the processing time is longer than when determining the correlation coordinates between all areas of each two-dimensional image. Can be greatly reduced.

  In the image processing apparatus of the present invention, it is preferable that the plurality of two-dimensional images are geometrically corrected images that are geometrically corrected based on position data and posture data at the time of photographing by the line sensor. In this way, a stereoscopic image is generated based on a geometrically corrected image obtained by geometrically correcting a plurality of two-dimensional images composed of a one-dimensional image group captured by the line sensor based on position data and posture data. Can be made.

  In the image processing apparatus of the present invention, any one of the plurality of two-dimensional images is preferably a geometrically corrected image that is geometrically corrected based on the vertical parallax between the correlation coordinates between the two-dimensional images. . In this way, it is possible to generate a stereoscopic image without using data including many errors that cause vertical parallax (for example, position data and orientation data at the time of shooting by the line sensor).

  The image processing apparatus according to the present invention also includes a left-eye image and a right-eye image based on a plurality of two-dimensional images composed of a one-dimensional image group captured while moving a plurality of line sensors having different shooting directions in the air. An image processing apparatus for displaying a stereoscopic image comprising: an index display means for displaying an index on a stereoscopic image; and a display means for displaying a stereoscopic image on the screen. When the index is set on the stereoscopic image, the stereoscopic image based on the plurality of two-dimensional images after correcting the vertical parallax that is the vertical parallax generated between the plurality of two-dimensional images is displayed on the screen. The stereoscopic image is displayed in a three-dimensional manner.

  According to the present invention, even when displaying a stereoscopic image based on a plurality of two-dimensional images in which vertical parallax has occurred due to various factors, a plurality of two-dimensional images after the vertical parallax has been corrected by the display means A stereoscopic image based on the image is stereoscopically displayed on the screen.

  The image processing apparatus according to the present invention can generate a stereoscopic image with reduced vertical parallax.

  Hereinafter, embodiments of an image processing apparatus according to the present invention will be described with reference to the drawings. In addition, in each figure, the same code | symbol is attached | subjected to the same element and the overlapping description is abbreviate | omitted.

[First embodiment]
First, a first embodiment of the present invention will be described. The image processing apparatus according to the present embodiment generates a stereoscopic image based on a plurality of two-dimensional images composed of a one-dimensional image group captured by a plurality of line sensors having different shooting directions, and displays the generated stereoscopic image on a screen. It is a device that displays on top. Here, in the present embodiment, a line sensor (camera) constituting a three-line scanner (TLS) is used as a plurality of line sensors having different shooting directions. The line sensor is a plurality of pixels arranged in a one-dimensional manner, and each line sensor is arranged at a predetermined interval with a direction perpendicular to the traveling direction of a flying object such as a helicopter as a longitudinal direction. .

  FIG. 1 shows the position of each line sensor at the moment of photographing a certain point Q on the building. As shown in FIG. 1, the first line sensor A1 captures a point Q on the building in a forward downward direction (arrow db in FIG. 1) with respect to the traveling direction of the flying object (arrow da in FIG. 1). . The second line sensor A2 captures the point Q on the building in the direction directly below the flying object (arrow dc in FIG. 1). The third line sensor A3 captures the point Q on the building in a downward downward direction (arrow dd in FIG. 1) closer to the traveling direction of the flying object. That is, a forward view image is generated based on the one-dimensional image group captured by the first line sensor A1, and a nadir view image is generated based on the one-dimensional image group captured by the second line sensor A2. A rear view image is generated based on the one-dimensional image group captured by the third line sensor A3.

  In the present embodiment, a front-view image, a direct-view image, and a rear-view image are used as a two-dimensional image composed of a one-dimensional image group. Further, two types of two-dimensional images are selected from these forward-view images, direct-view images, and rear-view images, and a stereoscopic image is generated based on the selected two types of two-dimensional images. Furthermore, the two-dimensional image may be ground surface data (DSM: Digital Surface Model) including buildings and vegetation covering the land, or ground surface data (DTM: Digital Terrain Model) not including buildings and vegetation. May be.

  FIG. 2 is a diagram illustrating a functional configuration of the image processing apparatus 1 in the present embodiment. As shown in FIG. 2, the image processing apparatus 1 includes a geometric correction unit 10, a stereoscopic unit 20, a vertical parallax correction unit 30 (vertical parallax correction unit), a display unit 40 (display unit), and an index display unit. 50 (index display means). The image processing apparatus 1 stores various types of information used for processing in the above-described units in a database. Examples of such databases include a flight trajectory database 60 and a correlation coordinate information database 70 (storage means).

  Here, the flight trajectory database 60 includes, as data items, for example, image coordinates (x, y), the position (X, Y, Z) of the line sensor at the time of shooting, and the attitude angle (ω, φ) of the line sensor at the time of shooting. , Κ) and photographing time (t). The x component of the image coordinates (x, y) is a line number of the line sensor, and the y component is a pixel number.

  The correlation coordinate information database 70 stores correlation coordinate information related to correlation coordinates between two-dimensional images. More specifically, the correlated coordinate information is information in which image coordinates having a correlation are associated with each other between the forward-viewed image, the nadir-viewed image, and the backward-viewed image. In the correlation coordinate information database 70, correlation coordinate information obtained by a matching process between the forward-view image, the nadir-view image, and the rear-view image executed by the image matching unit 33 described later is registered in advance.

  The geometric correction unit 10 includes two types of two-dimensional images composed of a one-dimensional image group captured by the line sensor, the position (X, Y, Z) of the line sensor at the time of shooting, and the attitude angle of the line sensor at the time of shooting ( Based on (ω, φ, κ), a geometrically corrected image of each two-dimensional image is generated.

  FIG. 3 is a diagram schematically illustrating a process in which a geometric correction image is generated by the geometric correction unit 10 based on a two-dimensional image and flight trajectory data including position / posture data. L1 is a flight trajectory line based on the flight trajectory data, and L2 is an ideal flight trajectory line on which the flying object actually flew. G1 is a two-dimensional image composed of a one-dimensional image group, H1 is an average ground height plane that is a horizontal plane provided at the average ground height, and G2 is a projection of the two-dimensional image G1 onto the average ground height plane H1. It is a later projected image. H2 is a geometrically corrected image plane, and G3 is a geometrically corrected image after the projection image G2 is projected onto the geometrically corrected image plane H2. L3 is a quasi-epipolar line on the geometric correction image G3.

First, the geometric correction unit 10 uses the position / posture data corresponding to the one-dimensional image to which each pixel belongs for each pixel on the two-dimensional image G1 shown in FIG. Project to the plane H1. That is, the projection image G2 to the average ground height plane H1 is generated based on the two-dimensional image G1.

Here, “Z ₀ ” in the above equation is the average ground height, (X _N , Y _N , Z _N ) is the position coordinates of the line sensor at the time of capturing a one-dimensional image, and (r ₁₁ , r ₁₂ , .., R ₃₃ ) is the attitude angle of the line sensor at the time of taking a one-dimensional image. (X _P , Y _P ) is the horizontal coordinate of the pixel on the projection image G2 projected on the average ground height plane H1, and (x _P , y _P ) is the horizontal coordinate of the pixel on the two-dimensional image G1. Is the coordinate, and f is the focal length of the line sensor.

Next, the geometric correction unit 10 obtains an ideal flight locus line L2 shown in FIG. 3 as follows. (1) Of the attitude angles r ′ corresponding to the flight trajectory line L2, (ω ′ _N , φ ′ _N , κ ′ _N ), “ω ′ _N ” and “φ ′ _N ” are set to 0, In κ ′ _N ”, an average value of“ κ _N ”, which is one element of the attitude angle r corresponding to the flight trajectory line L1, is set. (2) The position data (X ′ _N , Y ′ _N , Z ′ _N ) of the flight locus line L2 is a straight line that approximates the horizontal position (X _N , Y _N ) of the flight locus line L1, and the flight locus line L1 Is set to be a straight line represented by the average flying height (Z _N ). (3) The internal orientation parameters such as the focal length f of the line sensor are kept unchanged.

Next, the geometric correction unit 10 obtains the position data (X ′ _N , Y ′ _N , Z ′ _N ) and the attitude angle (r) of the flight trajectory line L2 obtained according to the above conditions (1) to (3). _{'11, r' 12, ···} , with r _'33), to produce a geometric correction image G3 having a substantially epipolar relationship with other geometric correction image. That is, each pixel on the projection image G2 projected on the average ground height plane H1 shown in FIG. 3 is projected on the geometric correction image plane H2 based on the following equation by using the position / posture data of the flight trajectory line L2. To do.

  Each pixel on the geometric correction image G3 projected based on the above formula is resampled in the geometric correction image G3 and irregularly distributed, and has the same pixel value as the corresponding pixel in the two-dimensional image G1. Have.

  The stereoscopic unit 20 generates a stereoscopic image based on the two types of geometric correction images generated by the geometric correction unit 10.

  The vertical parallax correction unit 30 performs vertical parallax correction processing for correcting vertical parallax generated between the two-dimensional images. That is, the vertical parallax generated between each of the forward-view image, the direct-view image, and the rear-view image is corrected. More specifically, when an arbitrary point on the stereoscopic image displayed on the screen is selected and determined (set) by the female mark (index), each point is determined based on the point specified by the female mark. Corrects vertical parallax that occurs between two-dimensional images. Note that what is specified by the female mark is not limited to a point, but may be a line or a predetermined region. In other words, a position serving as a reference for the vertical parallax correction process may be specified. The vertical parallax correction unit 30 generates a stereoscopic image based on the geometric correction images corresponding to the two types of two-dimensional images after the vertical parallax is corrected. That is, a stereoscopic image that is viewed stereoscopically is generated by viewing the two types of geometrically corrected images after correcting the vertical parallax with the left and right eyes.

  The vertical parallax correction unit 30 includes a back projection unit 31 (first coordinate determination unit), an image matching unit 33 (second coordinate determination unit), a stereo measurement unit 35 (third coordinate determination unit), and a process. And an end determination unit 37.

  The back projection unit 31 determines the ground coordinates corresponding to the point on the screen specified by the knife mark. The back projection unit 31 back-projects (back projections) the determined ground coordinates onto a two-dimensional image used as a reference image at the time of matching performed in the image matching unit 33 described later, among the two-dimensional images.

  The image matching unit 33 determines a coordinate (second coordinate) on another two-dimensional image corresponding to the coordinate (first coordinate) back-projected on the reference image by the back projection unit 31. That is, the image matching unit 33 extracts coordinates on another two-dimensional image corresponding to the backprojected coordinates from the correlation coordinate information database 70.

  Here, a method for obtaining correlation coordinates between two types of two-dimensional images using an area matching (area correlation) method, which is an example of an image matching method, will be described. In the area matching method, pixels in each two-dimensional image are not compared with each other to determine the corresponding coordinates, but the corresponding coordinates are determined by comparison between surfaces (windows) including the periphery of the pixel of interest. More specifically, for each coordinate in the surface, a difference is taken in both two-dimensional images, and the coordinate having the smallest difference is defined as the corresponding coordinate. In this way, corresponding coordinates (correlation coordinate information) between two types of two-dimensional images are determined at random.

  The image matching unit 33 resamples the random correlation coordinate information determined using the area matching method in units of lattices shown in FIG. Here, N shown in FIG. 4 is a direct view image, and F is a forward view image. FIG. 4 is a diagram showing in which position on the forward-view image F each lattice point constituting the square lattice is located with reference to a square lattice on the nadir image N. By re-sampling the random correlation coordinate information determined by the area matching method with reference to the lattice shown in FIG. 4, the corresponding coordinate FP of the coordinate NP in the square lattice on the nadir image N is forward-viewed. It can be obtained instantaneously by bilinear interpolation or the like in the deformation grid on the image F.

  By performing such area matching, vertical parallax hardly occurs between the coordinates on the geometric correction image corresponding to the first coordinates and the coordinates on the geometric correction image corresponding to the second coordinates. That is, both geometric correction images have a quasi-epipolar relationship.

  In the correlation coordinate information database 70 according to the present embodiment, for example, correlation coordinate information between two-dimensional images determined using the area matching described above is registered in advance. Therefore, when the position on the screen is specified by the female mark, the image matching unit 33 corresponds to the first coordinate from the correlation coordinate information database 70 without performing matching processing that requires a long time for processing each time. A second coordinate can be obtained. That is, the processing time can be greatly reduced.

  Based on the correlation coordinate information determined by the image matching unit 33 and the position / orientation data of the flight trajectory data, the stereo measurement unit 35 obtains ground coordinates by, for example, triangulation including the forward intersection method.

  The process end determination unit 37 determines whether or not the vertical parallax correction process has been performed on all the features other than the feature corresponding to the point specified by the female mark. When it is determined that the vertical parallax correction process has been performed on all the features, the process end determination unit 37 ends the vertical parallax correction process.

  The display unit 40 displays a stereoscopic image on the screen. That is, the stereoscopic image generated by the stereoscopic unit 20 and the stereoscopic image after the vertical parallax correction unit 30 has corrected the vertical parallax are displayed on the screen. More specifically, for example, the display unit 40 causes two types of images (left-eye image and right-eye image constituting a stereoscopic image) to be alternately switched at high speed and displayed on one screen. In this case, the user can view the image displayed on the screen in a three-dimensional manner by viewing the screen with the glasses for stereoscopic viewing. Here, a liquid crystal shutter is provided on each of the left and right lens portions of the stereoscopic glasses. By alternately opening and closing the liquid crystal shutter in synchronization with the switching timing of the two types of images, the image displayed on the screen can be displayed in three dimensions.

  The index display unit 50 displays a female mark on the stereoscopic image displayed by the display unit 40.

  Next, the operation of the image processing apparatus 1 in this embodiment will be described with reference to FIGS. First, an overview of image processing executed in the image processing apparatus 1 according to the present embodiment will be described with reference to FIG. Here, before the image processing is executed, the following processing is performed as preprocessing. First, two types of two-dimensional images (for example, a nadir view image and a forward view image) to be stereoscopically viewed are selected according to a user operation instruction. Next, one of the selected direct-view image and forward-view image is set as the reference image. After these pre-processing is performed, the image processing shown in FIG. 5 is started.

  First, the geometric correction unit 10 generates a geometric correction image corresponding to each image based on the nadir view image, the forward view image, and the flight trajectory data (step S1).

  Next, the stereoscopic unit 20 generates a stereoscopic image based on the two types of geometrically corrected images generated in step S1. Thereby, the display unit 40 displays the stereoscopic image generated by the stereoscopic unit 20 on the screen (step S2).

  Next, the vertical parallax correction unit 30 determines whether or not an arbitrary point on the stereoscopic image is specified by the female mark (step S3). If the result of this determination is NO (step S3; NO), the process proceeds to step S3. That is, in the determination in step S3, the determination in step S3 is repeated until the determination result is YES.

  On the other hand, when it is determined in step S3 that an arbitrary point on the stereoscopic image is specified by the female mark (step S3; YES), the vertical parallax correction unit 30 performs the direct view image, the forward view image, and the forward view image. A vertical parallax correction process for correcting the vertical parallax generated during the period is performed (step S4). Details of the vertical parallax correction processing will be described later. After the vertical parallax correction process is performed, the display unit 40 displays the stereoscopic image after the vertical parallax correction is performed on the screen (step S5). In actual image processing, after the stereoscopic image after the vertical parallax is corrected in step S5 is displayed on the screen, the processing in step S3 and subsequent steps is repeated again. Then, when an end instruction is input according to a user operation instruction, the above-described image processing ends.

  Next, the vertical parallax correction process performed in step S4 (see FIG. 5) described above with reference to FIGS. 6 and 7 will be described. Here, N shown in FIG. 7 is a direct view image, and F is a forward view image. Ng is a geometrically corrected image of the nadir image N, and Fg is a geometrically corrected image of the forward image F. M is a female mark, P1 indicates the ground coordinates, P2 indicates the coordinates on which the ground coordinates P1 are back-projected on the nadir image N, and P3 is the coordinates on the forward image F corresponding to the nadir image coordinates P2. Indicates. P4 indicates the coordinates on the geometric correction image Ng corresponding to the nadir view image coordinate P2, and P5 indicates the coordinates on the geometric correction image Fg corresponding to the forward view image coordinate P3.

  First, the backprojection unit 31 determines the ground coordinates P1 corresponding to the point on the screen specified by the female mark M (step S11), and backprojects the ground coordinates P1 onto the direct view image N that is the reference image. (Step S12). Thereby, the coordinate P2 on the nadir image N is obtained.

  Next, the image matching unit 33 determines, from the correlation coordinate information database 70, the forward view image coordinate P3 corresponding to the direct view image coordinate P2 to determine the coordinate on the forward view image F corresponding to the direct view image coordinate P2. Extract (step S13). That is, the forward view image coordinates P3 determined as the coordinates corresponding to the direct view image coordinates P2 by area matching are extracted from the correlation coordinate information database 70. As shown in FIG. 7, there is a vertical parallax between the coordinate P4 on the geometrically corrected image Ng corresponding to the nadir image coordinate P2 and the coordinate P5 on the geometrically corrected image Fg corresponding to the forward image coordinate P3. Hardly occurs.

  Next, the stereo measurement unit 35 uses the forward intersection method based on the direct view image coordinates P2 and the forward view image coordinates P3 associated with the image matching unit 33 and the position / posture data of the flight trajectory data. The coordinates are obtained (step S14).

  Next, the processing end determination unit 37 determines whether or not the vertical parallax correction processing described above has been performed on all the features other than the feature corresponding to the point specified by the female mark (step S15). If this determination is NO (step S15; NO), the process proceeds to step S11. That is, the above-described processing from step S11 to step S14 is repeated until the vertical parallax correction processing is performed for all the features.

  On the other hand, if it is determined in step S15 that the vertical parallax correction process described above has been performed for all features other than the feature corresponding to the point specified by the female mark (step S15; YES), The parallax correction process ends.

  In the above-described image processing apparatus 1 according to the first embodiment, the stereo measurement unit 35 has the coordinates back-projected on the reference image by the back-projection unit 31 and the other two-dimensional image determined by the image matching unit 33. Since the ground coordinates are obtained based on the coordinates, it is possible to generate a stereoscopic image with reduced vertical parallax.

  Further, in the image processing apparatus 1 according to the first embodiment described above, based on the correlation coordinate information stored in the correlation coordinate information database 70, the other corresponding to the coordinates back-projected on the reference image by the back projection unit 31. Since the coordinates on the two-dimensional image can be extracted, it is possible to shorten the processing time from when an arbitrary position on the stereoscopic image is specified by the female mark until the stereoscopic image is generated.

  In the first embodiment described above, when image matching is performed, corresponding points are determined using the correlation coordinate information database 70, but the correlation coordinate information database 70 is not necessarily used. That is, each time a point on the screen is specified by a female mark, image matching is performed based on the coordinates back-projected on the reference two-dimensional image, and corresponding coordinates existing on other two-dimensional images are obtained. Also good.

[Second Embodiment]
Next, a second embodiment of the present invention will be described. First, the functional configuration of the image processing apparatus 2 in the second embodiment will be described with reference to FIG. The functional configuration of the image processing device 2 in the second embodiment shown in FIG. 8 is different from the functional configuration of the image processing device 1 in the first embodiment, as one of the functional configurations of the vertical parallax correction unit 30S in the second embodiment. Is different from a part of the functional configuration of the vertical parallax correction unit 30 in the first embodiment, and the image processing apparatus 2 in the second embodiment does not include the correlation coordinate information database 70. Therefore, since the other functional configuration is the same as the functional configuration of the image processing apparatus 1 in the first embodiment, the same reference numerals are given to the respective components and the description thereof will be omitted, and the first embodiment will be described below. Differences from the form will be described in detail.

  As shown in FIG. 8, the vertical parallax correction unit 30S of the image processing apparatus 2 in the second embodiment includes a low resolution image generation unit 38 (low resolution image generation unit) and a corresponding region extraction unit 39 (corresponding region extraction unit). And an image matching unit 33S (correlation coordinate determination means) and a stereo measurement unit 35S (stereoscopic image coordinate determination means).

  The low-resolution image generation unit 38 generates a low-resolution image corresponding to each two-dimensional image by reducing the resolution of the plurality of two-dimensional images that are targets of the stereoscopic image. That is, the low-resolution image generation unit 38 reduces the resolution of the forward-view image, the direct-view image, and the backward-view image, for example, the low-resolution image of the forward-view image, the low-resolution image of the direct-view image, and the backward-view image. A low resolution image.

  Thus, by generating a low-resolution image, it is possible to reduce the vertical parallax that was noticeably generated between the high-resolution images between the low-resolution images.

  The corresponding area extracting unit 39 corresponds to each low resolution image by using the corresponding area corresponding to each predetermined area based on the predetermined area corresponding to each low resolution image generated by the low resolution image generating unit 38. Respectively extracted from the two-dimensional images. As the predetermined area corresponding to the low resolution images, for example, the peripheral area of the point on each low resolution image corresponding to the point on the screen specified by the female mark corresponds. Therefore, for example, when a point on the screen is specified by a female mark, the corresponding region extraction unit 39 extracts a region corresponding to the female mark from the low-resolution image of the nadir view image, and directly below the extracted region. An area on the visual image is extracted from the direct-view image as a corresponding area. In the same manner, the corresponding region is extracted from the front view image or the rear view image. As a method for obtaining a point on the low-resolution image corresponding to the point on the screen specified by the female mark, for example, the ground coordinates corresponding to the point on the screen specified by the female mark are determined, and this determined ground There is a method of back projecting coordinates onto a low resolution image.

  In this way, based on a predetermined area (each area corresponding to a point specified by a female mark) corresponding to each low-resolution image with reduced resolution and reduced influence of vertical parallax, it can be handled from the two-dimensional image By extracting the regions, the vertical parallax between the regions of each two-dimensional image after extraction is greater than when the corresponding region between each two-dimensional image is extracted directly from the two-dimensional image that is a high-resolution image. It becomes possible to reduce.

  The image matching unit 33S determines correlation coordinates that are correlated between the corresponding regions extracted by the corresponding region extraction unit 39. That is, the image matching unit 33 obtains correlation coordinates between two types of corresponding regions using, for example, an area matching (area correlation) method. Since this area matching method is the same as the area matching method described in the first embodiment, description thereof is omitted.

  As described above, since the comparison target when determining the correlation coordinates can be limited only to the corresponding region of the two-dimensional image, the processing time can be reduced as compared with the case where the entire region of the two-dimensional image is the comparison target. It can be greatly reduced.

  The stereo measurement unit 35S determines the coordinates on the stereoscopic image based on the correlation coordinates determined by the image matching unit 33S. That is, the stereo measurement unit 35S obtains the ground coordinates by triangulation including, for example, the forward intersection method based on the correlation coordinate information determined by the image matching unit 33S and the position / posture data of the flight trajectory data.

  As described above, since the coordinates on the stereoscopic image are determined based on the correlation coordinates between the corresponding regions after the reduction of the vertical parallax, it is possible to generate a stereoscopic image with further reduced vertical parallax. It becomes possible.

  Next, the vertical parallax correction process (step S4 in FIG. 5) performed in the image processing apparatus 2 in the second embodiment will be described with reference to FIG. That is, the vertical parallax correction process for correcting the vertical parallax generated between the direct-view image set as the reference image and the forward-view image will be described. The outline of the image processing executed in the image processing apparatus 2 in the second embodiment is the same as the outline of the image processing executed in the image processing apparatus 1 in the first embodiment (see FIG. 5). Description is omitted.

  First, the low-resolution image generation unit 38 generates a low-resolution image of the direct-view image and a low-resolution image of the forward-view image by reducing the resolution of the direct-view image and the forward-view image (step S21).

  Next, the corresponding area extraction unit 39 determines the ground coordinates corresponding to the point on the screen specified by the female mark, and uses the determined ground coordinates on the low resolution image of the direct view image and the low resolution of the forward view image. Backproject onto the image. Then, the corresponding region extraction unit 39 extracts a region near the back-projected point from the low-resolution image of the direct-view image and the low-resolution image of the forward-view image, and the direct-view image corresponding to each of the extracted regions The upper region and the region on the forward view image are extracted from the direct view image and the forward view image as corresponding regions (step S22).

  Next, the image matching unit 33S calculates the correlation coordinates between the corresponding region of the high resolution image corresponding to the nadir view image extracted by the corresponding region extracting unit 39 and the corresponding region of the high resolution image corresponding to the forward view image. Then, it is determined using an area matching (area correlation) method (step S23).

  Next, the stereo measurement unit 35S obtains the ground coordinates using the forward intersection method based on the correlation coordinates determined by the image matching unit 33S and the position / posture data of the flight trajectory data (step S24). Thereby, the vertical parallax correction process is completed.

  As described above, in the image processing apparatuses 1 and 2 in the above-described embodiments, the vertical parallax generated between two types of two-dimensional images can be corrected by the vertical parallax correction units 30 and 30S. Even when a stereoscopic image is generated based on a plurality of two-dimensional images in which vertical parallax has occurred, it is possible to generate a stereoscopic image with reduced vertical parallax.

  Further, in the image processing apparatuses 1 and 2 in each of the above-described embodiments, the vertical parallax correction units 30 and 30S correct vertical parallax with reference to the position specified by the female mark displayed by the index display unit 50. Every time an arbitrary position on the stereoscopic image is specified by the female mark, the stereoscopic image after the vertical parallax is corrected can be displayed on the screen.

  As described above, the image processing apparatuses 1 and 2 in each of the above-described embodiments enable the vertical parallax correction that has been conventionally difficult to be realized. In the conventional image processing technology, even if an attempt is made to generate a stereoscopic image based on a plurality of two-dimensional images in which vertical parallax has occurred, the epipolar lines do not match due to the influence of vertical parallax, so matching between two-dimensional images is not possible. I couldn't do it well. Therefore, it has been necessary to remove vertical parallax manually. However, even when the vertical parallax is manually removed, it has been difficult to obtain road edges continuously and to obtain contour lines. As a result, good measurement accuracy could not be obtained.

  On the other hand, in the image processing apparatuses 1 and 2 in each of the above-described embodiments, by using area matching, the coordinates (correlation) on the front view image or the back view image corresponding to the coordinates on the nadir image that is the reference image are used. Since the coordinate information is determined irrespective of the epipolar line, a stereoscopic image after correcting the vertical parallax can be generated by performing stereo measurement based on the corresponding coordinates.

  In each of the above-described embodiments, when performing vertical parallax correction, vertical parallax correction is performed based on two types of geometric correction images generated by the geometric correction unit 10. The two-dimensional image to be is not limited to the two types of geometric correction images generated by the geometric correction unit 10. For example, based on the two-dimensional image shown in FIG. 10 and another two-dimensional image, the geometrically corrected image shown in FIG. 11 is generated, and based on the geometrically corrected image shown in FIG. 11 and the other two-dimensional image. The vertical parallax correction process described above may be performed.

  FIG. 10 is a graph showing an arrangement state of pixels in one of the two-dimensional images when generating a stereoscopic image based on two types of two-dimensional images (for example, a forward-view image and a direct-view image). . FIG. 11 is a graph showing a pixel arrangement state in the geometrically corrected image after geometric correction of the two-dimensional image shown in FIG. In the following description, for convenience of explanation, the graph shown in FIG. 10 is a graph showing the pixel arrangement state in the forward-view image, and the graph shown in FIG. 11 is shown in the geometrically corrected image of the forward-view image. It is assumed that the graph shows the pixel arrangement state. 10 and 11, the X axis indicates the line number of the line sensor, and the Y axis indicates the pixel number.

  Here, the procedure ((1) to (3)) for converting the forward-view image shown in FIG. 10 into the geometrically corrected image shown in FIG. 11 will be described. (1) The correlation coordinates between the forward view image and the nadir view image are determined for each line number. (2) The vertical parallax between the correlation coordinates determined in (1) is calculated for each line number. (3) Based on the vertical parallax calculated in (2), the pixel column of the line number corresponding to the vertical parallax is shifted in the Y direction by the vertical parallax. The geometrically corrected image (see FIG. 11) converted by the above-described procedure is generated without using position / posture data including many errors that cause vertical parallax. Therefore, a stereoscopic image in which the vertical parallax is further reduced can be generated by performing the vertical parallax correction process in the above-described embodiment using such a geometrically corrected image.

  In the above-described embodiment, the display unit 40 displays the stereoscopic image stereoscopically by alternately switching the two types of images at high speed and displaying them on one screen. The means for making the image appear three-dimensional is not limited to this. For example, a liquid crystal shutter is provided in a screen portion for displaying a stereoscopic image, and two types of images to be displayed on the screen are converted into images having vertical and horizontal polarization planes and alternately displayed on the screen. Also good. In this case, the user can see the image displayed on the screen three-dimensionally by viewing the screen while wearing polarized glasses. Further, for example, in a goggle type display device having a display screen dedicated to the left eye and a display screen dedicated to the right eye, the image for the left eye and the image for the right eye constituting the stereoscopic image are displayed on the respective display screens. It is good also as showing it in three dimensions.

It is a figure for demonstrating the position of each line sensor in the moment which image | photographs a certain point on a building. It is a figure which illustrates the function structure of the image processing apparatus in 1st Embodiment. It is the figure which modeled the process in which a geometric correction image is produced | generated based on a two-dimensional image and flight locus data. It is a figure for demonstrating the method of calculating | requiring the corresponding point between two-dimensional images using area matching. It is a flowchart which shows the outline | summary operation | movement in an image process. It is a flowchart which shows the operation | movement in the vertical parallax correction process in 1st Embodiment. It is the figure which modeled the process of the vertical parallax correction process in 1st Embodiment. It is a figure which illustrates the function structure of the image processing apparatus in 2nd Embodiment. It is a flowchart which shows the operation | movement in the vertical parallax correction process in 2nd Embodiment. It is a figure which shows the arrangement | sequence state of the pixel in a front view image. It is a figure which shows the arrangement | sequence state of the pixel in the geometric correction image of a front view image. It is a figure for demonstrating the mechanism in which a vertical parallax generate | occur | produces. It is a figure for demonstrating the mechanism in which a vertical parallax generate | occur | produces.

Explanation of symbols

  DESCRIPTION OF SYMBOLS 1, 2 ... Image processing apparatus, 10 ... Geometric correction | amendment part, 20 ... Stereoscopic part, 30, 30S ... Vertical parallax correction part, 31 ... Back projection part, 33, 33S ... Image matching unit, 35, 35S ... stereo measurement unit, 37 ... processing end determination unit, 38 ... low resolution image generation unit, 39 ... corresponding region extraction unit, 40 ... display unit, 50... Indicator display section, 60... Flight trajectory database, 70.

Claims (8)

An image processing apparatus that generates a stereoscopic image based on a plurality of two-dimensional images composed of one-dimensional image groups captured by a plurality of line sensors having different shooting directions,
Vertical parallax correction means for correcting vertical parallax, which is vertical parallax generated between the plurality of two-dimensional images,
An image processing apparatus that generates the stereoscopic image based on an image after the vertical parallax is corrected by the vertical parallax correction unit. Display means for displaying the stereoscopic image on a screen;
Index display means for displaying an index on the stereoscopic image displayed by the display means,
The image processing apparatus according to claim 1, wherein the vertical parallax correction unit corrects the vertical parallax every time an arbitrary position on the stereoscopic image is specified by the index. The vertical parallax correction means includes
A first coordinate determining means for determining any one of the plurality of two-dimensional images as a reference image and determining a coordinate on the reference image as a first coordinate;
Second coordinate determining means for determining a second coordinate corresponding to the first coordinate determined by the first coordinate determining means from a two-dimensional image other than the reference image among the plurality of two-dimensional images; ,
The third coordinate determining unit that determines coordinates on the stereoscopic image based on the first coordinates and the second coordinates, and comprising: a third coordinate determining unit that determines coordinates on the stereoscopic image. Image processing device. Storage means for storing correlation coordinate information between the plurality of two-dimensional images;
The second coordinate determining means determines the second coordinates by extracting the second coordinates corresponding to the first coordinates from the correlated coordinate information stored in the storage means. The image processing apparatus according to claim 3, wherein: The vertical parallax correction means includes
Low-resolution image generation means for generating a low-resolution image corresponding to each of the two-dimensional images by reducing the resolution of the plurality of two-dimensional images;
Based on a predetermined area corresponding to each of the low resolution images, a corresponding area extracting unit that extracts a corresponding area corresponding to the predetermined area from the two-dimensional image corresponding to each of the low resolution images;
Correlation coordinate determining means for determining correlation coordinates between the corresponding regions of the two-dimensional image which is a high resolution image corresponding to each of the low resolution images;
The image processing apparatus according to claim 1, further comprising: a stereoscopic image coordinate determining unit that determines coordinates on the stereoscopic image based on the correlation coordinates.   The plurality of two-dimensional images are geometrically corrected images that are geometrically corrected based on position data and posture data at the time of photographing by the line sensor. Image processing device.   The one-dimensional two-dimensional image of the plurality of two-dimensional images is a geometrically corrected image that is geometrically corrected based on a vertical parallax between correlation coordinates between the two-dimensional images. The image processing apparatus according to any one of the above. Display on the screen a stereoscopic image consisting of an image for the left eye and an image for the right eye based on a plurality of two-dimensional images consisting of a group of one-dimensional images captured while moving a plurality of line sensors in different shooting directions in the air An image processing device for causing
Index display means for displaying an index on the stereoscopic image;
Display means for displaying the stereoscopic image on a screen,
When the indicator is set on the stereoscopic image, the display unit is based on the plurality of two-dimensional images after correcting vertical parallax that is vertical parallax generated between the plurality of two-dimensional images. An image processing apparatus that displays the stereoscopic image on a screen and makes the stereoscopic image appear stereoscopically.

Images