JP2001266110A - System and method for positioning picture - Google Patents

Abstract

PROBLEM TO BE SOLVED: To easily and appropriately decide a corresponding point for positioning a picture photographed in the air in spite of the object of the air- taken picture and to appropriately decide the corresponding point even if the resolution of the air-taken picture is insufficient. SOLUTION: When a user searches the same linear objects on an earth surface plane from two air-taken pictures I1 and I2 and finds them, at least four or more corresponding segments are decided in accordance with the various same linear objects. A corresponding segment input part 15 recognizes the coordinate system of its own of the air-taken pictures I1 and I2, recognizes the coordinate of the start point and the end point of the corresponding segments, shows the corresponding segments in expressions of linear lines and calculates the coordinate of the intersection of the two linear expressions.

Description

DETAILED DESCRIPTION OF THE INVENTION

[0001]

BACKGROUND OF THE INVENTION 1. Field of the Invention The present invention relates to a method and a method for aligning two images including the same subject, which are photographed from different directions and different places.

[0002]

2. Description of the Related Art In the case of updating map information or confirming the state of a disaster at the time of a disaster, two aerial images showing the same point are sometimes compared. When updating the map information, a current and a past aerial image is compared, and a changed area is extracted. In the case of checking the damage situation at the time of a disaster, the aerial image at normal times where no disaster has occurred and the aerial image at the time of the disaster are compared to extract a damaged area.

Usually, the positions and directions (shooting angles) of the cameras when two aerial images are photographed are different from each other, so that the display position and display direction of the same subject are different from each other in the two aerial images. I have. When comparing these two aerial images, the images are aligned so that the display positions of the same subject overlap each other. Image alignment is generally performed by a method called camera calibration. In the camera calibration, a point (hereinafter, referred to as “corresponding point” in the present specification) corresponding to the same point on the ground plane is determined for both aerial images, and the position of the image is determined based on the corresponding point. Matching is performed. At least five corresponding points need to be determined, and are manually determined by the operator.

[0004]

In order to perform accurate image registration by camera calibration,
Corresponding points must be appropriately determined so as to be present evenly on each aerial image. However, as described above, since the display position and display direction of the same subject differ from each other in the two aerial images, it is difficult to appropriately determine the corresponding points so as to be present without bias. In particular, it is more difficult in the case of an aerial image of an urban area. The reason will be described with reference to FIGS.

FIGS. 1A and 1B show two aerial images for confirming the state of a disaster during a disaster. FIG. 1A shows an aerial image during normal times, and FIG. 1B shows an aerial image during a disaster. is there. In order to perform image registration, corresponding points must be appropriately determined for both these images. However, as can be seen from a comparison between (a) and (b), when the subject is in an urban area, the buildings are densely packed, or the view is blocked by smoke from a fire, and the ground plane hardly appears. Since the shape of a building, a road, or the like may have been changed due to destruction due to a disaster, construction, or the like, it is more difficult to appropriately determine a corresponding point.

FIGS. 2A and 2B show two aerial images including the city area where tall buildings such as high-rise buildings are lined up as the same subject. FIGS. 2A and 2B show a photographing place and a photographing direction, respectively. Are different. In order to perform image registration, corresponding points must be appropriately determined for both these images. However, as can be seen from (a) and (b), if a tall building is included, the ground surface will be hidden behind a tall building, or a tall building will cause a large shadow on the ground surface. As a result, the number of places where the corresponding points can be determined decreases. For this reason, it is more difficult to appropriately determine the corresponding points.

Further, in the case of an aerial image in which a relatively large area is shown, it is difficult to determine an appropriate corresponding point in the large area unless the resolution of the aerial image is sufficiently high. .

Accordingly, an object of the present invention is to make it possible to easily and appropriately determine a corresponding point for aligning an aerial image, regardless of the subject of the aerial image.

Another object of the present invention is to make it possible to appropriately determine corresponding points even if the resolution of an aerial image is not sufficient.

[0010]

An image registration method according to the present invention is a method for aligning two images displaying the same subject, and each of the two images has its own coordinate system. And includes a line segment designation unit, an intersection coordinate calculation unit, and an image conversion processing unit. The line segment designating unit designates, on each of the two images, a plurality of line segments corresponding to the two images. The intersection coordinate calculating means calculates, for each of the two images, coordinates of a plurality of intersections generated by a plurality of line segments or extension lines of the line segments. The image conversion processing means corresponds between the two images,
Based on the calculated plurality of intersection coordinates, the coordinate system of one image is converted into the coordinate system of the other image so that the display positions of the same subject overlap.

According to the present invention, when a user specifies a line segment corresponding to each other based on a straight line portion of the same subject, an intersection generated by the corresponding line segment, that is, a corresponding point is automatically determined. For this reason, it is possible to determine the corresponding point more easily and appropriately than when a point is directly determined on an image as in the related art. In addition, since the coordinates of the intersection generated by the extension line of the line segment are used, even if the corresponding straight line segment having a sufficient length is not defined on the captured image because the straight line portion of the subject is short, the corresponding point is determined. Determined. In addition, since an intersection may be formed outside the display area of the captured image, a virtual corresponding point can be determined outside the display area of the captured image.

[0012]

DETAILED DESCRIPTION OF THE PREFERRED EMBODIMENTS An embodiment of an image registration system in which the present invention is applied to aerial image registration will be described below.

First, a general image registration method will be described with reference to FIGS. This method is also used in an embodiment of the present invention described later.

Each of the two aerial images I ₁ and I ₂ shown in FIG. 3 has its own coordinate system. These two
Since the two aerial images I ₁ and I ₂ have different positions and directions (shooting angles) of the cameras C1 and C2 when the respective images are taken, the display positions and display directions of the same subject are different from each other ( As will be described in detail later, in the present embodiment, for the two aerial images I ₁ and I ₂ , corresponding points p ₁ and p ₂ that show the same point P on the ground plane K are simply and appropriately set. It is characterized in that it can be specified in

Aerial image I₁Camera C1 during shooting and aerial photography
Image I₂The camera C2 at the time of shooting has its own coordinates
The system exists. The position of one camera C1 is
The origin (0,0,0) of the coordinate system of
On the other hand, the position of the other camera C2 is represented by the coordinate system of the camera C1.
In (C_x, C_y, C_z), The camera orientation of camera C2
Let (ω, φ, κ) be in the coordinate system. These ω, φ, κ are
It represents the rotation angle about the axis of the coordinate system of the camera C2. this
When C_x, C_y, C_z, Ω, φ, κ are camera C1
It can be seen as an unknown variable indicating the degree of displacement of these cameras C2.
We can, and consider finding these unknown variables. That
For example, the same point P on the ground plane K is in the sky of the camera C1.
Image I₁Point p₁To (x₁, Y₁) And above
The same point P is an aerial image I of the camera C2.₂Point p
₂To (x₂, Y₂). (X₁, Y₁) Aerial photography
Image I₁Coordinates of the coordinate system of (x₂, Y₂) Is an aerial image
I ₂In the coordinate system of.

The position of the camera _{C2 (C x, C y,} C z) at, since _{C x} is a variable that affects the scale, based on _{C x, C y / C x} = c y, C z / C x = c _z
The position of the camera C2 is (1, _cy , _cz ). The camera C2 position _(1, c y, _{c z)} the camera position C1 (0,0,0), since it is on p1, _{p 2} and coplanar, the following equation (1) holds.

[0017]

(Equation 1)

Here, (X₁, Y₁, Z₁) Is the next
According to equation (2), p₁Aerial image I ₁From the coordinate system
Converted to the coordinate system of Mera C1, (X₂, Y₂,
Z₂) Is given by the following equation (3).₂Aerial image I₂
From the coordinate system of camera C1 to the coordinate system of camera C1.
You. Equations (2) and (3) for conversion are
This is an expression when the focal lengths of the lenses C1 and C2 are f.

[0019]

(Equation 2)

[0020]

(Equation 3)

[0021] (2) and (3), the unknown variables to be _{determined, c y, c z, ω} , φ, since it is five kappa,
If at least five pairs of p ₁ (x ₁ , y ₁ ) and p ₂ (x ₂ , y ₂ ), that is, five pairs of corresponding points are determined, the unknown variables _cy , _cz , There are five equations related to ω, φ, and κ. By solving them as simultaneous equations, unknown variables c _y , c _z , ω, φ, and κ can be obtained (however, in practice, the solution is rarely uniquely determined. , A solution is analytically obtained by using five or more pairs of corresponding points by the least squares method), and the camera C1
From the camera C2 is estimated.

The calculated unknown variables c _y , c _z , ω, φ,
Based on the value of kappa, by the following equation, as shown in FIG. 4, to overlap all the same subject to the aerial image I ₁ and the aerial image I _2, Aerial image I ₂ pictures I ₂ To '. That is, first, by the following equation (4), p
_{2 (x} 2, _{y 2)} coordinates when rotating kappa the z-axis of the coordinate system of the camera C2 (x _κ, y _κ) obtained.

[0023]

(Equation 4)

Next, according to the following equation (5), (x _κ , y _κ ) obtained by the above equation (4) is converted to y in the coordinate system of the camera C2.
The coordinates (x _φ , y _φ ) when rotated by φ around the axis are obtained.

[0025]

(Equation 5)

Next, according to the following equation (6), (x _φ , y _φ ) obtained by the above equation (5) is converted into x in the coordinate system of the camera C2.
The coordinates ( _xω , _yω ) when rotated by ω around the axis are obtained.

[0027]

(Equation 6)

Next, the following equation (6) is used according to the following equation (7).
(X_ω, Y_ω) Is X in the coordinate system of camera C1.
Coordinates (x_x, Y_x).
Note that Z₀Is the horizon from camera C1 position (0,0,0)
The length of the camera axis of the camera C1 up to the plane K, s is
(X_ω, Y_ω) And p₁(X₁, Y₁) And the distance ratio (x
_w ²+ Y_ω ²)^1/2/ (X₁ ²+ Y₁ ²)^1/2In
You. This Z₀And s are given by the following equations (8) and (9).
The same applies to the case.

[0029]

(Equation 7)

Next, according to the following equation (8), (x _x , y _x ) obtained by the above equation (7) is converted into Y in the coordinate system of the camera C1.
The coordinates ( _xy , _yy ) at the time of moving in the axial direction are obtained.

[0031]

(Equation 8)

Next, according to the following equation (9), (x _y , y _y ) obtained by the above equation (8) is converted to the Z coordinate of the camera C1.
The coordinates ( _xz , _yz ) at the time of moving in the axial direction are obtained.
Note that the coordinates (x _z , _yz ) correspond to the converted image I ₂ ′
Are coordinates in the coordinate system.

[0033]

(Equation 9)

With the above-described processing, the above-described image conversion processing is performed, and the converted image is displayed on the display 3 or stored in a predetermined file.

In image alignment, as can be understood from the above description, since the corresponding points need to be on the same plane, the ground surface photographed in both aerial images I ₁ and I ₂ Is desirably flat. This is the same for the present embodiment described below.

Hereinafter, the present embodiment will be described in detail. still,
In the following description, the aerial image as a base image registration as Aerial image I ₁ is called "base Aerial image",
Is image registration as Aerial image I ₂ (the image to be converted) aerial image will be referred to as "converted Aerial image".

FIG. 5 is a functional block diagram of a computer to which the image registration method according to the present embodiment is applied.

The display 3 and a user input device 5 such as a mouse and a keyboard are connected to the computer 1.
The computer 1 is typically a general-purpose computer such as a personal computer or a workstation. The computer 1 stores an image database (image D) in which a base aerial image and a conversion target aerial image are stored.
B) 8 and the base aerial image and the conversion target aerial image are extracted from the image DB 8 and both aerial images are displayed on the display 3.
And an image registration application 11 for performing the above-described registration in accordance with an operation by a user. The base aerial image is, for example, the data of the “current” aerial image for image registration when updating map information, and the “at the time of disaster” aerial image data for image registration during a rescue operation in the event of a disaster. The aerial image data to be compared (to be compared). On the other hand, the conversion target aerial image is, for example, “past” in image alignment when updating map information.
The aerial image data is the data of the “normal” aerial image in the case of the image registration at the time of the rescue operation in the event of a disaster, and is the aerial image data on the side to be compared.

FIG. 6 shows the configuration of the image registration application 11.

The image registration application 11 includes an image acquisition unit 13, a corresponding line segment input unit 15, a camera calibration execution unit 23, an image conversion unit 25,
An image output unit 27 is provided.

The image acquisition unit 13 acquires a base aerial image and a conversion target aerial image from the image DB 8, and writes both aerial images acquired for subsequent processing into the memory of the computer 1.

As shown in FIG. 7, the corresponding line segment input unit 15 displays the obtained two aerial images side by side on the display 3 and displays a line on the displayed two aerial images for the user. Enter minutes. Then, a process of calculating a corresponding point based on the line segment input by the user is executed.

FIG. 7 shows an example on two aerial images displayed together on the display 3. Referring to this figure,
In the present embodiment, the processing performed by the corresponding line segment input unit 15 together with the processing performed by the user on two aerial images will be described.

First, the user selects two aerial images I ₁ and I _{1
From 2} , a straight line on the ground plane, for example, a straight line portion on the ground contact surface (bottom surface) of the same building, a straight line portion on the same road or the same track, etc. are searched. When the user finds the same straight-line object, the user uses the user input device 5 (typically a mouse), and does not use a point but a line segment corresponding to each other according to the same straight-line object (for example, so as to follow the same straight-line object) (hereinafter, referred to as a line segment). , Corresponding line segment).

For example, as shown in the figure, when the user finds the same roads 50a and 50b as the same straight object, a corresponding line segment is determined on each of the aerial images I ₁ and I ₂ . That is, air in the shooting image _{I 1,} determines the starting point _{S 1} in the straight section of the road 50a, decide the end point _{E 1} at a desired position in accordance with the straight portion, defining a corresponding segment _S 1 _{E 1.} Similarly, on the aerial image _{I 2,} a road 50
determining the starting point S ₂ in the linear portion of b, determined the end point E ₂ at a desired position in accordance with the straight portion, corresponding segment S
Define the ₂ E _2.

[0046] The user, another identical road 53a as same line thereof, even when you find 53b, by performing the same processing as this, the corresponding line segment _S 3 _{E 3} is on aerial image _{I 1,} empty A corresponding line segment S ₄ E ₄ is defined on the captured image I ₂ .

When two corresponding line segments are determined in this way, intersections (p ₁ and p _{2 in the} figure) are formed by the corresponding line segments unless the corresponding line segments are parallel to each other. Aerial image I ₁ ,
In I _2, each corresponding line segment, so line segments defined based on the same straight line of the ground plane, the intersection of each respective line segments, the same point on the ground plane, that is, "corresponding points". More specifically, the corresponding points in the present embodiment are determined by extending each corresponding line segment and treating it as a straight line, and finding the intersection of the straight lines, as described later. Accordingly, even if a straight line portion of the subject is short on the aerial image and a corresponding line segment having a sufficient length cannot be determined, a corresponding point can be determined. In addition, this may cause intersections outside the display area of the aerial image, so unlike the conventional method of directly defining the corresponding point on the image, define a virtual corresponding point outside the display area of the aerial image. Can be.

The corresponding points are used in the processing in the camera calibration execution unit 23. In the processing,
At least five corresponding points are required as already described with reference to FIGS. Since the number v of intersections (corresponding points) by n straight lines is determined by the equation v = n (n-1) / 2, at least five corresponding points are determined in each of the aerial images I ₁ and I ₂ . For this purpose, four or more corresponding line segments are required. That is, in the present embodiment, the user defines four or more corresponding line segments on each of the aerial images I ₁ and I ₂ .

The coordinates of each corresponding point defined by four or more corresponding line segments given on the aerial images I ₁ and I ₂ are calculated by the corresponding line segment input unit 15 shown in FIG.

The corresponding line segment input unit 15 outputs each aerial image I ₁ ,
It recognizes its own coordinate system of I _2. According to that coordinate system,
The corresponding line segment input unit 15 performs the following processing for each of the aerial photographed images I ₁ and I ₂ . That is, the corresponding line segment input unit 15 confirms the coordinates of the start point and the end point of each corresponding line segment, and expresses each corresponding line segment as a straight line formula (each corresponding line segment is extended and treated as a straight line). Then, the corresponding line segment input unit 15 obtains two linear equations, for example, the following equations (10) and (11), and the following equations (12) are used to convert the equations (10) and (11). Angle α to be calculated,
Check whether the two straight lines are parallel. In Expressions (10) and (11), m and n indicate the numbers of the corresponding line segments, and x and y indicate the coordinates in a coordinate system unique to the image.

[0051]

(Equation 10)

[0052]

[Equation 11]

[0053]

(Equation 12)

When the angle α = uπ (u is an integer), the corresponding line segment input line 15 is in parallel with the corresponding line segment (straight line) m and the corresponding line segment (straight line) n, and there is no corresponding point. In addition, the corresponding line segment is determined by the user. If the angle α is not u = uπ (u is an integer), the corresponding line segment input unit 15 calculates
From the following equation (13) obtained from the equation (11), the intersection of the corresponding line segment (straight line) m and the corresponding line segment (straight line) n, that is, the coordinates of the corresponding point is calculated, and the coordinates are stored.

[0055]

(Equation 13)

The corresponding line segment input unit 15 outputs each aerial image I ₁ ,
In I _2, to the number of coordinates calculating is established number (at least 5), the above processing is repeatedly, when it becomes the default number is a pair of coordinates of the calculated corresponding points, camera calibration execution unit Notify 23.

The camera calibration execution unit 23
The camera position and orientation of the aerial images I ₁ and I ₂ are estimated based on the pair of corresponding point coordinates notified from the corresponding line segment input unit 15. That is, the camera calibration execution unit 2
3 uses the five or more corresponding point coordinate pairs described above and
The unknown variables c _y , c _z , ω, φ, and κ are calculated from the equations (1) to (3) described above with reference to FIG. Notify.

The image conversion processing unit 25 uses the notified values of the unknown variables c _y , c _z , ω, φ, and κ to obtain the coordinates (x _z , y) from the above-described equations (4) to (9). _z ), and
As shown in FIG. 8, a conversion target Aerial image I _2, the image I _{2 'which} align the same subject and based aerial image I ₁ in the image I ₂ to the position of the base Aerial image I ₁ Convert.

The image output unit 27 simultaneously displays the base aerial image I ₁ and the converted image I ₂ ′ on the display 3, or converts the data of both images I ₁ and I ₂ ′ into a predetermined file. It saves or prints out via the printer if a printer is connected to the computer 1.

FIG. 9 shows the flow of the image positioning process according to this embodiment.

First, data of the conversion target aerial image and the base aerial image are obtained from the image DB 8 (step S).
1) Two aerial image data are stored in the memory of the computer 1 (S2). The two stored aerial image data are displayed on the display 3, and are displayed by the user.
A plurality of corresponding line segments are input (S3). For each of the input corresponding line segments, it is determined whether there is at least one or more non-parallel corresponding line segments (S4). If all of the input corresponding line segments are parallel (no in S4), the corresponding line segments are again set. User inputs. If there is a corresponding line segment that is not parallel (yes in S4), a straight line formula of the corresponding line segment is obtained, and the coordinates of the intersection are calculated (S4).
5), and the intersection coordinates are stored as corresponding point coordinates (S
6). The operations in steps S3 to S6 are repeated until a predetermined number of pairs (corresponding to at least five) of corresponding point coordinates are calculated. When a predetermined number of pairs of corresponding point coordinates are calculated (yes in S6), the coordinates of each corresponding point pair and the above (1) to
The camera position and orientation are estimated by equation (3) (S
7) Based on the estimated camera position and orientation, the image to be converted is image-converted by the above equations (4) to (9) (S8), and the converted image is output to a predetermined file or the display 3. (S9).

As described above, according to the above-described embodiment, the user determines the line segment on the aerial image based on the same straight-line object such as a building or a road, and the corresponding point is input by the corresponding line segment input unit 15. Determined automatically. For this reason, the corresponding point can be determined more appropriately than when a point is directly determined on an image as in the related art. In addition, each aerial image is an image in which the buildings are dense and the ground plane hardly appears, or a part of the ground surface is hidden by high-rise buildings, or the resolution is not sufficiently high. Even if it can grasp the same straight object on the ground surface, the corresponding point can be determined appropriately. For this reason, the task of determining the corresponding points becomes very simple for the user.

According to the above-described embodiment, the corresponding points are determined by extending each corresponding line segment and treating it as a straight line, and finding the intersection of the straight lines. Accordingly, even if a straight line portion of the subject is short on the aerial image and a corresponding line segment having a sufficient length cannot be determined, a corresponding point can be determined. In addition, since an intersection may be formed outside the display region of the aerial image, a virtual corresponding point can be determined outside the display region of the aerial image. Further, since the number v of intersections of n straight lines is v = n (n-1) / 2, a plurality of corresponding points can be determined by simply increasing the number of corresponding line segments, so that many corresponding points can be efficiently used. Well given.

Although some preferred embodiments of the present invention have been described above, these are merely examples for describing the present invention, and are not intended to limit the scope of the present invention only to these examples. . The present invention can be implemented in other various forms. That is, the camera calibration using the corresponding line segment performed in the process of the image registration method proposed in the present invention is not limited to the application to the aerial image, and the corresponding point assignment in the general camera calibration is performed. It is also effective for the problem of.

[Brief description of the drawings]

FIG. 1A is a diagram showing an aerial photograph image in a normal state in a case of confirming a disaster situation at the time of a disaster. (B) The figure which shows the aerial image at the time of disaster in the case of confirmation of the damage situation at the time of disaster.

FIG. 2A is a diagram illustrating an aerial image captured from a certain direction when a tall building such as a high-rise building is included; (B) A diagram showing an aerial image taken from a different direction from (a) and including an image of a tall building such as the same high-rise building.

FIG. 3 is a diagram modeling an outline of image alignment according to an embodiment of the present invention.

In Figure 4 3, a diagram of when converting aerial image I ₂ to the image I _{2 '.}

FIG. 5 is a functional block diagram of a computer to which the image alignment method according to the embodiment is applied.

FIG. 6 is a block diagram showing a configuration of an image registration application 11;

FIG. 7 is a view showing an example of two aerial images displayed on a display 3;

Figure when aerial image I ₂ in FIG. 8] FIG. 7 is converted into an image I _{2 '.}

FIG. 9 is a flowchart illustrating a flow of an image registration process according to the embodiment.

[Explanation of symbols]

 Reference Signs List 1 computer 3 display 5 user input device 8 image database (image DB) 11 image registration application 13 image acquisition unit 15 corresponding line segment input unit 23 camera calibration execution unit 25 image conversion unit 27 image output unit

Claims (3)

[Claims] 1. A method for aligning two images displaying the same subject, wherein each of the two images has its own coordinate system, and each of the two images has Line segment designating means for designating a plurality of line segments corresponding to the two images; and coordinates of a plurality of intersections generated by the plurality of line segments or extensions of the line segments for each of the two images. Intersection coordinate calculation means for calculating the coordinate system of one image based on the calculated plurality of intersection coordinates corresponding to the two images so that the display position of the same subject overlaps with the other image. An image alignment method comprising: image conversion processing means for converting an image into a coordinate system. 2. A method for aligning two images displaying the same subject, wherein each of the two images has its own coordinate system, and each of the two images has: A line segment designating step of designating a plurality of line segments corresponding between the two images; and, for each of the two images, coordinates of a plurality of intersections generated by the plurality of line segments or extensions of the line segments. And calculating the intersection coordinates corresponding to the two images, based on the calculated plurality of intersection coordinates, changing the coordinate system of one of the images so that the display positions of the same subject overlap each other. An image conversion processing step of converting the image into a coordinate system of the image. 3. A method for aligning two images displaying the same subject, wherein each of the two images has its own coordinate system, and each of the two images has: A line segment designating step of designating a plurality of line segments corresponding between the two images; and, for each of the two images, coordinates of a plurality of intersections generated by the plurality of line segments or extensions of the line segments. And calculating the intersection coordinates corresponding to the two images, based on the calculated plurality of intersection coordinates, changing the coordinate system of one of the images so that the display positions of the same subject overlap each other. A computer-readable recording medium storing a program for causing a computer to execute an image conversion processing step of converting an image into a coordinate system.