JP2000115639A - Image input device and method - Google Patents

Abstract

PROBLEM TO BE SOLVED: To produce a composited image at a high speed and with high accuracy, even when 3 or more sheets of images of object planes are obtained by providing a projection means which projects the images on an arbitrary same image screen and composites the images which are photographed at plural visual points based on the specific information. SOLUTION: A projection means 25 projects each of images which are stored in a frame memory 15 on an arbitrary same image screen, based on the plane information outputted from a three-dimensional position/plane calculation means 23 and also on the outputs of an extension coefficient calculation means 24 which calculates the extension coefficients of the translation component obtained from every image pair and a three-dimensional position, an attitude detection means 19, a translation motion detection means 21, etc. Then the means 25 projects each of images stored in the memory 15 on a plane to composite an image, according to the projection condition that is defined based on the outputs supplied from the means 19, 21, 23 and 24.

Description

DETAILED DESCRIPTION OF THE INVENTION

[0001]

[0001] 1. Field of the Invention [0002] The present invention relates to an image input apparatus for photographing a plurality of images, such as a digital still camera and a digital video camera, and an image input method thereof.

[0002]

2. Description of the Related Art Conventionally, an image input apparatus has been widely used, in which an image passing through an optical system is photoelectrically converted by an image pickup device such as a CCD sensor and an image signal obtained by the photoelectric conversion is converted into an A signal.
/ D converted and stored in an external storage device such as a memory card. Such an image input device can handle an image as a digital signal.
There is an advantage that processing and transmission can be easily performed.

On the other hand, there is a demand for a device capable of easily and accurately inputting flat characters and photographs such as signs and panels. A paper image of about A3 size can be easily read by a device such as a copying machine or a scanner. However, the image input device described above cannot read large-area paper information such as newspaper or information drawn on a panel wall. Impossible. Therefore,
A method has emerged in which a large image is divided and photographed using a portable image input device, and a plurality of obtained images are combined to synthesize a high-resolution panoramic image (conventional example 1). In this method, a panoramic image is formed by overlapping and projecting a subject image taken such that a part of each image overlaps or each image is connected seamlessly to a projection surface such as a spherical surface. Combined.

Further, Japanese Patent Application Laid-Open No.
In Japanese Patent No. 67, in order to improve continuity between images to be synthesized, partial images common to each other in images obtained from a plurality of image input means are determined, and a corresponding point pair in the partial images is extracted. 2. Description of the Related Art An image processing apparatus has been proposed in which a device for creating a composite image from a three-dimensional position of a corresponding point pair and dividing the composite image is provided.

In addition, Japanese Patent Application Laid-Open No. 9-32 as Conventional Example 3
In Japanese Patent Publication No. 2040, when combining images obtained by dividing and photographing a subject so as to partially overlap, based on a result of discriminating whether a series of images has been photographed at a long distance or a short distance. Therefore, an image generating apparatus has been proposed which selectively switches between short-range image synthesis and long-range image synthesis. In detail, in the case of close-up shooting, an image is synthesized based on the vertical and horizontal translation on the screen, the rotation angle around the optical axis, and the change in magnification due to the translation along the optical axis. The image is synthesized based on the rotation angle of the device between the captured image and the next captured image.

[0006]

However, the above conventional example 1
Is applicable only to a distant view image in which the subject image hardly changes even when the image input device is translated, and is not suitable for synthesizing a subject image at a short distance.

Further, according to the conventional example 2, since images obtained from a plurality of image input means mounted in the apparatus in advance are combined, it is impossible to photograph a subject under photographing conditions preferred by the user. . Also, there is no disclosure of a technique for creating a composite image from three-dimensional positions.

Further, according to the above-mentioned conventional example 3, complicated processing of switching between short-distance image synthesis and long-distance image synthesis is required, and the discrimination between the two is performed by threshold processing of the focus information and the magnitude of the rotation angle. Therefore, there is a problem that an unnatural seam is likely to occur in a portion where the synthesized image is switched between the short-range image synthesis and the long-range image synthesis. In addition, in the case of close-up shooting, only the rotation angle (roll angle) around the optical axis of the device is considered, so the vertical rotation angle (pitch angle) and the horizontal rotation angle (yaw angle) of the device. When images are distorted due to the above, it is impossible to bond the images seamlessly.

An object of the present invention is to solve these problems. Even when three or more images of a target plane are obtained, an image input device capable of creating a composite image with high accuracy and high speed. It is an object to provide an apparatus and a method thereof.

[0010]

SUMMARY OF THE INVENTION In order to solve the above-mentioned problems, the present invention provides an image pickup means for picking up an object plane from a plurality of viewpoints so as to partially overlap a reference image which is a previously picked-up image. A plurality of feature points in the reference image based on an image pair of an orientation detection unit that detects an orientation of the imaging unit at each imaging, and a reference image that is an image having a portion overlapping with the reference image. Correspondence detecting means for extracting and detecting a corresponding point indicating the same point as the feature point in the reference image, and translational motion for detecting a translational motion component of the imaging means based on the attitude, the feature point, and the corresponding point of the imaging means Detecting means, three-dimensional position measuring means for calculating a three-dimensional position of each feature point based on the attitude, feature point, corresponding point, and translational motion component of the imaging means, and each feature point measured by the three-dimensional position measuring means Are the same Plane calculating means for calculating information of a plane conforming to the three-dimensional position of each feature point as being on a plane, three-dimensional position of each feature point obtained by using each image pair, and plane calculating means Based on the information of the plane, based on the information of the plane calculated by the magnification factor calculating means for calculating each translation component and the expansion coefficient of each three-dimensional position, the orientation and the translation component of the imaging means, and the plane calculation means, A projection unit for projecting each image onto an arbitrary image plane and combining images captured from a plurality of viewpoints.

According to still another aspect of the present invention, there is provided an image pickup means for picking up an object plane from a plurality of viewpoints so as to partially overlap a reference image which is a previously picked-up image; And extracting a plurality of feature points in the reference image based on the image pair of the reference image and the reference image which is an image having a portion overlapping with the reference image, and extracting a plurality of feature points identical to the feature points in the reference image. Correspondence detection means for detecting a corresponding point indicating the position of the image, three-dimensional position measurement means for calculating the three-dimensional position of each feature point based on the attitude and translation component of the imaging means, and the feature point and the corresponding point; A plane calculating unit for calculating information on a plane conforming to the three-dimensional position of each feature point assuming that the three-dimensional positions of each feature point measured by the three-dimensional position measuring unit are on the same plane, and using each image pair Required features Magnification factor calculating means for calculating the translation component and the magnification factor of each three-dimensional position based on the three-dimensional position and the plane information calculated by the plane calculating means, the attitude and the translation component of the imaging means, and the plane calculation A projection unit for projecting each image onto an arbitrary image plane based on the information on the plane calculated by the unit, and synthesizing images captured from a plurality of viewpoints. Therefore, according to the above two inventions, even when a large number of target planes at arbitrary distances are photographed, plane information can be calculated with high accuracy by a simple calculation, and a beautiful composite image can be obtained. . Further, since the overlapping range of the images is not preliminarily forced, the user can photograph the target plane under relatively free conditions. Furthermore, since it is not necessary for the user to manually set the order of the captured images, it is possible to combine images with high operability and usability.

According to another aspect of the present invention, an image pickup means for picking up an object plane from a plurality of viewpoints so as to partially overlap a reference image which is a previously picked up image, and a posture of the image pickup means at each image pickup are detected. Based on an image pair of a posture detection unit, a reference image, and a reference image which is an image having a portion overlapping with the reference image, a plurality of feature points in the reference image are extracted and the same as the feature points in the reference image. Correspondence detection means for detecting a corresponding point indicating a location, translational motion detection means for detecting a translational motion component of the imaging means based on the attitude, characteristic points and corresponding points of the imaging means; The three-dimensional position measuring means for calculating the three-dimensional position of each feature point based on the corresponding point and the translational motion component, and the three-dimensional position of each feature point measured by the three-dimensional position measuring means on the same plane Each feature point A plane calculating means for calculating information on a plane conforming to the three-dimensional position, a plane optimizing means for optimizing plane information on the target plane based on the information on the plane calculated by the plane calculating means, and each image pair is used. Enlargement coefficient calculating means for calculating each translation component and an enlargement coefficient of each three-dimensional position based on the three-dimensional position of each feature point obtained as described above and information on the plane optimized by the plane optimizing means, Projection means for projecting each image onto an arbitrary same image plane and synthesizing images captured from a plurality of viewpoints, based on the posture and translation components of the image and the plane information optimized by the plane optimization means. There is a feature.

According to still another aspect of the present invention, there is provided an image pickup means for picking up an object plane from a plurality of viewpoints so that a part of the image is overlapped with a reference image which is a previously picked up image; And extracting a plurality of feature points in the reference image based on the image pair of the reference image and the reference image which is an image having a portion overlapping with the reference image, and extracting a plurality of feature points identical to the feature points in the reference image. Correspondence detection means for detecting a corresponding point indicating the position of the image, three-dimensional position measurement means for calculating the three-dimensional position of each feature point based on the attitude and translation component of the imaging means, and the feature point and the corresponding point; Plane calculating means for calculating information on a plane conforming to the three-dimensional position of each feature point, assuming that the three-dimensional position of each feature point measured by the three-dimensional position measuring means is on the same plane, and a plane calculated by the plane calculating means Feelings Plane optimization means for optimizing plane information of the target plane based on the three-dimensional position of each feature point obtained by using each image pair, and information on the plane optimized by the plane optimization means. A magnification factor calculating means for calculating a magnification factor of each translational component and each three-dimensional position; and an image of each image can be arbitrarily determined based on the orientation of the imaging means, the translational component, and the plane information optimized by the plane optimizing means. And projection means for projecting images onto the same image plane and combining images captured from a plurality of viewpoints. Therefore, according to the above two inventions, an accurate plane equation can be calculated even when an error is included in the posture or motion detection result of the image input device. As a result, a more beautiful composite image can be created.

[0014]

DESCRIPTION OF THE PREFERRED EMBODIMENTS An image pickup means for picking up an object plane from a plurality of viewpoints so as to partially overlap a reference image which is a previously picked up image, and a posture detecting means for detecting the posture of the image pickup means at each image pickup. Means for extracting a plurality of feature points in the reference image based on an image pair of the reference image and a reference image which is an image having a portion overlapping with the reference image, and extracting a same point as the feature point in the reference image. Correspondence detecting means for detecting a corresponding point shown, translational motion detecting means for detecting a translational motion component of the imaging means based on the attitude, feature point, and corresponding point of the imaging means; and posture, feature points, and corresponding points of the imaging means. And calculating the three-dimensional position of each feature point based on the translational motion component 3
Three-dimensional position measuring means, and plane calculating means for calculating information of a plane conforming to the three-dimensional position of each feature point assuming that the three-dimensional position of each feature point measured by the three-dimensional position measuring means is on the same plane; Enlargement coefficient calculation means for calculating each translation component and an expansion coefficient for each three-dimensional position based on the three-dimensional position of each feature point obtained by using each image pair and information on the plane calculated by the plane calculation means; A projection unit for projecting each image onto an arbitrary same image plane and synthesizing images captured from a plurality of viewpoints, based on the posture and translation component of the imaging unit and information on the plane calculated by the plane calculation unit. .

[0015]

Embodiments of the present invention will be described below with reference to the drawings. FIG. 1 is a diagram showing a state of photographing using the image input device of the present invention. As can be seen from the figure, using the image input device 1, different locations, that is, a plurality of
From K, still images of the same target plane are sequentially photographed such that part of the images overlap. Here, an image photographed at the viewpoint 1 is referred to as an image 1, and an image photographed at the viewpoint 2 is referred to as an image 2. Further, for simplicity of description, between an image j (not shown) and an image (j + 1) (not shown) taken in adjacent order, such as image 1 and image 2 and image 2 and image 3. Suppose that there is an overlap region in.

FIG. 2 is a block diagram showing the configuration of the image input device according to the first embodiment of the present invention. The image input device 1 shown in FIG. 1 mainly includes an imaging unit 11, a signal processing unit 12 for processing an image signal obtained by the imaging unit 11, and a memory to which an output processed by the signal processing unit 12 is supplied. The controller 13, the main controller 14 and an interface (hereinafter abbreviated as I / F) 16, a frame memory 15 for storing image signals according to instructions from the memory controller 13, and displaying image signals via the I / F 16. The display unit 17 includes a display unit 17 and an external storage unit 18 that reads / writes various signals including an image signal via the I / F 16. In addition, a posture detecting means 19 for detecting the posture of the image input device 1 at the time of image capturing, a correspondence detecting means 20 for extracting feature points and corresponding points between images having mutually overlapping areas accumulated in the frame memory 15 and Motion detecting means 2 for detecting the translation motion of the image input apparatus 1 from the outputs from the attitude detecting means 19 and the correspondence detecting means 20
1, three-dimensional position measuring means 22 for calculating the three-dimensional position of each feature point from the outputs from the attitude detecting means 19, the correspondence detecting means 20, and the translational motion detecting means 21, and the three-dimensional position measuring means 22 Plane calculating means 23 for calculating information of a plane to which the three-dimensional position of each feature point is suitable, and three-dimensional position of each feature point outputted from the three-dimensional position measuring means 22 and plane information outputted from the plane calculating means 23 On the basis of the,
Magnification factor calculating means 24 for calculating a translation component and a three-dimensional position magnification coefficient obtained from each image pair;
9. Projection means 25 for projecting each image stored in the frame memory 15 onto an arbitrary same image plane based on the outputs supplied from the translational motion detection means 21, the plane calculation means 23 and the magnification coefficient calculation means 24. are doing.

The details of each block will be described below. The imaging means 11 includes a lens 111, an aperture 112, a shutter 11
3, and a photoelectric conversion element 114 and a pre-processing means 115. For example, CC
D (charge coupled device) is used. Further, the pre-processing unit 115 includes an analog signal processing such as a preamplifier and an AGC (auto gain control) or an analog-to-digital converter (hereinafter, A / D converter).
D, which is abbreviated as D converter). The analog video signal output from the photoelectric conversion element 114 is subjected to preprocessing such as amplification and clamping, and then the analog video signal is converted into a digital video signal by an A / D converter. Convert to a signal. The signal processing unit 12 includes a digital signal processor (hereinafter abbreviated as a DSP processor) and the like.
Various digital image processing such as color separation, white balance adjustment, and γ correction are performed on the digital video signal obtained from Step 1.
The memory control unit 13 stores the image signal processed by the signal processing unit 12 in the frame memory 15 and reads the image signal stored in the frame memory 15 on the contrary.
The frame memory 15 can store at least two images, and generally uses a VRAM, an SRAM, a DRAM, or the like. The recording of the image signal read from the frame memory 15 is performed by performing signal processing such as image compression on the image signal in the signal processing unit 12 and storing the image signal in the external storage unit 18 via the I / F 16. Done. As the external storage means 18, an IC memory card, a magneto-optical disk, or the like can be used, but an image signal may be directly transmitted to a remote recording medium via a network using a modem card or an ISDN card. . Conversely, reading of the image signal recorded in the external storage device 18 is performed by the I / F
This is performed by transmitting an image signal to the signal processing unit 12 through the signal processing unit 16 and performing image expansion in the signal processing unit 12. On the other hand, the external storage means 18 and the frame memory 1
The display of the image signal read from the display unit 5 is performed by the signal processing unit 1.
In 2, the image signal is subjected to signal processing such as digital-analog conversion (hereinafter abbreviated as D / A conversion) or amplification, and then transmitted to the display unit 17 via the I / F 16. The display unit 17 is composed of, for example, a liquid crystal display device installed in the housing of the image input device 1.

Next, the attitude detecting means 19 determines the attitude of the image input device 1 at the time of shooting the reference image and at the time of shooting the reference image.
It is detected in the form of rotation angles around three orthogonal axes. As an element for detecting the attitude of the image input apparatus 1, an acceleration sensor, an angular velocity sensor, a magnetic sensor, or the like may be used alone or in combination. However, here, as shown in FIG. A description will be given of an operation in the case of being composed of a three-axis acceleration sensor 191 for detecting gravitational acceleration in three-axis directions, a three-axis magnetic sensor 192 for detecting terrestrial magnetism in three axes directions, a sensor signal processing means 193 and an attitude calculation means 194 I do. First, two coordinate systems, a device coordinate system and a world coordinate system, are defined, and are defined as follows. Here, it is assumed that there is no magnetic field other than geomagnetism.

(A) Apparatus coordinate system: xyz coordinate system (see FIG. 4) x-axis: positive right in image plane y-axis: positive downward in image plane z-axis: optical axis direction; direction toward object is positive Origin o: Optical center of the imaging unit 11 f and p will be described later.

(B) World coordinate system: XYZ coordinate system (see FIG. 5) X-axis: positive eastward direction Y-axis: positive gravitational acceleration direction Z-axis: positive magnetic direction (northward) Origin O: reference Viewpoint position when the image was taken

The three-axis acceleration sensor 191 calculates the gravitational acceleration x.
The three-axis magnetic sensor 192 determines the x-axis and y-axis components,
It is installed to detect the axis, y-axis, and z-axis components.
These sensor signals are subjected to signal processing such as filtering and amplification and A / D conversion in a sensor signal processing unit 193, and are sent to a posture calculation unit 194.

Then, the attitude calculating means 194 calculates the attitude of the image input apparatus 1 from the processed sensor signals. The attitude of the image input device 1 is described by the following rotation matrix based on the world coordinate system.

R = R _Y R _X R _Z (1) where

[0024]

(Equation 1)

## EQU1 ## Now, the gravitational acceleration vector and the geomagnetic vector are

[0026]

(Equation 2)

The acceleration vector and the terrestrial magnetic vector detected by the three-axis acceleration sensor and the three-axis magnetic sensor with reference to the apparatus coordinate system are respectively

[0028]

(Equation 3)

It is assumed that The above relationships between g and a, and between M and m, are described by the following equations using the rotation matrix R.

Ra = g (5) Rm = M (6)

From the equation (5), a rotation angle α around the X axis and a rotation angle γ around the Z axis are calculated.

[0032]

(Equation 4)

Using α and γ obtained by the equations (7) and (8), the rotation angle β around the Y axis is calculated from the geomagnetic vector m as follows.

[0034]

(Equation 5)

However,

[0036]

(Equation 6)

Is as follows. With the above calculations, α, β, γ and the rotation matrix R can be calculated from the detection values of the three-axis acceleration sensor and the three-axis magnetic sensor, and the attitude of the image input device 1 with respect to the world coordinate system can be described.

Next, as an example of the correspondence detecting means 20, a method of detecting corresponding points by correlation calculation in two images having mutually overlapping areas will be described. The correspondence detecting means 20 includes a feature point setting means 201 and a correlation calculating means 202 as shown in FIG. In the frame memory 15 of FIG. 2, an image (reference image) captured earlier and a reference image having an area overlapping the reference image are stored. Therefore, the feature point setting means 2
No. 01 determines the position of a feature point in the reference image, extracts a (2N + 1) (2P + 1) gray-scale pattern centered on the feature point, and creates a correlation window. The position of the feature point is determined by extracting a portion where the density pattern of the image is characteristic, such as a corner. Further, the correlation calculating means 202 detects, by a correlation calculation in the reference image, a portion that substantially matches the shading pattern of the correlation window created in the reference image, and determines this as a corresponding point. Here, an example in which a corresponding point is detected by block matching by a correlation operation will be described. FIG.
In the block matching using the (2N + 1) (2P + 1) correlation window, the i-th feature point P _i (x _i0 , y _i0 ) in the reference image and the point (x _i0 +
dx _i, the cross-correlation value S _i of y _i0 + dy _i) is calculated by the following equation.

[0039]

(Equation 7)

Here, the meaning of each symbol in the equation (11) is as follows. I _s (x, y): density at point (x, y) of reference image I _r (x, y): density at point (x, y) of reference image

MI _s (x, y): point (x, y) of the reference image
mean density in a (2N + 1) (2P + 1) correlation window centered on y) MI _r (x, y): average in a (2N + 1) (2P + 1) correlation window centered on point (x, y) of the reference image Concentration K: constant

Each feature point P_iFor the cross-correlation value S_iMost
Finding points where the maximum value is greater than or equal to a predetermined threshold
From the corresponding point (x_i0+ Dx_i, Y_i0+
dy_i ) Is required. S_iIf the maximum of is less than or equal to the threshold,
It is assumed that there is no corresponding point.

The translational motion detecting means 21 calculates the focal length f of the optical system of the imaging means 11, the attitude information detected by the attitude detecting means 19, and a set of feature points in the reference image and corresponding points in the reference image. The translational motion component of the image input device 1 is obtained by performing the following processing. First, a gaze vector based on the world coordinate system is calculated for a feature point in the reference image and a corresponding point in the reference image. Assuming that the optical system of the imaging unit 11 is a center projection model with a focal length f as shown in FIG.
The unit line-of-sight vector p for the point (x, y) on the image is
It is determined as follows.

[0044]

(Equation 8)

Here, p in equation (12) is a line-of-sight vector based on the imaging coordinate system. For example, as shown in FIG. 8, the characteristic point in the reference image (x _s, y _s) corresponding point in the reference image (x _r, y _r) with respect to a unit line-of-sight vector of the following is calculated.

[0046]

(Equation 9)

However, p in the expressions (13) and (14)_sIs the standard
Unit line-of-sight vector based on imaging coordinate system in image
Le, p_rIs a unit based on the imaging coordinate system of the reference image.
It is a position gaze vector. p_s, P_rBased on the world coordinate system
To convert to the standard line-of-sight vector,
And posture information at the time of photographing the reference image (that is, (1)
(Rotation matrix of the expression). However, in both images
Rotation matrix (R_s , R_rIs the attitude detecting means 1
9 so that p_s, P_rIs
Is the gaze vector P based on the world coordinate system._s, P_r
Is converted to That is,

[0048] _{P s = R s p s (} 15) P r = R r p r (16)

Is as follows. From the line-of-sight vector group based on the world coordinate system of the feature points and corresponding points determined above,
The translation component of the imaging unit 11 is calculated. For example, in FIG. 8, _assuming that the line-of-sight vectors P _s and _Pr are in a correspondence relationship,
Three vectors of P _s , P _r , and a unit vector t indicating a translation component exist on the same plane. When this is expressed by an equation, it is expressed by the following scalar triple product equation.

(P _s × t, _Pr ) = 0 (17)

Therefore, the vector t satisfying the expression (17) is a translational motion vector. However, the expression (17) is expressed by the line-of-sight vectors P
_{s, i (i = 1,} ···, N) and, line-of-sight vector P _r of the corresponding point, i (i = 1, ··· , N) does not hold against.
Therefore, t that minimizes the sum of the absolute values of the following scalar triple products may be obtained.

[0052]

(Equation 10)

From the above calculations, the translational movement of the image input device 1 from the time of photographing the reference image to the time of photographing the reference image can be obtained.

However, since the absolute value of the translational motion cannot be obtained, the magnitude of the translational motion vector t is set to 1.

Next, the three-dimensional position measuring means 22
And the line-of-sight vector P for the corresponding point_{s, i}, P_{r, i}And translation
Using the translational motion vector obtained by the motion detection means 21
And the three-dimensional position (X
_i, Y_i, Z_i) Is calculated. Where the world coordinate system
Since the origin is the viewpoint position when the reference image was taken,
(X_i, Y_i, Z_i) Is based on the viewpoint position when the reference image was taken.
Note that the point is.

The plane calculating means 23 determines that each feature point detected by the correspondence detecting means 20 is on the same plane, and calculates the three-dimensional position (X _i , Y _i , Z _i ) to calculate plane information. Specifically, the equation based on the world coordinate system of the plane to be obtained is

AX + bX + cZ + d = 0 (a ² + b ² + c ² = 1) (19)

In addition, four parameters (a, b, c, d), which are more plane information, are obtained by using (X _i , Y _i , Z _i ).
Is calculated by, for example, the least squares method.

The expansion coefficient calculating means 24 measures the three-dimensional position.
The three-dimensional position of each feature point (X_i,
Y_i, Z_i) And the plane information output from the plane calculating means 23
Translation component and 3D obtained from each image pair based on the information
Calculate the expansion factor of the position. For example, target plane
3D position of the feature point (X_i, Y_i, Z_i)
When calculating, (X) as shown in FIG._i, Y_i,
Z_i) Should be approximately coplanar. Only
The three-dimensional position measuring means 22 calculates (X_i, Y_i ,
Z_i) Indicates the average time from when the reference image was taken to when the reference image was taken.
Is calculated using the forward motion vector b as the unit vector
Therefore, as shown in FIG.
This is a three-dimensional position when the heights are equal. this
Is that the positional relationship of the image input device 1 at the time of shooting is unknown to each other.
When recovering the three-dimensional shape of a subject from two images
In this case, the actual size of the subject and the size of the translation
Is possible and the ratio between the size of the subject and the size of the translation
This is because a three-dimensional shape is required in this manner. like this
Also, in monocular vision, far and large and near and small
The inability to distinguish between
You. Hereinafter, the constraint condition that the subject is on the same plane
Of the translation and 3D position obtained from each image pair
The procedure for obtaining the enlargement factor will be described.

First, variables are defined as follows.

T _i : image j as reference image, image (j + 1)
Is a unit translational motion vector calculated as a reference image.

P ^m _{j, i} : 3 with respect to the i-th feature point in the image j, expressed in the world coordinate system with the viewpoint m as the origin.
Dimensional position.

The image j is obtained as a reference image, and the image (j + 1) is obtained as a reference image. k _j : magnification factor for image j. N _j : number of feature points in image j.

In FIG. 10, three-dimensional positions calculated using image 2 as a reference image and image 3 as a reference image

[0065]

[Equation 11]

The translational motion vector calculated using image 1 as a reference image and image 2 as a reference image

[0067]

(Equation 12)

And a plane obtained from the three-dimensional position

[0069] _{a 1 X + b 1 Y +} c 1 Z + d 1 = 0 (a 1 2 + b 1 2 + c 1 2 = 1) (22)

Explanation will be made using an example applied to the above. Here, the enlargement coefficient k _{2 to be obtained} is an unknown variable, and k ₂ is calculated in which the three-dimensional position k ₂ P ² _{2, j} with respect to N ₂ feature points best fits the plane of the equation (22). Here, the plane of equation (22) is a plane equation when image 1 is used as a reference image. Therefore, it is necessary to convert the three-dimensional position relative to the viewpoint 1 P 2 _{^2, j.} Using the translational motion vectors k ₁ t ₁ and k ₂ from viewpoint 1 to viewpoint 2, P ²
_{2, j} is converted to a three-dimensional position P ¹ _{2, j based} on viewpoint 1.

P ¹ _{2, j} = k ₁ t ₁ + k ₂ P ² _{2, j} (23)

Therefore, P ¹² _{, j} which best fits the plane equation of equation (22) may be calculated. For example, the following equation may be solved using the least square method.

[0073]

(Equation 13)

Where n is a normal vector of a plane,

[0075]

[Equation 14]

Is expressed as follows. Using k ₁ = 1 (24)
The following equation is obtained by further modifying the equation.

[0077]

(Equation 15)

From the above equation, the expansion coefficient k ₂ can be obtained by a simple linear calculation.

Although the calculation method of the enlargement factor when photographing from three viewpoints has been described above, the present invention can be applied to a case where photographing is performed from more viewpoints. Specifically, the enlargement coefficient for the image j (j = 1,..., K) can be sequentially obtained by the following iterative processing.

(1) The plane of the equation (22) obtained by using the image 1 as a reference image and the image 2 as a reference image is set as a fitting plane. Here, when obtaining the plane (22) the value of k ₁ can be arbitrarily set, for example, k ₁ = 1.

(2) For j = 2,..., K−1,
The following processes (3) to (5) are repeated.

(3) Corresponding to each feature point in image j
Three-dimensional position P with the viewpoint j as the origin^j _{j , i}(I = 1,
.., N_j) To k_jMultiply.

(4) Convert k _j P ^j _{j, i} to a three-dimensional position based on viewpoint 1.

[0084]

(Equation 16)

(5) By the following least square calculation, k _j satisfying the following equation is calculated.

[0086]

[Equation 17]

Next, the projecting means 25 comprises the attitude detecting means 1
9. Projecting each image stored in the frame memory 15 onto a plane using projection conditions based on the outputs supplied from the translational motion detecting means 21, the plane calculating means 23 and the magnification coefficient calculating means 24, Combine. Hereinafter, the operation of projecting each image on the same plane will be described with reference to FIG. A point at which a straight line connecting the point P _{j, i} on the image obtained at the viewpoint j and the optical center of this viewpoint intersects the plane G _c calculated by the plane calculating means 23 is defined as P _c . Further, an area surface on which an image to be newly synthesized is projected is defined as _Im , and the optical center is defined as _Om . Straight line connecting the point P _c and the optical center O _m is projected pixel to P _m point which intersects the image plane. This behavior
By performing for all necessary pixel, as shown in FIG. 12, to form a new image on the image plane I _m. This processing is performed for each image j (j = 1,...
-, it performed for K), the imaging means 11 by projecting each image captured in any of the same image plane I _m, combining images taken from a plurality of viewpoints. The composite image created in this way is stored in a separately provided storage unit (not shown) or displayed on the display unit 17.

Next, the operation of the image input apparatus of the first embodiment will be described with reference to the operation flow shown in FIG. First, a plane to be input is photographed by the imaging means 11, and this image is stored as a reference image, and at the same time, the attitude of the image input apparatus 1 at the time of photographing is detected (step S101). Then, the viewpoint and the attitude of the image input device 1 are changed so that a part of the image is overlapped with the reference image, the input target plane is photographed by the imaging means 11, and this image is stored as a reference image. The attitude of the input device 1 is detected (step S102). A feature point is extracted from the reference image, and a corresponding point indicating the same location as the feature point is detected in the reference image (step S103). Thus the corresponding points between the obtained feature points above (15) and (16) line-of-sight vector P _s relative to the world coordinate system using equation converts each P _r, at each image shooting Using the attitude of the image input device 1 of the above, the image input device 1
Is obtained (step S104).
Its translation vector t and a line-of-sight vector P _s, by using the P _r, the viewpoint at the time of the reference image photographing as the origin, calculating the three-dimensional position of each feature point by the principle of triangulation (Step S105). After the above processing, it is determined whether or not plane information has already been calculated (step S106). If not, a plane equation is calculated (step S106).
107), when the plane equation already exists, 3
An enlargement coefficient is calculated so that the dimensional position applies to the plane equation (step S108), and based on the plane equation and the enlargement coefficient, the reference image and the reference image are projected onto the same image plane Im by the method described above, A composite image is created (Step S109). After that, in order to increase the number of images to be photographed and combined, the image used as the reference image in the above processing is replaced with the reference image, and the operations from step S102 to step S109 are repeated. (Steps S110, S
111).

The above description is merely an example of the present invention. For example, as for the posture detecting means 19, an example constituted by a three-axis acceleration sensor and a three-axis magnetic sensor has been described, but another form such as using an angular velocity sensor may be employed. Also, it has been described that a process of combining images is performed each time one image is captured, but a combined image may be created by batch processing after all images have been captured. Further, it has been described that the three-dimensional position in each image is applied to the plane calculated by using the image 1 as the reference image and the image 2 as the reference image. However, the plane may be calculated by selecting an arbitrary image pair having an overlapping area with each other. I do not care.

FIG. 14 is a block diagram showing the configuration of an image input apparatus according to the second embodiment of the present invention, and FIG. 15 is a flowchart showing the operation of this embodiment. In FIG. 14, the image input device 1 includes an imaging unit 11 and an imaging unit 11.
Processing means 12 for processing the image signal obtained by
, A memory control unit 13 to which an output of the signal processing unit 12 is supplied, a main control unit 14 and an I / F 16, a frame memory 15 for storing image signals in accordance with a command from the memory control unit 13, and an I / F 16. Display means 17 for displaying image signals through the I / F 16 and external storage means 18 for reading / writing various signals including the image signals via the I / F 16. In addition, motion detection means 26 for detecting the position and orientation of the image input device 1 at the time of image capturing, and correspondence detection means for extracting feature points and corresponding points between images having mutually overlapping regions accumulated in the frame memory 15 20
And a three-dimensional position measuring means 2 for calculating a three-dimensional position of each feature point from outputs of the motion detecting means 26 and the correspondence detecting means 20
2 and a plane calculating means 2 for calculating information of a plane to which the three-dimensional position of the feature point output from the three-dimensional position measuring means 22 is suitable
3 and the translation component and the expansion coefficient of the three-dimensional position obtained from each image pair based on the three-dimensional position of each feature point output from the three-dimensional position measurement unit 22 and the plane information output from the plane calculation unit 23. Expansion coefficient calculating means 24 for calculating
There is also provided a projection unit 25 for projecting each image stored in the frame memory 15 onto an arbitrary same image plane based on outputs supplied from the motion detection unit 26, the plane calculation unit 23, and the enlargement coefficient calculation unit 24. . Further, an imaging unit 11, a signal processing unit 12, a memory control unit 13, a main control unit 1
4, frame memory 15, I / F 16, display means 17,
External storage means 18, correspondence detection means 20, three-dimensional position measurement means 22, plane calculation means 23, enlargement coefficient calculation means 24,
The operation of the projection means 25 is the same as that of the first embodiment, and the description is omitted. Here, the operation of the motion detecting means 26 will be described.

The motion detecting means 26 detects the translational motion component and the posture of the image input device 1 between the time of photographing the reference image and the time of photographing the reference image, and includes, for example, an acceleration sensor, an angular velocity sensor, a magnetic sensor, A sound wave sensor, an infrared sensor, or the like can be configured alone or in combination. When the acceleration sensor is used, the inclination of the image input device 1 can be known by detecting the direction of the gravitational acceleration as described above, and the acceleration component generated in the process of moving the image input device 1 between the viewpoints. By integrating, the translational motion component can be calculated. When using a magnetic sensor, set the geomagnetism to 3 as described above.
The posture can be detected by detecting in the axial direction and calculating the azimuth of the image input device 1. In addition, the position and orientation of the image input device 1 can be simultaneously detected from the strength and direction of the artificially generated magnetic field. When an ultrasonic sensor is used, for example, a transmitter that generates ultrasonic waves in an environment where the image input device 1 is used, and an ultrasonic sensor is installed on the device side, and the strength and phase difference of received ultrasonic waves are set. Thus, the position and orientation of the image input device 1 can be detected.
When an infrared sensor is used, for example, the image input device 1
A device for irradiating an infrared spotlight into an environment using a light emitting device, and a PSD (position sensitive)
A photoelectric conversion device such as a detector or a CCD is installed, and the reflected light of the infrared spot light applied to the image input device 1 is detected by the photoelectric conversion device, so that the distance to the image input device 1 is determined by the principle of triangulation. Can be calculated. By detecting this reflected light at three or more points, the position and orientation can be detected.

The three-dimensional position of each feature point is calculated by the three-dimensional position measuring means 22 based on the translational motion component of the image input device 1 detected by the above processing, and the feature points and corresponding points. Plane calculating means 23 using
Has information on the size as well as the direction of the translational motion, so that the problem of the unknown expansion coefficient described above has been solved. However, there is a possibility that an error is included in the translational motion component output from the motion detecting means 26 due to disturbance, sensor noise, or the like existing in the environment in which the environment is used. Therefore, the expansion coefficient calculating means 24 calculates the expansion coefficient. The calculated enlargement coefficient, plane equation, and the image stored in the frame memory 15 are sent to the projection unit 25, and a composite image is created by the method described above.

Next, the operation of the image input apparatus according to the second embodiment will be described with reference to FIG. First, the input target plane is projected by the imaging means 11, and this image is stored as a reference image, and at the same time, the position and orientation of the image input device 1 at the time of shooting are detected (step S201). continue,
The viewpoint and the attitude of the image input device 1 are changed so that a part of the image input device 1 overlaps with the reference image, and an image of the input target plane is captured by the image capturing means 11, and this image is stored as a reference image. Is detected (step S202). A feature point is extracted from the reference image, and a corresponding point indicating the same location as the feature point is detected in the reference image (step S203). Thus the obtained feature points obtained by the corresponding point (15) and (16) line-of-sight vector P _s relative to the world coordinate system using equation converts each P _r, the image at each image shooting Using the position and orientation of the input device 1, the three-dimensional position of each feature point is calculated based on the principle of triangulation using the viewpoint at the time of capturing the reference image as the origin (step S204). After the above processing, it is determined whether plane information has already been calculated (step S205). If it has not been calculated, a plane equation is calculated (step S206). If a plane equation already exists, it is calculated. An enlargement coefficient is calculated so that the three-dimensional position corresponds to the plane equation (step S20).
7) Based on the plane equation and the enlargement coefficient, the reference image and the reference image are projected onto a certain same image plane Im by the method described above to create a composite image (step S208). Thereafter, when the number of images to be captured and synthesized is further increased, the image used as the reference image in the above processing is replaced with the reference image, and steps S202 to S20 are performed.
8 is repeated, and if not, the process is terminated (steps S209 and S210).

The above description is only an example of the present invention. For example, although it has been described that a process of combining images is performed each time one image is captured, a combined image may be created by batch processing after all images have been captured. Further, it has been described that the three-dimensional position in each image is applied to the plane calculated by using the image 1 as the reference image and the image 2 as the reference image. However, the plane may be calculated by selecting an arbitrary image pair having an overlapping area with each other. I do not care.

FIG. 16 is a block diagram showing the configuration of an image input apparatus according to a third embodiment of the present invention.
9 is a flowchart showing the operation of the third embodiment. In the above-described first and second embodiments, a plane equation is calculated using one image pair as a reference image and the other as a reference image, and the three-dimensional position and translational motion component obtained from the other image pairs are calculated. Although it has been described that the enlargement coefficient is calculated sequentially, a plane equation may be calculated for each of all the image pairs, and the normal vector of the plane may be optimized based on these plane equations. For example, as shown in FIG. 16, the image input device 1 includes an imaging unit 11, a signal processing unit 12 that processes an image signal obtained by the imaging unit 11, and a memory control unit to which an output of the signal processing unit 12 is supplied. 13, a main control means 14 and an I / F 16, a frame memory 15 for storing image signals in accordance with instructions from the memory control means 13,
Display means 17 for displaying an image signal via F16;
An external storage unit 18 that reads / writes various signals including an image signal via the I / F 16 is provided. In addition, a posture detecting means 19 for detecting the posture of the image input device 1 at the time of image capturing, a correspondence detecting means 20 for extracting feature points and corresponding points between images having mutually overlapping areas accumulated in the frame memory 15 and , Posture detecting means 19
And a translational motion detecting means 21 for detecting the translational motion of the image input device 1 from the output of the correspondence detecting means 20, and the three-dimensional position of each characteristic point from the outputs of the attitude detecting means 19, the correspondence detecting means 20 and the translational motion detecting means 21. , A plane calculating unit 23 that calculates information of a plane to which a three-dimensional position of a feature point output by the three-dimensional position measuring unit 22 is suitable, and a plurality of planes output by the plane calculating unit 23. The plane optimizing means 27 for calculating an optimal plane equation based on the information on the plane, and the three-dimensional position of each feature point output from the three-dimensional position measuring means 22 and the plane information output from the plane calculating means 23 Magnification factor calculating means 24 for calculating a translation component and a three-dimensional position expansion coefficient obtained from each image pair based on each image pair, posture detecting means 19, translational motion detecting means 2
1. There is also provided a projecting unit 25 for projecting each image stored in the frame memory 15 onto an arbitrary image plane based on the outputs supplied from the plane calculating unit 23 and the enlargement coefficient calculating unit 24. Imaging means 11, signal processing means 12, memory control means 13, main control means 14, frame memory 1
5, I / F 16, display means 17, external storage means 18, attitude detection means 19, correspondence detection means 20, translational movement detection means 21, three-dimensional position measurement means 22, plane calculation means 23, enlargement coefficient calculation means 24, and The operation of the projection means 25 is the first
The description is omitted because it is the same as that of the embodiment. Here, the operation of the plane optimizing means 27 will be described.

As shown in FIG. 1, if the number of captured images is K, (K-1) sets of image pairs are obtained, and the plane calculating means 23 calculates (K-1) plane equations. It is possible. Assuming that the normal vectors of these plane equations are n _j (j = 1,..., K−1), n _j is based on the coordinate system shown in FIG. The plane optimizing unit 27 outputs the normal vector n of the fitted plane equation by averaging n _j , for example.

Using the normal vector n output by the above processing, the expansion coefficient calculation means 24 calculates the expansion coefficient by using the calculation method described in the first embodiment. The calculated enlargement coefficient, plane equation, and the image stored in the frame memory 15 are sent to the projection unit 25, and a composite image is created by the method described above.

Next, the operation of the image input apparatus according to the third embodiment will be described with reference to FIG. First, the input target plane is projected by the imaging means 11, and this image is stored as a reference image, and at the same time, the attitude of the image input device 1 at the time of shooting is detected (step S301). Then, the viewpoint and the attitude of the image input device 1 are changed so that a part of the image is overlapped with the reference image, the input target plane is photographed by the imaging means 11, and this image is stored as a reference image. The position and orientation of the input device 1 are detected (step S302). A feature point is extracted from the reference image, and a corresponding point indicating the same location as the feature point is detected in the reference image (step S303). Thus the obtained feature points obtained by the corresponding point (15) and (16) line-of-sight vector P _s relative to the world coordinate system using equation converts each P _r, the image at each image shooting Using the attitude of the input device 1, a translational motion vector t of the image input device 1 is obtained by Expression (18) (Step S304). Its translation vector t and a line-of-sight vector P _s, by using the P _r, the viewpoint at the time of the reference image photographing as the origin, calculating the three-dimensional position of each feature point by the principle of triangulation (Step S30
5). A plane equation is calculated from the three-dimensional position obtained after the above processing (step S306). When the number of images to be captured and synthesized is further increased, the image used as the reference image in the above processing is replaced with the reference image, and the operations from step S302 to step S306 are repeated. The photographing ends (steps S307 and S308). After the photographing, the plane normal vector n is optimized based on the plane equation obtained from each image pair to determine the plane equation (step S).
309). When n is obtained, an expansion coefficient is calculated so that the three-dimensional position obtained from each image pair is applied to the plane equation (step S310), and the reference image is obtained based on the plane equation and the expansion coefficient by the method described above. and the reference image projected on the same image plane I _m in the, to create a composite image (step S311).

The above description is only an example of the present invention. For example, although it has been described that the normal vector of the plane obtained from each image pair is averaged, other normal vector optimization processing may be used.

The present invention is not limited to the above embodiments, and needless to say, various modifications and substitutions can be made within the scope of the claims.

[0101]

As described above, according to the present invention,
Imaging means for imaging the target plane from a plurality of viewpoints so that a part of the reference image is a previously captured image partially overlaps, a posture detection means for detecting a posture of the imaging means at the time of each imaging, a reference image, Based on an image pair with a reference image which is an image having a portion overlapping with the reference image, a plurality of feature points in the reference image are extracted, and a corresponding point indicating the same point as the feature point in the reference image is detected. Detecting means, translational motion detecting means for detecting a translational motion component of the imaging means based on the attitude, feature points, and corresponding points of the imaging means, and based on the attitude, feature points, corresponding points, and translational motion components of the imaging means. Three-dimensional position measuring means for calculating the three-dimensional position of each feature point;
It is assumed that the three-dimensional position of each feature point measured by the three-dimensional position measurement means is on the same plane, and plane calculation means for calculating information of a plane conforming to the three-dimensional position of each feature point, and each image pair is used. Coefficient calculating means for calculating the translation component and the expansion coefficient of each three-dimensional position based on the three-dimensional position of each feature point obtained by the above and the plane information calculated by the plane calculating means; It is characterized by having projection means for projecting each image on an arbitrary same image plane based on the translation component and the information on the plane calculated by the plane calculation means, and synthesizing images captured from a plurality of viewpoints.

As still another invention, an image pickup means for picking up an object plane from a plurality of viewpoints so that a part of the image is overlapped with a reference image which is a previously picked-up image, and an attitude and a translation component of the image pickup means at each image pickup. And extracting a plurality of feature points in the reference image based on the image pair of the reference image and the reference image which is an image having a portion overlapping with the reference image, and extracting a plurality of feature points identical to the feature points in the reference image. Correspondence detection means for detecting a corresponding point indicating the position of the image, three-dimensional position measurement means for calculating the three-dimensional position of each feature point based on the attitude and translation component of the imaging means, and the feature point and the corresponding point; A plane calculating unit for calculating information of a plane conforming to the three-dimensional position of each feature point, assuming that the three-dimensional positions of each feature point measured by the three-dimensional position measuring unit are on the same plane, and using each image pair. Required features Magnification factor calculating means for calculating the translation component and the magnification factor of each three-dimensional position based on the three-dimensional position and the plane information calculated by the plane calculating means, the attitude and the translation component of the imaging means, and the plane calculation A projection unit for projecting each image onto an arbitrary image plane based on the information on the plane calculated by the unit, and synthesizing images captured from a plurality of viewpoints.

Therefore, according to the above two inventions, even when a large number of target planes at arbitrary distances are photographed, plane information can be calculated with high accuracy by a simple calculation, and a beautiful composite image is obtained. be able to. Further, since the overlapping range of the images is not preliminarily forced, the user can photograph the target plane under relatively free conditions. Furthermore, since it is not necessary for the user to manually set the order of the captured images, it is possible to combine images with high operability and usability.

As another invention, an image pickup means for picking up an object plane from a plurality of viewpoints so as to partially overlap a reference image which is a previously picked-up image, and an attitude of the image pickup means at each image pickup are detected. Based on an image pair of a posture detection unit, a reference image, and a reference image which is an image having a portion overlapping with the reference image, a plurality of feature points in the reference image are extracted and the same as the feature points in the reference image. Correspondence detection means for detecting a corresponding point indicating a location, translational motion detection means for detecting a translational motion component of the imaging means based on the attitude, characteristic points and corresponding points of the imaging means; The three-dimensional position measuring means for calculating the three-dimensional position of each feature point based on the corresponding point and the translational motion component, and the three-dimensional position of each feature point measured by the three-dimensional position measuring means on the same plane Each feature point A plane calculating means for calculating information on a plane conforming to the three-dimensional position, a plane optimizing means for optimizing plane information on the target plane based on the information on the plane calculated by the plane calculating means, and each image pair is used. Enlargement coefficient calculating means for calculating each translation component and an enlargement coefficient of each three-dimensional position based on the three-dimensional position of each feature point obtained as described above and information on the plane optimized by the plane optimizing means, And projection means for projecting each image onto an arbitrary same image plane and synthesizing images captured from a plurality of viewpoints, based on the posture and translational components of the image, and information on the plane optimized by the plane optimization means. There are features.

As still another invention, an image pickup means for picking up an object plane from a plurality of viewpoints so that a part of the image is overlapped with a reference image which has been picked up earlier, and an attitude and a translation component of the image pickup means at each image pickup. And extracting a plurality of feature points in the reference image based on the image pair of the reference image and the reference image which is an image having a portion overlapping with the reference image, and extracting a plurality of feature points identical to the feature points in the reference image. Correspondence detection means for detecting a corresponding point indicating the position of the image, three-dimensional position measurement means for calculating the three-dimensional position of each feature point based on the attitude and translation component of the imaging means, and the feature point and the corresponding point; Plane calculating means for calculating information on a plane conforming to the three-dimensional position of each feature point, assuming that the three-dimensional position of each feature point measured by the three-dimensional position measuring means is on the same plane, and a plane calculated by the plane calculating means Feelings Plane optimization means for optimizing plane information of the target plane based on the three-dimensional position of each feature point obtained by using each image pair, and information on the plane optimized by the plane optimization means. A magnification factor calculating means for calculating a magnification factor of each translational component and each three-dimensional position; and an image of each image can be arbitrarily determined based on the orientation of the imaging means, the translational component, and the plane information optimized by the plane optimizing means. And projection means for projecting images onto the same image plane and combining images captured from a plurality of viewpoints.

Therefore, according to the above two inventions, an accurate plane equation can be calculated even when an error is included in the posture or motion detection result of the image input device. As a result, a more beautiful composite image can be created.

[Brief description of the drawings]

FIG. 1 is a diagram showing a state of photographing using an image input device of the present invention.

FIG. 2 is a block diagram illustrating a configuration of an image input device according to a first embodiment of the present invention.

FIG. 3 is a block diagram illustrating a configuration of a posture detection unit in FIG. 2;

FIG. 4 is a diagram showing a device coordinate system.

FIG. 5 is a diagram showing a world coordinate system.

FIG. 6 is a block diagram illustrating a configuration of a correspondence detection unit in FIG. 2;

FIG. 7 is a diagram illustrating a state of matching between a reference image and a reference image by a correlation operation.

FIG. 8 is a diagram illustrating a line-of-sight vector of a reference image and a reference image with respect to a target plane.

FIG. 9 is a diagram showing translational motion vectors in translational motion between viewpoints.

FIG. 10 is a diagram showing a three-dimensional position based on a viewpoint 2;

FIG. 11 is a diagram illustrating an example in which each image is projected on the same plane.

FIG. 12 is a diagram illustrating a state of image synthesis.

FIG. 13 is a flowchart showing the operation of the first embodiment.

FIG. 14 is a block diagram illustrating a configuration of an image input device according to a second embodiment of the present invention.

FIG. 15 is a flowchart showing the operation of the second embodiment.

FIG. 16 is a block diagram illustrating a configuration of an image input device according to a third embodiment of the present invention.

FIG. 17 is a flowchart showing the operation of the third embodiment.

[Explanation of symbols]

 DESCRIPTION OF SYMBOLS 11 Imaging means 12 Signal processing means 13 Memory control means 14 Main control means 15 Frame memory 16 Interface 17 Display means 18 External storage means 19 Attitude detection means 20 Correspondence detection means 21 Translational motion detection means 22 Three-dimensional position measurement means 23 Plane calculation means 24 Enlargement coefficient calculation means 25 Projection means 26 Motion detection means 27 Planar optimization means

 ────────────────────────────────────────────────── ─── Continued on the front page F term (reference) 5B057 CA13 CB12 CE10 DA07 DB03 DC05 DC08 5C023 AA11 AA31 AA36 AA37 AA38 BA01 BA11 CA01 DA04

Claims (8)

[Claims] An image pickup means for picking up an object plane from a plurality of viewpoints such that a part of the image is overlapped with a reference image which is an image picked up earlier, and an attitude detecting means for detecting an attitude of the image pickup means at each image pickup. And extracting a plurality of feature points in the reference image based on an image pair of the reference image and a reference image which is an image having a portion overlapping with the reference image, and indicating a same point as the feature point in the reference image. Correspondence detecting means for detecting a corresponding point, a posture of the imaging means, a translational motion detecting means for detecting a translational motion component of the imaging means based on the feature points and the corresponding points, an attitude, a feature point of the imaging means, A three-dimensional position measuring means for calculating a three-dimensional position of each feature point based on the corresponding point and the translational motion component; and a three-dimensional position of each feature point measured by the three-dimensional position measuring means being on the same plane. As each feature A plane calculating means for calculating information on a plane conforming to the three-dimensional position, and a three-dimensional position of each feature point obtained by using each image pair, and information on the plane calculated by the plane calculating means, Magnification factor calculating means for calculating a magnification factor of each translation component and each three-dimensional position; and arbitrarily identifying each image based on information on the orientation and the translation component of the imaging means and the plane calculated by the plane calculation means. An image input device comprising: projection means for projecting onto an image plane and combining images captured from a plurality of viewpoints. 2. An image pickup means for picking up an object plane from a plurality of viewpoints so as to partially overlap a reference image which is an image picked up earlier, and an attitude detecting means for detecting an attitude of the image pickup means at each image pickup. And extracting a plurality of feature points in the reference image based on an image pair of the reference image and a reference image which is an image having a portion overlapping with the reference image, and indicating a same point as the feature point in the reference image. Correspondence detecting means for detecting a corresponding point, a posture of the imaging means, a translational motion detecting means for detecting a translational motion component of the imaging means based on the feature points and the corresponding points, an attitude, a feature point of the imaging means, A three-dimensional position measuring means for calculating a three-dimensional position of each feature point based on the corresponding point and the translational motion component; and a three-dimensional position of each feature point measured by the three-dimensional position measuring means being on the same plane. As each feature Plane calculating means for calculating information on a plane conforming to the three-dimensional position, plane optimizing means for optimizing plane information on the target plane based on the information on the plane calculated by the plane calculating means, Expansion factor calculation means for calculating the translation component and the expansion coefficient of each three-dimensional position based on the three-dimensional position of each feature point obtained by using the information and the plane information optimized by the plane optimization means. A projection for projecting each image onto an arbitrary same image plane and synthesizing images captured from a plurality of viewpoints, based on the attitude and translation component of the imaging unit and information on a plane optimized by the plane optimizing unit; And an image input device. 3. A target plane is imaged from a plurality of viewpoints so that a part of the target plane is overlapped with a reference image which is an image previously picked up by an image pickup means, and a posture of the image pickup means at each image pickup is detected. And extracting a plurality of feature points in the reference image based on an image pair of the reference image, which is an image having a portion overlapping with the reference image, and indicating the same location as the feature point in the reference image. A step of detecting a point; a step of detecting a translational motion component of the imaging unit based on a posture, a feature point, and a corresponding point of the imaging unit; a posture, a feature point, a corresponding point, and a translational motion of the imaging unit. Calculating the three-dimensional position of each feature point based on the component, and calculating plane information that matches the three-dimensional position of each feature point assuming that the measured three-dimensional position of each feature point is on the same plane Process and each Three-dimensional positions of the feature points obtained using the image pair, and based on the information of the calculated plane, each translation component and the 3
Calculating the magnification factor of the dimensional position; and the orientation and translation component of the imaging unit, and based on the calculated plane information, each image is projected onto any same image plane and imaged from a plurality of viewpoints. Combining an image. 4. An object plane is imaged from a plurality of viewpoints so as to partially overlap a reference image which is an image previously imaged by using the image pickup means, and an attitude of the image pickup means at each image pickup is detected. And extracting a plurality of feature points in the reference image based on an image pair of the reference image, which is an image having a portion overlapping with the reference image, and indicating the same location as the feature point in the reference image. A step of detecting a point; a step of detecting a translational motion component of the imaging unit based on a posture, a feature point, and a corresponding point of the imaging unit; a posture, a feature point, a corresponding point, and a translational motion of the imaging unit. Calculating the three-dimensional position of each feature point based on the component, and calculating plane information that matches the three-dimensional position of each feature point assuming that the measured three-dimensional position of each feature point is on the same plane Process and each A step of optimizing plane information of the target plane based on information of each plane obtained from the image pair, a three-dimensional position of each feature point obtained by using each image pair, and Calculating the translational component and the expansion coefficient of each three-dimensional position based on the information; and transforming each image into an arbitrary identical image based on the orientation and the translational component of the imaging unit and the information on the optimized plane. Projecting the image on a surface and combining images captured from a plurality of viewpoints. 5. An image pickup means for picking up an object plane from a plurality of viewpoints so as to partially overlap a reference image which is an image picked up earlier, and detecting an attitude and a translation component of the image pickup means at each image pickup. A plurality of feature points in the reference image are extracted based on the motion detection unit and the image pair of the reference image and the reference image which is an image having a portion overlapping with the reference image; Correspondence detecting means for detecting the corresponding point indicated by the image capturing means; three-dimensional position measuring means for calculating the three-dimensional position of each feature point based on the attitude and translation component of the image capturing means; and the feature point and the corresponding point; Plane calculating means for calculating information on a plane conforming to the three-dimensional position of each feature point, assuming that the three-dimensional position of each feature point measured by the position measuring means is on the same plane; 3 of each feature point Expansion factor calculation means for calculating each translation component and an expansion coefficient of each three-dimensional position based on the original position and information on the plane calculated by the plane calculation means; and an orientation and a translation component of the imaging means, and the plane An image input apparatus comprising: a projection unit that projects each image onto an arbitrary same image plane based on information on a plane calculated by a calculation unit and combines images captured from a plurality of viewpoints. 6. An image pickup means for picking up an object plane from a plurality of viewpoints so as to partially overlap a reference image which is an image picked up earlier, and detecting an attitude and a translation component of the image pickup means at each image pickup. A plurality of feature points in the reference image are extracted based on the motion detection unit and the image pair of the reference image and the reference image which is an image having a portion overlapping with the reference image; Correspondence detecting means for detecting the corresponding point indicated by the image capturing means; three-dimensional position measuring means for calculating the three-dimensional position of each feature point based on the attitude and translation component of the image capturing means; and the feature point and the corresponding point; Plane calculating means for calculating information of a plane conforming to the three-dimensional position of each feature point, assuming that the three-dimensional position of each feature point measured by the position measuring means is on the same plane; and a plane calculated by the plane calculating means Information A plane optimizing unit for optimizing plane information of the target plane, a three-dimensional position of each feature point obtained by using each image pair, and information on the plane optimized by the plane optimizing unit. A magnification factor calculating means for calculating a magnification factor of each translational component and each three-dimensional position; each image based on information on a plane and a translational component of the imaging means, and a plane optimized by the plane optimizing means. And a projection unit for projecting the image on an arbitrary image plane and combining images captured from a plurality of viewpoints. 7. A target plane is imaged from a plurality of viewpoints so that a part of the target plane is overlapped with a reference image which is an image previously picked up by an image pickup means, and a posture and a translation component of the image pickup means at each image pickup. And extracting a plurality of feature points in the reference image based on an image pair of the reference image and the reference image which is an image having a portion overlapping with the reference image, and extracting the same portion as the feature point in the reference image. Detecting the corresponding points indicating the following: calculating the three-dimensional position of each feature point based on the attitude and translational component of the imaging means, the feature points and the corresponding points; and measuring the three-dimensional position of each measured feature point Calculating the information of a plane that conforms to the three-dimensional position of each feature point assuming that they are on the same plane, and the three-dimensional position of each feature point obtained using each image pair, and the Each translation based on information Minute and each 3
Calculating the magnification factor of the dimensional position, and the orientation and translation component of the imaging unit, based on the information of the calculated plane, to project each image to any same image plane,
Synthesizing images captured from a plurality of viewpoints. 8. A target plane is imaged from a plurality of viewpoints so that a part of the target plane is overlapped with a reference image which is an image previously imaged by using an image pickup means, and a posture and a translation component of the image pickup means at each image pickup. And extracting a plurality of feature points in the reference image based on an image pair of the reference image and the reference image which is an image having a portion overlapping with the reference image, and extracting the same portion as the feature point in the reference image. Detecting the corresponding points indicating the following: calculating the three-dimensional position of each feature point based on the attitude and translational component of the imaging means, the feature points and the corresponding points; and measuring the three-dimensional position of each measured feature point Calculating plane information that conforms to the three-dimensional position of each feature point assuming that they are on the same plane, and optimizing plane information of the target plane based on information of each plane obtained from each image pair Process and each drawing A step of calculating a translation component and an expansion coefficient of each three-dimensional position based on the three-dimensional position of each feature point obtained using the pair and information on the optimized plane; Projecting each image on an arbitrary same image plane based on the translation component and the information of the optimized plane, and synthesizing images captured from a plurality of viewpoints. .