JPH11298837A - Image input device and image input method - Google Patents

Abstract

PROBLEM TO BE SOLVED: To allow a photographer to easily confirm photographing conditions by informing the photographer of whether or not photographing is proper in the case of compositing panorama images. SOLUTION: A 1st image photographed by an image pickup means 2 is used for a criterion image and a 2nd image that is being photographed by the image pickup means 2 at present is used for a reference image. An overlap area detection means 13 detects an overlapped area between the criterion image and the reference image and displays the size of the overlapped area on an overlapped area display indicator of a finder 11. When a shutter of the image pickup means 2 is depressed, the overlap area detection means 13 lights a warning indicator of the finder 11 when the displayed size of the overlapped area is a preset threshold value or below to instruct revision of a photographing position to the photographer.

Description

DETAILED DESCRIPTION OF THE INVENTION

[0001]

BACKGROUND OF THE INVENTION 1. Field of the Invention The present invention relates to an image input apparatus and an image input method for synthesizing a plurality of still images photographed by an image pickup means such as a digital still camera.

[0002]

2. Description of the Related Art Cameras are widely used in which an image passing through an optical system is photoelectrically converted by a CCD sensor or the like, and the obtained image signal is A / D converted and stored in an external storage device such as a memory card. Since a camera using such a CCD sensor or the like can handle a captured image as a digital signal, there is an advantage that image processing, processing, and transmission can be easily performed. A high-resolution panoramic image is synthesized by pasting together images obtained by photographing a plurality of the same subject with such a camera. In order to combine the panoramic images, it is necessary to photograph the subject such that a part of the plurality of images overlaps or the plurality of images are connected seamlessly.

For example, in the method disclosed in Japanese Patent Laid-Open No. Hei 5-161050, an image that has been input and an image that is currently being input are thinned out and reduced, and an image is displayed on an adjacent portion of the display unit, and a panoramic image is displayed. When obtaining, it is possible to confirm whether the current imaging position is appropriate. In addition, the method disclosed in Japanese Patent Application Laid-Open No. 9-266561 displays a part of an image captured earlier in a state of being arranged on a currently input image in accordance with a moving direction of a camera, and sets a state in which both images are connected. I have. For example, when the camera is rotated clockwise, the right end portion of the previously captured image is cut out by a predetermined area and projected on the left end of the display unit provided on the back of the camera,
The image of the current input is displayed side by side right next to it. Then, when the shutter is pressed in a state where the two images are connected, the combined continuous image is stored in the panorama memory.

[0004]

SUMMARY OF THE INVENTION However, Japanese Patent Application Laid-Open
In the method disclosed in Japanese Patent Application Laid-Open No. 161050, an image that is already input and an image that is currently being input are thinned out to display a reduced image. Therefore, connections and overlapping areas between images are confirmed by the thinned low-resolution image. Therefore, it is difficult to confirm this, and especially when the number of images to be pasted increases and many images are divided, this problem cannot be ignored. In addition, in the method disclosed in Japanese Patent Application Laid-Open No. 9-266561, a panoramic image cannot be obtained unless two images are completely connected, so that the operation of the camera is severely restricted.

The present invention has been made to solve the above-mentioned disadvantages, and when performing various types of image processing and synthesis such as panoramic image synthesis and three-dimensional shape restoration of a subject, the user is notified whether or not the shooting operation is appropriate. It is an object of the present invention to provide an image input device and an image input method capable of easily confirming an image.

[0006]

According to the present invention, there is provided an image input apparatus comprising: an image input device for inputting an image of a subject; and an image storage device for storing an image obtained from the image input device. The image capturing means has an overlapping area detecting means for automatically detecting an overlapping area between the reference image stored in the image storing means and the image currently input by the means.

The overlapping area detecting means preferably detects an overlapping area with the reference image from a motion vector in the currently input moving image.

Further, the overlapping area detecting means may detect an overlapping area from a motion vector between the reference image and the image to be picked up.

According to another aspect of the present invention, there is provided an image input device for inputting an image of a subject, an attitude detecting sensor for obtaining a signal representing the attitude of the image capturing means, and an image for storing an image obtained from the image capturing means. An image input apparatus having an overlapping area detecting means for automatically detecting an overlapping area between a reference image previously captured by an image capturing means and stored in an image storing means and an image currently input by the image capturing means, The overlapping area detecting means detects an overlapping area between the reference image and the image currently being input by the imaging means based on the posture information obtained from the posture detecting sensor signal.

In the image input method according to the present invention, an image previously taken by the image pickup means is stored as a reference image, and an overlapping area between the stored reference image and the image currently being input by the image pickup means is detected to convert the image. It is characterized by combining.

The overlapping area between the reference image and the image currently being input by the image pickup means is detected from the motion vector of the moving image currently input by the image pickup means, or the overlap area between the reference image and the image to be picked up. It is preferable to detect from the motion vector and detect from the attitude information of the imaging means.

[0012]

DETAILED DESCRIPTION OF THE PREFERRED EMBODIMENTS An image input device according to the present invention has a CC
It has an imaging means having a photoconductive conversion element such as D, a mode setting means, a main control means, a signal processing means, a frame memory, an external storage device, a finder and an overlap area detecting means.

[0013] When the overlapping photographing mode is selected by the mode setting means of the image input apparatus, the main control means displays an indicator in the finder indicating that the overlapping photographing mode has been selected, and indicates that the overlapping photographing mode has been set. The overlap area detection means is notified and the overlap area detection means is activated.
In this state, when one image of the subject is photographed by the imaging means,
The main control means stores the captured first image in a frame memory as a reference image, and also stores the image in an external storage device. After the photographing, when a predetermined minute time elapses while shifting the position of the imaging means for photographing the subject, the main control means fetches the second image currently input by the imaging means into the frame memory as a reference image. When the currently input image is taken into the frame memory as a reference image, the overlapping area detecting means detects an overlapping area between the reference image stored in the frame memory as the first image and the currently input reference image. Then, the size of the overlapping area is displayed on the overlapping area display indicator of the finder. This process is repeated at regular intervals until the shutter of the imaging means is depressed. When the shutter of the imaging means is depressed, the overlapping area detecting means checks whether the size of the displayed overlapping area is equal to or larger than a threshold value which is set in advance and is displayed on the indicator of the finder. As a result of this check, if the size of the overlapping area is equal to or smaller than the threshold, the warning indicator is turned on to request the photographer to change the photographing position. In this way, it is possible to select an appropriate overlapping area when combining a plurality of images.

When the photographer changes the photographing position and the size of the overlap area between the reference image and the currently inputted reference image becomes greater than or equal to the threshold value and the shutter of the image pickup means is depressed, the current image is inputted. The second image is stored in an external storage device, and the reference image stored in the frame memory is replaced with the second image.
This process is repeated while photographing is continued, and a plurality of images are synthesized in an appropriate overlapping area.

[0015]

FIG. 1 is a block diagram showing the configuration of an image input apparatus according to one embodiment of the present invention. The image input device 1 includes an imaging unit 2, a mode setting unit 3, a main control unit 4, a signal processing unit 5, a memory control unit 6, a frame memory 7, an external storage interface 8, an external storage device 9, a display interface 10, a finder 11, For example, the image input device 1 includes an external display device 12 made of liquid crystal and an overlapping area detection unit 13 installed in a housing of the image input device 1.

The imaging means 2 has a photoconductive conversion element such as a CCD on the optical path converged by the lens unit, and outputs an analog video signal representing a subject image. The mode setting means 3 is for setting a photographing mode of the image input apparatus 1, and is provided on, for example, a housing surface of the image input apparatus 1 so that a user can select a desired photographing mode. The mode setting means 3 also switches on / off of the overlapping photographing mode. The main control means 4 controls the basic processing such as shooting, display and recording, receives the set value of the shooting mode from the mode setting means 3, and controls the signal processing means 5, the memory control means 6 and the overlap area detecting means 13. And so on. The signal processing means 5 includes the imaging means 2
A / D conversion is performed on the analog video signal output from the A / D converter by performing preprocessing such as amplification and clamping, and then various image processing such as filtering and color component decomposition is performed. The memory control means 6 stores the image signal processed by the signal processing means 5 in the frame memory 7 and reads out the image signal stored in the frame memory 7. The frame memory 7 is a VRAM or SRA
M, a DRAM, and the like, in which an image previously captured by the imaging means 2 and an image currently captured and input are stored. The image signals stored in the frame memory 7 are stored in an external storage device 9.
When the image signal is stored in the external memory, the image signal stored in the frame memory 7 is read by the memory control means 6, and the read image signal is subjected to signal processing such as image compression or the like. The data is stored in the external storage device 9 via the interface 8. As the external storage device 9, an IC memory card, a magneto-optical disk, or the like is used, but an image signal may be directly transmitted to a remote recording medium via a network using a modem card or an ISDN card. . When reading an image signal stored in the external storage device 9, the image signal stored in the external storage device 9 is sent to the signal processing unit 5 via the external storage interface 8. The signal processing means 5 performs image expansion of the transmitted image signal. The image signal read from the external storage device 9 or the frame memory 7 is subjected to signal processing such as D / A conversion and amplification by the signal processing means 5, and then is transmitted via the display interface 10 to the finder 11 and the external display device 12. To display.
The overlapping area detecting means 12 detects the size and the position of the overlapping area between the previously captured image stored in the frame memory 7 and the image currently captured and input by the imaging means 2.

The operation in the case where the overlap mode is selected by the mode setting means 3 in the image input apparatus 1 configured as described above will be described with reference to the flowchart of FIG. First,
When the overlapping shooting mode is selected by the mode setting means 3, FIG.
As shown in the figure, the main control means 22 includes an indicator 11 in the viewfinder 11 indicating that the overlapping photographing mode has been selected.
1 is displayed to notify the overlapping area detecting means 13 that the overlapping shooting mode has been set (step S1). The overlap region detecting means 20 is activated by the notification of the overlapping shooting mode. When one image of the subject is photographed by the imaging means 2 in this state, the main control means 4 stores the captured first image 21a in the frame memory 7 and the external storage device 9 as a reference image as shown in FIG. (Step S2). After the photographing, when a predetermined minute time elapses while shifting the position of the imaging means 2 for photographing the subject, the main control means 4 stores the second image 21b currently input by the imaging means 2 in the frame memory 7 as a reference image. Import (step S3). The currently input image 21b is used as the reference image in the frame memory 7
The overlapping area detecting means 13 detects an overlapping area between the reference image stored in the frame memory 7 as the first image and the currently input reference image, and determines the size of the overlapping area in the finder. 11 is displayed on the overlapping area display indicator 112 (step S4). This process is repeated at regular intervals until the shutter of the imaging means 2 is depressed (step S5). When the shutter of the imaging means 2 is depressed, the overlap area detecting means 13 sets the size of the displayed overlap area in advance and sets
It is checked whether it is equal to or more than the threshold value displayed by the indicator 113 of step 1 (step S6). If the size of the overlap area is equal to or smaller than the threshold value as a result of the check, the warning indicator 114 is turned on to request the photographer to change the photographing position (step S7). Then, when the photographer changes the photographing position and the size of the overlapping area between the reference image and the currently input reference image becomes equal to or larger than the threshold value and the shutter of the imaging unit 2 is pressed down, the image that is currently input is obtained. And the second image 21b taken is stored in the external storage device 9,
The reference image stored in the frame memory 7 is replaced with the second captured image 21b (steps S8 and S9). After that, the process of taking a third image starts (step S1).
0, S3). This process is repeated while shooting is continued.

When a subject is continuously photographed and a panoramic image is synthesized in this way, during the period from the time of the previous shutter operation to the time when the next shutter is depressed, the photographer can determine the size of the overlap between the images to be photographed. Can be checked.

In the above embodiment, the case where the previously shot image is used as the reference image has been described. However, the overlap may be detected using the first shot image or an arbitrary image shot so far as the reference image. Also, a case has been described in which a reference image is input at regular time intervals to detect an overlapping area, but, for example, a reference image is input immediately before a next image is captured, such as when a shutter is half-pressed, and an overlapping area is detected. You may make it detect. Further, in an image input device specialized in image synthesis and restoration of a three-dimensional shape of a subject, it is not necessary to set an imaging mode.

In the above embodiment, the overlapping area detecting means 1
Although the case where the overlap area between the currently input image and the previously captured image is detected in 3 has been described, the image capturing unit 2 uses the motion vector of the moving image being input as the reference vector and the currently input reference image. An overlap area can also be determined.
In this case, the overlapping area detecting means 13 includes a feature point setting means 131, a motion vector calculating means 132, and an adding means 133, as shown in the block diagram of FIG. Then, the previously captured image is stored and stored in the frame memory 7 as a reference image, and the image signals of the two latest temporally continuous frames of the image currently input to the imaging unit 2 are stored in the frame memory 7. . Here, of the currently input images, images of the latest two frames that are temporally continuous are image (t-1) and image t, respectively, in ascending order of time.
Call. The image (t-1) and the image t are updated at regular time intervals.

As shown in FIG. 6, the feature point setting means 131 determines the positions of the feature points x _i0 and y _i0 on the screen in advance in the image (t-1), and sets the feature points Pi (x _i0 and y
_The correlation window 30 is created by extracting the (2N + 1) and (2P + 1) light and shade patterns centered on ( _i0 ). The motion vector calculation means 132 calculates a motion vector on the image between the image (t-1) and the image t. For example, the motion vector between the image (t-1) and the image t is a point at which the image t substantially coincides with the shading pattern of the correlation window 30 created by the image (t-1) (hereinafter referred to as a corresponding point). It is done by asking for. Here, an example in which a corresponding point is detected by a matching block based on a correlation operation will be described. As shown in FIG. 7, (2N + 1), (2P + 1)
The i-th feature point Pi (x _i0 , y _i0 ) in the image (t-1) by block matching using the correlation windows 30 and 31
Points in the image _{t Qi (x i0 + dx i} , y i0 + dy i)
Are performed, the cross-correlation value S _i is calculated by the following equation (1).

[0022]

(Equation 1)

[0023] In (1), I _t-1 (x, y) is the concentration at the point (x, y) of the image _{(t-1), I t} (x,
y) is the density at point (x, y) of image t, I
_t-1 (x, y) mean concentration in the correlation window 30 of the bar point in the image (t-1) (x, y) around the (2N + 1), (2P + 1), I t (x, y) bar Is the average density in the (2N + 1) and (2P + 1) correlation windows 31 centered on the point (x, y) of the image t.

A corresponding point Qi for which the maximum value of the cross-correlation value S _i is equal to or greater than a predetermined threshold value for each feature point P _i .
By successively obtaining the _{(x i0 + dx i, y} i0 + dy i), the motion vector v _i at the point _{P i = (dx i, dy} i)
Is required. When the maximum value of the cross-correlation value S _i is equal to or smaller than the threshold, it is determined that no corresponding point exists. The entire motion vector v _{t−1, t} is obtained by averaging the motion vector v _t of each feature vertex P _i , for example, as shown in the following equation (2). The adding means 133 adds the motion vectors v _{t−1 and t} calculated by the motion vector calculating means 132 as shown in the following equation (3), and calculates a motion vector v from the reference image to the next image to be shot. Is calculated.

[0025]

(Equation 2)

When the current motion vector v is obtained, the size of the overlapping area is displayed on the indicator 112 in the finder 11.

The operation when the overlap mode is selected by the mode setting means 3 in the image input apparatus 1 configured as described above will be described with reference to the flowchart of FIG. When the overlapping shooting mode is selected by the mode setting means 3, the main control means 22 displays an indicator 111 in the finder 11 indicating that the overlapping shooting mode has been selected, and indicates that the overlapping shooting mode has been set. 13 and activates the overlapping area detecting means 20 (step S1).
1). When one image of the subject is photographed by the imaging means 2 in this state, the main control means 4 stores the captured first image in the frame memory 7 and the external storage device 9 as a reference image. When the first image is stored in the frame memory 7 as a reference image, the time when the reference image was taken is represented by t.
= 1 and the motion vector v from the reference image is initialized to a zero vector (step S12). After the first image is captured, when the position of the imaging unit 2 for imaging the subject is moved, the main control unit 4 stores the moving image currently being input by the imaging unit 2 in the frame memory 7 at regular intervals. take in. When a moving image is fetched into the frame memory 7, the time is set as t, and the time at which the previous image is fetched is (t-
As 1), the two images currently input by the imaging unit 2 are taken into the frame memory 7 as reference images (step S1).
3). Feature point setting means 131 of overlapping area detecting means 13
Creates a correlation window 30 by setting a plurality of feature points in the image taken at the previous time (t-1) (step S14). The motion vector calculation means 132 calculates the time (t
-1) A motion vector v _{t-1, t} between the reference image at time t and the reference image at time t is obtained, and an overlap area is detected by sequentially adding the obtained motion vectors v _{t-1, t} , The size of the overlapping area is displayed on the overlapping area display indicator 112 of the finder 11 (step S15). This process is repeated at regular intervals until the shutter of the imaging means 2 is depressed (step S16). When the shutter of the imaging means 2 is depressed, the overlap area detecting means 13 sets the size of the displayed overlap area in advance and sets
It is checked whether the threshold value is equal to or greater than the threshold value displayed by the indicator 113 (step S17). As a result of this check, if the size of the overlapping area is equal to or smaller than the threshold, the warning indicator 114 is turned on to request the photographer to change the photographing position (step S18). Then, when the photographer changes the photographing position and the size of the overlapping area between the reference image and the currently input reference image becomes equal to or larger than the threshold value and the shutter of the imaging unit 2 is pressed down, the image that is currently input is obtained. Is stored in the external storage device 9,
The reference image stored in the frame memory 7 is replaced with the second captured image (steps S19 and S20). After that, the photographing process of the third image is started (step S21,
S3). This process is repeated while shooting is continued.

In each of the above embodiments, when the overlap mode is selected by the mode setting means 3, a moving image input to the image pickup means 2 is taken at predetermined time intervals after taking the first image. The case where the overlap region with the reference image is detected as the reference image has been described. However, the overlap region is directly detected from the motion vector between the image to be photographed by pressing the shutter of the imaging unit 2 half and the reference image. You may do it. When the overlap area between the image to be photographed by half-pressing the shutter and the reference image is detected in this way, during the period from the start of half-pressing the shutter to the next press of the shutter, the photographer determines the distance between the images to be photographed. Can be checked at any time. In this case, as shown in FIG.
It has a feature point setting means 131 and a motion vector calculation means 132. As shown in FIG. 6, the feature point setting unit 131 sets the feature point P in the reference image which is the image captured immediately before.
(2N + 1), (2P +) centered on i (x _i0 , y _i0 )
The correlation window 30 is created by extracting the shading pattern of 1).
The motion vector calculation means 132 calculates a motion vector between the reference image, which is the image currently being input, and the reference image.

The operation when the mode setting means 3 selects the overlapping mode in the image input apparatus 1 configured as described above will be described with reference to the flowchart of FIG. When the overlapping shooting mode is selected by the mode setting means 3, the main control means 22 displays an indicator 111 in the finder 11 indicating that the overlapping shooting mode has been selected, and indicates that the overlapping shooting mode has been set. 13 and activates the overlapping area detecting means 20 (step S3).
1). When one image of the subject is photographed by the imaging means 2 in this state, the main control means 4 stores the captured first image in the frame memory 7 and the external storage device 9 as a reference image (step S32). The feature point setting means 131 of the overlapping area detecting means 13 sets a plurality of feature points in the reference image stored in the frame memory 7 and creates a correlation window (step S33). After the first image is captured, when the shutter of the imaging unit 2 is half-pressed while moving the position of the imaging unit 2 for capturing the subject, the main control unit 4 is currently moving by the imaging unit 2. The image is taken into the frame memory 7 as a reference image (step S34). The motion vector calculation means 132 of the overlap area detection means 13 calculates a motion vector between the reference image and the reference image, detects the overlap area, and displays the size of the overlap area on the overlap area display indicator 112 of the finder 11. (Step S
35). This process is repeated at regular time intervals until the shutter of the imaging means 2 is depressed (steps S36, S3
4). When the shutter of the imaging means 2 is depressed, the overlapping area detecting means 13 sets the size of the overlapping area of the reference image and the reference image at that time in advance, and
It is checked whether it is equal to or larger than the threshold value displayed by the indicator 113 of step 1 (steps S36 and S37). As a result of this check, if the size of the overlapping area is equal to or smaller than the threshold, the warning indicator 114 is turned on to request the photographer to change the photographing position (step S38). Then, when the photographer changes the photographing position and the size of the overlapping area between the reference image and the currently inputted reference image becomes equal to or larger than the threshold value, the photographer photographs the currently inputted image, and takes the second photographed image. The image is stored in the external storage device 9, and the reference image stored in the frame memory 7 is replaced with the second captured image (Step S).
36, S37-S40). After that, the photographing process of the third image is started (steps S41 and S33). This process is repeated while shooting is continued.

In the above embodiment, the case where the overlapping area between the reference image and the currently input reference image is obtained from the motion vector of the image being inputted by the image pickup means 2 has been described. An overlapping area between the reference image and the currently input reference image may be detected from the posture.

In this case, as shown in the block diagram of FIG. 11, the image input device 1 includes an imaging means 2, a mode setting means 3, a main control means 4, a signal processing means 5, and a memory control means 6.
, Frame memory 7, external storage interface 8, external storage means 9, display interface 10, and finder 11
And an attitude detection sensor 14 provided in the imaging means 2 in addition to the external display means 12 and the overlap area detection means 13.
As shown in the block diagram of FIG. 12, the overlapping area detecting means 13 includes a posture calculating means 134 and a relative posture calculating means 135.
And motion vector calculating means 136 and buffer memory 13
Seven. The posture calculation means 134 is provided by the posture detection sensor 14.
Then, the attitude of the imaging means 2 at the time of shooting is calculated from the signal from the camera and stored in the buffer memory 137. Relative attitude calculation means 13
Numeral 5 calculates the relative attitude between the two at the time of the previous shooting, that is, the shooting of the reference image stored in the buffer memory 137, and the attitude at the time of inputting the current reference image output from the attitude calculating means 134. The motion vector calculation means 136 calculates a motion vector between the reference image and the reference image based on the relative attitude between the two images.

The posture detecting sensor 14 detects, for example, a three-axis acceleration sensor for detecting gravitational acceleration in three axes orthogonal to each other, a three-axis magnetic sensor for detecting geomagnetism in three axes, or a gravitational acceleration in three axes orthogonal to each other. It comprises a three-axis acceleration sensor for detecting and a three-axis angular velocity sensor for detecting angular velocities around three axes. Thus, the posture detection sensor 14 is
In the case of a three-axis magnetic sensor and a three-axis magnetic sensor,
When the imaging means 2 is stationary or when the acceleration is moved at such a speed that the acceleration can be ignored, the posture can be detected. Also, the posture detection sensor 14 is a three-axis acceleration sensor and a three-axis acceleration sensor.
In the case of using an axial angular velocity sensor, the attitude can be detected when the imaging means 2 is moved at such a speed that the angular velocity can be detected.

The operation in the case where the attitude detection sensor 14 is constituted by a three-axis acceleration sensor and a three-axis magnetic sensor to detect an overlapping area between the reference image and the reference image will be described. First,
As shown in FIG. 13, the positive direction of the z-axis of the xyz coordinate system of the imaging unit 2 is the optical axis direction, the positive direction of the x-axis is the rightward direction with respect to the image plane 22, and the positive direction of the y-axis is the image plane. 14, the positive direction of the Y axis in the world coordinate system is the direction of gravitational acceleration, the positive direction of the Z axis is the direction of magnetism, and the positive direction of the X axis is XYZ, as shown in FIG. The directions are to form a right-handed orthogonal system in order. Also, for simplicity, it is assumed that the movement speed caused by the movement of the imaging means 2 can be ignored, the gravitational acceleration and the magnetic field are orthogonal, and the magnetic field does not exist except for geomagnetism. Strictly speaking, there is a magnetic dip, and the geomagnetism and the gravitational acceleration are not orthogonal. However, if the dip is known, it can be calculated in the same way as when the geomagnetic direction and the gravitational acceleration are orthogonal. Further, if geomagnetism is detected on three axes, posture information can be calculated even if the dip angle is unknown. That is, the positive direction of the X axis is set to the east, and the positive direction of the Z axis is set to the north. The attitude detection sensor 14 is installed so as to detect the x-, y-, and z-axis components of the gravitational acceleration and the x-, y-, and z-axis components of the geomagnetism.

The posture signal output from the posture detection sensor 14 is subjected to signal processing such as filtering and amplification and A / D conversion in the signal processing means 5 and then sent to the overlapping area detection means 13. The attitude calculating means 137 of the overlapping area detecting means 13 calculates the attitude of the imaging means 2 from the sent attitude signal. The attitude of the imaging means 2 is described by a rotation matrix of the following equations (4) and (5) with reference to the world coordinate system.

[0035]

(Equation 3)

Now, the gravitational acceleration vector g and the terrestrial magnetic vector M are represented by the following equation (6) in the world coordinate system, and the acceleration vector based on the apparatus coordinate system detected by the three-axis acceleration sensor and the three-axis magnetic sensor is used. a and the geomagnetic vector m are represented by the following equation (7).

[0037]

(Equation 4)

The relationship between the gravitational acceleration vector g and the geomagnetic vector M and the relationship between the acceleration vector a and the geomagnetic vector m are described by the following equations (8) and (9) using the rotation matrix R.

[0039]

(Equation 5)

From the equation (8), the rotation angle α around the X axis and the rotation angle γ around the Z axis are calculated by the following equations (10) and (11). Using the rotation angle γ about the Z axis, the rotation angle β about the Y axis from the geomagnetic vector m
Is calculated by the following equations (12) and (13).

[0041]

(Equation 6)

In this manner, the rotation angle α about the X axis, the rotation angle β about the Y axis, the rotation angle γ about the Z axis, and the rotation matrix R are calculated from the detected values of the three-axis acceleration sensor and the three-axis magnetic sensor. , The attitude of the imaging means 2 with respect to the world coordinate system.

[0043] The relative orientation calculation unit 135, the rotation matrix R _r with respect to the world coordinate system in the rotation matrix R _s and the reference image with respect to the world coordinate system in the reference image, it calculates the relative posture of both. For example, a rotation matrix from the imaging coordinate system of the reference image to the imaging coordinate system of the reference image is expressed by the following equation (14) based on the reference image.

[0044]

(Equation 7)

The motion vector calculation means 136 calculates a motion vector from the reference image to the reference image from the rotation matrix calculated as described above. Assuming that the optical system of the imaging means 2 is a central projection model having a focal length f as shown in FIG. 13, a line-of-sight vector p _s for a point (x, y) on the reference image is expressed by the following equation (15). Is represented by

[0046]

(Equation 8)

[0047] Here, when compared to the imaging position change between the reference image distance between the imaging unit 2 and the subject is the reference image sufficiently large, the subject image represented in the reference image (15) of p _s is In the reference image, the image is projected to the coordinates of the following equation (16).

[0048]

(Equation 9)

Here, k is a coefficient for setting the z component of the line-of-sight vector p _r to f. Therefore, the motion vector v from the reference image to the reference image can be obtained by the following equation (17).

[0050]

(Equation 10)

Here, I is a unit matrix. The motion vector v in the equation (17) changes depending on the coordinates (x, y) in the reference image, but the motion vector v is hardly affected by the coordinates (x, y) unless the viewing angle of the imaging unit 2 is so large. ,. For example, (x, y) is set at the center of the screen, that is, (0,
0) is substituted into the expression (17), and the motion vector v can be obtained by the following expression (18).

[0052]

[Equation 11]

The operation when the mode setting means 3 selects the overlap mode in the image input apparatus 1 configured as described above will be described with reference to the flowchart of FIG. When the overlapping shooting mode is selected by the mode setting means 3, the main control means 22 displays an indicator 111 in the finder 11 indicating that the overlapping shooting mode has been selected, and indicates that the overlapping shooting mode has been set. 13 and activates the overlapping area detecting means 20 (step S5).
1). When one image of the subject is photographed by the imaging means 2 in this state, the main control means 4 stores the captured first image in the frame memory 7 and the external storage device 9 as a reference image. Further, a posture signal of the imaging unit 2 detected by the posture detection sensor 14 is sent to the overlapping area detection unit 13 via the signal processing unit 5, and the image of the first image calculated by the posture calculation unit 134 is captured. The attitude of the means 2 is stored in the buffer memory 137 (step S52). After the first image is captured, when the position of the image capturing means 2 for capturing an object is shifted, the main control means 4 uses the image capturing means 2 at regular time intervals to refer to the second image currently input by the image capturing means 2 as a reference image. When the attitude signal of the imaging means 2 detected by the attitude detection sensor 14 is sent to the overlapping area detection means 13 through the signal processing means 5 and the second image is captured by the attitude calculation means 134 Is calculated (step S53). The relative attitude calculation means 135 of the overlap area detection means 13 determines the relative attitude between the two based on the attitude at the time of shooting the reference image stored in the buffer memory 137 and the attitude at the time of inputting the current reference image output from the attitude calculation means 134. The motion vector is calculated and sent by the motion vector calculation means 136. The motion vector calculation means 136 calculates a motion vector v between the two images based on the relative orientation of the reference image and the reference image, adds the motion vectors v to each other, detects an overlap region, and indicates the size of the overlap region to the overlap region display indicator 112 of the finder 11. It is displayed (step S54). This process is repeated at regular intervals until the shutter of the imaging means 2 is pressed down (step S5).
5). When the shutter of the image pickup means 2 is depressed, the overlapping area detecting means 13 sets the size of the displayed overlapping area in advance and the indicator 113 of the finder 11.
(Step S5)
6). If the size of the overlapping area is equal to or smaller than the threshold value as a result of the check, the warning indicator 114 is turned on to request the photographer to change the photographing position (step S5).
7). Then, when the photographer changes the photographing position and the size of the overlapping area between the reference image and the currently input reference image becomes equal to or larger than the threshold value and the shutter of the imaging unit 2 is pressed down, the image that is currently input is obtained. Is stored in the external storage device 9, and the reference image stored in the frame memory 7 is replaced with the captured second image (steps S58 and S59). After that, the photographing process of the third image is started (steps S60 and S53). This process is repeated while shooting is continued. In this manner, the photographer can check the size of the overlap area between the captured image and the currently input image during the period from the previous shutter operation to the next shutter operation.

In the above embodiment, the case where the overlap area is detected by calculating the motion vector from the attitude information of the imaging means 2 has been described. For example, as shown in FIG. 16, the image plane 23a of the reference image and the image When it is detected that the rotation of the angle θ around the y-axis of the apparatus coordinate system occurs between the surfaces 23b, and if the focal length or the viewing angle of the imaging unit 2 is known, the overlapping area 24 has the angle θ and the focal length or the imaging distance. It can be easily detected from the viewing angle of the means 2. In this way, the overlapping area may be detected.

[0055]

As described above, according to the present invention, the image previously taken by the image pickup means is used as the reference image, and the overlapping area between the reference image and the image currently input by the image pickup means is automatically detected and displayed. Therefore, when performing various image processing and synthesis such as panoramic image synthesis and restoration of a three-dimensional shape of a subject, it is possible to notify the photographer whether or not the shooting operation is appropriate. Can be photographed in an appropriate overlapping area.

Since the overlapping range of the images is not preliminarily set, the photographer can photograph the subject under relatively free conditions.

Further, by detecting the overlapping area between the reference image and the image currently being input by the imaging means from the motion vector of the moving image currently being input by the imaging means, the overlapping area of a plurality of images can be accurately detected. Can be

Further, by detecting an overlapping area between the reference image and the image currently being input by the image pickup means from a motion vector between the reference image and the image to be picked up, a plurality of images can be easily processed. The overlapping area can be detected with high accuracy. Also, since the photographer does not need to capture the subject from the time of shooting the reference image to the time of the next image shooting,
The imaging means can be relatively freely operated.

Further, by detecting the overlap area between the reference image and the image currently being input by the image pickup means from the attitude of the image pickup means, the overlap area can be easily detected. Even when a subject having few characteristics is photographed, the overlapping region can be detected stably.

[Brief description of the drawings]

FIG. 1 is a block diagram illustrating a configuration of an image input device according to an embodiment of the present invention.

FIG. 2 is a flowchart showing the operation of the embodiment.

FIG. 3 is a configuration diagram showing an indicator of a finder.

FIG. 4 is an explanatory diagram showing an overlapping state of images.

FIG. 5 is a block diagram illustrating a configuration of an overlap area detecting unit according to a second embodiment.

FIG. 6 is an explanatory diagram showing a correlation window centering on a feature point.

FIG. 7 is an explanatory diagram showing correspondence of feature points.

FIG. 8 is a flowchart showing the operation of the second embodiment.

FIG. 9 is a block diagram illustrating a configuration of an overlapping area detecting unit according to a third embodiment.

FIG. 10 is a flowchart showing the operation of the third embodiment.

FIG. 11 is a block diagram illustrating a configuration of an image input device according to a fourth embodiment.

FIG. 12 is a block diagram illustrating a configuration of an overlap area detecting unit according to a fourth embodiment.

FIG. 13 is an explanatory diagram showing a device coordinate system of an imaging unit.

FIG. 14 is an explanatory diagram showing a world coordinate system.

FIG. 15 is a flowchart showing the operation of the fourth embodiment.

FIG. 16 is a layout diagram illustrating an example of an overlap between an image plane of a reference image and an image plane of a reference image.

[Explanation of symbols]

 REFERENCE SIGNS LIST 1 image input device 2 imaging means 3 mode setting means 4 main control means 5 signal processing means 6 memory control means 7 frame memory 8 external storage interface 9 external storage device 10 display interface 11 viewfinder 12 external display device 13 overlapping area detecting means 14 attitude Detection sensor 131 Feature point setting means 132 Motion vector calculation means 133 Addition means 134 Attitude calculation means 135 Relative attitude calculation means 136 Motion vector calculation means 137 Buffer memory

Claims (8)

[Claims] 1. An image input apparatus having an image pickup means for inputting an image of a subject and an image storage means for storing an image obtained from the image pickup means, wherein an image is first taken by the image pickup means and stored in the image storage means. An image input apparatus comprising: an overlap area detection unit that automatically detects an overlap area between a reference image and an image currently being input by an imaging unit. 2. The image input device according to claim 1, wherein said overlapping area detecting means detects an overlapping area with a reference image from a motion vector in a currently input moving image. 3. The image input device according to claim 1, wherein said overlapping area detecting means detects an overlapping area from a motion vector between a reference image and an image to be imaged. 4. An imaging means for inputting an image of a subject, an attitude detection sensor for acquiring a signal representing the attitude of the imaging means,
Image storage means for storing an image obtained from the imaging means, and overlap area detection for automatically detecting an overlap area between the reference image previously captured by the imaging means and stored in the image storage means and the image currently being input by the imaging means. An image input apparatus having an image capturing apparatus, wherein the overlapping area detecting means detects an overlapping area between a reference image and an image currently being input by the image capturing means from the attitude information obtained from the attitude detecting sensor signal. Input device. 5. An image input device, wherein an image previously taken by an imaging means is stored as a reference image, and an overlapping area between the stored reference image and an image currently being input by the imaging means is detected and displayed. Method. 6. The image input method according to claim 5, wherein an overlap area between the reference image and the image currently input by the image pickup means is detected from a motion vector of a moving image currently input by the image pickup means. 7. The image input method according to claim 5, wherein an overlapping area between the reference image and the image currently being input by the image pickup means is detected from a motion vector between the reference image and the image to be picked up. 8. The image input method according to claim 5, wherein an overlap area between the reference image and an image currently being input by the image pickup means is detected from posture information of the image pickup means.