JP2001250114A - Method and device for image processing and computer- readable recording medium - Google Patents

Abstract

PROBLEM TO BE SOLVED: To provide a method and a device for image processing which can easily obtain a more adequate image by correcting distortion, and a recording medium which actualizes the method. SOLUTION: This is an image processor 1 which corrects distortion of images obtained by photographing a subject so that the images overlap with one another and is equipped with a correspondence detection part 21 which detects the correspondence relation between overlap parts in the images obtained by the photography, a reference image setting part 20 which selects an object of distortion correction out of the images, and an image distortion correction part 22 which corrects the distortion of the image set by the reference image setting part 20 according to the correspondence relation detected by the correspondence detection part 21.

Description

DETAILED DESCRIPTION OF THE INVENTION

[0001]

BACKGROUND OF THE INVENTION 1. Field of the Invention The present invention relates to an image processing method, an image processing apparatus, and a recording medium, and more particularly, to an image processing method and an image processing apparatus for obtaining an appropriate image regardless of a shooting state, and the image processing. The present invention relates to a computer-readable recording medium for implementing the method.

[0002]

2. Description of the Related Art With the rapid progress of computer networks, business styles have diversified, and it has become necessary to quickly obtain important information in all aspects. Along with this, there is an increasing demand for easy and high-definition input of goods and document information necessary for business using portable input devices everywhere. In particular, with the rapid spread of digital still cameras and their high resolution, there has been an application of processing and processing of captured images to utilize them as effective electronic information for work and entertainment. Was.

As a first typical application example, when photographing a subject surface such as an A4 page or a large poster, if the photographing surface is not parallel to the subject surface, the image is distorted. Is also referred to as “tilt distortion”.) There is a technique for improving the legibility of the acquired document image information by correcting the distortion. Here, it is required that the desired range of the subject is photographed in the image for which the “distortion distortion” is to be corrected. That is, the user needs to photograph at least one of the plurality of photographed images so as to include a desired range of the subject. Therefore, there is a demand for an interface (I / F) capable of selecting an object for correcting a tilt distortion among a plurality of captured images, and an interface for prompting a user to pay due attention at the time of capturing.

As a second application, a portable image input device is used to divide and photograph a large area of paper information such as newspapers, a picture drawn on a panel or a wall, and obtain a plurality of images. In some cases, one composite image is created by pasting images together. That is, a CCD (Charge Coupled Dev
With the increase in the number of pixels of an imaging device represented by ice), the resolution of a digital camera has been improved in recent years. However, a subject having a fine pattern as described above, that is, a subject containing a high frequency component is photographed and digitized. Resolution is still insufficient. Therefore, an approach has been taken in which a high-definition image is artificially created by pasting images together to compensate for the lack of resolution of a digital camera.

An example of such an application is a technique in which when a subject can be regarded as a plane, an image obtained by dividing and photographing a part of the subject by using a geometric correction formula such as affine transformation or projective transformation is used. In this method, a subject image of an image is converted into the appearance of the subject in a reference image and attached. An outline of such a technology is described in the document "Computer Vision-Technical Review and Future Perspective-" (Takashi Matsuyama et al., New Technology Communications).

However, in such an application example, if the subject image is distorted in the reference image, there is a problem that the combined image includes the distorted distortion. That is, the magnitude of the tilt distortion in the generated composite image changes depending on which image is used as a reference among a plurality of images obtained by dividing and photographing. This problem will be described with reference to FIG.

As shown in FIG. 1A, for example, an object plane PL is photographed in three directions D _{1 to} D _3, and when images IM ₁ to IM 3 are obtained, respectively, when the images IM ₁ to IM 3 are obtained, the object plane PL is slanted to the left with respect to the object plane PL. When these three images are pasted together based on the image IM1 obtained by photographing from the direction, FIG.
A composite image IMA as shown in FIG. Further, when the same image is pasted on the basis of the image IM2 photographed substantially from the front direction, a composite image IMB as shown in FIG. 1C is obtained. Here, FIGS. 1B and 1C
As shown in FIG.
Are significantly different in the magnitude of the tilt distortion.

[0008] Therefore, it is desired that at least one of the plurality of captured images is one that is captured almost directly on the subject plane PL. There is a demand for an interface for selecting an image to be used as a reference and an interface for calling a user's attention when capturing an image serving as a reference for bonding.

[0009]

SUMMARY OF THE INVENTION The present invention has been made in view of the above points, and an image processing method and an image processing apparatus for easily obtaining a more appropriate image by correcting distortion.
And a computer-readable recording medium for implementing the image processing method.

[0010]

SUMMARY OF THE INVENTION The object of the present invention is to provide an image processing method for correcting a distortion of an image photographed from a plurality of directions so that at least a part of the image overlaps with a subject.
A first step of specifying a correspondence relationship between overlapping portions in a plurality of images obtained by shooting, a second step of selecting a target for correcting distortion from the plurality of images, and a first step And a third step of correcting distortion of the image selected in the second step according to the determined correspondence. According to such means, a target for which distortion is to be corrected is selected, so that an optimal image for obtaining an appropriate image can be set as a correction target.

Here, in the second step, an object to be corrected may be automatically selected according to the size of the area occupied by the subject in the image. According to such a means, it is possible to automatically correct the distortion of an image that requires the largest amount of document information.

In the second step, the correction target may be automatically selected according to the direction of the linear pattern detected in the image. According to such means, an image photographed from a position substantially facing the subject can be automatically set as a correction target.

[0013] In the second step, if the object to be corrected is automatically selected according to the correspondence specified in the first step, highly accurate correction can be executed reliably. .

In the second step, if the object to be corrected is automatically selected in accordance with the direction of the subject detected for each photographing, the photographing is performed from a position almost directly facing the subject. Images can be automatically corrected.

It is another object of the present invention to provide an image processing method for correcting distortion of an image photographed from a plurality of directions so that at least a part of the image overlaps with a subject. A first step of respectively specifying the correspondence of the overlapping portions in the, and a second step of selecting an image with the least distortion from a plurality of images,
Depending on the correspondence identified in the first step,
This is achieved by providing an image processing method, comprising: a third step of correcting distortions of a plurality of images, respectively, and combining the images selected in the second step with the images. According to such means, a composite image is generated based on the image with the least distortion, so that a more appropriate composite image can be obtained.

It is another object of the present invention to provide an image processing apparatus for correcting distortion of an image photographed from a plurality of directions so that at least a part of the image overlaps with a subject. Correspondence detecting means for detecting the correspondence of the overlapping portions in the image, selecting means for selecting an image having the least distortion from the plurality of images, and correspondence between the plurality of images according to the correspondence detected by the correspondence detecting means. This is attained by providing an image processing apparatus comprising: an image synthesizing unit that corrects each of the distortions and synthesizes the image selected by the selecting unit with the image.

It is another object of the present invention to provide an image processing apparatus for correcting distortion of an image photographed from a plurality of directions so that at least a part of the image overlaps with a subject. Selecting means for pre-selecting an image to be used as a reference when correcting distortion, and notifying means for notifying to a user that an image to be taken next will be a reference image in accordance with the selection made by the selecting means And, corresponding detection means for detecting the correspondence of overlapping portions between the reference image obtained by shooting and the other image, according to the correspondence detected by the correspondence detection means,
This is attained by providing an image processing apparatus comprising: a correction unit that corrects distortion of an image used as a reference. According to such means, it is possible to call the user's attention in photographing an image that is used as a reference when correcting distortion.

It is another object of the present invention to provide an image processing apparatus for correcting distortion of an image photographed from a plurality of directions so that at least a part of the image overlaps with a subject. Correspondence detecting means for detecting a correspondence relation between overlapping portions in the image, selecting means for selecting an object for which distortion is to be corrected from among a plurality of images, and selection by the selection means in accordance with the correspondence relation detected by the correspondence detection means And a correcting means for correcting the distortion of the image obtained.

Another object of the present invention is to provide an image processing apparatus for correcting distortion of an image photographed from a plurality of directions so that at least a part of the image overlaps with the object. Means, a selecting means for selecting an image to be corrected from among the images taken by the plurality of optical means, and detecting a correspondence relationship of an overlapping portion between the image selected by the selecting means and another image This is attained by providing an image processing apparatus comprising: a correspondence detection unit; and a correction unit that corrects a distortion of a selected image in accordance with the correspondence detected by the correspondence detection unit. According to such a means, since a plurality of subject images photographed from a plurality of directions can be obtained by one photographing, the number of photographing times required to correct the distortion can be reduced.

Here, the selecting means automatically selects an object to be corrected in accordance with the size of the area occupied by the subject in the image, or selects an object in accordance with the direction of the linear pattern detected in the image. In this case, the correction target is automatically selected. In addition, the selecting means automatically selects a correction target according to the correspondence detected by the correspondence detection means, or automatically selects a correction target according to the direction of the subject detected for each photographing. May be selected.

It is another object of the present invention to provide a computer-readable recording medium which stores a program for correcting a distortion of an image photographed from a plurality of directions so that at least a part of the image overlaps a subject. The program causes the computer to specify a correspondence relationship between overlapping portions in a plurality of images obtained by shooting, select a target to correct distortion from among the plurality of images, and specify the specified correspondence relationship. The computer-readable recording medium is characterized in that the distortion of the selected image is corrected according to the following. According to such means, an optimal image for obtaining an appropriate image can be easily set as a correction target.

It is another object of the present invention to provide a computer-readable recording medium on which a program for correcting a distortion of an image photographed from a plurality of directions by a computer so as to at least partially overlap the subject is recorded. Then, the program causes the computer to specify the correspondence between overlapping portions in the plurality of images obtained by shooting, select an image with the least distortion from among the plurality of images, and specify the specified correspondence. This is achieved by providing a computer-readable recording medium characterized in that distortions of a plurality of images are respectively corrected according to the relationship and combined with a selected image. According to such means, a more appropriate synthesized image can be easily obtained.

[0023]

Embodiments of the present invention will be described below in detail with reference to the drawings. In the drawings, the same reference numerals indicate the same or corresponding parts. [First Embodiment] FIG. 2 is a diagram for explaining an image processing method and an image processing apparatus according to a first embodiment of the present invention. Here, in the present embodiment, as shown in FIG. 2, two images 3 and 4 are photographed by image processing device 1 so that at least a part of subject plane PL overlaps, and these two images are taken. An example will be described in which one of the images 3 and 4 is corrected, for example, the tilt distortion of the image 3 is corrected to finally obtain the distortion corrected image 5.

FIG. 3 is a diagram showing a configuration of the image processing apparatus 1 according to the first embodiment of the present invention. As shown in FIG. 3, the image processing device 1 includes an imaging unit 11 and a signal processing unit 12
A memory control unit 13, a main control unit 14, a frame memory 15, an interface 16, a display unit 17, an external storage unit 18, a shooting mode setting unit 19, a reference image setting unit 20, A section 21 and an image distortion correction section 22 are provided. Then, the imaging unit 11 includes a lens 111, an aperture 112, a shutter 113, and a photoelectric conversion element 114.
And a preprocessing unit 115.

Here, the signal processing unit 12 includes a pre-processing unit 11
5 and the memory control unit 13, the main control unit 14, and the interface 16. Further, the memory control unit 13 is further connected to the frame memory 15 and the reference image setting unit 20. The main control unit 14 is further connected to the memory control unit 13, the shooting mode setting unit 19, and the reference image setting unit 20.

The frame memory 15 is connected to the memory control unit 13, the correspondence detection unit 21, and the image distortion correction unit 22. The interface 16 is further connected to a display unit 17 and an external storage unit 18. Then, the reference image setting unit 20 is further connected to the image distortion correction unit 22. The correspondence detection unit 21 is further connected to the image distortion correction unit 22.

On the other hand, in the imaging section 11, the lens 11
1, aperture 112, shutter 113, photoelectric conversion element 11
4 are arranged in this order on the optical axis, and the photoelectric conversion element 114 is connected to the preprocessing unit 115.

In the above, the photographing mode is switched by the photographing mode setting section 19, and the image to be corrected by the reference image setting section 20 is set. In addition, the correspondence detection unit 21 extracts a feature point and a corresponding point between two images at least partially overlapping each other. Then, the image distortion correction unit 22 corrects the tilt distortion in the captured image according to the signal supplied from the correspondence detection unit 21. The setting of the reference image, the operation of the correspondence detection unit 21, and the correction of the tilt distortion will be described later in detail.

Further, for example, a CCD is used as the photoelectric conversion element 114 of the imaging unit 11. The pre-processing unit 115 includes a preamplifier and an automatic gain control circuit (Auto Gain Control circuit).
-AGC) and an analog-to-digital converter (A / D converter). The analog video signal output from the photoelectric conversion element 114 is subjected to pre-processing such as amplification and clamping. After that, the analog video signal is converted into a digital video signal.

The signal processing unit 12 is constituted by a digital signal processor (DSP processor) or the like, and performs various image processing such as color separation, white balance adjustment, and γ correction on the digital video signal obtained by the imaging unit 11. Is applied. Further, the memory control unit 13 stores the image signal processed in this manner in the frame memory 15 and reads out the image signal stored in the frame memory 15 on the contrary. The main control unit 14 is configured by a microcomputer or the like. The frame memory 15 stores at least two images, and a semiconductor memory such as a VRAM, an SRAM, or a DRAM is generally used.

Here, the image signal read from the frame memory 15 is subjected to signal processing such as image compression in the signal processing unit 12 and then stored in the external storage unit 18 via the interface 16. The external storage unit 18 reads and writes various signals such as image signals supplied through the interface 16, and is configured by an IC memory card, a magneto-optical disk, or the like. Here, as the external storage unit 18,
If a modem card or an ISDN card is used, an image signal can be directly transmitted to a remote recording medium via a network.

Conversely, reading of the image signal recorded in the external storage unit 18 is performed by transmitting the image signal to the signal processing unit 12 via the interface 16 and performing image expansion in the signal processing unit 12. Done. on the other hand,
The display of the image signal read from the external storage unit 18 and the frame memory 15 is performed by performing signal processing such as digital-analog conversion (D / A conversion) and amplification on the image signal in the signal processing unit 12. This is performed by transmitting the data to the display unit 17 via the interface 16. Here, the display unit 17 displays an image according to an image signal supplied via the interface 16 and is configured by, for example, a liquid crystal display device installed in a housing of the image processing apparatus 1.

FIG. 4 is a perspective view showing the image processing apparatus shown in FIG. As shown in FIG. 4, the image processing apparatus 1 according to the present embodiment includes a power switch 101, a shutter 102, a finder 103, and a shooting mode setting key 104 for setting a shooting mode in the shooting mode setting unit 19. An upward scroll key 201 for scrolling an image projected on the display unit 17 upward, a downward scroll key 202 for scrolling an image projected on the display unit 17 downward, and an enter key 203.

The operation of the image processing apparatus having the above configuration will be described below with reference to the flowcharts shown in FIGS. First, the image processing apparatus 1 is started by switching the power switch 101, and the photographing mode is selected. Here, the shooting modes include a normal mode for shooting a normal snapshot and a tilt correction mode for generating an image in which tilt distortion of a shot image is corrected. The photographing mode is selected by operating the photographing mode setting key 104 by the user. Note that the image processing apparatus 1
Is provided with a shooting mode setting key 104. However, the shooting mode setting unit 19 may be configured by hardware or software provided separately from the main body.

Then, as shown in FIG.
In step 1, it is determined whether or not the tilt correction mode is to be selected. If the tilt correction mode is not selected and the normal mode is selected, the process proceeds to step S10, where the user takes a snapshot of a desired subject.

On the other hand, when the tilt correction mode is selected in step S1, the process proceeds to step S2. Then, the user captures at least two images of the object plane PL by the imaging unit 11, and the image processing apparatus 1 captures the object image.
At this time, each image needs to be photographed so that a part of the subject image overlaps with each other.

Next, in step S3, it is determined whether or not the input of the subject image has been completed. If the input has not been completed, the process returns to step S2, and the input of the subject image is continued. On the other hand, if the input of the subject image has been completed by the user's instruction to end the shooting, the process proceeds to step S4. Note that the above instruction is performed by pressing the shooting mode setting key 104 again to switch to the normal mode, and a switch for instructing the end of shooting may be separately provided.

During shooting, as shown in FIG. 6, the display unit 17 displays the current number of shots, such as "first shot", and a message, such as "pressing a shooting mode setting key to end". An appropriate shooting end method may be displayed as an overlay.

When the photographing is completed as described above,
In step S4 and subsequent steps, the operation for correcting the tilt distortion is started. First, in step S4, which image is to be corrected for the tilt distortion is set. Note that the image selected at this time is also referred to as a “reference image” below. The setting of the reference image is performed by the reference image setting unit 2.
0. In the following, the reference image setting unit 2
0 will be described in detail.

FIG. 7 shows the reference image setting unit 2 shown in FIG.
FIG. 7 is a diagram showing a layout of No. 0; As shown in FIG. 7, the reference image setting unit 20
And a down scroll key 202 and an enter key 203. Then, as shown in FIG. 7, when the user gives an instruction to end shooting in step S3, the characters "reference image setting" are overlaid on the display unit 17, and the user is instructed to select the reference image. .

Then, the user operates the upward scroll key 201 and the downward scroll key 202, and the images captured in step S2 are displayed while being sequentially switched. By pressing the upward scroll key 201, an image taken immediately before the currently displayed image is displayed.
By pressing 02, an image taken immediately after the currently displayed image is displayed. When the enter key 203 is pressed in a state where the image selected as the reference image is displayed on the display unit 17, the image displayed at that time is determined as the reference image.

As the reference image, the subject plane PL is captured over the widest region in the plurality of captured images, and the angle between the imaging plane and the subject plane during photography is as small as possible. The image is selected by the user.

Here, the object area and the above-mentioned inclination angle are calculated inside the apparatus, and an object to be corrected for the tilt distortion, that is, an optimum image as a reference image is automatically selected according to the calculation result. It is also possible. FIG. 8 is a diagram illustrating a configuration of the image processing apparatus 2 that realizes such an operation. As shown in FIG. 8, the image processing apparatus 2 includes
Reference image setting unit 2 included in image processing apparatus 1 shown in FIG.
A subject area determination unit 23 is provided instead of 0.

The subject area determination unit 23 performs processing for detecting the area occupied by the subject in the captured image. For example, the document “Image Processing and Recognition” (Takeshi Aiin
As described in Tomoharu Nagao, Shokodo)
(A) a method of performing clustering on an image such as a region growing method or a region division method, (b) a method of performing clustering on a feature space such as region division by a histogram,
(C) a method using edges in an image, such as contour tracing,
(D) An area division method such as texture analysis is applied. However, when the object surface is rectangular, the vertex coordinates of the four corners of the object on the image are input externally,
The region of the subject is uniquely determined.

The data indicating the area of the subject obtained by the subject area determining section 23 is supplied to the image distortion correcting section 22, and the image having the widest area is stored in the image distortion correcting section 22.
Is selected. Further, as will be described in detail below, the image distortion correction unit 22 calculates the angle formed by the imaging plane with respect to the subject plane PL. In the case where there are a plurality of images showing, the image having the smallest angle is finally selected as the reference image.

The function of automatically selecting the reference image as described above can be similarly applied to any of the following embodiments.

Next, proceeding to step S5, feature points are detected in the reference image determined as described above, and an image overlapping at least a part of the reference image (hereinafter, also referred to as a "reference image"). In the parentheses, corresponding points indicating the same locations as the above feature points are detected. The detection of such feature points and corresponding points is performed by the correspondence detection unit 21 shown in FIG. Therefore, the configuration and operation of the correspondence detection unit 21 will be described in detail below.

The correspondence detecting section 21 detects the same photographed portion in two images having mutually overlapping areas as described above. Here, a method using a correlation operation will be described.

FIG. 9 is a diagram showing a configuration of the correspondence detecting section 21 shown in FIG. As shown in FIG. 9, the correspondence detection unit 21 includes a feature point setting unit 211 connected to the frame memory 15, and a feature point setting unit 211 and the frame memory 15.
And a correlation operation unit 212 connected to. Note that FIG.
The reference image and the reference image are stored in the frame memory 15 shown in FIG.

Here, the feature point setting section 211 determines the position of the feature point in the reference image, and then extracts (2N + 1) × (2P + 1) gray-scale patterns centered on the feature point and is called a correlation window. Create area data. Note that the positions of the feature points are determined by extracting a portion where the density pattern of the image is characteristic, such as a corner.

Further, the correlation calculation unit 212 detects, by executing a correlation calculation in the reference image, a portion that substantially matches the shading pattern of the correlation window created based on the reference image, and determines this as a corresponding point. . Here, an example of detecting a corresponding point by block matching by correlation calculation will be described with reference to FIG.

As shown in FIG. 10, (2N + 1) ×
Correlation window 215 composed of (2P + 1) gray-scale patterns
In the block matching of 216, the reference image 7
Coordinates (x_i0, Y_i0) I-th feature point 213 having
And coordinates (x _i0+ Dx_i, Y_i0+ D
y_i), The cross-correlation value S of the corresponding point 217_iIs
Is calculated by

[0053]

(Equation 1) In Expression (1), I _s (x, y) represents the density at the coordinate point (x, y) of the reference image 7; I
_r (x, y) indicates the density at the coordinate point (x, y) of the reference image 9. Also, / I _s (x, y) indicates the average density in (2N + 1) × (2P + 1) patterns centered on the coordinate point (x, y) in the correlation window 215 in the reference image 7, and / I _r (X, y) is centered on a coordinate point (x, y) in the correlation window 216 in the reference image 9 (2N +
1) Indicates the average density in (2P + 1) patterns. K indicates a constant.

Then, according to the above equation (1), each feature point 21
Against 3, by obtaining the point the maximum value of the cross-correlation value S _i is the threshold value or more predetermined corresponding point 217 in the reference image 9 is obtained. The cross-correlation value S _i
Is less than or equal to the threshold value, it is determined that no corresponding point exists.

When the detection of the feature point and the corresponding point is completed in this way, a parameter for correcting the tilt distortion of the reference image 7 is calculated in step S6 shown in FIG. 5, and based on the parameter in step S7. Then, an image in which the tilt distortion of the image is corrected is created, and the operation is terminated. In the following, each of the above parameters is also referred to as a “distortion correction parameter”, and the image in which the tilt distortion is corrected is also referred to as a “distortion corrected image”.

The calculation of the distortion correction parameters and the generation of the distortion corrected image are executed by the image distortion correction unit 22. Hereinafter, the configuration and operation of the image distortion correction unit 22 will be described in detail.

The image distortion correction unit 22 corrects the tilt distortion by converting the object surface into an image photographed from the front using the relationship between the feature points detected by the correspondence detection unit 21 and the corresponding points. The configuration of the image distortion correction unit 22 is shown in FIG. As shown in FIG. 11, the image distortion correction unit 22 includes a three-dimensional calculation unit 221 and a parameter calculation unit 2.
22 and a coordinate conversion unit 223. Here, the three-dimensional calculation unit 221 is connected to the correspondence detection unit 21 and the reference image setting unit 20, and the parameter calculation unit 222 is connected to the three-dimensional calculation unit 221. The coordinate conversion unit 223 includes the parameter calculation unit 222, the frame memory 15, and the reference image setting unit 20.
Connected to.

The operation of the image distortion correcting section 22 will be described below. In the following, as shown in FIG. 12, the reference image 7 and the reference image 9
As shown in FIG. 13, the optical system of the imaging unit 11 has a positive rightward direction of the image plane 224 with respect to the x-axis, a positive downward direction of the image plane 224 with respect to the y-axis, and z in the optical axis direction. Regarding the axis, the origin O which is the optical center of the imaging unit 11
The direction toward the image plane 224 is positive, and the focal length of the optical system is f.
n model) will be described as an example.

The three-dimensional operation unit 221 shown in FIG.
The following three-dimensional parameters are calculated based on the relationship between the feature points 213 and the corresponding points 217. That is, a rotation matrix R indicating a change in the direction of the imaging unit 11 at the time of reference image shooting based on the reference image shooting shown in FIG. 12 and the imaging unit 11 at the time of reference image shooting based on the reference image shooting
Translational motion vector t indicating the change in the position of
A normal vector n indicating the direction of L is calculated. Then, methods for obtaining these three three-dimensional parameters {R, t, n} are mainly the following two methods.

That is, as a first method, the position and orientation of the camera at the time of capturing each image and the three-dimensional coordinates of each corresponding point are calculated based on a set of eight or more feature points and corresponding points. Assuming that the plane is a plane, there is a method of applying the obtained three-dimensional coordinates to one plane.

As a second method, a projection transformation matrix (homography matri
x) is calculated, and the position and orientation of the camera and the orientation of the object plane at the time of capturing each image are calculated from the obtained projection transformation matrix.

Here, the first method is a general-purpose motion stereoscopic vision technique, and the above-mentioned parameters {R,
t, n} is uniquely obtained, and details thereof are recorded in general literature on three-dimensional measurement and computer vision (for example, “3D Vision” co-authored by Xu Go and Saburo Tsuji, Kyoritsu Shuppan). On the other hand, the second method is to calculate the direction of the camera and the direction of the subject plane after obtaining a coordinate transformation formula (projection transformation matrix) that is satisfied under the constraint that the subject is a plane. . Then, the three-dimensional operation unit 22
Needless to say, the method 1 can employ any of the first and second methods. Here, the operation based on the second method will be described.

First, the procedure for calculating the projection transformation matrix will be described in detail. Here, the projective transformation from the reference image to the reference image is, as shown in FIG. 14, converted into an image obtained when the subject image shown in the reference image 7 is photographed from the same direction as the reference image 9. To obtain the image 10. When representing this projective transformation in a formula, if the point in the reference image (x _{s, y s)} and the reference point in the image (x _{r, y r)} and are in correspondence, the following equation .

[0064]

(Equation 2) Then, the eight unknowns b ₁ to b in the above equation (2)
₈

[0065]

(Equation 3)And a matrix B, which is called a projective transformation matrix. This
To find the projection transformation matrix B of
Is (x_si, Y_si) And the reference image
Coordinates are (x_ri, Y_ri) Corresponding points (i = 1,
.., N; N ≧ 4). Where
Mark (x_si, Y_si) And coordinates (x_ri, Y _ri) Above
Substituting into equation (2), b₁~ B₈It is sufficient to find the solution of
Actually, the equation (2) is obtained by an error such as noise superimposed on the image.
Does not hold, so the solution is calculated using the least squares operation shown below.
Will be.

[0066]

(Equation 4) Then, the above equation (4) is modified as follows.

[0067]

(Equation 5) Using the constraint condition that the value of the derivative obtained by varying and differentiating the left side of the above equation (5) with b _{1 to} b ₈ becomes 0, b _{1 to} b ₈ can be solved by solving simultaneous linear equations Is calculated. That is, the projective transformation matrix B can be obtained by a simple linear operation using the pair associated with each other.

Next, a procedure for obtaining a three-dimensional parameter {R, t, n} from the projection transformation matrix B will be described. The normal vector n of the object plane is

[0069]

(Equation 6) And the equation of the subject plane with respect to the time of shooting the reference image is

[0070]

(Equation 7)Here, | d | indicates the distance from the origin to the subject plane PL.
And r = [xyz] ^TIt is put. Also focal length
Using f, equation (2) can be rewritten as:

[0071]

(Equation 8) Further, equation (8) is

[0072]

(Equation 9) Is transformed. However,

[0073]

(Equation 10) It is. At this time, the matrix H of Equation (9) and the parameter {R,
t, n, d} is as follows.

[0074]

[Equation 11] Here, s ′ is a constant, and each element of the matrix H has a degree of freedom of the scale. The conversion from the projection transformation matrix B in Expression (3) to the matrix H in Expression (9) can be performed by the following expression.

[0075]

(Equation 12) In the following, parameters {R, t,
n, d} will be described. However, the scale of the variable d representing the distance to the object plane and the magnitude of the translational motion vector t are indefinite,

[0076]

(Equation 13) Assume that Here, the calculation process for deriving the solution of the parameter {R, t, n, d} is described in detail in the document "Image understanding-Mathematics of three-dimensional recognition-" (Kenichi Kanaya, Morikita Publishing). The results are summarized as follows. (I) Multiply each element of the matrix H of the equation (9) by an appropriate constant so that det [H] = 1. (Ii) symmetric matrix HH eigenvalues of _{^{T σ 1 2, σ 2 2}} , and sigma ₃ ^2, perpendicular to the corresponding eigenvectors _{u 1,} u 2, _{u 3} together, take unit vector to make right-handed in this order. Where σ ₁ ≧ σ
₂ ≧ σ ₃ > 0. (Iii) If σ ₁ = σ ₂ = σ ₃ , the motion parameter is

[0077]

[Equation 14] And the parameters {n, d} of the object plane are undefined. Otherwise, two sets of solutions are obtained as follows.
(Iv) The parameters {n, d} of the object plane are determined as follows.

[0078]

(Equation 15)

[0079]

(Equation 16) However, ε = ± 1, and ε is selected so that c> 0. (V) The unit translational motion vector t is determined as follows.

[0080]

[Equation 17] Further, the rotation matrix R is determined as follows.

[0081]

(Equation 18) Therefore, two solutions of the parameters {R, t, n, d} can be obtained from the matrix B. In most cases, the true solution can be determined from the derived values. Note that the focal length f of the imaging unit 11 is used in the intermediate calculation, and the value of the focal length f can be easily obtained by a method such as storing the optical system parameters of the imaging unit 11 in an internal memory (not shown). Is possible. Further, even when the focal length of the optical system of the imaging unit 11 is variable and the focal lengths of the reference image and the reference image are different, if the focal lengths of both images are known, the above three-dimensional parameter calculation procedure can be applied as it is, The focal length can be detected by a method such as installing an encoder in the optical system.

Next, the parameter calculating section 222 shown in FIG. 11 performs a tilt distortion based on the relationship between the imaging section 11 and the orientation of the subject plane calculated by the three-dimensional calculation section 221 when the subject plane is photographed. Is calculated. Here, in the present embodiment, as shown in FIG. 15, a plane parallel to the subject plane calculated by the three-dimensional calculation unit 221 is set as the projection plane 31, and the image plane 30 including the subject image having the tilt distortion is displayed. By performing the projective transformation of projecting, the tilt distortion of the image is corrected. Here, for example, a point P1 on the image plane 30 is projected on a point P2 on the projection plane 31. Hereinafter, a method of calculating the parameters for correcting the tilt distortion will be described.

First, as shown in FIG. 16, a rotation matrix R ′ indicating coordinate transformation for making the z-axis in the apparatus coordinate system 33 coincide with the unit normal vector of the projection target surface 31 is obtained. In this case, the following relational expression holds.

[0084]

[Equation 19] There are many rotation matrices R ′ that satisfy the above equation (9). Here, the rotation matrix R ′ is defined as the following equation.

[0085]

(Equation 20) Here, _R'x and _R'y are represented as follows, respectively.

[0086]

(Equation 21) This is achieved by rotating the device coordinate system (xyz coordinate system) 33 in the following order as shown in FIG.
This is equivalent to converting to a coordinate system. (I) Device coordinate system 3
3 by β rotation only about the y-axis. Then, a coordinate system obtained by this rotation is defined as an x ₁ y ₁ z ₁ coordinate system.
(Ii) rotates by α the device coordinate system about the x ₁ axis.

Here, using equations (19) and (20), the rotation angle is derived as follows.

[0088]

(Equation 22) Then, the matrix R ′ can be uniquely determined by substituting the rotation angles obtained in this way into Expressions (20) and (21).

Next, the coordinates on the image plane 30 are
Transform coordinates upward. That is, in FIG. 15, a point P2 that intersects with the projection plane 31 when the three-dimensional vector p corresponding to the point P1 on the image plane 30 is extended is set as coordinates after coordinate conversion. Then, a point P1 based on the device coordinate system 33
Is represented by the following equation.

[0090]

(Equation 23) Here, k is an expansion coefficient representing the distance from the optical center o of the imaging unit 11 to the projection surface, and thus k represents the size of the distortion-corrected image to be created. Also, a three-dimensional vector p
Is the point P based on the device coordinate system 33 at the time of shooting the reference image.
1 is converted into a coordinate by using the rotation matrix R ′ as in the following equation.
Is converted to a three-dimensional vector p ′ when the object is directly opposed to the object plane.

[0091]

(Equation 24) Therefore, by setting the x coordinate and the y coordinate of Expression (25) as the coordinates obtained after the coordinate conversion, an image in which the tilt distortion has been corrected can be obtained. Then, according to the above-described procedure, it is possible to calculate the parameter for correcting the tilt distortion of the reference image by using the equation (25).

Next, the coordinate conversion section 223 performs coordinate conversion of the reference image based on the tilt correction parameter calculated by the parameter calculation section 222 to create a distortion corrected image. Specifically, coordinates after coordinate transformation (X, Y) before conversion of the coordinate _(x s, _{y s)} corresponding to the calculated according to equation (25), the calculated coordinates _(x s, _{y s} The pixel value at the coordinates (X, Y) is determined by interpolation based on the pixel values in the vicinity of ()). Note that this interpolation operation may be performed using an existing method such as a bilinear interpolation method or a B-spline interpolation method.

As described above, according to the image processing apparatus according to the first embodiment of the present invention, when two or more subject surfaces are photographed so that at least a part thereof overlaps each other,
It is possible to obtain a more appropriate overall image of the subject by correcting the tilt distortion in the photographed arbitrary image.

That is, in the above procedure, it is not necessary to photograph the entire range of the subject in all the images. For example, as shown in FIG. 14, an image of the whole subject and an image of only a part of the subject are taken. Is enough. In such a case, an image obtained by photographing the entire subject by the reference image setting unit 20 can be selected as the reference image, so that the entire image of the subject in which the tilt distortion has been corrected can be generated.

Further, in order to generate a distortion-corrected image by the above-described method, at least two subject images are required. Therefore, if the number of images to be photographed is limited to two, the user can input an image, perform photographing, and Not only can the setting of the reference image be simplified, but also the memory capacity required for storing the image, the calculation cost required for detecting feature points and corresponding points, and calculating the projection transformation matrix can be reduced. [Second Embodiment] FIG. 18 is a diagram for describing an image processing method and an image processing apparatus according to a second embodiment of the present invention. As shown in FIG. 18 (a), the still picture of the same object plane PL is, when taken from a plurality of directions d ₁ to d _k so as to overlap each at a part of the image, the present embodiment The operation of the image processing apparatus 6 according to the first embodiment will be described.

Here, as shown in FIG.
Image imn (n = 1 by photographing from the direction _{D n,}
, J,..., K) are obtained. For example, an image imj and an image im (j + 1) (1 ≦ j ≦) obtained by shooting in adjacent directions such as an image im1 and an image im2.
It is assumed that there is an overlap area between the data and k-1).

In the image processing apparatus according to the second embodiment, as shown in FIG. 18B, another image is matched with one of the images imj selected as the reference image. The composite image IMC is obtained by pasting.

FIG. 19 is a diagram showing a configuration of the image processing apparatus 6 according to the second embodiment. As shown in FIG. 19, the image processing device 6 has the same configuration as the image processing device 1 according to the first embodiment shown in FIG. 24 is provided. Here, the image synthesizing unit 24 pastes images having overlapping regions by performing coordinate conversion based on the relationship between the feature points and the corresponding points obtained by the correspondence detecting unit 21. This operation will be described later in detail. explain.

FIG. 20 is a flowchart showing the operation of the image processing apparatus 6 according to the second embodiment shown in FIG. First, the image processing apparatus 1 is started by switching the power switch 101, and the photographing mode is selected. Here, the shooting modes include a normal mode for shooting a normal snapshot and a tilt correction mode for generating an image in which tilt distortion of a shot image is corrected. When the user selects the shooting mode, the shooting mode setting key 104
This is done by operating. Note that a shooting mode setting key 104 is provided as a shooting mode setting unit 19 in the main body of the image processing apparatus 1. However, the shooting mode setting unit 19
May be configured by hardware or software provided separately from the main body.

Then, as shown in FIG.
In step 1, it is determined whether or not the tilt correction mode is to be selected. If the tilt correction mode is not selected and the normal mode is selected, the process proceeds to step S10, where the user takes a snapshot of a desired subject.

On the other hand, when the tilt correction mode is selected in step S1, the process proceeds to step S2. Then, the user captures at least two images of the object plane PL using the imaging unit 11, and the image processing device 6 captures the object image.
At this time, each image needs to be photographed so that a part of the subject image overlaps with each other.

Next, in step S3, it is determined whether or not the input of the subject image has been completed. If the input has not been completed, the process returns to step S2, and the input of the subject image is continued. On the other hand, if the input of the subject image has been completed by the user's instruction to end the shooting, the process proceeds to step S4. Note that the above instruction is performed by pressing the shooting mode setting key 104 again to switch to the normal mode, and a switch for instructing the end of shooting may be separately provided.

When the photographing of the subject is completed as described above, an operation for generating a composite image in which a plurality of images are combined is started from step S4. First, in step S4, an image on which to generate a composite image is selected. It should be noted that the image selected at this time is the above-described reference image, and a case where the image imj shown in FIG. 18 is selected as the reference image will be described below as an example. The reference image is set in the reference image setting unit 2.
0, the configuration and operation of the reference image setting unit 20 are the same as those in the first embodiment.

Next, in step S5, FIG.
Image pairs taken from adjacent directions shown in (a),
That is, between the image n and the image (n + 1) (1 ≦ n ≦ k−1), a feature point and a corresponding point indicating the same location as the feature point are detected. The detection of the feature points and the corresponding points is performed by the correspondence detection unit 21. Here, the configuration and operation of the correspondence detection unit 21 are the same as those in the first embodiment.

Then, when the detection of the feature points and the corresponding points is completed, the images are pasted after coordinate transformation so as to match the reference image based on the obtained relationship between the two points. Here, when projective transformation is used as coordinate transformation, a projective transformation matrix is calculated in step S6, and a composite image is generated by the above-described bonding in step S7. The calculation of the projection transformation matrix and the generation of the composite image are performed by the image compositing unit 24. Hereinafter, the configuration and operation of the image combining unit 24 will be described in detail.

FIG. 21 shows the image synthesizing unit 2 shown in FIG.
4 is a diagram showing a configuration of FIG. As shown in FIG. 21, the image synthesizing unit 24 includes a projective transformation calculating unit 231 and a coordinate transforming unit 23.
2 is included. Here, the projection conversion calculation unit 231 is connected to the reference image setting unit 20 and the correspondence detection unit 21, and the coordinate conversion unit 232 is connected to the reference image setting unit 20, the frame memory 15, and the projection conversion calculation unit 231.

Then, the projective transformation calculating section 231 computes the projective transformation matrix B shown in the equation (3) using a set of four or more feature points and corresponding points. The calculation procedure is the same as that in the first embodiment, and the least square calculation of Expression (5) may be performed. Here, in the present embodiment, (k
-1) It is necessary to calculate a projective transformation matrix between pairs of images, and to further calculate a projective transformation matrix from each image to a reference image.

More specifically, as shown in FIG. 18, the image imn (n = 1 to k-1) to the image im (n + 1)
The projection transformation matrix to B _n and the image imn to image im
the projective transformation matrix to j

[0109]

(Equation 25) In other words, this projective transformation matrix can be calculated based on the following equation.

[0110]

(Equation 26) Here, the coordinate transformation unit 232 is attached to the image imj based on the image n based on the projection transformation matrix represented by the equation (26) calculated by the projection transformation calculation unit 231. Specifically, first, the coordinates (x, y) of the image n before the coordinate conversion corresponding to the coordinates (X, Y) of the reference image after the coordinate conversion are calculated based on the following equation.

[0111]

[Equation 27] Here, s is a constant for setting the third component of the column vector on the left side in Expression (27) to 1. Next, based on the pixel values near the coordinates (x, y) before the coordinate conversion, the coordinates (X,
The pixel value of Y) is determined by an interpolation operation. The interpolation operation may be performed using an existing method such as a bilinear interpolation method or a B-spline interpolation method.

As described above, according to the image processing apparatus 6 of the present embodiment, two or more subject planes PL are photographed so as to partially overlap each other, and another image is taken with respect to the selected reference image. By pasting, a composite image over a wide area of the subject plane PL is generated. In this case, the reference image setting unit 20 can select, as the reference image, an image photographed from a direction substantially opposite to the subject, that is, an image with small tilt distortion. An image can be obtained as the composite image. [Embodiment 3] In the image processing apparatus according to Embodiments 1 and 2, after two or more subject images have been input or photographed first, they are set as distortion correction targets or as a reference for a composite image. A reference image is selected. On the other hand, in the image processing apparatus according to the third embodiment, the reference image is set beforehand before photographing the subject.

FIG. 22 is a diagram showing a configuration of the image processing device 8 according to the third embodiment of the present invention. As shown in FIG. 22, the image processing device 8 according to the present embodiment has the same configuration as the image processing device 1 according to the first embodiment shown in FIG. 3, but further includes a notification unit 26. They differ in the point. Here, the notification unit 26 is connected to the main control unit 14 and the reference image setting unit 20.

Note that the shutter 113 and the interface 16 included in the image processing apparatus 8 are connected to the main control unit 14, and the finder 241 is connected to the interface 16.

Next, the operation of the image processing apparatus 8 according to Embodiment 3 of the present invention will be described with reference to the flowchart of FIG. First, in step S1, it is determined whether or not the tilt correction mode is selected. When the tilt correction mode is not selected and the normal mode is selected, the process proceeds to step S10, where the user takes a snapshot of a desired subject.

On the other hand, when the tilt correction mode is selected in step S1, the process proceeds to step S2. In step S <b> 2, the image to be subjected to the tilt distortion correction, that is, the reference image is set in the reference image setting unit 20. Here, the reference image setting unit 20 includes a cursor key 204 and an enter key 203 as shown in FIG.
The display section 17 displays an overlay of the text “reference image setting”, and specifies to the user the number of images to be taken in the future and the reference image designation to specify the image to be taken as the reference image in the shooting. You are prompted to set a value. Here, for example, the user enters the cursor key 20
4 By switching the number of shots and the reference image by operating the up or down scroll key, and by increasing or decreasing the set value in the frame by operating the left or right scroll key, The number of sheets and the designated value of the reference image can be set. The enter key 20
By pressing 3, the setting of the reference image is completed.

Next, in step S3, the user starts photographing at least two or more subject images. At this time, each image needs to be photographed so that a part thereof overlaps with each other. Each time an image is shot, a shooting signal is supplied from the shutter 113 to the main control unit 14, so that the counter built in the main control unit 14 is incremented, and the next shot from the counter is the number of the next shot. Is supplied to the notification unit 26. Here, a signal indicating the reference image designation value is supplied from the reference image setting unit 20 to a register built in the notification unit 26, and the reference image designation value is stored in the register. Always compares the specified value of the reference image stored in the register with the value indicated by the number-of-images specifying signal to determine whether or not the image to be captured this time is the reference image.

If the two values match in the above comparison, and the image to be taken is the reference image, the process proceeds to step S5, where the notification unit 26
Notifies the user that the reference image has been captured. That is, at this time, a notification signal is supplied from the notification unit 26 to the interface 16 via the main control unit 14, and as shown in FIG.
Accordingly, the indicator 242 on the side of the subject image 35 in the viewfinder 241 is turned on. Therefore, at the time of photographing, the user can easily recognize whether or not the photographing of the reference image is performed. The above notification may be performed by displaying a predetermined text or symbol on the display unit 17 or the like.

Next, the process proceeds to step S6, but in step S4
If it is determined in step that the image to be taken next is not the reference image, the process directly proceeds to step S6. Then, in this step S6, it is determined whether or not the shooting of the subject (input of the subject image) has been completed. If it is determined that the shooting has not been completed, the process returns to step S3. On the other hand, if it is determined to end, the process proceeds to step S7. Here, the end of the photographing is determined according to the user's instruction, and the instruction is made when the setting in the photographing mode setting unit 19 is switched to the normal mode, or a switch provided for terminating the photographing is pressed. It can be.

Next, in step S7, a feature point is detected in the reference image, and in at least a part of the reference image, a corresponding point indicating the same point as the above-mentioned feature point is detected in an overlapping image. Such feature points and corresponding points are detected by the correspondence detecting unit 21 shown in FIG. The configuration and operation of the correspondence detection unit 21 are the same as those in the first embodiment.

Then, in step S8, parameters for correcting the tilt distortion of the reference image are calculated, and in step S9, an image in which the tilt distortion is corrected based on the above parameters is generated, and the operation ends. In addition,
The calculation of the parameters and the generation of the image in which the tilt distortion has been corrected are performed by the image distortion correction unit 22. The configuration and operation of the image distortion correction unit 22 are the same as those in the first embodiment.

As described above, according to the image processing apparatus 8 according to the third embodiment, the reference image designation value is set before photographing, so that the next image to be photographed by the user during image photographing by the user The user is notified by the notification unit 26 whether or not the reference image is taken, so that the user can easily recognize the shooting of the reference image. Then, the user can pay particular attention so that a desired area of the subject falls within the shooting range when the reference image is shot. Still further, it is possible to reduce shooting errors of the reference image.

It is to be noted that the technique of notifying the user of the photographing of the reference image according to the setting of the reference image designated value by the notifying unit 26 is also applicable to the image processing apparatus 6 according to the second embodiment. Not even. [Embodiment 4] The image processing apparatus according to the above-described embodiment needs to photograph a subject at least twice from different directions by moving a single optical system included in an imaging unit. In the image processing apparatus according to the present embodiment, two or more optical systems are arranged side by side in the imaging unit 41 so that a plurality of subject images photographed from different directions can be obtained by one photographing operation.

FIG. 26 is a diagram showing a configuration of an image processing apparatus 40 according to Embodiment 4 of the present invention. As shown in FIG. 26, the image processing device 40 according to Embodiment 4
It has the same configuration as the image processing apparatus 1 according to the first embodiment shown in FIG.
A and 11B are different.

Hereinafter, the operation of the image processing apparatus according to the present embodiment will be described with reference to the flowchart of FIG. First, the image processing device 40 is activated, and a photographing mode is selected. Here, the shooting modes include a normal mode for shooting a normal snapshot and a tilt correction mode for generating an image in which tilt distortion of a shot image is corrected. The photographing mode is selected by operating the photographing mode setting key by the user. Note that a shooting mode setting key 104 is provided as a shooting mode setting unit 19 in the main body of the image processing apparatus 40. However,
The shooting mode setting unit 19 may be configured by hardware or software provided separately from the main body.

As shown in FIG. 27, it is determined in step S1 whether or not the tilt correction mode is to be selected. If the tilt correction mode is not selected and the normal mode is selected, the process proceeds to step S10, where the user selects a desired mode. A snapshot of the subject is taken.

On the other hand, if the tilt correction mode is selected in step S1, the process proceeds to step S2, where a reference image as a target for correcting tilt distortion is set in the reference image setting section 20. Here, the reference image setting unit 20 includes an upward scroll key 201, a downward scroll key 202, and an enter key 203, as shown in FIG. In addition, a character string “reference image setting” is displayed on the display unit 17 as an overlay, and any optical system 1 is displayed.
The user is required to select whether to use the images captured in 1A and 11B as the reference images.

Then, the user operates the upward scroll key 2
By operating the “01” or the downward scroll key 202, for example, a pointer indicated by a triangle is moved on the display unit 17 to display “Camera 1” for selecting the optical system 11A or “Camera 2” for selecting the optical system 11B. ". When the enter key 203 is pressed while the pointer designates one of the optical systems, an image photographed by the designated optical system is used as the reference image. Here, the setting information of the reference image is supplied from the reference image setting unit 20 to the main control unit 14.

Next, in step S3, the user shoots a subject. At this time, the main controller 1 is displayed on the display 17.
If only the image captured by the optical system selected in step S2 is to be displayed by the control by step 4, it is determined by the user whether or not the image whose tilt is to be corrected includes a desired range of the subject. It is easily confirmed.

In step S4, a feature point is detected in the reference image, and a corresponding point indicating the same point as the feature point is detected in an image having an area overlapping with the reference image. In addition, detection of this feature point and corresponding point
This is executed by the correspondence detection unit 21.
The configuration and operation of the first embodiment are the same as those of the first embodiment.

Next, in step S5, a parameter for correcting the tilt distortion of the reference image is calculated, and in step S6, an image in which the tilt distortion has been corrected based on the parameter is generated, and the operation ends. here,
The calculation of the parameters and the generation of the image in which the tilt distortion has been corrected are executed by the image distortion correction unit 22. The configuration and operation of the image distortion correction unit 22 are the same as those in the first embodiment.

As described above, according to the image processing apparatus 40 according to the fourth embodiment, since the imaging section 41 has at least two optical systems, the user can perform the operation according to the above-described embodiment by performing a single photographing operation. A corrected image obtained by the image processing device can be obtained. In addition, if only the image captured by the selected optical system is displayed on the display unit 17, the user can easily confirm whether or not the image to be corrected for the tilt distortion includes a desired range of the subject. Since the user can do so, the user can pay more attention to the photographing of the reference image, and the possibility of photographing errors can be reduced.

The imaging section 41 shown in FIG. 26 may include three or more optical systems, and the imaging section including such a plurality of optical systems may be used in the image processing according to the second embodiment. It goes without saying that the device 6 may be provided.

In the image processing apparatuses according to all of the above-described embodiments, two or more sheets stored in a storage device such as a hard disk or a storage medium such as a CD-ROM are used instead of photographing a subject with an imaging unit. The subject images may be loaded into the external storage unit 18 or the like, and a corrected image may be generated using these subject images. Further, a shooting mode setting unit 19, a reference image setting unit 20, a correspondence detection unit 21, an image distortion correction unit 22
The case where the image synthesizing unit 24 and the like are housed in a housing different from the imaging units 11 and 41, for example, a computer or the like, is also considered in the same manner as in the above embodiment.

Although the correspondence detection unit 21 has been described as detecting corresponding points by density matching by the correlation method, it may be performed by another method such as a spatiotemporal differentiation method. Further, the calculation of the tilt distortion parameter is not limited to the coordinate transformation shown in the above equation (20), and another parameter calculation method may be applied. [Embodiment 5] In the above-described embodiment, a coordinate conversion parameter is calculated for a reference image to be subjected to image processing, and a distortion-corrected image is generated by using an interpolation operation.
It is preferable that the reference image is an image that includes abundant document information and has little deterioration of the image due to the interpolation operation, that is, an image that is photographed while relatively facing the subject. Therefore, hereinafter, an image processing method for automatically selecting an optimal reference image and an image processing apparatus for executing the method will be described.

FIG. 29 is a diagram showing a first configuration example of the image processing apparatus according to the fifth embodiment of the present invention. FIG.
As shown in FIG. 9, the image processing apparatus 50 according to the fifth embodiment has the same configuration as the image processing apparatus 1 according to the first embodiment shown in FIG. Instead, the subject area determination unit 25 and the reference image automatic selection unit 2
7 is provided.

Here, the subject area determination section 25 and the reference image automatic selection section 27 are connected to the main control section 14, respectively, and the output terminal of the subject area determination section 25 is connected to the reference image automatic selection section 27. The output end of the reference image automatic selection unit 27 is connected to the image distortion correction unit 22.

Note that the image processing apparatus 50 according to the fifth embodiment can also have the same configuration as the image processing apparatus 1 according to the first embodiment, as shown in FIG.

Hereinafter, the operation of the image processing apparatus 50 according to the fifth embodiment will be described with reference to the flowchart shown in FIG. Here, since the image processing device 50 according to the fifth embodiment operates in the same manner as the image processing device 1 according to the first embodiment, the following description will focus on the differences.

First, in step S1, it is determined whether or not the tilt correction mode is selected by the user. If the tilt correction mode is not selected and the normal mode is selected, the process proceeds to step S10, and the user selects a desired snap mode. A picture is taken.

On the other hand, when the tilt correction mode is selected by the user in step S1, step S2
Proceed to. Then, in step S2, the imaging unit 11
Thus, the subject image photographed at least twice is taken into the frame memory 15. At this time, each image needs to be photographed such that a part of the subject image overlaps with each other.

Next, in step S3, it is determined whether or not the input of the subject image has been completed. If the input has not been completed, the process returns to step S2, and the input of the subject image is continued. On the other hand, if the input of the subject image has been completed by the user's instruction to end the shooting, the process proceeds to step S4.

Then, in step S4, the correspondence detection unit 2
A feature point is detected in the subject image by 1 and a corresponding point indicating the same place as the feature point is detected in a reference image overlapping at least a part of the subject image. Next, when the detection of the feature point and the corresponding point is completed, the operation for correcting the tilt distortion starts below. First, in step S5, the reference image automatic selection unit 27 automatically selects a reference image to be subjected to tilt distortion correction.
Hereinafter, the reference image automatic selection unit 27 will be described in detail.

As in the case of the image processing apparatus 1 according to the first embodiment, in step S6, step S5
In step S7, a parameter for correcting the tilt distortion is calculated for the selected reference image, and in step S7, an image in which the tilt distortion of the image is corrected is created based on the parameter, and the operation ends.

When correcting the tilt distortion, as the reference image, the object plane is shown over the widest region among the plurality of images taken, and the amount of document information required by the user is abundant. Preferably, an image is selected. In addition, it is preferable to select an image in which the angle formed by the imaging surface with respect to the object plane during photographing (hereinafter, also referred to as “tilt angle”) is as small as possible. The reason will be described with reference to FIG.

The image distortion correction section 22 shown in FIG.
As described above, the tilt correction is performed by executing the coordinate conversion based on the equation (25), but the correction operation changes according to the tilt angle φ. Here, for the sake of simplicity, it is assumed that the tilt angle φ is limited around the y-axis, and that the size of the projection surface 31 parallel to the subject surface is equal to the size of the imaging surface 32.

As shown in FIG. 31 (a), in the coordinate conversion when the tilt angle φ is relatively small,
A point near the left end of the imaging surface 32 is projected onto the projection surface 31 by variation indicated by a vector toward the origin o. Note that the resolution of the subject image is reduced in the hatched portion in the projection target surface 31 due to such coordinate conversion.

On the other hand, as shown in FIG. 31B, when the tilt angle φ is relatively large in the coordinate conversion,
A point near the left end of the imaging surface 32 involves a variation larger than the variation shown in FIG. 31A, that is, the projection surface 31 such that the position vector with respect to the origin o is further reduced.
Projected to Then, it can be seen that the area where the resolution of the subject image is reduced by the coordinate conversion is larger than that in the case of FIG.

Accordingly, it is understood from the above that the smaller the tilt angle φ is, the less the resolution is deteriorated due to the coordinate transformation by the image distortion correction unit 22.

The subject area determination unit 25 shown in FIG. 29 detects the area occupied by the subject in the photographed image. For example, see the document “Image Processing and Recognition” (Takeharu Yasui and Tomoharu Nagao, Shoko Temple),
(A) a method of performing clustering on an image such as a region growing method or a region division method, (b) a method of performing clustering on a feature space such as region division by a histogram,
(C) a method using edges in an image, such as contour tracing,
(D) An area division method such as texture analysis is applied. Then, the subject area determination unit 25 selects an image obtained by photographing a wide range of the subject as the reference image according to the determination result. As a result, an image including abundant document information can be automatically selected as the reference image.

Here, the image processing apparatus according to Embodiment 5 of the present invention may have a configuration as shown in FIG. That is, the image processing device 51 shown in FIG.
Has a configuration similar to that of the image processing apparatus 50 shown in FIG. 29, but differs in that a linear pattern detection unit 28 is provided instead of the subject area determination unit 25.

Generally, on a subject surface such as a document, there are many linear patterns such as character strings and ruled lines which are in parallel with each other. However, when the tilt angle at the time of photographing is large, linear patterns that should be originally parallel are projected on the image as linear patterns having different directions. Therefore, by examining the variation in the direction of the linear pattern projected on the image, the magnitude of the tilt angle can be determined, and the image with the small tilt angle can be automatically selected as the reference image.

Here, the linear pattern detecting section 28 shown in FIG. 32 performs a process of detecting a linear pattern in a plurality of captured images. An example of a method for detecting the linear pattern will be described below.

First, an edge image is created by taking a derivative of the plurality of images. Next, the fragmentary edge point group in the created edge image is divided into linear segments, and each linear segment is applied to a linear equation represented by the following equation (28). ax + by + c = 0 (a ² + b ² = 1) (28) Here, the above equation (28) is applied by applying the least square method using the point group forming the linear segment. Then, when the fitting of the straight line to the straight segment is completed, the variation of (a, b) which is the parameter indicating the direction of the straight line is obtained. In this way, by obtaining the variation of the parameter (a, b) for all of the plurality of images, it is possible to know the variation in the direction of the straight line in each image. Then, the image having the smallest variation in the direction of the straight line can be automatically selected as the reference image.

In the automatic selection of the reference image, the following method can be applied. First, an edge image is created for the plurality of images in the same manner as described above. Next, Hough is applied to all the points in the edge image.
Perform the conversion. Here, the Hough transform is a mathematical transform used to detect a straight line from a fragmentary edge,
This is a method of performing clustering in a space formed by parameters of an expression representing a line to be detected. More specifically, the point sequence on the image is projected onto a θ-ρ space indicated by an angle θ forming the x-axis and a length ρ of the straight line.

FIG. 33 is a diagram for explaining the Hough transform from the image space shown in FIG. 33 (a) to the θ-ρ space shown in FIG. 33 (b). As shown in FIG.
The points P1 to P3 on the image are converted into corresponding curves L1 to L3 by the h conversion. When such a conversion is performed for all points on the image, a point CP where the locus of the curve is concentrated in the θ-ρ space occurs.
This point CP corresponds to a straight line passing through many edge points in the edge image. Here, the coordinates of this point CP are represented by (θ,
ρ), a straight line corresponding to the following equation (29) is detected. ρ = xcos θ + ysin θ (29) Then, for all of the plurality of images, a large number of points CP at which the curve is concentrated are extracted, and variations in θ at these points are obtained, whereby variations in the direction of the straight line in each image are known. be able to. In this way, the image having the smallest variation in the direction of the straight line may be automatically selected as the reference image.

FIG. 34 is a diagram showing a third configuration example of the image processing apparatus according to the fifth embodiment of the present invention. FIG.
As shown in FIG. 4, the image processing apparatus according to the fifth embodiment includes the subject area determination unit 25 shown in FIG.
The reference image automatic selection unit 27 may be connected to the correspondence detection unit 21 without including the linear pattern detection unit 28 shown in FIG.

The set of feature points and corresponding points detected by the correspondence detection section 21 is used for calculating a projection transformation matrix B, that is, a parameter for correcting tilt distortion. At this time, generally, the parameter is calculated with higher accuracy as the number of sets of the feature points and the corresponding points is larger and the parameters are more widely dispersed. Therefore, for a plurality of captured images, a set of feature points and corresponding points detected by the correspondence detection unit 21 and a variance in the images are checked, and an image having the maximum value is automatically determined as a reference. It may be selected as an image.

In the correspondence detection based on the above-described correlation method, generally, the more feature patterns in an image, the more feature points and corresponding points are detected. And since there are many patterns with features, it is highly likely that many useful document information for the user is included in the image, so if an image rich in feature points and corresponding points is automatically selected as a reference image, It is preferable that an image including necessary document information can be subjected to the tilt distortion correction.

FIG. 35 is a diagram showing a fourth configuration example of the image processing apparatus according to the fifth embodiment of the present invention. FIG.
As shown in FIG. 5, the image processing apparatus according to the fifth embodiment includes the subject area determination unit 25 shown in FIG.
Instead of the linear pattern detection unit 28 shown in FIG. 2, a plane measurement unit 29 may be provided.

Here, the plane measuring section 29 measures the direction of the subject with respect to the image pickup section 11 at the time of photographing each image. FIG. 36 is a diagram illustrating a configuration example of the plane measurement unit 29 illustrated in FIG. As shown in FIG. 36, the plane measuring unit 29 includes a spot light source 271, a light receiving element 272,
A dimension coordinate calculator 273 and a plane calculator 274;
The spot light source 271 includes a light source 271a including a light emitting diode or a semiconductor laser, a scanning mirror 271b such as a polygon mirror, and a driving unit 271c for controlling the movement of the scanning mirror 271b.

Here, in the spot light source 271, the scanning mirror 271b is driven by the driving unit 27 so that the spot light generated by the light source 271a impinges on the object plane PL.
1c. The light receiving element 272 is a PSD (Position sensitive detector) installed at a location where the position with respect to the spot light source 271 is measured in advance.
And a photoelectric conversion element such as a CCD.
The direction of the reflected light from L is detected. Note that the photoelectric conversion element 114 included in the imaging unit 11 may be used as the light receiving element 272 in the above.

The three-dimensional coordinate calculator 273 determines the direction of the spot light emitted by the spot light source 271, the positional relationship between the spot light source 271 and the light receiving element 272, and the direction of the reflected light detected by the light receiving element 272. Then, the three-dimensional coordinates (X, Y, Z) of the subject plane PL with respect to the image processing device 53 are calculated by using the principle of triangulation.
Then, the plane calculation unit 274 includes a three-dimensional coordinate calculation unit 273.
The plane equation is estimated using the three-dimensional coordinates of three or more points that are not on the same straight line calculated by the above. For example, the plane equation to be obtained is set as aX + bY + cZ + d = 0 (a ² + b ² + c ² = 1, c> 0) (30), and the four parameters (a , B, c, d) are calculated by the least squares method. As a result, the tilt angle φ is calculated by the following equation (31). φ = cos ⁻¹ c (31) Accordingly, the orientation of the object plane is measured for all of the plurality of captured images, and the image having the smallest tilt angle φ may be automatically selected as the reference image.

In addition, as described in the description of the image distortion correction unit 22, the orientation of the subject can be obtained by using a set of the feature points and the corresponding points detected by the correspondence detection unit 21. Therefore, in the image processing apparatus according to the fifth embodiment, the image distortion correction unit 22 once executes Expressions (14) to (18) to determine the direction of the subject, and uses the obtained result as the reference image automatic selection unit. 27. [Sixth Embodiment] The following description is based on the premise that, as shown in FIG. 18, still images on the same subject plane PL are photographed from a plurality of directions so as to partially overlap each other in a part of the image. An embodiment will be described. Here, as shown in FIG. 18, the image imj (1 ≦ j ≦ K) is obtained by photographing from the direction d _j,
For example, an image imj and an image i obtained by shooting in adjacent directions like an image im1 and an image im2
It is assumed that there is an overlap area between m (j + 1) (1 ≦ j ≦ K−1). Then, in the image processing apparatus according to the sixth embodiment, another image is pasted so as to match any one of the images selected as the reference image, and a combined image is generated.

FIG. 37 is a diagram showing a configuration of an image processing apparatus 60 according to Embodiment 6 of the present invention. As shown in FIG. 37, the image processing device 60 according to Embodiment 6
It has the same configuration as the image processing device 6 according to the second embodiment shown in FIG. 19, but differs in that a reference image automatic selection unit 27 is provided instead of the reference image setting unit 20.

Embodiment 6 Having the Configuration As Above
Operates in the same manner as the image processing apparatus 6 according to the second embodiment, but the operation of the image processing apparatus 60 according to the sixth embodiment will be described below with reference to the flowchart shown in FIG. The differences will be mainly described with reference to FIG.

First, in step S1, it is determined whether or not the user selects the image combining mode. If the image combining mode is not selected and the normal mode is selected, the process proceeds to step S10, and the user selects a desired snap mode. A picture is taken.

On the other hand, when the image synthesis mode is selected by the user in step S1, the process proceeds to step S2. Then, in step S2, a plurality of subject images photographed by the imaging unit 11 are loaded into the frame memory 15. At this time, each image needs to be photographed such that a part of the subject image overlaps with each other.

Next, in step S3, it is determined whether or not the input of the subject image has been completed. If the input has not been completed, the process returns to step S2, and the input of the subject image is continued. On the other hand, if the input of the subject image has been completed by the user's instruction to end the shooting, the process proceeds to step S4.

Then, in step S4, the correspondence detection unit 2
1, a feature point is detected between the image imj and the image im (j + 1), and in the image im (j + 1) overlapping at least a part of the image imj, a corresponding point indicating the same location as the above feature point Is detected.

Next, when the detection of the feature point and the corresponding point is completed, an operation for generating a composite image in which a plurality of images are combined will be described below. First, in step S5, the reference image automatic selection unit 27 automatically selects a reference image to be a reference in the synthesis. Here, the reference image automatic selection unit 27 automatically selects an image having the smallest possible tilt angle as a reference image. By making such a selection, it is possible to obtain a composite image with small tilt distortion.

In step S6, a projective transformation matrix for performing composition based on the reference image selected in step S5 is calculated, and in step S7, a combined image is generated using the projective transformation matrix. End the operation. [Seventh Embodiment] FIG. 39 is a diagram showing a configuration of an image processing apparatus according to a seventh embodiment of the present invention. As shown in FIG. 39, the image processing device 70 according to the seventh embodiment includes:
It has the same configuration as the image processing apparatus 1 according to the first embodiment shown in FIG. 3, but differs in that it further includes a reference image automatic selection unit 27 and a switching unit 45.

Here, the reference image automatic selection section 27 is controlled by the main control section 14. The switching unit 45 has an input terminal whose reference end is the reference image setting unit 20 and the reference image automatic selection unit 2.
7, and an output terminal is connected to the image distortion correction unit 22.

The switching section 45 switches the method of selecting the reference image, and includes an upward scroll key 201, a downward scroll key 202, and an enter key 203, as shown in FIG. Here, when the user selectively presses a menu key (not shown) displayed on the display unit 17, a screen on which the character “selection of reference image” is overlaid as shown in FIG. 40 is displayed. Displayed on the display unit 17.

When the user operates the upward scroll key 201 or the downward scroll key 202, the triangular cursor shown in FIG. 40 moves up and down. At this time, when the enter key 203 is pressed while the cursor is pointing to the character “AUTO”, it is determined that the reference image is automatically selected, and FIG.
Are activated selectively. On the other hand, when the enter key 203 is pressed while the cursor is pointing to the character "MANUAL", it is determined that the reference image is to be manually set, and the reference image setting unit 20 shown in FIG. Be activated.

Therefore, according to the image processing apparatus of the seventh embodiment, either the reference image setting section 20 or the reference image automatic selection section 27 is selectively switched by the switching section 45 to the image distortion correction section. 22, the user can arbitrarily select either automatic or manual as a method of selecting the reference image.

In any of the above-described embodiments, the image processing method according to the embodiment can be described as a computer program. And FIG.
As shown in FIG. 1, the image processing can be easily realized by mounting the recording medium 301 storing the program in the image processing apparatus 1 and causing the image processing apparatus 1 to execute the program.

As shown in FIG. 42, the image processing is also realized by mounting the CD-ROM 302 storing the program on a personal computer (PC) PC and executing the program on the PC. be able to. The recording medium mounted on the personal computer PC and storing the program is the CD-R described above.
It goes without saying that the present invention is not limited to the OM 302 but may be, for example, a DVD-ROM.

Hereinafter, a specific example in the case where the above-described image processing method is realized by executing the program will be described.
In this case, the image processing apparatus 1 and the personal computer PC include a storage device such as a built-in memory and a hard disk and a C
Two or more subject images stored in a recording medium such as a D-ROM are loaded into the signal processing unit 12 via various interfaces mounted on a computer.

The setting of the photographing mode in the above is as follows.
Press a predetermined key on the keyboard of the personal computer PC,
Alternatively, this is performed by clicking a mouse on an icon displayed on the screen. On the other hand, when manually selecting a reference image, a plurality of input images are thinned out and displayed on a screen, and an up / down cursor key on a keyboard used for a calculator is pressed, or an icon is clicked with a mouse. Thus, the thinned image is selected. Then, when a line feed key is pressed while a desired image is selected, the image is set as a reference image.

As for the focal length f of the optical system constituting the image pickup unit 11, the focal length of the optical system is measured in advance and recorded in the recording medium. Then, on the screen, the user selects the focal length of the optical system used at the time of actual shooting. Here, the focal length can be recorded as header information. That is, for example, when the Exif format is used as the image data, the focal length at the time of shooting can be recorded as the header information. Then, the image processing apparatus according to the present embodiment can obtain the focal length by reading the header information. If the above method cannot be performed, a dialog box or the like may be displayed on a screen of a personal computer PC or the like, and the user may manually input the focal length directly in the dialog box.

The orientation of the subject in the above can be handled in the same manner as the focal length. That is, the orientation of the subject measured by the plane measurement unit 29 or the image distortion correction unit 22 is recorded in advance as header information in the image data file, and the image processing apparatus according to the present embodiment reads the header information. Thus, the orientation of the subject may be obtained.

At this time, for example, when the Exif format is used as image data, there is no field for recording the direction of the subject, but the field called Maker Note, which can be freely used by the manufacturer, is used to record the direction of the subject. Can be recorded.

If the above method cannot be used, a dialog box or the like is displayed on the screen of a personal computer PC or the like, as in the case of the focal length, so that the user can directly orient the subject. It is good to input.

The above embodiment of the present invention can be applied to image processing such as input of paper surface information using a digital still camera or digital video camera, bonding and combining of divided images, and a non-contact handy scanner or other It can also be applied to imaging equipment.

As described above, according to the present invention, in an image processing method for correcting distortion of an image photographed from a plurality of directions so that at least a part of the image overlaps with a subject,
By selecting a target for which distortion is to be corrected from among a plurality of images, an optimum image for obtaining an appropriate image can be set as a correction target, so that more accurate correction can be realized.

Further, by selecting an image having the least distortion from a plurality of images and combining the image with the corrected distortion with the selected image, a more appropriate synthesized image can be obtained.

Here, if an object to be corrected is automatically selected in accordance with the size of the area occupied by the subject in the image, the distortion of the image requiring the largest amount of document information is automatically corrected. Therefore, an appropriate image can be reliably obtained.

If an object to be corrected is automatically selected according to the direction of a linear pattern detected in an image, an image photographed from a position almost directly facing the subject is automatically set as an object to be corrected. Therefore, an appropriate image having a high resolution can be obtained.

Further, by automatically selecting an object to be corrected in accordance with the specified correspondence, highly accurate correction can be reliably performed, so that the reliability of image processing can be improved.

If the object to be corrected is automatically selected according to the direction of the subject detected for each photographing, an image photographed from a position almost directly facing the subject is automatically set as the correction object. Therefore, an appropriate image having a high resolution can be obtained.

Further, according to the image processing apparatus provided with the notifying means for notifying the user that the image to be taken next will be the reference image for correcting the distortion, the reference value is used for correcting the distortion. Attention is drawn to the user in capturing the image to be taken, so that a capturing error due to the carelessness of the user can be avoided, and the reliability of operation and the quality of the corrected image can be improved.

According to the image processing apparatus provided with a plurality of optical means for simultaneously photographing a subject and a selecting means for selecting an image to be corrected from among the images photographed by the plurality of optical means, Since a plurality of subject images taken from a plurality of directions can be obtained by photographing, the operation can be simplified by reducing the number of times of photographing required for correcting the distortion, and the distortion can be more easily obtained. A corrected image can be obtained.

[Brief description of the drawings]

FIG. 1 is a diagram for explaining a problem in a conventional technique.

FIG. 2 is a diagram for explaining an image processing method and an image processing apparatus according to Embodiment 1 of the present invention.

FIG. 3 is a diagram showing a configuration of an image processing apparatus according to Embodiment 1 of the present invention.

FIG. 4 is a perspective view illustrating the image processing apparatus illustrated in FIG. 3;

FIG. 5 is a flowchart illustrating an operation of the image processing apparatus according to the first embodiment of the present invention.

6 is a diagram showing a display example at the time of shooting on the display unit shown in FIG. 3;

FIG. 7 is a diagram showing a layout of a reference image setting unit shown in FIG. 3;

FIG. 8 is a diagram showing another example of the image processing device according to the first embodiment of the present invention.

FIG. 9 is a diagram illustrating a configuration of a correspondence detection unit illustrated in FIG. 3;

FIG. 10 is a diagram illustrating the operation of the correlation operation unit shown in FIG.

FIG. 11 is a diagram illustrating a configuration of an image distortion correction unit illustrated in FIG. 3;

FIG. 12 is a first diagram illustrating an operation of the image distortion correction unit illustrated in FIG. 3;

FIG. 13 is a diagram illustrating an optical system of the imaging unit illustrated in FIG.

FIG. 14 is a diagram illustrating a projective transformation by the three-dimensional operation unit shown in FIG. 11;

FIG. 15 is a second diagram illustrating an operation of the image distortion correction unit illustrated in FIG. 3;

FIG. 16 is a first diagram illustrating an operation of the parameter calculation unit illustrated in FIG. 11;

FIG. 17 is a second diagram illustrating the operation of the parameter calculator shown in FIG. 11;

FIG. 18 is a diagram for explaining an image processing method and an image processing apparatus according to Embodiment 2 of the present invention.

FIG. 19 is a diagram showing a configuration of an image processing apparatus according to Embodiment 2 of the present invention.

FIG. 20 is a flowchart showing an operation of the image processing apparatus according to the second embodiment of the present invention.

FIG. 21 is a diagram illustrating a configuration of an image combining unit illustrated in FIG. 19;

FIG. 22 is a diagram showing a configuration of an image processing device according to a third embodiment of the present invention.

FIG. 23 is a flowchart showing an operation of the image processing apparatus according to Embodiment 3 of the present invention.

FIG. 24 is a diagram illustrating setting of a reference image in a reference image setting unit illustrated in FIG. 22;

25 is a diagram illustrating the operation of the notification unit shown in FIG.

FIG. 26 is a diagram showing a configuration of an image processing apparatus according to Embodiment 4 of the present invention.

FIG. 27 is a flowchart showing an operation of the image processing apparatus according to Embodiment 4 of the present invention.

FIG. 28 is a diagram illustrating the setting of a reference image for a reference image setting unit illustrated in FIG. 26;

FIG. 29 is a diagram illustrating a first configuration example of the image processing apparatus according to Embodiment 5 of the present invention;

FIG. 30 is a flowchart showing an operation of the image processing apparatus according to Embodiment 5 of the present invention.

FIG. 31 is a diagram for explaining a tilt angle dependency in resolution degradation.

FIG. 32 is a diagram illustrating a second configuration example in the image processing device according to the fifth embodiment of the present invention;

FIG. 33 is a diagram illustrating Hough transform used in the image processing method according to the fifth embodiment of the present invention.

FIG. 34 is a diagram illustrating a third configuration example in the image processing device according to the fifth embodiment of the present invention;

FIG. 35 is a diagram illustrating a fourth configuration example of the image processing device according to the fifth embodiment of the present invention;

FIG. 36 is a diagram illustrating a configuration of a plane measurement unit illustrated in FIG. 35;

FIG. 37 is a diagram showing a configuration of an image processing device according to Embodiment 6 of the present invention.

FIG. 38 is a flowchart showing the operation of the image processing apparatus according to Embodiment 6 of the present invention.

FIG. 39 is a diagram showing a configuration of an image processing apparatus according to Embodiment 7 of the present invention.

40 is a diagram illustrating the operation of the switching unit shown in FIG. 39.

FIG. 41 is a diagram illustrating an image processing apparatus and a recording medium according to an embodiment of the present invention.

FIG. 42 is a diagram showing a computer and a computer-readable recording medium according to the embodiment of the present invention.

[Explanation of symbols]

1, 2, 6, 8, 40, 50 to 53, 60, 70 Image processing device 3, 4, 10, IM1 to IM3, im1, im2, im
j, imk image 5 distortion-corrected image 7 reference image 9 reference image 11, 41 imaging unit 11A, 11B optical system 12 signal processing unit 13 memory control unit 14 main control unit 15 frame memory 16 interface (I / F) 17 display unit 18 External storage unit 19 Shooting mode setting unit 20 Reference image setting unit 21 Correspondence detection unit 22 Image distortion correction unit 23 Subject area determination unit 24 Image synthesis unit 25 Subject area determination unit 26 Notification unit 27 Reference image automatic selection unit 28 Linear pattern detection Unit 29 plane measurement unit 30 image plane 31 projection plane 32 imaging plane 33 device coordinate system 35 subject image 45 switching unit 101 power switch 102 shutter 103 finder 104 shooting mode setting key 111 lens 112 aperture 113 shutter 114 photoelectric conversion element 115 preprocessing Part 201 top Direction scroll key 202 Downward scroll key 203 Enter key 204 Cursor key 211 Feature point setting unit 212 Correlation calculation unit 213 Feature point 215, 216 Correlation window 217 Corresponding point 221 3D calculation unit 222 Parameter calculation unit 223 Coordinate conversion unit 224 Image plane 231 Projection conversion calculation unit 232 Coordinate conversion unit 241 Finder 242 Indicator 271 Spot light source 271a Light source 271b Scanning mirror 271c Driving unit 272 Light receiving element 273 Three-dimensional coordinate calculation unit 274 Flat surface calculation unit 301 Recording medium 302 CD-ROM PL Object plane PC Personal computer _{(PC) D 1 ~D 3, d 1} , d 2, d j, d k direction IMA, IMB, IMC composite image

 ────────────────────────────────────────────────── ─── Continuing on the front page (72) Inventor Shin Aoki 1-3-6 Nakamagome, Ota-ku, Tokyo F-term in Ricoh Co., Ltd. 5B057 BA02 CD12 CD20 CE10 5C023 AA10 AA11 AA31 AA34 AA37 CA01 DA01 EA03 5C076 AA12 AA23 AA40 BA01 BA06

Claims (16)

[Claims] 1. An image processing method for correcting a distortion of an image photographed from a plurality of directions so that at least a part of the image overlaps with a subject, wherein an overlapping portion in the plurality of images obtained by the photographing is provided. A first step of identifying the correspondence of the; and a second step of selecting a target for correcting the distortion from among the plurality of images, and according to the correspondence identified in the first step, A third step of correcting the distortion of the image selected in the second step. 2. The image processing method according to claim 1, wherein in the second step, the correction target is automatically selected in accordance with a size of a region occupied by the subject in the image. 3. The image processing method according to claim 1, wherein in the second step, the correction target is automatically selected according to a direction of a linear pattern detected in the image. 4. The image processing method according to claim 1, wherein in the second step, the correction target is automatically selected in accordance with the correspondence specified in the first step. 5. The image processing method according to claim 1, wherein in the second step, the correction target is automatically selected in accordance with the direction of the subject detected for each photographing. 6. An image processing method for correcting a distortion of an image photographed from a plurality of directions so that at least a part of the image overlaps with a subject, wherein an overlapping portion in the plurality of images obtained by the photographing is provided. A first step of specifying each of the correspondence relationships; a second step of selecting the image with the least distortion from the plurality of images; and a response to the correspondence relationship specified in the first step. And correcting each of the distortions of the plurality of images,
A third step of combining with the image selected in the second step. 7. An image processing apparatus for correcting distortion of an image photographed from a plurality of directions so that at least a part of the image overlaps with a subject, wherein an overlapping portion in the plurality of images obtained by the photographing is provided. Correspondence detecting means for detecting a correspondence relationship between the plurality of images; selecting means for selecting the image having the least distortion from the plurality of images; and An image processing apparatus comprising: an image synthesizing unit that corrects the distortion of an image and synthesizes the image with the image selected by the selecting unit. 8. An image processing apparatus for correcting distortion of an image photographed from a plurality of directions so as to at least partially overlap a subject, wherein the distortion is corrected among a plurality of the images to be photographed from now on. Selecting means for pre-selecting the image to be used as a reference when notifying, and notifying to the user that the image to be taken next will be the reference image in accordance with the selection made by the selecting means Notification means; correspondence detection means for detecting a correspondence relationship between the reference image obtained by photographing and another image; and the correspondence detected by the correspondence detection means. An image processing apparatus comprising: a correction unit configured to correct the distortion of the image serving as the reference in accordance with the relationship. 9. An image processing apparatus for correcting distortion of an image photographed from a plurality of directions so that at least a part of the image overlaps with a subject, the image processing apparatus comprising: an overlapping portion in the plurality of images obtained by the photographing; A correspondence detecting means for detecting a correspondence relationship between the plurality of images; a selection means for selecting an object for which the distortion is to be corrected from the plurality of images; and the selection means according to the correspondence detected by the correspondence detection means. An image processing apparatus comprising: a correction unit configured to correct the distortion of the selected image. 10. An image processing apparatus for correcting distortion of an image photographed from a plurality of directions so that at least a part thereof overlaps a subject, wherein: a plurality of optical means for photographing the subject simultaneously; Selecting means for selecting the image to be corrected from among the images photographed by the optical means, and a correspondence relationship between overlapping portions between the image selected by the selecting means and another image. An image processing apparatus comprising: a correspondence detection unit for detecting; and a correction unit for correcting the distortion of the selected image in accordance with the correspondence detected by the correspondence detection unit. 11. The image processing apparatus according to claim 9, wherein the selection unit automatically selects the correction target in accordance with a size of a region occupied by the subject in the image. 12. The image processing apparatus according to claim 9, wherein the selection unit automatically selects the correction target according to a direction of a linear pattern detected in the image. 13. The image processing apparatus according to claim 9, wherein the selection unit automatically selects the correction target according to the correspondence detected by the correspondence detection unit. 14. The image processing apparatus according to claim 9, wherein the selection unit automatically selects the correction target according to the orientation of the subject detected for each photographing. 15. A computer-readable recording medium on which a program for correcting, by a computer, distortion of an image photographed from a plurality of directions so that at least a part of the image overlaps with a subject is recorded. Causing the computer to specify a correspondence between overlapping portions in the plurality of images obtained by the photographing, to select a target to correct the distortion from among the plurality of images, A computer-readable recording medium for correcting the distortion of the selected image according to 16. A computer-readable recording medium in which a program for correcting, by a computer, distortion of an image photographed from a plurality of directions so that at least a part thereof overlaps with a subject is recorded, wherein the program is Causing the computer to specify the correspondence between overlapping portions in the plurality of images obtained by the photographing, and to select the image with the least distortion from among the plurality of images; A computer-readable recording medium, wherein the distortion of each of the plurality of images is corrected according to the correspondence, and the images are combined with the selected image.