JP2024021204A - Image processing device, imaging device, image processing method, program and recording medium - Google Patents

Abstract

[Problem] In a method for obtaining an image equivalent to imaging with an aperture larger than the aperture of an imaging device from a plurality of images taken at different positions, the focus position (focus position) and the distribution of the focal position within the screen ( It was not possible to determine this independently of the focus surface shape (focus surface shape).
SOLUTION: The image processing device includes a first calculation unit that calculates a projective transformation amount for a plurality of images captured from a plurality of viewpoints using a plurality of first points in the plurality of images; a second calculation means for calculating a translation amount using at least one second point different from the first point in the plurality of images; and a second calculation means for calculating the translation amount using the at least one second point different from the first point in the plurality of images; and a deforming means for deforming the object.
[Selection diagram] Figure 4

Description

The present invention relates to an image processing device, and more particularly to an image processing device that performs compositing of a plurality of images.

In recent years, a method has been developed in which a plurality of images captured by an imaging device are combined to obtain characteristics that cannot be obtained with a single image. Patent Document 1 discloses a technique for adjusting a focusing range by combining a plurality of images taken at different positions.

As described in Patent Document 1, optical characteristics (blur, resolution) equivalent to a system with a large aperture can be obtained by combining a plurality of images taken at different positions. In this way, an image equivalent to that of a large diameter optical system can be obtained with a small diameter optical system, making it possible to achieve higher resolution, smaller size, and lower cost.

JP2012-253444A

However, because the feature points used to obtain the amount of image deformation and the feature points used to obtain the amount of image translation were common, the degree of freedom in setting the focus position and focus plane was limited, and the focus The degree of freedom in adjusting the range is limited.

The present invention has been made in view of the above problems, and an object of the present invention is to provide an image processing device that can independently calculate the focus position and the shape of the focus surface.

The present invention provides a first calculation means for calculating a projective transformation amount using a plurality of first points in the plurality of images, and a second calculation means for calculating a translation amount using at least one second point different from the first point; and deforming the plurality of images using the projective transformation amount and the translation amount. An image processing apparatus is provided, characterized in that it has a deformation means for performing the deformation.

According to the present invention, in a method for obtaining an image equivalent to imaging with an aperture larger than the aperture of an imaging device from a plurality of images taken at different positions, the in-focus position and the distribution of focal positions within the screen are Can be decided individually.

FIG. 6 is a diagram for explaining enlargement of an aperture by combining images captured from a plurality of viewpoints in an embodiment of the present invention. FIG. 2 is a diagram for explaining the arrangement of a subject and an imaging device in an embodiment of the present invention. FIG. 3 is a diagram for explaining a focus position and a state of a focus surface in an embodiment of the present invention. FIG. 3 is a diagram for explaining the relationship between the focus position and the shape of the focus surface in the embodiment of the present invention. 1 is a diagram for explaining each part of an imaging device in an embodiment of the present invention. FIG. It is a flowchart for explaining the operation in the first embodiment of the present invention. It is a flowchart for explaining another operation in a 1st embodiment of the present invention. It is a flow chart for explaining calculation of a translation amount in a 2nd embodiment of the present invention.

DESCRIPTION OF THE PREFERRED EMBODIMENTS Preferred embodiments of the present invention will be described below with reference to the drawings. Note that the embodiments described below are merely examples for carrying out the present invention, and the present invention is not limited to the following embodiments.

(First embodiment)
<About aperture synthesis>
First, the principle of a method for synthesizing images captured from a plurality of viewpoints to obtain an image equivalent to imaging with an aperture larger than the aperture of the imaging device will be explained.

FIG. 1 is a diagram for explaining enlargement of an aperture by combining images taken from a plurality of viewpoints in this embodiment. A situation will be described when subjects 101 and 102 having different subject distances are imaged from different positions as shown in FIG. 1. FIG. 1 is a diagram showing the positional relationship between an imaging device and a subject when viewed from above. In the imaging device shown in FIG. 1, only lenses 103 and 104 and imaging elements 105 and 106 are depicted, and other structures are omitted.

An image captured when the imaging device is located at position 107 is referred to as image 109, and an image captured when the imaging device is located at position 108 is designated as image 110. In each image, it is assumed that the subject 101 and the subject 102 are within the depth of field. Since the subject 101 is located in front of the subject 102 when viewed from the imaging device, when the imaging device is at position 107, the distance between the subjects 101 and 102 is wider than when the imaging device is at position 108, and the distance between the subjects 101 and 102 is recorded in the image. has been done.

When image 109 and image 110 are translated and combined so that object 101 overlaps, object 102 does not overlap as shown in image 111. Even if subjects 101 and 102 are within the depth of field in the image before addition, the image of subject 102 after addition is shifted, and subject 102 becomes blurred in image 111.

In other words, the above technique can obtain an image that appears to have a shallow depth of field because the aperture of the imaging lens is enlarged by moving the imaging device and combining the captured images.

Focusing on the above phenomenon in terms of the overlapping state of images and blurring, when combining multiple images, the position of the subject image matches and the overlapping part is in focus, and the position of the subject image shifts. The overlapping parts become blurred.

FIG. 2 is a diagram for explaining the arrangement of a subject and an imaging device in this embodiment. In order to easily recognize the focus plane, as shown in FIG. 2, an example is shown in which the subject 201 is placed at an angle with respect to the imaging device, and the subject 202 is placed in front of the subject 201. It will be explained as follows. Regarding the imaging device, only a lens 203 and an imaging element 204 are shown, and the housing of the imaging element is indicated at positions 205 to 207.

FIG. 3 is a diagram for explaining the focus position and the state of the focus surface in this embodiment. Images captured by the imaging device at positions 205 to 207 are images 301 to 303 in FIG. 3. At position 206, the imaging device is moving in a direction directly facing the subject 201, so the left and right perspective of the subject 201 appears weakened in the image. At position 207, the subject 201 is viewed from a larger angle, so the left and right perspective of the subject 201 is emphasized in the image.

Here, it is assumed that an image processing unit 507 (described later) or a unit having the same function performs projective transformation on images 302 and 303 using an image 301 captured at a position 205 as a reference. The procedure of projective transformation will be described below with reference to FIG.

First, in the image 301, the coordinates of four points on the subject 201 specified by the photographer for setting the shape of the focus plane are acquired. When four points are set at the four corners of the subject 201, points 304 to 307 are shown in a star shape in the image 301.

Next, in image 302, it is searched where feature points corresponding to points 304 to 307 in image 301 are located. A known vector detection method can be used to search for feature points among multiple images. Feature points corresponding to points 304 to 307 in image 302 are designated as points 308 to 311, and coordinates of points 308 to 311 are obtained.

The coordinates of the four points on the subject 201 for setting the focus plane may be selected by the photographer, or may be automatically selected by the imaging device based on the distance information of the subject, or from the options presented by the imaging device. It may be selected by the photographer.

Here, the origin is the upper left of the image, the coordinates of points 304 to 307 in image 301 are (x1, y1), (x2, y2), (x3, y3), (x4, y4), and the coordinates of points 308 to 307 in image 302 are Let the coordinates of 311 be (x5, y5), (x6, y6), (x7, y7), and (x8, y8). Points 308 to 311 where the image processing unit 507 (described later) or a unit having the same function are located at (x5, y5), (x6, y6), (x7, y7), and (x8, y8) of the image 302 are A projective transformation is performed on the image 302 so as to correspond to the positions of (x1, y1), (x2, y2), (x3, y3), and (x4, y4), respectively. As long as the motion vectors before and after the projective transformation are not in the same direction, and there are four pairs of corresponding feature points, all the coefficients of the projective transformation can be determined.

Similarly, in the image 303, the locations of feature points corresponding to the points 304 to 307 in the image 301 are searched to identify the points 312 to 315, and the coordinates of the points 312 to 315 are obtained. A projective transformation is performed on the image 303 so that the points 312 to 315 correspond to the positions of the points 304 to 307 in the image 301.

By performing the above processing, images 316 to 318 in which the subject 201 has substantially the same shape are generated. If image 301 is used as a reference image, even if projective transformation is performed on image 301, image 316 after projective transformation is itself, that is, image 301 and image 316 are the same image. Of course, it is not necessary to perform projective transformation processing on the image 301, which is the reference image.

Next, generation of an image focused on a desired position using images 316 to 318 will be explained. The point on which the photographer wants to focus, specified by the photographer, is shown in the image 316 as a point 319 on the subject 202 in the shape of a cross. The point 319 may be specified after the projective transformation or before the projective transformation. If specified at the time of image capture, it is possible to intentionally focus on the subject corresponding to the position of point 319 during image capture, and blur can be suppressed.

First, an image processing unit 507 (described later) or a unit having the same function acquires the coordinates of a point 319 in the image 316. Next, in the image 317, the position of the point 320 to which the point 319 corresponds is obtained using the coefficients of projective transformation.

Next, an image processing unit 507 (described later) or a unit having the same function translates the image 317 by the difference between the coordinates of the point 319 and the point 320, and adds the images 316 and 317. Similarly, the same process is performed on the image 318, and the image 318 is translated and added by the difference in coordinates between a point 319 and a point 321 corresponding to the point 319. As a result, the image of the point 319 specified by the photographer remains clear because the same part of the subject 202 is superimposed. On the other hand, the object 201 had the same shape after projective transformation, but with the addition of the translational movement, each part of the object 201 is shifted by the amount of the translational movement and added, resulting in blurring. Become. Further, the amount of shift due to translational movement is equal for each part because the object 201 has the same shape after projective transformation. This indicates that the plane of focus of the image is parallel before translation.

As described above, the process for projective transformation and the process for translation can be performed independently.

FIG. 4 is a diagram for explaining the relationship between the focus position and the shape of the focus surface in this embodiment. In FIG. 4, the origin is the position of the imaging device when the reference image was captured, the optical axis of the optical system of the imaging device is the z direction, and there are objects f1 and f2 at different distances from the imaging device. do. θh is the horizontal viewing angle of the imaging device. FIG. 4A shows a state in which the plane passing through the subject f1 and the subject f2 is the plane of focus, and the subject f1 and the subject f2 are in focus. On the other hand, FIG. 4(b) shows that the focus plane and focus position are set individually. The focal plane is not perpendicular to the optical axis of the optical system of the imaging device, but becomes a plane parallel to the plane passing through the subjects f1 and f2 when projective transformation is performed using the subjects f1 and f2 as a reference. When the focus position is the position of the subject f3, the subjects f1 and f2 are blurred by the same amount. Subject f3 is used for translational position alignment. The focus plane is set to be inclined at an angle corresponding to the distance difference between the subject f1 and the subject f2. In this way, the focus position can be determined by the position of f3, regardless of f1 and f2, and the shape of the focus surface can be set by the positional relationship between f1 and f2, regardless of the position of f3. It can be said that the position can be set individually.

<Configuration of imaging device>
FIG. 5 is a diagram for explaining each part of the imaging device in this embodiment.

First, the configuration of the camera housing section 501 will be explained. The light flux from the subject is imaged and received on the image sensor 506 via the imaging optical system 503 of the lens unit 502. The imaging optical system includes an aperture, and the drive unit 504 also includes a drive mechanism for moving the imaging optical system to change the focal position and control the aperture.

Microlenses are arranged in a grid on the surface of the image sensor 506. A large number of microlenses constitute a microlens array 513, which functions as a pupil dividing means. Information related to the distance to the subject (distance information) is obtained by acquiring signals from a plurality of photoelectric conversion units corresponding to each microlens and performing a correlation calculation between the signals. For example, there is a distance image that represents the distribution of distance information of a subject within a captured image. The distance image information is two-dimensional information in which the value indicated by each pixel represents distance information of a subject existing in a region of the captured image corresponding to the pixel. Examples of depth information representing the depth of a subject in the depth direction include an image shift amount map and a defocus amount map. The image shift amount map is calculated from a plurality of viewpoint images from different viewpoints, and the defocus amount map is calculated by multiplying the image shift amount by a predetermined conversion coefficient. A distance map obtained by converting the amount of defocus into object distance information represents a distance distribution from the imaging device to the object.

In this embodiment, an example of measuring the object distance based on the imaging plane phase difference detection method is shown. can be obtained.

Further, since signals representing the evaluation amount for focus adjustment and the exposure amount are obtained from the output of the image sensor 506, AF control and AE (automatic exposure) control of the imaging optical system 503 can be performed based on these signals. .

The image processing unit 507 includes an analog/digital converter (A/D converter), a white balance circuit, a gamma correction circuit, an interpolation calculation circuit, and the like, and can generate image data for recording. In the present invention, the image processing unit may perform projective transformation, translation, and image addition, or the image data may be read out to an external image processing device such as a computer and processed on processing software. Further, the image processing unit 507 may be built into a system control unit 505, which will be described later, or the system control unit 505 may also serve as the image processing unit 507.

The memory unit 508 includes a storage unit and a processing circuit necessary for storing image data and the like. Further, the memory unit 508 performs compression processing and expansion processing of data such as images, moving images, and audio using a predetermined method. The image data stored in the memory section 508 is read out and output to the display section 509 or the recording and reproducing section 510. The display unit 509 is an LCD (liquid crystal display) or the like, and displays images according to control commands from the system control unit 505. Further, the display unit 509 displays a predetermined message on the screen to notify the user.

The recording and reproducing unit 510 performs processing for recording image data and the like on a predetermined recording medium, or reading and reproducing data from the recording medium, in accordance with control commands from the system control unit 505. The predetermined recording medium is, for example, a semiconductor memory device that can be used by being attached to the camera body.

The system control unit 505 includes a central processing unit (CPU) and is a central unit that controls each component of the imaging system. The system control unit 505 generates timing signals and the like during imaging and outputs them to each unit, and also controls each component of the imaging system, image processing system, and recording/reproducing system in response to an operation instruction signal. The system control unit 505 performs processing for displaying or recording image data processed by the image processing unit 507, processing for transmitting it to an external device using an output device, and the like.

The operation detection unit 511 detects a user's operation performed using an operation switch, a touch panel, etc., and outputs an operation instruction signal to the system control unit 505. For example, the operation detection unit 511 detects a user's imaging operation instruction and notifies the system control unit 505. The system control unit 505 controls driving of the image sensor 506, operation of the image processing unit 507, compression processing by the memory unit 508, and controls display of images and information on the screen of the display unit 509.

Next, the configuration of the lens section 502 will be explained. The lens section 502 includes an imaging optical system 503 and a drive section 504. The imaging optical system 503 is composed of optical members such as a lens group and an aperture, and an optical axis 512 is shown in FIG. 5 by a dashed line.

The drive unit 504 moves the imaging optical system according to a control signal from the system control unit 505. All or part of the imaging optical system is moved in the optical axis direction for focusing, and the aperture blades are opened and closed.

<Operation of imaging device>
Next, the operation of the imaging device in this embodiment will be explained. FIG. 6 is a flowchart for explaining the operation in this embodiment. FIG. 7 is a flowchart for explaining another operation in this embodiment.

In step S601, the system control unit 505 selects the blur enlargement mode through an operation by the photographer.

In step S602, the system control unit 505 determines the composition according to the operation of the photographer.

In step S603, the image sensor 506 acquires the first image.

In step S604, the photographer specifies a point to focus on (focus point), and the system control unit 505 controls the imaging optical system 503 to adjust the focus position.

In step S605, the system control unit 505 selects a point for obtaining projective transformation. As described above, the points for determining the projective transformation are the points for determining the shape of the focal plane, and at least four points are required.

In step S606, the imaging device or the imaging optical system 503 moves to the imaging position in order to capture the next image.

In step S607, the image sensor 506 acquires the second and subsequent images.

In step S608, the image processing unit 507 calculates a vector using the points specified by the system control unit 505 in steps S604 and S605.

In step S609, the image processing unit 507 performs projective transformation so that the points selected by the system control unit 505 in step S605 become the same.

In step S610, the image processing unit 507 performs translational movement so that the points selected by the system control unit 505 in step S604 overlap.

In step S611, the system control unit 505 determines whether a predetermined number of sheets or a predetermined time has been reached, and if it is determined that the number has not been reached, the same operation is repeated from step S606. If it is determined that the threshold has been reached, a composite image is output in step S612, and the series of operations ends.

In step S612, the image processing unit 507 performs addition processing on the image after projective transformation and translation, and generates a composite image. The system control unit 505 outputs the composite image to the recording/reproducing unit 510.

FIG. 7 is a flowchart for explaining another example for implementing this embodiment. In the example shown in FIG. 7, a case will be described in which already captured images are stored in the memory unit 508 or the like, and the system control unit 505 and the image processing unit 507 perform composition processing. In the example shown in FIG. 7, the step of capturing an image is not provided.

In step S701, the system control unit 505 designates one image from among the plurality of images as a compositing target image.

In step S702, the system control unit 505 specifies a point on which the photographer wants to focus (focus point).

In step S703, the system control unit 505 selects a point for obtaining projective transformation.

In step S704, the image processing unit 507 uses the points specified by the system control unit 505 in steps S702 and S703 to calculate a vector from the two images including the compositing target image.

In step S705, the image processing unit 507 performs projective transformation on the compositing target image so that the points selected by the system control unit 505 in step S703 are the same.

In step S706, the image processing unit 507 performs translational movement based on the compositing target image so that the points selected by the system control unit 505 in step S702 overlap.

In step S707, the image processing unit 507 performs addition between the image after projective transformation and translation and the compositing target image. If a composite image has already been recorded, the image processing unit 507 performs addition on the composite image and the image after projective transformation and translation to generate a new composite image. In step S708, the system control unit 505 overwrites the memory unit 508 with the composite image newly generated by the image processing unit 507.

In step S709, the system control unit 505 determines whether all the images to be combined have been combined, and if it is determined that the combined images have not been combined, the same operation is repeated from step S704. If it is determined that the threshold has been reached, a composite image is output in step S710, and the series of operations ends.

Note that in this embodiment, the method of determining the reference image is arbitrary, and the flow described above is just an example.

<Projective transformation/translation>
Below, detailed algorithms for projective transformation and translation in this embodiment will be explained.

Projective transformation of an image can be expressed as a matrix as shown in (Equation 1) and (Equation 2) below.

The transformation that moves the point (x ₁ , y ₁ ) to the point (x ₁ ', y ₁ ') is

The unknown matrix has 8 elements, and each element can be found if there are 4 pairs of x and y.

In the projective transformation performed on each image in this embodiment, the four points (x1, y1) to (x4, y4) in the image to be transformed are converted to the four points (x1'y1') to (x4'y4') in the reference image. ), create and solve four equations. As a result, each of the matrix elements h11 to h32 can be determined. The projective transformation is completed by performing matrix calculations of (Formula 1) and (Formula 2) on all pixels of the image to be transformed.

In the explanation using FIGS. 6 and 7, first, projective transformation was performed to generate an intermediate image, and translational movement was performed on the image. By devising a matrix for projective transformation, it is possible to complete translational movement with only one projective transformation without generating intermediate images. That is, step S608 and step S609 or step S705 and step S706 can be made into one process. The method is shown below.

First, the matrix elements of (Equation 2), which is a matrix of projective transformation, are obtained from the movement of four specified points for calculating the amount of projective transformation between the reference image and the image to be transformed. Next, it is determined how much the coordinates of the focus point in the image to be transformed have moved between before and after the projective transformation, and the amount of movement is set to (+Δx2, +Δy2). Let the amount of translation of the point to be focused on without projective transformation be (+Δx1, +Δy1). Then, considering that the deformation target (x1, y1,) is superimposed on the reference (x1', y1') by projective transformation and translation, the projective transformation and translation are expressed by (Equation 1) and the following (Equation It can be expressed as 3).

When the matrices on the right side are calculated and integrated, it can be expressed as (Equation 4) to (Equation 6) below.

By converting the image to be converted using (Equation 4) and (Equation 1), it is possible to perform a transformation equivalent to sequentially performing projective transformation and translation in one transformation. can.

Therefore, resources for calculation such as memory and processing time can be reduced, and rounding errors when arranging images at respective pixels after conversion can also be reduced.

In the above explanation, vector detection technology is used to find the projective transformation amount and translation amount, but if different points on the image are confused with the same point, there is a possibility that undesirable vector detection values may be obtained. sell. When images are synthesized using the projective transformation amount or translation amount determined from such vector detection values, the quality of the synthesized image may be degraded. Therefore, if the vector detection amount is monitored, and if it falls outside a certain range, that is, if the value deviates significantly compared to other images, or if the value deviates significantly from the imager's movement of the imaging device, , images may be excluded from the compositing target.

According to the first embodiment, in an image equivalent to imaging with an aperture larger than the aperture of the imaging optical system, the focus position (focus position) and the distribution of the focus position within the screen (focus surface shape) can be individually determined. Can be done.

(Second embodiment)
The second embodiment will be described below. In the second embodiment, unlike the first embodiment, the shape of the focus plane is determined by specifying a virtual focus plane. Below, the second embodiment will be described in detail using mathematical formulas.

FIG. 8 is a flowchart for explaining calculation of the amount of translation in this embodiment.

First, in step S801, the image processing unit 507 creates a distance map for a plurality of images taken from different positions using a known distance estimation technique. Specifically, the photographer specifies a plurality of subjects forming a desired focus plane, and the image processing unit 507 calculates the subject distance of each subject specified by the photographer.

In step S802, the image processing unit 507 sets the plane passing through the subject specified by the photographer as the virtual focus plane f(x,y). f(x, y) is a function that includes information on the distance of the focal plane from the imaging device at each point within the imaging field of view of the imaging device, and f(x, y) is used to calculate the properties of the focal plane shape. shows.

In step S803, the image processing unit 507 obtains the position (x0, y0) of the subject to be set as the focus position. Furthermore, the image processing unit 507 obtains the distance d0 between the position (x0, y0) and the imaging device.

In step S804, the system control unit 505 assigns the position (x0, y0) of the subject to the function f(x, y) of the virtual focus plane, and calculates the value f(x ₀ , y ₀ ).

In step S805, the system control unit 505 acquires the position (si, ti) of the imaging device when each image is captured, using a known position and orientation estimation technique using a distance map, for example.

Next, the image captured at the coordinates (s, t) is used as a reference image, and processing for deforming the plurality of images is performed.

In step S806, the system control unit 505 defines a new function f'(x, y) of (Equation 7) below.

In step S807, the system control unit 505 calculates the amount of translation.

Let i, j be the coordinates of a pixel on the image, and Δi, Δj be the translation amount [pix] of each pixel in the x direction and y direction, then the following (Equation 8) and (Equation 9) can be used to calculate the translation. Able to calculate quantity.

The meanings of the characters used in (Formula 8) and (Formula 9) above are as follows.
W: Horizontal image size H: Vertical image size θw: Horizontal viewing angle of the imaging device θh: Vertical viewing angle of the imaging device (sm, tm): Coordinates of the imaging device at the time of capturing each image ( s, t): Coordinates of the imaging device at the time when the reference image was captured By performing the above translation on each image captured at a different position using the calculated translation amount, and then adding and combining the images, , the shape of the focus plane is determined by the position of the subject from which the virtual focus plane is determined, and the focus position can be determined separately by the specified subject.

According to the second embodiment, the focus position can be individually determined while freely specifying the focus plane shape depending on how the virtual focus plane is determined.

(Other embodiments)
Note that the "imaging device" in the above embodiments can be applied not only to personal digital cameras but also to mobile devices, smartphones, network cameras, and the like.

Note that the present invention provides a system or device with a program that implements one or more functions of the above-described embodiments via a network or a storage medium, and one or more processors in a computer of the system or device executes the program. This can also be realized by reading and activating the process. It can also be realized by a circuit (for example, an ASIC) that realizes one or more functions.

501 Camera housing section 502 Lens section 503 Imaging optical system 504 Drive section 505 System control section 506 Image sensor 507 Image processing section 508 Memory section 509 Display section 510 Recording/reproduction section 511 Operation detection section 512 Optical axis 513 Microlens array

Claims (19)

a first calculation means for calculating a projective transformation amount using a plurality of first points in the plurality of images with respect to a plurality of images captured from a plurality of viewpoints;
a second calculation means for calculating a translation amount using at least one second point different from the first point in the plurality of images;
An image processing device comprising: a deformation unit that deforms the plurality of images using the projective transformation amount and the translation amount. 2. The image processing apparatus according to claim 1, further comprising a compositing means for compositing the plurality of images after the deformation has been performed by the deformation means. 3. The image processing apparatus according to claim 2, wherein the composite image generated by the composition by the composition means is equivalent to an image captured with a larger aperture than the aperture used when capturing the plurality of images. 3. The image processing apparatus according to claim 2, wherein the compositing means individually sets the shape of the focus plane and the focus position when performing the compositing. 5. The image processing apparatus according to claim 4, wherein the synthesizing means sets the shape of the focus plane based on the projective transformation amount. 5. The image processing apparatus according to claim 4, wherein the synthesizing means sets the focus position based on the translation amount. The first calculation means does not use the second point in calculating the projective transformation amount,
The image processing apparatus according to claim 1, wherein the second calculation means does not use the first point in calculating the translation amount. The image processing apparatus according to claim 1, wherein the first calculation means and the second calculation means each independently calculate the projective transformation amount and the translation amount. The transformation means performs a first transformation on the plurality of images using the projective transformation amount calculated by the first calculation means, and then calculates the translation amount calculated by the second calculation means. The image processing apparatus according to claim 1, wherein the image processing apparatus performs the second transformation using the image processing apparatus. The image processing apparatus according to claim 9, wherein the second deformation is a translational movement. The image processing apparatus according to claim 1, wherein the first calculation means calculates the projective transformation amount using four first points in each of the plurality of images. The transformation means is configured to transform an image among the plurality of images in which the projective transformation amount calculated by the first calculation means and the translation amount calculated by the second calculation means have values within a certain range. The image processing apparatus according to claim 1, wherein the deformation is performed on a target. specifying means for specifying a virtual focus plane based on a first point on the subject;
Calculation means for calculating a translation amount using a second point different from the first point on the subject;
An image processing device comprising: a deformation unit that deforms the plurality of images using the virtual focus plane and the translation amount. comprising an acquisition means for acquiring a subject distance of the subject;
14. The image processing apparatus according to claim 13, wherein the deformation means performs the deformation using the subject distance. 14. The image processing apparatus according to claim 13, further comprising a compositing means for compositing the plurality of images after the deformation has been performed by the deformation means. an imaging means for capturing a plurality of images from a plurality of viewpoints;
a first calculation unit that calculates a projective transformation amount for the plurality of images using a plurality of first points in the plurality of images;
a second calculation means for calculating a translation amount using at least one second point different from the first point in the plurality of images;
An image processing device comprising: a deformation unit that deforms the plurality of images using the projective transformation amount and the translation amount. a first calculation step of calculating a projective transformation amount for a plurality of images captured from a plurality of viewpoints using a plurality of first points in the plurality of images;
a second calculation step of calculating a translation amount using at least one second point different from the first point in the plurality of images;
An image processing method comprising: a deforming step of deforming the plurality of images using the projective transformation amount and the translation amount. A program that causes a computer to operate an image processing device,
a first calculation step of calculating a projective transformation amount for a plurality of images captured from a plurality of viewpoints using a plurality of first points in the plurality of images;
a second calculation step of calculating a translation amount using at least one second point different from the first point in the plurality of images;
A program for performing a deforming step of deforming the plurality of images using the projective transformation amount and the translation amount. A computer readable storage medium storing the program according to claim 18.

Images