JP2015200918A - Image processor, method, and program - Google Patents

Abstract

PROBLEM TO BE SOLVED: To achieve photographic expression which enables the feeling of the flow of times by compositing an image (image A) photographed in the past and the current image (image B) photographed from the almost same position without a sense of incongruity.SOLUTION: A relation between positions of photographing the image A and the image B is acquired (photographing position calculation part 13), a virtual photographing position transiting with time, with the photographing position of the image A as a start point and with the photographing position of the image B as an end point is determined (virtual photographing position calculation part 14), an image from the virtual photographing position is created by deforming the image A (deformation image creation part 15), further, the image B and the deformed image A are composited (composite image creation part 16), and display is performed in an image display part 17.

Description

The present technology relates to an image processing method, an apparatus, and a program for creating one image group from a plurality of images.

There is a field called “Re-photography” as an image representation method. Rephotography is an image of a historical building or natural landscape taken in the past (photographs) taken from the same place, and these two images (the past image and the image just taken) are combined. To make one image. As a result, it is possible to make a photographic expression that can feel the flow of the times. In order to realize this, it is necessary to shoot from the same shooting position as the place where the past image was shot. This is because if the image is taken from a different shooting position, a misalignment occurs when the image is superimposed on the past image, resulting in an unbearable image. However, it is difficult to specify the position where the image was taken by looking at the past image.

Therefore, Patent Document 1 or Non-Patent Document 1 discloses a method for instructing where to start photographing by automatically analyzing past images. In this method, the photographer first takes a picture near the position where the past image was taken, with the photographing equipment. By analyzing the past image and the image taken now, for example, an instruction “move right 5 m” is given. However, in this method, the photographing position is simply instructed, and the photographer must move with the photographing equipment in accordance with the instruction. In particular, there are many tourists around a historical building, and it is difficult to move without disturbing other tourists. In addition, for example, in the above example, if the vehicle moves 5 m to the right, it may move on a road with a lot of traffic, and it may be dangerous to take a picture.

If possible, we would like to finish the photographer's hassle by shooting once from the “same” position as the previous shooting position. In other words, there is a need for a photographic expression method that allows you to feel the flow of the times by fusing past images and images you have just taken, just by shooting from about the same position as the past shooting position. There was no photographic expression technique. Also, there has been no apparatus or method for realizing the photographic expression technique.

JP 2011-239361 A

Soonmin Bae, Aseem Agarwala, Fredo Durand. "Computational Re-Photography", ACM Transactions on Graphics (Presented at SIGGRAPH Asia 2010), 29 (3), pp. 24: 1-24: 15.

It is desirable to have a photographic expression technique that allows you to feel the flow of the times by fusing past images and images you have just taken, just by shooting from about the same position as the previous shooting position. There was no expression method. Also, there has been no apparatus or method for realizing the photographic expression technique.

The present technology has been made in view of such a situation, and since an image taken from almost the same place and an image taken in the past can be combined without a sense of incongruity, the photographer moves. It is not necessary. The synthesized image is a photographic expression technique that allows you to feel the flow of the times.

An image processing apparatus according to an aspect of the present technology is an image processing apparatus that generates a third image using a first image and a second image, and the first image and the second image A shooting position calculation unit that obtains a positional relationship between the first shooting position where the first image was shot and the second shooting position where the second image was shot by matching, and the first shooting position. And a virtual position determination unit that determines a position between the second shooting position (referred to as a third shooting position) and an image when it is assumed that the shooting is performed from the third shooting position. An image generation unit that generates an image of the image, and the virtual position determination unit includes a path starting from one of the first shooting position or the second shooting position and having the other as an end point. The third shooting position moves with time, and the image generator The deformation in accordance with the positional relationship of the third imaging position with respect to the first imaging position, by performing with respect to the first image, and generates the third image.

As a result, it is possible to synthesize an image taken in the past and a current image taken from almost the same place without any sense of incongruity, and realize a photographic expression that can feel the flow of the times.

In the image processing apparatus, the first image further includes depth information for each pixel position, and the shooting position calculation unit uses the depth information to determine the first shooting position and the first image. The positional relationship of the second shooting position is obtained.

This improves the accuracy.

In the image processing apparatus, the first image further includes mask information for each pixel position, and the photographing position calculation unit stores information on the first image for a position without the mask information. It is characterized in that matching is performed using the information of the first image only for a position where the mask information is present without being used.

This improves the accuracy.

An image processing method or an image processing program according to an aspect of the present technology is an image processing method or an image processing program that generates a third image using a first image and a second image, A shooting position calculation step for obtaining a positional relationship between a first shooting position where the first image is shot and a second shooting position where the second image is shot by matching the image with the second image. And a virtual position determination step for determining a position between the first shooting position and the second shooting position (referred to as a third shooting position), and shooting from the third shooting position. An image generation step for generating a case image as the third image, and in the virtual position determination step, one of the first shooting position and the second shooting position is set as a starting point, and the other On the route that ends at The third shooting position is moved with time, and in the image generation step, the first image is deformed according to the positional relationship of the third shooting position with respect to the first shooting position. In this case, the third image is generated.

As a result, it is possible to synthesize an image taken in the past and a current image taken from almost the same place without any sense of incongruity, and realize a photographic expression that can feel the flow of the times.

Image processing according to one aspect of the present technology is image processing for generating a third image using the first image and the second image, and is based on matching between the first image and the second image. The positional relationship between the first shooting position at which the first image is shot and the second shooting position at which the second image is shot is obtained, and the first shooting position and the second shooting position are obtained. Is determined as a third imaging position, and an image when it is assumed that the image is captured from the third imaging position is generated as the third image. The third photographing position moves with time on a route having one photographing position or one of the second photographing positions as a starting point and the other as an ending point, with respect to the first photographing position. The deformation corresponding to the positional relationship of the third shooting position is applied to the first image. By it performed, and generates the third image.

As a result, it is possible to synthesize an image taken in the past and a current image taken from almost the same place without any sense of incongruity, and realize a photographic expression that can feel the flow of the times.

According to one aspect of the present technology, it is possible to synthesize an image taken in the past and a current image taken from almost the same place without a sense of incongruity, and realize a photographic expression that can feel the flow of the times.

It is a block diagram which shows the structure of the Example of this invention. It is a flowchart explaining the process of the Example of this invention. It is a figure explaining the concrete imaging | photography condition which can apply the Example of this invention. It is a figure which shows the image image | photographed in the past which can apply the Example of this invention. It is a figure explaining the image image | photographed in the past which can apply the Example of this invention. It is a figure explaining the image image | photographed in the past which can apply the Example of this invention. It is a figure explaining the concrete imaging | photography condition which can apply the Example of this invention. It is a figure which shows the image image | photographed now at which the Example of this invention is applicable. It is a figure which shows the image produced | generated when the Example of this invention is applied. It is a figure which shows the image produced | generated when the Example of this invention is applied. It is a figure explaining regarding the Example of this invention. It is a figure which shows the image produced | generated when the Example of this invention is applied. It is a block diagram which shows the structural example of the computer apparatus which performs the process of the Example of this invention.

<< Examples of the Invention >>
Examples of the present invention will be described. First, a typical usage method of the present invention and an example of processing in the apparatus of the present invention will be briefly described.
<User Action 1> The user first prepares an image (image A) taken in the past. The image A is, for example, a photograph taken of a historic building several decades ago.
<User Operation 2> While viewing the image A, the user goes to the site where the image A was shot and takes a picture from almost the same shooting position. Let the captured image be an image B.
<User action 3> The user inputs the image A and the image B to the apparatus of the present invention.
The apparatus of the present invention performs the following processing.
<Process 1> Image A and image B are matched, and the positional relationship between the shooting position where image A was shot (referred to as shooting position PA) and the shooting position where image B was shot (referred to as shooting position PB) is determined. Ask.
<Process 2> Next, a virtual photographing position PC (t) is introduced. t is a parameter representing time, and the value of t is an integer from 0 to 100. PC (t) is changed with t so that it is the same as the shooting position PA at t = 0 and the same as the shooting position PB at t = 100. The reason for setting the value of t from 0 to 100 is to generate 101 images, and when it is desired to use a number other than 101, it is sufficient to take a value corresponding to t.
<Process 3> At each time t, an image on the assumption that the image is taken from the position of PC (t) is created by deforming the image A. Let this image be C (t).
<Process 4> The image C (100) and the image B are combined. This image is designated as an image D.
<Process 5> Finally, the image display unit continuously displays the image C (0) to the image C (100), and finally displays the image D.
Through such a series of processing, the images finally displayed in <Processing 5> (102 images that are continuously played back) are expressed so that they can feel the flow of the times. . This is because the image C (t) (t = 0 to 100) created in <Process 3> is created from the image A and is an image taken in the past. To be precise, it is an image obtained by transforming an image taken in the past. That is, C (t) (t = 0 to 100) is an image representing a past landscape. Furthermore, C (100) is an image when it is assumed that the shooting position PC (100) = PB, that is, it is shot from the same shooting position as the image B. Therefore, when the image C (100) and the image B are overlapped, there is no positional deviation, and no breakage occurs in the composition processing of the image C (100) and the image B. That is, the image D is an unbroken image in which the current landscape and the past landscape are synthesized. Displaying image C (0) to image C (100) in order, and then displaying image D causes image A (a past photo of a historical building) to be slowly deformed. , "Past historic buildings" can be expressed in a fusion of the current landscape. In this way, it is possible to express photographs that can feel the flow of the times.

The above is a brief description of typical usage of the present invention and processing in the apparatus of the present invention.

In general, if there are two images, it can be reduced to a matching problem in stereo vision. Therefore, the positional relationship between the photographing position PA and the photographing position PB is obtained by a matching method between two images related to existing stereo vision. (<Process 1> described above). However, the obtained positional relationship is often inaccurate because the solution process is unstable. Therefore, in the embodiment of the present invention, the accuracy of the positional relationship is improved by giving depth information to the image A. In other words, for an arbitrary position (x, y) in the image A, the pixel value A (x, y) has a value, and the depth information Z (x, y) also has a value. To do. Further, the focal length when the image A is taken is assumed to be fA. That is, A (x, y), Z (x, y), and fA are known.

If the image A is given, it is the same value that the pixel value A (x, y) is given to an arbitrary position (x, y), and therefore A (x, y) is known. it is obvious. fA can be known from the information of the camera that captured the image A in the past. Depth information Z (x, y) is automatically created by using the method described in the paper D. Hoiem, AA Efros, and M. Hebert, "Automatic Photo Pop-up", ACM SIGGRAPH 2005, for example. I can do it. Alternatively, the depth information Z (x, y) may be created manually by the user.

In addition, it is assumed that information on a subject that is currently present among the subjects projected on the image A is also known. This may be created manually by comparing the current scenery with the image A at the same time as the above-described <User action 2>. This information is referred to as matching mask information M (x, y). Note that the value of M (x, y) is 0 or 1. That is, if an object at position (x, y) still exists in image A, M (x, y) = 1. If the subject at position (x, y) does not currently exist in image A, M (x, y) = 0.

In addition to the historical building, the image A includes unnecessary subjects. For example, the sky or the ground. Therefore, the user creates information that excludes unnecessary portions. This information is set as subject mask information N (x, y). Note that the value of N (x, y) is 0 or 1. That is, if the subject at the position (x, y) in the image A is an important subject (such as a historical building), N (x, y) = 1. If the subject at the position (x, y) in the image A is not an important subject, N (x, y) = 0.

In addition, since it is possible that a plurality of users combine with the current photo taken by each user using the same “historical building past photo”, a person can obtain depth information Z ( x, y), matching mask information M (x, y), and subject mask information N (x, y) are created. By sharing the information, other users can use Z (x, y) y), M (x, y), and N (x, y) can be saved.

Further, since the user can know the focal length of his camera when taking the image B, the focal length (referred to as fB) is also known.

In summary, the information input to the apparatus according to the embodiment of the present invention is as follows.
(Input 1) Image A, that is, a value of pixel value A (x, y) for each position (x, y) (Input 2) Depth information about image A, that is, depth information Z for each position (x, y) Value (x, y) (input 3) Focal length fA when image A is photographed
(Input 4) Information relating to a subject that is currently present among subjects projected on the image A, that is, a value of matching mask information M (x, y) for each position (x, y).
(Input 5) Information on whether or not the subject is an important subject among the subjects projected on the image A, that is, a value of subject mask information N (x, y) for each position (x, y).
(Input 6) Image B, that is, a focal length fB when a value (input 7) image B of a pixel value B (x, y) for each position (x, y) is photographed.
It is.

Details of the embodiments of the present invention will be described below.

An apparatus according to an embodiment of the present invention is shown in FIG. Reference numeral 10 in FIG. 1 denotes an image processing apparatus according to an embodiment of the present invention. In the image processing apparatus 10, an input terminal 11, a feature point matching processing unit 12, a shooting position calculation unit 13, a virtual shooting position calculation unit 14, a deformed image creation unit 15, a composite image creation unit 16, and an image display unit 17. There is. Also, the lines connecting the parts in the figure show the main data flow. The details of each part will be described below, and the approximate processing contents will be described here.

The data of the above (input 1) to (input 7) is input from the input terminal unit 11.

The feature point matching unit 12 receives the pixel value A (x, y), the matching mask information M (x, y), and the pixel value B (x, y), and the positional relationship between the paired feature points (Xi, yi) and (ui, vi) are output.

In the photographing position calculation unit 13, the depth information Z (x, y), the focal length fA, the focal length fB, and (xi, yi) and (ui, vi), which are the positional relationships among the paired feature points, are obtained. Received, and outputs the shooting position (R0, R1, R2, Tx, Ty, and Tz) of the image B when the shooting position of the image A is used as a reference.

The virtual shooting position calculation unit 14 receives the focal length fA, the focal length fB, and the shooting positions (R0, R1, R2, Tx, Ty, and Tz) of the image B and receives the shooting position (R00) corresponding to the time t. (T), R10 (t), R20 (t), Tx0 (t), Ty0 (t), Tz0 (t), and fB0 (t)) are output.

In the deformed image creation unit 15, the photographing position (R00 (t), R10 (t), R20 (t) corresponding to the pixel value A (x, y), depth information Z (x, y), focal length fA, and time t. ), Tx0 (t), Ty0 (t), Tz0 (t), and fB0 (t)), and outputs an image C (t). Here, t is an integer from 0 to 100.

In the composite image creation unit 16, pixel value A (x, y), depth information Z (x, y), subject mask information N (x, y), focal length fA, pixel value B (x, y), time An imaging position (R00 (t), R10 (t), R20 (t), Tx0 (t), Ty0 (t), Tz0 (t), and fB0 (t)) corresponding to t = 100 is received and an image D is output.

The image display unit 17 receives the image C (t) (t = 0 to 100) and the image D and displays them.

Next, the flow of processing will be described, but before that, the correspondence relationship between the image A and the image B will be described. At an arbitrary position (x, y) of the image A, a subject at a three-dimensional position expressed by Expression 1 is projected in a coordinate system based on the optical center of the camera when the image A is captured.

This is because the light beam coming from the direction (x, y, fA) is exposed to the position (x, y) of the image A and the distance (depth) in the z-axis direction from the camera to the subject is Z (x, It is clear from y).

Consider the position in the image B where the subject located in Equation 1 above is projected. In general, when two images are captured, the relationship between the capturing positions is represented by a three-dimensional rotation matrix and a three-dimensional translation vector. Each element of the rotation matrix is described by a trigonometric function. However, in the case of the embodiment of the present invention, since the image B is taken from almost the same position as the image A, the rotation matrix can be approximated by a linear expression. . That is, Equation 2 is obtained.

R 0, R 1, R 2, Tx, Ty, and Tz in Formula 2 are appropriate numerical values (real numbers). The subject projected at the position (x, y) of the image A is projected at the position (u, v) of the image B. Here, (u, v) is a position calculated by Equation 2. In other words, (u, v) is obtained by rotating the position shown by Equation 1 by the rotation matrix and moving the position by the translation vector.

When Expression 2 is transformed, Expression 3 is obtained.

FIG. 2 shows the flow of processing according to the embodiment of the present invention.

First, in step 1 of FIG. 2, the above-described (input 1) image A, that is, the depth information regarding the value (input 2) image A of the pixel value A (x, y) for each position (x, y), that is, Depth information Z (x, y) value (input 3) for each position (x, y) Focal length fA when image A is photographed
(Input 4) Information relating to a subject that is currently present among subjects projected on the image A, that is, a value of matching mask information M (x, y) for each position (x, y).
(Input 5) Information on whether or not the subject is an important subject among the subjects projected on the image A, that is, a value of subject mask information N (x, y) for each position (x, y).
(Input 6) Image B, that is, a focal length fB when a value (input 7) image B of a pixel value B (x, y) for each position (x, y) is photographed.
Is input from the input terminal 11. Then, the process proceeds to Step 2.

In step 2, feature points are detected from each of image A and image B. However, the feature point detection from the image A excludes the position (x, y) where M (x, y) = 0, and the position (x, y) where M (x, y) = 1. Only ask. Then, the feature points obtained from the image A and the feature points obtained from the image B are matched. That is, a feature point having the same (or similar) feature as each feature point of the image A is selected from the feature points of the image B. Thereby, pairing of feature points is performed. The positions of these paired feature points are (xi, yi) and (ui, vi). Here, i is a subscript (parameter) for specifying the paired feature points. That is, for an arbitrary i, the feature of the position (xi, yi) of the image A and the feature of the position (ui, vi) of the image B are the same (or similar), and the position of the image A (xi , Yi) and the subject projected at the position (ui, vi) of the image B are considered to be the same. This process is performed by the feature point matching unit 12.

In step 2, “exclude position (x, y) where M (x, y) = 0” does not currently exist in the subject projected on image A (that is, Subjects that do not exist in the image B can be excluded, and erroneous determination of pairing can be suppressed.

After step 2, proceed to step 3.

In step 3, information is obtained as to what position the shooting position of image B was when the shooting position of image A was used as a reference. That is, R0, R1, R2, Tx, Ty, and Tz that satisfy Expression 3 (described above) are obtained. Since there is actually an error, R0, R1, R2, Tx, Ty, and Tz that minimize Equation 4 are obtained.

Here, the summation (Σ) for i is performed for all the paired feature points obtained in step 2. This processing is performed by the photographing position calculation unit 13.

After step 3, proceed to step 4.

In step 4 and subsequent steps, an image obtained by deforming the image A with time is created. Therefore, in step 4, 0 is set to t as a parameter representing time. Then, the process proceeds to Step 5.

In step 5, the photographing position corresponding to time t is calculated. That is, the virtual shooting position calculation unit 14 calculates Expression 5.

At time t, the rotation represented by R00 (t), R10 (t), R20 (t), and Tx0 (t), Ty0 (t), Tz0 (t) with reference to the shooting position of the image A It is assumed that the image is taken from the position where the expressed parallel movement is performed. The focal length is fB0 (t). As can be seen from Equation 5, at t = 0, it is the same as the shooting position of the image A. At t = 100, it is the same as the shooting position of the image B.

After step 5, proceed to step 6.

In step 6, an image C (t) corresponding to time t is generated by transforming the image A. That is, for each position (x, y) in the image A, Equation 6 is calculated to obtain the position (u (x, y, t), v (x, y, t)).

Next, prepare a new canvas. Then, the pixel value A (x, y) at each position (x, y) of the image A is written to the position (u (x, y, t), v (x, y, t)) on the canvas. After writing all the positions of the image A, an image formed on the canvas is defined as an image C (t). This process is performed by the deformed image creation unit 15.

As can be seen from Equations 5 and 6, the image A is created as it is at t = 0, and the image is assumed to be taken from the same shooting position as the image B at t = 100. Further, as can be seen from Equation 5, when t increases continuously, the shooting positions corresponding to time t (R00 (t), R10 (t), R20 (t), Tx0 (t), Ty0 (t ), The position determined by Tz0 (t), and the focal length fB0 (t)) continuously change.

After step 6, proceed to step 7.

In step 7, it is determined whether the value of t is 100. If not 100, it is necessary to create an image C (t) at the next time t. If it is 100, since all images C (t) have been created, the process proceeds to step 9.

In step 8, t is incremented by 1 for processing at the next time. Then, the process returns to step 5.

In step 9, the last image D is created. That is, first, a new canvas is prepared. Then, image B is copied to this canvas. That is, the pixel value B (x, y) at each position (x, y) of the image B is written to the position (x, y) on the canvas. Then, in the image A, the pixel value A (x, y) at each position (x, y) where N (x, y) = 1 is set to the position (u (x, y, 100), v (x, y, 100)) is overwritten. Note that (u (x, y, 100), v (x, y, 100)) is a value when t = 100 in Equation 6. An image formed on the canvas is referred to as an image D. This process is performed by the composite image creation unit 16.

The image D created in this way will be described below. The pixel value A (x, y) is written at a position (u (x, y, 100), v (x, y, 100)) on the canvas. The written image is an image of the subject projected on the image A when it is assumed that the image is taken from the photographing position at time t = 100. Since the shooting position at time t = 100 is the same shooting position as that of the image B, the subject image has no positional deviation from the image B that has already been copied to the canvas. Since only the position where N (x, y) = 1 is written, only the important subject is written. Therefore, the image D is an image in which an important subject portion in the image B is replaced by the subject image projected on the image A.

After step 9, proceed to step 10.

In step 10, the created image is displayed. That is, the image display unit 17 displays the image C (0) to the image C (100) in order, and finally displays the image D. As a result, the image A (a past photograph of a historical building) is slowly deformed, and finally, a scene where the “historic building of the past” is merged with the current landscape can be expressed. In this way, it is possible to express photographs that can feel the flow of the times.

Now, the process is finished. Of course, the image C (t) (t = 0 to 100) and the image D may be stored in a data storage unit (not shown) so that they can be viewed again later.

In order to deepen the understanding of the present invention, the same description as the above processing contents will be described again with reference to the drawings.

FIG. 3 shows an example of a situation when the image A is taken. A historic building 20A at that time, a historic building 21A at that time that does not currently exist, and a tree 22A that does not currently exist are arranged as shown in the figure. For convenience of explanation, it is assumed that 20A has four white windows and one white door as shown in the figure. In order to simplify the drawing, the object is represented by a quadrangular prism or a cone. It is assumed that the image projected onto the screen 31A as viewed from the coordinate system 30A is the image A. Here, the distance from the origin of the coordinate system 30A to the screen 31A is fA.

An image A photographed in the situation shown in FIG. 3 is shown in FIG. As shown in FIG. 4, projected images of a historical building 20A at that time, a historical building 21A at that time that does not currently exist, and a tree 22A that does not currently exist are shown in image A.

Before performing the processing of the embodiment of the present invention, as described above, for example, the technique described in the paper D. Hoiem, AA Efros, and M. Hebert, “Automatic Photo Pop-up”, ACM SIGGRAPH 2005 is used. The depth information Z (x, y) is created by using it. Each position (x, y) of the image A is a position of (x, y, fA) when viewed from the coordinate system 30A. Therefore, it can be seen that the subject projected at the position (x, y) is actually at the position represented by Expression 1 (described above) when viewed from the coordinate system 30A. This state is shown in FIG.

In FIG. 3, it is assumed that 21A and 22A do not currently exist. It is assumed that 20A still exists. Therefore, the information regarding the subject that is present in the subject projected on the image A, that is, the matching mask information M (x, y) for each position (x, y) is shown in FIG. It becomes. In FIG. 5, the shaded area is an area where M (x, y) = 1, and the other areas are M (x, y) = 0.

Further, 20A and 21A are historically meaningful, and 22A is meaningless. Therefore, information on whether or not the subject is an important subject among the subjects projected on the image A, that is, subject mask information N (x, y) for each position (x, y) is shown in FIG. It becomes. In FIG. 6, the shaded area is an area where N (x, y) = 1, and the other areas are N (x, y) = 0.

FIG. 7 shows an example of a situation when the image B is captured. The current historic building 20B, the person 23B, and the building 24B that was built at the time but was not built at that time are arranged as shown in the figure. The current historic building 20B is the same as the historic building 20A of FIG. 3 at the time, but the windows and doors have become black due to aging. As described above, 21A and 22A do not exist in FIG. 7 (present). Assume that the image projected onto the screen 31B as viewed from the coordinate system 30B in FIG. Here, the distance from the origin of the coordinate system 30B to the screen 31B is fB.

The coordinate system 30A and the coordinate system 31A are substantially the same position, but are illustrated with a considerable distance for easy understanding.

An image B shot in the situation shown in FIG. 7 is shown in FIG. As shown in FIG. 8, a projected image of each of the current historic building 20B, the person 23B, and the building 24B that was not built at the time but is now built is shown in the image B.

The image A shown in FIG. 4 (however, only the region where M (x, y) shown in FIG. 5 is 1) and the image B shown in FIG. 8 are matched (steps 2 and 3 in FIG. 2). As a result, the positional relationships R0, R1, R2, Tx, Ty, and Tz between the coordinate system 30A and the coordinate system 30B are obtained. FIG. 3 illustrates this positional relationship. That is, the coordinate system 30B is a position where the rotation represented by R0, R1, and R2 and the parallel movement represented by Tx, Ty, and Tz are performed on the coordinate system 30A. Of course, the coordinate system 30B (FIG. 3) obtained in step 2 and step 3 is the same as the coordinate system 30B shown in FIG.

When the positional relationship between the coordinate system 30A and the coordinate system 30B is obtained, a coordinate system that smoothly changes in time from the coordinate system 30A to the coordinate system 30B is obtained using the time parameter t. That is, R00 (t), R10 (t), R20 (t), Tx0 (t), Ty0 (t), and Tz0 (t) obtained in step 5 of FIG. In FIG. 3, with respect to the coordinate system 30A, the rotation represented by R00 (t), R10 (t), R20 (t), and Tx0 (t), Ty0 (t), Tz0 (t) are represented. The position where the parallel movement is performed is shown as a coordinate system 30C. In FIG. 3, an image projected onto the screen 31C when viewed from the coordinate system 30C is an image C (t) (step 6 in FIG. 2). Here, the distance from the origin of the coordinate system 30C to the screen 31C is fB0 (t).

FIG. 9 shows an image (image projected onto the screen 31C as viewed from the coordinate system 30C) C (t) taken from a virtual position in the situation shown in FIG. As shown in FIG. 9, in the image C (t), each projected image is shown as a modified projected image as compared with the image A.

An image C (t) when t = 100 is shown in FIG. As in FIG. 9, the image C (100) shows each projection image as a modified projection image as compared to the image A. Since the photographing position at time t = 100 coincides with the coordinate system 30B, “20B projection image” in the image B shown in FIG. 8 and “deformation” in the image C (100) shown in FIG. Note that the “projected image of 20A” is coincident in position.

Next, the image D generated in such a situation will be described. As can be seen from the description in step 9 above, the image D is created by replacing part of the image B with the image A (more precisely, the image C (100) obtained by deforming the image A). The part to be replaced is a part (see FIG. 6) where N (x, y) = 1 in the image A. That is, in the image C (100) shown in FIG. 10, it is a portion of a “deformed projection image of 20A” and a “deformed projection image of 21A”. This is illustrated in FIG. FIG. 11 illustrates data obtained by deforming the image A only at a position where N (x, y) = 1 in order to create the image D.

The image D is the image shown in FIG. 12 because the image shown in FIG. 11 is overwritten on the image B (FIG. 8). It should be noted in the image D shown in FIG. 12 that the projected image of 20B (currently projected image of the historical building 20B) is compared with the projected image of 20A (more precisely, compared to the image B (FIG. 8)). , 20A is replaced with a deformed projection image). In addition, a projected image of a historical building 21A at that time that does not currently exist is also shown. In the image D, the color of the windows and doors of the historical building is white, not black, and is expressed in the state of the historical building 20A at that time. The image D is an image in which 20A and 21A are shown together with the current person 23B and the building 24B that was not built at that time but is now built.

Thus, after displaying image C (0) to image C (100) in order, and finally displaying image D, image A (a past photo of a historic building) is slowly deformed. Finally, you can express a scene where “historic buildings of the past” merged with the current landscape. In this way, it is possible to express photographs that can feel the flow of the times.

<< Other Examples of the Present Invention >>
In the above-described embodiment, an example in which processing is performed by the dedicated image processing apparatus 10 has been described. However, it is also possible to send an image to a computer device and generate and display the image in the computer device. In this case, although detailed description is omitted, the image C (t) and the image D can be created and displayed in the computer apparatus by the same process as the process in the image processing apparatus 10 described above.

FIG. 13 shows a configuration example of the computer apparatus 40 that creates the image C (t) and the image D and displays them.

The computer apparatus 40 includes a CPU (Central Processing Unit) 4011, a memory 4012, a display controller 4013, an input device interface 4014, and a network interface 4015. Further, the computer device 40 includes an external device interface 4016 and a digital camera interface 4018. These are connected to the bus 4017.

The display 4019 is connected to the bus 4017 via the display controller 4013. A keyboard (KBD) 4020 and a mouse 4021 are connected to the bus 4017 via the input device interface 4014. Further, the hard disk drive (HDD) 4022 and the media drive 4023 are connected to the bus 4017 via the external device interface 4016. The digital camera is connected to the bus 4017 via the digital camera interface 4018. The computer device 40 is connected to a network such as the Internet via a network interface 4015.

The CPU 4011 is a main controller of the computer device 40. The CPU 4011 executes various applications under the control of an operating system (OS). The CPU 4011 can execute an application program that processes an image (for example, the above-described image B) once downloaded from the digital camera to the hard disk drive 4022, for example. A program for performing the processing shown in FIG. 2 is implemented in the application program. The CPU 4011 is interconnected with other devices by a bus 4017.

A memory 4012 is a storage device used for storing program codes executed by the CPU 4011 and temporarily storing work data being executed. The memory 4012 includes, for example, a nonvolatile memory such as a ROM and a volatile memory such as a DRAM.

A display controller 4013 is a dedicated controller for actually processing a drawing command issued by the CPU 4011. The drawing data processed in the display controller 4013 is once written in a frame buffer (not shown), for example, and then output on the screen by the display 4019. For example, an image read from the hard disk drive 4022 is displayed on the screen of the display 4019 so that the user can see and enjoy it.

The input device interface 4014 is a device for connecting user input devices such as a keyboard 4020 and a mouse 4021 to the computer device 40. The user can input a command for executing a program for performing the processing shown in FIG. 2 via the keyboard 4020 and the mouse 4021.

The network interface 4015 connects the computer device 40 to a local network such as a LAN (Local Area Network) and further to a wide area network such as the Internet according to a predetermined communication protocol such as Ethernet (registered trademark).

On the network, a plurality of host terminals and servers (not shown) are connected in a transparent state, and a distributed computing environment is constructed. On the network, distribution services such as software programs and data contents can be provided. For example, image data can be downloaded to the hard disk drive 4022 from another server in which the image A is stored.

The digital camera interface 4018 is a device for capturing an image supplied from the digital camera. The external device interface 4016 is a device for connecting external devices such as the hard disk drive 4022 and the media drive 4023 to the computer device 40.

As conventionally known, the hard disk drive 4022 is an external storage device in which a magnetic disk as a storage carrier is fixedly mounted, and is superior to other external storage devices in terms of storage capacity and data transfer speed. The hard disk drive 4022 can also be randomly accessed. Placing the software program on the hard disk drive 4022 in an executable state is called “installation” of the program in the system. Usually, the hard disk drive 4022 stores program codes of an operating system to be executed by the CPU 4011, application programs, device drivers, and the like in a nonvolatile manner. For example, a program for performing the processing shown in FIG. 2 can be installed on the hard disk drive 4022.

The media drive 4023 is a device for loading portable media such as CD (Compact Disc), MO (Magneto-Optical disc), DVD (Digital Versatile Disc), etc. and accessing the data recording surface. The portable media is mainly used for the purpose of backing up software programs, data files, and the like as data in a computer-readable format, and for moving them between systems (that is, including sales, distribution, and distribution). Application programs for performing image processing can be physically distributed and distributed among a plurality of devices by using these portable media.

As mentioned above, although embodiment of this invention was explained in full detail, the concrete structure is not restricted to this embodiment, The design change etc. of the range which does not deviate from the summary of this invention are included.

Furthermore, this technique can also be set as the following structures.

[1]
An image processing device that generates a third image using a first image and a second image,
By matching the first image and the second image, the positional relationship between the first photographing position where the first image is photographed and the second photographing position where the second image is photographed is obtained. A shooting position calculator,
A virtual position determination unit that determines a position (referred to as a third shooting position) between the first shooting position and the second shooting position;
An image generation unit that generates an image when it is assumed that the image is captured from the third imaging position as the third image;
In the virtual position determination unit, the third photographing position moves with time on a route starting from one of the first photographing position or the second photographing position and ending at the other. West,
The image generation unit generates the third image by performing deformation on the first image according to a positional relationship of the third shooting position with respect to the first shooting position. A featured image processing apparatus.
[2]
The first image further has depth information for each pixel position;
The image processing apparatus according to [1], wherein the shooting position calculation unit obtains a positional relationship between the first shooting position and the second shooting position using the depth information.
[3]
The first image further has mask information for each pixel position;
The imaging position calculation unit performs matching using the information of the first image only for the position where the mask information exists without using the information of the first image for the position where the mask information does not exist. The image processing apparatus according to [1] or [2].
[4]
An image processing method for generating a third image using a first image and a second image,
By matching the first image and the second image, the positional relationship between the first photographing position where the first image is photographed and the second photographing position where the second image is photographed is obtained. A shooting position calculation step;
A virtual position determining step for determining a position (referred to as a third shooting position) between the first shooting position and the second shooting position;
An image generation step of generating an image when it is assumed that the image is captured from the third imaging position as the third image;
In the virtual position determination step, the third shooting position moves with time on a path starting from one of the first shooting position or the second shooting position and ending at the other. West,
In the image generation step, the third image is generated by performing deformation on the first image according to the positional relationship of the third shooting position with respect to the first shooting position. A featured image processing method.
[5]
An image processing program for generating a third image using a first image and a second image,
By matching the first image and the second image, the positional relationship between the first photographing position where the first image is photographed and the second photographing position where the second image is photographed is obtained. A shooting position calculation step;
A virtual position determining step for determining a position (referred to as a third shooting position) between the first shooting position and the second shooting position;
An image generation step of generating an image when it is assumed that the image is captured from the third imaging position as the third image;
In the virtual position determination step, the third shooting position moves with time on a path starting from one of the first shooting position or the second shooting position and ending at the other. West,
In the image generation step, the third image is generated by performing deformation on the first image according to the positional relationship of the third shooting position with respect to the first shooting position. A program to be executed by a characteristic computer.

DESCRIPTION OF SYMBOLS 10 ... Image processing apparatus 11 ... Input terminal 12 ... Feature point matching process part 13 ... Shooting position calculation part 14 ... Virtual shooting position calculation part 15 ... Deformation image creation part 16 ... Composite image creation part 17 ... Image display part 20A, 21A, 22A, 20B, 23B, 24B ... Subjects 30A, 30B, 30C ... Coordinate systems 31A, 31B, 31C in a three-dimensional space ... Screen 40 ... Computer device

Claims (5)

An image processing device that generates a third image using a first image and a second image,
By matching the first image and the second image, the positional relationship between the first photographing position where the first image is photographed and the second photographing position where the second image is photographed is obtained. A shooting position calculator,
A virtual position determination unit that determines a position (referred to as a third shooting position) between the first shooting position and the second shooting position;
An image generation unit that generates an image when it is assumed that the image is captured from the third imaging position as the third image;
In the virtual position determination unit, the third photographing position moves with time on a route starting from one of the first photographing position or the second photographing position and ending at the other. West,
The image generation unit generates the third image by performing deformation on the first image according to a positional relationship of the third shooting position with respect to the first shooting position. A featured image processing apparatus. The first image further has depth information for each pixel position;
The image processing apparatus according to claim 1, wherein the photographing position calculation unit obtains a positional relationship between the first photographing position and the second photographing position using the depth information. The first image further has mask information for each pixel position;
The imaging position calculation unit performs matching using the information of the first image only for the position where the mask information exists without using the information of the first image for the position where the mask information does not exist. The image processing apparatus according to claim 1 or 2. An image processing method for generating a third image using a first image and a second image,
By matching the first image and the second image, the positional relationship between the first photographing position where the first image is photographed and the second photographing position where the second image is photographed is obtained. A shooting position calculation step;
A virtual position determining step for determining a position (referred to as a third shooting position) between the first shooting position and the second shooting position;
An image generation step of generating an image when it is assumed that the image is captured from the third imaging position as the third image;
In the virtual position determination step, the third shooting position moves with time on a path starting from one of the first shooting position or the second shooting position and ending at the other. West,
In the image generation step, the third image is generated by performing deformation on the first image according to the positional relationship of the third shooting position with respect to the first shooting position. A featured image processing method. An image processing program for generating a third image using a first image and a second image,
By matching the first image and the second image, the positional relationship between the first photographing position where the first image is photographed and the second photographing position where the second image is photographed is obtained. A shooting position calculation step;
A virtual position determining step for determining a position (referred to as a third shooting position) between the first shooting position and the second shooting position;
An image generation step of generating an image when it is assumed that the image is captured from the third imaging position as the third image;
In the virtual position determination step, the third shooting position moves with time on a path starting from one of the first shooting position or the second shooting position and ending at the other. West,
In the image generation step, the third image is generated by performing deformation on the first image according to the positional relationship of the third shooting position with respect to the first shooting position. A program to be executed by a characteristic computer.