JP2002095011A - Method and device for generating pseudo-three dimensional image - Google Patents

Abstract

PROBLEM TO BE SOLVED: To provide a pseudo-three dimensional image generating method for displaying solid images, by using encoding and decoding technology which efficiently compresses image data. SOLUTION: The image is photographed from two or more directions, and matching is calculated between two or above original images including a viewpoint with respect to an object. The image is interpolated, based on the result of matching and a composited image viewed from a setting eye direction is generated and data of the composited image is displayed, so that a natural image which does not have unnatural divided parts or steps is displayed.

Description

DETAILED DESCRIPTION OF THE INVENTION

[0001]

BACKGROUND OF THE INVENTION 1. Field of the Invention The present invention relates to an image data processing technique, and more particularly to a method and apparatus for generating a pseudo three-dimensional image by presenting an object three-dimensionally by synthesizing an image viewed from a direction different from the photographing direction. About.

[0002]

2. Description of the Related Art In recent years, with the advance of image technology, instead of moving a camera in accordance with a change in a line of sight, images taken by capturing an object from various directions by many cameras are used, and a field of view in editing after shooting is used. By synthesizing the video when the image is moved, it is possible to present a realistic image. In addition, image data is distributed online such as the Internet, and the image data is processed and displayed as a movie, a moving image, or a 3D stereoscopic image, or can be viewed as a panoramic photograph or a continuous photograph from the viewpoint specified by the viewer. There are also things. In particular, in online commerce and the like, it is natural to present an image and advertise it. However, it is convenient to be able to observe a target product from any direction, and such a technology is used. In online shopping, it is known that the desire to purchase is increased several times only by displaying an image three-dimensionally.

As described above, a method of providing an image that follows a change in the visual field using a plurality of screens is to prepare a large number of images having overlapping visual fields and switch the displayed image according to the movement of the line of sight, or change the displayed image adjacent to each other. The images are joined so that there is no deviation, and the images are cut out and displayed according to the movement of the visual field. In such a method, there is an image shift between adjacent photographed images, so that when moving the line of sight from one photographed image to an adjacent image, it is possible to make the image partition feel natural without being aware of the partition of the image. It is necessary to make the dimensional difference in the portion sufficiently small, or to precisely adjust the size using the portion where the two images overlap.

[0004] Also, when image data is distributed online and displayed on a receiver device as a moving image or a 3D stereoscopic image on the Internet or the like, a natural image corresponding to a moving line of sight moves as the difference between adjacent images increases. Is difficult to generate. Therefore, it is necessary to prepare and use an extremely large number of image data obtained by slightly changing the photographing angles of adjacent images. As described above, stereoscopic display requires an extremely large amount of image data, which is not practical on the Internet or the like where the amount of information transmission is limited. In addition, in order to adjust the overlap between two images, it is necessary to accurately specify the correspondence between the video positions on the two screens, but it is difficult to automatically perform this.

[0005]

SUMMARY OF THE INVENTION The present invention has been made in view of the above circumstances, and has as its object to present a stereoscopic image using an encoding and decoding technique for achieving efficient compression of image data. Another object of the present invention is to provide a pseudo three-dimensional image generation method.

[0006]

SUMMARY OF THE INVENTION A pseudo three-dimensional image generating method according to the present invention calculates matching between two or more original images photographed from two or more directions and including a viewpoint for an object in the image. And a step of generating a composite image viewed from a line of sight set by interpolation based on the result of the matching, and a step of presenting data of the composite image.

According to the pseudo three-dimensional image generation method of the present invention, a composite image viewed from the line of sight, which is an intermediate point, is generated by interpolating based on the result of matching between the original images. According to the present invention, instead of simply selecting and switching and displaying a plurality of original images, matching between the original images is automatically performed, and a new image is formed and displayed by interpolating the image using the result. Therefore, a natural image display without unnatural partitions and steps as in the conventional method is possible.

Further, according to the present invention, a large number of original images of a target object are prepared so as to include a large number of expected viewpoints, and prepared when a viewer designates a viewpoint in the image. Select the original image containing the viewpoint from the original image,
Since the image centered on the designated viewpoint is synthesized and displayed, the viewer can see the object from any direction, and thus has the same feeling as observing a stereoscopic image.

Further, a plurality of original images are obtained from different positions by a camera arranged around the subject or a camera moving around the subject, and an original image including a designated viewpoint is selected from these, and these original images are selected. By performing matching calculation between the images, it is possible to generate and present a composite image having the set viewpoint at the center.

In the matching calculation step of the present invention, a multi-resolution singularity filter is applied to each original image to generate a series of hierarchical images having different resolutions. Computation is calculated, and the composite image generation step uses the positional relationship between the set viewpoint and the viewpoint of the original image, based on the pixel-to-pixel correspondence based on the position and brightness of matched pixels in the original image. An intermediate image may be generated by performing an interpolation calculation.

The image data processed in the present invention may be a moving image such as a movie or an animation, or a still image obtained by converting a three-dimensional object into a two-dimensional image such as a medical image or a product photograph. Furthermore, all the images of an arbitrary dimension corresponding to the image data for each image can be processed. Since matching is used in this method, a natural intermediate image can be obtained even if the deviation between the original images is considerably large. Therefore, the number of original images prepared to obtain an intermediate image viewed from an arbitrary viewpoint is small, and the amount of information to be handled can be small.
In addition, since the matching calculation can be performed at a high speed, it is also possible to perform image processing in real time, perform image synthesis accompanying the movement of the viewpoint, and display the moving image or movie.

Since the amount of image data required for the calculation is small,
It is possible to provide a sufficiently natural moving image even when the communication amount is restricted as in the case of the Internet or on-demand movie broadcasting. Of course, online commerce (EC)
A great effect can be brought about by applying the present invention to a product display using a 3D video, which is required in the above. Note that, in the pseudo three-dimensional image generation method of the present invention, output as encoded data of image data relating to the generated hierarchical image,
By storing or transmitting, the amount of electronic information transmission can be reduced.

[0013] A pseudo three-dimensional image generating apparatus according to the present invention includes a unit for acquiring image data of a plurality of images by photographing from a plurality of positions around a subject, and two or more images including a line of sight to an object in the image. A unit for selecting an original image, a unit for applying a multi-resolution singularity filter to each image data of the original image to generate image data for a series of hierarchical images having different resolutions, and a unit for generating image data for hierarchical images having different resolution levels. A unit that calculates matching between the original images based on the positional relationship between the set pixels and the matched pixels in the original image using the positional relationship between the set viewpoint and the viewpoint of the original image. A unit that performs interpolation calculation on a pixel-by-pixel basis to generate image data of an intermediate image, and generates a composite image viewed from a direction including the set viewpoint. Characterized in that it presented in.
Each of these units can be realized by any combination of software and hardware.

The image data of the original image used to generate the pseudo three-dimensional image can be obtained by a robot. By using a robot to acquire the required image data, the robot can run independently or operate the robot arm without having to prepare an imaging device with a large number of cameras. It is possible to automatically acquire image data of an image in which a photographing position and a photographing direction are faithfully reproduced as required.

According to the present invention, it is possible to automatically synthesize an image from a viewpoint different from that at the time of shooting by using an already-acquired image, so that an object is not stereoscopically displayed on a display device. However, since an image viewed from substantially any direction is obtained, a pseudo three-dimensional image display can be realized. As described above, it is possible to accurately transmit information of a three-dimensional object over the Internet or the like using the pseudo three-dimensional image according to the method of the present invention. It is possible to display the object so that the object can be observed from any viewpoint to stimulate purchase will.

Further, according to the present invention, since a composite image can be generated by matching calculation and interpolation at a very high speed, it can be displayed as a movie or a moving image. Furthermore, according to the matching based on the hierarchical image,
Since the information amount of the image data is sufficiently small, the original image data can be transmitted almost online and processed at the receiver side to display a pseudo three-dimensional image.

[0017]

DESCRIPTION OF THE PREFERRED EMBODIMENTS In describing a pseudo three-dimensional image generation technique according to the present invention, first, a multi-resolution singularity filter technique used in the embodiment and a technique using the same will be described with reference to FIGS. "Prerequisite technology" for image matching
It will be described in detail. These techniques have already been obtained by the present applicant in Japanese Patent No. 2927350, and are suitable for combination with the present invention. However, it goes without saying that the image matching technique that can be adopted in the embodiment is not limited to this.

[Prior Art Related to Prerequisite Technology] Automatic matching of two images, that is, correspondence between image regions and pixels, is one of the most difficult and important themes in computer vision and computer graphics. For example, if matching can be achieved between images of a certain object from different viewpoints, an image from another viewpoint can be generated. If the matching between the right-eye image and the left-eye image can be calculated, photogrammetry using a stereoscopic image is also possible. When a model of a face image is matched with another face image, characteristic face portions such as eyes, nose, and mouth can be extracted. For example, when matching is correctly performed between images of a human face and a cat face, morphing can be completely automated by automatically generating an intermediate image between them.

However, conventionally, generally, the corresponding point between two images has to be designated by a person, and a large number of work steps are required. To solve this problem, a number of corresponding point automatic detection methods have been proposed. For example, there is a method of reducing the number of corresponding point candidates by using an epipolar straight line. However, even in that case, the processing is extremely complicated. To reduce complexity, the coordinates of each point in the left-eye image are usually assumed to be approximately at the same position in the right-eye image. However, with such constraints, it is very difficult to achieve matching that satisfies both global and local features at the same time.

In volume rendering, a series of cross-sectional images is used to construct a voxel. In this case, conventionally, it is generally assumed that the pixel in the upper cross-sectional image corresponds to the pixel at the same position in the lower cross-sectional image,
These pixel pairs are used for the interpolation calculation. Since such a very simple method is used, if the distance between consecutive sections is long and the cross-sectional shape of the object changes greatly, the object constructed by volume rendering tends to be unclear.

There are many matching algorithms that use edge detection, such as the three-dimensional photogrammetry method. But in this case,
Since the resulting number of corresponding points is small, the value of the disparity must be interpolated to fill the gap between the matched corresponding points. In general, all edge detectors have difficulty determining if this really indicates the presence of an edge when the brightness of a pixel changes within the local window they use. Edge detectors are all high-pass filters in nature, and pick up noise simultaneously with edges.

As another method, an optical flow is known. When two images are given, the optical flow detects the movement of an object (rigid body) in the images. At this time, it is assumed that the brightness of each pixel of the object does not change. In the optical flow, the motion vector (u, v) of each pixel is added together with some additional conditions, for example, the smoothness of the vector field of (u, v).
Calculate v). However, the optical flow cannot detect a global correspondence between images. It focuses only on local changes in pixel brightness, and when the displacement of the image is large, system errors become significant.

Many multi-resolution filters have been proposed to recognize the global structure of an image. They are classified into linear filters and non-linear filters. Wavelets are an example of the former, but linear filters are generally
It is not very useful for image matching. This is because the information on the luminance of the pixel having the extreme value becomes gradually blurred together with the positional information. FIGS. 1A and 1B show the results of applying an averaging filter to the face images shown in FIGS. 19A and 19B, respectively. As shown in the figure, the brightness of the pixel having the extreme value gradually decreases due to the averaging, and the position shifts under the influence of the averaging. As a result, the luminance and position information of the eyes (minimum luminance points) become ambiguous at such a coarse resolution level, and correct matching cannot be calculated at this resolution. Thus, while providing a coarse resolution level is for global matching, the matching obtained here does not correspond exactly to the real features of the image (eye, i.e. the minimum point). Even if the eyes appear sharper at a finer resolution level, the errors introduced during global matching are no longer irreparable. It has also been pointed out that adding smoothing processing to an input image causes stereo information in a texture area to be reduced.

On the other hand, a one-dimensional "sieve" is used as a nonlinear filter which has recently been used in the field of topography.
There are operators. This operator applies a smoothing process to an image by selecting a local minimum (or local maximum) in a one-dimensional window of a predetermined size while preserving the causal relationship between the scale and the space. The resulting image is the same size as the original image, but is simpler because small wave components have been removed. This operator can be classified as a "multi-resolution filter" in a broad sense in terms of dropping image information, but it does not actually hierarchize the image while changing the image resolution like a wavelet. (Ie not a multi-resolution filter in the narrow sense) and cannot be used to detect correspondence between images. [Problems to be solved by the underlying technology] The following problems are recognized by summarizing the above.

1. There have been few image processing methods for accurately grasping the features of images with relatively simple processing. In particular, there have been few effective proposals regarding an image processing method capable of extracting a feature while maintaining information on a characteristic point, for example, a pixel value and a position. 2. When the corresponding points are automatically detected based on the features of the image, there are generally disadvantages such as complicated processing or low noise resistance. In addition, it is necessary to set various restrictions upon processing, and it has been difficult to perform matching that simultaneously satisfies global features and local features. 3. Even if a multi-resolution filter was introduced to recognize the global structure or features of the image, if the filter was a linear filter, the luminance and position information of the pixel would be ambiguous. As a result, it was easy for the grasp of the corresponding points to be inaccurate. The one-dimensional sieve operator, which is a non-linear filter, does not hierarchize images, and thus cannot be used for detecting corresponding points between images. 4. As a result, in order to correctly grasp the corresponding points, there was no effective means other than relying on manual designation.

The present invention has been made for the purpose of solving these problems, and provides a technique that enables accurate grasp of the characteristics of an image in the field of image processing. [Means for Solving the Problem by the Base Technology] For this purpose, an embodiment of the present invention proposes a new multi-resolution image filter. This multi-resolution filter extracts a singular point from an image. Therefore, it is also called a singular point filter. A singular point is a point having a feature on an image. As an example,
In a certain area, a pixel value (a pixel value indicates an arbitrary numerical value related to an image or a pixel such as a color number and a luminance value) is a maximum point where the pixel value is maximum, a minimum point where the pixel value is minimum, and maximum in one direction but in another direction Has a minimum saddle point. The singular point may be a concept on topological geometry. However, it may have any other features. It is not essential for the present invention what kind of property is considered a singular point.

In this embodiment, image processing using a multi-resolution filter is performed. First, in a detection step, a singular point is detected by performing a two-dimensional search on the first image. Next, in the generation step, a detected image is extracted to generate a second image having a lower resolution than the first image. The second image inherits the singular point of the first image. Since the second image has a lower resolution than the first image, it is suitable for grasping global characteristics of the image.

Another embodiment of the present invention relates to an image matching method using a singular point filter. In this embodiment, matching between the start image and the end image is performed. The start point image and the end point image are names given for convenience in distinguishing the two images, and there is no essential difference.

In this embodiment, in the first step, a singular point filter is applied to the starting image to generate a series of starting hierarchical images having different resolutions. In the second step, a series of end point hierarchical images having different resolutions are generated by applying a singularity filter to the end point image. The start hierarchical image and the end hierarchical image refer to an image group obtained by hierarchizing the start image and the end image, respectively, and each include at least two images. Next, in the third step, matching between the start hierarchical image and the end hierarchical image is calculated in the resolution level hierarchy. According to this aspect, the features of the image associated with the singularity are extracted and / or clarified by the multi-resolution filter,
Matching becomes easy. No particular constraint is required for matching.

Still another preferred embodiment according to the present invention relates to matching of a start image and an end image. In this aspect, an evaluation formula is provided in advance for each of the plurality of matching evaluation items,
The comprehensive evaluation formula is defined by integrating the evaluation formulas, and the optimum matching is searched by focusing on the vicinity of the extreme value of the comprehensive evaluation formula. The comprehensive evaluation formula may be defined as a sum of those evaluation formulas after multiplying at least one of the evaluation formulas by the coefficient parameter. In this case, the comprehensive evaluation formula or one of the evaluation formulas takes an extreme value. The parameter may be determined by detecting a state. The reason why "approximate to an extreme value" or "approximately takes an extreme value" is that an error may be included to some extent. Some errors are not a problem for the present invention.

Since the extreme value itself also depends on the parameters, there is room for determining a parameter considered to be optimal based on the behavior of the extreme value, that is, the manner of change of the extreme value. This embodiment makes use of that fact. According to this aspect,
This opens up ways to automate the determination of parameters that are inherently difficult to adjust. [Embodiment of Base Technology] First, the basic technology of the base technology will be described in detail in [1], and the processing procedure will be specifically described in [2].
The results of the experiment are reported in [3]. [1] Details of Elemental Technology [1.1] Introduction A new multi-resolution filter called a singularity filter is introduced to accurately calculate matching between images. No prior knowledge of the object is required. The calculation of matching between images is calculated at each resolution while proceeding through the resolution hierarchy. At that time, the hierarchy of the resolution is sequentially traced from a coarse level to a fine level. The parameters required for the calculations are set completely automatically by dynamic calculations similar to the human visual system. It is not necessary to manually specify corresponding points between images.

The prerequisite technique can be applied to, for example, completely automatic morphing, object recognition, stereogrammetry, volume rendering, generation of a smooth moving image from images with small intervals. When used for morphing,
A given image can be automatically transformed. When used for volume rendering, an intermediate image between sections can be accurately reconstructed. The same applies to the case where the distance between the sections is long and the shape of the section greatly changes.

[1.2] Hierarchy of Singularity Filter The multi-resolution singularity filter according to the base technology can reduce the resolution of an image while preserving the brightness and position of each singularity included in the image. Here, the width of the image is N and the height is M. For simplicity, N = M = 2n
(N is a natural number). Also, the interval [0, N] ⊂R
Is described as I. Let the pixel of the image at (i, j) be p
Described as (i, j) (i, j∈I).

Here, a multi-resolution hierarchy is introduced. The hierarchized image group is generated by a multi-resolution filter. The multi-resolution filter performs a two-dimensional search on the original image to detect a singular point, and extracts the detected singular point to generate another image having a lower resolution than the original image. .
Here, the size of each image at the m-th level is 2mx2m
(0 ≦ m ≦ n). The singularity filter recursively constructs the following four types of new hierarchical images in a direction descending from n.

[0035]

(Equation 1) But where

(Equation 2) And Hereinafter, these four images are called sub-images (sub-images). If min _{x ≦ t ≦ x + 1} and max _{x ≦ t ≦ x + 1} are described as α and β, respectively, the sub-images can be described as follows.

P (m, 0) = α (x) α (y) p (m + 1,0) P (m, 1) = α (x) β (y) p (m + 1,1) P (m, 2 ) = Β (x) α (y) p (m + 1,2) P (m, 3) = β (x) β (y) p (m + 1,3) That is, these are like the tensor product of α and β. It is considered something. Each sub-image corresponds to a singular point.
As is apparent from these equations, the singularity filter detects a singularity for each block composed of 2 × 2 pixels in the original image. At this time, a point having a maximum pixel value or a minimum pixel value is searched for in two directions of each block, that is, in the vertical and horizontal directions. As the pixel value, luminance is adopted in the base technology, but various numerical values relating to an image can be adopted. The pixel with the maximum pixel value in both directions is the local maximum point, the pixel with the minimum pixel value in both directions is the local minimum point, the maximum pixel value in one of the two directions, and the minimum pixel value in the other. Are detected as saddle points.

The singular point filter represents an image of a singular point (here, one pixel) detected inside each block by representing the image of the block (here, four pixels).
Decrease the image resolution. From a theoretical point of view of the singularity, α (x) α (y) preserves the minimum point and β (x) β
(Y) preserves the maxima, and α (x) β (y) and β
(X) α (y) preserves the saddle point.

First, a singular point filter process is separately performed on a start point (source) image and an end point (destination) image to be matched to generate a series of image groups, that is, a start point hierarchical image and an end point hierarchical image. Keep it. The start point hierarchical image and the end point hierarchical image are each generated in four types corresponding to the type of the singular point.

Thereafter, the start hierarchical image and the end hierarchical image are matched in a series of resolution levels. First, the minimum point is matched using p (m, 0). Next, based on the result, a saddle point is matched using p (m, 1), and another saddle point is matched using p (m, 2). And finally p (m, 3)
Is used to match the maximum point.

FIGS. 1 (c) and 1 (d) correspond to FIG.
(A) and the sub-image p (5,0) of FIG.1 (b) are shown. Similarly, FIGS. 1E and 1F show p (5,1),
FIGS. 1 (g) and 1 (h) show p (5, 2), and FIGS. 1 (i) and 1 (j) show p (5, 3). As can be seen from these figures, the matching of the characteristic portions of the image is facilitated by using the sub-image. First, eyes become clear by p (5,0). This is because the eyes are the minimum point of luminance in the face. According to p (5,1), the mouth becomes clear. This is because the mouth has low brightness in the horizontal direction. According to p (5,2), the vertical lines on both sides of the neck become clear. Finally, p (5,3) defines the brightest points on the ears and cheeks. This is because these are the maximum points of luminance.

According to the singular point filter, the features of the image can be extracted.
By comparing the features of several objects recorded in advance, the subject reflected on the camera can be identified.

[1.3] Calculation of Mapping Between Images The pixel at the position (i, j) of the starting image is represented by p (n) (i, j).
And the pixel at the position (k, l) of the end point image is q
(N) Described by (k, l). i, j, k, l∈I. Defines the energy of mapping between images (described below). This energy is determined by the difference between the luminance of the pixel of the start image and the luminance of the corresponding pixel of the end image, and the smoothness of the mapping. First, p (m, 0) and q with minimum energy
Mapping f (m, 0) between (m, 0): p (m, 0) → q
(M, 0) is calculated. Based on f (m, 0), a mapping f (m, 1) between p (m, 1) and q (m, 1) having the minimum energy is calculated. This procedure is p (m, 3)
And until the calculation of the mapping f (m, 3) between q (m, 3) is completed. Each mapping f (m, i) (i = 0, 1, 2,
...) is called a submap. For the convenience of calculating f (m, i), the order of i can be rearranged as follows: The reason why the sorting is necessary will be described later.

(Equation 3) Here, σ (i) {0, 1, 2, 3}.

[1.3.1] Bijection When the matching between the start image and the end image is expressed by a mapping, the mapping should satisfy the bijection condition between both images. This is because there is no conceptual advantage between the two images, and each pixel should be connected in a bijective and injective manner. However, unlike the usual case, the mapping to be constructed here is a digital version of the bijection. In the base technology, pixels are specified by grid points.

The mapping from the start point sub-image (sub-image provided for the start-point image) to the end-point sub-image (sub-image provided for the end-point image) is f (m, s): I / 2n-m × I
/ 2n-m → I / 2n-m × I / 2n-m (s = 0,
1,...). Where f (m, s)
(I, j) = (k, l) is p (m, s) of the starting image
This means that (i, j) is mapped to q (m, s) (k, l) of the destination image. For simplicity, f (i, j) =
When (k, l) is satisfied, the pixel q (k, l) is represented by qf
(I, j).

When the data is discrete like pixels (grid points) handled in the base technology, the definition of bijection is important. Here, it is defined as follows (i, i ', j, j', k,
1 is an integer). First, each square area represented by R in the plane of the starting image,

(Equation 4) (I = 0, ..., 2m-1, j = 0, ..., 2m-
1). Here, the direction of each side (edge) of R is determined as follows.

[0046]

(Equation 5) This square must be mapped by mapping f to a quadrilateral in the destination image plane. a quadrilateral indicated by f (m, s) (R),

(Equation 6) Must satisfy the following bijection conditions:

1. The edges of the quadrilateral f (m, s) (R) do not intersect each other. 2. The directions of the edges of f (m, s) (R) are equal to those of R (clockwise in FIG. 2). 3. Shrinkage mapping (retraction: retrac
allowtions).

This is because there is only a unit mapping that completely satisfies the bijection condition unless some relaxation condition is provided. Here, the length of one edge of f (m, s) (R) may be 0, that is, f (m, s) (R) may be a triangle. However, the figure must not be a figure having an area of 0, that is, one point or one line segment. FIG. 2 (R)
2 (A) and FIG. 2 (D) satisfy the bijection condition, but FIG. 2 (B), FIG. 2 (C), FIG.
(E) is not satisfied.

In an actual implementation, the following condition may be further imposed to easily guarantee that the mapping is surjective. That is, each pixel on the boundary of the start image is mapped to a pixel occupying the same position in the end image. That is, f (i, j) = (i,
j) (where i = 0, i = 2m-1, j = 0, j = 2m
-1 on four lines). This condition is hereinafter also referred to as “additional condition”.

[1.3.2] Energy of mapping [1.3.2.1] Cost related to luminance of pixel Define energy of mapping f. The goal is to find a mapping that minimizes energy. The energy is mainly determined by the difference between the luminance of the pixel of the start image and the luminance of the corresponding pixel of the end image. That is, the energy C (m, s) (i, j) at the point (i, j) of the mapping f (m, s)
Is determined by the following equation.

[0051]

(Equation 7) Here, V (p (m, s) (i, j)) and V (q
(M, s) f (i, j)) is a pixel p (m, s)
(I, j) and q (m, s) are the luminances of f (i, j). The total energy C (m, s) of f is one evaluation formula for evaluating the matching, and the following C
(M, s) (i, j).

[0052]

(Equation 8)

[1.3.2.2] Cost related to pixel position for smooth mapping In order to obtain a smooth mapping, another energy Df for the mapping is introduced. This energy is independent of the brightness of the pixel and is independent of p (m, s) (i, j) and q (m, s) f
(I = 0, ..., 2m-) determined by the position of (i, j)
1, j = 0, ..., 2m-1). The energy D (m, s) (i, j) of the mapping f (m, s) at the point (i, j) is defined by the following equation.

[0054]

(Equation 9) Here, the coefficient parameter η is a real number equal to or greater than 0, and

(Equation 10)

[Equation 11] And here,

(Equation 12) And for i ′ <0 and j ′ <0, f (i ′,
j ′) is determined to be 0. E0 is (i, j) and f (i,
j) is determined by the distance. E0 prevents pixels from being mapped to pixels that are too far apart. However, E0 will be replaced by another energy function later. E1 guarantees the smoothness of the mapping. E1 represents the distance between the displacement of p (i, j) and the displacement of its adjacent point. Based on the above consideration, energy Df, which is another evaluation expression for evaluating matching, is determined by the following expression.

[0055]

(Equation 13) [1.3.2.3] Total energy of mapping The total energy of the mapping, that is, the total evaluation formula relating to the integration of a plurality of evaluation formulas, is defined by λC (m, s) f + D (m, s) f. Here, the coefficient parameter λ is a real number of 0 or more. The purpose is to detect a state where the comprehensive evaluation expression takes an extreme value, that is, to find a mapping giving the minimum energy represented by the following expression.

[Equation 14]

It should be noted that for λ = 0 and η = 0, the mapping is a unit mapping (ie, all i =
0, ..., 2m-1 and j = 0, ..., 2m-1
(M, s) (i, j) = (i, j)). As will be described later, in the base technology, since the case where λ = 0 and η = 0 are evaluated first, the mapping can be gradually deformed from the unit mapping. Suppose that the position of λ in the comprehensive evaluation formula is changed and C
Assuming that (m, s) f + λD (m, s) f,
When λ = 0 and η = 0, the comprehensive evaluation formula is C (m, s) f
And the pixels that have no relation at all are simply associated with each other simply because of the close luminance, and the mapping becomes meaningless. There is no point in transforming a mapping based on such a meaningless mapping. For this reason, consideration is given to how to give the coefficient parameters so that the unit mapping is selected as the best mapping at the start of the evaluation.

The optical flow is also similar to this base technology.
Consider the brightness difference and smoothness of the pixels. However, optical flow cannot be used for image conversion. This is because only the local motion of the object is considered.
Global correspondence can be detected by using the singularity filter according to the base technology.

[1.3.3] Determination of mapping by introducing multiple resolutions Mapping fm that satisfies bijection condition by giving minimum energy
in is obtained using a multi-resolution hierarchy. At each resolution level, the mapping between the starting sub-image and the ending sub-image is calculated. Starting from the highest level (the coarsest level) in the resolution hierarchy, the mapping of each resolution level is determined taking into account the mapping of the other levels. The number of mapping candidates at each level is limited by using higher, or coarser, levels of mapping. More specifically, when determining a mapping at a certain level, a mapping obtained at a level lower than that is imposed as a kind of constraint condition.

First,

(Equation 15) Holds, p (m−1, s) (i ′, j ′), q
(M-1, s) (i ', j') is respectively p (m, s)
Parents of (i, j) and q (m, s) (i, j) will be called. [X] is the maximum integer not exceeding x. Also, p (m, s) (i, j), q (m, s) (i,
j) is p (m−1, s) (i ′, j ′), q
It is called the child of (m-1, s) (i ', j'). The function parent (i, j) is defined by the following equation.

[0060]

(Equation 16) The mapping f (m, s) between p (m, s) (i, j) and q (m, s) (k, l) is determined by performing the energy calculation and finding the minimum one. Is done. f (m, s)
The value of (i, j) = (k, l) is f (m-1, s) (m =
, N) is determined as follows. First, q (m, s) (k, l) imposes a condition that it must be inside the following quadrilateral, and narrows down the mappings that satisfy the bijection condition with high reality.

[0061]

[Equation 17] But where

(Equation 18) It is. The quadrilateral thus determined is expressed as p (m, s)
Let's call it the (i, j) inherited quadrilateral. The pixel which minimizes the energy inside the inherited quadrilateral is determined.

FIG. 3 shows the above procedure. In the figure, pixels A, B, C, and D of the start image are represented by A ', B', C ', and C' of the end image at the m-1 level.
It is mapped to D '. The pixel p (m, s) (i, j) is a pixel q existing inside the inherited quadrilateral A'B'C'D '.
(M, s) f (m) (i, j). With the above considerations, the mapping from the (m-1) th level mapping to the mth level mapping is performed.

The energy E0 defined above is replaced by the following equation in order to calculate the submap f (m, 0) at the m-th level.

[0064]

[Equation 19] The following equation is used to calculate the submap f (m, s).

(Equation 20) In this way, a mapping is obtained in which the energies of all submappings are kept low. According to Equation 20, the submappings corresponding to different singularities are associated within the same level so that the submappings have a high degree of similarity. Equation 19 gives f (m, s)
It shows the distance between (i, j) and the position of the point to be projected at (i, j) when considered as a part of the pixel of the (m-1) th level.

If no pixel satisfying the bijection condition exists inside the inherited quadrilateral A'B'C'D ', the following measures are taken. First, a pixel whose distance from the boundary line of A'B'C'D 'is L (initially L = 1) is examined. If the one with the smallest energy satisfies the bijection condition, it is selected as the value of f (m, s) (i, j). L is increased until such a point is found or L reaches its upper limit L (m) max. L (m) max is fixed for each level m.
If no such point is found, a mapping in which the area of the destination quadrilateral becomes zero by temporarily ignoring the third condition of bijection is also recognized, and f (m, s) (i , J). If a point satisfying the condition is still not found, the first and second conditions for bijection are removed.

The approximation using multiple resolutions is essential for determining the global correspondence between images while avoiding the mapping being affected by image details. Unless an approximation method based on multiple resolutions is used, it is impossible to find a correspondence between pixels at a long distance. In that case, the size of the image must be limited to a very small one, and only an image with a small change can be handled. Further, since the mapping is usually required to be smooth, it is difficult to find the correspondence between such pixels. This is because the energy of mapping from a pixel at a distance to the pixel is high. According to the approximation method using multiple resolutions, an appropriate correspondence between such pixels can be found. This is because those distances are small at the upper level (coarse level) of the resolution hierarchy.

[1.4] Automatic Determination of Optimum Parameter Values One of the main drawbacks of the existing matching techniques is the difficulty of parameter adjustment. In most cases, parameter adjustments are made manually and it is extremely difficult to select the optimal value. According to the method according to the base technology, the optimal parameter value can be completely automatically determined.

The system according to the base technology includes two parameters, λ and η. In short, λ is the weight of the difference in luminance between pixels, and η indicates the rigidity of the mapping. The initial values of these parameters are 0. First, η is fixed to 0 and λ is gradually increased from 0. When increasing the value of λ and minimizing the value of the comprehensive evaluation expression (Expression 14), the value of C (m, s) f for each submapping generally decreases. This basically means that the two images have to match better. However, when λ exceeds the optimum value, the following phenomenon occurs.

1. Pixels that should not correspond to each other are erroneously associated simply because the luminance is close. 2. As a result, the correspondence between pixels becomes strange,
The mapping begins to collapse. 3. As a result, D (m, s) f tends to increase sharply in equation (14). 4. As a result, the value of equation (14) tends to increase sharply, so that f (m, s) f is suppressed so as to suppress a sharp increase.
(M, s) changes and consequently C (m, s) f increases. Therefore, while maintaining the state where Expression 14 takes the minimum value while increasing λ, a threshold value at which C (m, s) f changes from decreasing to increasing is detected, and the λ is set as the optimal value at η = 0. Next, the behavior of C (m, s) f is inspected by gradually increasing η, and η is automatically determined by a method described later. Its η
Is also determined corresponding to

This method is similar to the operation of the focus mechanism of the human visual system. In the human visual system, the image of the left and right eyes is matched while moving one eye. When an object is clearly recognizable, its eyes are fixed.

[1.4.1] Dynamic determination of λ λ is increased from 0 at a predetermined interval, and each time the value of λ changes, the submap is evaluated. As in Equation 14, the total energy is defined by λC (m, s) f + D (m, s) f. D (m, s) f in Equation 9 represents smoothness, and theoretically becomes minimum in the case of unit mapping, and E0 and E1 increase as the mapping becomes more distorted. Since E1 is an integer, the minimum step size of D (m, s) f is 1. Therefore, the change (decrease amount) of the current λC (m, s) (i, j)
If is not greater than 1, the total energy cannot be reduced by changing the mapping. Because D (m, s) f increases by 1 or more with the change of the mapping,
This is because the total energy does not decrease unless C (m, s) (i, j) decreases by 1 or more.

Under these conditions, it is shown that C (m, s) (i, j) decreases in a normal case as λ increases.
The histogram of C (m, s) (i, j) is described as h (l). h (l) is the energy C (m, s) (i, j)
Is the number of pixels with l2. In order to satisfy λ12 ≧ 1, let us consider, for example, the case of 12 = 1 / λ. When λ changes by a small amount from λ1 to λ2,

[0073]

(Equation 21) A pixels represented by

(Equation 22) Changes to a more stable state with the energy of Here, it is assumed that all of the energies of these pixels become zero. This equation shows that the value of C (m, s) f is

[0074]

(Equation 23) Only change, so that

[0075]

(Equation 24) Holds. Since h (l)> 0, C (m,
s) f decreases. However, when λ exceeds the optimum value, the above phenomenon, that is, an increase in C (m, s) f occurs. By detecting this phenomenon, the optimum value of λ is determined.

When H (h> 0) and k are constants,

(Equation 25) Assuming that

(Equation 26) Holds. At this time, if k ≠ -3,

[0077]

[Equation 27] Becomes This is the general formula of C (m, s) f (C is a constant).

When detecting the optimum value of λ, the number of pixels that violates the bijection condition may be examined for safety. Here, when determining the mapping of each pixel, it is assumed that the probability of violating the bijection condition is p0. in this case,

[0079]

[Equation 28] Holds, the number of pixels that violate the bijection condition increases at the following rate.

(Equation 29) Therefore,

[Equation 30] Is a constant. Assuming h (l) = Hlk,
For example,

(Equation 31) Becomes a constant. However, when λ exceeds the optimal value, the above value increases rapidly. This phenomenon is detected, and B0λ3 / 2 + k
It is possible to determine whether the value of / 2 / 2m exceeds the abnormal value B0thres and determine the optimal value of λ. Similarly, the value of B1λ3 / 2 + k / 2 / 2m is an abnormal value B1th
By checking whether or not res is exceeded, the rate of increase B1 of pixels that violates the third condition of bijection is confirmed. The reason for introducing the factor 2m will be described later. This system is not sensitive to these two thresholds. These thresholds can be used to detect excessive distortion of the mapping that cannot be detected by observation of the energy C (m, s) f.

In the experiment, when the submap f (m, s) is calculated, if λ exceeds 0.1, the calculation of f (m, s) is stopped and the calculation is shifted to the calculation of f (m, s + 1). did. λ> 0.
When 1, the difference of only “3” in the 255 levels of luminance of the pixel affected the calculation of the submapping, and λ> 0.1
It was difficult to get the correct result at that time.

[1.4.2] Histogram h (l) The inspection of C (m, s) f does not depend on the histogram h (l). When testing for bijection and its third condition, h (l)
Can be affected. When (λ, C (m, s) f) is actually plotted, k is usually around 1. In the experiment, k = 1
Was used to examine B0λ2 and B1λ2. If the true value of k is less than 1, B0λ2 and B1λ2 do not become constants, but gradually increase according to the factor λ (1-k) / 2. If h (l) is a constant, for example, the factor is λ
It is 1/2. However, such a difference is due to the threshold B0thre
It can be absorbed by setting s correctly.

Here, as shown in the following equation, the center of the starting image is (x
(0, y0) and a circular object with a radius r.

(Equation 32) On the other hand, the end point image has the center (x1, y1) as in the following equation,
Assume that the object has a radius of r.

[0083]

[Equation 33] Here, it is assumed that c (x) has a form of c (x) = xk. If the centers (x0, y0) and (x1, y1) are far enough,
The histogram h (l) has the form shown below.

[0084]

(Equation 34) When k = 1, the image shows objects with sharp boundaries embedded in the background. This object is darker in the center and brighter toward the periphery. When k = -1, the image represents an object with ambiguous boundaries. This object is brightest in the center and gets darker as it goes around. The generality of objects can be considered to be intermediate between these two types of objects without loss of generality. Therefore, k can cover most cases as −1 ≦ k ≦ 1, and it is guaranteed that Equation 27 is generally a decreasing function.

It should be noted that as can be seen from Equation 34, r is affected by the resolution of the image, that is, r is proportional to 2 m. For this, [1.4.1]
Introduced a factor of 2 m.

[1.4.3] Dynamic Determination of η The parameter η can be automatically determined in the same manner. First, η = 0, and the final mapping f at the finest resolution
Calculate (n) and energy C (n) f. Subsequently, η is increased by a certain value Δη, and again the final mapping f (n) and the energy C (n) f at the finest resolution
Is recalculated. This process is continued until the optimum value is obtained.
η indicates the rigidity of the mapping. This is because the weight of the following equation.

[0087]

(Equation 35) When η is 0, D (n) f is determined independently of the immediately preceding submap, and the current submap is elastically deformed and excessively distorted. On the other hand, when η is a very large value, D
(N) f is almost completely determined by the immediately preceding submapping. At this time, the sub-mapping is very rigid, and the pixels are projected at the same place. As a result, the mapping becomes a unit mapping. When the value of η gradually increases from 0, C (n) f gradually decreases as described later. However, when the value of η exceeds the optimum value, FIG.
As shown in the figure, the energy starts to increase. In the figure, the X axis is η, and the Y axis is Cf.

In this way, an optimum value of η that minimizes C (n) f can be obtained. However, as compared with the case of λ, various factors affect the calculation, and as a result, C (n) f changes while slightly fluctuating. In the case of λ, the submapping is only recalculated once every time the input changes by a small amount.
In the case of, all submappings are recalculated. Therefore, it is not possible to immediately determine whether the obtained value of C (n) f is the minimum. If a candidate for the minimum value is found, it is necessary to search for the true minimum value by setting a finer section.

[1.5] Super Sampling When determining the correspondence between pixels, the range of f (m, s) can be extended to R × R in order to increase the degree of freedom (R is a real number). set). In this case, the brightness of the pixel of the end point image is interpolated, and the non-integer point,

[0090]

[Equation 36] F (m, s) with a luminance at is provided. That is, super sampling is performed. In the experiment, f (m,
s) is allowed to take integer and half-integer values,

(37) Is

(38) Given by

[1.6] Normalization of Luminance of Pixels in Each Image When the start image and the end image include extremely different objects, it is difficult to use the original pixel luminance as it is for calculating the mapping. This is because the energy C (m, s) f relating to the luminance is too large due to a large difference in luminance, and it is difficult to make a correct evaluation.

For example, consider a case where a human face and a cat face are matched as shown in FIGS. 20 (a) and 20 (b). The cat's face is covered with hair and contains very bright and very dark pixels. In this case, the sub-image is first normalized in order to calculate the sub-map between the two faces. That is, the brightness of the darkest pixel is set to 0, the brightness of the brightest pixel is set to 255, and the brightness of the other pixels is obtained by linear interpolation.

[1.7] Implementation A recursive method in which the calculation proceeds linearly according to the scanning of the starting point image is used. First, the value of f (m, s) is determined for the top left pixel (i, j) = (0,0). Next, while increasing i by 1, each f (m, s)
Determine the value of (i, j). When the value of i reaches the width of the image, the value of j is increased by 1 and i is returned to 0. Thereafter, f (m, s) (i, j) is determined in accordance with the scanning of the starting point image. If the correspondence of pixels is determined for all points, one mapping f (m, s) is determined.

For a given p (i, j), the corresponding point qf
Once (i, j) is determined, the corresponding point qf (i, j + 1) of p (i, j + 1) is next determined. At this time, qf
The position of (i, j + 1) is q
Limited by the position of f (i, j). Therefore,
The priority of this system is higher as the corresponding point is determined first. If the state where (0,0) always has the highest priority continues, an extra deflection is added to the final mapping required. In the base technology, f (m, s) is used to avoid this state.
Is determined by the following method.

First, when (s mod 4) is 0, (0,
0) is determined as a starting point while gradually increasing i and j. When (s mod 4) is 1, the right end point of the uppermost line is set as a start point, and i is decreased and j is increased. When (s mod 4) is 2, the right end point of the bottom row is set as a starting point, and the values are determined while decreasing i and j. (S
When mod 4) is 3, the left end point of the bottom line is set as a start point, and i is increased and j is decreased while j is determined. Since the concept of submapping, that is, the parameter s does not exist at the n-th level having the finest resolution, s = 0 and s =
The two directions were continuously calculated as being 2.

In an actual implementation, a penalty is given to a candidate that violates the bijection condition, so that f (m, s) ( i, j) (m = 0,..., n). The energy D (k,
l) is multiplied by φ, while candidates that violate the first or second condition are multiplied by ψ. This time φ = 2, ψ = 100000
Was used.

For checking the above-mentioned bijection condition, the following test was performed when (k, l) = f (m, s) (i, j) was determined as an actual procedure. That is, f (m,
s) Each lattice point (k, k) included in the inherited quadrilateral of (i, j)
For 1), it is checked whether or not the z component of the outer product of the following equation becomes 0 or more.

[0098]

[Equation 39] But where

(Equation 40)

[Equation 41] (Where the vector is a three-dimensional vector, and the z-axis is defined in an orthogonal right-handed coordinate system). If W is negative, then for that candidate D (m, s) (k, l)
To give a penalty and avoid choices as much as possible.

FIGS. 5A and 5B show the reason for checking this condition. FIG. 5A shows a candidate without a penalty, and FIG. 5B shows a candidate with a penalty. Mapping f (m, s) for adjacent pixel (i, j + 1)
When determining (i, j + 1), if the z component of W is negative, no pixel satisfies the bijection condition on the source image plane. This is because q (m, s) (k, l) crosses the boundary of an adjacent quadrilateral.

[1.7.1] Order of submapping In the implementation, when the resolution level is even, σ (0) = 0, σ (1) = 1, σ (2) = 2, σ
(3) = 3, σ (4) = 0 is used, and when the number is odd, σ
(0) = 3, σ (1) = 2, σ (2) = 1, σ (3) =
0 and σ (4) = 3 were used. As a result, the sub-mapping was shuffled appropriately. Note that there are originally four types of submappings,
s is any of 0 to 3. However, actually, s = 4
Was performed. The reason will be described later.

[1.8] Interpolation Calculation After the mapping between the start image and the end image is determined, the luminance of the corresponding pixels is interpolated. In the experiment, trilinear interpolation was used. The square p (i,
j) p (i + 1, j) p (i, j + 1) p (i + 1, j
+1) is a quadrangle qf (i, j) qf on the destination image plane
(I + 1, j) qf (i, j + 1) qf (i + 1, j +
Assume that it is projected to 1). For simplicity, the distance between images is set to 1. The distance from the starting image plane is t (0 ≦ t ≦
1) pixel r (x, y, t) of the intermediate image (0 ≦ x ≦
N−1, 0 ≦ y ≦ M−1) is obtained in the following manner.
First, the position of the pixel r (x, y, t) (where x, y, tｙ)
R) is determined by the following equation.

[0102]

(Equation 42) Subsequently, the luminance of the pixel at r (x, y, t) is determined using the following equation.

[0103]

[Equation 43] Here, dx and dy are parameters and change from 0 to 1.

[1.9] Mapping when constraint conditions are imposed The determination of a mapping when no constraint conditions exist without any change has been described. However, when the correspondence between the specific pixels of the start point image and the end point image is defined in advance, the mapping can be determined based on the constraint.

The basic idea is that the starting image is roughly transformed by a rough mapping that moves a specific pixel of the starting image to a specific pixel of the end image, and then the mapping f is calculated accurately.

First, a specific mapping of a specific pixel of the start image to a specific pixel of the end image and a projection of another pixel of the start image to an appropriate position are determined. That is, a pixel that is close to a particular pixel is a mapping that is projected near where the particular pixel is projected. Here, the rough mapping of the m-th level is described as F (m).

A rough mapping F is determined in the following manner. First, mappings are specified for some pixels. Ns pixels for the source image,

[Equation 44] When specifying, determine the following values:

[Equation 45] The displacement amount of the other pixels of the starting image is p (ih, jh) (h
= 0,..., Ns-1). That is, the pixel p (i, j) is projected to the following pixel of the end point image.

[0108]

[Equation 46] But where

[Equation 47]

[Equation 48] And

Next, the energy D (m, s) (i, j) of the mapping f is changed so that the candidate mapping f close to F (m) has less energy. To be precise, D
(M, s) (i, j) is

[Equation 49] It is. However,

[Equation 50] And κ, ρ ≧ 0. Finally, f is completely determined by the automatic mapping calculation process described above.

Here, f (m, s) (i, j) is F (m)
When they are close enough to (i, j), that is, their distance is

(Equation 51) Note that E2 (m, s) (i, j) goes to 0 when The reason for such a definition is
This is because, as long as each f (m, s) (i, j) is sufficiently close to F (m) (i, j), its value is to be automatically determined so as to settle to an appropriate position in the end point image. For this reason, the exact correspondence need not be specified in detail, and the start image is automatically mapped to match the end image.

[2] Specific Processing Procedure The flow of processing according to each element technology of [1] will be described. FIG.
Is a flowchart showing the overall procedure of the base technology. As shown in the figure, first, processing using a multi-resolution singularity filter is performed (S1), and then, matching between the start image and the end image is performed (S2). However, S2 is not essential,
Processing such as image recognition may be performed based on the features of the image obtained in S1.

FIG. 7 is a flowchart showing the details of S1 in FIG. Here, it is assumed that the start image and the end image are matched in S2. Therefore, the starting point image is first hierarchized by the singular point filter (S1).
0), obtain a series of starting hierarchical images. Subsequently, the end image is hierarchized by the same method (S11), and a series of end hierarchical images is obtained. However, the order of S10 and S11 is arbitrary, and the start hierarchical image and the end hierarchical image can be generated in parallel.

FIG. 8 is a flowchart showing the details of S10 of FIG. The size of the original start image is 2n × 2n. Since the starting hierarchical image is created in order from the one with the smallest resolution, the parameter m indicating the resolution level to be processed is set to n (S100). Subsequently, the m-th level images p (m, 0), p (m, 1), p (m,
2), a singular point is detected from p (m, 3) using a singular point filter (S101), and the images p (m-1, 0) and p (m-1, 1) at the (m-1) th level, respectively. , P (m−1,
2), p (m-1, 3) is generated (S102). Here, since m = n, p (m, 0) = p (m, 1) =
p (m, 2) = p (m, 3) = p (n), and four types of sub-images are generated from one start-point image.

FIG. 9 shows a part of the m-th level image and the m-th level image.
The correspondence of a part of the one-level image is shown. Numerical values in the figure show the luminance of each pixel. In the figure, p (m, s) is p
Symbolizes four images (m, 0) to p (m, 3). When generating p (m-1, 0), p (m, 0)
s) is considered to be p (m, 0). According to the rule shown in [1.2], p (m−1,0) is, for example, “3”, p (m−1,1) out of four pixels included in a block in which luminance is written in FIG. ) Is “8”, p (m−1,
In 2), “10” is obtained for “6” and p (m−1,3), respectively, and this block is replaced with one obtained pixel. Therefore, the size of the sub-image at the (m-1) th level is 2m-1 × 2m-1.

Subsequently, m is decremented (S1 in FIG. 8).
03), and confirm that m is not negative (S10).
4) Returning to S101, a sub-image having a coarse resolution is generated next. As a result of this repetition processing, S = 0 ends when m = 0, that is, when the 0th level sub-image is generated. The size of the 0th level sub-image is 1 × 1.

FIG. 10 exemplifies the starting hierarchical image generated in S10 when n = 3. Only the first starting point image is common to the four series, and thereafter, sub-images are generated independently according to the types of singularities. Note that the processing in FIG. 8 is common to S11 in FIG. 7, and the destination hierarchical image is also generated through the same procedure. Thus, S in FIG.
1 is completed.

In the base technology, a preparation for matching evaluation is made in order to proceed to S2 in FIG. FIG. 11 shows the procedure. As shown in the figure, first, a plurality of evaluation expressions are set (S30). Energy C (m, s) f for the pixel introduced in [1.3.2.1] and energy D (m, s) for the smoothness of the mapping introduced in [1.3.2.2].
f is that. Next, these evaluation expressions are integrated to form a comprehensive evaluation expression (S31). Total energy λC (m, s) f + D (m, s) f introduced in [1.3.2.3]
And using η introduced in [1.3.2.2], 、 (λC (m, s) (i, j) + ηE0 (m, s) (i, j) + E1 (m, s ) (I, j)) (Equation 52). However, the sum is 0, 1 for i and j, respectively.
... Calculated as 2m-1. The preparation for the matching evaluation is now completed.

FIG. 12 is a flowchart showing the details of S2 in FIG. As described in [1], matching between the start hierarchical image and the end hierarchical image is performed between images having the same resolution level. In order to obtain good global matching between images, matching is calculated in order from the level with the lowest resolution. Since the start point hierarchical image and the end point hierarchical image are generated using the singular point filter, the position and luminance of the singular point are clearly preserved even at a coarse resolution level, and the result of global matching is lower than in the past. It will be very good.

As shown in FIG. 12, first, the coefficient parameter η is set to 0 and the level parameter m is set to 0 (S20). Subsequently, matching is calculated between each of the four sub-images at the m-th level in the start hierarchical image and each of the four sub-images at the m-th level in the destination hierarchical image. Are obtained such that four kinds of submappings f (m, s) (s = 0, 1, 2, 3, 3) that minimize (S2)
1). The bijection condition is checked using the inheritance quadrilateral described in [1.3.3]. At this time, as shown in Expressions 17 and 18, the sub-mappings at the m-th level are constrained by those at the (m-1) -th level, so that matching at a lower resolution level is sequentially used. This is a vertical reference between different levels. Note that m = 0 now and there is no coarser level, but this exceptional processing will be described later with reference to FIG.

On the other hand, horizontal reference within the same level is also performed. As shown in Equation 20 of [1.3.3], f (m,
3) is f (m, 2) and f (m, 2) is f (m, 1)
F (m, 1) is determined to be similar to f (m, 0). The reason is that even if the type of singularity is different,
This is because it is unnatural that the submappings are completely different as long as they are originally included in the same start image and end image. As can be seen from Equation 20, the closer the submappings are, the smaller the energy is, and the matching is considered to be good.

Since there is no submapping that can be referred to at the same level for f (m, 0) to be determined first, a coarser level is referred to as shown in Expression 19. However, in the experiment, after f (m, 3) was obtained, a procedure was performed in which f (m, 0) was updated once using this as a constraint. This is equivalent to substituting s = 4 into Equation 20, and replacing f (m, 4) with a new f (m, 0). f (m, 0)
And f (m, 3) in order to avoid the tendency of the relevance to become too low, and this measure improved the experimental result. In addition to this measure, the experiment also shuffled the submap shown in [1.7.1]. This is also intended to keep the degree of relevance of submappings originally determined for each type of singularity closely. The point at which the position of the start point is changed according to the value of s in order to avoid the deflection depending on the start point of the processing is as described in [1.7].

FIG. 13 is a diagram showing how the submapping is determined at the 0th level. At the 0th level, since each sub-image is composed of only one pixel, four sub-maps f
(0, s) is automatically determined as a unit mapping. FIG.
FIG. 8 is a diagram showing how a sub-mapping is determined at the first level. At the first level, each sub-image is composed of four pixels. In the figure, these four pixels are indicated by solid lines. Now, when searching for a corresponding point of point x of p (1, s) in q (1, s), the following procedure is taken.

1. An upper left point a, an upper right point b, a lower left point c, and a lower right point d of the point x are obtained at the first level of resolution. 2. A pixel to which the points a to d belong at one coarse level, that is, the 0th level is searched. In the case of FIG. 14, points a to d belong to pixels A to D, respectively. However, the pixels A to C are virtual pixels that do not originally exist. 3. The corresponding points A ′ to D ′ of the pixels A to D already determined at the 0th level are plotted in q (1, s). Pixels A ′ to C ′ are virtual pixels, and pixels A to C, respectively.
At the same position as 4. Assuming that the corresponding point a 'of the point a in the pixel A is in the pixel A', the point a 'is plotted. At this time, the position occupied by the point a in the pixel A (in this case, the lower right) and the point a ′
Occupy the same position in pixel A ′. The corresponding points b ′ to d ′ are plotted in the same manner as in 5.4,
An inheritance quadrilateral is created at points a ′ to d ′. 6. A corresponding point x ′ of the point x is searched so that the energy is minimized in the inherited quadrilateral. The candidates for the corresponding point x ′ may be limited to, for example, those whose pixel centers are included in an inherited quadrilateral. In the case of FIG. 14, all four pixels are candidates.

The above is the procedure for determining the corresponding point of the point x. The same processing is performed for all other points to determine a submapping. At the second and higher levels, the shape of the inherited quadrilateral is considered to gradually collapse, so that a situation occurs in which the pixels A ′ to D ′ are spaced apart as shown in FIG.

When the four sub-maps of the m-th level are determined, m is incremented (S2 in FIG. 12).
2) Check that m does not exceed n (S2
3) Return to S21. Hereinafter, each time the process returns to S21, a sub-mapping of a finer resolution level is obtained, and when the process returns to S21, an n-th level mapping f (n) is determined. Since this mapping is determined for η = 0, f
Write (n) (η = 0).

Next, in order to obtain mappings for different η, η is shifted by Δη and m is cleared to zero (S2).
4). It is confirmed that the new η does not exceed the predetermined search cutoff value ηmax (S25), and the process returns to S21 to obtain the mapping f (n) (η = Δη) for the current η. This process is repeated, and in S21, f (n) (η = iΔη) (i =
0, 1, ...). When η exceeds ηmax, the process proceeds to S26, where the optimal η = ηopt is determined by a method described later, and f (n) (η = ηopt) is finally mapped f (n).
And

FIG. 15 is a flowchart showing the details of S21 in FIG. According to this flowchart, the submapping at the m-th level is determined for a certain η. In determining the submapping, in the base technology, the optimum λ is independently determined for each submapping.

As shown in the figure, first, s and λ are cleared to zero (S210). Next, the submap f that minimizes the energy for λ (and implicitly for η) at that time
(M, s) is obtained (S211), and this is f (m, s)
Write (λ = 0). In order to obtain a mapping for a different λ, λ is shifted by Δλ, and it is confirmed that the new λ does not exceed the predetermined search cutoff value λmax (S21).
3), returning to S211 and performing f (m,
s) (λ = iΔλ) (i = 0, 1,...) is obtained. When λ exceeds λmax, the process proceeds to S214, and the optimal λ = λo
pt is determined, and f (m, s) (λ = λopt) is finally set as a mapping f (m, s) (S214).

Next, λ is cleared to zero and s is incremented in order to obtain another submapping at the same level (S215). Confirm that s does not exceed 4 (S216), and return to S211. When s = 4, f (m, 0) is updated using f (m, 3) as described above,
The determination of the sub-mapping at that level ends.

FIG. 16 shows that f (m, s) (λ = iΔλ) (i = 0,
It is a figure which shows the behavior of energy C (m, s) f corresponding to (1, ...). As described in [1.4], as λ increases, C (m, s) f usually decreases. However, when λ exceeds the optimum value, C (m, s) f starts to increase. Therefore, in the base technology, λ when C (m, s) f takes a minimum value is determined as λopt. Even if C (m, s) f becomes smaller again in the range of λ> λopt as shown in the figure, the mapping is already distorted at that point and it is meaningless. Good. λopt is determined independently for each submapping, and finally one for f (n).

FIG. 17 is a diagram showing the behavior of energy C (n) f corresponding to f (n) (η = iΔη) (i = 0, 1,...) Obtained while changing η. . Here, as η increases, C (n) f usually decreases, but when η exceeds the optimum value, C (n) f starts to increase. So C
(N) η when f takes a minimum value is determined as η opt.
FIG. 17 can be considered as an enlarged view of the vicinity of zero on the horizontal axis in FIG. Once ηopt is determined, f (n) can be finally determined.

As described above, according to the base technology, various merits can be obtained. First, since there is no need to detect an edge, the problem of the conventional edge detection type technology can be solved. Also,
No a priori knowledge of the objects included in the image is required, and automatic detection of corresponding points is realized. According to the singular point filter, the luminance and position of the singular point can be maintained even at a coarse resolution level, which is extremely advantageous for object recognition, feature extraction, and image matching. As a result, it is possible to construct an image processing system that significantly reduces manual work.

The following prerequisite technology can be considered as the prerequisite technology. (1) In the base technology, parameters are automatically determined when matching is performed between the start hierarchical image and the end hierarchical image. However, this method performs matching between two normal images, not between hierarchical images. Available for all cases.

For example, between two images, energy E0 relating to the difference in luminance of pixels and energy E1 relating to positional deviation of pixels are used as an evaluation expression, and the linear sum Etot = αE0 + E1 is used as an overall evaluation expression. Attention is paid to the vicinity of the extreme value of this comprehensive evaluation formula, and α is automatically determined. That is, a mapping that minimizes Etot for various α is obtained. Among these mappings, α when E1 takes a minimum value with respect to α is determined as an optimal parameter. The mapping corresponding to the parameter is finally regarded as the optimal matching between the two images.

In addition to the above, there are various methods for setting the evaluation formula. For example, a formula having a larger evaluation value such as 1 / E1 and 1 / E2 may be adopted as the evaluation result becomes better. The overall evaluation formula does not necessarily need to be a linear sum, and n
A sum of squares (n = 2, 、, −1, −2, etc.), a polynomial, an arbitrary function, or the like may be appropriately selected.

The parameter may be any parameter, such as α alone, two cases of η and λ as in the base technology, and more than two cases. If the parameter is 3 or more, it is determined by changing one by one. (2) In the base technology, after a mapping is determined so that the value of the overall evaluation formula is minimized, a point at which C (m, s) f, which is one of the evaluation formulas constituting the overall evaluation formula, becomes minimum is detected. The parameters were determined. However, in place of such a two-step process, depending on the situation, it is effective to simply determine the parameters so that the minimum value of the comprehensive evaluation formula is minimized. In this case, for example, αE0 + βE1 is used as a comprehensive evaluation formula, and α + β =
One constraint condition may be provided to take measures such as treating each evaluation formula equally. This is because the essence of automatic parameter determination is to determine parameters so that energy is minimized. (3) In the base technology, four types of sub-images related to four types of singular points are generated at each resolution level. However, one, two, and three of the four types may be used selectively. For example, if there is only one bright point in the image, a corresponding effect should be obtained even if a hierarchical image is generated only with f (m, 3) relating to the maximum point. In this case, different submappings at the same level are not required, so that there is an effect that the amount of calculation regarding s is reduced. (4) In the base technology, when the level is advanced by one by the singular point filter, the number of pixels is reduced to 1/4. For example, 3 × 3 and 1
A configuration in which a block is used to search for a singular point in the block is also possible. In this case, if the level advances by one, the pixel becomes 1/9. (5) If the start and end images are color, they are first converted to black and white images and the mapping is calculated. The starting color image is converted using the mapping obtained as a result. As another method, a submapping may be calculated for each component of RGB. [4] Results of Experiment Various images can be interpolated using this base technology. By interpolating images from two different viewpoints, an image from an intermediate viewpoint can be generated. This is WWW
Is very advantageous in This is because an arbitrary viewpoint image can be generated from a limited number of images. Morphing can be performed by interpolating the images of the faces of the two people. If a cross-sectional data image of a three-dimensional object is used, such as CT or MRI data, an accurate shape of the three-dimensional object for volume rendering can be reconstructed as a result of the interpolation.

FIGS. 18 (a), 18 (b), 18
(C) shows a case where the mapping is used to generate an intermediate viewpoint image. Here, the right eye image and the left eye image are interpolated. 18 (a) is a starting image viewed from the left eye, FIG. 18 (b) is a starting image viewed from the right eye, and FIG. 18 (c) is t = 0.5 in [1.8] for simplicity. Each of the intermediate images is shown.

FIGS. 19 (a), 19 (b), 19
(C) and FIG. 19 (d) show a case where a human face is morphed using a mapping. Here, the two faces were interpolated. FIG. 19A shows a starting point image, and FIG.
19C shows an end image, FIG. 19C shows an image obtained by superimposing the start image on the end image, and FIG. 19D shows an intermediate image when t = 0.5.

FIGS. 20A and 20B show a case where a mapping is used for interpolation of human and cat faces. FIG.
(A) shows a cat face, and FIG. 20 (b) shows a morphing image of a human face and a cat face. Figure 1 as a human face
9 (a) was used. The luminance normalization described in [1.6] is used only in this example.

FIGS. 21 (a), 21 (b), 21
(C) shows an example in which the present method is applied to an image including many objects. FIG. 21 (a) is the start point image, FIG. 21 (b) is the end point image, and FIG.
An intermediate image at 0.5 is shown.

FIGS. 22 (a), 22 (b) and 22
(C) and FIG. 22 (d) show the results of using a mapping to interpolate a cross-sectional image of the human brain obtained by MRI. FIG. 22A shows a start point image, FIG. 22B shows an end point image (upper section), and FIG. 22C shows an intermediate image when t = 0.5. FIG. 22D shows the result of performing volume rendering using four cross-sectional images viewed from an oblique direction. The object is completely opaque and has a luminance of 51 (= 255) as a result of the interpolation.
× 0.2) are displayed. The reconstructed object is cut vertically near the center, showing the interior.

In these examples, the MRI images were 25
6 × 256 pixels, all other images are 512 × 512
Pixel. The luminance of the pixel takes any value from 0 to 255. The additional conditions described in [1.3.1] are the same as those in FIG.
Except for the cases of (a) to FIG. 21 (c), they are used in all examples. In all these examples, B0thres =
0.003 and B1thres = 0.5 were used and there was no need to change these values. The luminance of the pixel of each sub-image was normalized only in the case of FIGS. 20 (a) and 20 (b).

[Pseudo Three-Dimensional Image Generation Technology] A pseudo three-dimensional image generation technology using the base technology described in detail above will be described. The pseudo three-dimensional image generation technology is an application of the processing technology for generating an intermediate image in the base technology. FIG. 23 is a flowchart illustrating a procedure for generating a pseudo three-dimensional image. The object is photographed from each direction (S
2010). FIG. 24 is an arrangement diagram showing an imaging device in which imaging devices a, b,... The visual field of the video is set to a size such that the viewpoint s when viewed as a pseudo three-dimensional image later is included in any of the captured images.

The photographing may be performed with the imaging device fixed as in the photographing device shown in FIG. 24, or may be performed from an appropriate direction with a person holding the camera, for example. However, it is the same that a viewpoint desired to be viewed as a pseudo three-dimensional image is included in any image. Even if the shooting distance or the magnification changes, there is no problem because it is corrected by matching calculation or interpolation calculation performed later. Alternatively, the imaging device may be held by the robot, and the imaging point may be determined by walking or operating the arm in accordance with the program so that the imaging device can perform the imaging. When a robot is used, a photographing position and a photographing direction required for interpolation can be optimized in advance so that an accurate image can be obtained at any time even if the target object changes.

Next, image data is input from the imaging device (S
2012), encoded image data is generated for each image (S2014). In the coded image data generation step, a two-dimensional search is performed on the image data of the original image by a multi-resolution filter, and a maximum point, a minimum point, a saddle point, and other singular points having unique characteristics of a pixel value in a certain area are obtained. Is detected, and a hierarchical image including the singular point and having a lower resolution than the original image is generated. By repeating such a procedure, a hierarchical image group is generated. The image data of these hierarchical image groups is stored as encoded image data in a storage device, and is used for generating a pseudo three-dimensional image. In addition, the image can be transmitted to a device that performs image display (S201).
6). In this way, encoded image data including a series of hierarchical image data is obtained for each original image.

In order to display a pseudo three-dimensional image of the object, the pseudo three-dimensional image generation device acquires viewpoint information (S
2020). A viewer observing an object using the image display device can specify a different viewpoint for the object being viewed. The pseudo three-dimensional image generation device inputs information on a designated viewpoint position via a pointing device or the like. The encoded image data is searched based on the viewpoint position information to detect an image including the designated viewpoint (S202).
2).

Using the hierarchical image data of the detected images, a singular point shared between the images is detected, and matching between the images is performed (S2024). Image matching is performed for each resolution level of the target image. At this time, as described in the base technology, the energy of the mapping between the images is defined, and the matching can be expressed by a mapping that minimizes this energy. The optimal matching can be automatically determined by using the energy of the mapping as an evaluation function relating to the difference between the position and the luminance, and introducing parameters. Further, interpolation is performed on the mapping based on the viewpoint information to determine the position and luminance of the corresponding pixel in the composite image centered on the specified viewpoint, and generate a new image.

For example, as shown conceptually in FIG.
The position of the new viewpoint V with respect to the viewpoint Va of the image A captured by the camera a and the viewpoint Vb of the image B captured by the camera b is obtained and subjected to interpolation calculation, so that the viewpoint V is viewed from the intermediate position s through which the viewpoint V is seen. An intermediate image S is synthesized. Assuming that the viewpoint is at a position that internally divides the viewpoint between the two images into t: (1-t), the intermediate image is also at a position that internally divides the space between the original images into t: (1-t). Assuming and interpolating. For example, if the new viewpoint V is in the middle of the viewpoints Va and Vb in the acquired image, the image data K of the intermediate image S is K = (Ka + Kb) / K with respect to the image data Ka and Kb of the images A and B. It has a relationship of 2.

Since the composite image generated in this way represents the state of the object viewed from an arbitrary direction other than the photographing direction, the same image as when the face is moved or the object is rotated and observed is obtained. And a pseudo three-dimensional image can be presented (S2026). When the position of the new viewpoint V is out of the straight line connecting the viewpoints Va and Vb of the acquired image, the component in the vertical direction with respect to the straight line must be considered. Also in this case, a synthesized image can be obtained by performing distance calculation in the vertical direction using the original image. However, more preferably, the third image K having a vertically offset viewpoint Vc
c to obtain an image K as represented by K = ΣaiKi
What is necessary is just to get. Where Ki = Ka, K
b, Kc, Σai = 1. In addition, it is preferable that the line-of-sight directions at which each image is taken intersect at one point. However, even if there is some error, it can sufficiently withstand practical use.

The image data of the composite image thus obtained is sent to the image display device, where the composite image is displayed. The image data of the composite image may be formed by image data of a hierarchical image using a hierarchical image of the original image. An apparatus for realizing the above processing is constituted by units performing the respective functions.
It can also be realized by a program loaded into a C (personal computer) from a recording medium such as a CD-ROM.

According to the pseudo three-dimensional image generating apparatus of this embodiment, since the image matching is used, a good composite image can be obtained even if the distortion between the images is large. In fact, if there is an image of the periphery of the object taken at about every 30 degrees, depending on the image, it is possible to present an image that looks natural as a stereoscopic image from any direction. It has been confirmed that an image that can hardly be distinguished from a captured image is generated.

Since the operation speed of this procedure is extremely high, even when using a computer of the personal computer level,
Real-time processing that does not cause a delay when the user changes the viewpoint by operating the mouse is realized. It is also possible to present a moving image or a movie having a different viewpoint from the created or photographed original image. By using the technology of the present embodiment, for example, even in the case of sales using the Internet, it is possible to display a stereoscopic image of a product by transmitting a small amount of encoded image data, thereby evoking consumers' willingness to purchase and improving efficiency. Business can be performed.

As shown in FIG. 26, if the image pickup device F is arranged at appropriate intervals on the spherical surface G covering the object O and an image is taken from the entire periphery of the object, the viewpoint of the viewer can be improved. Z
It is possible to cope with the case of moving to any position in the direction, and a more complete pseudo three-dimensional image providing device can be provided. At this time, the imaging device may not be fixedly arranged, and may be photographed while being moved to an appropriate position by the hand of a person or a robot. In addition, by processing images taken at appropriate intervals while rotating and moving the imaging apparatus outward at a predetermined position, it is possible to synthesize and present an external view image viewed in an arbitrary direction. The image can be displayed as a panoramic photograph in which the image moves according to the moving viewpoint.

FIG. 27 is a diagram conceptually illustrating a case where a composite photograph s having different eyes is generated from two landscape photographs a and b to make a panoramic photograph. FIG.
In (a), images A and B having different lines of sight are acquired.
In the image A and the image B, a part of the scene is overlapped and photographed. It is assumed that the viewer requests to display the image S with a viewpoint different from the two images. The pseudo three-dimensional image generation device performs the image A and the image B as shown in FIG.
A singular point is detected for each of the image data, and the two are matched based on this. Further, an interpolation calculation is performed based on the viewpoint information on the image S to be synthesized, and FIG.
A new image S as shown in FIG.

According to the apparatus of this embodiment, since these pseudo three-dimensional images are presented by performing image processing at a high speed using extremely small encoded image data, they are utilized for WWW or the like for displaying images via a communication path. be able to. Note that, in the above-described embodiment, an example in which an intermediate position image closer to the actual position is synthesized from a smaller number of original images using a matching procedure using a multi-resolution filter has been described. Needless to say, an effect can be obtained.

[0156]

As described above, the method and apparatus for generating a pseudo three-dimensional image according to the present invention convert a plurality of images into coded image data, and automatically perform matching calculation and interpolation calculation, thereby obtaining a small amount of information. It is possible to combine and present an image with an arbitrary viewpoint, and to display an online image of a three-dimensional object in an electric commerce (EC) or a moving image, or to produce a movie composed of an image viewed from a different direction from shooting. It becomes possible.

[Brief description of the drawings]

FIG. 1A is an image obtained by applying an averaging filter to the faces of two persons, and FIG.
1 (d) is an image of p (5,0) obtained by the base technology with respect to the faces of two persons, FIG. 1 (e) and FIG.
(F) is an image of p (5,1) obtained by the base technology with respect to the faces of two persons, and FIGS. 1 (g) and 1 (h)
P required by the underlying technology for the faces of two people
The images of (5, 2) and FIGS. 1 (i) and 1 (j) show p (5,3) obtained by the premise technique for the faces of two persons.
2 is a photograph of a halftone image in which each image is displayed on a display.

FIG. 2 (R) shows an original quadrilateral, FIG.
(A), FIG. 2 (B), FIG. 2 (C), FIG. 2 (D), FIG.
(E) is a figure which shows an inheritance quadrangle.

FIG. 3 is a diagram illustrating a relationship between a start point image and an end point image and a relationship between an m-th level and an (m-1) th level using an inherited quadrilateral.

FIG. 4 is a diagram showing a relationship between a parameter η and an energy Cf.

5 (a) and 5 (b) are diagrams showing how to determine from a cross product calculation whether or not a mapping for a certain point satisfies the bijection condition.

FIG. 6 is a flowchart showing an overall procedure of the base technology.

FIG. 7 is a flowchart showing details of S1 of FIG. 6;

FIG. 8 is a flowchart showing details of S10 in FIG. 7;

FIG. 9 is a diagram illustrating a correspondence relationship between a part of an m-th level image and a part of an (m−1) th level image;

FIG. 10 is a diagram showing a starting hierarchical image generated by the base technology.

FIG. 11 is a diagram showing a procedure for preparing a matching evaluation before proceeding to S2 of FIG. 6;

FIG. 12 is a flowchart illustrating details of S2 in FIG. 6;

FIG. 13 is a diagram showing how a sub-mapping is determined at the 0th level.

FIG. 14 is a diagram showing how a sub-mapping is determined at the first level.

FIG. 15 is a flowchart showing details of S21 in FIG. 12;

FIG. 16 is a diagram showing a behavior of energy C (m, s) f corresponding to f (m, s) (λ = iΔλ) obtained while changing λ for a certain f (m, s).

FIG. 17 shows f (n) obtained while changing η (η = i
Δη) energy C corresponding to (i = 0, 1,...)
It is a figure which shows the behavior of (n) f.

18 (a), 18 (b), 18 (c)
Are photographs of a left-eye image, a right-eye image, and a halftone image of an interpolated image generated by the base technology on a display, respectively.

19 (a), 19 (b), 19
(C) and FIG. 19 (d) are photographs of a halftone image in which a face of a certain person, a face of another person, a superimposed image thereof, and a morphing image generated by the base technology are displayed on a display. .

FIGS. 20 (a) and 20 (b) are photographs of a halftone image in which a morphing image of a face of a cat, a face of a person and a face of a cat are displayed on a display, respectively.

FIG. 21 (a), FIG. 21 (b), FIG. 21 (c)
Are left-eye images containing many objects,
It is a photograph of a right-eye image and a halftone image in which an interpolation image generated by the base technology is displayed on a display.

FIG. 22 (a), FIG. 22 (b), FIG.
(C) and FIG. 22 (d) respectively show a halftone image in which a start point image, an end point image, an interpolated image generated by the base technology, and a volume rendering image generated based on the interpolated image are displayed on a display. It is a photograph of an image.

FIG. 23 is a flowchart showing a procedure for generating a pseudo three-dimensional image according to the embodiment.

FIG. 24 is a layout diagram showing a photographing apparatus according to an embodiment.

FIG. 25 is a diagram conceptually illustrating an intermediate image combining process according to the embodiment.

FIG. 26 is a layout diagram showing another photographing apparatus according to the embodiment.

FIG. 27 is a conceptual diagram illustrating a process of combining panoramic photos according to the embodiment.

[Explanation of symbols]

 O Target a, b Camera s New line of sight A, B Original image S Interpolated image Va, Vb, Vs Viewpoint F Camera G Camera arrangement coordinates

────────────────────────────────────────────────── ───

[Procedure amendment]

[Submission date] July 19, 2001 (2001.7.1)
9)

[Procedure amendment 1]

[Document name to be amended] Drawing

[Correction target item name] All figures

[Correction method] Change

[Correction contents]

FIG. 6

FIG. 7

FIG. 10

FIG. 11

FIG. 13

FIG.

FIG. 5

FIG.

FIG.

FIG. 2

FIG. 3

FIG. 8

FIG. 9

FIG. 14

FIG. 25

FIG. 4

FIG.

FIG. 24

FIG. 26

FIG.

FIG.

FIG.

FIG.

FIG. 23

FIG. 27

FIG. 21

FIG.

Continued on front page F term (reference) 5B057 CA01 CA08 CA12 CA16 CB01 CB08 CB12 CB16 CE06 CE08 CE10 CH09 5C023 AA03 AA09 AA10 AA32 AA37 AA38 BA02 BA13 CA01 DA04 DA08 5C061 AA21 AB04 AB08 AB12 5C076 AA19 BA09

Claims (6)

[Claims] 1. A step of calculating matching between a plurality of original images which are photographed from two or more directions and include a viewpoint set for an object in the image, and interpolation based on the result of the matching. A method for generating a pseudo three-dimensional image, comprising: generating a composite image viewed from a direction including the viewpoint; and presenting data of the composite image. 2. A step of acquiring a plurality of images photographed from a plurality of positions around a subject, and a step of selecting, from the acquired images, two or more original images including a set viewpoint in the images. Calculating a matching between the two or more original images; generating a composite image viewed from a direction including the set viewpoint based on the result of the matching; A pseudo three-dimensional image generation method including a step of presenting. 3. The matching calculating step includes the steps of: applying a multi-resolution singularity filter to each original image to generate a series of hierarchical images having different resolutions; Calculating the matching between the two, wherein the synthetic image generating step includes the step of: using the positional relationship between the set viewpoint and the viewpoint of the original image, and the correspondence between the position and the luminance of the matched pixels in the original image. 3. The method according to claim 1, further comprising the step of performing an interpolation calculation on a pixel-by-pixel basis to generate an intermediate image.
The pseudo three-dimensional image generation method described in the above. 4. A unit for photographing from a plurality of positions around a subject and acquiring image data of a plurality of images,
A unit for selecting two or more original images that are photographed from two or more directions and include a viewpoint set for the object in the image, and a multi-resolution singularity filter is applied to each of the image data of the original images. A unit for generating image data of a series of hierarchical images having different resolutions, a unit for calculating matching between original images based on the image data of the hierarchical images having different resolution levels, A unit for generating image data of an intermediate image by performing an interpolation calculation on a pixel basis based on the correspondence relationship between the position and luminance of the matched pixels in the original image using the positional relationship of the viewpoint of the image, A pseudo three-dimensional image generation apparatus, which generates and presents a composite image viewed from a direction including a set viewpoint. 5. The robot according to claim 1, wherein the unit for acquiring the plurality of images is a robot equipped with an imaging device, and the robot is operated by a built-in program or remote control to acquire image data of images by photographing from a plurality of positions. The pseudo three-dimensional image generation device according to claim 4, wherein 6. A recording medium storing a computer-executable program, the program comprising: a step of selecting two or more original images including a viewpoint for an object in the image; Applying a multi-resolution singularity filter to each image data to generate image data of a series of hierarchical images having different resolutions; and performing matching between original images based on the image data of the hierarchical images having different resolution levels. Calculating the image data of the intermediate image by performing an interpolation calculation on a pixel-by-pixel basis based on the correspondence between the positions of the matched pixels and the luminance using the positional relationship between the set viewpoint and the viewpoint of the original image. A computer-readable recording medium, comprising: