JP3387900B2 - Image processing method and apparatus - Google Patents

Description

Detailed Description of the Invention

[0001]

BACKGROUND OF THE INVENTION 1. Field of the Invention The present invention relates to an image processing apparatus and method for interpolating and generating an image at a user's viewpoint position from an existing image.

[0002]

2. Description of the Related Art A omnidirectional image in which lens centers coincide with each other is effective as a data format in a storage / replay telepresence mixed reality system, and various attempts have been made.

The example is as follows. For example, (1) "Construction of a cyber visual city space using a camera mounted on a moving vehicle (2) -Generation of a wide-area virtual space using a real image-
(Proceedings of the 2nd Annual Meeting of the Virtual Reality Society of Japan, pp
67-70,1997 (Hirose, Watanabe, Tanigawa, Endo, Katayama, Tamura)
Proposes an imaging system in which a plurality of video cameras are radially arranged on the roof of an automobile in the horizontal direction. (2) In the iMove Spherical Video introduced at http://www.imoveinc.com, it is shown that the entire sky circumference is shot at the same time using multiple cameras. (3) QuickTime VR (An Image-Based by Chen SE
Approach to Virtual Environment Navigation, Proc.
SIGGRAPH'95, pp.29-38, 1995) makes it possible to freely look around all directions from a certain point. (4) "Creation of high-resolution stereo panoramic video image by omnidirectional stereo image sensor using hexagonal pyramid mirror (Public technique PRMU97-118, 1997 (Kawanishi, Yamazawa, Iwasa, Takemura, Yokoya))" and "Ray Ray space projection method for efficient recording of three-dimensional real space by information (Bidding Technique IE95-119, 1996
(Naemura, Yanagisawa, Kaneko, Harashima)) ", if you use one camera and a rotating hyperboloidal mirror, or if you use multiple cameras and a polygonal pyramid mirror, you can get a full-circle image with the same lens center. It is shown that the images can be taken at the same time.

[0004]

However, since the camera and the photographing equipment have a physical size, it is impossible to photograph the whole sky image from a certain point at the same time.

For example, in the above (1), the lens centers of the cameras cannot be made to coincide with each other, and the camera does not correspond to the entire circumference. Also, in the above (2), the lens centers of a plurality of cameras cannot be matched. Further, in the above (3), the omnidirectional image from a certain point is composed from images sequentially shot by rotating one camera, and images at the same time cannot be generated. Furthermore, it is physically impossible to extend the method described in (4) above to support imaging of the entire sky.

The present invention has been made in view of the above problem, and based on a plurality of images obtained from a physically arrangable number of cameras arranged in the direction of the whole sky, The purpose is to be able to interpolate images
To do . In particular, the multiple images taken by the cameras arranged discretely
Image of desired viewpoint position and line-of-sight direction is obtained from several images
The purpose is to be able to.

[0007]

An image processing apparatus according to the present invention for achieving the above object has, for example, the following configuration. That is, the images are taken by the cameras arranged discretely.
An image processing apparatus for generating an interpolated image of a desired viewpoint position and viewing direction based on a plurality of images, an arbitrary viewpoint
Of the input means for inputting position and line-of-sight direction information and the plane forming a polyhedron having the lens center of the camera that has photographed each of the plurality of images as the vertex, the input means is used.
Based on a plurality of images having a lens center at the apex of the plane extracted by the extracting means, and an extracting means for extracting a plane having an intersection between a viewpoint position and a ray connecting the target pixel in the interpolation image, Interpolation means for interpolating and generating a virtual image having the intersection point as a viewpoint position; drawing means for acquiring a pixel corresponding to the pixel of interest from the virtual image and drawing the pixel of interest using the pixel; and the interpolation image for each of the in pixels, said by extracting means and said interpolation means and said drawing means <br/> repeating Succoth, viewpoint position input by the input means
And a generation means for generating the interpolation image according to the position and the line-of-sight direction information .

Further, the image processing method according to the present invention for achieving the above object includes, for example, the following steps. That is, it is an image processing method for generating an interpolated image of a desired viewpoint position and a line-of-sight direction based on a plurality of images captured by discretely arranged cameras , and an arbitrary viewpoint position and
Among the input step of inputting the line direction information and the planes forming the polyhedron whose vertex is the lens center of the camera that has captured each of the plurality of images, the viewpoint position input in the input step
Position and an extraction step of extracting a plane having an intersection of a ray connecting the pixel of interest in the interpolated image, based on a plurality of images with the vertex of the plane extracted in the extraction step as the lens center, An interpolation step of interpolating and generating a virtual image as a viewpoint position, a drawing step of acquiring a pixel corresponding to the target pixel from the virtual image, and drawing the target pixel using the pixel, and a pixel in the interpolation image for each, score Repeat the extraction step and the interpolation step and the drawing step
And the viewpoint position and line of sight input by the input means
And a generation step of generating the interpolation image according to the direction information .

[0009]

Preferred embodiments of the present invention will be described below with reference to the accompanying drawings.

[Outline and Principle of System] Even if it is not possible to capture an all-round image in which the lens centers of the cameras are coincident with each other, it is sufficient that the images can be reconstructed by post-processing. Therefore, in the present embodiment, it is possible to interpolate and generate a vast amount of images based on the images captured by a small number of cameras, and to interpolate and generate all-sky images.

Each pixel forming an image is a recording of one ray of light that has come to the center of the lens of the camera. Therefore, it can be considered that the image is a record of light rays that fly to the center of the lens of the camera. here,
Assuming that each ray goes straight without being attenuated during the flight, the color of the ray can be observed at any position on the flight path of the ray. Therefore, assuming that the image to be generated is I _e and the viewpoint position thereof is e, the same number of cameras as the number of pixels of the image I _e are prepared, and the cameras are arranged at positions apart from the viewpoint position e, and each camera has By recording an appropriate ray for each book, I _e can be reconstructed as shown in FIG. 1 without physically placing a camera at the viewpoint position e.

Although FIG. 1 shows the case of a plane image, the whole sky image can be reconstructed in the same manner.

However, in order to realize such an idea, a very large number of cameras corresponding to each pixel are required, which is not realistic. However, if it is possible to interpolate and generate an image that should be captured by a camera corresponding to each pixel from images captured by more discretely arranged cameras, as shown in FIG. The same result can be obtained as when the camera is arranged.

That is, a virtual image at an arbitrary viewpoint position on the surface of a polyhedron whose vertex is the lens center of each camera is generated from images captured by discretely arranged cameras, and an appropriate pixel is selected from those images. For example, it is possible to generate an image (interpolation image) at an arbitrary viewpoint position inside the polyhedron.

FIG. 3 is a view for explaining the principle of generating an interpolated image from an arbitrary viewpoint according to this embodiment. As described above, if an image at an arbitrary viewpoint position on the surface of the polyhedron S with the center of each camera lens as the vertex can be generated,
By selecting the appropriate pixels from those images,
The interpolated image I _e at the viewpoint position e inside the polyhedron S can be generated.

Specifically, for all the pixels P _e forming the interpolated image I _e , the intersection X _F between the ray representing the pixel P _e and the polygonal plane F forming the polyhedron S is obtained, and X _F the virtual image I _F to the viewpoint position generated by the interpolation from the image to the lens center to the vertices of the polygon plane F, determine the pixel P _F on the virtual image I _F corresponding to the pixel P _e, this Interpolated image I _e
The pixel P _e is drawn.

The generation of an interpolated image from an arbitrary viewpoint according to this embodiment will be described below.

[Description of System Configuration] FIG. 4 is a block diagram showing a schematic configuration of the camera system of the present embodiment. In FIG. 4, reference numerals 41-1 to 41-n denote a plurality (n) of cameras (hereinafter, referred to as n cameras 41) arranged substantially in the entire circumferential direction. In this example, CCD cameras are used as the n cameras 41. 42
Is a camera controller, which controls the simultaneous shooting by the n cameras 41 and provides image information obtained by these shootings to the image storage processing unit.
An image storage processing unit 43 includes a camera controller 42.
The respective images of the n cameras 41 obtained by the above are stored in the storage device 44.

Reference numeral 45 is a viewpoint position and line-of-sight direction input unit, which gives a desired viewpoint position and line-of-sight direction to the apparatus. 46
Is an interpolation processing unit, and is a viewpoint position and line-of-sight direction input unit 45.
An interpolated image to be observed according to the viewpoint position and the line-of-sight direction set in step 1 is generated based on the image information stored in the storage device 44. A display device 47 displays the interpolated image generated by the image generation unit 46 on a display such as a CRT or a liquid crystal display. The image storage processing unit 43 and the interpolation processing unit 46 may be configured by one computer.

In order to generate a virtual image, it is necessary to find correspondences between feature points between input images. For that purpose, all objects in the omnidirectional direction are observed with a plurality of input images. Must have been In this embodiment, since interpolation is performed using three input images as described later, it is necessary to observe at least three input images. FIG. 5 is a diagram for explaining the relationship between the camera arrangement and the angle of view according to this embodiment. In FIG. 5, a two-dimensional camera arrangement is described for convenience of explanation, but this may be expanded to three dimensions.

It is now assumed that the vertical angle of view of the camera is α, and that the camera is arranged on a spherical surface having a radius R so that the angular interval between the camera and the closest camera is θ. When the distance to the closest object is D from the center of the sphere, the conditions for observing all the objects in the omnidirectional direction in at least two input images are as follows.

[0022]

[Equation 7]

According to the above equation (1), for example, θ = 30
In the case of R = 1 m and D = 3 m, α ≧ 86.4 degrees. Here, if the number of cameras is small, the photographing system can be downsized and the amount of data can be reduced. However, if the number of cameras is reduced, the angle of view of each camera must be widened, and the resolution of the obtained image will be reduced. Also, various aberrations become a problem. Further, since the cameras are separated from each other, it becomes difficult to associate the feature points. Therefore, when constructing the omnidirectional photographing system, the arrangement intervals of the cameras are set in consideration of these conditions.

[Explanation of All-Surface Image Reconstruction Processing] All-Surround Image Reconstruction Process by the All-Surface Imaging System Having the Above-mentioned Structure will be Described. FIG. 6 is a flow chart showing an outline of processing of the omnidirectional photographing system according to the present embodiment. In step S601, the image storage processing unit 43
Stores n images captured by the n cameras 41 in the storage device 44 (hereinafter, this process is referred to as preprocessing). Further, various information for interpolating and generating the virtual image I _F that can be observed from any position on each plane of the polyhedron with the lens center of the camera as the vertex is stored in association with each image. The information for generating the interpolation will be apparent from the description below.

When the viewpoint position / line-of-sight direction input unit 45 gives the viewpoint position and line-of-sight direction of the observer, the interpolation processing unit 46 uses the image and the information for interpolation generation stored in the storage device 44. , An interpolated image corresponding to this is generated and displayed on the display device 47 (step S60).
2). When the above-described generation of the interpolated image ends, this process ends (step S603).

FIG. 7 is a flow chart for explaining a specific processing procedure of the preprocessing (step S601) according to this embodiment. In step S <b> 701, the n cameras 41 arranged in the omnidirectional direction capture the entire omnidirectional image. At this time, the positions and orientations of the n cameras 41 are also measured. Then, in step S702,
The image obtained from each camera and the shooting information indicating the position and orientation of each camera are stored in the storage device 44 in association with each other, and an image database is created. Here, the information indicating the position of the camera directly or indirectly represents the coordinates of the lens center (vertex of the polyhedron S). The information indicating the posture of the camera directly or indirectly represents the optical axis direction of each camera and the rotation angle around the optical axis.

Subsequently, in step S703, each image in the image database generated in step S702 is divided into an image set including three images as one set. As will be described later, in the present embodiment, each plane forming the polyhedron S is a triangular plane as shown in FIG. 3, and three images having the three vertices forming the selected triangular plane as lens centers are used to form an image I. _{Produces F.} Therefore, each image set is
It is composed of three images with each vertex of the triangular plane forming the polyhedron S as the lens center. Then, the image set is registered in the image database in correspondence with all the triangular planes forming the polyhedron S.

Next, in steps S704 to S708,
For each image set, the correspondence of the same area between the three images is registered in the image database. In addition, in order to eliminate the influence of hiding by an object, in the present embodiment, the image is divided into layers for each object to generate a virtual image. Therefore, the image database contains 3 for each image set.
Each of the images is registered and held in a plurality of layers, and information indicating a corresponding area between the images is held for each layer.

The above processing will be described step by step with the steps shown in FIG. 7. First, in step S704, a variable i indicating a number (image set number) for specifying an image set is set to 1. Then, step S705
Then, the three images included in the image set i are divided into layers for each object.

FIG. 8 is a flow chart for explaining the details of the division processing into layer images, which is executed in step S705. First, in step S801, an object to be divided into layers is determined from the three images included in the image set, and how many layer images are to be divided based on the number of the objects, that is, the total number m of layers is determined. Decide Then, in step S802, the variable j indicating the layer number is set to 1.

Next, in step S803, the object (j) of the jth layer to be drawn on the layer (j
Areas other than the object belonging to the second layer) are painted black. In step S804, the user retouches the portion of the object belonging to the j-th layer, which is hidden by the objects belonging to other layers.

When the editing of the jth layer image is completed as described above, 1 is added to the variable j in order to process the next layer image (step S805). Here, when the variable j becomes larger than the number of all layers set in step S801, it means that the processing of all layers has been completed for the image set, and thus this processing ends (step S8).
06). On the other hand, if the variable j is equal to or less than the total number of layers, the process is performed to process the layer indicated by the variable j in step S.
Return to 803.

The above black filling in step S803 and the retouching in step S804 can be executed by, for example, a commercially available general-purpose image authoring tool software.

The layer is designated by, for example, designating a characteristic point of the image (usually the vertex of the contour line). In this case, the division into the layer images results in assigning the layer value i to the point sequence P _i (k) of the feature points of the outline of the image of the object. Here, i represents a layer value (the smaller the value, the farther the layer is from the viewpoint), and k represents the object included in the same layer i.

For example, when the user divides the image RI of FIG. 12 into layer images, the image is divided into three layer images in order from the layer farthest from the viewpoint position. Layer I (i = I): Whole background image layer II (I = II): Point sequence PI (1) = 10 → 11 → 12 → 13 → 14 Layer III (i = III): Point sequence PI (1) = 1 → 2 → 3 → 4 → 5 → 6 To be Although the vertex 13 of the triangular prism cannot be seen by the user, the user knows that the vertex 13 of the triangular prism is hidden by the square prism, and therefore gives the virtual point 13 to retouch.

As a result of the above operation, the images of FIG. 12 are obtained as shown in FIG. 13 (layer I), FIG. 14 (layer II), and FIG. 15 (layer III), which are registered in the image database. It

After the division into the layer images is completed as described above, in step S706 of FIG. 7, the corresponding regions of the three images are formed by the triangular mesh for each layer of the same layer of the image set i. Correspondence (this point will be described in the interpolation processing of steps S903 to S906). Then, in order to process the next image set, the variable i indicating the image set number is incremented by 1 (step S707). Here, if the variable i becomes larger than the number of all registered image sets, it means that the processing has been completed for all the image sets, so this processing is ended (step S708). On the other hand, if the variable i is equal to or less than the total number of image sets, the process returns to step S705 to process the image set indicated by the variable i.

When the pretreatment is completed as described above,
An image set for each triangular plane forming the polyhedron S will be registered. Then, when an arbitrary viewpoint position and line-of-sight direction in the polyhedron S are given, an image observed from the viewpoint position and line-of-sight direction is generated by interpolation. Less than,
The generation of the interpolation image shown in step S602 will be described. FIG. 9 is a flowchart illustrating the generation of the interpolated image in step S602.

First, in step S901, the viewpoint position and line-of-sight direction of the user are given. Now, as shown in FIG. 3, it is assumed that the viewpoint position is given to the position e in the polyhedron S and the line-of-sight direction is given to generate the interpolated image I _e .
Hereinafter, each pixel of the interpolated image I _e to be generated is represented by a pixel i (i =
1 to the number of pixels of the interpolated image I _e ), and its value is P _e .

In step S902, 1 is assigned to the pixel number i, and the processes in steps S903 to S907 are executed to draw the pixel i. This is performed for all the pixels of the interpolation image I _e (steps S908 and S90).
9) Obtain the interpolated image I _e .

In steps S903 to S906, the pixel i
Is calculated, a virtual plane I _F parallel to the triangular plane is obtained, and a pixel corresponding to a pixel of the interpolated image I _e is obtained as a virtual image I _F.
Get from and draw. In the present embodiment, as a virtual image generation method, "Cybey city Walker -Layered Morphing Metho
d-, Proc. HCI'99, Vol.2, pp.1044-1048, 1999 (T.En
do, A.Katayama, H.Tamura, M.Hirose, T.Tanikawa) ''
Use the method proposed in.

According to this method, it is possible to generate a virtual image without distortion at an arbitrary viewpoint position in a triangle having the lens centers of the three cameras as vertices based on the three input images. . Specifically, the corresponding areas of the three images are previously associated with each other by a triangular mesh (step S706), the coordinates of the vertices of these triangular meshes are interpolated to be arranged, and the interpolated triangular meshes are arranged. A virtual image is generated by texture mapping the image of the corresponding region in the input image. Here, it is possible to prevent the generated image from being distorted by performing linear interpolation after normalizing the optical axes of the three input images to be parallel.

Each face of the polyhedron having the apex of the lens center of each camera in the direction of the whole sky is formed by a triangle having the apex of the lens centers of the three cameras. Therefore, by applying the above-described method to each surface, it is possible to generate an image at an arbitrary viewpoint position on the surface of the polyhedron without distortion.

It is assumed that the external parameters of each camera are accurately obtained in advance, and that the corresponding areas of the three input images are associated with each other by the triangular mesh on each triangular plane forming the polyhedron S. . Furthermore, it is assumed that the focal lengths of the cameras are the same.

In step S903 of FIG. 9, the pixel i
It is determined which triangle plane of the polyhedron S having the vertex of the lens center of each camera intersects, and which triangle plane of the polyhedron S is used to generate the virtual image. That is, in FIG. 3, from the relationship between the direction of the light ray 1 representing the pixel P _e , the position of each camera, and the optical axis direction,
Determine a triangular plane F having an intersection with the ray l. Then, three images (IA, IB, IC respectively) having the three vertices (X ₁ , X ₂ , X ₃ ) of the triangular plane as the lens centers are provided.
And decide).

Next, the coordinates (X _F , Y _F , Z _F ) of the intersection of the triangular plane F and the ray 1 are obtained, and the interpolation coefficient w _n (n = 1, n = 1,
2, 3) is calculated. The interpolation coefficient w _n is _calculated by the following equation (2). Here, (X _m , Y _m , Z _m )
(M = 1, 2, 3) are the coordinates of the centers of the three lenses forming the plane F. Also, Σw _n becomes 1.

[0047]

[Equation 8]

Next, steps S905 and S9.
At 06, a virtual image I _F is generated. That is, the lens center is (X _F , Y _F , Z _F ) and the optical axis direction is the plane F.
A virtual image I _F that is the normal direction of is generated.

This processing will be described with reference to FIG. First, in step S905, each image IA, I
For B and IC, the images IA, IB, and IC are rotated (normalized) so that their optical axes coincide with the normal direction of the triangular plane, and IA ′, IB ′, and IC ′ are obtained (FIG. 10A).
→ (b)). Then, in step S906, the interpolation coefficients w _n and IA ′, IB ′ obtained in step S904,
The virtual image I _F is obtained by linear interpolation using IC ′ (FIG. 10B). This processing can be performed using the following equation (3).

[0050]

[Equation 9]

However, (x, y) is the coordinates of the feature point determined in the virtual image I _F , and (x ′ _n , y ′ _n ) (n =
1, 2, 3) are the coordinates of the feature points in the images IA ′, IB ′, IC ′ obtained by normalizing the images IA, IB, IC, respectively. The conversion from the coordinates in the original image (IA, IB, IC) to the coordinates in the image after the rotation conversion is

[0052]

[Equation 10] Represented by Here, P and P ′ are coordinates of corresponding points in the original image and the image after rotation conversion (homogeneous coordinates having the center of the image as an origin), A is a camera internal parameter matrix of the original image, and A ′. Is a camera internal parameter matrix of the image after the rotation conversion, R is a rotation matrix from the world coordinates to the camera coordinates of the original image, and R ′ is a rotation matrix from the world coordinates to the camera coordinates of the image after the rotation conversion. By this calculation, the coordinates of each feature point in the images IA, IB, and IC are
It can be converted into the coordinates of the feature points in the rotated images IA ′, IB ′, and IC ′.

As described above, the images IA, IB, IC
Each feature point in the inside is arranged on the virtual image I _F , and the texture of the corresponding triangular mesh of the images IA, IB, and IC is blended and mapped to the obtained triangular mesh.

When the virtual image I _F is obtained as described above, in step S907, which pixel in the virtual image I _F should be selected for the pixel P _e is obtained by the following equation (5), and The value of the pixel P _F is drawn as the value of the pixel P _e on the interpolated image I _e .

[0055]

[Equation 11] However, P _F and P _e are the coordinates of the corresponding points in I _F and I _e (homogeneous coordinates with the center of the image as the origin), respectively.
A _F is a camera internal parameter matrix of the virtual image, A _e is a camera internal parameter matrix of the interpolated image, R _F is a rotation matrix from the world coordinates to the camera coordinates of the virtual image,
R _e is a rotation matrix from world coordinates to camera coordinates of the interpolated image.

When the above steps S903 to S907 are performed for all the pixels forming the interpolated image I _e (steps S908 and S909), the interpolated image I _e is completed.

As described above, each image in each image set is divided into layers and held.
The virtual image I _F in 06 is generated based on the virtual images for the number of layers. For example, the final virtual image I _F can be obtained by drawing and synthesizing the virtual images corresponding to the number of layers in order from the viewpoint position.

As described above, according to the present embodiment, it is possible to obtain an interpolated image in an arbitrary line-of-sight direction from images obtained by a limited number of cameras. That is,
The whole sky image can be obtained.

The generation of the virtual image by the linear interpolation in the above processing has dealt with the case where the viewpoint position is within the triangular plane. However, even if the viewpoint position is outside the triangular plane, it is possible to perform extrapolation to perform linear interpolation from an image with the three vertices of the triangular plane as the lens centers.

FIG. 11 is a diagram showing this state. As shown in FIG. 11B, the linear interpolation processing using the complementary number w _n (n = 1, 2, 3) described above is performed on the image at the position 1 and at the position 2
Image is linearly interpolated at 1-s: s to form a first virtual image at position 4, and the virtual image at position 4 and the image at position 3 are t:
This is equivalent to linearly interpolating at 1-t to obtain the image at position e '.

However, even when the viewpoint position is outside the triangular plane like the position e ″, the virtual image at the position 4 and the image at the position 3 may be linearly interpolated at t ′: 1−t ′ ( In this case, t'is negative). In the above-described embodiment, since the intersection of the corresponding ray and the triangular plane is obtained in order to obtain the pixel in the interpolated image, the viewpoint position cannot go out of the triangular plane. However,
In order to obtain pixels in the interpolated image, a method different from the “method for obtaining the intersection of the ray and the plane” is used (for example, a case where a triangular plane is determined in consideration of the camera orientation as well) In some cases, the viewpoint may go out of the plane.

Another object of the present invention is to supply a storage medium storing a program code of software for realizing the functions of the above-described embodiments to a system or apparatus, and to supply a computer (or CPU) of the system or apparatus.
It is needless to say that it can be achieved by reading and executing the program code stored in the storage medium.

In this case, the program code itself read from the storage medium realizes the functions of the above-described embodiments, and the storage medium storing the program code constitutes the present invention.

As a storage medium for supplying the program code, for example, a floppy disk, a hard disk, an optical disk, a magneto-optical disk, a CD-ROM, a CD
-R, magnetic tape, non-volatile memory card, ROM, etc. can be used.

Further, not only the functions of the above-described embodiment are realized by executing the program code read by the computer, but also the OS (operating system) running on the computer based on the instructions of the program code. It is needless to say that this also includes a case where the above) performs a part or all of the actual processing and the processing realizes the functions of the above-described embodiments.

Further, after the program code read from the storage medium is written in the memory provided in the function expansion board inserted into the computer or the function expansion unit connected to the computer, based on the instruction of the program code, It goes without saying that a case where the CPU or the like included in the function expansion board or the function expansion unit performs some or all of the actual processing and the processing realizes the functions of the above-described embodiments is also included.

[0067]

As described above, according to the present invention,
It becomes possible to interpolate and generate the omnidirectional image based on a plurality of images obtained from the physically arrangable number of cameras arranged in the omnidirectional direction. In particular, discretely placed
Desired viewpoint position from multiple images taken by the selected camera
It is possible to obtain an image of the position and the direction of the line of sight.

[Brief description of drawings]

FIG. 1 is a diagram illustrating a method of generating an arbitrary viewpoint image from a super multi-viewpoint image.

FIG. 2 is a diagram illustrating a method of generating an appointment viewpoint image based on interpolation of a multi-viewpoint image.

FIG. 3 is a diagram illustrating a principle of generating an interpolated image from an arbitrary viewpoint according to the present embodiment.

FIG. 4 is a block diagram showing a schematic configuration of a camera system of the present embodiment.

FIG. 5 is a diagram illustrating a relationship between a camera arrangement and an angle of view according to the present embodiment.

FIG. 6 is a flowchart showing an outline of processing of the omnidirectional imaging system according to the present embodiment.

FIG. 7: Pre-processing according to the present embodiment (step S601)
6 is a flowchart illustrating a specific processing procedure of.

FIG. 8 is a flowchart illustrating details of a process of dividing into layer images, which is executed in step S705.

FIG. 9 is a flowchart illustrating generation of an interpolation image in step S602.

FIG. 10 is a diagram illustrating a procedure for generating an interpolated image from three images.

FIG. 11 is a diagram illustrating a case where extrapolation is permitted.

FIG. 12 is a diagram showing an example of an image which is a target of division into layers.

13 is a diagram showing an example of an image obtained by layer division from the image of FIG.

FIG. 14 is a diagram showing an example of an image obtained by layer division from the image of FIG.

FIG. 15 is a diagram showing an example of an image obtained by layer division from the image of FIG.

Continuation of the front page (56) References JP-A-9-190550 (JP, A) Tkaaki Endo, Akihiro Katayama, Hideyuki ki Tamura, Michitaka moiraker, poker-yaker ed. HCI'99, 1999, Vol. 2, pp. 1044-1048 (58) Fields investigated (Int.Cl. ⁷ , DB name) G06T 1/00 315 G06T 3/00 300 JISST file (JOIS)

Claims (9)

(57) [Claims] An image processing apparatus for generating an interpolated image in a desired viewpoint position and a line-of-sight direction based on a plurality of images captured by discretely arranged cameras, the arbitrary viewpoint position And an input means for inputting the gaze direction information
When, among the planes constituting a polyhedron whose vertices lens center of the cameras taking each of the plurality of images, said input means
Binding in the inputted visual point position on the target pixel in the interpolation image
Extraction means for extracting a plane having a department intersection of a ray based on a plurality of images to the lens center to the vertices of the extracted plane by said extraction means, for generating interpolated virtual image to the viewpoint position of the intersection point Interpolation means, a drawing means for obtaining a pixel corresponding to the pixel of interest from the virtual image, and drawing the pixel of interest using the pixel, and the extraction means and the interpolation for each of the pixels in the interpolation image. the return Succoth repeat the drawing means and means, prior to
The viewpoint position and line-of-sight direction information input by the input means are used.
An image processing apparatus comprising: a generation unit configured to generate the same interpolated image. 2. An image processing method for generating an interpolated image in a desired viewpoint position and a line-of-sight direction based on a plurality of images captured by discretely arranged cameras, the arbitrary viewpoint position And the input process to input the eye direction information
And the input step among the planes forming a polyhedron whose vertex is at the lens center of the camera capturing each of the plurality of images.
Binding in the inputted visual point position on the target pixel in the interpolation image
An extraction step of extracting a plane having a department intersection of a ray based on a plurality of images to the lens center to the vertices of the plane that is extracted by the extraction step, to generate interpolated virtual image to the viewpoint position of the intersection point An interpolation step, a drawing step of acquiring a pixel corresponding to the target pixel from the virtual image, and drawing the target pixel using the pixel, and the extraction step and the interpolation for each pixel in the interpolation image. the process as repeat the drawing process Succoth, before
The viewpoint position and line-of-sight direction information input by the input means are used.
And a generation step of generating the same interpolated image. 3. The interpolating step rotates a plurality of images having a vertex of a plane extracted in the extracting step as a lens center so that an optical axis of each image coincides with a normal direction of the plane,
The image processing method according to claim 2 , wherein a virtual image that is parallel to the plane having the point of view at the intersection is generated by linear interpolation using a rotated image. 4. The image processing method according to claim 2, wherein each plane forming the polyhedron is a triangular plane having three lens centers as vertices. 5. The drawing step sets the coordinates of pixels on the virtual image to P _F and the coordinates of corresponding pixels on the interpolated image to P _e , where these coordinates are the origin of the image. Next order coordinates, A _F is a camera internal parameter matrix of the virtual image, A _e is a camera internal parameter matrix of the interpolated image, R _F is a rotation matrix from the world coordinates to the camera coordinates of the virtual image, and R _e is a world. When the rotation matrix from the coordinates to the camera coordinates of the interpolation image, Claims 2 to 4 noise and obtains a point on the virtual image corresponding to the pixel on the interpolated image by
The image processing method according to either Re. 6. A plurality of images obtained by simultaneously shooting with a plurality of cameras oriented substantially in the omnidirectional direction are stored together with shooting condition information corresponding to the lens center and optical axis of the camera that shot each image. The image processing method according to claim 2 , further comprising a storage step of providing each of the steps. 7. A plane forming the polyhedron is a triangular plane, and in the storing step, a set of three adjacent images of the plurality of images is set, and the same set of three images is used for use in the interpolation step. 7. The image processing method according to claim 6 , further comprising storing correspondence information that associates corresponding areas in one image with each other. 8. The storing step divides each of the three images of the same set into a plurality of layer images for drawing a specific object and stores the plurality of layer images, wherein each layer image includes another layer. The part of the object hidden by the object belonging to is retouched, the area other than the object is an invalid area, the generating step generates an interpolation image for each layer,
The image processing method according to claim 7 , wherein a final interpolated image is obtained by sequentially drawing the interpolated image of a layer distant from the viewpoint position to one closer to the viewpoint position. 9. A storage medium, which stores a control program for realizing the image processing method according to claim 2 by a computer.