JP2016178658A - Synthesis method of light field - Google Patents

Abstract

PROBLEM TO BE SOLVED: To present a composite stereoscopic image free from discomfort, by mutually synthesizing a plurality of sets of multi-view image information, including the stereoscopic information about a separate subject.SOLUTION: Based on a light field Fαr including first subject information and a light field Fβr including second subject information, a third light field Fγr is created on a composite surface W. In order to determine the pixel value of a specific light beam vector Lγ passing a specific point P(xγ, yγ) and directing a specific direction D, a light beam vector Lα passing the intersection P(xα, yα) of a reference straight line R including the specific light beam vector Lγ and the Fαr and directing the specific direction D, and a light beam vector Lβ passing the intersection P(xβ, yβ) of the reference straight line R and the Fβr and directing the specific direction D are identified. Based on at least one of the pixel value of the light beam vector Lα and the pixel value of the light beam vector Lβ, the pixel value of the specific light beam vector Lγ is determined.SELECTED DRAWING: Figure 19

Description

The present invention relates to a light field synthesis method, and more particularly to a technique for synthesizing multi-viewpoint image information used for presenting a stereoscopic image of a subject.

  In order to present a stereoscopic image, various methods have been proposed for a long time, and some of them have already been put into practical use. For example, in Patent Document 1 below, a plurality of parallel projection images obtained by parallel projection of the same subject in different directions are divided into a large number of strip regions, rearranged in a predetermined order, and printed on a medium. A method of presenting a stereoscopic image so that a specific parallel projection image is observed from a specific viewpoint position by disposing a lenticular lens thereon is disclosed.

  Patent Document 2 below discloses a technique for acquiring a multi-viewpoint image by shooting a single subject from multiple directions and displaying a stereoscopic image on a display screen based on the real-view multi-viewpoint image. ing. Such live-action shooting is usually performed while a linear or arc-shaped rail is laid near the subject and the camera is moved along the rail.

  In any of these methods, multi-viewpoint image information is constituted by a plurality of two-dimensional image groups obtained by projecting the same subject in different directions or photographing from different directions, and the subject is based on the multi-viewpoint image information. The method of presenting a stereoscopic image is adopted. That is, the method according to Patent Document 1 specifically presents multi-viewpoint image information by a combination of a print medium and a lenticular lens, whereas the method according to Patent Document 2 performs image processing by a computer. As a result, the multi-viewpoint image information is specifically presented as an image displayed on the display screen.

  Recently, the multi-viewpoint image information composed of a collection of parallel projection images in different directions of the subject is based on the concept of “Light Field” in which light vectors pointing in various directions are defined on a plane. A handling method has been proposed. For example, in the following Non-Patent Document 1, in order to completely describe the behavior of light rays in space, “a point indicated by coordinate values (x, y, z) in space is represented by an azimuth angle φ and an elevation angle θ. The light beam having the wavelength λ has passed through the direction indicated by the symbol “p” at the time t using the seven-dimensional function P called Plenoptic, P (x, y, z, φ, θ, λ, t ) Is disclosed, and as a form of using such a description format for stereoscopic image presentation technology, the following non-patent document 2 describes the concept of “Light Field”. Has been proposed. Further, Patent Documents 3 and 4 below disclose a specific method for handling a three-dimensional image using such a light field.

JP 2007-147737 A JP 2010-226500 A JP 2004-199702 A JP 2008-257686A

Edward H. Adelson and James R. Bergen "The Plenoptic Function and the Elements of Early Vision" Computational Models of Visual Processing (pp. 3-20). Cambridge, MA: MIT Press (1991). Marc Levoy and Pat Hanrahan "Light Field Rendering" Proceedings of ACM SIGGRAPH '96, ACM, 1996, pp.43-54.

  As described above, if multi-viewpoint image information can be prepared for a specific subject, a print medium capable of stereoscopic display of the subject can be created using this multi-viewpoint image information. It is also possible to perform stereoscopic display of the subject.

  However, in order to prepare such multi-viewpoint image information by shooting, it is necessary to shoot a real subject from various directions, and as described above, a rail for moving the camera is laid. Large-scale equipment is required. In addition, a drive control device or the like for accurately controlling the position of the camera at the time of actual shooting is required. When the subject is a small article, live-action shooting is possible in a small studio, but when the subject is a group of people consisting of multiple members, rails and materials are attached to a large facility such as a gymnasium. It is necessary to carry in and assemble these to construct a photographing facility. In addition, when an outdoor scenery or the like is used as a background image of a subject, it is necessary to shoot with a rail installed outside, but in reality, such shooting is very difficult.

  Of course, if a method of preparing multi-viewpoint image information as a CG image is used, since the actual shooting is not necessary, the large-scale equipment described above is not necessary. However, in order to create a subject image of the same quality as a real image with CG, a certain amount of cost and production time are required. For this reason, the multi-viewpoint image information composed only of the CG image is practically suitable only for limited use.

  Accordingly, an object of the present invention is to provide a method for synthesizing a plurality of sets of multi-viewpoint image information including information on separate subjects and presenting a synthesized stereoscopic image without any sense of incongruity.

The present invention relates to a light field synthesis method, but the basic idea of the invention originates in a new method for synthesizing multi-viewpoint image information. Therefore, here, for convenience of description, the multi-viewpoint image information synthesizing method and synthesizing apparatus related to the present invention will be described as a reference mode.

A first reference aspect of the present invention is a multi-viewpoint image information synthesis method for synthesizing multi-viewpoint image information composed of a plurality of n two-dimensional images obtained by parallel projecting subjects in different directions.
The computer inputs the first multi-view image information E (α) in which the first subject α is recorded and the second multi-view image information E (β) in which the second subject β is recorded. An image information input stage;
A synthesis condition setting stage in which a computer sets a synthesis condition for synthesizing the first multi-view image information E (α) and the second multi-view image information E (β);
The i-th (1 ≦ i ≦ n) two-dimensional image Pαi constituting the first multi-view image information E (α) and the i-th constituting the second multi-view image information E (β). An image synthesis step of repeatedly executing, for i = 1 to n, a process of creating an i-th synthesized two-dimensional image Pγi on a predetermined synthesis plane W by synthesizing the second two-dimensional image Pβi.
An image information output stage in which the computer outputs multi-viewpoint image information composed of the n synthesized two-dimensional images Pγ1 to Pγn obtained in the image synthesis stage as synthesized multi-viewpoint image information E (γ);
And do
The i-th two-dimensional image Pαi constituting the first multi-viewpoint image information E (α) projects the first subject α on a predetermined projection plane in a direction parallel to the i-th observation direction Di. The i-th two-dimensional image Pβi forming the parallel projection image obtained in this way and constituting the second multi-viewpoint image information E (β) is obtained by observing the second subject β on the predetermined projection plane. A parallel projection image obtained by projecting in a direction parallel to the direction Di is formed,
The composition condition is the position of the two-dimensional image constituting the first multi-view image information E (α) with respect to the composition surface W and the composition surface W of the two-dimensional image constituting the second multi-view image information E (β). This is a condition indicating the position, and the depth value Zα for the two-dimensional image constituting the first multi-view image information E (α) and the two-dimensional image constituting the second multi-view image information E (β). Depth value Zβ, and
In the image composition stage, the i-th two-dimensional image Pαi constituting the first multi-viewpoint image information E (α) is parallel to the composition surface W and a predetermined position at which the distance to the composition surface W becomes the depth value Zα. And the i-th two-dimensional image Pβi constituting the second multi-viewpoint image information E (β) is placed in a predetermined position parallel to the composite surface W and the distance to the composite surface W is the depth value Zβ. A parallel projection image obtained by projecting the two-dimensional image Pαi in a direction parallel to the i-th observation direction Di and the two-dimensional image Pβi parallel to the i-th observation direction Di on the composite plane W. The i-th synthesized two-dimensional image Pγi is created by forming parallel projected images obtained by projecting in various directions and combining the parallel projected images in consideration of the positional relationship based on the depth values Zα and Zβ. It is what you do.

A second reference aspect of the present invention is a multi-viewpoint image information synthesis method according to the first reference aspect described above,
A plurality of n sets of direction vectors passing through the reference point Q on a plane including the predetermined reference point Q are defined as first multi-view image information E (α) and second multi-view image information E (β). Multi-viewpoint image information in which the directions of n sets of direction vectors are the individual observation directions is used.

According to a third reference aspect of the present invention, in the method for synthesizing multi-viewpoint image information according to the first reference aspect described above,
A vector passing through the origin O of the XYZ three-dimensional coordinate system, which is a direction vector D () specified by an azimuth angle φ formed by an orthogonal projection image on the XY plane and the Y axis, and an elevation angle θ with respect to the XY plane. φ, θ) are defined as a × b by changing θ to a and φ to b, and the first multi-view image information E (α) and the second multi-view image information E (β ), Multi-viewpoint image information having a total of n (however, n = a × b) direction vector directions as individual observation directions is used.

A fourth reference aspect of the present invention is the above-described multi-viewpoint image information synthesis method according to the first to third reference aspects .
In the synthesis condition setting stage, as the values of the depth values Zα and Zβ, a positive value or a negative value or zero is set for one, a positive value or a negative value is set for the other,
In the image composition stage, composition is performed such that a parallel projection image of an image having a small depth value is observed in front.

A fifth reference aspect of the present invention is the above-described method for synthesizing multi-viewpoint image information according to the fourth reference aspect .
In the image composition stage, on the composite surface W, the formed parallel projection image is recorded as it is for the region where the formed parallel projection image does not overlap, and the region where the formed parallel projection image is overlapped is forward. The composition is performed by recording only the observed parallel projection images.

A sixth reference aspect of the present invention is a multi-viewpoint image information synthesis method according to the fifth reference aspect described above.
At least one of n two-dimensional images Pα1 to Pαn constituting the first multi-view image information E (α) and n two-dimensional images Pβ1 to Pβn constituting the second multi-view image information E (β). Is an image containing a pixel with a pixel value indicating the background attribute,
In the image composition stage, a parallel projection image is not formed for an area composed of pixels having a pixel value indicating the background attribute, and when there is an area where no parallel projection image is formed on the synthesis surface W, For the pixels in the region, a synthesized two-dimensional image is created by giving a pixel value indicating a background attribute.

A seventh reference aspect of the present invention is the above-described multi-viewpoint image information synthesis method according to the first to sixth reference aspects .
At least one of n two-dimensional images Pα1 to Pαn constituting the first multi-view image information E (α) and n two-dimensional images Pβ1 to Pβn constituting the second multi-view image information E (β). Are constituted by the same common image.

An eighth reference aspect of the present invention is a stereoscopic image presentation method using the method for synthesizing multi-viewpoint image information according to the first to seventh reference aspects described above.
N pieces of two-dimensional images Pα1 to Pαn constituting the first multi-viewpoint image information E (α) and n pieces of two-dimensional images Pβ1 to Pβn constituting the second multi-viewpoint image information E (β) are prepared. Multi-viewpoint image information preparation stage,
A multi-view image information combining step of creating the combined multi-view image information E (γ) by executing the multi-view image information combining method;
The i-th (1 ≦ i ≦ n) composite two-dimensional image Pγi presents n composite two-dimensional images Pγ1 to Pγn constituting the multi-viewpoint image information E (γ) mainly in the i-th observation direction Di. An image presentation stage to be presented in such a manner;
Is to do.

According to a ninth reference aspect of the present invention, in the stereoscopic image presentation method using the multi-view image information combining method according to the eighth reference aspect described above,
As a multi-viewpoint image information preparation stage,
A first photographing stage for obtaining a total of ζ first live-action photographed image groups by photographing the first subject α from a plurality of ζ-directions;
A second photographing stage for obtaining a total of η second photographed images by photographing the second subject β from a plurality of η directions;
Based on a total of ζ first live-action photographed image groups, the first subject α is placed on a predetermined projection plane in n observation directions from the first observation direction D1 to the nth observation direction Dn. First multi-view image information creation for obtaining parallel projection images obtained by projecting in parallel directions and creating n two-dimensional images Pα1 to Pαn constituting the first multi-view image information E (α) Stages,
Based on a total of η second actual photographed image groups, the second subject β is placed on a predetermined projection plane in n observation directions from the first observation direction D1 to the nth observation direction Dn. Second multi-view image information creation for obtaining n two-dimensional images Pβ1 to Pβn constituting second multi-view image information E (β), respectively, by obtaining parallel projection images obtained by projecting in parallel directions Stages,
Is to do.

Tenth reference aspect of the present invention is a method of creating a three-dimensional image recording medium utilizing a synthetic method of the multi-view image information according to the first to seventh reference embodiment described above,
N pieces of two-dimensional images Pα1 to Pαn constituting the first multi-viewpoint image information E (α) and n pieces of two-dimensional images Pβ1 to Pβn constituting the second multi-viewpoint image information E (β) are prepared. Multi-viewpoint image information preparation stage,
A multi-view image information combining step of creating multi-view image information E (γ) by executing a multi-view image information combining method;
The n composite two-dimensional images Pγ1 to Pγn constituting the multi-viewpoint image information E (γ) are each divided into a plurality of m partial images, and the recording surface on the recording medium is divided into a plurality of m sections K1 to Km. A medium recording step of dividing and recording each of the i-th partial images of the n synthesized two-dimensional images Pγ1 to Pγn in the i-th section Ki on the recording medium;
An optical element disposing step of disposing an optical element on each section on the recording medium such that a partial image of the i-th synthesized two-dimensional image Pγi is mainly presented in the i-th observation direction Di;
Is to do.

In an eleventh reference aspect of the present invention, the above-described method for synthesizing multi-view image information according to the first to seventh reference aspects is executed by incorporating a program in a computer.

A twelfth reference aspect of the present invention is a multi-viewpoint image information synthesizing apparatus that synthesizes multi-viewpoint image information composed of a plurality of n two-dimensional images obtained by parallel projecting subjects in different directions.
Image information storage for storing first multi-view image information E (α) in which the first subject α is recorded and second multi-view image information E (β) in which the second subject β is recorded And
A synthesis condition storage unit that stores a synthesis condition for synthesizing the first multi-view image information E (α) and the second multi-view image information E (β) based on an operator setting instruction;
The i-th (1 ≦ i ≦ n) two-dimensional image Pαi constituting the first multi-view image information E (α) and the i-th two constituting the second multi-view image information E (β). An image compositing unit that synthesizes the dimensional image Pβi and repeatedly executes the process of creating the i-th composite two-dimensional image Pγi on a predetermined composite surface W for i = 1 to n.
A composite image storage unit that stores multi-viewpoint image information composed of n composite two-dimensional images Pγ1 to Pγn created by the image composition unit as composite multi-viewpoint image information E (γ);
Provided,
The first multi-view image information E (α) stored in the image information storage unit is composed of an aggregate of n two-dimensional images, and the i-th two-dimensional image Pαi of the first multi-view image information E (α) A parallel projection image obtained by projecting the subject α on a predetermined projection plane in a direction parallel to the i-th observation direction Di;
The second multi-viewpoint image information E (β) stored in the image information storage unit is composed of a set of n two-dimensional images, and the i-th two-dimensional image Pβi is the second A parallel projection image obtained by projecting the subject β on a predetermined projection plane in a direction parallel to the i-th observation direction Di;
The composition condition setting unit at least obtains a depth value Zα indicating the position of the two-dimensional image constituting the first multi-view image information E (α) with respect to the composition plane W and the second multi-view image information E (β). A depth value Zβ indicating a position with respect to the composite plane W of the two-dimensional image to be configured, and
The image composition unit converts the i-th two-dimensional image Pαi constituting the first multi-viewpoint image information E (α) stored in the image information storage unit to be parallel to the composition surface W and at a distance from the composition surface W. The i-th two-dimensional image Pβi that is arranged at a predetermined position so as to have the depth value Zα and constitutes the second multi-viewpoint image information E (β) stored in the image information storage unit is displayed on the composite plane W. A parallel position obtained by projecting the two-dimensional image Pαi onto the composite surface W in a direction parallel to the i-th observation direction Di is arranged at a predetermined position so that the distance to the composite surface W is the depth value Zβ. Forming a projection image and a parallel projection image obtained by projecting the two-dimensional image Pβi in a direction parallel to the i-th observation direction Di, and regarding the parallel projection image based on the depth values Zα and Zβ The i-th composite 2D image Pγi is created by combining And stored in the composite image storage unit.

The multi-viewpoint image information synthesizing method and synthesizing apparatus related to the present invention has been described above as the reference mode. The present invention provides a new method for synthesizing a light field by utilizing the technical idea described as the reference embodiment. Hereinafter, an aspect of the light field synthesis method according to the present invention will be described.

(1) A first aspect of the present invention is a plane on which a plurality of points P are arranged, and each of the individual points P has a plurality of ray vectors that are directed in different directions through the point P. In a light field composition method for composing a light field defined as a plane in which L is defined and a specific feature value is assigned to each ray vector L,
The first light field Fα in which the computer indicates individual feature values by a function having the coordinate value (x, y) of the point P on the XY two-dimensional coordinate plane and the direction D of the ray vector L passing through the point P as variables. A light field input stage for inputting (x, y, D) and a second light field Fβ (x, y, D);
A synthesis condition setting stage in which a computer sets a synthesis condition for synthesizing the first light field and the second light field;
The computer synthesizes the first light field Fα (x, y, D) and the second light field Fβ (x, y, D), and the coordinate value (x of the point P on the XY two-dimensional coordinate plane) , Y) and a light field synthesis stage for creating a third light field Fγ (x, y, D) indicating individual feature values by a function having the direction D of the light vector L passing through the point P as a variable,
A light field output stage in which the computer outputs a third light field Fγ (x, y, D) obtained in the light field synthesis stage;
And
The composition condition includes an arrangement condition for arranging the first light field and the second light field at a predetermined position so as to be parallel to the predetermined composition surface W.
In the light field synthesis stage, the first light field and the second light field are arranged based on the arrangement condition, and the feature value of the specific light vector passing through the specific point P on the composite surface W and facing the specific direction D is obtained. , The characteristic value of the light vector passing through the intersection of the reference straight line including the specific light vector and the first light field and pointing in the specific direction D, and the specific direction D passing through the intersection of the reference straight line and the second light field. The composite surface W is determined based on at least one of the feature values of the light vector, and a predetermined feature value is given to each light vector passing through each point, and the third light field Fγ (x, y, D).

(2) According to a second aspect of the present invention, in the light field synthesis method according to the first aspect described above,
In the light field synthesis stage,
If only one of the intersection between the reference line and the first light field and the intersection between the reference line and the second light field exists, the ray vector of the light vector directed through the existing intersection and pointing in the specific direction D Determining the feature value of the third light field based on the feature value;
When both the intersection of the reference line and the first light field and the intersection of the reference line and the second light field exist, the specific direction D passes through one of the intersections selected based on the arrangement condition. The feature value of the third light field is determined on the basis of the feature value of the ray vector directed toward the.

(3) According to a third aspect of the present invention, in the light field synthesis method according to the first or second aspect described above,
For each of a large number of pixels having a predetermined area arranged on the XY two-dimensional coordinate plane, the first to third light fields are pixel feature values of a plurality of ray vectors passing through the representative point P of the pixel. Consists of image data as values,
In the light field synthesis stage, the pixel values of the individual pixels constituting the third light field are set to the pixel values of the pixels on the first light field including the intersection of the reference line and the first light field, and the reference line. It is determined based on at least one of the pixel values of the pixels on the second light field including the intersection with the second light field.

(4) According to a fourth aspect of the present invention, in the light field synthesis method according to the first to third aspects described above,
A combination of pixel values of the three primary colors R, G, and B is used as a feature value to be assigned to each light vector.
The first light field is configured by a combination of a primary color R component Fαr (x, y, D), a primary color G component Fαg (x, y, D), and a primary color B component Fαb (x, y, D),
The second light field is configured by a combination of a primary color R component Fβr (x, y, D), a primary color G component Fβg (x, y, D), and a primary color B component Fβb (x, y, D),
The third light field is configured by a combination of a primary color R component Fγr (x, y, D), a primary color G component Fγg (x, y, D), and a primary color B component Fγb (x, y, D),
In the light field composition stage, the feature value is determined independently for each primary color component.

(5) According to a fifth aspect of the present invention, in the light field synthesis method according to the first to fourth aspects described above,
At least one of the first light field and the second light field is a light field to which the same feature value is assigned regardless of the direction D with respect to a ray vector passing through the same point P. is there.

(6) According to a sixth aspect of the present invention, the above-described light field composition method according to the first to fifth aspects is executed by incorporating a program in a computer.

  According to the light field combining method of the present invention, the first light field and the second light field are arranged at predetermined positions so as to be parallel to the predetermined composite surface W, and A characteristic value of a specific light vector passing through a specific point P and pointing in a specific direction D is determined, and a composite plane W to which a predetermined characteristic value is given to each light vector passing through each point is used as a third light field. Since Fγ (x, y, D) is set, it becomes possible to synthesize any light field. For this reason, it is possible to present a synthetic stereoscopic image that includes information about individual subjects and has no sense of incongruity.

It is a front view which shows an example of the multi-viewpoint image information E comprised by the two-dimensional parallel projection image P1-P5 which parallel-projected the to-be-photographed object (alpha) to the different direction, respectively. 6 is a top view showing a state in which the subject α is being projected in parallel with respect to the projection plane S. FIG. 6 is a top view showing a state in which two subjects α1, α2 are projected in parallel with respect to the projection plane S. FIG. FIG. 6 is a plan view showing a difference between a parallel projection image (FIG. (A)) and a photographed image (FIG. (B)) of two subjects α1, α2. FIG. 2 is a plan view (upper and middle) and a top view (lower) showing a process of creating a stereoscopic image recording medium based on the multi-viewpoint image information E shown in FIG. 1. FIG. 6 is a perspective view showing angle parameters φ and θ for specifying a direction vector D. It is a top view which shows an example of the multi-viewpoint image information comprised by the two-dimensional parallel projection image P11-P35 which parallel-projected the to-be-photographed object to the up-down and left-right different directions. FIG. 8 is a plan view showing a process for creating a stereoscopic image recording medium based on the multi-viewpoint image information shown in FIG. 7. It is a top view which shows an example of the equipment which performs a real photography with respect to the to-be-photographed object (alpha). FIG. 10 is a block diagram illustrating a process of creating multi-viewpoint image information from a large number of captured images obtained by the real-image shooting shown in FIG. 9. It is a top view which shows the basic principle which synthesize | combines 1st multi-view image information E ((alpha)) and 2nd multi-view image information E ((beta)), and produces | generates 3rd multi-view image information E ((gamma)). . FIG. 12 is a plan view showing a general method for obtaining a composite image Pγ3 by combining the two-dimensional images Pα3 and Pβ3 shown in FIG. 11. It is a top view which shows the synthetic | combination method of each two-dimensional image in this invention. It is a flowchart which shows the basic procedure of the synthetic | combination method of the multi viewpoint image information which concerns on this invention. It is a flowchart which shows the basic procedure of the stereo image presentation method which concerns on this invention. It is a block diagram which shows the basic composition of the synthesizing | combining apparatus of the multiview image information which concerns on this invention. It is a top view which shows another example of 1st multi-view image information E ((alpha)) used as the synthesis target by this invention, and 2nd multi-view image information E ((beta)). It is the perspective view and mathematical formula which show the concept of a general light field. It is a perspective view which shows the basic principle of the synthetic | combination method of the light field based on this invention. It is a perspective view which shows the specific method of the synthetic | combination method of the light field based on this invention.

Hereinafter, the present invention will be described based on the illustrated embodiments. As described above, the present invention relates to a light field composition method, but the basic idea of the present invention originates from a new method for compositing multi-viewpoint image information. Therefore, for convenience of explanation, first, a method for synthesizing multi-viewpoint image information related to the present invention will be described.

<<< §1. Presentation method of stereoscopic image using multi-viewpoint image information >>>
The present invention is a technique that can be used to present a stereoscopic image of a subject using multi-viewpoint image information. Here, the multi-viewpoint image information is a collection of two-dimensional images obtained by parallel projecting subjects to be presented in different directions. The technique itself for presenting a stereoscopic image using such multi-viewpoint image information is known as disclosed in the above-mentioned Patent Documents 1 and 2, etc., but here, for convenience of explanation of the present invention, The basic principle will be briefly explained.

  FIG. 1 is a front view showing an example of multi-viewpoint image information E composed of two-dimensional parallel projection images P1 to P5 obtained by parallel projecting a subject α (coffee cup in this example) in different directions. In this example, a projection surface S having a finite area is disposed in front of the subject α, and the subject α is projected onto the projection surface S. Arrows D1 to D5 originating from the reference point Q indicate the projection direction on the projection plane S, and viewpoints V1 to V5 are set in the tip direction, respectively. Eventually, the arrows D1 to D5 indicate the directions from the reference point Q toward the viewpoints V1 to V5. Here, the directions indicated by the arrows are referred to as observation directions D1 to D5. In FIG. 1, the reference point Q is set at a position slightly before the subject α. However, if the reference point Q is set at the center point of the subject α, the subject α having a more effective stereoscopic effect is presented. It becomes possible to do.

  The two-dimensional images P1 to P5 shown in the lower part of FIG. 1 are two-dimensional images obtained on the projection plane S when the subject α is projected in parallel in the observation directions D1 to D5, respectively, and are generally called “slices”. ing. If the projection plane S is considered as a window, these two-dimensional images P1 to P5 can be said to be images obtained by observing the subject α through the window from the positions of the viewpoints V1 to V5, respectively. The multi-viewpoint image information E is information constituted by an aggregate of these two-dimensional images P1 to P5, and specifically, an aggregate of image data indicating these five two-dimensional images P1 to P5. Become.

  It should be noted that each of the two-dimensional images P1 to P5 is not a two-dimensional photographed image obtained by photographing the subject α from the positions of the viewpoints V1 to V5, but the subject α is projected in parallel in different observation directions D1 to D5. This is the projected image. FIG. 2 is a top view showing a state in which the subject α is being projected in parallel with respect to the projection plane S. FIG. 2A shows a state of parallel projection in the observation direction D3 (direction toward the viewpoint V3 at the front position in FIG. 1), and a parallel projection image of the subject α on the portion indicated by the bold line on the projection plane S. P3 will be obtained. Similarly, FIG. 2 (b) shows a state of parallel projection in the observation direction D2 (direction toward the viewpoint V2 at an oblique left position in FIG. 1), and the subject α is shown in a portion indicated by a thick line on the projection plane S. The parallel projection image P2 is obtained.

  In general, a photographed image obtained by photographing using an optical system such as a lens is an image that expresses a subject on a two-dimensional plane using a perspective method, but constitutes multi-viewpoint image information E. The individual two-dimensional images P1 to P5 are not expressed using such a perspective method, but are merely projected onto the projection plane S by parallelly projecting each point on the subject toward a predetermined observation direction. This is an image to be obtained.

  For example, as in the example shown in FIG. 3, two subjects α1, α2 of the same size (in the example shown, two sets of the same coffee cup) are arranged with subject α1 in front and subject α2 in the back. When parallel projection is performed on the projection surface S toward the observation direction D3, parallel projection images P3 of the subjects α1 and α2 (parts of the subjects α1 and α2 partially overlap each other) are obtained in the portions indicated by thick lines. FIG. 4A shows a parallel projection image obtained on the projection surface S in this way. On the other hand, FIG. 4B shows a photographed image obtained by photographing two subjects α1 and α2 from the front. In the photographed image shown in FIG. 4 (b), the subject α2 in the back is smaller than the subject α1 in the foreground, but this is because the photographing is performed using an optical system such as a lens. This is because an applied two-dimensional image (a so-called parsed image) is formed.

  When multi-viewpoint image information E composed of five two-dimensional images P1 to P5 (parallel projection images) as shown in the lower part of FIG. 1 is prepared, each two-dimensional image P1 is placed at the position of the projection plane S. When .about.P5 is presented, the subject .alpha. Can be presented as a stereoscopic image. However, at this time, the individual two-dimensional images P1 to P5 are presented only in the corresponding observation directions D1 to D5, respectively, using some means. For example, the two-dimensional image P1 is presented only in the corresponding observation direction D1, and is not presented in the other observation directions D2 to D5. Then, when viewed from the illustrated viewpoint V1, the two-dimensional image P1 appears to be displayed on the projection plane S, and when viewed from the illustrated viewpoint V2, the two-dimensional image is displayed on the projection plane S. P2 appears to be displayed, and the two-dimensional images obtained when observed from individual viewpoints are different from each other, and a stereoscopic effect is obtained.

  Here, for convenience of explanation, an example in which five observation directions D1 to D5 are defined and five two-dimensional images P1 to P5 are created is shown. However, in practical use, finer observation is performed. By defining direction variations (for example, 70 different observation directions D1 to D70) and creating parallel projection images (for example, 70 two-dimensional images P1 to P70) corresponding to the individual observation directions, It is possible to present a stereoscopic image that does not feel strange even when moved.

  In short, the subject α is projected in parallel in different observation directions D1 to Dn to form two-dimensional images P1 to Pn on the same projection plane S, and the multi-viewpoint image information E is configured by these n two-dimensional images. By doing so, it is possible to present a stereoscopic image using the multi-viewpoint image information E.

  Specifically, the i-th two-dimensional image Pi formed on the projection surface S may be presented only in the i-th observation direction Di by using some means. However, in reality, it is difficult and strictly unnecessary to present the i-th two-dimensional image Pi strictly only in the i-th observation direction Di. Therefore, practically, the i-th two-dimensional image Pi may be presented not only in the i-th observation direction Di but also in the vicinity thereof. In short, if a method is employed in which the i-th two-dimensional image Pi is presented mainly in the i-th observation direction Di, a stereoscopic image that is practically unhindered can be presented.

  Next, a specific example using a lenticular lens will be described as an example of a technique in which the i-th two-dimensional image Pi is mainly presented in the i-th observation direction Di. Here, a specific process for creating a stereoscopic image recording medium based on multi-viewpoint image information E constituted by five two-dimensional images P1 to P5 as shown in the lower part of FIG. 1 will be described with reference to FIG. While explaining.

  The upper part of FIG. 5 is a plan view showing an example in which six strip-shaped partial images are formed by dividing the two-dimensional images P1 to P5 shown in the lower part of FIG. For example, the two-dimensional image P1 is divided into six partial images P1a to P1f, the two-dimensional image P2 is divided into six partial images P2a to P2f, and so on. In FIG. 5, for convenience of explanation, the constituent areas of the two-dimensional images P <b> 1 to P <b> 5 are hatched differently so that they can be distinguished from each other.

  Thus, when 30 pieces of strip-shaped partial images are prepared in total by dividing the five two-dimensional images P1 to P5 into six equal parts, these 30 sets of partial images are shown in the middle diagram of FIG. In this way, printing is performed on the recording medium M. Here, when an area in which five sets of partial images are adjacently arranged is defined as one section, and six sections Ka to Kf are defined as shown in the drawing, each of the first sections Ka located on the leftmost side has two dimensions. The partial images P1a, P2a, P3a, P4a, and P5a located on the leftmost side of the images P1 to P5 are arranged in this order. Similarly, in the second section Kb, the partial images P1b, P2b, P3b, P4b, and P5b that are located second from the left of the two-dimensional images P1 to P5 are arranged in this order, and so on. In the section Kf, the partial images P1f, P2f, P3f, P4f, and P5f located on the rightmost side of the two-dimensional images P1 to P5 are arranged in this order.

  After all, if the multi-viewpoint image information E is composed of a plurality of n two-dimensional images P1 to Pn, and each individual two-dimensional image is divided into a plurality of m partial images, recording on the recording medium M is performed. The surface may be divided into a plurality of m sections K1 to Km, and the i-th partial images of the n two-dimensional images P1 to Pn may be recorded in the i-th section Ki (FIG. 5). In this example, n = 5 and m = 6 are set).

  Subsequently, when the lenticular lens L is arranged on the recording medium M subjected to such printing, a stereoscopic image recording medium on which the subject α is recorded is obtained. The lower part of FIG. 5 is a top view of the stereoscopic image recording medium thus created. The lenticular lens L is a collection of lenses having a bowl-shaped cross section arranged for each of the sections K1 to Km, and the lens arranged on one section is the i-th printed on the section. This function serves to refract light so that the partial image of the two-dimensional image Pi is presented in the i-th observation direction Di.

  For example, in the example shown in the lower part of FIG. 5, five sets of partial images P1a, P2a, P3a, P4a, and P5a are printed in the leftmost section Ka, but the lenticular lens L arranged thereon is also shown. By function, the light from the partial image P1a is refracted in the observation direction D1, the light from the partial image P2a is refracted in the observation direction D2, and the light from the partial image P5a is refracted in the observation direction D5. Become. Such a refraction phenomenon occurs in all of the illustrated sections Ka to Kf. Eventually, the i-th two-dimensional image Pi is presented in the i-th observation direction Di.

  5 shows an example in which strip-shaped partial images P1a to P5f taken out from the individual two-dimensional images P1 to P5 are printed on the recording medium M at the same scale. Can be enlarged or reduced at an arbitrary magnification during printing. In particular, with respect to the horizontal direction, if printing is performed on the recording medium M at the same scale, the stereoscopic image of the observed subject α is an image that is stretched five times in the horizontal direction. Therefore, in practice, when printing on the recording medium M, it is preferable to perform a process of reducing the strip-shaped individual partial images P1a to P5f to 1/5 in the horizontal direction.

  In the stereoscopic image recording medium created in this way, a stereoscopic image in which parallax is generated by moving the viewpoint in the horizontal direction of the figure is presented, but when the viewpoint is moved in the vertical direction of the figure, the parallax is displayed. Will not occur. As shown in FIG. 1, the five observation directions D1 to D5 are set to the observation directions that are changed in the horizontal direction, and the two-dimensional images P1 to P5 are horizontally displayed as shown in the upper part of FIG. This is because a vertically long strip-shaped partial image is formed by dividing the image into six in the direction, and the partial images P1a to P5f are arranged in the horizontal direction as shown in the middle of FIG.

  Of course, if necessary, it is also possible to create a stereoscopic image recording medium capable of presenting a stereoscopic image in which parallax occurs even if the viewpoint is moved in any direction. For this purpose, it is only necessary to set a variation that changes the observation direction not only horizontally but also vertically. That is, in the case of the example shown in FIG. 1, when the observation directions D1 to D5 are each considered as a direction vector indicating a predetermined direction, all the direction vectors are arranged on the same plane including the reference point Q. By setting a direction vector pointing in an arbitrary direction in a three-dimensional space as an observation direction, multi-viewpoint image information E capable of presenting a stereoscopic image in which parallax occurs even when the viewpoint is moved in either the vertical or horizontal direction. Can be created.

  In general, in the XYZ three-dimensional coordinate space, a direction vector that points to an arbitrary direction through the origin O, as shown in FIG. 6, uses two angle parameters φ and θ, which are an azimuth angle φ and an elevation angle θ, and is a vector D (Φ, θ) can be represented (in the description and the like, the arrow symbol indicating a vector is omitted) due to restrictions of the electronic application. Here, the azimuth angle φ is given as an angle formed between the orthogonal projection image D ′ of the direction vector D on the XY plane and the Y axis (of course, it may be defined as an angle formed with the X axis), The elevation angle θ is given as an angle formed by the direction vector D and the XY plane.

  The example shown in FIG. 1 can be said to be an example in which the reference point Q is defined at the position of the origin O and the observation directions D1 to D5 are defined using only the direction vector included on the XY plane. The vector is represented by a vector D (φ) using only the azimuth angle φ. The direction vector D (φ, θ) defined by adding the elevation angle θ to this can point to an arbitrary direction in the three-dimensional coordinate space. If the variation of the observation direction is defined using such a direction vector D (φ, θ), it is possible to present a stereoscopic image in which parallax occurs even if the viewpoint is moved in any direction.

  Here, an example in which the azimuth angle φ is changed in five ways as a variation in the horizontal direction, and an elevation angle θ is changed in three ways as a variation in the vertical direction, as in the example shown in FIG. In this example, 15 observation directions D11 to D15, D21 to D25, and D31 to D35 are eventually defined. That is, the observation directions D11 to D15 are obtained by changing the azimuth angle φ to φ1 to φ5 while the elevation angle θ is fixed to the first value θ1, and the observation directions D21 to D25 are obtained by changing the elevation angle θ to the second value. The observation direction D31 to D35 is obtained by changing the azimuth angle φ to φ1 to φ5 with the value θ2 being fixed, and the observation direction D31 to D35 is the azimuth angle φ being φ1 with the elevation angle θ being fixed to the third value θ3. It is obtained by changing to ~ φ5.

  FIG. 7 is a plan view showing multi-viewpoint image information E composed of 15 kinds of two-dimensional images P11 to P35 obtained by setting such 15 kinds of observation directions. Each of the two-dimensional images P11 to P35 is provided with a projection plane S having a finite area in front of the subject α, in the same manner as in the example shown in FIG. 1, and the subject α is in the direction indicated by the observation directions D11 to D35, respectively. It is a parallel projection image formed on the projection surface S by performing parallel projection. Such multi-viewpoint image information E includes not only parallax information in the horizontal direction (direction in which the azimuth angle φ is changed) but also parallax information in the vertical direction (direction in which the elevation angle θ is changed). It is possible to present a stereoscopic image in which parallax occurs even when the viewpoint is moved in any direction.

  FIG. 8 is a plan view showing a process of creating a stereoscopic image recording medium based on the multi-viewpoint image information E shown in FIG. The upper half of FIG. 8 shows an example in which 24 rectangular partial images are formed by dividing the two-dimensional images P11 to P35 shown in FIG. 7 into 6 equal parts in the horizontal direction and 4 equal parts in the vertical direction. It is a top view. Thus, when a total of 360 sets of rectangular partial images are prepared by dividing the 15 two-dimensional images P11 to P35 into 24 equal parts, these 360 sets of partial images are shown in the lower half of FIG. In this way, printing is performed on the recording medium M.

  Here, the entire area constituting the recording medium M is divided into six equal parts in the horizontal direction and four equal parts in the vertical direction, and a total of 24 sections Ka to Kx are defined. Each section is further divided into five equal parts in the horizontal direction and three equal parts in the vertical direction, and is divided into a total of 15 minute regions. Each rectangular partial image constituting the two-dimensional images P11 to P35 is assigned to each minute region. For example, partial images P11a, P12a, P13a, P14a, and P15a are allocated to the minute area in the first row constituting the upper left section Ka of the recording medium M, and the partial images P21a, P15a, are assigned to the minute area in the second row. P22a, P23a, P24a, and P25a are allocated, and partial images P31a, P32a, P33a, P34a, and P35a are allocated to the minute region in the third row.

  In other words, the first row and first column partial images P11a to P35a of the two-dimensional images P11 to P35 are allocated to the section Ka of the first row and first column on the recording medium M. . Similarly, partial images P11b to P35b in the first row and second column of the two-dimensional images P11 to P35 are allocated to the section Kb in the first row and second column on the recording medium M. In short, the jth row and kth column partial images of the two-dimensional images P11 to P35 are allocated to the jth row and kth column of the recording medium M.

  Subsequently, if a microlens array is arranged on the recording medium M subjected to such printing, a stereoscopic image recording medium on which the subject α is recorded can be obtained. This microlens array is an assembly of independent microlenses for each of the sections Ka to Kx, and the microlens arranged on one section is the i-th two-dimensional printed on the section. It fulfills the function of refracting light so that the partial image of the image Pi is presented in the i-th observation direction Di. For example, in the illustrated section Ka, 15 sets of partial images P11a to P35a are printed, but the light from the partial image P11a is refracted in the observation direction D11 by the function of the microlens arranged thereon, The light from the partial image P12a is refracted in the observation direction D12, and the light from the partial image P35a is refracted in the observation direction D35. Such a refraction phenomenon occurs in all of the illustrated sections Ka to Kx. Eventually, the i-th two-dimensional image Pi is presented in the i-th observation direction Di.

  In the stereoscopic image recording medium created in this way, a stereoscopic image in which parallax occurs is presented regardless of whether the viewpoint is moved in the horizontal direction of the figure or the viewpoint is moved in the vertical direction of the figure.

<<< §2. General multi-view image information creation method >>
As described in §1, individual two-dimensional images (for example, two-dimensional images P1 to P5 shown in the lower part of FIG. 1) constituting the multi-viewpoint image information E are normal captured images obtained by capturing the subject α with a camera or the like. Instead, as illustrated in FIG. 2, it is a parallel projection image obtained by parallel projecting the subject α in a specific observation direction.

  One method of creating the multi-viewpoint image information E composed of such a set of parallel projection images is a method using a CG technique. For example, if a subject α as shown in FIG. 2 is prepared as a virtual CG image by a computer, a projection image obtained by projecting the subject α in parallel in an arbitrary direction on the projection plane S defined in the virtual space. Can be obtained by calculation. Therefore, it is possible to generate two-dimensional images P1 to P5 as shown in the lower part of FIG.

  However, in order to create a high-quality subject image with CG, a certain amount of cost and production time are required. Therefore, in view of commercial use, its use must be limited. Therefore, for practical purposes, real shooting was performed on the subject α from various directions, and geometric calculation processing based on a large number of obtained image data was performed, so that the quality was equivalent to that of the real shooting image. A method of creating multi-viewpoint image information E composed of a collection of parallel projection images is used.

  FIG. 9 is a top view showing an example of equipment for performing such a real-life shooting on the subject α. In this example, a linear shooting path T from the shooting start point A to the shooting end point B is provided in the vicinity of the subject α. Actually, a rail is laid along the photographing route T, and the camera can move on the rail. The shooting route T does not necessarily have to be a straight route, and may be a route along an arbitrary curve. However, in order to simplify the conversion calculation processing described later as much as possible, a straight route as shown in the figure is used. Or an arcuate path.

  If the subject α is photographed while moving the camera from the photographing start point A to the photographing end point B along such a photographing route T, a photographed image taken from any photographing point on the photographing route T is obtained. be able to. The figure shows a state where the subject α is being photographed from the i-th photographing point T (i). Of course, since such shooting is performed by a camera using a normal lens or the like, the obtained shot image is an image expressed using a perspective method.

  If 1000 shooting points are set on the shooting path T, the shooting start point A is the first shooting point T (1), and the shooting end point B is the 1000th shooting point T (1000). A photographed image is obtained. Here, 1000 photographed images obtained in this way will be referred to as actual photographed images U1 to U1000. By accurately controlling the movement of the camera during shooting, it is possible to accurately grasp the geometric shooting conditions from which each live-action shot image was obtained. A plurality of parallel projection images constituting the multi-viewpoint image information E can be obtained by performing geometric calculation processing on the images U1 to U1000.

  FIG. 10 is a block diagram showing a process of creating the two-dimensional images P1 to P70 constituting the multi-viewpoint image information E by a predetermined conversion calculation process based on the photographed images U1 to U1000 obtained by the photographed shooting shown in FIG. FIG. In the case of this example, 70 two-dimensional images P1 to P70 are generated based on 1000 photographed images U1 to U1000. The 70 two-dimensional images P1 to P70 are projection images obtained on a predetermined projection plane S when the subject α is projected in parallel toward predetermined observation directions D1 to D70, respectively.

  For example, Hiroshi Yoshikawa and Yasuhiro Takagi “Development of a three-dimensional camera for high-density display of directional images” (execution of a three-dimensional image conference) Issued by the Committee: 3D Image Conference Proceedings Vol. 2003, p. 229-232), etc., and is not a process directly related to the important features of the present invention. Detailed description is omitted.

  Eventually, if an actual subject is photographed with respect to an arbitrary subject α using a photographing facility as shown in FIG. 9, and conversion processing as shown in FIG. Multi-viewpoint image information E (two-dimensional image group on which parallel projection images are recorded) can be obtained. However, the larger the subject α, the larger the necessary photographing equipment. For example, when a person's whole body image is used as the subject α, a studio having a certain size is required in order to install facilities necessary for live-action shooting. Therefore, when the subject is an aggregate of persons consisting of a plurality of members, a large facility as large as a gymnasium must be prepared. Further, when it is desired to use an outdoor scenery as the background image of the subject, it is necessary to provide a photographing facility outdoors, which is very difficult in practice.

  The present invention proposes a specific method for solving such a problem, and an object of the present invention is to synthesize a plurality of sets of multi-viewpoint image information E each including information on separate subjects. Then, it is to be able to present a synthetic stereoscopic image without a sense of incongruity.

<<< §3. Principle of composition of multi-view image information according to the present invention >>>
FIG. 1 shows multi-viewpoint image information for stereoscopically displaying a subject α composed of a coffee cup. And as described in §2, such multi-viewpoint image information can be created by photographing the subject α using a photographing facility as shown in FIG. Of course, instead of the subject α made of a coffee cup, for example, the same image is taken using the subject β made of an apple, multi-viewpoint image information for stereoscopic display of the subject β can be obtained. The basic concept of the present invention is to obtain multi-viewpoint image information for stereoscopic display of a plurality of subjects by mutually synthesizing a plurality of sets of multi-viewpoint image information including information on such separate subjects. It is in.

  For example, if multi-viewpoint image information for presenting a three-dimensional image in which coffee cups and apples are juxtaposed is created based on live-action shooting, a shooting facility with such a large shooting space is required. However, if the present invention is used, since the multi-view image information for only the coffee cup and the multi-view image information for only the apple are separately created and synthesized, they are taken. Even in this case, it is sufficient to prepare a relatively small-scale photography facility.

  Therefore, for example, even when presenting a stereoscopic image of a group of all 40 students in a class consisting of 40 students in a class, a live-action shooting with each student as a subject is performed using a relatively small shooting facility. If a total of 40 sets of multi-viewpoint image information is created and these 40 sets of multi-viewpoint image information are synthesized using the present invention, a composition that can present a stereoscopic image of the aggregate of all students Multi-viewpoint image information can be obtained.

  Hereinafter, one embodiment of a method for synthesizing multi-viewpoint image information according to the present invention will be described with reference to multi-viewpoint image information E (α) for a subject α made of a coffee cup and multi-viewpoint image information E ( A specific example of synthesizing β) will be described with reference to FIG.

  The first multi-view image information E (α) shown in the upper part of FIG. 11 is the same as the multi-view image information E shown in the lower part of FIG. Consists of two-dimensional images Pα1 to Pα5 obtained by parallel projection on D5. Here, the two-dimensional images Pα1 to Pα5 are the same images as the two-dimensional images P1 to P5 shown in FIG. 1, but are denoted by symbols Pα1 to Pα5 to indicate that they are images about the subject α. . Here, for convenience of describing the composition processing for two sets of images, the background portion around the projection image of the subject α made of coffee cups is shown as a hatched area.

  On the other hand, the second multi-viewpoint image information E (β) shown in the middle of FIG. 11 is composed of two-dimensional images Pβ1 to Pβ5 obtained by projecting the subject β made of apples in parallel in five observation directions D1 to D5. Is done. Again, the background portion around the projected image of the subject β made of apples is hatched.

  Here, the positional relationship between the subject α and the projection plane Sα for obtaining the first multi-viewpoint image information E (α), and the subject β and the projection plane for obtaining the second multi-viewpoint image information E (β). The positional relationship with Sβ is not necessarily the same. Of course, the distance between the projection surface Sα and the subject α and the distance between the projection surface Sβ and the subject β need not be the same. Further, the sizes of the projection surfaces Sα and Sβ do not have to be the same. However, the observation directions D1 to D5 with respect to the projection surfaces Sα and Sβ are common to both. That is, in FIG. 1, as long as the positional relationship between the projection surface S and the observation directions D1 to D5 is fixed, the positions at which the subject α and the subject β are arranged may be arbitrary.

  In the lower part of FIG. 11, the third multi-view image information E (γ) obtained by combining the first multi-view image information E (α) and the second multi-view image information E (β). It is shown. The third multi-viewpoint image information E (γ) is a two-dimensional image showing a state in which the subject α made of coffee cup and the subject β made of apple are arranged from the above five observation directions D1 to D5. The images Pγ1 to Pγ5 are configured. The third multi-viewpoint image information E (γ) is information obtained by the synthesis process, and can be used for presenting a state in which the subjects α and β are juxtaposed as a stereoscopic image without a sense of incongruity.

  Specifically, for example, by using a method similar to the method described with reference to FIG. 5, the two-dimensional images Pγ1 to Pγ5 are printed on the recording medium M, and the lenticular lens L is arranged thereon, whereby the subject α , Β can be created, and a stereoscopic image recording medium capable of presenting a stereoscopic image in a state where β is juxtaposed can be created. That is, by the function of the lenticular lens L, a two-dimensional image Pγ1 is presented in the observation direction D1, a two-dimensional image Pγ2 is presented in the observation direction D2, and a two-dimensional image Pγ5 is presented in the observation direction D5. Thus, a state where the coffee cup and the apple are juxtaposed (in the illustrated example, the state where the apple is arranged behind the coffee cup) is presented as a stereoscopic image.

  In order to generate the third multi-view image information E (γ) by synthesizing the first multi-view image information E (α) and the second multi-view image information E (β), a common It can be intuitively understood that two-dimensional images relating to the observation direction may be synthesized. For example, for the observation direction D1, the two-dimensional images Pα1 and Pβ1 are synthesized to generate a new two-dimensional image Pγ1, and for the observation direction D2, the two-dimensional images Pα2 and Pβ2 are synthesized to create a new one. A two-dimensional image Pγ2 is generated, and for the observation direction D5, a new two-dimensional image Pγ5 may be generated by synthesizing the two-dimensional images Pα5 and Pβ5. A white “+” mark and a white arrow in FIG. 11 indicate such a composition procedure.

  In general, two-dimensional images are synthesized by superimposing two images at the same position, and for an area where the images overlap, the image arranged in front is given priority in consideration of the depth of both. Done by the method. FIG. 12 is a plan view showing a general method for obtaining the synthesized image Pγ3 by synthesizing the two-dimensional images Pα3 and Pβ3 shown in FIG. Here, the synthesis is performed on the assumption that the image Pα3 is arranged in front of the image Pβ3. As described above, the hatched area in FIG. 11 is a background area of each two-dimensional image and is an area unrelated to the projected image of the subject. Therefore, this background region is not considered as a synthesis target when a two-dimensional image is synthesized.

  The process of performing image composition while distinguishing the background region is generally known as chroma key composition processing. When shooting a subject, the background color area (area consisting of pixels with a background attribute) in the captured image is obtained by shooting using a screen with a specific background color such as blue or green as the background. ) Can be recognized as the background area. The hatched area in FIG. 11 indicates an area recognized as a background area by such a method. When the two-dimensional images Pα3 and Pβ3 are synthesized as in the example illustrated in FIG. 12, a portion other than the background region of the image Pα3 (portion of the subject α) and a portion other than the background region of the image Pβ3 (portion of the subject β) And a synthesized image Pγ3 may be generated using a portion other than the synthesized subject as a background region.

  The example shown in FIG. 12 is an example in which two two-dimensional images Pα3 and Pβ3 to be synthesized are images of the same size (images having the same pixel arrangement), and the two pixels are exactly The overlapping pixels at the same position correspond one-to-one. Here, the pixel G1 is a pixel in the background area, the pixel G2 is a pixel that forms part of the subject α, the pixel G3 is a pixel in the overlapping region of the subjects α and β (hatched by dots in the figure), and the pixel G4 is a pixel constituting a part of the subject β. In this case, in the composite image Pγ3, the pixel value of the subject α may be adopted as the pixel value of the pixel G2, and the pixel value of the subject β may be adopted as the pixel value of the pixel G4. On the other hand, as the pixel value of the pixel G3 in the area hatched with dots, the pixel value of the subject located in the foreground (subject α in this example) is adopted, and the pixel value of the pixel G1 is the background value. A pixel value indicating an attribute may be employed.

  Of course, the method of synthesizing two two-dimensional images by the method shown in FIG. 12 is an ordinary method, and is widely used in various fields. The inventor of the present application also initially performed processing for generating third multi-viewpoint image information E (γ) by combining two-dimensional images related to a common observation direction using this general combining method. . However, based on the third multi-viewpoint image information E (γ) obtained by such a synthesis method, a stereoscopic image recording medium using a lenticular lens is created by a method as shown in FIG. Unnaturalness was felt in the stereoscopic images. Specifically, the depth relationship between the subjects α and β, in particular, the depth relationship of the contour portion, resulted in a sense of incongruity.

  The inventor of the present application believes that the reason why such a sense of incongruity is caused is that the two-dimensional images Pα3 and Pβ3, which are parallel projection images of the subject, are synthesized by a conventional method similar to a normal planar image. . Therefore, when the present inventor tried to synthesize by the following novel method, the third multi-view image information E (γ) capable of presenting a good synthesized stereoscopic image without a sense of incongruity is generated. I was able to. The technical idea that forms the basis of the present invention resides in this novel synthesis method.

  Hereinafter, a method for synthesizing a two-dimensional image according to the present invention will be described with reference to FIG. FIG. 13A is a top view showing the principle of creating a composite two-dimensional image Pγ1 by combining the two-dimensional image Pα1 and the two-dimensional image Pβ1 on the composite surface W. Here, the images Pα1, Pβ1, and Pγ1 correspond to the three images shown in the column on the left side of FIG. 11, and in the top view of FIG. 13, all are flat images perpendicular to the paper surface of the drawing. Become. That is, the two-dimensional image Pα1 is a projection image obtained by parallel projection of the subject α in the observation direction D1, and the two-dimensional image Pβ1 is a projection obtained by parallel projection of the subject β in the observation direction D1. It is an image. Note that the composite two-dimensional image Pγ1 is an image formed on the composite surface W, but is shown by a thick line in the figure for convenience. Since the images Pα1, Pβ1 are all arranged at positions parallel to the composite surface W, the images Pα1, Pβ1, Pγ1 are all images formed on a plane parallel to each other. Become.

  Here, as shown in the figure, a depth value Z is defined, and a mutual depth relationship is defined for the images Pα1, Pβ1, and Pγ1. Specifically, it is assumed that the viewpoint is located at the maximum value on the negative side of the depth value Z, and that the smaller the depth value Z is, the closer the position is, and the larger the depth value Z is, the deeper the position is located. I will show that. In the case of the illustrated example, the position of the composite plane W (the position of the composite two-dimensional image Pγ1) is the depth value Z = 0, the position of the image Pα1 is the depth value Z = Zα, and the position of the image Pβ1 is the depth value Z = Zβ. It is said. Here, the depth value Zα takes a negative value, and the depth value Zβ takes a positive value. Therefore, the image Pα1 is disposed on the front side and the image Pβ1 is disposed on the back side.

  The depth values Zα and Zβ do not necessarily need to be set to one positive and the other negative, and both may be set to positive values or both may be set to negative values. Alternatively, one may be set to zero. In short, different depth values Z may be set for the images Pα1 and Pβ1. Of course, the absolute values of the depth values Zα and Zβ can also be set to arbitrary values. These depth values Zα and Zβ can be set regardless of the distance between the subjects α and β and the projection surfaces Sα and Sβ when the parallel projection images constituting the two-dimensional images Pα1 and Pβ1 are created. In other words, the depth values Zα and Zβ do not directly indicate the arrangement of the subjects α and β but function as parameters indicating the arrangement of the two-dimensional images Pα1 and Pβ1.

  Thus, the vertical positions (distances to the composite surface W) of the two-dimensional images Pα1 and Pβ1 are determined by the depth values Zα and Zβ set as arbitrary values. Similarly, the position in the left-right direction in the figure of the two-dimensional images Pα1 and Pβ1 and the position in the direction perpendicular to the drawing sheet (position in the direction orthogonal to the depth axis Z) can be arbitrarily set.

  FIG. 11 shows an example in which the two-dimensional image groups Pα1 to Pα5 and the two-dimensional image groups Pβ1 to Pβ5 use images of exactly the same size (images having the same pixel arrangement). The group and the latter image group may be different image groups. That is, the five two-dimensional images Pα1 to Pα5 are parallel projection images obtained by parallel projecting the same subject α onto the same projection surface Sα, and the size thereof is determined by the size of the projection surface Sα. Similarly, the five two-dimensional images Pβ1 to Pβ5 are parallel projection images obtained by parallel projecting the same subject β onto the same projection surface Sβ, and the size thereof is determined by the size of the projection surface Sβ. Accordingly, the five two-dimensional images Pα1 to Pα5 are images of the same size, and the five two-dimensional images Pβ1 to Pβ5 are also images of the same size.

  However, if setting is made such that the size of the projection surface Sα and the size of the projection surface Sβ are different, the sizes of the two-dimensional image groups Pα1 to Pα5 and the sizes of the two-dimensional image groups Pβ1 to Pβ5 are different. The synthesis target by the synthesis method described here may be two-dimensional images having different sizes as described above. Therefore, FIG. 13A illustrates a case where the two-dimensional image Pα1 and the two-dimensional image Pβ1 are images of different sizes.

  Further, if necessary, a process of enlarging or reducing the two-dimensional images Pα1 and Pβ1 may be performed before synthesis. In order to enlarge an image, a process of interpolating pixels is necessary, and in order to reduce an image, a process of thinning out pixels is necessary, so that the image size is naturally changed. Therefore, when different scaling processes are performed on the two-dimensional images Pα1 and Pβ1, even if the original size is the same, the images will have different sizes after the scaling process. There will be no hindrance in the implementation. Of course, if necessary, trimming processing may be performed on the two-dimensional images Pα1 and Pβ1.

  In short, the essential requirement for carrying out the synthesis method described here is that the two-dimensional images Pα1 and Pβ1 are parallel projection images obtained by parallel projection of the respective subjects in the same observation direction D1, and the two-dimensional images. The only difference is that the images Pα1 and Pβ1 are arranged at predetermined positions so as to be parallel to the composite surface W. If these essential requirements are satisfied, the positions of both images can be set arbitrarily, and the sizes of both images may be different.

  If the arrangement shown in FIG. 13A is performed, the process of generating the synthesized two-dimensional image Pγ1 on the synthesized surface W is very simple. That is, a parallel projection image obtained by projecting the two-dimensional image Pα1 onto the composite plane W in a direction parallel to the observation direction D1 and a parallel projection obtained by projecting the two-dimensional image Pβ1 into a direction parallel to the observation direction D1. The combined two-dimensional image Pγ1 may be created by forming a projected image and combining these parallel projected images in consideration of the positional relationship based on the depth values Zα and Zβ. Dashed arrows in the figure indicate the projection direction of each two-dimensional image.

  When such projection is performed, a parallel projection image of the image Pα1 and a parallel projection image of the image Pβ1 are formed on the composite surface W. Therefore, these two parallel projection images are formed on the composite surface W. May be combined in consideration of the positional relationship based on the depth values Zα and Zβ. Here, as described above, since an image having a small depth value Z is set to the front, the image Pα1 is an image positioned closer to the front. Therefore, the image Pα1 positioned closer to the front is set to the back. Combining is performed with priority over the positioned image Pβ1.

  Specifically, for the region where the parallel projection image of the image Pα1 and the parallel projection image of the image Pβ1 overlap on the composite plane W, the pixel value of the parallel projection image of the image Pα1 is adopted, and the parallel projection of the image Pα1 is performed. For the region where only the image is formed, the pixel value of the parallel projection image of the image Pα1 is adopted, and for the region where only the parallel projection image of the image Pβ1 is formed, the pixel value of the parallel projection image of the image Pβ1 is adopted. That's fine. Such a synthesis process is as described with reference to FIG.

  In the example shown in FIG. 13A, the parallel projection image of the image Pβ1 formed on the composite plane W is included in the parallel projection image of the image Pα1. In such a case, since the image Pα1 positioned in front is given priority, the pixel value of the parallel projection image of the image Pα1 is adopted as the composite two-dimensional image Pγ1 formed on the composite surface W. It will be. Accordingly, in FIG. 13 (a), the combined two-dimensional image Pγ1 indicated by the black thick line is constituted by a parallel projection image of the image Pα1.

  On the other hand, in the example shown in FIG. 13B, the two-dimensional image Pα3 obtained by parallel projection of the subject α in the observation direction D3 and the two-dimensional image obtained by parallel projection of the subject β in the observation direction D3. FIG. 6 is a top view showing the principle of creating a synthesized two-dimensional image Pγ3 by synthesizing Pβ3 on a synthesis surface W. In this example, since the projection onto the composite surface W is performed in a direction parallel to the observation direction D3, the parallel projection image of the image Pα3 and the parallel projection image of the image Pβ3 formed on the composite surface W are only a part. Overlapping form.

  Therefore, with respect to the overlapping area, the image Pα3 positioned in front is prioritized and the pixel value of the parallel projection image of the image Pα3 is adopted, and the area where only the parallel projection image of the image Pα3 is formed is the image Pα3. For the region where only the parallel projection image of the image Pβ3 is formed, the pixel value of the parallel projection image of the image Pβ3 is employed. In FIG. 13 (b), of the synthesized two-dimensional image Pγ3 obtained on the synthesized surface W, the portion adopting the pixel value of the parallel projected image of the image Pα3 is indicated by a thick black line, and the parallel projected image of the image Pβ3 is shown. The portion where the pixel value is adopted is indicated by a white thick line.

  In the example shown in FIG. 13C, a two-dimensional image Pα5 obtained by parallel projection of the subject α in the observation direction D5 and a two-dimensional image obtained by parallel projection of the subject β in the observation direction D5. It is a top view showing the principle of creating a synthesized two-dimensional image Pγ5 by synthesizing Pβ5 on the synthesis surface W. In this example, since the projection onto the composite surface W is performed in a direction parallel to the observation direction D5, the parallel projection image of the image Pα5 and the parallel projection image of the image Pβ5 formed on the composite surface W overlap. It becomes a form that does not occur.

  Therefore, as the composite two-dimensional image Pγ5 obtained on the composite plane W, the pixel value of the parallel projection image of the image Pα5 is adopted for the region where the parallel projection image of the image Pα5 is formed, and the parallel projection image of the image Pβ5 is obtained. For the area where only the image is formed, the pixel value of the parallel projection image of the image Pβ5 is adopted. In FIG. 13 (c), of the composite two-dimensional image Pγ5 obtained on the composite surface W, the portion adopting the pixel value of the parallel projection image of the image Pα5 is indicated by a thick black line, and the parallel projection image of the image Pβ5 is shown. The portion where the pixel value is adopted is indicated by a white thick line.

  It should be noted that an area where no parallel projection image is formed on the composite surface W (for example, an area sandwiched between a black thick line area and a white thick line area in FIG. 13C) should be recorded. Since the image does not exist, a pixel value indicating that no valid image exists or a pixel value indicating the background attribute may be given.

  Further, the examples shown in FIGS. 13A, 13B, and 13C are examples in which the entire area of the two-dimensional image to be synthesized is parallel projected onto the synthesis plane W. However, when chroma key synthesis is performed, In the two-dimensional image to be synthesized, an area composed of pixels having pixel values indicating the background attribute is not formed, and an area where no parallel projection image is formed on the synthesis plane W If it exists, a synthesized two-dimensional image may be created by giving a pixel value indicating a background attribute to the pixels in the region.

  As described above, with reference to FIG. 13, an example (FIG. 13A) in which the two-dimensional images Pα1 and Pβ1 in the observation direction D1 are combined to create the combined two-dimensional image Pγ1, and the two-dimensional image Pα3 in the observation direction D3. , Pβ3 are combined to create a combined two-dimensional image Pγ3 (FIG. 13B), and two-dimensional images Pα5 and Pβ5 in the observation direction D5 are combined to generate a combined two-dimensional image Pγ5 (FIG. 13). (c)) has been described, but an example of creating a synthesized two-dimensional image Pγ2 by synthesizing the two-dimensional images Pα2 and Pβ2 in the observation direction D2, or by synthesizing the two-dimensional images Pα4 and Pβ4 in the observation direction D4 The example of creating the synthesized two-dimensional image Pγ4 is exactly the same.

  In short, in general terms, the i-th two-dimensional image Pαi constituting the first multi-view image information E (α) is parallel to the composite surface W and the distance to the composite surface W is the depth value Zα. The i-th two-dimensional image Pβi that is arranged at such a predetermined position and constitutes the second multi-viewpoint image information E (β) is parallel to the composite surface W and the distance to the composite surface W is the depth value Zβ. The parallel projection image obtained by projecting the two-dimensional image Pαi in a direction parallel to the i-th observation direction Di and the two-dimensional image Pβi on the composite plane W and the i-th observation A parallel projection image obtained by projecting in a direction parallel to the direction Di, and combining the parallel projection images in consideration of the positional relationship based on the depth values Zα and Zβ. An image Pγi may be created.

  Here, the position of the i-th two-dimensional image Pαi with respect to the composite surface W is set to be the same position in common for i = 1 to n, and the position of the i-th two-dimensional image Pβi with respect to the composite surface W is also i = 1 to n are set to the same common position. Also in the example illustrated in FIG. 13, the positions of the images Pα1, Pα3, and Pα5 with respect to the composite surface W are common, and the positions of the images Pβ1, Pβ3, and Pβ5 with respect to the composite surface W are common. Further, here, for convenience of explanation, an example is shown in which n = 5 is set and two-dimensional images are synthesized for each of the five observation directions D1 to D5. However, in practice, n is a larger value. By setting to, it is preferable that a more detailed stereoscopic image can be presented. For example, when n = 70 is set, two-dimensional images are synthesized in the same manner as the method shown in FIG. 13 for the 70 observation directions D1 to D70, respectively, and a total of 70 synthesized two-dimensional images Pγ1 to Pγ70. Thus, the third multi-viewpoint image information E (γ) is created.

<<< §4. Procedure for synthesizing multi-viewpoint image information according to the present invention >>
Here, based on the principle described in §3, a basic procedure for synthesizing two sets of multi-viewpoint image information will be described with reference to the flowchart of FIG. The basic procedure shown in this flowchart is a procedure of a method for synthesizing multi-viewpoint image information composed of a plurality of n two-dimensional images obtained by parallel projecting subjects in different directions, and is a procedure executed by a computer. Therefore, in practice, each process shown in the figure is performed by incorporating a dedicated program for executing such a procedure into the computer and executing the program.

  First, in step S1, the computer records first multi-view image information E (α) in which the first subject α is recorded and second multi-view image information E (β in which the second subject β is recorded. ) And the image information input step are executed. Here, the first multi-viewpoint image information E (α) is configured by an aggregate of a plurality of n two-dimensional images Pα1 to Pαn about the first subject α, for example, as shown in the upper part of FIG. The second multi-viewpoint image information E (β) is constituted by an aggregate of a plurality of n two-dimensional images Pβ1 to Pβn for the second subject β, for example, as shown in the middle part of FIG.

  Here, in the i-th (1 ≦ i ≦ n) two-dimensional image Pαi constituting the first multi-viewpoint image information E (α), the first subject α is placed on the predetermined projection plane Sα. The i-th (1 ≦ i ≦ n) two-dimensional image Pβi forming a parallel projection image obtained by projecting in a direction parallel to the observation direction Di of the second multi-viewpoint image information E (β) is Then, a parallel projection image obtained by projecting the second subject β onto the predetermined projection surface Sβ in a direction parallel to the i-th observation direction Di is formed.

  In the next step S2, the computer determines a composition condition for arranging the first multi-view image information E (α) and the second multi-view image information E (β) (as shown in FIG. 13). The synthesis condition setting stage for setting the conditions for the above is executed. Actually, a predetermined synthesis condition is set based on a setting instruction given to the computer by the operator. The composition condition is that the position of the two-dimensional image constituting the first multi-view image information E (α) with respect to the composition surface W and the composition surface W of the two-dimensional image constituting the second multi-view image information E (β). The depth value Zα for the two-dimensional image constituting the first multi-view image information E (α) and the two-dimensional constituting the second multi-view image information E (β). A depth value Zβ for the image.

  If necessary, in addition to the depth values Zα and Zβ, a condition for determining a position in a direction orthogonal to the depth axis Z of the two-dimensional image may be set as a synthesis condition. For example, in the example shown in FIG. 13, if the conditions for determining the positions of the images Pα1 to Pα5 and the images Pβ1 to Pβ5 in the horizontal direction in the figure are set, when the subjects α and β are combined, they are orthogonal to the depth. It is possible to arbitrarily adjust the positional relationship with respect to the direction to be performed. Of course, if it is not necessary to adjust the positional relationship, it is necessary to always make arrangements such as aligning the upper left corner of the image or aligning the center of the image. The condition for determining the position in the direction orthogonal to the depth axis Z is not necessary. Of course, when a scaling process such as enlargement / reduction is performed on each two-dimensional image at the time of synthesis, the magnification relating to the scaling process may be set as a synthesis condition.

  In subsequent steps S3 to S6, the computer performs the i-th (1 ≦ i ≦ n) two-dimensional image Pαi constituting the first multi-view image information E (α) and the second multi-view image information E ( The process of creating the i-th synthesized two-dimensional image Pγi on the synthesis plane W by synthesizing the i-th (1 ≦ i ≦ n) two-dimensional image Pβi constituting β), i = 1 An image composition step is repeatedly executed for .about.n. That is, first, in step S3, the value of the parameter i is set to an initial value 1, and in step S4, the i-th two-dimensional image Pαi and the i-th two-dimensional image Pβi are combined to generate a combined plane W. A process for creating the i-th synthesized two-dimensional image Pγi is executed on the top. In step S5, it is determined whether or not the parameter i has reached n. Until the parameter i reaches n, the parameter i is updated to +1 in step S6, and the process in step S4 is repeatedly executed. Will be.

  The specific contents of the image composition stage performed in step S4 are as described in section 3 with reference to FIG. That is, the i-th two-dimensional image Pαi constituting the first multi-viewpoint image information E (α) is arranged at a predetermined position parallel to the composite surface W and having a depth value Zα at a distance to the composite surface W. The i-th two-dimensional image Pβi constituting the second multi-viewpoint image information E (β) is arranged at a predetermined position parallel to the composite surface W and the distance to the composite surface W is the depth value Zβ. A parallel projection image obtained by projecting the two-dimensional image Pαi in a direction parallel to the i-th observation direction Di and a two-dimensional image Pβi in a direction parallel to the i-th observation direction Di on the composite surface W. Forming a i-th synthesized two-dimensional image Pγi by forming a parallel projection image obtained by projection and synthesizing these parallel projection images in consideration of the positional relationship based on the depth values Zα and Zβ. Done.

  In the case of the embodiment described in §3, in the synthesis condition setting stage in step S2, as values of the depth values Zα and Zβ, a positive value, a negative value, or zero is set for one, and a positive value for the other Alternatively, a negative value is set, and in the image composition stage of step S4, composition is performed such that a parallel projection image of an image with a small depth value is observed in front. More specifically, in the image synthesizing stage in step S4, the formed parallel projection image is recorded as it is on the synthesis surface W where the formed parallel projection images do not overlap, and the formed parallel projection image is recorded as it is. For the region where the two overlap, the composition may be performed by recording only the parallel projection image observed in front.

  When performing chroma key composition, n two-dimensional images Pα1 to Pαn constituting the first multi-view image information E (α) and n sheets constituting the second multi-view image information E (β). At least one of the two-dimensional images Pβ1 to Pβn is set to an image including a pixel having a pixel value indicating the background attribute. Then, in the image composition stage of step S4, a parallel projection image is not formed for an area composed of pixels having a pixel value indicating the background attribute, and there is an area where no parallel projection image is formed on the synthesis plane W. In this case, a synthesized two-dimensional image may be created by giving a pixel value indicating a background attribute for the pixels in the region.

  In the final step S7, the multi-viewpoint image information constituted by n synthesized two-dimensional images Pγ1 to Pγn obtained through the image synthesis step repeated n times as steps S3 to S6 on the computer, An image information output stage for outputting the synthesized multi-viewpoint image information E (γ) is executed. In practice, it is preferable that information indicating the corresponding observation directions D1 to Dn is added to the synthesized two-dimensional images Pγ1 to Pγn for output. Each of the synthesized two-dimensional images Pγ1 to Pγn itself is a simple planar image, but the multi-viewpoint image information E (γ) is obtained by converting these images Pγ1 to Pγn into a direction parallel to the observation directions D1 to Dn. The information is bundled as a collection of projection images formed on a common projection plane by performing parallel projection on the image, and each image Pγ1 to Pγn is associated with a predetermined observation direction D1 to Dn, respectively. is important.

  As described above, the multi-viewpoint image information E (α), E (β), and E (γ) in the present invention is an aggregate of n two-dimensional images each associated with a predetermined observation direction D1 to Dn. It has an important meaning in terms. Because the observation directions D1 to Dn are associated with each other in this way, by using the method illustrated in FIG. 5 or FIG. 8, it is possible to present each two-dimensional image in the corresponding observation direction. Thus, it is possible to present the subject recorded on the medium as a stereoscopic image.

  The n viewing directions can be defined by n sets of direction vectors. For example, the embodiment shown in FIGS. 1 to 5 defines a plurality of n sets of direction vectors passing through the reference point Q on a plane including the predetermined reference point Q (in the example shown, n = 5 is set as an example). ), An example using multi-view image information in which the direction of the n sets of direction vectors is an individual observation direction as the first multi-view image information E (α) and the second multi-view image information E (β). It is. In this case, each observation direction can be indicated by a direction vector D (φ) having only the azimuth angle φ as a parameter.

  On the other hand, the embodiment shown in FIGS. 6 to 8 is a vector passing through the origin O of the XYZ three-dimensional coordinate system, and the azimuth angle φ formed by the orthogonal projection image on the XY plane and the Y axis. The direction vector D (φ, θ) specified by the elevation angle θ with respect to the XY plane is defined as a × b by changing θ to a and φ to b (in the example shown, a = 3, b = 5), the first multi-view image information E (α) and the second multi-view image information E (β) are a total of n (where n = a × b). This is an example using multi-viewpoint image information in which the direction of a direction vector is an individual observation direction.

<<< §5. Method for Presenting Synthetic Stereo Image According to the Present Invention >>
As already described in §1, the multi-viewpoint image information handled in the present application is used for the purpose of presenting a stereoscopic image of a subject. Therefore, here, the procedure of the method for presenting a composite stereoscopic image according to the present invention will be described with reference to the flowchart shown in FIG. This method is a method of presenting a stereoscopic image using the multi-view image information combining method shown in FIG.

  The procedure of the stereoscopic image presenting method shown in FIG. 15 includes a multi-view image information preparation stage composed of steps S11 to S14, a multi-view image information composition stage composed of step S15, and an image composed of step S16. And a presentation stage. Here, in the multi-view image information preparation stage (steps S11 to S14), the n two-dimensional images Pα1 to Pαn and the second multi-view image information E (the first multi-view image information E (α)). In this step, n two-dimensional images Pβ1 to Pβn constituting β) are prepared, and the multi-view image information synthesis step (step S15) includes the first multi-view image information E (α) and the second For the multi-view image information E (β), the multi-view image information E (γ) synthesized by executing the multi-view image information combining method constituted by steps S1 to S7 in FIG. In the image creation stage (step S16), the n-th synthesized two-dimensional images Pγ1 to Pγn constituting the multi-viewpoint image information E (γ) are converted into the i-th (1 ≦ i ≦ n). The synthesized two-dimensional image Pγi is the i-th observation method A step of presenting in a manner as presented in Di.

  Steps S11 to S14 shown in FIG. 15 show a procedure in the case of performing real-time shooting on the subject as the multi-viewpoint image information preparation stage. First, in the first shooting stage of step S11, the first subject α (in the case of the above-described embodiment, coffee cups) is shot in real time from a plurality of ζ directions, so that a total of ζ first shots are taken. A process for obtaining a group of photographed images is performed, and in the subsequent second photographing stage of step S12, the second subject β (in the case of the above-described embodiment, apples) is photographed in real time from a plurality of η directions. A process of obtaining a total of η second actual photographed image groups is performed.

  Any photographing stage can be carried out by using photographing equipment as shown in FIG. Of course, it is sufficient to shoot only the first subject α in the first shooting stage and only the second subject β in the second shooting stage. When chroma key composition is performed, shooting is performed in a state where a screen of a specific color such as blue or green (a color not included in the subject as much as possible) is arranged on the background of the subject.

  Subsequently, in the first multi-viewpoint image information creation stage in step S13, the first subject α is placed on a predetermined projection plane based on the total of ζ first live-action photographed image groups obtained in step S11. The parallel projection images obtained by projecting in the directions parallel to the n observation directions of the first observation direction D1 to the nth observation direction Dn are respectively obtained, and the first multi-viewpoint image information E (α) is obtained. A process of creating n two-dimensional images Pα1 to Pαn constituting the image is performed. Further, in the second multi-viewpoint image information creation stage in step S14, the second subject β is placed on a predetermined projection plane based on the total of η second actual photographed image groups obtained in step SS12. Parallel projection images obtained by projecting in the directions parallel to the n observation directions of the first observation direction D1 to the nth observation direction Dn are obtained, respectively, and second multi-viewpoint image information E (β) is obtained. A process of creating n two-dimensional images Pβ1 to Pβn to be configured is performed. Such processing can be carried out by a known method for converting a large number of photographed images as described in section 2 with reference to FIG.

  In the multi-viewpoint image information synthesis step in step S15, the steps S1 to S7 in FIG. 14 are performed on the first multi-viewpoint image information E (α) and the second multi-viewpoint image information E (β) thus prepared. Is applied to create multi-viewpoint image information E (γ), the details of which are as already described in Section 4.

  In the final image presentation stage of step S16, the n composite two-dimensional images Pγ1 to Pγn constituting the multi-viewpoint image information E (γ) obtained in this way are converted to the i-th (1 ≦ i ≦ n) by some method. The composite two-dimensional image Pγi in (2) may be presented mainly in the i-th observation direction Di. As a specific presentation method, as described in the embodiment of FIGS. 5 and 8, a stereoscopic image recording medium may be created and a stereoscopic image may be presented using this stereoscopic image recording medium.

  Therefore, in the stereoscopic image presenting method shown in FIG. 15, instead of the image presenting step of step S16, a medium recording step of recording a composite two-dimensional image on the recording medium M by a method such as printing, and the recording medium M If the optical element arranging step of arranging the optical elements is performed, the procedure shown in FIG. 15 is a procedure showing a method for creating a stereoscopic image recording medium using a method for synthesizing multi-viewpoint image information.

  In this case, in the medium recording stage, the n composite two-dimensional images Pγ1 to Pγn constituting the multi-viewpoint image information E (γ) are divided into a plurality of m partial images, and the recording surface on the recording medium M is divided. A process of dividing a plurality of m sections K1 to Km and recording the i-th partial images of the n composite two-dimensional images Pγ1 to Pγn in the i-th section Ki on the recording medium M (for example, A process of printing a partial image on the recording medium M of FIG. 5 or the recording medium M of FIG. 8 may be performed. In the optical element arrangement stage, optical elements are arranged on the individual sections on the recording medium M so that the partial image of the i-th synthesized two-dimensional image Pγi is presented mainly in the i-th observation direction Di. (For example, a process of arranging a lenticular lens on the recording medium M in FIG. 5 or a process of arranging a microlens array on the recording medium M in FIG. 8) may be performed.

<<< §6. Multi-viewpoint image information synthesizing apparatus according to the present invention >>>
Next, the configuration of the multi-viewpoint image information synthesizing apparatus according to the present invention will be described with reference to the block diagram of FIG. This synthesizing apparatus is an apparatus for synthesizing multi-viewpoint image information composed of a plurality of n two-dimensional images obtained by projecting subjects in parallel in different directions. The storage unit 20, the image composition unit 30, and the composite image storage unit 40 are configured. Each of these components is actually constructed by incorporating a dedicated program into the computer.

  Here, the image information storage unit 10 includes the first multi-view image information E (α) in which the first subject α is recorded and the second multi-view image information E (in which the second subject β is recorded. β) and the function of storing. As already described, the first multi-viewpoint image information E (α) is composed of an aggregate of n two-dimensional images. Here, the i-th (1 ≦ i ≦ n) two-dimensional image Pαi is obtained by projecting the first subject α onto a predetermined projection plane in a direction parallel to the i-th observation direction Di. Make a projected image. Similarly, the second multi-viewpoint image information E (β) is also composed of an aggregate of n two-dimensional images. Here, the i-th (1 ≦ i ≦ n) two-dimensional image Pβi is obtained by projecting the second subject β onto a predetermined projection plane in a direction parallel to the i-th observation direction Di. Make a projected image.

  Eventually, the image information storage unit 10 stores image data of n two-dimensional images constituting the first multi-view image information E (α) and n constituting the second multi-view image information E (β). The image data of the two two-dimensional images are stored. These image data may be in any format. For example, in the case of a general color image, the color components of the three primary colors R, G, and B for a large number of pixels arranged vertically and horizontally are shown. It can be constituted by aggregated data of pixel values.

  On the other hand, the synthesis condition storage unit 20 stores a synthesis condition for synthesizing the first multi-view image information E (α) and the second multi-view image information E (β) based on an operator setting instruction. A depth value Zα indicating the position of the two-dimensional image constituting the first multi-view image information E (α) with respect to the composite plane W and the second multi-view image information E (β). And a depth value Zβ indicating the position of the two-dimensional image to the composite plane W is stored.

  The image composition unit 30 includes the i-th (1 ≦ i ≦ n) two-dimensional image Pαi constituting the first multi-viewpoint image information E (α) stored in the image information storage unit 10 and image information. The i-th two-dimensional image Pβi constituting the second multi-viewpoint image information E (β) stored in the storage unit 10 is synthesized, and the i-th synthesized two-dimensional image on the synthesis plane W. It fulfills the function of repeatedly executing the process of creating the image Pγi for i = 1 to n.

  More specifically, as described in detail in §4, the i-th two-dimensional image Pαi constituting the first multi-viewpoint image information E (α) stored in the image information storage unit 10 is synthesized. The i-th image forming the second multi-viewpoint image information E (β) stored in the image information storage unit 10 is arranged at a predetermined position parallel to the surface W and the distance to the composite surface W is the depth value Zα. The second two-dimensional image Pβi is arranged at a predetermined position parallel to the composite surface W and the distance to the composite surface W is the depth value Zβ, and the two-dimensional image Pαi is placed on the composite surface W in the i-th observation direction. A parallel projection image obtained by projecting in a direction parallel to Di and a parallel projection image obtained by projecting the two-dimensional image Pβi in a direction parallel to the i-th observation direction Di are formed, and these parallel projections are formed. The image is synthesized by considering the positional relationship based on the depth values Zα and Zβ. The i-th synthesized two-dimensional image Pγi is generated and stored in the synthesized image storage unit 40.

  The composite image storage unit 40 stores the multi-viewpoint image information composed of the n composite two-dimensional images Pγ1 to Pγn thus created by the image composition unit 30 as the composite multi-viewpoint image information E (γ). The multi-viewpoint image information E (γ) is output to the outside as necessary.

<<< §7. Modification of the present invention >>
Here, some modified examples of the embodiments described so far will be described.

(1) Subject Variation In the embodiments described so far, an example in which an actual object having a three-dimensional shape is used as a subject, such as a coffee cup as the first subject α and an apple as the second subject β. However, the subject handled in the present invention is not necessarily a so-called solid body having a three-dimensional shape. For example, strictly speaking, objects such as stamps and postcards are objects with a three-dimensional shape, but for ordinary observers, the thickness portion has no substantial meaning, and only a two-dimensional shape. It is grasped as an object. Even such an object having only a two-dimensional shape can be used as a subject in the present invention. In this case, of course, a three-dimensional effect cannot be generated for a stamp or a picture postcard, but it can be presented as a three-dimensional image in the sense of a two-dimensional object arranged in a three-dimensional space.

  Further, as described in §2, the subject handled in the present invention is not necessarily a real object, and a virtual CG image created using a computer can be used as a subject (original image). is there. When such a CG image is used as a subject, the subject α and the projection plane S shown in FIG. 1 are defined in a virtual space on the computer, and a two-dimensional image P1 as shown in the lower part of FIG. ~ P5 can be obtained. Therefore, it is not necessary to perform the real-time photographing by the method as shown in FIG.

  Of course, the subject (original image) made of a CG image is not necessarily a three-dimensional stereoscopic image, and may be a two-dimensional planar image. Further, the two subjects to be synthesized may both be real objects, both may be CG images, or may be a mixture of both.

(2) Multi-viewpoint image information composed of the same common image In the present invention, as described above, the first multi-viewpoint image information E (α) is composed of a plurality of n two-dimensional images Pα1 to Pαn, The multi-viewpoint image information E (β) is similarly composed of a plurality of n two-dimensional images Pβ1 to Pβn, and the i-th (1 ≦ i ≦ n) two-dimensional images Pαi and Pβi are combined to generate a combined two-dimensional image Pγi. The operation of creating is repeated for i = 1 to n.

  Here, the n two-dimensional images Pα1 to Pαn are generally different two-dimensional images. Similarly, the n two-dimensional images Pβ1 to Pβn are generally different two-dimensional images. For example, in the case of the example shown in FIG. 11, the five two-dimensional images Pα1 to Pα5 are different from each other, and the five two-dimensional images Pβ1 to Pβ5 are also different from each other. This is because these two-dimensional images are constituted by parallel projection images obtained by parallel projecting the subjects α and β on the projection plane S in five directions.

  However, the n two-dimensional images constituting the multi-viewpoint image information handled by the present invention are not necessarily different from each other, and may be the same common image. In general, when a certain subject is projected in different directions, the obtained projection images are often different from each other. Accordingly, the n two-dimensional images constituting the multi-viewpoint image information are usually different from each other. However, under special conditions, the multi-viewpoint image information may be composed of n identical common images.

  For example, when a two-dimensional image (a real object such as a postcard or a CG image) may be prepared as a subject and the subject is arranged on the projection plane S, the projection image of the subject on the projection plane S is Regardless of the projection direction, the same common image as the subject is obtained. Therefore, in this case, the n two-dimensional images constituting the multi-viewpoint image information are common images on the subject, and the same common image is observed regardless of the observation direction. It will be. This can be easily understood by considering a state in which a picture postcard is arranged on the projection plane S in FIG.

  Therefore, when the multi-view image information combining method according to the present invention is implemented, n two-dimensional images Pα1 to Pαn or second multi-view image information E constituting the first multi-view image information E (α). The n two-dimensional images Pβ1 to Pβn constituting (β) can be composed of the same common image. In FIG. 17, n two-dimensional images Pα1 to Pαn constituting the first multi-view image information E (α) are configured by different images as in the previous embodiments, and the second multi-view image In this example, n pieces of two-dimensional images Pβ1 to Pβn constituting information E (β) are configured by the same common image.

  That is, in the example shown in FIG. 17, the first multi-viewpoint image information E (α) is composed of n different two-dimensional images obtained by projecting a subject made of a coffee cup in n directions. However, the second multi-viewpoint image information E (β) is substantially composed of one common image on which a landscape is drawn. Of course, when the multi-view image information combining method according to the present invention (the procedure shown in the flowchart of FIG. 14) is executed, the first multi-view image information E (α) is composed of n two-dimensional images. The second multi-view image information E (β) is also composed of n two-dimensional images, and it is necessary to perform a process of synthesizing the i-th images. Therefore, in the example shown in FIG. 17 as well, for convenience, the second multi-viewpoint image information E (β) is handled as being constituted by n two-dimensional images. The entity of one two-dimensional image is one common image on which a landscape is drawn.

  In the example shown in FIG. 17, the two-dimensional image constituting the first multi-view image information E (α) is arranged in front of the two-dimensional image constituting the second multi-view image information E (β). If composition conditions are set (for example, if the former depth value Zα is set to a negative value and the latter depth value Zβ is set to a positive value or zero), coffee is placed in front of the background image that is the scenery. It becomes possible to present a composite stereoscopic image in which the cup is placed. In this case, the coffee cup is presented as a stereoscopic image, but the background scene is simply presented as a flat image. Nevertheless, a sense of depth is created between the coffee cup in the foreground and the scenery in the back, and it is possible to present a composite stereoscopic image without any sense of incongruity.

  In the modified example shown in FIG. 17, since a very general two-dimensional image (for example, a picture of a postcard) can be used as a background image to be a landscape, an arbitrary landscape is provided on the background of the coffee cup that is the main subject. Can be easily created.

  In addition, the modification shown in FIG. 17 is further simplified, and the first multi-view image information E (α) is also a general two-dimensional image (for example, a photograph of a coffee cup) with a coffee cup as a subject. ) May be used as a common image. In this case, the first multi-view image information E (α) is substantially constituted by a two-dimensional image of one coffee cup, and the second multi-view image information E (β) is substantially one sheet. It is composed of two-dimensional images of the scenery. Of course, in this case, in executing the method for synthesizing the multi-viewpoint image information according to the present invention, for convenience, n two-dimensional images constituting the first multi-viewpoint image information E (α) are coffee cup photos. N two-dimensional images constituting the second multi-viewpoint image information E (β) are constituted by one identical common image consisting of a landscape photograph. Will be handled.

  In the stereoscopic image presented based on the third multi-viewpoint image information E (γ) obtained by executing such a synthesis process, both the coffee cup and the scenery are presented as simple planar images. However, there is a sense of depth between the coffee cup in the foreground and the scenery in the back (for example, a signboard with a single coffee cup placed behind the distant view). In that sense, presentation as a stereoscopic image is possible.

(3) Variation of synthetic stereoscopic image presentation method Based on the third multi-viewpoint image information E (γ) obtained by executing the synthetic processing according to the present invention, the synthetic stereoscopic image is actually displayed to the observer. As a specific method to be presented, an example in which a stereoscopic image recording medium in which an optical element such as a lenticular lens or a microlens array is arranged on a recording medium has been described so far has been described. However, the method of presenting a composite stereoscopic image based on the multi-viewpoint image information E (γ) is not limited to the method using such a stereoscopic image recording medium.

  As already described, in the image presentation stage in step S16 in the flowchart of FIG. 15, the i-th two-dimensional image Pγi is mainly presented in the i-th observation direction Di by using some means. Good. For example, it is possible to present a composite stereoscopic image based on the multi-viewpoint image information E (γ) by using a technique such as a parallax barrier method or glassless 3D that is generally used as a stereoscopic image presentation method. It is. Of course, as a medium used for presentation, not only a fixed medium such as a printed matter but also a display screen can be used. For example, it is possible to sequentially display n two-dimensional images on a display screen in a time-sharing manner, and sequentially switch the direction of light from the displayed image toward the eyes of the observer by some optical means. It is.

  In addition, the stereoscopic image presentation method according to the present invention is a technique that not only presents a single stereoscopic image to the observer unilaterally, but also interactively changes the presentation content in response to the operation of the observer. It can also be applied to.

(4) Variation of composition method In the embodiment described so far, as a method of compositing the parallel projection image of the subject α and the parallel projection image of the subject β on the composition plane W, the overlapping area is synthesized. The example of giving priority to the subject in the foreground in consideration of the positional relationship based on the depth values Zα and Zβ set as conditions has been described. However, by setting the transparency to the subject, It is also possible to perform composition so that the subject can be seen through.

  For example, in the example shown in FIG. 12, for the pixels in the overlapped area that is hatched with dots in the figure, the pixel values of the coffee cup image in the foreground are adopted in the previous embodiments. For example, if a setting of 20% transparency is made for the coffee cup, the pixel value of the coffee cup image in the foreground and the pixel value of the apple image in the back are set to 80 for the pixels in the overlapping region. : Pixel values obtained by blending at a ratio of 20 can be adopted. Thereby, it becomes possible to present the three-dimensional image in the state where the apple is arranged behind the translucent coffee cup.

  In the embodiments so far, the example in which two sets of multi-view image information, that is, the multi-view image information E (α) for the subject α and the multi-view image information E (β) for the subject β has been described. However, if the present invention is used, it is of course possible to synthesize three or more sets of multi-viewpoint image information. The basic concept is exactly the same as the method of combining the two sets of multi-viewpoint image information described so far. That is, the i-th two-dimensional images constituting three or more sets of multi-viewpoint image information are arranged at predetermined positions so as to be parallel to the synthesis plane W based on the depth values set as the synthesis conditions, A parallel projection image obtained by projecting each two-dimensional image in a direction parallel to the i-th observation direction Di is formed on the composite surface W, and the parallel projection image is considered in a positional relationship based on each depth value. The process of creating the i-th synthesized two-dimensional image by synthesizing is repeated for i = 1 to n.

<<< §8. Light Field Synthesis Method According to the Present Invention >>
As described in the background art, Non-Patent Document 1 described above uses a 7-dimensional function P called Plenoptic in the form of P (x, y, z, φ, θ, λ, t), “in space” Discloses a theory describing a phenomenon that a light beam having a wavelength λ has passed through a point indicated by the coordinate value (x, y, z) of λ toward a direction indicated by an azimuth angle φ and an elevation angle θ at a time t. In Non-Patent Document 2 mentioned above, the introduction of the concept of “Light Field” is proposed as one form in which such a description format is used for the technique of presenting a stereoscopic image. Patent Documents 3 and 4 listed above disclose a specific method for handling a three-dimensional image using such a “light field”.

  From this point of view, the multi-viewpoint image information according to the present invention is information that can be expressed by the concept of “light field”, and the present invention can also be regarded as a “light field composition method”. is there. Therefore, here, the description will be made assuming that the present invention is a “light field composition method”.

  FIG. 18 is a perspective view and a mathematical diagram showing the concept of a general light field. Now, as shown in the upper perspective view, let the ray vector L pass through an arbitrary point P (x, y) on the XY two-dimensional coordinate plane. Here, if the direction of the light vector L is D, this light vector L is specified by (x, y, D) using the coordinate value (x, y) indicated by XY two-dimensional coordinates and the direction D as parameters. Will be. The direction D can be expressed by an azimuth angle φ and an elevation angle θ as in the example shown in FIG. 6, for example, but can also be expressed by various methods. Therefore, here, a parameter “direction D” is simply used.

  Now, a certain feature value is assigned to the ray vector thus identified, and the feature value is represented by a function F (x, y, D) having three parameters x, y, and D as variables. That is, the function F (x, y, D) passes through the predetermined point P indicated by the coordinate value (x, y), and the feature value given to the specific light vector L facing the direction D. This is a function indicating

  Here, a plurality of points P are defined in the closed region on the XY plane, and for each point P, a plurality of ray vectors L passing through the point P and facing in different directions are defined, and each ray vector is defined. If a specific feature value is assigned to each of L, the closed region in which such a definition is made is a field that falls within the concept of “light field” referred to in the above-mentioned literature.

  Therefore, here, the “light field” is a plane on which a plurality of points P are arranged, and each of the individual points P has a plurality of ray vectors L that pass through the point P and are directed in different directions. , And a plane to which a specific feature value is assigned for each ray vector L. Then, the present invention can be understood as a light field combining method for combining two sets of light fields.

  As the feature value of the light field, the wavelength or luminance value of the light vector can be used, but it has a function of recognizing one pixel constituting the image when the light vector reaches the eyes of the observer. In view of the above, in the context of the present invention, it is most appropriate to use a pixel value as a feature value.

  Thus, if the method for synthesizing multi-viewpoint image information described so far is regarded as a light field synthesis method, the method includes four steps: a light field input stage, a synthesis condition setting stage, a light field synthesis stage, and a light field output stage. It consists of stages. Actually, each of these steps is executed by a function of a dedicated program incorporated in the computer.

  First, in the light field input stage, each feature value is calculated by a computer using a function whose variable is the coordinate value (x, y) of the point P on the XY two-dimensional coordinate plane and the direction D of the ray vector L passing through the point P. The first light field Fα (x, y, D) and the second light field Fβ (x, y, D) indicating the above are input. For example, the two-dimensional image Pα1 constituting the first multi-viewpoint image information E (α) shown in FIG. 11 is given to the light vector L1 passing through the point P on the projection surface S toward the first observation direction D1. Information indicating the characteristic amount (that is, the pixel value of the pixel at the position of the point P), and the two-dimensional image Pα2 directs the point P on the projection surface S in the second observation direction D2. It has information indicating the feature amount (that is, the pixel value of the pixel at the position of the point P) assigned to the passing light vector L2. Accordingly, the five two-dimensional images Pα1 to Pα5 constituting the first multi-view image information E (α) shown in FIG. 11 are information indicating the first light field Fα (x, y, D). It turns out that. Similarly, the five two-dimensional images Pβ1 to Pβ5 constituting the second multi-viewpoint image information E (β) are information indicating the second light field Fβ (x, y, D).

  In the next synthesis condition setting stage, the computer sets a synthesis condition for synthesizing the first light field Fα (x, y, D) and the second light field Fβ (x, y, D). It is. In this synthesis condition, the first light field Fα (x, y, D) and the second light field Fβ (x, y, D) are placed at predetermined positions so as to be parallel to the predetermined synthesis surface W. Includes arrangement conditions (specifically, depth values Zα and Zβ are set as synthesis conditions).

  In the subsequent light field synthesis step, the computer synthesizes the first light field Fα (x, y, D) and the second light field Fβ (x, y, D) on the XY two-dimensional coordinate plane. A third light field Fγ (x, y, D) indicating individual feature values is generated by a function having the coordinate value (x, y) of the point P and the direction D of the light vector L passing through the point P as variables. It is a stage. Of course, the third light field Fγ (x, y, D) corresponds to the five two-dimensional images Pγ1 to Pγ5 constituting the third multi-viewpoint image information E (γ) shown in FIG. To do.

  Here, if the combination of the pixel values of the three primary colors R, G, and B is used as the feature value to be assigned to each light vector, the first light field Fα (x, y, D) as shown in FIG. Can be configured by a combination of a primary color R component Fαr (x, y, D), a primary color G component Fαg (x, y, D), and a primary color B component Fαb (x, y, D). The light field Fβ (x, y, D) is a combination of a primary color R component Fβr (x, y, D), a primary color G component Fβg (x, y, D), and a primary color B component Fβb (x, y, D). The third light field Fγ (x, y, D) includes a primary color R component Fγr (x, y, D), a primary color G component Fγg (x, y, D), and a primary color B component Fγb. It can be configured by a combination with (x, y, D). In this case, the feature value may be determined independently for each primary color component in the light field synthesis stage.

  FIG. 19 is a perspective view showing the basic principle of the light field composition method according to the present invention. In FIG. 19, only the primary color R component synthesis method is shown for the sake of convenience, but the primary color G component and primary color B component synthesis methods are exactly the same. A closed region Fγr shown in FIG. 19 is a primary color R component plane of the third light field Fγ (x, y, D) newly created by the synthesis process, and is arranged on the synthesis surface W. Of course, at this time, no feature value (that is, the pixel value of the primary color R) has been assigned to each light vector of the light field Fγr.

  On the other hand, the closed region Fαr is a primary color R component plane of the first light field Fα (x, y, D) to be synthesized, and is a predetermined position (for example, a depth value Zα) parallel to the synthesis surface W. (Position indicated by). Similarly, the closed region Fβr is a plane of the primary color R component of the second light field Fβ (x, y, D) to be synthesized, and a predetermined position (for example, a depth value) parallel to the synthesis surface W (Position indicated by Zβ).

  Now, a specific procedure for creating the primary color R component plane Fγr of the third light field Fγ (x, y, D) in a state where such an arrangement is performed will be described. Here, it is considered to determine the characteristic value (pixel value of the primary color R) of the specific light vector Lγ that passes through the specific point P (xγ, yγ) on the composite surface W and faces the specific direction D. For this purpose, a reference straight line R (a straight line indicated by a broken line in the figure) including the specific light vector Lγ is defined, and the specific direction D passes through the intersection P (xα, yα) between the reference straight line R and the first light field Fαr. , And a light vector Lβ facing the specific direction D through the intersection point P (xβ, yβ) of the reference straight line R and the second light field Fβr. Then, based on at least one of the two feature values of the feature value (primary color R pixel value) given to the light vector Lα and the feature value (primary color R pixel value) given to the light vector Lβ. Then, the characteristic value (pixel value of the primary color R) of the specific light vector Lγ is determined.

  Specifically, as in the example shown in FIG. 19, the intersection point P (xα, yα) between the reference straight line R and the first light field Fαr and the intersection point P (xβ between the reference straight line R and the second light field Fβr , Yβ) both exist, the feature value of the third light field Fγr based on the feature value of the light vector passing through one of the intersections selected based on the arrangement condition and facing the specific direction D Can be determined.

  That is, in the case of the example shown in FIG. 19, when the first light field Fαr arranged above is given priority, the feature value (primary color R pixel value) assigned to the light vector Lα is specified. The feature value of the light vector Lγ (the pixel value of the primary color R) may be used. When the priority is given to the second light field Fβr arranged below, the feature value given to the light vector Lβ. The (primary color R pixel value) may be the characteristic value of the specific light vector Lγ (primary color R pixel value).

  Further, depending on the direction D of the specific light vector Lγ, there may be only one of the intersection of the reference straight line R and the first light field Fαr and the intersection of the reference straight line R and the second light field Fβr. . In such a case, the feature value of the third light field Fγr may be determined based on the feature value of the light vector passing through the existing intersection and facing the specific direction D. Note that if neither the intersection of the reference straight line R and the first light field Fαr nor the intersection of the reference straight line R and the second light field Fβr exists, such a characteristic value for the specific direction D is determined. I can't.

  In the case of performing the synthesis considering the transparency as described above, the feature value obtained by blending the feature value given to the light vector Lα and the feature value given to the light vector Lβ at a predetermined ratio. The value may be a feature value of the specific light vector Lγ.

  The example in which the characteristic value of the specific light vector Lγ that passes through the specific point P (xγ, yγ) on the composite plane W and faces the specific direction D has been described. If this processing is performed, the feature values for all the light vectors passing through the specific point P (xγ, yγ) can be determined. Further, if such processing is performed for a large number of points in the closed region Fγr on the composite surface W, predetermined feature values are assigned to individual light vectors passing through the individual points in the closed region Fγr. A plane Fγr (x, y, D) of the primary color R component of the third light field can be obtained. By the same method, it is possible to obtain the primary light component G component plane Fγg (x, y, D) and the primary color B component plane Fγb (x, y, D) of the third light field. A third light field Fγ (x, y, D), which is an aggregate of the primary color component planes, is obtained. If this is output from a computer, it can be used for various purposes as a synthesized light field.

  As described above, the basic principle of creating the third light field Fγr by combining the first light field Fαr and the second light field Fβr with respect to the primary color R component has been described. Each light field defines a light vector of a finite number of directions for a finite number of points arranged in a closed region having a finite area, and each light vector has a predetermined feature value (pixel value). ) Is given as information. In other words, each point P on each light field is actually defined as a representative point of pixels arranged with a small area, and discretely at positions corresponding to the pitch of the pixel arrangement. It will be a placed point. For this reason, each light field actually needs to be handled as an array of pixels having a very small area.

  FIG. 20 is a perspective view showing a specific method of a light field composition method incorporating such a pixel concept. As shown in the figure, the first to third light fields Fαr, Fβr, and Fγr related to the primary color R component are representative of each of a large number of pixels each having a predetermined area arranged on an XY two-dimensional coordinate plane. Image data having each feature value of a plurality of ray vectors passing through a point P (for example, the center point of a minute region constituting the pixel) as a pixel value is configured.

  Therefore, in the light field synthesis stage, the pixel values of the individual pixels Gγr (representative points are P (xγ, yγ)) constituting the third light field Fγr are used as the intersections of the reference straight line R and the first light field Fαr. On the second light field Fβr including the pixel value of the pixel Gαr on the first light field Fαr including P (xα, yα) and the intersection P (xβ, yβ) between the reference straight line R and the second light field Fβr It may be determined based on at least one of the pixel values of the pixel Gβr.

  Here, the reference straight line R is defined as a straight line that includes the specific light vector Lγ that passes through the representative point P (xγ, yγ) of the pixel Gγr and faces the specific direction D. The reference straight line R and the first light field Fαr A point P (xα, yα) is defined as the intersection point of and a point P (xβ, yβ) is defined as the intersection point of the reference straight line R and the second light field Fβr. Of course, the intersection point P (xα, yα) is not necessarily the representative point of the pixel Gαr, and the light vector Lα is not always defined at the position of the intersection point P (xα, yα). Similarly, the intersection point P (xβ, yβ) is not necessarily a representative point of the pixel Gβr, and the light vector Lβ is not always defined at the position of the intersection point P (xβ, yβ). However, the pixel Gαr is extracted as a pixel that includes the intersection point P (xα, yα) at any position in the minute region, and the pixel Gβr has the intersection point P (xβ, yβ) at any position in the minute region. The extracted light vectors Lα and Lβ are defined at positions slightly deviated from the reference straight line R (positions where deviation occurs within the range of the minute area constituting the pixel). Even if it is a vector, since the feature-value (pixel value) provided to these vectors is extracted without trouble, there is no problem.

  Note that, as exemplified in FIG. 17 as the multi-viewpoint image information E (β), the n two-dimensional images constituting the multi-viewpoint image information used in the present invention may be the same common image. Therefore, also in the light field combining method described here, the light vector in which one or both of the first light field Fα and the second light field Fβ pass through the same point P is the same regardless of the direction D. A light field to which a feature value (pixel value) is assigned may be used.

10: Image information storage unit 20: Composition condition storage unit 30: Image composition unit 40: Composite image storage unit A: Shooting start point B: Shooting end point D: Direction vector / ray beam vector direction D ′: Direction vector projection image D1 D5: Observation direction E: Multi-viewpoint image information E (α): Multi-viewpoint image information for subject α E (β): Multi-viewpoint image information for subject β E (γ): Multi-synthesized for subjects α and β Viewpoint image information F: light field Fα (x, y, D): first light field Fαr (x, y, D): primary color R plane Fαg (x, y, D) of the first light field: first Light field primary color G plane Fαb (x, y, D): first light field primary color B plane Fβ (x, y, D): second light field Fβr (x, y, D): second Light field primary color R Lane Fβg (x, y, D): second light field primary color G plane Fβb (x, y, D): second light field primary color B plane Fγ (x, y, D): third light Field Fγr (x, y, D): primary color R plane Fγg (x, y, D) of the third light field: primary color G plane Fγb (x, y, D) of the third light field: third light Field primary colors B-plane G1-G4: Pixels Gαr, Gβr, Gγr: Pixels Ka-Kx: Section L: Lenticular lenses L, Lα, Lβ, Lγ: Ray vector M: Recording medium O: Origins P1-P70 of the coordinate system Two-dimensional image (parallel projection image)
P1a to P5f: Partial images P11a to P35a: Partial images P (x, y): One point P (xα, yα), P (xβ, yβ), P (xγ, yγ) on the light field 1 point Pα1 to Pα5: two-dimensional image (parallel projection image) of the subject α
Pβ1 to Pβ5: Two-dimensional images (parallel projection images) about the subject β
Pγ1 to Pγ5: two-dimensional images (parallel projection images) synthesized for the subjects α and β
Q: Reference point R: Reference straight line S: Projection planes S1 to S16: Steps T in the flowchart: Shooting path T (i): i-th shooting points U1 to U1000: Live-shot shot images V1 to V5: Viewpoint W: Composition Surface X, Y, Z: coordinate axes x, y: coordinate values Zα, Zβ: depth values α, α1, α2: subject β: subject θ: elevation angle φ: azimuth angle

Claims (6)

A plane in which a plurality of points P are arranged, and each of the individual points P is defined with a plurality of ray vectors L passing through the point P and facing in different directions. A method of synthesizing a light field defined as a plane with a specific feature value,
The first light field Fα in which the computer indicates individual feature values by a function having the coordinate value (x, y) of the point P on the XY two-dimensional coordinate plane and the direction D of the ray vector L passing through the point P as variables. A light field input stage for inputting (x, y, D) and a second light field Fβ (x, y, D);
A synthesis condition setting step in which a computer sets a synthesis condition for synthesizing the first light field and the second light field;
A computer synthesizes the first light field Fα (x, y, D) and the second light field Fβ (x, y, D) to obtain the coordinate value of the point P on the XY two-dimensional coordinate plane. A light field synthesis stage for generating a third light field Fγ (x, y, D) indicating individual feature values by a function having (x, y) and the direction D of the light vector L passing through the point P as a variable; ,
A light field output stage in which a computer outputs the third light field Fγ (x, y, D) obtained in the light field synthesis stage;
Have
The composition condition includes an arrangement condition for arranging the first light field and the second light field at a predetermined position so as to be parallel to a predetermined composition surface W.
In the light field combining step, the first light field and the second light field are arranged based on the arrangement condition, and the specific light beam that passes through the specific point P on the composite plane W and faces the specific direction D. The feature value of the vector is the feature value of the light vector that faces the specific direction D through the intersection of the reference straight line including the specific light vector and the first light field, and the reference straight line and the second light field. The composite plane W is determined on the basis of at least one of the feature values of the ray vectors passing through the intersections and directed in the specific direction D, and a predetermined feature value is given to each ray vector passing through each point. A light field composition method, wherein the third light field Fγ (x, y, D) is used. The light field synthesis method according to claim 1 ,
In the light field synthesis stage,
If only one of the intersection between the reference line and the first light field and the intersection between the reference line and the second light field exists, the ray vector of the light vector directed through the existing intersection and pointing in the specific direction D Determining the feature value of the third light field based on the feature value;
When both the intersection of the reference line and the first light field and the intersection of the reference line and the second light field exist, the specific direction D passes through one of the intersections selected based on the arrangement condition. A light field composition method, comprising: determining a feature value of a third light field based on a feature value of a ray vector facing the light. The light field synthesis method according to claim 1 or 2 ,
For each of a large number of pixels having a predetermined area arranged on the XY two-dimensional coordinate plane, the first to third light fields are pixel feature values of a plurality of ray vectors passing through the representative point P of the pixel. Consists of image data as values,
In the light field synthesis stage, the pixel values of the individual pixels constituting the third light field are set to the pixel values of the pixels on the first light field including the intersection of the reference line and the first light field, and the reference line. A light field composition method, comprising: determining based on at least one of pixel values of a pixel on a second light field including an intersection with the second light field. The light field synthesis method according to any one of claims 1 to 3 ,
A combination of pixel values of the three primary colors R, G, and B is used as a feature value to be assigned to each light vector.
The first light field is configured by a combination of a primary color R component Fαr (x, y, D), a primary color G component Fαg (x, y, D), and a primary color B component Fαb (x, y, D),
The second light field is configured by a combination of a primary color R component Fβr (x, y, D), a primary color G component Fβg (x, y, D), and a primary color B component Fβb (x, y, D),
The third light field is configured by a combination of a primary color R component Fγr (x, y, D), a primary color G component Fγg (x, y, D), and a primary color B component Fγb (x, y, D),
A light field composition method, wherein, in the light field composition stage, feature values are determined independently for each primary color component. In the light field synthesis method according to any one of claims 1 to 4 ,
At least one of the first light field and the second light field is a light field to which the same feature value is given regardless of the direction D with respect to a ray vector passing through the same point P. Light field composition method. A program for causing a computer to execute the light field synthesis method according to claim 1 .

Images