JP2006203668A - Image creation system and image creation method - Google Patents

Abstract

<P>PROBLEM TO BE SOLVED: To easily create a multi-eye cubic image which gives natural cubic feeling by making it possible to confirm the cubic feeling before finally completing the multi-eye cubic image. <P>SOLUTION: In an image creation system 100 which creates the multi-eye cubic image for cubic vision based on an image of a cubic vision object seen from at least 3 or more visual point positions, an image processing device 1 establishes a view point position of an observer for the multi-eye cubic image, creates a two-eye cubic image according to the established view point position, and makes the user confirm the cubic feeling of the displayed image in a display unit 3 by displaying the created two-eye cubic image in the display unit 3. When the user discriminates the cubic feeling to be unsuitable, view point positions (or, observation points) for the multi-eye cubic image are adjusted, and the two-eye cubic image corresponding to the adjusted view point is displayed in the display unit 3. <P>COPYRIGHT: (C)2006,JPO&NCIPI

Description

  The present invention relates to an image generation system and an image generation method for generating an image for stereoscopic viewing.

  2. Description of the Related Art Conventionally, a binocular stereoscopic image created from an image from a left-eye viewpoint and an image from a right-eye viewpoint is well known as a stereoscopic image. This is a method of observing images obtained from both viewpoints separately with the left and right eyes, polarized glasses method, two-color glasses method (anaglyph), head mounted display (HMD) method, parallax barrier. Various methods such as a method are known. In binocular stereo images, a stereoscopic effect can be obtained by binocular parallax, but it is assumed that the viewpoint of the observer is fixed, and the image obtained in response to a change in viewpoint does not change. Therefore, it is not possible to obtain a realistic three-dimensional effect.

Therefore, by preparing viewpoint images obtained at three or more different viewpoint positions in advance, and using a multi-view stereoscopic image that can view the viewpoint image according to the viewpoint of the observer from among them, more Realistic three-dimensional effect can be obtained. Integral photography is a typical example of such a multi-view stereoscopic image (see Non-Patent Document 1, for example).
Kazuhisa Yanaka and 5 others "Synthesis of integral photography images by Shade (TM)" 3D Image Conference 2004, p.173-176

  In multi-view stereoscopic images, the stereoscopic effect changes greatly when various viewpoint parameters are adjusted. Therefore, the optimal stereoscopic effect can be obtained while checking the stereoscopic effect when creating a multi-view stereoscopic image. It is desirable to select viewpoint parameters. At this time, in the work process of printing a multi-view stereoscopic image (hard copy) and confirming the stereoscopic effect and adjusting the viewpoint parameters and the like, the time required for the work, the production cost, etc. will increase. There's a problem.

  An object of the present invention is to make it possible to easily generate a multi-view stereoscopic image that gives a natural stereoscopic effect by making it possible to confirm the stereoscopic effect before finally completing the multi-view stereoscopic image. is there.

  In order to solve the above problems, the invention according to claim 1 is an image generation for generating a multi-view stereoscopic image for stereoscopic viewing based on an image of a stereoscopic object viewed from at least three different viewpoint positions. In the system, setting means for setting the observer's viewpoint position with respect to the multi-view stereoscopic image, and binocular stereoscopic image generating means for generating a binocular stereoscopic image corresponding to the viewpoint position set by the setting means; And a display means for displaying the binocular stereoscopic image generated by the binocular stereoscopic image generating means.

  According to a second aspect of the present invention, in the image generation system according to the first aspect, the binocular stereoscopic image generating means is set from the viewpoint images that constitute the multi-view stereoscopic image. The binocular stereoscopic image is generated by selecting a left-eye image and a right-eye image corresponding to a viewpoint position.

  According to a third aspect of the present invention, in the image generation system according to the second aspect, the binocular stereoscopic image generating means is set in the viewpoint images constituting the multi-view stereoscopic image. When the left-eye image and / or the right-eye image corresponding to the viewpoint position does not exist, the left-eye image and / or the right-eye is obtained by performing interpolation processing on data of a plurality of viewpoint images existing in the vicinity of the viewpoint direction at the viewpoint position. It is characterized by generating an image for use.

  According to a fourth aspect of the present invention, in the image generation system according to any one of the first to third aspects, the multi-view stereoscopic image is observed based on the binocular stereoscopic image displayed by the display means. Adjusting means for adjusting the distance; and reconstructing means for reconstructing the multi-view stereoscopic image so that the binocular stereoscopic image corresponding to the observation distance adjusted by the adjusting means can be viewed at another observation distance; It is characterized by providing.

According to a fifth aspect of the present invention, in the image generation system according to the fourth aspect, the multi-view stereoscopic image is visually recognized according to a multi-view reconstructed image created based on each viewpoint image and a viewpoint direction. Consists of visual image selection means for selecting possible image information,
The visual image selection means is a shielding plate having a microhole array,
The reconstruction unit changes pixels on the multi-view stereoscopic image that can be viewed in the viewpoint direction passing through the microhole array in accordance with a change in the distance between the multi-view reconstruction image and the shielding plate. The multi-view stereoscopic image is reconstructed.

According to a sixth aspect of the present invention, in the image generation system according to the fourth aspect, the multi-view stereoscopic image is visually recognized according to a multi-view reconstructed image created based on each viewpoint image and a viewpoint direction. Consists of visual image selection means for selecting possible image information,
The visual image selection means is a shielding plate having a microhole array,
The reconstructing means is on a multi-view stereoscopic image that is visible in the viewpoint direction passing through the microhole array in accordance with a change in the refractive index of the gap between the multi-view reconstructed image and the shielding plate. The multi-view stereoscopic image is reconstructed by changing pixels.

According to a seventh aspect of the present invention, in the image generation system according to the fourth aspect, the multi-view stereoscopic image is visually recognized in accordance with a multi-view reconstructed image created based on each viewpoint image and a viewpoint direction. Consists of visual image selection means for selecting possible image information,
The visual image selection means is a micro lens array,
The reconstruction means changes pixels on a multi-view stereoscopic image that can be viewed in a viewpoint direction passing through the micro lens array according to a change in a distance between the multi-view reconstruction image and the micro lens array. Thus, the multi-view stereoscopic image is reconstructed.

The invention according to claim 8 is the image generation system according to claim 4, wherein the multi-view stereoscopic image is visually recognized according to a multi-view reconstructed image created based on each viewpoint image and a viewpoint direction. Consists of visual image selection means for selecting possible image information,
The visual image selection means is a micro lens array,
The reconstruction unit is configured to display a multi-view stereoscopic image that is visible in a viewpoint direction passing through the micro lens array according to a change in a refractive index of a gap between the multi-view reconstruction image and the micro lens array. The multi-view three-dimensional image is reconstructed by changing the pixels.

The invention according to claim 9 is the image generation system according to claim 4, wherein the multi-view stereoscopic image is visually recognized according to a multi-view reconstructed image created based on each viewpoint image and a viewpoint direction. Consists of visual image selection means for selecting possible image information,
The visual image selection means is a micro lens array,
The reconstruction means changes the pixels on the multi-view stereoscopic image visible in the viewpoint direction passing through the micro-lens array in accordance with the adjustment of the focal length of the micro-lens array, so that the multi-view stereoscopic It is characterized by reconstructing images.

According to a tenth aspect of the present invention, in the image generation system according to the fourth aspect, the multi-view stereoscopic image is constituted by a multi-view reconstructed image generated based on each viewpoint image,
The reconstruction unit reconstructs the multi-view stereoscopic image by spatially enlarging or reducing the pixel data in an area centered on a predetermined position on the multi-viewpoint reconstructed image. It is characterized by doing.

  In addition, the method of reconstructing each multi-view stereoscopic image described in claims 5 to 10 is not limited to the method described in each claim, but arbitrarily combines the methods described in each claim. May be realized.

  The invention according to claim 11 is the image generation system according to any one of claims 1 to 10, wherein the display means is adjusted according to adjustment of a viewpoint position with respect to a multi-view stereoscopic image. It is characterized by switching to a binocular stereoscopic image corresponding to the viewpoint position and displaying it.

  A twelfth aspect of the present invention is the image generation system according to any one of the first to eleventh aspects, wherein the display means is configured to visually recognize a left-eye image for the left eye and a right-eye image for the right eye. It is characterized by performing display processing.

  The invention according to claim 13 is an image generation method for generating a multi-view stereoscopic image for stereoscopic viewing based on an image of a stereoscopic object viewed from at least three different viewpoint positions. A setting step for setting the observer's viewpoint position with respect to the stereoscopic image, a binocular stereoscopic image generating step for generating a binocular stereoscopic image corresponding to the viewpoint position set in the setting step, and the binocular stereoscopic image And a display step for displaying the binocular stereoscopic image generated in the generation step.

  According to a fourteenth aspect of the present invention, in the image generating method according to the thirteenth aspect, in the binocular stereoscopic image generating step, the setting is made from among the viewpoint images constituting the multi-view stereoscopic image. The binocular stereoscopic image is generated by selecting a left-eye image and a right-eye image corresponding to a viewpoint position.

  According to a fifteenth aspect of the present invention, in the image generation method according to the fourteenth aspect, in the binocular stereoscopic image generation step, the setting is made in each viewpoint image constituting the multi-view stereoscopic image. When the left-eye image and / or the right-eye image corresponding to the viewpoint position does not exist, the left-eye image and / or the right-eye is obtained by performing interpolation processing on data of a plurality of viewpoint images existing in the vicinity of the viewpoint direction at the viewpoint position. It is characterized in that a working image is generated.

  The invention according to claim 16 is the image generation method according to any one of claims 13 to 15, wherein the multi-view stereoscopic image is observed based on the binocular stereoscopic image displayed in the display step. An adjustment step of adjusting the distance, and a reconstruction step of reconstructing the multi-view stereoscopic image so that the binocular stereoscopic image corresponding to the observation distance adjusted in the adjustment step can be viewed at another observation distance; It is characterized by including.

According to a seventeenth aspect of the present invention, in the image generation method according to the sixteenth aspect, the multi-view stereoscopic image is visually recognized according to a multi-view reconstructed image created based on each viewpoint image and a viewpoint direction. Consists of visual image selection means for selecting possible image information,
The visual image selection means is a shielding plate having a microhole array,
In the reconstruction step, by changing pixels on the multi-view stereoscopic image that can be viewed in the viewpoint direction passing through the microhole array according to a change in the distance between the multi-view reconstruction image and the shielding plate. The multi-view stereoscopic image is reconstructed.

According to an eighteenth aspect of the present invention, in the image generation method according to the sixteenth aspect, the multi-view stereoscopic image is visually recognized according to a multi-view reconstructed image created based on each viewpoint image and a viewpoint direction. Consists of visual image selection means for selecting possible image information,
The visual image selection means is a shielding plate having a microhole array,
In the reconstruction step, on the multi-view stereoscopic image that is visible in the viewpoint direction passing through the microhole array in accordance with a change in the refractive index of the gap between the multi-view reconstruction image and the shielding plate. The multi-view stereoscopic image is reconstructed by changing pixels.

According to a nineteenth aspect of the present invention, in the image generating method according to the sixteenth aspect, the multi-view stereoscopic image is visually recognized according to a multi-view reconstructed image created based on each viewpoint image and a viewpoint direction. Consists of visual image selection means for selecting possible image information,
The visual image selection means is a micro lens array,
In the reconstruction step, the pixels on the multi-view stereoscopic image that are visible in the viewpoint direction passing through the micro lens array are changed according to a change in the distance between the multi-view reconstruction image and the micro lens array. Thus, the multi-view stereoscopic image is reconstructed.

According to a twentieth aspect of the present invention, in the image generation method according to the sixteenth aspect, the multi-view three-dimensional image is visually recognized according to a multi-view reconstructed image created based on each viewpoint image and a viewpoint direction. Consists of visual image selection means for selecting possible image information,
The visual image selection means is a micro lens array,
In the reconstruction step, on a multi-view stereoscopic image that is visible in the viewpoint direction passing through the micro lens array according to a change in the refractive index of the gap between the multi-view reconstruction image and the micro lens array. The multi-view three-dimensional image is reconstructed by changing the pixels.

According to a twenty-first aspect of the present invention, in the image generating method according to the sixteenth aspect, the multi-view stereoscopic image is visually recognized according to a multi-viewpoint reconstructed image created based on each viewpoint image and a viewpoint direction. Consists of visual image selection means for selecting possible image information,
The visual image selection means is a micro lens array,
In the reconstruction step, the multi-view three-dimensional image is changed by changing pixels on the multi-view three-dimensional image that can be viewed in the viewpoint direction passing through the small lens array in accordance with the adjustment of the focal length of the small lens array. It is characterized in that the image is reconstructed.

According to a twenty-second aspect of the present invention, in the image generation method according to the sixteenth aspect, the multi-view stereoscopic image is configured by a multi-view reconstructed image generated based on each viewpoint image.
In the reconstruction step, the multi-view stereoscopic image is reconstructed by spatially enlarging or reducing the pixel data in a region centered on a predetermined position on the multi-viewpoint reconstructed image. It is characterized by being.

  In addition, the method of reconstructing each multi-view stereoscopic image described in claims 17 to 22 is not limited to the method described in each claim, but arbitrarily combines the methods described in each claim. May be realized.

  According to a twenty-third aspect of the present invention, in the image generation method according to any one of the thirteenth to twenty-second aspects, in the display step, the adjustment is performed according to the adjustment of the viewpoint position with respect to the multi-view stereoscopic image. It is characterized by switching to a binocular stereoscopic image corresponding to the viewpoint position and displaying it.

  According to a twenty-fourth aspect of the present invention, in the image generation method according to any one of the thirteenth to twenty-third aspects, in the display step, a left eye image is visually recognized by the left eye and a right eye image is visually recognized by the right eye. A display process is performed.

  According to the present invention, before the multi-view stereoscopic image that is the final product is generated, the binocular stereoscopic image is displayed so that the stereoscopic effect can be confirmed. In addition, it is possible to easily generate a multi-view stereoscopic image that gives a natural stereoscopic effect without recreating.

Hereinafter, embodiments of the present invention will be described in detail with reference to the drawings.
First, the configuration in the present embodiment will be described.

  FIG. 1 shows a configuration of an image generation system 100 according to an embodiment of the present invention. As shown in FIG. 1, the image generation system 100 includes an image processing device 1, a CG / CAD creation device 7, an imaging device 8, and a printing device 9.

  The image processing apparatus 1 includes an operation unit 2, a display unit 3, a storage device 4, an external interface 5, and a control unit 6.

  The operation unit 2 includes a key input device including a character input key, a numeric keypad, a cursor key, and various function keys, and a pointing device such as a mouse, and outputs an operation signal generated by operating the key input device or the pointing device to the control unit 6. To do. The operation unit 2 may be configured to include a touch panel. In this case, the touch panel of the operation unit 2 is provided so as to cover the display display of the display unit 3, and detects and detects coordinates instructed by a coordinate reading principle such as an electromagnetic induction type, a magnetostriction type, and a pressure sensitive type. The coordinates are output to the control unit 6 as a position signal.

  The display unit 3 includes a display such as an LCD (Liquid Crystal Display) and a CRT (Cathode Ray Tube), and performs a required display process according to a display control signal input from the control unit 6.

  The storage device 4 is configured by an HD (Hard Disc) or the like, and stores data such as an image input via the external interface 5 and an image processing result executed by the control unit 6.

  The external interface 5 exchanges data with external devices (CG / CAD creation device 7, imaging device 8, and printing device 9) connected to the image processing device 1.

  The control unit 6 includes a CPU (Central Processing Unit), a RAM (Random Access Memory), a ROM (Read Only Memory), and the like. The CPU executes various control programs stored in the ROM using a predetermined area of the RAM as a work area, and controls the operation of each unit constituting the image processing apparatus 1. In the present embodiment, the control unit 6 converts a multi-view stereoscopic image based on images (hereinafter referred to as “multi-view images”) of stereoscopic objects viewed from a plurality of (at least three or more) different viewpoint positions. A multi-view stereoscopic image generation process (see FIG. 5) to be generated is executed. The multi-view stereoscopic image will be described in detail later with reference to FIGS.

  The CG / CAD creating device 7 creates three-dimensional data and rendering data by CG (Computer Graphics) or CAD (Computer-Aided Design).

  The imaging device 8 has cameras (digital cameras or the like) arranged at a plurality of different positions, and each camera captures a shooting optical axis simultaneously toward a specific point of a subject (stereoscopic object) and is obtained by shooting. The obtained image data is output to the image processing apparatus 1. As a method for arranging the cameras, for example, there are a method in which the cameras are arranged in a line in the horizontal direction, a method in which the cameras are arranged in a lattice shape or a honeycomb shape, and the like. By photographing with the imaging device 8, viewpoint images corresponding to the number of cameras included in the imaging device 8 are acquired, and a multi-viewpoint image is obtained by arranging the acquired viewpoint images according to the arrangement of the cameras. In addition, as shown in FIG. 2, the camera is not arranged toward a specific point, and the camera is arranged so that the optical axes of the cameras are parallel to each other. There is also a method of selecting and interpolating for each pixel.

  The printing device 9 is a printer that prints out an image generated by the image processing device 1 and is configured by an ink jet printer, a thermal sublimation printer, or the like.

  Although FIG. 1 shows the case where the CG / CAD creation device 7, the imaging device 8, and the printing device 9 are connected to the image processing device 1, they are not necessarily connected. For example, image data obtained by the CG / CAD creation device 7 and the imaging device 8 is recorded on a recording medium such as a memory card, and the image processing device 1 reads the image data recorded on the recording medium and performs image processing. You may make it perform. Alternatively, the print data created by the image processing apparatus 1 may be recorded on a recording medium such as a memory card, and the printing apparatus 9 may read the print data recorded on the recording medium and perform print processing.

Next, a multi-view stereoscopic image will be described with reference to FIGS.
As the multi-viewpoint image, for example, as shown in FIG. 2, an image obtained by photographing a subject (stereoscopic object) by a plurality of cameras included in the imaging device 8, or CG, CAD in the CG / CAD creation device 7 The image obtained by rendering from the above is included.

  FIG. 3A shows an example of a multi-viewpoint image in which viewpoint images are arranged in a grid pattern and a multi-viewpoint reconstructed image generated from the multi-viewpoint image. FIG. 3A shows an example in which the multi-viewpoint reconstructed image is an IP (Integral Photography) image. In this multi-viewpoint image, the i-th viewpoint position from the left and the j-th viewpoint position from the top is (i, j), the viewpoint image at the viewpoint position (i, j) is V (i, j), and the viewpoint image V (i, The coordinate in j) is (x, y), and the pixel (pixel) at the coordinate (x, y) of the viewpoint image V (i, j) is I (i, j, x, y). A multi-viewpoint image is composed of viewpoint images V (i, j) (i = 1, 2,..., M, j = 1, 2,..., N) having m pieces in the horizontal direction and n pieces in the vertical direction. The pixels I (i, j, x, y) of the same coordinates (x, y) of each viewpoint image V (i, j) are gathered together in a rectangular area composed of m × n pixels. As a result, a pixel r (x, y) in the multi-viewpoint reconstructed image is formed. Hereinafter, as shown in FIG. 3B, each pixel I (i, j, x, y) constituting the pixel r (x, y) in the multi-viewpoint reconstructed image is referred to as a “sub-pixel”. That is, each pixel of the multi-viewpoint reconstructed image is composed of subpixels arranged in m in the horizontal direction and n in the vertical direction. Each sub-pixel I (i, j, x, y) in the pixel r (x, y) is arranged according to the sequence of viewpoint positions (i, j). In FIG. 3A, it is assumed that each viewpoint image is composed of 180 × 180 pixels, and 9 viewpoints (m = 9) in the horizontal direction and 9 viewpoints (n = 9) in the vertical direction, that is, An example is shown in which one multi-viewpoint reconstructed image (IP image) is generated from a multi-viewpoint image composed of 9 × 9 = 81 different viewpoint images.

  It should be noted that the configuration of the sub-pixels constituting each pixel of the multi-viewpoint reconstructed image does not need to match the configuration (m × n) of the viewpoint images constituting the multi-viewpoint image. For example, it is possible to make the configuration smaller than m × n by thinning out the sub-pixel data in the pixel or extracting only a part (center part) of the pixel. Further, by inserting sub-pixel data in a pixel, a configuration larger than m × n can be achieved.

  Next, the principle of integral photography using a printing technique will be described with reference to FIG.

  As shown in FIG. 4, a pinhole array is printed in advance on the upper transparent sheet, and an IP image as a multi-viewpoint reconstructed image is printed in advance on the lower transparent sheet. In order to keep the distance between the two transparent sheets constant, an intermediate transparent plate is inserted between the two transparent sheets. Further, in order to illuminate the IP image, a light source (backlight) is disposed below the IP image. These constitute the “multi-view stereoscopic image” as a whole.

  The observer H sees the light that has passed through the pinhole array, but cannot distinguish whether the light that enters the right eye has come out from the point Q of the IP image or the point S on the three-dimensional object. Similarly, it cannot be distinguished whether the light entering the left eye has come out from the point R of the IP image or the point S on the three-dimensional object. Therefore, the observer H appears as if there is an object at the point S due to the binocular parallax. In this way, stereoscopic viewing from an IP image becomes possible. The IP image is composed of sub-pixels (point Q, point R, etc.) in which information of light emitted from the point S on the three-dimensional object is appropriately assigned to pixel coordinates for each viewpoint direction.

Next, the operation in this embodiment will be described.
As an operation in the present embodiment, a multi-view stereoscopic image generation process executed in the image processing apparatus 1 will be described with reference to a flowchart of FIG.

  First, the viewpoint position for the multi-view stereoscopic image is set (step S1). The setting of the viewpoint position in step S1 may be set by operating the operation unit 2, or a viewpoint position registered in advance in the image processing apparatus 1 or a viewpoint position registered in advance in multi-viewpoint image data is set. You may do it.

  Next, the left-eye viewpoint direction and the right-eye viewpoint direction are calculated based on the viewpoint position set in step S1 (step S2). As shown in FIG. 6A, when the viewpoint position, which is the middle point between both eyes, is determined, if the distance between the left eye and the right eye (the distance between both eyes) is constant, the left eye is determined according to the viewpoint position. A viewpoint direction and a viewpoint direction of the right eye are determined.

  Next, viewpoint images corresponding to the left-eye viewpoint direction and the right-eye viewpoint direction calculated in step S2 are selected from the multi-viewpoint images as the left-eye image and the right-eye image, respectively (step S3). FIG. 6B shows an enlarged view of the circular portion (dotted line) in FIG. In step S3, as shown in FIG. 6B, the viewpoint image to which the sub-pixel in the viewpoint direction of the left eye belongs is selected as the left-eye image on the multi-view reconstructed image, and the right eye on the multi-view reconstructed image is selected. The viewpoint image to which the subpixel in the viewpoint direction belongs is selected as the right-eye image. FIG. 7 shows an example in which the image of the viewpoint 35 and the image of the viewpoint 75 are selected as the left-eye image and the right-eye image from the multi-viewpoint images composed of 9 × 9 viewpoint images, respectively. Yes.

  If there is no viewpoint image that exactly matches the viewpoint direction of the left eye and / or the viewpoint direction of the right eye in the multi-viewpoint image, interpolation processing is performed on pixel data of a plurality of viewpoint images existing in the vicinity of the viewpoint direction. Thus, the pixel data of the left-eye image and / or the right-eye image may be determined.

For example, as shown in FIG. 8, it is assumed that pixel data corresponding to the viewpoint A direction, the viewpoint B direction, the viewpoint C direction, and the viewpoint D direction exist in the vicinity of the viewpoint direction of the left eye on the multi-viewpoint image. In FIG. 8, when pixel data in the viewpoint A direction, viewpoint B direction, viewpoint C direction, and viewpoint D direction are Pa, Pb, Pc, and Pd, respectively, the pixel data Pl in the viewpoint direction of the left eye is linear interpolation. And is calculated as shown in Equation (1).
Pl = (1-r) (1-s) Pa + r (1-s) Pb + (1-r) sPc + rsPd (1)

  When the left-eye image and the right-eye image are selected in step S3, a binocular stereoscopic image is generated by a predetermined method using the left-eye image and the right-eye image (step S4). An ocular stereoscopic image is displayed on the display unit 3 (step S5). The generation method of the binocular stereoscopic image in step S4 follows the display method in step S5. A method for displaying a binocular stereoscopic image will be described later with reference to FIGS.

  When the user visually confirms the binocular stereoscopic image displayed on the display unit 3 and determines that the stereoscopic effect of the binocular stereoscopic image is inappropriate, the operation distance of the operation unit 2 is changed. By adjusting (step S6; NG), the viewpoint position is set again (step S1), and the processes of steps S2 to S5 are repeated for the set viewpoint position. Here, the observation distance indicates the distance between the viewpoint position of the observer and the multi-view stereoscopic image, as shown in FIG.

  When it is determined by the operation of the operation unit 2 that the stereoscopic effect of the binocular stereoscopic image displayed on the display unit 3 is appropriate (step S6; OK), the viewpoint adjustment parameter is set according to the adjustment of the observation distance and the like. Calculated (step S7). Based on the viewpoint adjustment parameter calculated in step S7, the adjusted multi-view stereoscopic image is generated, that is, the multi-view stereoscopic image is reconstructed (step S8).

  The “viewpoint adjustment parameter” calculated in step S7 is a parameter for changing a viewpoint direction angle (an angle formed between a vertical direction and a viewpoint direction with respect to a multi-view stereoscopic image) for observing a specific viewpoint image. As shown in FIG. 6, by adjusting the observation distance, the viewpoint direction of the left eye and the viewpoint direction of the right eye change, and the observed binocular stereoscopic image also changes. For example, as shown in FIG. 9, a binocular stereoscopic image 1 (FIG. 9A) is observed at an observation distance L1 in a certain multi-view stereoscopic image, and a binocular stereoscopic image 2 (FIG. 9) is also observed at an observation distance L2. 9 (b)) is observed. As shown in FIG. 9C, in order to observe the binocular stereoscopic image 2 observed at the observation distance L2 at the observation distance L1, on the multi-viewpoint reconstructed image constituting the multi-view stereoscopic image. It is necessary to reconstruct a multi-view stereoscopic image by changing the viewing direction angle for each sub-pixel. Here, the observation distance L1 is, for example, the observation distance initially set in the multi-view stereoscopic image generation process of FIG. 5, and the observation distance L2 is the observation distance adjusted by the visual confirmation in step S6. is there. For example, when converting the viewing direction angle θ of a certain sub-pixel on the multi-viewpoint reconstructed image constituting the multi-view stereoscopic image to θ ′ = kθ, k is the view adjustment parameter. In general, when transformation of θ ′ = f (θ, k1, k2,...) Is performed by the function f, k1, k2,.

  For example, as shown in FIG. 10, the reconstruction of the multi-view stereoscopic image in step S8 is realized by expanding / contracting the distance from the sub-pixel center of the sub-pixel of the multi-view reconstructed image at a constant ratio. That is, subpixels are reconfigured by enlargement processing or reduction processing with the subpixel center as the origin. In FIG. 10, the subpixels (FIG. 10A) on the multi-viewpoint reconstructed image viewed in the viewpoint direction 2 are reconfigured so that the subpixels can be viewed in the viewpoint direction 1 by the subpixel enlargement process. (FIG. 10B) shows the case.

  One method for realizing reconstruction of a multi-view stereoscopic image is a method of adjusting a shielding plate in a configuration using a shielding plate. As shown in FIG. 11, a shielding plate (pinhole, beer, etc.) having a microhole array is provided between the multi-viewpoint reconstructed image and the viewpoint position with respect to the multi-viewpoint reconstructed image. On a multi-view stereoscopic image that can be viewed in a specific viewpoint direction passing through the microhole array in accordance with a change in the distance to the plate (the thickness of the gap) (FIG. 11 (a) → (b)). Subpixels can be changed.

  Further, as another method for realizing reconstruction of a multi-view stereoscopic image, there is a method of adjusting a lens in a configuration using a lens (microlens array) such as a fly-eye lens or a lenticular lens. As shown in FIG. 12, a lens is provided between the multi-viewpoint reconstructed image and the viewpoint position with respect to the multi-viewpoint reconstructed image, and the change in the distance (thickness of the gap) between the multi-viewpoint reconstructed image and the lens ( The sub-pixels on the multi-view stereoscopic image that can be viewed in a specific viewpoint direction passing through the lens can be changed according to FIGS. Also, as shown in FIG. 12 (c), by adjusting the focal length (lens shape) of the lens together, the focus state by the lens is appropriately maintained, and the multi-view type with excellent subpixel selectivity. A stereoscopic image can be obtained.

  Note that any transparent member including air or a lens material can be used for the gap between the lens and the multi-viewpoint reconstructed image (corresponding to the intermediate transparent plate in FIG. 4). Moreover, it is possible to reconstruct a multi-view stereoscopic image by changing the refractive index of the gap between the shielding plate or lens and the multi-viewpoint reconstructed image.

  Further, the process of enlarging or reducing the distance from the subpixel center of the subpixel of the multi-viewpoint reconstructed image as shown in FIG. 10 is realized as follows.

In FIG. 13, it is assumed that one pixel of the multi-viewpoint reconstructed image is a rectangular region (m = n = 9 in FIG. 13) composed of m × n subpixels. The sub-pixel center is the center position of the pixel area of the multi-viewpoint reconstructed image (in FIG. 13, the fifth pixel from the left and the fifth pixel from the top). With the subpixel center as the origin, the subpixel data at the relative position (Δx, Δy) from the origin is P (Δx, Δy), and the distance from the origin is changed by k times (k> 1, The subpixel data P ′ (Δx, Δy) when 0 <k <1 in the reduction process is expressed as in Expression (2).
P ′ (Δx, Δy) = P (Δx / k, Δy / k) (2)
In general, the value of the right side P (Δx / k, Δy / k) of Equation (2) is calculated by interpolation from data of adjacent subpixels. The example of FIG. 13 shows a case where the distance from the origin is increased by 1.3 times, and subpixel data is calculated by linear interpolation using data of four adjacent subpixels.

Further, when the distance from the origin is reduced by a factor of k (0 <k <1), if the relative position (Δx / k, Δy / k) from the origin deviates from the pixel area, for example, In addition, it is possible to substitute the nearest neighboring data.
If Δx / k <− (m−1) / 2, Δx / k ← − (m−1) / 2;
If Δx / k> + (m−1) / 2, Δx / k ← + (m−1) / 2;
If Δy / k <− (n−1) / 2, Δy / k ← − (n−1) / 2;
If Δy / k> + (n−1) / 2, Δy / k ← + (n−1) / 2.

  In this way, the multi-view stereoscopic image design is reconstructed based on the viewpoint adjustment parameter, and the multi-view stereoscopic image generation is performed through the image creation / printing step and the assembly step with the constituent members based on the reconstruction design. The process ends.

Next, with reference to FIGS. 14 to 19, a display method of a binocular stereoscopic image will be described.
As shown in steps S1 to S3 of FIG. 5, by adjusting the observation distance for the multi-view stereoscopic image, the viewpoint direction of the left eye and the viewpoint direction of the right eye are determined, and the determined viewpoint direction of both eyes is determined. The corresponding left-eye image and right-eye image are selected (or interpolated) from the multi-viewpoint images. Therefore, as shown in FIG. 14, the left-eye image and the right-eye image are switched according to the adjustment state of the observation distance (or viewpoint position), so that the binocular stereoscopic image displayed on the display unit 3 is also the observation distance. It is switched according to the adjustment status.

  At this time, as shown in FIG. 15, the display unit 3 displays the left eye with only the left-eye image and the right eye with only the right-eye image (ie, displays different images for the left and right eyes). Display process). Hereinafter, the display method of a binocular stereoscopic image will be described with reference to four specific examples.

  The first method is a polarized glasses method. As shown in FIG. 16, the polarized glasses method projects left-eye images and right-eye images simultaneously or in a time-division manner with polarization characteristics corresponding to the polarization selection characteristics of the polarization glasses. The projected image is viewed through polarized glasses made with polarizing filters having different polarization selection characteristics on the left and right eye sides.

  The second method is an anaglyph method. In the anaglyph method, as shown in FIG. 17, the image for the left eye and the image for the right eye are projected simultaneously or in a time-division manner with spectral characteristics corresponding to the spectral transmission characteristics of the color glasses. The resulting image is viewed through color glasses made with color filters having different spectral transmission characteristics on the left eye side and the right eye side.

  The third method is a parallax barrier method. As shown in FIG. 18, the parallax barrier method arranges a parallax barrier in front of an image in which left-eye pixels (sub-pixels) and right-eye pixels (sub-pixels) are alternately arranged. The left eye pixel can be viewed from the viewpoint direction, and the right eye pixel can be viewed from the right eye viewpoint direction.

  The fourth method is a lenticular method. In the lenticular method, as shown in FIG. 19, the lenticular lens is arranged in front of an image in which left-eye pixels (sub-pixels) and right-eye pixels (sub-pixels) are alternately arranged, so that the viewpoint direction of the left eye From the left eye pixel, and from the right eye viewpoint direction, the right eye pixel can be viewed.

  As a simple method that does not depend on the above method, there is a method in which the left-eye image and the right-eye image are arranged side by side and stereoscopically viewed by the observer's own line-of-sight control (parallel method or intersection method). In the case of the parallel method, a shielding plate for avoiding tangling of the line of sight may be used. Moreover, you may use the light guide by an optical fiber bundle, without depending on observer's own gaze control. Furthermore, a method of directly displaying a left-eye image near the left eye and a right-eye image near the right eye using a head-mounted display (HMD) as a display device is also conceivable.

  As described above, according to the image generation system 100 of the present embodiment, before the multi-view stereoscopic image that is the final product in the image processing apparatus 1 is generated, the binocular stereoscopic image is displayed and the stereoscopic display is performed. By making it possible to confirm the feeling, it is possible to easily generate a multi-view stereoscopic image that gives a natural stereoscopic effect without recreating the multi-view stereoscopic image many times.

1 is a block diagram showing a configuration of an image generation system according to an embodiment of the present invention. The figure which shows the positional relationship between each viewpoint position and a to-be-photographed object. The figure which shows the sub-pixel (b) which comprises the multi-view three-dimensional image (a) produced | generated from a multi-view image (a several viewpoint image from which a viewpoint position differs), and the multi-view three-dimensional image. The figure for demonstrating the principle of integral photography. The flowchart which shows the multi-view type | mold stereoscopic image generation process performed in an image processing apparatus. The figure which shows the method of selecting the image for left eyes, and the image for right eyes. The figure which shows the image for left eyes and the image for right eyes selected from a multi-viewpoint image for a binocular stereo image production | generation. The figure which shows the image for left eyes (or image for right eyes) calculated by the interpolation process of the pixel data of a near viewpoint image. The figure which shows the relationship between an observation distance, a binocular stereo image, and a multi-view stereo image. The figure which shows an example of a reconstruction of a multi-view type stereo image. The figure which shows the reconstruction method of the multi-view type stereo image using a shielding board. The figure which shows the reconstruction method of the multi-view type stereo image using a lens. The figure explaining the reconstruction of the sub pixel by the enlargement / reduction process with the sub pixel center as the origin. The figure which shows switching of the binocular stereo image by adjustment of observation distance (or viewpoint position). Conceptual diagram of visual selection. The figure which shows the display system using a polarized glasses system. The figure which shows the display system using an anaglyph system. The figure which shows the display system using a parallax barrier system. The figure which shows the display system using a lenticular system.

Explanation of symbols

DESCRIPTION OF SYMBOLS 1 Image processing apparatus 2 Operation part 3 Display part 4 Memory | storage device 5 External interface 6 Control part 7 CG / CAD production apparatus 8 Imaging apparatus 9 Printing apparatus 100 Image generation system

Claims (24)

In an image generation system that generates a multi-view stereoscopic image for stereoscopic vision based on an image of a stereoscopic object viewed from at least three different viewpoint positions,
Setting means for setting an observer's viewpoint position with respect to the multi-view stereoscopic image;
Binocular stereoscopic image generating means for generating a binocular stereoscopic image corresponding to the viewpoint position set by the setting means;
Display means for displaying a binocular stereoscopic image generated by the binocular stereoscopic image generating means;
An image generation system comprising:   The binocular stereoscopic image generation means selects a left-eye image and a right-eye image corresponding to the set viewpoint position from each viewpoint image indicating an image of a stereoscopic object at each viewpoint position. The image generation system according to claim 1, wherein the binocular stereoscopic image is generated by the method.   The binocular stereoscopic image generation means includes a left-eye image and / or a right-eye image corresponding to the set viewpoint position in each viewpoint image indicating an image of a stereoscopic object at each viewpoint position. If not, a left-eye image and / or a right-eye image is generated by interpolating data of a plurality of viewpoint images existing in the vicinity of the viewpoint direction at the viewpoint position. Image generation system. Adjusting means for adjusting an observation distance for the multi-view stereoscopic image based on the binocular stereoscopic image displayed by the display means;
Reconstructing means for reconstructing the binocular stereoscopic image so that the binocular stereoscopic image corresponding to the observation distance adjusted by the adjusting means can be viewed at another observation distance;
The image generation system according to any one of claims 1 to 3, further comprising: The multi-view stereoscopic image is composed of a multi-view reconstructed image created based on each viewpoint image, and visual image selection means for selecting image information that can be viewed according to the viewpoint direction,
The visual image selection means is a shielding plate having a microhole array,
The reconstruction unit changes pixels on the multi-view stereoscopic image that can be viewed in the viewpoint direction passing through the microhole array in accordance with a change in the distance between the multi-view reconstruction image and the shielding plate. The image generation system according to claim 4, wherein the multi-view stereoscopic image is reconstructed. The multi-view stereoscopic image is composed of a multi-view reconstructed image created based on each viewpoint image, and visual image selection means for selecting image information that can be viewed according to the viewpoint direction,
The visual image selection means is a shielding plate having a microhole array,
The reconstructing means is on a multi-view stereoscopic image that is visible in the viewpoint direction passing through the microhole array in accordance with a change in the refractive index of the gap between the multi-view reconstructed image and the shielding plate. The image generation system according to claim 4, wherein the multi-view stereoscopic image is reconstructed by changing pixels. The multi-view stereoscopic image is composed of a multi-view reconstructed image created based on each viewpoint image, and visual image selection means for selecting image information that can be viewed according to the viewpoint direction,
The visual image selection means is a micro lens array,
The reconstruction means changes pixels on a multi-view stereoscopic image that can be viewed in a viewpoint direction passing through the micro lens array according to a change in a distance between the multi-view reconstruction image and the micro lens array. The image generation system according to claim 4, wherein the multi-view stereoscopic image is reconstructed by the following. The multi-view stereoscopic image is composed of a multi-view reconstructed image created based on each viewpoint image, and visual image selection means for selecting image information that can be viewed according to the viewpoint direction,
The visual image selection means is a micro lens array,
The reconstruction unit is configured to display a multi-view stereoscopic image that is visible in a viewpoint direction passing through the micro lens array according to a change in a refractive index of a gap between the multi-view reconstruction image and the micro lens array. The image generation system according to claim 4, wherein the multi-view stereoscopic image is reconstructed by changing a pixel of the image. The multi-view stereoscopic image is composed of a multi-view reconstructed image created based on each viewpoint image, and visual image selection means for selecting image information that can be viewed according to the viewpoint direction,
The visual image selection means is a micro lens array,
The reconstruction means changes the pixels on the multi-view stereoscopic image visible in the viewpoint direction passing through the micro-lens array in accordance with the adjustment of the focal length of the micro-lens array, so that the multi-view stereoscopic The image generation system according to claim 4, wherein an image is reconstructed. The multi-view stereoscopic image is configured by a multi-view reconstructed image generated based on each viewpoint image,
The reconstruction unit reconstructs the multi-view stereoscopic image by spatially enlarging or reducing the pixel data in an area centered on a predetermined position on the multi-viewpoint reconstructed image. The image generation system according to claim 4, wherein:   The display unit according to any one of claims 1 to 10, wherein the display unit switches to a binocular stereoscopic image corresponding to the adjusted viewpoint position according to the adjustment of the viewpoint position with respect to the multi-view stereoscopic image. An image generation system according to claim 1.   The image generation system according to any one of claims 1 to 11, wherein the display means performs display processing for visually recognizing a left-eye image for the left eye and a right-eye image for the right eye. In an image generation method for generating a multi-view stereoscopic image for stereoscopic vision based on an image of a stereoscopic object viewed from at least three or more different viewpoint positions,
A setting step for setting the viewpoint position of the observer with respect to the multi-view stereoscopic image;
A binocular stereoscopic image generating step for generating a binocular stereoscopic image corresponding to the viewpoint position set in the setting step;
A display step of displaying the binocular stereoscopic image generated in the binocular stereoscopic image generation step;
An image generation method comprising:   In the binocular stereoscopic image generation step, a left-eye image and a right-eye image corresponding to the set viewpoint position are selected from the viewpoint images indicating the stereoscopic object image at each viewpoint position. The image generation method according to claim 13, wherein the binocular stereoscopic image is generated by.   In the binocular stereoscopic image generation step, a left-eye image and / or a right-eye image corresponding to the set viewpoint position are present in each viewpoint image indicating an image of a stereoscopic object at each viewpoint position. 15. If not, a left-eye image and / or a right-eye image are generated by interpolating data of a plurality of viewpoint images existing in the vicinity of the viewpoint direction at the viewpoint position. Image generation method. An adjustment step of adjusting an observation distance for the multi-view stereoscopic image based on the binocular stereoscopic image displayed in the display step;
A reconstruction step of reconstructing the multi-view stereoscopic image so that the binocular stereoscopic image corresponding to the observation distance adjusted in the adjustment step can be viewed at another observation distance;
The image generation method according to claim 13, further comprising: The multi-view stereoscopic image is composed of a multi-view reconstructed image created based on each viewpoint image, and visual image selection means for selecting image information that can be viewed according to the viewpoint direction,
The visual image selection means is a shielding plate having a microhole array,
In the reconstruction step, by changing pixels on the multi-view stereoscopic image that can be viewed in the viewpoint direction passing through the microhole array according to a change in the distance between the multi-view reconstruction image and the shielding plate. The image generation method according to claim 16, wherein the multi-view stereoscopic image is reconstructed. The multi-view stereoscopic image is composed of a multi-view reconstructed image created based on each viewpoint image, and visual image selection means for selecting image information that can be viewed according to the viewpoint direction,
The visual image selection means is a shielding plate having a microhole array,
In the reconstruction step, on the multi-view stereoscopic image that is visible in the viewpoint direction passing through the microhole array in accordance with a change in the refractive index of the gap between the multi-view reconstruction image and the shielding plate. The image generation method according to claim 16, wherein the multi-view stereoscopic image is reconstructed by changing a pixel. The multi-view stereoscopic image is composed of a multi-view reconstructed image created based on each viewpoint image, and visual image selection means for selecting image information that can be viewed according to the viewpoint direction,
The visual image selection means is a micro lens array,
In the reconstruction step, the pixels on the multi-view stereoscopic image that are visible in the viewpoint direction passing through the micro lens array are changed according to a change in the distance between the multi-view reconstruction image and the micro lens array. The image generation method according to claim 16, wherein the multi-view stereoscopic image is reconstructed. The multi-view stereoscopic image is composed of a multi-view reconstructed image created based on each viewpoint image, and visual image selection means for selecting image information that can be viewed according to the viewpoint direction,
The visual image selection means is a micro lens array,
In the reconstruction step, on a multi-view stereoscopic image that is visible in the viewpoint direction passing through the micro lens array according to a change in the refractive index of the gap between the multi-view reconstruction image and the micro lens array. The image generation method according to claim 16, wherein the multi-view stereoscopic image is reconstructed by changing a pixel of the image. The multi-view stereoscopic image is composed of a multi-view reconstructed image created based on each viewpoint image, and visual image selection means for selecting image information that can be viewed according to the viewpoint direction,
The visual image selection means is a micro lens array,
In the reconstruction step, the multi-view three-dimensional image is changed by changing pixels on the multi-view three-dimensional image that can be viewed in the viewpoint direction passing through the small lens array in accordance with the adjustment of the focal length of the small lens array. The image generation method according to claim 16, wherein the image is reconstructed. The multi-view stereoscopic image is configured by a multi-view reconstructed image generated based on each viewpoint image,
In the reconstruction step, the multi-view stereoscopic image is reconstructed by spatially enlarging or reducing the pixel data in a region centered on a predetermined position on the multi-viewpoint reconstructed image. The image generation method according to claim 16, wherein:   23. The display step according to any one of claims 13 to 22, wherein, in accordance with the adjustment of the viewpoint position with respect to the multi-view stereoscopic image, the display is switched to a binocular stereoscopic image corresponding to the adjusted viewpoint position. An image generation method according to claim 1.   The image generation method according to any one of claims 13 to 23, wherein in the display step, display processing is performed to visually recognize a left-eye image for the left eye and a right-eye image for the right eye.

Images