JP4440066B2 - Stereo image generation program, stereo image generation system, and stereo image generation method - Google Patents

Description

  The present invention relates to a system, a computer program, and a method for generating a stereoscopic image that is stereoscopically observed via a stereoscopic display device.

As a method for stereoscopically viewing images, and a method of using the integral photography, parallax barrier, a method had use a stereoscopic display device such as a lenticular plate are known. In order to generate such a stereoscopically observable image (stereoscopic image), it is necessary to prepare an image (perspective image) group obtained by photographing the subject from a plurality of viewpoints.

  In this case, a method of generating a viewpoint image corresponding to a viewpoint that has not been captured from a viewpoint image group captured from a relatively small number of viewpoints by an interpolation method so that stereoscopic images can be observed from a large number of viewpoints. Has been proposed. For example, in the method proposed in Patent Document 1, a parallax map representing the depth distribution of the subject is generated from a viewpoint image group obtained by photographing the subject from a plurality of viewpoints, and an arbitrary one is generated based on the viewpoint image group and the parallax map. A virtual viewpoint image similar to that captured with the viewpoint is generated, and a stereoscopic image is generated using the virtual viewpoint image.

  When displaying a stereoscopic image, it is necessary to consider the size of the screen of the stereoscopic display device. When the size of the stereoscopic image displayed in accordance with the screen size changes, the parallax (that is, the stereoscopic effect) between the viewpoint images presented to the left and right eyes of the observer via the stereoscopic display device changes. If the parallax becomes too small, a satisfactory stereoscopic effect cannot be obtained, and if the parallax becomes too large, an unnatural stereoscopic effect is given.

In order to deal with such problems, mainly in binocular stereoscopic image display, as proposed in Patent Document 2, simple translation processing such as shifting the relative positions of a plurality of viewpoint images is performed. Using the affine processing, the parallax range between the viewpoint images presented to the left and right eyes of the observer is scaled to a fixed-value parallax amount called a general fusion range.
JP 2001-346226 A (paragraphs 0048 to 0085, FIG. 3 etc.) JP-A-9-121370 (paragraphs 0012 to 0026, FIG. 1, etc.)

  However, even if the method proposed in Patent Document 2 is applied to multi-view stereoscopic display, there are disadvantages from the following two points.

  First, since the method proposed in Patent Document 2 mainly assumes binocular stereoscopic display, the allowable value of the amount of parallax on the display surface, that is, the allowable amount of display parallax is set as the fusion range. Or it is the point set with the fixed range according to it. Since the method aims to present a small number of parallax images as in the case of a binocular system, crosstalk between adjacent viewpoint images, which depends on the performance of individual optical elements included in the stereoscopic display device, That is, the problem caused by the resolving power of the individual optical elements of the stereoscopic display device and the turbidity between adjacent viewpoint images that occur when the observer's eyes deviate from the optimum observation distance are not considered.

  Here, the problem of crosstalk caused by the resolving power of the individual optical elements of the stereoscopic display device will be described. First, FIG. 12 shows (a) a parallax barrier and (b) a lenticular lens as examples of a stereoscopic display device. As shown in the figure, by arranging the viewpoint images in order relative to such an optical element, different images can be presented to the left and right eyes at the observation position. In the figure, 1, 2, 3,... 6 represent index numbers of the respective viewpoint images.

  Next, FIG. 13 shows a relationship between a change in the number of viewpoint images constituting the stereoscopic image and a light flux corresponding to each viewpoint image emitted from the stereoscopic display device. (A) schematically shows the case of 2 eyes, (b) of 5 eyes, and (c) of 10 eyes.

  Reference numeral 1301 denotes individual optical elements of the stereoscopic display device, and here, a case of a lenticular lens is shown. Reference numeral 1302 denotes an observer's eye placed at a certain viewpoint position, and a slit range set in front indicates a finite range in which a light beam enters the eye.

Reference numerals 1303, 1304, and 1305 denote areas of individual viewpoint images constituting the stereoscopic image. Further, 130 6 is a light beam from resolution range r _exp of stereoscopic image plane depends on the performance of the individual optical elements that schematically shows. P is a pitch width P of each optical element of the stereoscopic display device, and p is a width of each viewpoint image constituting the stereoscopic image. That is, when the area p of each viewpoint image is wider than the resolution range r _exp of the optical element as in (a), only each viewpoint image is incident on the eye 1302 of the observer. However, when the area p of each viewpoint image is narrower than the resolution range r _exp of the optical element as shown in (c), only the luminous flux from each viewpoint image can be incident on the eyes of the person observing. The light flux from the adjacent viewpoint image cannot enter the observer's eye 1302.

  Further, FIG. 14A shows a light beam emitted from an optical element having an ideal shape, and FIG. 14B shows a light beam emitted from an optical element with poor shape accuracy. When these optical elements are arranged in parallel and an image is observed, an ideal parallel light beam is emitted in (a). Therefore, if observed near the optimum observation distance, an image with small crosstalk is observed. When the emitted light beam diverges as shown in (b), even if observed near the optimum observation distance, images of a plurality of adjacent viewpoint images are observed due to crosstalk. As can be seen from the change in the width of the light beam emitted from each viewpoint image with respect to the change in the number of images in FIG. 13, the influence is avoided as the number of images becomes more than a certain value and the width of each viewpoint image becomes narrower. It becomes a problem that can not be.

  Next, the opacity of the parallax image that occurs when the optimum observation distance is deviated will be described. FIG. 15 shows the state of a light beam when an image is displayed with the above optical elements arranged in parallel. Reference numeral 801 denotes a stereoscopic display device (stereoscopic image), and reference numerals 802 to 806 denote a certain optimum observation distance. When the optical element ideally satisfies the required performance, the images are separated at the optimum observation distance as shown in the figure, and only the viewpoint image represented by a certain index is observed. However, when an image is observed at a position outside the optimum observation distance, viewpoint images represented by a plurality of indexes are simultaneously incident on the eye. When the number of viewpoints for stereoscopic display is small, this crosstalk is generally a problem.

  FIG. 16 shows the relationship between the change in the number of images and the width of the light beam incident on the eye. Similarly to FIG. 13, reference numeral 1601 denotes individual optical elements of the stereoscopic display device, and here, a case of a lenticular lens is shown. Reference numeral 1602 denotes an observer's eye placed at a certain viewpoint position, and a slit range set in front indicates a finite range in which a light beam enters the eye. Reference numerals 1603 and 1604 denote areas of individual viewpoint images constituting the stereoscopic image. Reference numerals 1605 and 1606 denote light beams that are ideally emitted toward the eye 1602 via the optical element 1601 from the width of each viewpoint image region. As can be inferred from the relationship between the change in the width of the light flux in FIGS. 16A and 16B and the incidence range of the observation eye, the occurrence of crosstalk becomes the optimum observation distance as the width of each viewpoint image becomes narrower. It becomes sensitive to it. Therefore, in multi-view type stereoscopic image display, when the distance from the display surface is allowed to some extent with respect to the observation viewpoint, crosstalk occurs. Therefore, when the width of each viewpoint image becomes narrow, that is, many When a stereoscopic image is composed of viewpoint images, crosstalk between adjacent viewpoint images must be considered more strictly.

  By the way, the resolving power of each optical element varies greatly depending on the type, structure, material, and manufacturing method of the stereoscopic display device. Therefore, specifically, when the stereoscopic display device is changed, such as when the stereoscopic display device is replaced from a very high precision glass lenticular sheet to a simple one made of plastic, the stereoscopic display device There arises a problem that crosstalk is likely to occur due to the difference in the performance of the optical elements constituting the. For this reason, when the same stereoscopic image is observed without adjusting the amount of presentation parallax, the viewer feels uncomfortable, such as making the viewer feel blurry images or overlapping images (overlapping images).

  Second, the parallax adjustment method proposed in Patent Document 2 cannot generate motion parallax peculiar to the multi-view system. In the method of Patent Document 2, parallax between viewpoint images is adjusted by relatively translating the positional relationship between presented stereo images (a pair of viewpoint images). However, such a simple method is not effective when a stereoscopic image is composed of three or more viewpoint images. This is because, in multi-view stereoscopic display, motion parallax is generated by moving the viewpoint, so that it is possible to create a stereoscopic image that does not cause discomfort to the observer while realizing natural changes in motion parallax. This is because it is necessary to change the viewpoint position itself of the viewpoint images constituting the stereoscopic image.

  The present invention automatically adjusts the viewpoint position of the viewpoint image constituting the stereoscopic image according to the stereoscopic display device to be used, and reduces the crosstalk at the time of observation of the stereoscopic image to enable natural stereoscopic observation. One of the purposes is to generate an image.

The stereoscopic image generation program according to one aspect of the present invention provides a first size information related to a size of the first viewpoint image from a plurality of first viewpoint images and a parallax related to a parallax between the plurality of first viewpoint images. A first information acquisition step for acquiring information, a second information acquisition step for acquiring device information relating to a stereoscopic display device used for observation of a stereoscopic image, the first size information, the parallax information, and the device A viewpoint determination step for determining a plurality of viewpoints corresponding to the stereoscopic display device based on the information, and a plurality of second viewpoints corresponding to the plurality of viewpoints determined in the viewpoint determination step using the first viewpoint image. Generating a second viewpoint image, and causing the computer to execute an image generation step of generating the stereoscopic image using the plurality of second viewpoint images. To.

A stereoscopic image generation program according to another aspect of the present invention includes a three-dimensional model information of an object, parameter information related to shooting parameters of a virtual camera virtually arranged at a viewpoint corresponding to a viewpoint image, and a view of the virtual camera. A first information acquisition step for acquiring first size information regarding size; a second information acquisition step for acquiring device information regarding a stereoscopic display device used for observation of a stereoscopic image; the parameter information; A viewpoint determination step for determining a plurality of viewpoints corresponding to the stereoscopic display device based on one size information and the device information; and a plurality of viewpoints determined in the viewpoint determination step using the three-dimensional model information A plurality of viewpoint images corresponding to the image and generating the stereoscopic image using the plurality of viewpoint images. Characterized in that to execute a flop, to the computer.
  A stereoscopic image generation system according to another aspect of the present invention relates to first size information relating to a size of the first viewpoint image from a plurality of first viewpoint images, and parallax between the plurality of first viewpoint images. A first information acquisition unit that acquires parallax information; a second information acquisition unit that acquires device information relating to a stereoscopic display device used for stereoscopic image observation; the first size information; the parallax information; Based on device information, viewpoint determining means for determining a plurality of viewpoints corresponding to the stereoscopic display device, and using the first viewpoint image, a plurality of viewpoints corresponding to the plurality of viewpoints determined by the viewpoint determining means. Image generating means for generating a second viewpoint image and generating the stereoscopic image using the plurality of second viewpoint images.
  A stereoscopic image generation system according to another aspect of the present invention includes a three-dimensional model information of an object, parameter information regarding shooting parameters of a virtual camera virtually arranged at a viewpoint corresponding to a viewpoint image, and a view of the virtual camera. First information acquisition means for acquiring first size information relating to size, second information acquisition means for acquiring device information relating to a stereoscopic display device used for observation of a stereoscopic image, the parameter information, the first information Viewpoint determining means for determining a plurality of viewpoints corresponding to the stereoscopic display device based on one size information and the device information; and a plurality of viewpoints determined by the viewpoint determining means using the three-dimensional model information Image generating means for generating a plurality of viewpoint images corresponding to the three-dimensional image and generating the stereoscopic image using the plurality of viewpoint images. To.
  A stereoscopic image generation method according to another aspect of the present invention relates to first size information relating to a size of the first viewpoint image from a plurality of first viewpoint images, and parallax between the plurality of first viewpoint images. A first information acquisition step of acquiring parallax information; a second information acquisition step of acquiring device information relating to a stereoscopic display device used for observation of a stereoscopic image; the first size information; the parallax information; A viewpoint determination step for determining a plurality of viewpoints corresponding to the stereoscopic display device based on device information, and a plurality of viewpoints corresponding to the plurality of viewpoints determined in the viewpoint determination step using the first viewpoint image. An image generation step of generating a second viewpoint image and generating the stereoscopic image using the plurality of second viewpoint images.
  A stereoscopic image generation method according to another aspect of the present invention includes three-dimensional model information of an object, parameter information related to shooting parameters of a virtual camera virtually arranged at a viewpoint corresponding to a viewpoint image, and a view of the virtual camera A first information acquisition step for acquiring first size information regarding size; a second information acquisition step for acquiring device information regarding a stereoscopic display device used for observation of a stereoscopic image; the parameter information; A viewpoint determination step for determining a plurality of viewpoints corresponding to the stereoscopic display device based on one size information and the device information; and a plurality of viewpoints determined in the viewpoint determination step using the three-dimensional model information Generating a plurality of viewpoint images corresponding to the image, and generating the stereoscopic image using the plurality of viewpoint images; Characterized in that it has.

  According to the present invention, even if the stereoscopic display device is changed, the viewpoint positions of the plurality of second viewpoint images that are the sources of the stereoscopic images are automatically adjusted, so that the natural stereoscopic with less crosstalk during observation is obtained. A stereoscopic image capable of presenting a feeling can be generated.

  Embodiments of the present invention will be described below with reference to the drawings.

  FIG. 2 shows a configuration of a stereoscopic image generation system that is Embodiment 1 of the present invention. In FIG. 2, 201 is an image input device, and an input stereo image composed of a plurality of images (first viewpoint image) relating to the same subject photographed from different viewpoints is converted into a stereoscopic image composition device (image generation means) described later. ) 202. The image input device 201 is an input / output device that reads an image file recorded on a fixed recording medium such as a CF card (registered trademark) or smart media (registered trademark), a digital camera having an input / output interface, a digital video camera, or the like. Image input devices that digitize and read image information from analog media, such as imaging devices, scanners, and film scanners.

  Further, the image input device 201 can be connected to a network, and a stereo image that is captured and stored and transmitted by a remote imaging device can be input to the stereoscopic image composition device 202.

  The stereoscopic image synthesis device 202 selects at least two viewpoint images from the input stereo images, generates virtual viewpoint images as necessary, and synthesizes stereoscopic image data. The stereoscopic image synthesizing device 202 is constituted by, for example, a general-purpose personal computer.

  Reference numeral 203 denotes an operation display for displaying information necessary for stereoscopic image synthesis in an interactive procedure with the user, assisting the acquisition, and displaying the processing status of the stereoscopic image synthesis process. To do. Furthermore, it is possible to display a menu screen and a viewpoint image group constituting a stereoscopic image. The display 203 is configured by a computer display or the like.

An operation input unit 204 includes a mouse, a keyboard, a joystick, and the like, and is used by the user to select a menu while viewing a menu image displayed on the display 203 .

  An input / output unit 205 is connected to an external device such as the stereoscopic display 210 or another computer, and outputs stereoscopic image data and the like as a file. Here, the stereoscopic display device 210 illustrated in FIG. 2 includes a display unit 210a configured by a CRT, an LCD display, and the like, and each of the stereoscopic images displayed on the display unit 210a that are disposed on the front surface of the display unit 210a. It is composed of an optical member 210b such as a lenticular plate and a parallax barrier that guides the light flux from the viewpoint image to the left and right eyes E of the observer arranged at a predetermined viewpoint.

Incidentally, the said input output unit 205, an Ethernet (registered trademark) network connected network ports and floppy such (registered trademark) disk, MO (registered trademark), ZIP (Noboru Rokusho mark), such as a CD-ROM You may comprise with a fixed recording medium. The input / output unit 205 may be configured by the same device as the image input device 201. In this case, the image input device 201 and the input / output unit 205 can be shared.

  The printer 206 prints the stereoscopic image synthesized by the stereoscopic image synthesis apparatus 202. By overlaying an optical member as a stereoscopic display device such as a lenticular plate on the printed stereoscopic image, a stereoscopic image can be observed.

  Next, the internal configuration of the stereoscopic image synthesis apparatus 202 will be described. Reference numeral 2021 denotes a central processing unit as a controller that controls the entire apparatus, and is hereinafter referred to as a CPU.

Reference numeral 2022 denotes a large-capacity storage device that stores a stereo image read from the image input device 201 or the like, a generated virtual viewpoint image or stereoscopic image data, or is empirically related to the stereoscopic display 210 or the number of images. The parameter value of the display parallax range set analytically is stored. The mass storage device 2022 is constituted by a hard disk or the like.

  Reference numeral 2023 denotes a main storage device, which includes a RAM and the like. The main storage device 2023 expands a stereo image read from the image input device 201 or a stereo image stored in the large-capacity storage device 2022 on the main storage area, generates parameters for generating a stereoscopic image, and the like. The virtual viewpoint image is temporarily stored for generating a stereoscopic image. Further, the main storage device 2023 stores the generated stereoscopic image data in the large-capacity storage device 2022, outputs it from the input / output unit 205, or temporarily stores it before printing by the printer 206.

  Next, a processing flow of the stereoscopic image synthesizing apparatus 202 (mainly the CPU 2021) in this embodiment will be described with reference to FIG. This process is executed according to a stereoscopic image synthesis program that is a computer program stored in the CPU 2021.

In step S <b> 101, the CPU 2021 inputs a stereo image from the image input device 201 and accumulates it in the mass storage device 2022. The plurality of viewpoint images (first viewpoint images) constituting the input stereo image are, for example, as shown in FIG. 3, with respect to the stationary subjects 303 to 305, It is obtained by photographing the subjects 303 to 305 while changing the viewing direction (imaging direction). In FIG. 3, the imaging device 301 is a digital camera or a video camera. 302 shows the IMAGING device 301 after changing the viewpoint and viewing direction. Instead of shooting while moving the imaging device in this way, a stereo camera is used to take a stereo image using a multi-lens camera of two or more eyes, or a so-called stereo adapter is attached to or incorporated in the camera's shooting optical system. May be used.

  In the present embodiment, it is assumed that parallel vision processing for eliminating geometric distortion between viewpoint images is performed on a plurality of viewpoint images that are input stereo images. Parallel vision processing is a so-called parallel vision arrangement (so that the camera optical axis directions of the cameras at a plurality of positions are parallel to each other in order to simply perform the geometric arrangement between the viewpoint images and various processes between the viewpoint images. Is a processing method that performs line-of-sight transformation on the viewpoint image group so that the arrangement is equivalent to that taken with a camera that has been placed in a different position), such as “Multiple View Geometry, Hartley and Zisserman, Cambridge Press, (2000)”, etc. This is a known technique introduced in.

  In step S102, information on the size of the stereo image (each viewpoint image) and information on the maximum value and the minimum value (existing parallax range information) on the images of the parallax occurring between the viewpoint images are acquired. First, in order to acquire a stereo image size, a reference view range used for generating a stereo image is set. The view range is adjusted to the aspect ratio of the output image size set in the next step S103 so that the image is not distorted, or an unnecessary subject included in the captured image in the shooting of the stereo image is excluded. It is set by limiting the scene area of the stereo image and narrowing the range of the parallax included in the stereo image. Further, the angle of view of the input stereo image may be set as the input stereo image size without setting the view range.

  In addition, in order to obtain the presence parallax range, stereo matching is performed between a plurality of viewpoint images constituting the input stereo image, thereby obtaining correspondence information of individual pixels between the viewpoint images, and the difference between the correspondence information. To calculate the maximum value and the minimum value of the parallax occurring between the viewpoint images to obtain the existing parallax range.

  Here, an outline of stereo matching will be described with reference to FIG. There are various methods for stereo matching, such as region-based methods, feature-based methods, and gradient-based methods, but any method that can find the parallax for each pixel in the two viewpoint images to be matched. Any method may be used.

  In the present embodiment, a case where a simple template matching method using a difference sum, which is a region-based method, is used for feature point association will be described. Here, of the two viewpoint images, the left image corresponding to the left side of the captured viewpoint position is set as the reference image 501, the right image corresponding to the right side of the viewpoint position is set as the reference image 502, and template matching is performed using the left image as a reference. I will do it.

  In this case, first, a certain feature point in the left image is selected. This point is referred to as an attention point (attention pixel) 504. Then, a region having a predetermined size centered on the attention point 504 is cut out as a template 503.

Next, the corresponding point search area 507 in the reference image 502 is arbitrarily determined in consideration of the amount of movement of the rough viewpoint and the change in the viewpoint and line-of-sight direction due to camera shake. Then, reference points 505 that are feature points existing in the corresponding point search area 507 are sequentially selected, and a window area 506 having the same size as the template 503 around the reference point 505 is selected in the standard image 501 . The correlation value with the template 503 is obtained.

  For example, when the camera moves horizontally, the corresponding point search area 507 is usually given as a rectangular area having long sides in the horizontal direction of the image coordinates.

  The corresponding point in the reference image corresponding to the point of interest is determined as the reference point having the largest correlation value. However, when the correlation value at the corresponding point is smaller than the predetermined value, when the difference between the correlation value at the corresponding point and the second smallest correlation value is smaller than the predetermined value, or when the change of the correlation value near the corresponding point is predetermined. If the value is smaller than the value, it is considered that the reliability of the corresponding point search process is low, so that point is not determined as the corresponding point.

  By the stereo matching process as described above, the correspondence information of the right image based on the left image is obtained. The above association is also performed for other feature points in the left image.

  Although the case where the left image is set as the reference image has been described here, the same processing may be performed using the right image as the reference image and the left image as the reference image. In addition, using the symmetry of the corresponding point information obtained by performing both the processing based on the left image and the processing based on the right image, the result is such that the correspondence between the feature points becomes 1: 1. May be corrected.

  Difference information of corresponding point information obtained by stereo matching is calculated as a parallax value. For example, when the left image is considered as the reference image, the parallax is represented by a positional deviation (parallax) vector between each pixel (each pixel of interest) of the left image and a corresponding point in the right image. The disparity vector is

It is obtained by.

  In the present embodiment, a case where a stereoscopic display device having an optical member capable of expressing only one-dimensional parallax change, such as a lenticular lens, will be described as the stereoscopic display device 210. Specifically, as the parallax, attention is paid only to the parallax in the one-dimensional direction, which represents a positional deviation in the horizontal direction with respect to each pixel of the left image from the position of the corresponding point of the right image. Therefore, the disparity vector is represented by a scalar value (disparity value). However, when a device that expresses two-dimensional parallax, such as integral photography, is used as the stereoscopic display device 210, the parallax is treated as a vector, and vertical positional deviation is also considered.

  The minimum value and the maximum value of the parallax between the viewpoint images obtained in this way are acquired, and the range represented by the minimum parallax value and the maximum parallax value is set as the existing parallax range of the input stereo image. However, the maximum value and the minimum value of the parallax are obtained only within the selected view range. Therefore, the presence parallax range of the input stereo image changes as the view range changes. As described above, in step S102, the input stereo image size and the existing parallax range are acquired.

  In step S103, the user is made to select a stereoscopic display device via the operation input unit 204, and the size of a stereoscopic image that is finally output corresponding to the selected stereoscopic display device is designated.

  In this embodiment, as shown in FIG. 5A, a stereoscopic display having a glass lenticular plate and a plastic lenticular plate for stereoscopic viewing of a stereoscopic image printed by the printer 206, a CRT, an LCD, etc. 15-inch, XGA size display A using a parallax barrier and 15-inch, XGA size display B using a lenticular plate, etc. The configuration, display format, number of viewable viewpoint images (number of images), stereoscopic image Assume that devices having different device information regarding specifications such as the size of the displayable area and the performance of the individual optical elements constituting the device can be selected as the stereoscopic display device.

  In addition, as schematically shown in the stereoscopic display device 210 in FIG. 2, the stereoscopic image is a pixel in which the pixel constituting the pixel vertical line at one end in the left-right direction displays the viewpoint image 1 and the pixel constituting the next pixel vertical line. Displays the viewpoint image 2, and further displays the viewpoint image 3 as the pixels constituting the next image vertical line, so that the pixel vertical that is part of the viewpoint images 1 to n (n = 5 in FIG. 2) is displayed. Lines (stripe images) are arranged in a circular manner in the horizontal direction. That is, the stereoscopic image of the present embodiment is synthesized as one image by dividing each viewpoint image into vertical lines of one pixel and arranging the divided stripe images in a cycle reverse to the order of the viewpoints. It is an image. Hereinafter, such a stereoscopic image is referred to as a striped stereoscopic image. A specific method for synthesizing the striped stereoscopic image will be described later.

  However, the format of the stereoscopic image is not limited to this, and the other is the one in which the pixels of each viewpoint image are circularly arranged according to the viewpoint arrangement in the horizontal direction and the viewpoint circulation position is shifted for each pixel horizontal line. It may be of the form

  Then, by selecting a device to be used for observing a stereoscopic image from among the stereoscopic display devices, based on the device information of the device, between two adjacent viewpoint images in the stereoscopic image corresponding to the device. A display parallax allowable amount is determined.

  In this embodiment, the stereoscopic display device to be used is selected interactively using the operation display 203 and the operation input unit 204 as shown in FIG.

Display parallax allowable amount corresponding to each of the specifications of the stereoscopic display device, in advance, experimentally determined or analytically, the individual optical elements stereoscopic display device has (e.g., lenticular lenses constituting the lenticular plate) Calculated based on the resolution.

  The resolution of each optical element of the stereoscopic display device is stored as a database in the large-capacity storage device 2022. In addition, when the stereoscopic display device 210 is selected from the operation input unit 204, it may be input as a parameter.

  For example, when a stereoscopic display device is interactively selected from a menu, a display parallax allowable amount corresponding to the stereoscopic display device is given as follows.

  First, whether the display parallax allowable amount is limited to a range determined by a value resulting from the user's adjustment and binocular parallax mismatch, as in the case of the conventional binocular system, or the like of the optical element constituting the stereoscopic display device A procedure for determining whether or not to limit within a range determined by a value caused by crosstalk caused by performance is performed.

  First, the width p of each viewpoint image constituting the stereoscopic image is obtained from the parameter (pitch width P of the optical element) of the selected stereoscopic display device and the image number n.

p = P / n (2)
Next, the width p of each viewpoint image and the resolving power r _exp of each optical element of the stereoscopic display device are compared. Each parameter corresponds to P, p, r _exp in FIG. In FIG. 13A, n = 2.

However, r _exp should be set to a value smaller than the actually measured value in consideration of the margin when comparing. Further, when a stereoscopic image is observed at an observation position outside the optimum observation distance, it is better to set r _exp as a smaller value.

When the width p of each viewpoint image constituting the stereoscopic image is wider than the resolving power r _exp of each optical element constituting the stereoscopic display device, the adjustment by the observer and the binocular are performed as in the conventional binocular system. The display parallax allowable amount is set in a parallax range determined by a value caused by the parallax mismatch, that is, a range called a fusion range.

  The amount of parallax in the fusion range has been studied for a long time in the shooting of stereoscopic images on the premise of a twin-lens system, and is described in documents such as “about 3D video design (Kumada Noriaki, Broadcasting Technology, 1999.11)”. A method for setting the amount of parallax is shown.

On the contrary, when the width p of each viewpoint image constituting the stereoscopic image is narrower than the resolution r _exp of each optical element constituting the stereoscopic display device, the parallax between the viewpoint images presented to the left and right eyes Rather than the problem of whether the amount is within the fusion range of the observer, light from multiple adjacent viewpoint images is incident on one eyeball due to crosstalk even when the optimal distance between viewpoints is taken. The effect of the problem is greater. When light from a plurality of adjacent viewpoint images is incident on one eyeball at the same time, a double contour or the like caused by an image shift due to a parallax amount between viewpoint image images leads to a strong discomfort to the observer. The number of overlapping images is determined based on the relationship between the resolving power r _exp and the pitch width p of the viewpoint image, and the parallax tolerance is determined as follows.

  However, when a stereoscopic image is observed at an observation position far from the optimum observation distance, crosstalk with an adjacent image gradually becomes a problem, as can be seen from the distribution of light beams shown in FIG. For this reason, stereoscopic image observation at an observation position greatly deviating from the optimum observation distance is assumed, and when the crosstalk caused by the deviation from the optimum observation distance of the actual observation position is regarded as important for the crosstalk, particularly a stereoscopic display device It is not necessary to consider the resolving power of the individual optical elements constituting the.

  Instead of reading out the resolving power of each optical element of the stereoscopic display device from the database, the number of multiple images may be determined directly according to the degree of deviation from the optimum observation distance.

  When the display parallax tolerance is determined by crosstalk caused by the performance of the optical elements that make up the stereoscopic display device, how much is the stereoscopic image viewed at the optimal viewing distance using each stereoscopic display device? The number of images can be observed at one time, and how much deviation between the viewpoint images is allowed when a plurality of viewpoint images are presented at one time.

  FIG. 6 is a diagram illustrating the influence of crosstalk that occurs during stereoscopic observation of a certain stereo image. (A) is an input stereo image, (b) is an image affected by crosstalk that is seen when a stereoscopic image generated from the input stereo image of (a) is observed through a stereoscopic display device. Further, (c) is a diagram showing the influence of the crosstalk of (b). (C) shows that three adjacent viewpoint images are incident on an eye arranged at a single viewpoint. That is, as shown in (b), a multiple image having a shift due to the influence of parallax is observed.

  In (b), reference numeral 1201 denotes a shift amount on the display surface of the stereoscopic image. The influence of multiple images is related to the amount of parallax between viewpoint images observed at the same viewpoint at the same time. That is, the allowable amount of display parallax is determined depending on the extent to which the amount of misalignment of the multiple images can be allowed.

  Further, as is apparent from the relationship between the change in the number of viewpoint images constituting the stereoscopic image shown in FIG. 13 and the light flux corresponding to each viewpoint image emitted from the stereoscopic display device, the individual optical characteristics of the stereoscopic display device are as follows. The display parallax allowable amount is inversely proportional to a certain number of images or more with respect to the number of images inserted into the element. That is, when the number of images increases, the display parallax tolerance decreases as the number of images increases. For example, when the number of images doubles, the display parallax tolerance between adjacent viewpoint images is approximately halved.

  Since the allowable amount of the influence of the multiple images is a subjective amount, the display allowable parallax amount for each stereoscopic display device is determined by experimentally changing the parallax range between the number of images and the adjacent viewpoint images. However, the display allowable parallax amount is also affected by the fineness of the texture of the part having parallax. That is, the allowable parallax amount on the sinking side of a distant view in which the texture is blurred can be made large in a normal case, and conversely, the parallax value on the pop-out side where the texture clearly appears is limited to a small value.

  Next, selection (designation) of the size of the stereoscopic image observed by the stereoscopic display device will be described. First, when a stereoscopic display device is selected, information on a displayable area of the stereoscopic display device is acquired, and a stereoscopic image size is determined within a range within the displayable area. When the relationship between the selected stereoscopic display device and its displayable area is a one-to-one relationship as in a normal stereoscopic display device, it is not necessary to repeatedly specify the stereoscopic image size.

  When a lenticular plate or the like that is used by being superimposed on a print image is selected as the stereoscopic display device, the stereoscopic display device and the printable area are naturally independent. Therefore, the size of the effective print area determined by selecting the printer and the paper to be printed becomes the size of the displayable area.

  Normally, for the selection of the stereoscopic display device and the designation (selection) of the stereoscopic image size, as shown in FIG. 5A, the display 203 and the operation input unit 204 of the stereoscopic image generation system in this embodiment are used. , Can be selected interactively at once.

  Next, the stereoscopic image size is determined within the displayable area. The displayable area may be used as the stereoscopic image size as it is, but if the aspect ratio of the input viewpoint image (input stereo image) and the displayable area size is different, it is presented when the stereoscopic image is displayed. The image will be distorted. Therefore, if you want to ensure the aspect ratio of the input stereo image size and the size of the displayable area, or if you do not want to display a 3D image to the limit of the displayable area, set the margin area to determine the 3D image size. To do.

  FIG. 5B shows, as a specific example, how the stereoscopic image size is changed in the displayable area in the case of printing. Reference numeral 601 denotes a displayable area, reference numeral 602 denotes a stereoscopic image size before a certain size editing, and reference numeral 603 denotes a stereoscopic image size after a certain size editing.

  Note that the input stereo image size and the existing parallax range are acquired in step S102, and the stereoscopic image size is selected in step S103, and independent parameters are obtained, so the order of these processes is reversed. Also good.

In step S104, the position of the viewpoint (hereinafter referred to as virtual viewpoint) of the viewpoint image (second viewpoint image, hereinafter referred to as virtual viewpoint image) newly generated from the input stereo image is determined. The virtual viewpoint position indicates a shooting position by a camera (virtual camera) that has virtually shot a virtual viewpoint image. In this embodiment, when a virtual viewpoint image is converted into a stereoscopic image, the parallax between adjacent viewpoint images in the entire area of the stereoscopic image presented through the stereoscopic display device 210 is the maximum within a range that does not exceed the display parallax allowable amount . The virtual viewpoint position is determined under the condition of parallax. Specifically, as described below, after determining a reference virtual viewpoint position, other virtual viewpoint positions are sequentially determined under the above conditions.

  First, using the size of the input stereo image obtained in step S102 and the size of the stereo image obtained in step S103, a size ratio between the input stereo image and the output stereo image, that is, a scaling factor is obtained.

Next, using the obtained scaling factor, the existing parallax range of the reference viewpoint image obtained in step S102, and the display parallax allowable amount between adjacent viewpoints obtained in step S103, the generation is performed in step S105. A virtual viewpoint position corresponding to the virtual viewpoint image to be determined is determined.

Here, first by integrating the inverse of the scaling factor to display parallax allowable amount between virtual viewpoint adjacent to determine the range of the parallax tolerance between adjacent viewpoint corresponding to the scale of the input stereo images.

Next, a reference virtual viewpoint is determined. Although the position of the reference virtual viewpoint may be any position, in this embodiment, an intermediate point on the base line connecting the viewpoints of the input stereo image is set as the reference virtual viewpoint. Then, taking the displacement of the virtual viewpoint position in a range not exceeding the parallax allowable amount between adjacent viewpoints corresponding to the scale of the stereo image, successively, it will determine the other virtual viewpoint position.

  FIG. 7 shows the relationship between the amount of parallax between input stereo images and the amount of parallax of a newly generated virtual viewpoint image. The viewpoints of the input stereo images 1001 and 1002 are connected to the parallax amount calculated between the input stereo images (here, two viewpoint images) 1001 and 1002 converted into a parallel view arrangement due to the geometrical relationship. When the virtual viewpoint image 1003 having the same viewing direction as the input stereo images 1001 and 1002 is generated at the virtual viewpoint position on the straight line, the parallax between the generated virtual viewpoint image and one viewpoint image of the input stereo images The value is uniquely determined from the relative positional relationship.

  That is, if the distance between the viewpoints of the input stereo image is S, and the viewpoint of the virtual viewpoint image 1003 is at the position t with respect to one viewpoint image 1001 of the input stereo image, the one viewpoint The disparity value between the image 1001 and the virtual viewpoint image 1003 is

Can be obtained in the form of

  D (x, y) in the equation (3) is obtained by associating the parallax value obtained by the equation (1) with the coordinate value of one viewpoint image. Here, it is a parallax value obtained between one viewpoint image 1001 and the other viewpoint image 1002 as a reference. D ′ (x, y) corresponds to a parallax value generated between the viewpoint image 1001 and the virtual viewpoint image 1003 as a reference.

  When the line-of-sight direction of the virtual viewpoint image to be generated is different from the line-of-sight direction of the input stereo image, the displacement of the corresponding point position is obtained by obtaining a parallax value in parallel viewing arrangement with the input stereo image and then performing line-of-sight conversion It is necessary to calculate each pixel position of the generated virtual viewpoint image by adding to the parallax.

The virtual viewpoint position is calculated while taking the displacement of the virtual viewpoint position within a range in which the parallax value between the adjacent virtual viewpoint images obtained in this way is within the allowable parallax amount between adjacent viewpoints on the scale of the input stereo image. Determine sequentially.

FIG. 8A shows the virtual viewpoint position determined when the display parallax allowable amount is small, and FIG. 8B shows the virtual viewpoint position determined when the display parallax allowable amount is large.

Reference numeral 901 denotes one viewpoint image of the input stereo images, and reference numeral 902 denotes the other viewpoint image. Reference numerals 903 and 904 denote virtual viewpoint groups in the cases of FIGS. 8A and 8B, respectively. Reference numeral 905 denotes a reference position of the virtual viewpoint taken at the midpoint connecting the two viewpoints of the input stereo image. The positions of the virtual viewpoint groups 903 and 904 are determined so as to be evenly distributed in the horizontal direction around the reference virtual viewpoint. Of course, the imaginary viewpoint group extent when the display parallax allowable amount of (a) is small, becomes larger towards the virtual viewpoint groups spread when larger display parallax allowable amount of (b).

  In step S105, a virtual viewpoint image corresponding to the virtual viewpoint determined in the previous step S104 is generated.

In this embodiment, a virtual viewpoint image is generated directly from an input stereo image using the principle of view morphing (View Morphing, Steven M. Seitz, CR Dyer, Proc. SIGGRAPH 96 (1996)) . Here, the case of using the virtual viewpoint generation method proposed in Japanese Patent Laid-Open No. 2001-346226 will be described with reference to the flowchart of FIG.

  In step S1101, the disparity information in the existing disparity range obtained in S102 is interpolated so that the disparity value corresponds to each pixel of the input stereo image, and a disparity map that is dense disparity distribution information is calculated. If you want to reproduce the effects of changes in the viewpoint position in virtual viewpoint image generation more precisely, you can re-execute stereo matching called hierarchical matching or adaptive matching to obtain more accurate parallax values between viewpoint images. .

  In addition, with respect to the interpolation of disparity information, in this embodiment, the weighted average is performed using the calculated coordinate position of the non-corresponding point to be interpolated and the distance parameter at the corresponding point position for which the disparity value is obtained by the corresponding point extraction process as a weight, The parallax at the position of the uncorresponding point for which is not obtained is obtained by the following equation (4).

  Here, n is an arbitrarily set parameter that influences the interpolation result. If the value of n is too small, the parallax to be obtained is close to the average from the entire image, and the parallax distribution becomes uniform. If the value of n is too large, it is greatly affected by nearby corresponding points. In consideration of calculation time, n = 1 is desirable.

  Through the above processing, dense parallax distribution information for the left image, that is, a parallax map is obtained. A dense parallax distribution corresponding to the right image is obtained by exchanging the coordinate positions (x, y) and (x ′, y ′) in the equation (4).

  Next, in step S1102, warping processing is performed to generate virtual viewpoint images corresponding to the respective virtual viewpoint positions that are relative positions to the input stereo image obtained in step S104.

  Specifically, an image of the virtual viewpoint can be generated by warping each pixel of the input stereo image as a warp source in accordance with the warp amount of each pixel obtained by the equation (4). A virtual viewpoint image corresponding to the determined virtual viewpoint is obtained by performing post-processing such as occlusion processing and smoothing on the generated virtual viewpoint image.

  In step S1103, line-of-sight conversion is performed when the line-of-sight direction of the generated virtual viewpoint image is desired to be different from the line-of-sight direction of the input stereo image after parallel vision processing. Specifically, a line-of-sight conversion image is obtained by calculating a line-of-sight matrix and applying it to the image.

  By performing such a procedure for each determined virtual viewpoint, a virtual viewpoint image corresponding to each virtual viewpoint is generated.

  In this embodiment, a virtual viewpoint for generating a virtual viewpoint image necessary for synthesizing a three-dimensional image from an input stereo image, a size information of the input stereo image, and a device related to specifications such as the type of stereoscopic display device and the number of images A stereoscopic image having an appropriate parallax is generated by generating a virtual viewpoint image corresponding to the determined virtual viewpoint based on the information and the display size information of the stereoscopic image and then generating the virtual viewpoint image based on the input stereo image . However, the method itself for generating the virtual viewpoint image from the input stereo image is not limited.

  As proposed in Japanese Patent Application Laid-Open No. 9-121370, which is a prior art, it is presented to the left and right eyes of the observer by simple affine processing of viewpoint images, such as shifting the relative position between already existing viewpoint images. When adjusting the parallax range between the viewpoint images, the change in the amount of parallax that is presented only affects the amount of projection and the amount of sinking of the solid so as to translate approximately vertically to the tube surface. In particular, as in the case of central vision, there is a large difference in influence when the line-of-sight directions between viewpoint images are not parallel to each other due to the geometric relationship. In such a parallax adjustment method, an incorrect correction in the geometric relationship is performed for the change in parallax, and thus the parallax change is distorted when the adjusted viewpoint image is observed.

  On the other hand, as in the present embodiment, for example, a view morphing method is used to generate a virtual viewpoint image, and a virtual viewpoint image is newly generated for parallax adjustment of the image, thereby FIG. , (B), it is possible to generate a virtual viewpoint image group that can reproduce motion parallax more favorably with a wider viewpoint interval, that is, a base length between virtual viewpoints. Further, even when the line-of-sight directions between the viewpoint images are not parallel as in the case of central vision, the viewpoint image group for stereoscopic image synthesis is performed by performing a line-of-sight conversion process that changes the line-of-sight direction after the virtual viewpoint image generation process. As a result, it is possible to finally generate a virtual viewpoint image group that provides a natural stereoscopic effect without distortion in the parallax change.

Next, in step S106 in FIG. 1, the virtual viewpoint image group generated in step S105 is synthesized to generate a stereoscopic image. The generated stereoscopic image is output to the printer 206 for printing, or output to the operation display 203 for display. In some cases, the image data is output from the input / output unit 205 as stereoscopic image data.

  Here, as shown in FIG. 2, as the stereoscopic display device 210, a lenticular plate (optical member) 210b is arranged on the front surface of a display unit 210a that displays a stereoscopic image, and the above-described striped stereoscopic image is displayed on the display unit 210a. A description will be given of the synthesis of the striped stereoscopic image.

  The stripe-type stereoscopic image is synthesized by arranging pixels at the same coordinates of each image in the generated virtual viewpoint image group in the reverse order to the viewpoint arrangement.

First, assuming that the pixel value of the virtual viewpoint image corresponding to the viewpoint j is P _jmn (where m and n are indices of the pixel array in the horizontal and vertical directions, respectively), the virtual viewpoint image has the following pixel value data. It is represented by a two-dimensional array.

  Next, each virtual viewpoint image is decomposed into stripes for each line extending in the vertical direction, and the stripe images are arranged by the number of viewpoints in the order opposite to the actual viewpoint arrangement order. Therefore, the combined image is a stripe image as shown below.

  However, the above arrangement represents a stereoscopic image corresponding to the positional relationship where the viewpoint 1 is the left end and N is the right end. Here, the reason why the arrangement order of the stripe images is reversed from the arrangement order of the viewpoints is that the images are observed to be reversed left and right within one pitch of the lenticular lens constituting the lenticular plate 210b. This striped stereoscopic image has a size of H × v in each virtual viewpoint image, and has a size of X (= N × H) × v when N virtual viewpoint images are included.

  Next, pitch adjustment (scaling process) with the lenticular plate 210b is performed on the striped stereoscopic image. Since one pitch of RP (dpi) pixels exists for N pixels, one pitch is N / RP (inch). However, since the pitch of the lenticular lens on the lenticular plate 210b is RL (inch), the image is horizontal. The pitch is adjusted by multiplying RL × RP / N in the direction.

  In addition, the number of pixels in the vertical direction needs to be (RL × RP / N) × Y pixels when it is desired to preserve the aspect ratio, and therefore (RL × RP × Y) / (N × v) in the vertical direction. It becomes the size.

  Here, RP (dpi) is the printing resolution, XP × YP is the printing size, X (RP × XP) × Y (RP × YP) (pixel) is the size of the image to be printed, and RL (inch) is the lenticular plate 210b. Is the pitch of the lenticular lens.

  In this way, the scaling process is performed on the stereoscopic image, and the final stereoscopic image is displayed on the stereoscopic display device 210. The scaling process is performed by, for example, bilinear interpolation. When a stereoscopic image is printed by the printer 206, an optical member such as a lenticular plate is superimposed on the printed stereoscopic image for observation.

  According to the present embodiment, the viewpoint position (virtual viewpoint position) in the generation of the virtual viewpoint image that is the generation source image of the striped stereoscopic image, the information such as the type and number of images of the stereoscopic display device, and the display size information of the stereoscopic image Automatically adjusts based on image quality, and generates a 3D image that has sufficient 3D effect and does not cause discomfort due to crosstalk during stationary observation or viewpoint movement observation. It becomes possible.

  In the above embodiment, a method for synthesizing a stereoscopic image of a pixel array corresponding to a stereoscopic display device using a lenticular plate has been described. However, the present invention is applicable to a so-called oblique lenticular sheet, parallax stereogram, holographic stereogram. The present invention can also be applied to a case where a stereoscopic image having a pixel arrangement corresponding to a stereoscopic display device using the above is synthesized.

  In the above embodiment, the case where only the virtual viewpoint image group newly generated from the input stereo image is used for the synthesis of the stereoscopic image has been described. However, the viewpoint corresponding to the viewpoint image in the input stereo image is a step. If the virtual viewpoint determined in S104 coincides, the viewpoint image included in the input stereo image may be included as a viewpoint image for generating a stereoscopic image.

  Furthermore, in the above embodiment, as a method for generating a virtual viewpoint image, a case has been described in which a virtual viewpoint image in which the line-of-sight direction is parallel is generated by a view morphing method in parallel viewing arrangement. Then, by adding affine processing such as line-of-sight conversion, a central viewpoint virtual viewpoint image group in which the line-of-sight direction of each virtual viewpoint image is directed toward a specific object in the scene may be generated. That is, the viewpoint position and the line-of-sight direction of the virtual viewpoint image may be arbitrary.

In the first embodiment, each time the display size or the type of stereoscopic display device or the number of images is changed using a stereo image as an input, the largest stereoscopic effect (parallax) within a range in which the viewer does not feel uncomfortable due to crosstalk. ), The viewpoint position of the virtual viewpoint image for generating the stereoscopic image is automatically adjusted to generate a stereoscopic image that gives an appropriate stereoscopic effect, but the input may be 3D model information. Is possible .
The configuration of the stereoscopic image generation system according to the present embodiment is basically the same as the system configuration according to the first embodiment. However, the processing in the stereoscopic image composition device 202 is changed as shown in the flowchart of FIG. This process is executed according to a stereoscopic image composition program that is a computer program stored in the CPU 2021 of the stereoscopic image composition apparatus 202.

  The three-dimensional model (information) referred to in the present embodiment is composed of polygons formed by vertex coordinates and vertex rows on the surface of an object, and a two-dimensional texture image representing attributes such as color and reflectance of the surface. This is information obtained by modeling a real object in three dimensions. By placing the three-dimensional model in a certain three-dimensional virtual space and changing the placement in the virtual space by performing operations such as movement, rotation, and enlargement on the three-dimensional model, a desired scene can be obtained. Can be built. In this way, in the virtual space where the three-dimensional model is arranged, a virtual viewpoint image can be generated by further rendering the image by installing a virtual camera at an arbitrary virtual viewpoint position.

  In step S1401, detailed three-dimensional model information including a three-dimensional model of one or a plurality of objects and arrangement information of the three-dimensional model in the virtual space is input from the input / output unit 205 with the external device, and the main storage device 2023 or the mass storage device 2022.

Further, when it is desired to change the shape and arrangement of the three-dimensional model is input, displays the arrangement of the three-dimensional model in the virtual space on the operation display 203, modified through the operation input unit 204, moving, rotating, inverting, Perform operations such as enlargement and reduction.

  In step S1402, the imaging parameters of the virtual camera installed in the virtual space and the view size on the imaging surface of the virtual camera are determined. By determining the shooting parameters of the virtual camera, the scale between the virtual space and the real space is determined. The shooting parameters of the virtual camera are internal parameters of the camera such as a shooting direction (gaze direction), a shooting magnification, and the like.

  In step S1403, as in step S103 of the first embodiment, the display parallax allowable amount is obtained from the device information of the selected stereoscopic display device, and the display stereoscopic image size is designated.

  In step S1404, the position of the virtual camera that captures the virtual viewpoint image, that is, the virtual viewpoint position is determined. First, based on the virtual camera shooting parameters and view size determined in step S1402, and the display parallax allowance and display stereoscopic image size acquired in step S1403, in the scale of the virtual viewpoint image generated at the viewpoint of the virtual camera. Then, the range of the allowable parallax amount between the virtual viewpoint images is obtained.

  Next, a virtual viewpoint position is determined in order to sequentially generate virtual viewpoint images. In FIG. 11, reference numeral 1501 denotes a reference viewpoint virtual camera, 1502 denotes a virtual camera after the viewpoint is moved, and 1503, 1504, and 1505 denote three-dimensional models in the virtual space.

  First, as in the first embodiment, the viewpoint position of the reference virtual camera is determined. In this embodiment, the viewpoint position 1501 in FIG. Next, the viewpoint position of the next virtual camera is determined using the viewpoint position 1501 as a reference. In the determination of the viewpoint position of the virtual camera, the virtual viewpoint image corresponding to the viewpoint position determined at the previous time point of the sequentially generated virtual viewpoint image (first reference viewpoint position 1501) and the next selected viewpoint The condition is that the amount of parallax with the virtual viewpoint image at the position does not exceed the allowable amount of parallax on the scale of the virtual viewpoint image.

  That is, first, the viewpoint position of the next virtual camera is arbitrarily selected. Then, the amount of parallax generated between the virtual viewpoint image generated at the reference viewpoint position is obtained. Furthermore, the parallax amount of the entire image is compared with the allowable parallax amount on the scale of the virtual viewpoint image. If the parallax amount between the generated virtual viewpoint images does not exceed the allowable parallax amount, the virtual viewpoint at the viewpoint position Allows selection as position. On the other hand, when the allowable parallax amount is exceeded, the viewpoint position of the virtual camera for one virtual viewpoint image is determined in such a manner that the viewpoint position of the next virtual camera is newly selected while moving the smaller viewpoint. To go.

  Further, the parallax information is obtained by installing each virtual camera in a virtual space as shown in FIG. 11, performing ray tracing to an object installed in the virtual space, and based on the shooting parameters and the view range. The parallax information with reference to one image between the two virtual viewpoint images taken by the above is obtained. In addition, since the geometric position information of the three-dimensional model and virtual camera installed in the virtual space is known, the disparity information may be partially obtained by geometric calculation.

  When the viewpoint position of one virtual viewpoint image is determined, the viewpoint position becomes the reference viewpoint position in determining the next viewpoint position. In this way, virtual viewpoint positions are sequentially determined, and virtual viewpoint positions that satisfy the number of virtual viewpoint images constituting the stereoscopic image are determined.

  In step S1405, the virtual viewpoint images corresponding to the virtual viewpoint position determined in step S1404 are sequentially rendered, thereby generating the required number of virtual viewpoint images for synthesizing the stereoscopic image.

  In step S1406, a stereoscopic image is synthesized from the virtual viewpoint image group and output to the stereoscopic display device 210, the printer 206, and the like.

  According to the present embodiment, the 3D model information is input, and each time the device information on the specifications such as the type of the stereoscopic display device and the number of viewpoint images (number of images) and the display stereoscopic image size are changed, the stereoscopic image is generated. Since the viewpoint position (virtual viewpoint position) in the generation of the virtual viewpoint image that is the original image is automatically adjusted, it has a sufficient three-dimensional effect and causes discomfort due to crosstalk during stationary observation and viewpoint movement observation. It is possible to generate a stereoscopic image that allows more natural stereoscopic observation.

  The present invention is not limited to the system configuration of each of the embodiments described above, and may be applied to a system constituted by a plurality of devices or a system constituted by one device.

The flowchart which shows the process of the stereo image production | generation system which is Example 1 of this invention. 1 is a diagram illustrating a configuration of a stereoscopic image generation system according to Embodiment 1. FIG. FIG. 3 is a diagram illustrating a method for capturing an input stereo image according to the first embodiment. FIG. 3 is a diagram for explaining a stereo matching method performed in the first embodiment. FIG. 6 is a diagram illustrating a state in which a stereoscopic display device and a stereoscopic image size are selected in the first embodiment. FIG. 6 is a diagram for explaining the relationship between the influence of crosstalk and the allowable amount of display parallax in Embodiment 1. FIG. 4 is a diagram illustrating a relationship between an input stereo image and a virtual viewpoint position in the first embodiment. FIG. 6 is a diagram for explaining a difference in virtual viewpoint position due to a difference in stereoscopic display devices in the first embodiment. 9 is a flowchart for explaining a virtual viewpoint image generation procedure according to the first embodiment. The flowchart which shows the process of the stereo image production | generation system which is Example 2 of this invention. FIG. 9 is a diagram for explaining generation of a virtual viewpoint image from a three-dimensional model in Embodiment 2. The figure which shows the example of a stereoscopic display device. The figure explaining the relationship between the change of the viewpoint image number which comprises a stereo image, and the light beam inject | emitted. The figure explaining the change of the emitted light beam by the shape of the optical element of a three-dimensional display device. The figure explaining the relationship between the optimal observation position of a stereo image, and generation | occurrence | production of crosstalk. The figure explaining the relationship between the change of the number of parallax images which comprise a stereo image, and the width | variety of the light beam which injects into an observation eye.

Explanation of symbols

DESCRIPTION OF SYMBOLS 201 Image input apparatus 202 Three-dimensional image composition apparatus 203 Operation display 204 Operation input part 205 Input / output part with external apparatus 206 Printer 210 Stereoscopic display device

Claims (11)

A first information acquisition step of acquiring, from a plurality of first viewpoint images, first size information relating to the size of the first viewpoint image and disparity information relating to parallax between the plurality of first viewpoint images;
A second information acquisition step of acquiring device information relating to a stereoscopic display device used for observation of a stereoscopic image;
A viewpoint determination step of determining a plurality of viewpoints corresponding to the stereoscopic display device based on the first size information, the parallax information, and the device information;
A plurality of second viewpoint images corresponding to a plurality of viewpoints determined in the viewpoint determination step are generated using the first viewpoint image, and the stereoscopic image is converted using the plurality of second viewpoint images. An image generation step to generate ;
It characterized by causing a computer to execute the standing body image generator. And that in said second information obtaining step obtains the second size information about the size of the three-dimensional image,
Determining the plurality of viewpoints based on the first and second size information, the parallax information, and the device information in the viewpoint determination step ;
3. The stereoscopic image generation program according to claim 1, wherein the computer is executed . First information for acquiring three-dimensional model information of an object, parameter information regarding shooting parameters of a virtual camera virtually arranged at a viewpoint corresponding to a viewpoint image, and first size information regarding a view size of the virtual camera An information acquisition step;
A second information acquisition step of acquiring device information relating to a stereoscopic display device used for observation of a stereoscopic image;
A viewpoint determination step of determining a plurality of viewpoints corresponding to the stereoscopic display device based on the parameter information, the first size information, and the device information;
Generating a plurality of viewpoint images corresponding to a plurality of viewpoints determined in the viewpoint determination step using the three-dimensional model information, and generating the stereoscopic image using the plurality of viewpoint images ; and
It characterized by causing a computer to execute the standing body image generator. And that in said second information obtaining step obtains the second size information about the size of the three-dimensional image,
Determining the plurality of viewpoints based on the parameter information, the first size information, the device information, and the second size information in the viewpoint determination step ;
4. The stereoscopic image generating program according to claim 3, wherein the computer is executed . Determining the plurality of viewpoints so that the maximum parallax is obtained in the parallax allowable range of the stereoscopic display device in the viewpoint determination step ;
5. The three-dimensional image generation program according to claim 1, wherein the three-dimensional image generation program according to claim 1 is executed . The stereoscopic image generation program according to claim 5, wherein in the viewpoint determination step, the computer is caused to determine the plurality of viewpoints based on an allowable amount of parallax of the stereoscopic display device. The stereoscopic image generation program according to claim 5, wherein in the viewpoint determination step, the computer executes the determination of the plurality of viewpoints based on the resolution of the stereoscopic display device. First information acquisition means for acquiring, from a plurality of first viewpoint images, first size information relating to the size of the first viewpoint image and disparity information relating to parallax between the plurality of first viewpoint images;
Second information acquisition means for acquiring device information relating to a stereoscopic display device used for observation of a stereoscopic image;
Viewpoint determining means for determining a plurality of viewpoints corresponding to the stereoscopic display device based on the first size information, the parallax information, and the device information;
A plurality of second viewpoint images corresponding to a plurality of viewpoints determined by the viewpoint determination unit are generated using the first viewpoint images, and the stereoscopic image is converted using the plurality of second viewpoint images. A stereoscopic image generation system comprising: an image generation means for generating. First information for acquiring three-dimensional model information of an object, parameter information regarding shooting parameters of a virtual camera virtually arranged at a viewpoint corresponding to a viewpoint image, and first size information regarding a view size of the virtual camera Information acquisition means;
Second information acquisition means for acquiring device information relating to a stereoscopic display device used for observation of a stereoscopic image;
Viewpoint determining means for determining a plurality of viewpoints corresponding to the stereoscopic display device based on the parameter information, the first size information, and the device information;
Image generating means for generating a plurality of viewpoint images corresponding to a plurality of viewpoints determined by the viewpoint determining means using the three-dimensional model information, and generating the stereoscopic image using the plurality of viewpoint images; A stereoscopic image generation system characterized by comprising: A first information acquisition step of acquiring, from a plurality of first viewpoint images, first size information relating to the size of the first viewpoint image and disparity information relating to parallax between the plurality of first viewpoint images;
A second information acquisition step of acquiring device information relating to a stereoscopic display device used for observation of a stereoscopic image;
A viewpoint determination step of determining a plurality of viewpoints corresponding to the stereoscopic display device based on the first size information, the parallax information, and the device information;
A plurality of second viewpoint images corresponding to a plurality of viewpoints determined in the viewpoint determination step are generated using the first viewpoint image, and the stereoscopic image is converted using the plurality of second viewpoint images. A three-dimensional image generation method comprising: an image generation step of generating the image. First information for acquiring three-dimensional model information of an object, parameter information regarding shooting parameters of a virtual camera virtually arranged at a viewpoint corresponding to a viewpoint image, and first size information regarding a view size of the virtual camera An information acquisition step;
A second information acquisition step of acquiring device information relating to a stereoscopic display device used for observation of a stereoscopic image;
A viewpoint determination step of determining a plurality of viewpoints corresponding to the stereoscopic display device based on the parameter information, the first size information, and the device information;
An image generation step of generating a plurality of viewpoint images corresponding to the plurality of viewpoints determined in the viewpoint determination step using the three-dimensional model information, and generating the stereoscopic image using the plurality of viewpoint images. A three-dimensional image generation method comprising:

Images