JP6300346B2 - IP stereoscopic image estimation apparatus and program thereof - Google Patents

Description

  The present invention relates to an IP stereoscopic video estimation apparatus and program for estimating the resolution of an IP stereoscopic video (element image group).

  In the conventional integral photography (IP) system, a subject is imaged through a lens array in which a plurality of microlenses are arranged with one camera. At this time, since the camera takes an image of the focal plane of the lens array, each micro lens constituting the lens array has the same function as the micro camera. As a result, an image obtained by photographing the subject through the lens array becomes an element image group in which minute images (element images) corresponding to the positions of the minute lenses are arranged. This elemental image group is an image in which light information from a subject is recorded, and the number of light rays that can be recorded depends on the resolution of the camera. For this reason, a high-resolution camera is required for shooting using the IP method.

  Conventionally, a technique for generating an IP element image group using computer graphics (CG) has been studied. In this conventional technology, a virtual three-dimensional space is generated in a computer, a three-dimensional subject and a background (set) are reproduced in the same manner as in the real world, and light rays from the subject are tracked in the virtual three-dimensional space. Generate element images. As methods for generating element images by CG, methods described in Patent Documents 1 and 2 and Non-Patent Documents 1 and 2 below are known. In these element image generation methods, light rays from a subject are traced through a depth control lens that controls light rays from the subject.

  In the element image generation method described above, it is necessary to photograph with the camera from the viewer side as many times as the number of pixels of the element image group. That is, since the element image generation method of Non-Patent Document 1 uses a ray tracing method for each pixel, it is equivalent to a camera that performs imaging for each pixel, and the number of imaging is the same as the number of pixels in the element image group. .

  In order to further increase the speed, as a conventional technique, there is also known a technique in which a CG object is photographed by a camera by orthographic projection and this image is converted into an element image group (Patent Document 3 and Non-Patent Document 3). In the conventional element image conversion method, the number of times of photographing with the camera is the number of pixels forming the element image.

Japanese Patent No. 4567422 No. 00/059235 (Japanese Patent Application No. 2000-608621) JP 2011-234142 A

Spyros S. Athineos et al., “Physical modeling of a microlens array setup for use in computer generated IP,” Proc. Of SPIE-IS & T Electronic Imaging, Vol.5664 pp.472-479,2005 Nakajima, et al., “High-speed image creation method for 3D displays using the principle of Integral Photography”, The Institute of Image Information and Television Engineers Vol. 54, No. 3, pp.420-425 (2000) M. Katayama et al., “A method for converting three-dimensional models into auto-stereoscopic images based on integral photography”, Proc of SPIE-IS & T Vol. 6805 68050Z-1 68050Z-8

  In the IP system, the resolution decreases as the distance from the display screen increases, and a stereoscopic image located in front of or behind the display screen is blurred. At this time, the degree of blur of the three-dimensional image is not known unless shooting is actually performed. However, in the IP system, photographing cannot be performed unless the lens array and the depth control lens are arranged at appropriate positions. As a result, in the IP method, in order to obtain a desired degree of blur in a three-dimensional image, photographing is repeatedly performed, which is very troublesome.

  Here, the binocular stereoscopic method can generate a stereoscopic image in a short time as compared to the conventional IP image method as well as the conventional element image conversion method. Therefore, in order to shorten the photographing time in the IP method, the use of the binocular stereoscopic method is examined.

  In the first place, the IP method has a great difference in the reproduction of motion parallax and the resolution characteristics of a stereoscopic image, compared with the binocular stereoscopic method. If the viewer is viewing without moving at a specific position, the influence of motion parallax is eliminated. That is, under the above-described conditions, the IP method has a large difference only in the resolution characteristics of the stereoscopic video as compared with the binocular stereoscopic method. From this, it is possible to estimate the resolution of the IP stereoscopic video by converting the resolution of the binocular stereoscopic system image, and the video producer should check the degree of blur of the stereoscopic image before actually shooting with the IP system. Can do.

  Therefore, an object of the present invention is to provide an IP stereoscopic video estimation apparatus and a program thereof that can estimate the resolution of an IP stereoscopic video without taking an image using the IP method.

  In view of the above-described problems, an IP stereoscopic video estimation device according to the present invention is photographed with a binocular stereoscopic imaging device in accordance with the resolution of an element image group displayed on an IP stereoscopic video display device including a display element and a lens array. IP stereo image estimation that converts the resolution of the stereo image and outputs the stereo image with the converted resolution as a viewpoint element image group in which the resolution of the element image group is estimated for each viewpoint representing the right eye and the left eye of the viewer An apparatus comprising: a resolution characteristic determining unit; a depth value calculating unit; a first pixel position calculating unit; a corresponding pixel resolution determining unit; a pixel acquisition region calculating unit; and a pixel value assigning unit. And

  According to such a configuration, the IP stereoscopic video estimation apparatus determines, as the resolution characteristic, the smaller one of the maximum resolution or viewing resolution of the element image group within the preset depth range by the resolution characteristic determination unit. In the IP stereoscopic video estimation apparatus, a stereo value is input by the depth value calculation means, and the depth value is calculated for each pixel of the input stereo image.

  In addition, the IP stereoscopic video estimation apparatus uses a first pixel position calculation unit to generate a viewpoint element image group positioned on a virtual line that passes through the center of the element lens from the viewpoint and reaches the display element by the ray tracing method for each viewpoint. The first reference pixel is calculated. Then, the IP stereoscopic video estimation apparatus obtains the corresponding pixel of the stereo image at the position corresponding to the first reference pixel by the corresponding pixel resolution determining unit, and determines the resolution according to the depth value of the corresponding pixel from the resolution characteristics. .

In the IP stereoscopic image estimation device, the pixel acquisition region calculation unit calculates a pixel acquisition region in which the diameter of the element lens is enlarged with the ratio of the maximum resolution and the resolution of the corresponding pixel around the corresponding pixel. Then, the IP stereoscopic video estimation device acquires the pixels included in the pixel acquisition region from the stereo image for each viewpoint by the pixel value assigning unit, and based on the acquired pixel values of the pixels, A pixel value of each pixel of the viewpoint element image to which the first reference pixel belongs is assigned.
In this way, the IP stereoscopic video estimation device can generate a viewpoint element image group equivalent to the IP stereoscopic video from the stereo image.

  The viewpoint element image group is an image having the same structure as the element image group displayed on the IP stereoscopic video display device corresponding to the viewpoint (both eyes) of the viewer. That is, the viewpoint element image group has the same shape and the same arrangement as the IP system element image group, and includes a right viewpoint element image group and a left viewpoint element image group corresponding to the left and right eyes of the viewer.

  The IP stereoscopic video estimation apparatus according to the present invention can also be realized by an IP stereoscopic video estimation program that causes hardware resources such as a CPU, a memory, and a hard disk included in a computer to operate cooperatively as the above-described means. This program may be distributed through a communication line, or may be distributed by writing in a recording medium such as a CD-ROM or a flash memory.

  According to the present invention, since a viewpoint element image group equivalent to an IP stereoscopic video can be generated from a stereo image in a short time, if this viewpoint element image group is displayed, the viewer can confirm the resolution of the IP stereoscopic video. Can do. As a result, according to the present invention, it is possible to estimate the resolution of the IP stereoscopic video without taking an image by the IP method, and it is easy to produce the IP stereoscopic video and shorten the production time.

In 1st Embodiment, it is the schematic which shows the outline of an IP stereo image estimation system. It is a block diagram which shows the structure of the resolution converter of FIG. It is explanatory drawing explaining calculation of the maximum resolution in 1st Embodiment. In 1st Embodiment, it is explanatory drawing explaining calculation of viewing resolution. FIG. 6 is an explanatory diagram for explaining determination of resolution characteristics in the first embodiment. FIG. 6 is an explanatory diagram illustrating calculation of a reference pixel position in the first embodiment. In 1st Embodiment, it is explanatory drawing explaining the 1st example of calculation of a pixel acquisition area, and allocation of a pixel value. In 1st Embodiment, it is explanatory drawing explaining the 2nd example of calculation of a pixel acquisition area, and allocation of a pixel value. In 1st Embodiment, (a) is explanatory drawing of a left viewpoint element image group, (b) is explanatory drawing of a right viewpoint element image group. In 1st Embodiment, it is explanatory drawing explaining the display method of a viewpoint element image group. It is a flowchart which shows operation | movement of the resolution converter of FIG. In 2nd Embodiment, it is the schematic which shows the outline of an IP stereo image estimation system. It is a block diagram which shows the structure of the image converter of FIG. It is explanatory drawing explaining the production | generation of an element image in 2nd Embodiment. It is a flowchart which shows operation | movement of the image converter of FIG. In 3rd Embodiment, it is the schematic which shows the outline of an IP stereoscopic image estimation system. It is a block diagram which shows the structure of the image conversion apparatus and viewing position tracking apparatus of FIG. In 3rd Embodiment, it is explanatory drawing explaining the change of a viewpoint boundary line. In 3rd Embodiment, it is explanatory drawing explaining the change of the display position of an element image. It is a flowchart which shows operation | movement of the viewing position tracking apparatus of FIG.

  Hereinafter, embodiments of the present invention will be described in detail with reference to the drawings as appropriate. In each embodiment, means having the same function are denoted by the same reference numerals and description thereof is omitted.

(First embodiment)
[Outline of IP stereoscopic video estimation system]
With reference to FIG. 1, an outline of the IP stereoscopic video estimation system 1 according to the first embodiment of the present invention will be described.
The IP stereoscopic video estimation system 1 estimates the resolution of an IP stereoscopic video from a stereo image captured by a binocular stereoscopic camera 20, and includes an IP stereoscopic video estimation device 2 and a binocular stereoscopic camera (binocular stereoscopic imaging device). 20, a recording / reproducing apparatus 30, and a twin-lens stereoscopic television 50.

  The IP stereoscopic video estimation device 2 converts the resolution of a stereo image captured by the binocular stereoscopic camera 20 and estimates the resolution of the IP stereoscopic video. In the first embodiment, the IP stereoscopic video estimation device 2 is configured by a resolution conversion device 40.

  The binocular stereoscopic camera 20 is a stereo camera that captures the subject 10 in a binocular stereoscopic manner. For example, the two-lens stereoscopic camera 20 includes two identical photographing cameras 21R and 21L arranged on the left and right with a predetermined baseline. Then, the binocular stereoscopic camera 20 outputs the captured stereo image to the recording / reproducing device 30.

A stereo image taken by the right camera 21R is called a right shot image, and a stereo image taken by the left camera 21L is called a left shot image. Further, the right photographing camera 21R is made to correspond to the right eye of the viewer, and the left photographing camera 21L is made to correspond to the left eye of the viewer.
The viewer is a person (for example, a video producer) who confirms the resolution of the IP stereoscopic video using the image generated by the IP stereoscopic video estimation device.

  The recording / reproducing device 30 records a stereo image input from the binocular stereoscopic camera 20 and outputs the recorded stereo image to the resolution conversion device 40. For example, the recording / reproducing apparatus 30 includes a general recording apparatus.

The resolution conversion device 40 converts the resolution of the stereo image input from the recording / playback device 30 in accordance with the resolution of the IP stereoscopic video. Then, the resolution conversion device 40 outputs a stereo image (viewpoint element image group) whose resolution has been converted to the binocular stereoscopic television 50.
Details of the resolution conversion device 40 will be described later.

The viewpoint element image group includes a right viewpoint element image group and a left viewpoint element image group having the same structure as the element image group displayed on the IP stereoscopic video display device corresponding to the viewpoint of the viewer. In the present embodiment, each viewpoint element image included in the viewpoint element image group has each pixel having the same pixel value.
The viewpoint represents a position corresponding to the right eye and left eye of the viewer.

  The binocular stereoscopic television 50 is a stereoscopic display device that displays the viewpoint element image group input from the resolution conversion device 40. For example, the binocular 3D television 50 may be a lenticular 3D television. In addition, the binocular stereoscopic television 50 may be a stereoscopic television that alternately displays the right viewpoint element image group and the left viewpoint element image group in a time division manner.

A method of using the IP stereoscopic video estimation system 1 will be briefly described.
As described above, in the IP system, the degree of blur of a stereoscopic image is not known unless actual shooting is performed. Therefore, in the IP system, in order to obtain an appropriate degree of stereoscopic image blur for the viewer, photographing is often repeated. On the other hand, in the IP method, since the photographing is performed after the lens array and the depth control lens are arranged at appropriate positions, it takes much time for photographing.

  Therefore, the IP stereoscopic video estimation system 1 uses the binocular stereoscopic imaging method, which requires less labor than the IP method, to resolve the resolution of the IP stereoscopic video displayed on the IP stereoscopic video display device 100 (FIG. 4). Is estimated. Specifically, the IP stereoscopic video estimation system 1 captures the subject 10 with the binocular stereoscopic camera 20, and converts the resolution of the captured stereo image in accordance with the resolution of the IP stereoscopic video. In other words, the IP stereoscopic video estimation system 1 can be said to be a simulator that estimates the degree of blurring of a stereoscopic image by the IP method.

  The IP stereoscopic video display device 100 is a stereoscopic video display device that is assumed to display IP stereoscopic video. The IP stereoscopic video display device 100 includes a lens array 110 and a display element 200.

[Configuration of resolution converter]
The configuration of the resolution conversion device 40 will be described with reference to FIG.
As shown in FIG. 2, the resolution conversion apparatus 40 includes a resolution characteristic determination unit 41, a depth value calculation unit 42, a pixel position calculation unit (first pixel position calculation unit) 43, and a viewpoint element image group generation unit 44. Prepare.

The resolution characteristic determining unit 41 determines the resolution characteristic of the element image group displayed on the IP stereoscopic video display device 100. The maximum resolution calculating unit 41a, the viewing resolution calculating unit 41b, and the resolution characteristic calculating unit 41c. With.
Here, the resolution characteristic determining means 41 is a parameter necessary for determining the resolution characteristic of the element image group (for example, the distance between the element lenses, the viewing distance, the distance between the display element and the lens array, and the pixel interval of the element image group). ) Is entered.

<Determination of resolution characteristics>
The determination of resolution characteristics will be described in detail with reference to FIGS. 3 to 5 (see FIG. 2 as appropriate).
In FIG. 3, the horizontal axis Z represents the depth value with reference to the lens array 110, and “0” is the position of the lens array 110.

  The maximum resolution calculation unit 41 a calculates the maximum resolution (maximum spatial frequency) of the element image group based on the distance P between the element lenses 120 constituting the lens array 110 and the viewing distance L between the lens array 110 and the viewpoint 300. To do.

  The maximum resolution is a resolution determined depending on the interval P between the element lenses 120 and is a resolution at a position (depth value z) of the lens array 110.

  As shown in FIG. 3, the angle φ represents an angle with respect to the interval P between the element lenses 120 as in the following formula (1). In other words, the angle φ is an approximate value of an angle formed by a line connecting the end of the element lens 120 and the viewpoint 300.

Therefore, the maximum resolution β _n can be expressed by the following equation (2). In other words, the maximum resolution calculating unit 41a calculates the maximum resolution β _n by dividing the viewing distance L by a value obtained by doubling the interval P between the element lenses 120. Then, the maximum resolution calculating unit 41a outputs the calculated maximum resolution β _n to the resolution characteristic calculating unit 41c.

As shown in FIG. 4, the viewing resolution calculation unit 41 b uses the viewing resolution (viewing) of the element image group 410 based on the distance g between the display element 200 and the lens array 110 and the pixel interval p of the element image 420. (Spatial frequency).
In FIG. 4, the distance g is negative because it is on the far side of the lens array 110.

The viewing resolution is the resolution when the projected stereoscopic image 400 is viewed from the viewpoint 300.
In FIG. 4, the pixels in the stereoscopic image 400 and the pixels in the element image 420 are alternately shown in black and white for easy viewing of the drawing.

  The pixel interval t represents the pixel interval in the stereoscopic image 400 as in the following formula (3). Here, the stereoscopic image 400 is formed at a position indicated by the depth value z.

  The angle θ represents an angle with respect to the pixel interval t as in the following formula (4). In other words, the angle θ is an approximate value obtained by dividing the pixel interval t by the value obtained by subtracting the depth value z from the viewing distance L.

Accordingly, the viewing resolution β can be expressed by the following equation (5). That is, the viewing resolution calculation means 41b is a product of a value obtained by dividing the distance g by twice the pixel interval p and a value obtained by subtracting the depth value z from the viewing distance L and dividing by the absolute value of the depth value z. The viewing resolution β is calculated. Then, the viewing resolution calculation unit 41b outputs the calculated viewing resolution β to the resolution characteristic calculation unit 41c.
The viewing resolution calculating unit 41b calculates the viewing resolution β while changing the depth value z of the stereoscopic image 400 within the depth range. The depth range is a range in which the depth value z of the stereoscopic image 400 can be set in advance.

As shown in FIG. 5, in the depth range, the resolution characteristic calculation unit 41c has a smaller maximum resolution β _n input from the maximum resolution calculation unit 41a or viewing resolution β input from the viewing resolution calculation unit 41b. Is a resolution characteristic γ that represents. Then, the resolution characteristic calculation unit 41 c outputs the calculated resolution characteristic γ to the viewpoint element image group generation unit 44.

Here, the resolution characteristic γ is the smaller of the maximum resolution β _n calculated by the maximum resolution calculation unit 41a or the viewing resolution β calculated by the viewing resolution calculation unit 41b, as in the following equation (6). Become. In Expression (6), “min” is a function that selects the smallest argument among a plurality of arguments.

In the example of FIG. 5, the resolution characteristic γ represents the maximum resolution β _n in the vicinity of the lens array 110, and represents the viewing resolution β when separated from the lens array 110 to some extent. The resolution characteristic γ represents that the resolution decreases as the distance from the lens array 110 increases.

In addition, since the determination method of the resolution characteristic is described in the following references 1 and 2, further explanation is omitted.
Reference 1: H. Hoshino et al., “Analysis of Resolution Limitation of Integral Photography”, J. Opt. Soc. Am. A, Vol. 15, No. 8, pp. 2059-2065, 1998
Reference 2: Yuichi Iwabuchi, “Studies on multi-view stereoscopic images in NHK”, http://www.3dc.gr.jp/jp/act_rep/121211/NHK_20121211.pdf

Returning to FIG. 2, the description of the configuration of the resolution conversion apparatus 40 will be continued.
The depth value calculating means 42 receives a stereo image (right captured image and left captured image) from the binocular stereoscopic camera 20 and calculates a depth value for each pixel of the input stereo image. Then, the depth value calculating unit 42 outputs the right captured image and the left captured image in which the depth value is associated with each pixel to the viewpoint element image group generating unit 44.

For example, the depth value calculation unit 42 searches for a corresponding point between the right captured image and the left captured image, and calculates a distance (parallax) of the searched corresponding point as a depth value.
The depth value calculation means 42 may receive a depth image indicating a depth value for each pixel from a depth camera (not shown) arranged at the same position as the binocular stereoscopic camera 20. In this case, the depth value calculating unit 42 associates the depth value indicated by the depth image with each pixel of the right captured image and the left captured image.

The pixel position calculation means 43 calculates the position of the reference pixel by the ray tracing method for each viewpoint.
In FIG. 6, the reference pixels are pixels of the right viewpoint element image group and the left viewpoint element image group that are located on a virtual line that passes from the viewpoint 301 through the center of the element lens 120 and reaches the display element 200 ( Reference numeral 312 in FIG. 6).

<Calculation of reference pixel position>
The reference pixel position calculation will be described in detail with reference to FIG. 6 (see FIG. 2 as appropriate).
In FIG. 6, reference numeral 310 represents the principal point position of the central element lens 120 </ b> C in the lens array 110.
Reference numeral 311 represents the principal point position of the uppermost element lens 120U in the lens array 110.
Reference numeral 312 represents the pixel position of the reference pixel of the element image 420 corresponding to the element lens 120U.
Reference numeral 313 represents the pixel position of the central pixel of the element image 420 at the same position as the principal point position 311 of the element lens 120U. In FIG. 6, reference numerals 312 and 313 represent pixel positions in the same element image 420.

In FIG. 6, the distance y represents the distance from the principal point position 310 to the principal point position 311. The distance d represents the distance from the pixel position 313 of the central pixel to the pixel position 312 of the reference pixel.
In FIG. 6, the viewer's right eye (calculated origin 301) is located on the normal line of the center position of the lens array 110. The element lens 120 and the element image 420 correspond 1: 1.

The right triangle tr1 is a triangle having the principal point position 311, the pixel position 312 of the reference pixel, and the pixel position 313 of the center pixel as vertices.
The right triangle tr2 is a triangle having the calculated origin 301, the principal point position 310, and the principal point position 311 as vertices.

  As shown in FIG. 6, since the right triangles tr1 and tr2 have a similar relationship, L: y = g: d and Ld = yg is established. The distances g and y are constants determined by the IP stereoscopic video display device 100. Therefore, if the viewing distance L is known, the distance d can be calculated using the following equation (7).

  As described above, the viewpoint element image has the same structure as a general IP stereoscopic video (element image group 410). Accordingly, the pixel position calculation unit 43 sets the pixel position of the reference pixel in the right viewpoint element image group corresponding to the viewer's right eye using the above-described method with the calculation origin 301 as the viewer's right eye (viewpoint). It can be calculated. Then, the pixel position calculation unit 43 adds the distance d to the pixel position 313 of the central pixel to obtain the pixel position 312 of the reference pixel.

Thereafter, the pixel position calculation unit 43 outputs the calculated position of the reference pixel to the viewpoint element image group generation unit 44.
The pixel position calculation unit 43 calculates the position of the reference pixel from the element lenses 120 other than the uppermost element lens 120U. The pixel position calculation unit 43 also calculates the reference pixel of the left viewpoint element image group as in the right viewpoint element image group.

Returning to FIG. 2, the description of the configuration of the resolution conversion apparatus 40 will be continued.
The viewpoint element image group generation unit 44 generates a viewpoint element image group by converting the resolution of the stereo image input from the depth value calculation unit 42 in accordance with the resolution characteristic input from the resolution characteristic determination unit 41. Is. For this reason, the viewpoint element image group generation unit 44 includes a corresponding pixel resolution determination unit 44a, a pixel acquisition region calculation unit 44b, and a pixel value allocation unit 44c.

  Corresponding pixel resolution determining means 44a obtains a corresponding pixel of the stereo image at a position corresponding to the reference pixel input from the pixel position calculating means 43, and determines the resolution according to the depth value of the corresponding pixel from the resolution characteristics. It is.

Specifically, the corresponding pixel resolution determining unit 44a has a one-to-one correspondence between the right viewpoint element image group and the right photographed image pixel, and therefore determines the right photographed image pixel corresponding to the reference pixel of the right viewpoint element image group. Obtained as a corresponding pixel. Then, the corresponding pixel resolution determining unit 44a acquires the depth value because the depth value is added to the corresponding pixel of the right captured image. Further, the corresponding pixel resolution determining unit 44a refers to the resolution characteristic indicating the resolution of each depth value, and determines the resolution corresponding to the acquired depth value for each corresponding pixel. Thereafter, the corresponding pixel resolution determination unit 44a outputs the determined resolution for each corresponding pixel to the pixel acquisition region calculation unit 44b.
Note that the corresponding pixel resolution determining unit 44a determines the resolution for each corresponding pixel of the left captured image as in the right captured image.

  The pixel acquisition area calculation unit 44b calculates a pixel acquisition area obtained by enlarging the diameter of the element lens with the ratio between the resolution of the corresponding pixel input from the corresponding pixel resolution determination unit 44a and the maximum resolution centered on the corresponding pixel of the stereo image. To do. Then, the pixel acquisition region calculation unit 44b outputs the calculated pixel acquisition region to the pixel value assignment unit 44c.

  The pixel value assigning unit 44c acquires, for each viewpoint, a pixel included in the pixel acquisition region input from the pixel acquisition region calculating unit 44b from the stereo image, and based on the acquired pixel value of the pixel of the viewpoint element image group. Among them, the pixel value of each pixel of the viewpoint element image to which the reference pixel belongs is assigned.

<Calculation of Pixel Acquisition Area and Assignment of Pixel Value: First Example>
Two specific examples of the calculation of the pixel acquisition region and the assignment of the pixel value will be described with reference to FIGS. 7 and 8 (see FIG. 2 as appropriate).
In FIG. 7, the position indicated by the depth value of the corresponding pixel is indicated by an arrow, and the configuration irrelevant to the calculation of the pixel acquisition region is indicated by a broken line (the same applies to FIG. 8).

  As shown in FIG. 7, in this first example, the depth value of the corresponding pixel indicates the position of the lens array 110. In this case, since the resolution of the IP stereoscopic video is the same as the density of the lens array 110, the maximum resolution is obtained.

  Therefore, the pixel acquisition area calculation unit 44b calculates a ratio “1” between the maximum resolution and the resolution of the corresponding pixel. Accordingly, the pixel acquisition region calculation unit 44b has the same shape and the same diameter as the element lens 120 (the interval P between the element lenses 120) around the corresponding pixel 520 (shown in black in FIG. 7) of the right captured image 510. The calculated range is calculated as a pixel acquisition region. Here, since the element lens 120 has a size corresponding to five pixels of the right captured image 510, the pixel acquisition region has a size corresponding to five pixels.

  Next, the pixel value assigning unit 44c acquires all the pixels 520 of the right captured image 510 included in the calculated pixel acquisition region, and sets the pixels 630 of the right viewpoint element image 620 based on the acquired pixel values of the pixels 520. A pixel value to be assigned is calculated.

Here, the pixel value assigning unit 44c may calculate the median value (median value) of the acquired pixel values of the pixel 520. When the median value is used, since the pixel value of the acquired pixel 520 is formed into a histogram, an accurate pixel value is not acquired. Therefore, there is a possibility that the right viewpoint element image 620 has a lower continuity of pixel values between adjacent right viewpoint element images 620 and becomes a folded (moire) image.
Therefore, it is preferable that the pixel value assigning unit 44c obtains an average value of the obtained pixel values of the pixels 520 because more accurate pixel values can be obtained.

  Next, the pixel value assigning unit 44c assigns the calculated pixel value to all the pixels 630 of the right viewpoint element image 620 to which the reference pixel belongs in the right viewpoint element image group 610. In the example of FIG. 7, in the right viewpoint element image 620, the pixel values of the five pixels 630 are the same. In this way, the pixel value assigning unit 44c can generate the right viewpoint element image 620 to which the pixel value is assigned at the position of the lens array 110.

<Calculation of Pixel Acquisition Area and Assignment of Pixel Value: Second Example>
As shown in FIG. 8, in this second example, the depth value of the corresponding pixel represents the front side of the lens array 110, and the resolution of the IP stereoscopic video is ½ of the maximum resolution at this depth position. .

  In this case, the pixel acquisition region calculation unit 44b calculates a ratio “2” between the maximum resolution and the resolution of the corresponding pixel. Accordingly, the pixel acquisition area calculation unit 44b acquires a range that is the same shape as the element lens 120 and includes twice the diameter around the corresponding pixel 520 (shown in black in FIG. 8) of the right captured image 510. Calculate as a region. Here, since the element lens 120 has a size corresponding to 5 pixels of the right captured image 510, the pixel acquisition area has a size corresponding to 10 pixels, which is twice the 5 pixels.

  Next, the pixel value assigning unit 44c acquires all the pixels 520 of the right captured image 510 included in the calculated pixel acquisition region, and sets the pixels 630 of the right viewpoint element image 620 based on the acquired pixel values of the pixels 520. A pixel value to be assigned is calculated. Here, as in the first example, the pixel value assigning unit 44c calculates the average value or the median value of the pixel values of the pixels 520 included in the pixel acquisition region.

  Then, the pixel value assigning unit 44c assigns the calculated pixel value to all the pixels 630 of the right viewpoint element image 620 to which the reference pixel belongs in the right viewpoint element image group 610. That is, the pixel value assigning unit 44c calculates how much the resolution is reduced at the depth position of the target pixel on the basis of the maximum resolution at the position of the lens array 110, and thereby the right viewpoint element image group reflecting the degree of blur. 610 is generated.

  Even when the depth position of the corresponding pixel is the back side of the lens array 110, the procedure is the same as in the second example. Further, similarly to the right viewpoint element image group 610, a left viewpoint element image group 611 (FIG. 9) is also generated.

  Thereafter, the pixel value assigning unit 44 c outputs the right viewpoint element image group 610 and the left viewpoint element image group 611 generated by the technique of the first example or the second example to the binocular stereoscopic television 50. The right viewpoint element image group 610 and the left viewpoint element image group 611 are stereoscopically displayed on the binocular stereoscopic television 50.

<Display of viewpoint element image group>
The display of the viewpoint element image group will be described with reference to FIGS.
FIG. 9A illustrates the left viewpoint element image group 611, and FIG. 9B illustrates the right viewpoint element image group 610. In FIG. 9, in the left viewpoint element image group 611 and the right viewpoint element image group 610, a triangular object 10 is captured.
In FIG. 9, black-painted element image groups 620 and 621 represent the subject 10.

  In general IP stereoscopic video (element image group), an element image is composed of a plurality of pixels and represents a small image of the subject 10. The element image is different in pixel values of pixels constituting the element image in order to reproduce motion parallax.

  As shown in FIG. 9, the left viewpoint element image group 611 and the right viewpoint element image group 610 have the same structure (the same shape and the same arrangement) as a general IP stereoscopic video. Further, in the left viewpoint element image group 611 and the right viewpoint element image group 610, the same subject 10 is captured from different viewpoints, and therefore the position of the subject 10 is shifted. That is, the left viewpoint element image at the position α in FIG. 9A has a different pixel value as compared with the right viewpoint element image at the same position α as shown in FIG. 9B. .

As shown in FIG. 10, the twin-lens stereoscopic TV 50 includes a set of a display element 51 </ b> L corresponding to the left viewpoint and a display element 51 </ b> R corresponding to the right viewpoint for one lenticular lens 53. Then, the binocular stereoscopic television 50 displays the left viewpoint element image group 611 on the display element 51L and the right viewpoint element image group 610 on the display element 51R. Then, the viewer views different images with both eyes, and can obtain a stereoscopic view.
Note that the left viewpoint element image group 611 and the right viewpoint element image group 610 are different from general IP stereoscopic video, and the pixel values of the pixels constituting the viewpoint element image are the same, so that the images are not switched due to motion parallax.

[Operation of resolution converter]
The operation of the resolution conversion apparatus 40 will be described with reference to FIG. 11 (see FIG. 2 as appropriate).
The resolution conversion device 40 receives parameters necessary for determining the resolution characteristics of the IP stereoscopic video display device 100 (step S1).

  The resolution conversion device 40 determines the resolution characteristic by the resolution characteristic determination means 41. Specifically, the resolution characteristic determination unit 41 calculates the maximum resolution by the maximum resolution calculation unit 41a, and calculates the viewing resolution by the viewing resolution calculation unit 41b. In addition, the resolution characteristic determining unit 41 calculates the smaller one of the maximum resolution or viewing resolution as the resolution characteristic by the resolution characteristic calculating unit 41c (step S2).

The resolution conversion device 40 calculates the depth value for each pixel of the stereo image captured by the binocular stereoscopic camera 20 by the depth value calculation means 42 (step S3).
In the resolution conversion device 40, the pixel position calculation unit 43 calculates the pixel position of the reference pixel using the ray tracing method (step S4).

  The resolution conversion device 40 generates viewpoint element image groups by the viewpoint element image group generation means 44. Specifically, the viewpoint element image group generation unit 44 determines the resolution according to the depth value of the corresponding pixel by the corresponding pixel resolution determination unit 44a. Further, the viewpoint element image group generation unit 44 calculates the pixel acquisition region by the pixel acquisition region calculation unit 44b. Further, the viewpoint element image group generation unit 44 acquires pixels included in the pixel acquisition region from the stereo image by the pixel value allocation unit 44c, and the pixels of each pixel of the viewpoint element image based on the acquired pixel values of the pixels. A value is assigned (step S5).

  The IP stereoscopic video estimation system 1 according to the first embodiment of the present invention generates a viewpoint element image group equivalent to an IP stereoscopic video from a stereo image in a short time, and displays this viewpoint element image group on the binocular stereoscopic television 50. Thus, the viewer can confirm the resolution of the IP stereoscopic video. As described above, the IP stereoscopic video estimation system 1 can estimate the resolution of the IP stereoscopic video without taking an image by the IP method, and can easily produce the IP stereoscopic video and shorten the production time.

  Furthermore, the IP stereoscopic video estimation system 1 can also estimate the resolution of the IP stereoscopic video by separating the performance on the imaging side and the display side. For example, the IP stereoscopic video estimation system 1 sets the appropriate parameters and sets the resolution of the IP stereoscopic video when shooting with an IP stereoscopic camera (not shown) having a resolution lower than that of the binocular stereoscopic camera 20. Can be simulated. The IP stereoscopic video estimation system 1 can also simulate the resolution of an IP stereoscopic video even when an IP stereoscopic camera is used on the imaging side and a binocular stereoscopic television 50 is used on the display side.

(Second Embodiment)
[Outline of IP stereoscopic video estimation system]
With reference to FIG. 12, the IP stereoscopic video estimation system 1A according to the second embodiment of the present invention will be described while referring to differences from the first embodiment.

The IP stereoscopic video estimation system 1A includes an IP stereoscopic video estimation device 2A and an IP stereoscopic television 70 in order to convert a viewpoint element image group into an IP stereoscopic video (element image group) and display it on the IP stereoscopic television 70. However, this is different from the first embodiment.
In the second embodiment, the IP stereoscopic video estimation device 2A is configured by a resolution conversion device 40 and an image conversion device 60.

[Configuration of image converter]
The configuration of the image conversion device 60 will be described with reference to FIG. 13 (see FIG. 12 as appropriate).
The image conversion device 60 converts the viewpoint element image group (the right viewpoint element image group and the left viewpoint element image group) input from the resolution conversion device 40 into a general IP stereoscopic video (element image group). . Therefore, the image conversion apparatus 60 includes a pixel position calculation unit (second pixel position calculation unit) 61 and an image composition unit 62 as shown in FIG.

The pixel position calculation means 61 calculates the position of the second reference pixel by the ray tracing method.
The second reference pixel is a pixel of an element image group located on an imaginary line passing through the center of the element lens 120 from the viewpoint intermediate position and reaching the display element 200 (FIG. 6). This viewpoint intermediate position represents the intermediate position of the right eye and left eye of the viewer.

  In the first embodiment, in order to generate two viewpoint element image groups 610 and 611 corresponding to the left and right eyes of the viewer, the pixel position of the reference pixel is calculated for each viewpoint. On the other hand, in the second embodiment, since there is one element image group 410, the pixel position of the second reference pixel may be calculated at the middle position between the eyes of the viewer. As described above, the pixel position calculation unit 61 can calculate the pixel position of the second reference pixel with the calculation origin 301 of FIG. 6 as the viewpoint intermediate position.

  The image compositing means 62 sets a divided area obtained by dividing the element image to which the second reference pixel belongs by the viewpoint boundary line, extracts an image area corresponding to the divided area from the viewpoint element image group for each viewpoint, and uses different viewpoints. Element images are generated by combining the extracted image regions.

<Generation of element images>
The generation of element images will be described with reference to FIG. 14 (see FIG. 13 as appropriate).
When viewing the IP stereoscopic television 70 in front, the range in which the viewer can obtain a stereoscopic view is half the distance between the eyes of the viewer in the horizontal direction. Accordingly, the image composition unit 62 sets the viewpoint boundary line 430 so that the element image 420 located behind the element lens 120 when viewed from the center position of both eyes of the viewer is divided into the left and right halves. Further, the image composition unit 62 sets divided regions 440R and 440L obtained by dividing the element image 420 by the viewpoint boundary line 430.

  Next, the image composition means 62 assigns an image region corresponding to the right viewpoint element image 620 (FIG. 9) to the left divided region 440R. Further, the image composition unit 62 assigns an image region corresponding to the left viewpoint element image 621 to the right divided region 440L. In this way, the image composition unit 62 synthesizes one element image 420 from the right viewpoint element image 620 and the left viewpoint element image 621. Then, the image composition means 62 outputs the element image group 410 composed of the composed element images 420 to the IP stereoscopic television 70.

[Operation of image converter]
The operation of the image conversion apparatus 60 will be described with reference to FIG. 15 (see FIG. 13 as appropriate).
The image conversion device 60 receives a viewpoint element image group from the resolution conversion device 40 (step S11).

The image conversion device 60 calculates the pixel position of the second reference pixel by the pixel position calculation unit 61 by the ray tracing method (step S12).
The image conversion device 60 generates an element image group by the image composition means 62 (step S13).

As in the first embodiment, the IP stereoscopic video estimation system 1A according to the second embodiment of the present invention makes it possible to estimate the resolution of an IP stereoscopic video without taking an image using the IP method, and easily produce an IP stereoscopic video. And the production time can be shortened.
Furthermore, since the IP stereoscopic video estimation system 1A can simulate the influence of reflection of external light by the lens array 71 included in the IP stereoscopic television 70 in addition to the resolution of the IP stereoscopic video, the IP stereoscopic video can be more easily produced. Can do.

(Third embodiment)
[Outline of IP stereoscopic video estimation system]
With reference to FIG. 16, an IP stereoscopic image estimation system 1B according to the third embodiment of the present invention will be described while referring to differences from the second embodiment.

The IP stereoscopic image estimation system 1B is further provided with an IP stereoscopic image estimation device 2B, a binocular stereoscopic camera 20B, and a viewing position detection camera 80 in order to follow the viewing position of the viewer. Different from the embodiment.
In the third embodiment, it is assumed that the IP stereoscopic video estimation device 2B includes a resolution conversion device 40, an image conversion device 60B, and a viewing position tracking device 90.

  In addition to the same functions as the binocular stereoscopic camera 20 in FIG. 13, the binocular stereoscopic camera 20B is a robot capable of shooting following the viewing position of the viewer in accordance with a command from the viewing position tracking device 90 described later. It is a camera. For example, the binocular stereoscopic camera 20B is a robot camera including a drive mechanism for moving the imaging floor back and forth, right and left, and a pan / tilt pan head mounted with the imaging cameras 21R and 21L.

  In addition to the same processing as the image conversion device 60 in FIG. 13, the image conversion device 60B changes the position of the viewpoint boundary line and the display position of the element image in accordance with a command from the viewing position tracking device 90. .

  The viewing position detection camera 80 is a general photographing camera that photographs a viewer. The viewing position detection camera 80 outputs a viewer image obtained by capturing the viewer to the viewing position tracking device 90. As shown in FIG. 16, the viewing position detection camera 80 is installed at the center of the upper end portion of the IP stereoscopic television 70. The installation position of the viewing position detection camera 80 is not particularly limited, but is known.

  The viewing position tracking device 90 detects the viewer's viewing position using the viewer image input from the viewing position detection camera 80, and follows the detected viewing position. The viewing area of the IP stereoscopic video displayed on the television 70 is controlled. Specifically, the viewing position tracking device 90 instructs the image conversion device 60B to change the position of the viewpoint boundary line and the display position of the element image so as to track the detected viewing position. Furthermore, the viewing position tracking device 90 commands the binocular stereoscopic camera 20B to move the shooting position and change the shooting posture so as to follow the detected viewing position.

[Configuration of Image Conversion Device]
The configuration of the image conversion device 60B will be described with reference to FIG.
As shown in FIG. 17, the image conversion device 60B includes a pixel position calculation unit 61 and an image composition unit 62B.

In addition to the same processing as the image composition unit 62 in FIG. 13, the image composition unit 62B changes the position of the viewpoint boundary line and the display position of the element image in accordance with a command from the viewing position tracking control unit 92 described later. Is.
Details of the change between the position of the viewpoint boundary line and the display position of the element image will be described later.

[Configuration of viewing position tracking device]
The configuration of the viewing position tracking device 90 will be described.
As shown in FIG. 17, the viewing position tracking device 90 includes a viewing position detection unit 91, a viewing position tracking control unit 92, and a robot camera control unit 93.

The viewing position detection unit 91 detects the viewing position of the viewer with respect to the center position of the lens array 110 from the viewer image.
Specifically, the viewing position detection unit 91 performs a face area detection process (reference document 3) and a binocular position detection process (reference document 4) on the viewer image, so that the viewer image is displayed. The position of both eyes included in is detected.
Reference 3: PAUL VIOLA, MICHAEL J. JONES: “Robust Real-Time Face Detection”, International Journal of Computer Vision, Vol.57, No.2, pp.137-154 (2004)
Reference 4: JP 2013-105384 A

  Next, the viewing position detection unit 91 obtains the detected shift amount and shift direction of the middle position of both eyes with respect to the detection reference position set in advance so as to represent the center position of the lens array 110. For example, when the viewer's head moves to the right with respect to the center position of the lens array 110, the shift direction is to the left. Naturally, the amount of movement of the viewer increases in proportion to the amount of deviation. In this way, the viewing position detection unit 91 calculates the viewing position from the calculated shift amount and shift direction, and outputs the determined viewing position to the viewing position tracking control unit 92 and the robot camera control unit 93.

  The viewing position follow-up control means 92 follows the viewing position input from the viewing position detection means 91 to the image composition means 62B, and changes the viewpoint boundary line and the element image display position. It is a command.

<Change of viewpoint boundary>
The change of the viewpoint boundary line will be described with reference to FIG. 18 (see FIGS. 6 and 17 as appropriate).
In FIG. 18, it is assumed that the viewing position (the middle position between the eyes of the viewer) has moved to the left by the distance x from the element lens 120 </ b> C positioned at the center of the lens array 110.

  As shown in FIG. 18, when the viewing position moves to the left side, the viewer's right eye views most of the elemental image 420 and the viewer's left eye views the rest of the elemental image 420. become. Therefore, the viewing position tracking control unit 92 moves the viewpoint boundary line 430 in the opposite direction (right side) of the viewing position movement direction so that the divided area 440R is widened and the divided area 440L is narrowed. 430 is moved.

  Here, a method of calculating the movement amount of the viewpoint boundary line 430 will be described. As described above, the right triangles tr1 and tr2 in FIG. 6 have a similar relationship. Therefore, when the viewer moves by the distance x, the movement amount of the viewpoint boundary line 430 is a value obtained by multiplying the distance x by the ratio of the sizes of the right triangles tr1 and tr2. That is, the viewing position follow-up control unit 92 determines the movement amount of the viewpoint boundary line 430 using the equation (7) in the same manner as the pixel position calculation unit 61 with the calculated origin 301 as the intermediate position between the eyes of the viewer. . In this case, the distance d in Expression (7) represents the movement amount of the viewpoint boundary line 430. Then, the viewing position follow-up control unit 92 obtains the position moved by the distance d from the center of the element image 420 as the position of the viewpoint boundary line 430.

  Hereinafter, another method will be described. The viewing position tracking control unit 92 calculates the pixel position of the element image in which light is incident on the right eye and the left eye, respectively, and sets a line that bisects the straight line connecting the pixel positions of the right eye and the left eye vertically to the viewpoint boundary line. Obtained as position 430. Here, since the position of the viewpoint boundary line 430 becomes more accurate, the viewing position follow-up control unit 92 preferably uses this other method.

  Thereafter, the viewing position tracking control unit 92 instructs the image composition unit 62B to change the viewpoint boundary line 430 to the obtained position. Then, the image composition unit 62B changes the viewpoint boundary line 430 in accordance with this command, and generates an element image group.

  Note that the viewing position tracking control unit 92 may instruct the image composition unit 62B to tilt the viewpoint boundary line 430 in accordance with the tilt of the viewer's face. For example, the viewing position tracking control unit 92 draws a perpendicular line at the midpoint of the line segment connecting the eyes of the viewer, and obtains the inclination angle of the viewpoint boundary line 430 so as to match the perpendicular line.

<Change of element image display position>
With reference to FIG. 19, the change in the display position of the element image will be described (see FIGS. 6 and 17 as appropriate).

  As shown in FIG. 19, the viewing position follow-up control unit 92 moves the display position of the element image 420 in the direction opposite to the moving direction of the viewing position (right side) because the viewing position moves to the left side with respect to the element lens 120C. Let Here, the viewing position follow-up control unit 92 can obtain the display position of the element image 420 as in the change of the viewpoint boundary line described above. That is, the viewing position follow-up control unit 92 moves the display position of the element image 420 by the distance d in Expression (7).

Note that the viewing position follow-up control unit 92 may set in advance whether to change the viewpoint boundary line 430 or to change the display position of the element image 420.
Further, the viewing position tracking control unit 92 commands the change of the viewpoint boundary line 430 when the movement amount of the viewpoint boundary line 430 is less than the radius of the element image 420, and the movement amount of the viewpoint boundary line 430 is set to the element image 420. If the radius is greater than or equal to the radius, a change in the display position of the element image 420 may be commanded.

Returning to FIG. 17, the description of the configuration of the viewing position tracking device 90 will be continued.
The robot camera control means 93 instructs the binocular stereoscopic camera 20B to take a stereo image following the viewing position input from the viewing position detection means 91. For example, the robot camera control means 93 calculates the moving distance of the viewer and the viewing angle of the IP stereoscopic video based on the viewing position, and instructs to change the shooting position and shooting posture of the binocular stereoscopic camera 20B. .

[Operation of viewing position tracking device]
The operation of the viewing position tracking device 90 will be described with reference to FIG. 20 (see FIG. 17 as appropriate).
The viewing position tracking device 90 receives a viewer image from the viewing position detection camera 80 (step S21).

The viewing position tracking device 90 detects the viewing position of the viewer from the viewer image by the viewing position detection unit 91 (step S22).
The viewing position tracking device 90 tracks the viewing position by the viewing position tracking control unit 92 and instructs to change the viewpoint boundary line and the display position of the element image (step S23).
The viewing position tracking device 90 instructs the binocular stereoscopic camera 20B to capture a stereo image following the viewing position by the robot camera control means 93 (step S24).

  The IP stereoscopic video estimation system 1B according to the third embodiment of the present invention displays the IP stereoscopic video that is easy for the viewer to view because it can follow the viewing position of the viewer in addition to the same effects as the second embodiment. can do.

(Modification 1)
The IP stereoscopic video estimation system according to the present invention is not limited to the above-described embodiments, and modifications can be made without departing from the spirit of the system.

  In the second embodiment described above, as shown in FIG. 14, the divided areas 440L and 440R are set in the element image 420, and the pixels of the divided areas 440L and 440R have the same pixel value. Here, for example, when the viewer moves 1 mm to the left side, in the element image, the pixel at the position obtained by multiplying this 1 mm by the ratio of the sizes of the right triangles tr1 and tr2 is viewed (FIG. 6). ). For this reason, although the different pixels are viewed before and after the viewer moves, since the pixel values of the different pixels are the same, the stereoscopic image viewed by the viewer is the same.

  In the IP method, an optical image of the subject 10 is reproduced. In other words, in the IP system, the stereoscopic light is realized by photographing the reflected light from the subject 10 with the IP stereoscopic photographing device and displaying it on the IP stereoscopic television 70. Here, the IP stereoscopic imaging device captures light within a certain range in order to realize stereoscopic viewing even if the viewer moves to some extent. Then, the IP stereoscopic television 70 reproduces the optical image of the subject 10 in a range corresponding to the IP stereoscopic photographing apparatus.

  When the viewer does not move, the IP stereoscopic imaging device and the IP stereoscopic television 70 capture and reproduce a wide range of light because the range of capturing and reproducing the light from the subject 10 is limited to the position of the viewer. There is no need to do it. From this, it is thought that the binocular stereoscopic camera 20 should just image | photograph the light which injects into the range of the pupil diameter of a viewer.

Therefore, the robot camera control means 93 instructs the binocular stereoscopic camera 20B to take a stereo image for each preset distance.
For example, the robot camera control means 93 gives a command to shoot a stereo image at a preset distance (for example, 1 mm) within the range of the pupil diameter (for example, 6 mm) of the viewer. Output to 20B.

  Then, the binocular stereoscopic camera 20B outputs a plurality of stereo images taken from different viewpoints to the resolution conversion device 40. Then, the resolution conversion device 40 generates a set of viewpoint element image groups from each stereo image, and outputs them to the image conversion device 60B.

  Hereinafter, a method for converting a plurality of stereo images into a set of viewpoint element image groups will be described. Here, the right captured image before the binocular stereoscopic camera 20B moves is referred to as a reference right captured image, and the right captured image after the binocular stereoscopic camera 20B moves 1 mm to the right is referred to as a 1 mm right captured image.

  The pixel value assigning unit 44c assigns the pixel value (center pixel value) of the corresponding pixel of the reference right captured image to the pixel of the right viewpoint element image at the position where light is emitted to the right eye of the viewer before moving. Also, the pixel value assigning unit 44c applies the 1 mm right captured image at the same position as the corresponding pixel of the reference right captured image to the pixel of the viewpoint element image at the position where light is emitted to the right eye of the viewer after moving 1 mm. The pixel value of the pixel (1 mm right moving pixel value) is assigned.

Further, the pixel value assigning unit 44c divides the right viewpoint element image by a line that bisects a straight line connecting the pixel positions to which the center pixel value and the 1 mm right moving pixel value are assigned. Then, when the divided region includes a pixel to which the central pixel value or the 1 mm moving pixel value is assigned, the image composition unit 62B assigns the central pixel value or the 1 mm moving pixel value to each pixel in the region.
Thus, the IP stereoscopic video estimation system 1B according to the first modification of the present invention can reproduce the optical image of the subject 10 at the viewing position of the viewer.

(Other variations)
In the second and third embodiments described above, the IP stereoscopic image estimation devices 2A and 2B are configured by a plurality of devices, but these devices may be integrated.

  In each of the above-described embodiments, the devices constituting the IP stereoscopic video estimation devices 2, 2A, 2B have been described as independent hardware, but the present invention is not limited to this. For example, each device can be realized by a program that causes hardware resources such as a CPU, a memory, and a hard disk included in a computer to operate in cooperation with each other as described above.

1, 1A, 1B IP stereoscopic video estimation system 2, 2A, 2B IP stereoscopic video estimation device 20 Binocular stereoscopic camera (Binocular stereoscopic imaging device)
30 Recording / playback device 40 Resolution conversion device 41 Resolution characteristic determining means 41a Maximum resolution calculating means 41b Viewing resolution calculating means 41c Resolution characteristic calculating means 42 Depth value calculating means 43 Pixel position calculating means (first pixel position calculating means)
44 Viewpoint element image group generation means 44a Corresponding pixel resolution determination means 44b Pixel acquisition area calculation means 44c Pixel value allocation means 50 Binocular stereoscopic television 60, 60B Image conversion devices 61, 61B Pixel position calculation means (second pixel position calculation means)
62 Image composition means 70 IP stereoscopic television 80 Viewing position detection camera 90 Viewing position tracking device 91 Viewing position detection means 92 Viewing position tracking control means 93 Robot camera control means

Claims (7)

The stereo image obtained by converting the resolution of the stereo image captured by the binocular stereoscopic imaging device in accordance with the resolution of the elemental image group displayed on the IP stereoscopic video display device including the display element and the lens array, and converting the resolution. Is output as a viewpoint element image group in which the resolution of the element image group is estimated for each viewpoint representing the right eye and the left eye of the viewer,
Resolution characteristic determining means for determining a smaller one of the maximum resolution or viewing resolution of the element image group as a resolution characteristic in a preset depth range;
A depth value calculating means for inputting the stereo image and calculating a depth value for each pixel of the input stereo image;
For each of the viewpoints, by ray tracing, a first pixel position for calculating a first reference pixel of the viewpoint element image unit located virtual line reaches the display device passes through the center of the perspectives RaKaname element lens A calculation means;
A corresponding pixel resolution determining unit for determining a corresponding pixel of the stereo image at a position corresponding to the first reference pixel, and determining a resolution according to a depth value of the corresponding pixel from the resolution characteristic;
A pixel acquisition area calculation means for calculating a pixel acquisition area obtained by enlarging the diameter of the element lens at a ratio between the maximum resolution and the resolution of the corresponding pixel around the corresponding pixel;
For each viewpoint, a pixel included in the pixel acquisition region is acquired from the stereo image, and a viewpoint element image to which the first reference pixel belongs in the viewpoint element image group based on the acquired pixel value of the pixel Pixel value assigning means for assigning a pixel value of each of the pixels;
An IP stereoscopic video estimation apparatus comprising: The second reference pixel of the element image group located on the virtual line passing through the center of the element lens and reaching the display element from the viewpoint intermediate position representing the intermediate position of the right eye and the left eye of the viewer by the ray tracing method Second pixel position calculating means for calculating
A divided region obtained by dividing the element image to which the second reference pixel belongs is divided by a viewpoint boundary line, an image region corresponding to the divided region is extracted from the viewpoint element image group for each viewpoint, and extracted from different viewpoints. Image synthesizing means for generating the element image by synthesizing the image area;
The IP stereoscopic video estimation apparatus according to claim 1, further comprising: A viewer image obtained by photographing the viewer, and viewing position detection means for detecting a viewing position of the viewer with respect to a center position of the lens array from the input viewer image;
Viewing position tracking control means for instructing the image composition means to change the viewpoint boundary line according to the viewing position;
The IP stereoscopic video estimation apparatus according to claim 2, further comprising:   4. The IP solid according to claim 3, wherein the viewing position tracking control unit further instructs the image composition unit to change a display position of the element image group in accordance with the viewing position. Video estimation device. Robot camera control means for instructing a stereoscopic imaging apparatus that is a movable robot camera to capture the stereo image following the viewing position;
The IP stereoscopic video estimation apparatus according to claim 3 or 4, further comprising:   The IP stereoscopic video estimation apparatus according to claim 5, wherein the robot camera control unit instructs the robot camera to take the stereo image at a predetermined distance.   An IP stereoscopic video estimation program for causing a computer to function as the IP stereoscopic video estimation device according to claim 1.

Images