JP2020129773A - Stereoscopic image generation system, stereoscopic image generation device, and program thereof - Google Patents

Abstract

To provide a stereoscopic image generation device capable of compressing the depth of a stereoscopic image even when a lens array of the stereoscopic image display device is a delta array.SOLUTION: A stereoscopic image generation device 30 includes parameter input means 31 that inputs various parameters, pixel position conversion table storage means 32 that stores a pixel position conversion table, arithmetic means 33 that calculates the amount of axis deviation between an imaging lens and an imaging element included in an imaging camera, imaging camera control means 34 that shifts the axes of the imaging lens and the imaging element by an amount of axis deviation, and instructs the imaging camera to capture a subject image, subject image input means 35 that inputs the subject image, and element image group generation means 36 that generates an element image group from the subject image captured by an imaging camera array.SELECTED DRAWING: Figure 15

Description

The present invention relates to a stereoscopic image generation system that generates an element image group displayed by a stereoscopic image display device from a subject image according to the depth of the stereoscopic image display device, a stereoscopic image generation device, and a program thereof.

In the conventional integral photography (IP) method, a single camera images a subject through a lens array in which a plurality of microlenses are arranged. At this time, since the camera images the focal plane of the lens array, each minute lens forming the lens array has the same function as a minute camera. As a result, the image of the subject captured through the lens array becomes an element image group in which minute images (element images) corresponding to the positions of the minute lenses are arranged. The elemental image group is an image in which light ray information from the subject is stored, and the number of light rays that can be stored depends on the resolution of the camera. For this reason, a high-resolution camera is required for image capturing in the IP method. It is also necessary to correct the influence of chromatic aberration and distortion of the image pickup optical system.

Further, conventionally, a technique for generating an IP type element image group by using computer graphics (CG) has been studied. In the conventional technology, a virtual three-dimensional space is generated in a computer, a three-dimensional subject and a background (set) are reproduced in the same manner as in the real world, and rays of light from the subject are traced in the virtual three-dimensional space. Generate an image group. The methods described in the following Patent Documents 1 and 2 and Non-Patent Documents 1 and 2 are known as methods for generating an element image by CG. These elemental image generation methods trace a light ray from a subject through a depth control lens that controls the light ray from the subject. In these methods, it is necessary to capture images by the camera from the observer side as many times as the number of pixels of the element image group. That is, since the element image generation method of Non-Patent Document 1 uses the ray tracing method for each pixel, it is equivalent to a camera that captures an image for each pixel, and the number of images is the same as the number of pixels of the element image group. .. Therefore, for further speeding up, as a conventional technique, there is also known a method of capturing a CG object with a camera by orthographic projection and converting these images into a group of elemental images (Patent Document 3 and Non-Patent Document 3). In such an element image conversion method, the number of times of imaging with a camera is the number of pixels forming the element image.

Japanese Patent No. 4567422 International Publication No. 00/059235 JP, 2011-234142, A

Spyros S. Athineos et al., “Physical modeling of a microlens array setup for use in computer generated IP,” Proc of SPIE-IS&T Electronic Imaging, Vol. 5664 pp.472-479, 2005 Nakajima K. et al., “High-speed image creation method for 3D display using the principle of Integral Photography”, Journal of Image Information Media Vol. 54, No. 3, pp.420-425 (2000). M. Katayama et al., “A method for converting three-dimensional models into auto-stereoscopic images based on integral photography,” Proc of SPIE-IS&T Vol.6805 68050Z-1 68050Z-8

Here, in the IP method, the resolution of the stereoscopic image decreases as the stereoscopic image moves away from the lens array of the display device in the depth direction. Therefore, in the IP method, it is better to compress the depth of the stereoscopic image within a range where a sufficient resolution can be obtained on the display device. However, the above-mentioned conventional technique has a problem that the depth of the stereoscopic image is not compressed. Furthermore, when the lens array of the display device is a delta array and each pixel of the imaging camera is a square array, both array structures are different, and thus it is not possible to simply generate element images.

Therefore, it is an object of the present invention to provide a stereoscopic image generation system capable of compressing the depth of a stereoscopic image, a stereoscopic image generation device, and a program thereof even when the lens array of the stereoscopic image display device is a delta array.

In view of the above-mentioned problems, a stereoscopic image generation system according to the present invention is an optical system in which an image pickup camera array including an image pickup camera in which pixels of an image pickup device are squarely arranged, a display element, and a plurality of optical elements are arranged in a delta shape. According to the depth of the stereoscopic image display device including the element array, the stereoscopic image generation device generates an element image group displayed by the stereoscopic image display device.

In such a stereoscopic image generation system, since the image pickup camera corresponds to an optical element array in which optical elements are arranged in a delta shape, the number of pixels of the image pickup element is equal to or more than the number of optical elements.
The stereoscopic image generation device uses the parameter input means to determine the maximum depth display distance of the stereoscopic image display device, the focal length of the optical element, and the center pixel of the element images forming the element image group to the calculation target pixel of each imaging camera. The inter-pixel distance, the differential distance from the optical element array to the maximum rear depth display position, the size of the image sensor included in the image capturing camera, and the size of the display element are input.

The stereoscopic image generation device calculates the position of the imaging camera based on the maximum depth display distance, the focal length of the optical element, and the inter-pixel distance by the position calculation means.
In the stereoscopic image generation apparatus, the image pickup lens included in the image pickup camera based on the inter-pixel distance, the difference distance, the focal length of the optical element, the size of the image pickup element, and the size of the display element by the axis shift amount calculation unit. And the amount of axis deviation between the image sensor and the image sensor is calculated.

The stereoscopic image generation device moves the imaging camera to the position calculated by the position calculation means by the imaging camera control hand, and shifts the imaging lens and the imaging element by the axial deviation amount calculated by the axial deviation amount calculation means.
The stereoscopic image generation device generates, by the stereoscopic image generation means, an element image group in which the element images are arranged in a delta shape from the subject image captured by the imaging camera array.
In this way, the stereoscopic image generation apparatus can compress the depth of the stereoscopic image by shifting the optical axis of the imaging lens included in the imaging camera and the central axis of the imaging element.

Further, in view of the above-mentioned problems, the stereoscopic image generation device according to the present invention is configured to display an image sensor according to the depth of a stereoscopic image display device including a display element and an optical element array in which a plurality of optical elements are arranged in a delta shape. The element image group displayed by the stereoscopic image display device is generated from the subject image generated by the image pickup camera array including the virtual image pickup camera in which each pixel is squarely arranged.

Such a stereoscopic image generation device uses the parameter input means to determine the maximum depth display distance of the stereoscopic image display device, the focal length of the optical element, and the center pixel of the element images forming the element image group to the calculation target pixel of each imaging camera. The inter-pixel distance, the difference distance from the optical element array to the maximum rear depth display position, the size of the image sensor included in the image capturing camera, and the size of the display element are input.

The stereoscopic image generation device calculates the position of the imaging camera based on the maximum depth display distance, the focal length of the optical element, and the inter-pixel distance by the position calculation means.
In the stereoscopic image generation apparatus, the image pickup lens included in the image pickup camera based on the inter-pixel distance, the difference distance, the focal length of the optical element, the size of the image pickup element, and the size of the display element by the axis shift amount calculation unit. And the amount of axis deviation between the image sensor and the image sensor is calculated.

The stereoscopic image generation device sets the position calculated by the position calculation unit and the axial shift amount calculated by the axial shift amount calculation unit by the imaging camera control unit. Since this virtual imaging camera corresponds to an optical element array in which optical elements are arranged in a delta shape, the number of pixels of the imaging element is equal to or more than the number of optical elements.
The stereoscopic image generation device generates a subject image via the imaging camera array by the stereoscopic image generation means, and generates an element image group in which element images are arranged in a delta shape from the generated subject image.
As described above, the stereoscopic image generation apparatus can compress the depth of the stereoscopic image by shifting the optical axis of the imaging lens of the virtual imaging camera and the central axis of the imaging element.

Here, the stereoscopic image generation device may generate the element image group by acquiring the pixel value of the pixel position corresponding to the amount of the axis shift from the subject image, instead of shifting the axis of the image pickup lens and the image pickup element.
As described above, the stereoscopic image generation apparatus can compress the depth of the stereoscopic image by performing the same processing as that of shifting the optical axis of the imaging lens included in the imaging camera and the central axis of the imaging element.

The stereoscopic image generation device described above can also be realized by a program that causes hardware resources such as a CPU, a memory, and a hard disk included in a computer to operate as the stereoscopic image generation device described above.

According to the present invention, the following excellent effects are exhibited.
According to the present invention, an element image group in which the depth of a stereoscopic image is compressed is generated from a plurality of subject images acquired by an imaging camera array in which the optical axis of the imaging lens and the central axis of the imaging element are displaced, and thus the stereoscopic image is generated. Even when the lens array of the display device is a delta array, it is possible to display a stereoscopic image in a high-resolution depth range.

Further, according to the present invention, the pixel value of the pixel position corresponding to the amount of axis deviation is obtained from the subject image, and the element image group in which the depth of the stereoscopic image is compressed is generated. Even in the case of the array, a stereoscopic image can be displayed in the depth range with high resolution.

It is a schematic block diagram which shows the outline of a stereoscopic image display apparatus. It is a schematic block diagram which shows the outline of a stereoscopic image pickup device. It is explanatory drawing explaining the light ray which injects into each pixel of an image sensor in the stereo image imaging device of FIG. It is explanatory drawing explaining the light ray which injects into the lowest pixel in each element image imaged by an image sensor in FIG. It is explanatory drawing explaining the light ray which injects into the pixel of the center in each element image which an image sensor image|photographs in FIG. It is explanatory drawing explaining the light ray which injects into the uppermost pixel in each element image imaged by an image sensor in FIG. It is explanatory drawing explaining the imaging camera arrange|positioned at the intersection of the light ray of FIG. 4 and the principal plane of a depth control lens. It is explanatory drawing explaining the imaging camera arrange|positioned at the intersection of the light ray of FIG. 3 and the principal plane of a depth control lens. It is explanatory drawing explaining the imaging camera which left|separated in FIG. 8 only the predetermined distance from the main plane of the depth control lens. It is explanatory drawing explaining arrangement|positioning of a virtual lens array. (A)-(d) is a figure explaining the light ray which injects into an image sensor. It is a graph showing the non-linear compression characteristic by the depth control lens. FIG. 7 is an explanatory diagram illustrating compression of depth due to an axis shift in the first embodiment. FIG. 7 is an explanatory diagram illustrating compression of depth due to an axis shift in the first embodiment. It is a block diagram which shows the structure of the stereo image display system which concerns on 1st-4th embodiment. FIG. 3 is an external view showing the external appearance of the imaging camera array in the first embodiment. In the first embodiment, it is an explanatory diagram for explaining the axis deviation of the imaging camera, (a) is a diagram when there is no axis deviation, (b) is a diagram when there is an axis deviation. In the first embodiment, (a) is an explanatory diagram illustrating a pixel array of an image sensor, (b) is an explanatory diagram illustrating a lens array of a display lens array, and (c) is an array of element images. It is an explanatory view explaining. In 1st Embodiment, (a) is explanatory drawing explaining a square lens and a circular lens, (b) is explanatory drawing explaining a rectangular lens and a honeycomb lens. In the first embodiment, it is an explanatory diagram for explaining the setting of the pixel position conversion table in the case of a square lens, (a) is a diagram showing an image sensor, (b) is a diagram showing an element image group. In the first embodiment, it is an explanatory diagram for explaining the setting of the pixel position conversion table in the case of a rectangular lens, (a) is a diagram showing an image sensor, (b) is a diagram showing a group of element images. 3 is a flowchart showing an operation of the stereoscopic image generation device according to the first embodiment. In the second embodiment, it is an explanatory diagram for explaining the setting of the pixel position conversion table in the case of a square lens, (a) is a diagram showing an image sensor, (b) is a diagram showing a group of element images. In the second embodiment, it is an explanatory view for explaining the setting of the pixel position conversion table in the case of a rectangular lens, (a) is a diagram showing an image sensor, (b) is a diagram showing an element image group. In the third embodiment, it is an explanatory diagram for explaining the setting of the pixel position conversion table in the case of a square lens, (a) is a diagram showing an image sensor, (b) is a diagram showing an element image group. In the third embodiment, it is a diagram for explaining the size of the image sensor in the case of a rectangular lens, (a) is a diagram showing a normal size of the image sensor, (b) is an image pickup in which the horizontal pixel size is doubled. It is a figure which shows an element, and (c) is a figure which shows the imaging element which reduced the vertical pixel size to 1/2. In 4th Embodiment, it is explanatory drawing explaining the setting of a pixel position conversion table, (a) is a figure which shows the image sensor which doubled the horizontal pixel size, (b) is a vertical pixel size 1/. It is a figure which shows the image sensor which was set to 2, (c) is a figure which shows an image sensor, (d) is a figure which shows an element image group. It is a block diagram which shows the structure of the stereo image display system which concerns on 5th Embodiment. It is a flowchart which shows operation|movement of the stereo image generation apparatus which concerns on 5th Embodiment. In 6th Embodiment, it is explanatory drawing explaining the acquisition of the pixel value of the pixel position which considered the amount of axis shifts, (a) is a figure when there is no axis shift, (b) is a figure when there is a axis shift. FIG.

Hereinafter, each embodiment of the present invention will be described in detail with reference to the drawings as appropriate. In addition, in each embodiment, the same reference numerals are given to the same means and processes, and the description thereof will be omitted.
After explaining the depth compression principle, the stereoscopic image display system according to the present embodiment will be described.

[Principle of depth compression]
Hereinafter, as the depth compression principle, the relationship between the stereoscopic image display device and the stereoscopic image pickup device, the replacement by the image pickup camera, the non-linear compression characteristic of the stereoscopic image pickup device, and the depth compression due to the axis shift will be described in order.

<Relationship Between Stereoscopic Image Display Device and Stereoscopic Image Imaging Device>
The stereoscopic image display device 10 is a display that displays a stereoscopic image of the subject α by the IP method. For example, the stereoscopic image display device 10 includes a display element 11 and a display lens array 12, as shown in FIG.

The display element 11 is composed of pixels arranged in a square, for example, and displays a group of elemental images (stereoscopic image) via the display lens array 12.
The display lens array 12 includes display element lenses 13 (for example, minute convex lenses) arranged in a delta shape (barrel-stacking arrangement).

As shown in FIG. 2, the stereoscopic image pickup device 90 is a camera that picks up an image of a subject α by an IP method, and includes an image pickup element 91, an image pickup lens array 92, and a depth control lens 94. Here, in the stereoscopic image pickup device 90, the image pickup lens array 92 is arranged in front of the image pickup element 91, and the depth control lens 94 is arranged between the image pickup lens array 92 and the subject α.

The image pickup element 91 is a general image pickup element that picks up an element image group via the image pickup lens array 92 and the depth control lens 94. The image pickup element 91 is located at the focal length LAf of the image pickup element lens 93 described later.
The imaging lens array 92 is a two-dimensional array of imaging element lenses 93 (for example, minute convex lenses).
The depth control lens 94 is a general convex lens. The focal length of the depth control lens 94 is Df. Here, the imaging lens array 92 is located at the focal length Df.

Since the stereoscopic image capturing device 90 captures a high quality stereoscopic image, an image capturing element 91 having a large number of pixels and a high density image capturing lens array 92 are required. Further, the stereoscopic image capturing apparatus 90 needs a large-diameter, short-focus depth control lens 94 to capture a wide range. As described above, since the stereoscopic image pickup device 90 has a large restriction, a method of replacing the stereoscopic image pickup device 90 with one or more general image pickup cameras 21 (FIG. 17) will be examined.

The stereoscopic image display device 10 and the stereoscopic image capturing device 90 are greatly different in the presence or absence of the depth control lens 94. On the other hand, the size ratio of the display element 11 and the image pickup element 91 and the ratio of the distance and the focal length of the display lens array 12 and the image pickup lens array 92 are usually the same. Due to such a relationship, the parameters of the stereoscopic image display device 10 that does not include the depth control lens 94 can be treated as the parameters of the stereoscopic image capturing device 90.

<Replacement with imaging camera>
FIG. 3 illustrates how the image pickup device 91 obtains a light ray from the subject α. Note that, in FIG. 3, only the light rays passing through the principal points of the respective imaging element lenses 93 are illustrated and the light rays are illustrated by arrows in the direction from the image sensor 91 to the subject α for easy understanding of the description (FIG. 4). ~Similar to FIG. 9).

It is assumed that the lens interval LAp of the imaging lens array 92 is an integral multiple of the pixel interval PlXp of the image sensor 91. As shown in FIG. 3, the light rays incident on the respective pixels of the image pickup element 91 have a point where they intersect with each other on the main plane β of the depth control lens 94 on the subject α side. Therefore, a ray group as shown in FIG. 4 is incident on the bottom pixel in each element image captured by the image sensor 91. A ray group as shown in FIG. 5 is incident on the pixel at the center of each element image captured by the image sensor 91. Further, a ray group as shown in FIG. 6 is incident on the uppermost pixel in each element image captured by the image sensor 91.

Here, focusing on FIG. 4, light rays intersect at a single point on the main plane β of the depth control lens 94 on the subject α side. As shown in FIG. 7, by arranging the image pickup camera 21 so that the principal point of the image pickup lens 27 of the image pickup camera 21 and the intersection point coincide with each other, a light ray incident on the bottom pixel in each element image. Can be imaged by one imaging camera 21. The light rays incident on the pixels other than the lowest pixel in each element image can be imaged by the single imaging camera 21 similarly to the lowest pixel.

Further, the number of imaging cameras 21 is the same as the number of pixels of the elemental image. For example, when the element image has 10 pixels in the vertical direction and 10 pixels in the horizontal direction, that is, 100 pixels in total, the stereoscopic image capturing device 90 can be replaced by 100 image capturing cameras 21. In FIG. 8, the bottom, center, and top pixels in each element image have been described, so the three-dimensional image capturing device 90 is replaced by three image capturing cameras 21.

<Non-linear compression characteristic of stereoscopic image pickup device>
In order to compress (compress) the depth of the stereoscopic image, as shown in FIG. 9, the image pickup device 91 and the image pickup lens array 92 are separated from the main plane γ of the depth control lens 94 on the image pickup device 91 side by Δf. The depth of the stereoscopic image is compressed according to the distance Δf from the principal plane γ.

As shown in FIG. 10, in the stereoscopic image capturing apparatus 90, a virtual lens array (virtual lens array) 95 is generated by the depth control lens 94 on the opposite side of the imaging lens array 92 with the depth control lens 94 interposed therebetween. It A is the distance from the depth control lens 94 to the virtual lens array 95. The distance from the depth control lens 94 to the imaging lens array 92 is B.

As shown in FIG. 11, a ray group when the distance Δf from the principal plane γ is changed to 1.25 times, 1.5 times, 1.75 times, and 2.0 times the focal length Df of the depth control lens 94. Is illustrated. In FIG. 11, the horizontal axis represents the position in the depth direction, and the vertical axis represents the position in the vertical direction. Further, 0 on the horizontal axis is the depth position of the image sensor 91, and 0 on the vertical axis is the lowest pixel position in the image sensor 91.

The distances A and B can be obtained by the following equation (1) when the distance Δf from the principal plane γ changes from 0 to the focal length Df.

As shown in FIG. 12, the nonlinear compression characteristic of the depth control lens 94 can be obtained from the relationship between the distances A and B. In the graph of FIG. 12, the horizontal axis represents the value obtained by normalizing the distance A with the focal length Df, and the vertical axis shows the value obtained by normalizing the distance B with the focal length Df.

In the graph of FIG. 12, when the center of the subject α is located at a position twice the focal length Df (distance A=2.0) on the subject α side with the depth control lens 94 as a reference, the image sensor 91 side This indicates that the center of the optical image of the subject α is formed at a position twice the focal length Df (distance B=2.0). In the graph of FIG. 12, when the center of the subject α is arranged at a position three times the focal length Df on the subject α side (distance A=3.0) with the depth control lens 94 as a reference, the image sensor 91 is arranged. It indicates that the center of the optical image of the subject α is formed at a position 1.5 times the focal length Df (distance B=1.5) on the side.

That is, the display position of the optical image of the subject α depends on the distance from the depth control lens 94 to the imaging lens array 92. As a result, the range from the position of infinity to the position of the focal length Df (main plane β) on the subject α side (range of distance A−focal length Df) is the position of the focal length Df on the imaging element 91 side ( It falls within the range from the principal plane γ) to the position twice the focal length Df. In other words, in the stereoscopic image pickup device 90, an infinite space can be compressed to the range of the focal length Df. Therefore, by adjusting the focal length Df of the depth control lens 94 and the distance Δf from the principal plane γ, the depth of the stereoscopic image can be compressed in the high resolution range in the stereoscopic image display device 10.

<Compression of depth due to misalignment>
As shown in FIG. 13, consider a case where the distance from the depth control lens 94 to the imaging lens array 92 is L1. In this case, the light ray incident on the uppermost pixel in each elemental image passes through the point P on the principal plane β at an angle θ.
As shown in FIG. 14, let us consider a case where the distance from the depth control lens 94 to the imaging lens array 92 is L2 (here, distance L1<distance L2). In this case, the light ray incident on the uppermost pixel in each elemental image passes through the same point P on the principal plane β at an angle φ. The angles θ and φ are angles formed by the light beam and the optical axis of the stereoscopic image pickup device 90 (θ≠φ).

Here, when the distance from the depth control lens 94 to the image pickup lens array 92 is changed from L1 to L2, the angle of the light beam that is incident on the uppermost pixel in each elemental image after passing through the depth control lens 94. The angle changes from θ to φ. That is, changes in the distances L1 and L2 from the depth control lens 94 to the imaging lens array 92 appear as changes in the angles θ and φ, and the shooting range of the stereoscopic image pickup device 90 (in accordance with the changes in the angles θ and φ). The angle of view) changes.

On the other hand, since the stereoscopic image display device 10 (FIG. 1) does not include the depth control lens 94, the angle change of the light beam unlike the stereoscopic image capturing device 90 does not occur. Therefore, the depth of the stereoscopic image can be compressed by shifting the axis of the imaging camera 21 on the main plane β located at the focal length Df.

The depth compression principle described above is summarized as follows.
By adjusting the distance Δf of the stereoscopic image pickup device 90, the depth of the stereoscopic image can be compressed. Further, the stereoscopic image capturing device 90 can be replaced by one or more image capturing cameras 21, and the axis of the image capturing camera 21 is shifted so that the depth compression of the stereoscopic image is performed as in the case where the distance Δf is adjusted by the stereoscopic image capturing device 90. The effect of is obtained. Then, the parameters of the stereoscopic image display device 10 can be treated as the parameters of the stereoscopic image capturing device 90 to calculate the axis shift of the image capturing camera 21.

Here, in the image pickup device 21, the pixels are arranged in a square, whereas in the display lens array 12, the display element lenses 13 are arranged in a delta shape. In this way, since the array structure is different on the image capturing side and the display side, when acquiring each pixel of the subject image captured by the image sensor 21, it is necessary to devise a method described in each of the following embodiments.

(First embodiment)
[Stereoscopic image display system]
The stereoscopic image display system (stereoscopic image generation system) 1 according to the first embodiment of the present invention will be described with reference to FIG.
The stereoscopic image display system 1 displays a stereoscopic image of the subject α, and includes a stereoscopic image display device 10, an imaging camera array 20, and a stereoscopic image generation device 30, as shown in FIG. 15.

The image pickup camera array 20 is an array of a plurality of image pickup cameras 21 as shown in FIG. Specifically, the image pickup camera array 20 has as many image pickup cameras 21 as the number of horizontal pixels of the element image arranged in the horizontal direction, and has as many image pickup cameras 21 as the number of vertical pixels of the element image arranged in the vertical direction. .. Note that in FIG. 16, the illustration of the imaging camera 21 is partially omitted in order to make the drawing easier to see.

The imaging camera array 20 can change the position (position in the horizontal direction, the vertical direction, and the depth direction) of each imaging camera 21 according to a command from the stereoscopic image generation device 30 (imaging camera control unit 34).

In the example of FIG. 16, the imaging camera array 20 moves in the depth direction along a rail formed on the floor surface in response to a command from the imaging camera control unit 34. Further, the image pickup camera array 20 includes 10 stages of elevating members 22 on which 10 image pickup cameras 21 are mounted. Each elevating member 22 moves in the vertical direction along the rail formed on the two columns 23 in response to a command from the imaging camera control means 34. Each imaging camera 21 moves in the horizontal direction along the rail formed on the elevating member 22 in response to a command from the imaging camera control means 34.

The imaging camera 21 captures the subject α and outputs the captured image (subject image) to the stereoscopic image generation device 30 (subject image input means 35). In the present embodiment, 100 imaging cameras 21 capture subject images from different positions and output them to the subject image input means 35.

Here, the imaging camera 21 includes an imaging element 25 and an imaging lens 27, as shown in FIG. The number of pixels of the image sensor 25 is equal to or more than the number of the display element lenses 13, and the pixels are arranged in a square. Further, since the image pickup camera 21 compresses the depth, it is possible to shift the central axis of the image pickup element 25 and the optical axis of the image pickup lens 27 in accordance with a command from the image pickup camera control means 34 (lens shift function). .. The central axis is an axis that passes through the center position of the image pickup device 25 and is perpendicular to the image pickup surface of the image pickup device 25. Further, the image pickup camera 21 can change the angle of view according to a command from the image pickup camera control means 34.

In FIG. 17A, the central axis of the image pickup device 25 and the optical axis of the image pickup lens 27 are aligned with each other and are not misaligned. On the other hand, in FIG. 17B, the central axis of the image pickup element 25 and the optical axis of the image pickup lens 27 do not coincide with each other, and the axes are deviated (axis deviation amount dy). Here, in the image pickup camera 21, the optical axis of the image pickup lens 27 is perpendicular to the image pickup surface of the image pickup device 25, regardless of the axis shift amount dy. Further, the image pickup camera 21 may move only one of the image pickup device 25 and the image pickup lens 27, or may move both the image pickup device 25 and the image pickup lens 27. In this embodiment, the image sensor 25 is moved.

[Configuration of stereoscopic image generation device]
The stereoscopic image generation device 30 generates an element image group displayed by the stereoscopic image display device 10 from a subject image captured by the imaging camera array 20. As shown in FIG. 15, the stereoscopic image generation device 30 includes a parameter input unit 31, a pixel position conversion table storage unit 32, a calculation unit 33, an imaging camera control unit 34, a subject image input unit 35, and an element image. And a group generation means (stereoscopic image generation means) 36.

The parameter input means 31 is for inputting various parameters necessary for generating the element image group. For example, the parameter input unit 31 inputs the specifications of the stereoscopic image display device 10 and the specifications of the imaging camera array 20 as parameters. Then, the parameter input unit 31 outputs the input parameter to the calculation unit 33.

The specifications of the stereoscopic image display device 10 include the maximum depth display distance of the stereoscopic image display device 10, the focal length of the display element lens 13, the differential distance from the display lens array 12 to the rear maximum depth display position, and the display. The size of the element 11 is included.
The specifications of the image pickup camera array 20 include the inter-pixel distance from the center pixel of the element image to the calculation target pixel and the size of the image pickup element 25.

The pixel position conversion table storage means 32 is a storage device such as a memory or an HDD (Hard Disk Drive) that stores the pixel position conversion table. In the present embodiment, the pixel position conversion table storage means 32 stores the pixel position conversion table for each imaging camera 21.

The pixel position conversion table is information on a pixel-by-pixel basis, and is information in which each pixel of the subject image captured by the imaging camera 21 is associated with each pixel of the element images that form the element image group. That is, in the pixel position conversion table, each pixel of the subject image is associated with each pixel of the element image in consideration of the difference in array structure between the imaging side and the display side.
Further, it can be referred to when the specifications of the stereoscopic image display device 10 do not change. In other words, the pixel position conversion table needs to be regenerated when the specifications of the stereoscopic image display device 10 change. Details of this pixel position conversion table will be described later.

The calculation unit 33 performs various calculations required by the imaging camera control unit 34, and includes a position calculation unit 331, an axis deviation amount calculation unit 333, a magnification calculation unit 335, and an angle of view calculation unit 337. ..

The position calculation unit 331 refers to the parameter input from the parameter input unit 31, and based on the maximum depth display distance, the focal length of the display element lens 13, and the inter-pixel distance, an image forming the image pickup camera array 20. The position of the camera 21 is calculated.

Specifically, the position calculation unit 331 calculates the position y in the vertical direction of each imaging camera 21 using the following formula (2). In Expression (2), the maximum depth display distance is Df (unit is meter), the focal length of the display element lens 13 is DLAf (unit is meter), and the inter-pixel distance is Py (unit is meter). ..
It should be noted that the position calculation means 331 calculates the positions of the respective imaging cameras 21 in the horizontal direction and the depth direction, as in the vertical direction.

The maximum depth display distance Df is the maximum distance that the stereoscopic image display device 10 can display in the depth direction from the display element lens 13. That is, the rear maximum depth display position is the maximum depth display position from the display element lens 13 to the rear (the display element 11 side in FIG. 1) within the maximum depth display distance Df.
Here, the maximum depth display distance Df corresponds to the focal length Df of the depth control lens 94. For example, when the stereoscopic image display device 10 and the stereoscopic image capturing device 90 have the same size, the maximum depth display distance Df and the focal length Df also have the same value. When the sizes of the stereoscopic image display device 10 and the stereoscopic image capturing device 90 are different, the maximum depth display distance Df and the focal length Df have a relationship according to the size ratio (distance Δf from the main plane γ and differential distance. Δf is the same).

The inter-pixel distance Py is the distance from the central pixel of the elemental images forming the elemental image group to each pixel to be calculated. The number of imaging cameras 21 is the same as the number of pixels of the elemental images, and each imaging camera 21 and each pixel to be calculated have a one-to-one correspondence. That is, since the inter-pixel distance Py is included in the equation (2), the position y of the imaging camera 21 is different.

The axis shift amount calculation means 333 refers to the parameter input from the parameter input means 31, and refers to the inter-pixel distance, the difference distance, the focal length of the display element lens 13, the size of the image sensor 25, and the display element 11. The amount of axis deviation of each imaging camera 21 is calculated based on the size.

Specifically, the axis deviation amount calculating unit 333 calculates the axis deviation amount dy of the image pickup device 25 in the vertical direction using the following formula (3) (unit is meter). In Expression (3), the difference distance is Δf (unit is meter), the size of the image sensor 25 is Cs (unit is meter), and the size of the display element 11 is Ds (unit is pixel). The difference distance Δf is the distance from the display element lens 13 to the maximum rear depth display position. Here, since the inter-pixel distance Py is included in the equation (3), the axial shift amount dy of the imaging camera 21 is different.

The magnification calculation means 335 refers to the parameter input from the parameter input means 31, and ratios the distance A from the depth control lens 94 to the virtual lens array 95 and the distance B from the depth control lens 94 to the imaging lens array 92. Is calculated.

<Calculation of magnification>
Calculation of the magnification m will be described with reference to FIG.
As shown in FIG. 10, the distance B is the sum (total distance) of the maximum depth display distance Df and the difference distance Δf. The distance A is the distance from the position (Z3) separated from the imaging lens array 92 by the distance B to the virtual lens array 95.
Here, the depth position of the image sensor 91 is Z1, and this Z1 is the origin. The depth position of the imaging lens array 92 is Z2, and the depth position of the depth control lens 94 is Z3. Further, the depth position of the principal plane β is Z4, and the depth position of the virtual lens array 95 is Z5. In this case, the depth positions Z1 to Z5 can be expressed by the following equation (4).

Consider a case where the element image group is imaged in a state where the subject α is located closer to the image sensor 91 than the virtual lens array 95. In this case, when the element image group is displayed on the display element 11, a stereoscopic image in which the subject α is projected is formed.
On the other hand, consider a case where the element image group is imaged in a state where the subject α is located on the opposite side of the image sensor 91 with respect to the virtual lens array 95. In this case, when the element image group is displayed on the display element 11, a stereoscopic image in which the subject α is depressed is formed.

In an optical system having a nonlinear depth compression characteristic, the size of the subject α changes simultaneously with the depth compression. Therefore, it is necessary to enlarge or reduce the subject α so that the size of the subject α does not change before and after the depth compression.

Here, the size of the virtual lens array 95 is a value obtained by multiplying the size of the imaging lens array 92 by the magnification m. Then, in the optical system having the nonlinear depth compression characteristic, the size of the subject α is also enlarged or reduced according to the size of the virtual lens array 95. Therefore, the magnification calculation means 335 calculates the magnification m, which is the ratio of the distances A and B, using the following equation (5), and multiplies the calculated magnification m by the size of the subject α, so that the virtual lens array 95 is obtained. On the lens surface of, the subject α has the same size as before the depth compression.

The angle-of-view calculation unit 337 refers to the parameter input from the parameter input unit 31, and based on the maximum depth display distance Df, the size Cs of the image sensor 25, and the size Ds of the display device 11, the image of the imaging camera 21. The angle is calculated.

Specifically, the angle-of-view calculation unit 337 calculates the focal length Cf of the image pickup camera 21 using the following equation (6), and calculates the angle of view of the image pickup camera 21 from the focal length Cf. In the imaging camera array 20, all imaging cameras 21 have the same focal length Cf.

The calculating unit 33 outputs the calculated position of the image pickup camera 21 (positions in the vertical direction, the horizontal direction, and the depth direction), the amount of misalignment, the magnification, and the angle of view to the image pickup camera control unit 34 as control signals. ..

The image capturing camera control unit 34 controls the image capturing camera array 20 based on the control signal input from the calculating unit 33, and instructs the image capturing camera array 20 to capture a subject image. In the present embodiment, the imaging camera control unit 34 controls each imaging camera 21 that constitutes the imaging camera array 20.

Specifically, the imaging camera control means 34 moves each imaging camera 21 to the position of the control signal.
Further, the image pickup camera control means 34 shifts the image pickup element 25 of each image pickup camera 21 by the amount of axis deviation of the control signal. Further, the imaging camera control means 34 matches the angle of view of each imaging camera 21 with the angle of view of the control signal.
At this time, the imaging camera control means 34 makes it difficult to change the size of the subject α, and when the magnification m is larger than 1, the distance between the adjacent imaging cameras 21 and the axial deviation amount are reduced by the magnification m. Then, the position of each imaging camera 21 is corrected.

The subject image input means 35 is for inputting a subject image captured by the image capturing camera array 20. In the present embodiment, the subject image input means 35 inputs a subject image from each of the image pickup cameras 21 forming the image pickup camera array 20. Then, the subject image input means 35 outputs the subject images from the respective imaging cameras 21 to the element image group generation means 36.

The element image group generation means 36 is for generating an element image group from the subject image input from the subject image input means 35. In the present embodiment, the element image group generation unit 36 refers to the pixel position conversion table and converts the pixel position of the subject image captured by each imaging camera 21 to generate the element image group. That is, the element image group generation means 36 allocates each pixel of the subject image to each pixel of the element images forming the element image group in consideration of the difference in array structure between the imaging side and the display side.

Here, the element image group generation means 36 allocates each pixel at the top of the subject image to each pixel at the bottom of each element image. Further, the element image group generation means 36 assigns each pixel in the second row from the top of the subject image to each pixel in the second row from the bottom of each element image. Further, the element image group generation means 36 similarly inverts the horizontal direction as well, and assigns the pixels of the subject image to the pixels of the element image. In this way, the element image group generation unit 36 can prevent the depth inversion of the stereoscopic image by inverting the pixels of the subject image vertically and horizontally and assigning the pixels to the pixels of the element image.
After that, the element image group generation means 36 outputs the generated element image group to the stereoscopic image display device 10.

<Setting of pixel position conversion table: Square lens>
The setting of the pixel position conversion table when the display element lens 13 is a square lens will be specifically described below with reference to FIGS. 18 to 20.

As shown in FIG. 18A, in the image pickup device 25 of the image pickup camera 21, the pixels 26 are arranged in a square array, and the pixels 26 are aligned in the horizontal direction and the vertical direction.
As shown in FIG. 18B, in the display lens array 12 of the stereoscopic image display device 10, the display element lenses 13 are arranged in a delta shape. In the horizontal direction (one axis direction), the positions of the adjacent display element lenses 13 are displaced, and the display element lenses 13 are arranged in a zigzag. On the other hand, in the vertical direction (the other axial direction), the adjacent display element lenses 13 are aligned.
Therefore, the pixel positions of the respective pixels of the image sensor 25 arranged in a square are converted according to the display element lenses 13 arranged in a delta, and the element images 81 are arranged in a delta as shown in FIG. 18C. It is necessary to generate the element image group 80. Note that the element image 81 is shown by a thick line to make the drawing easy to see.

Here, it is assumed that the display element lens 13 is a square lens (for example, a square convex lens) as shown in FIG. Further, the display element lens 13 may be a circular lens (for example, a circular convex lens) as long as it has the same height and width.
The rectangular lens of FIG. 19B will be described later.

Hereinafter, the number (horizontal number) of the display element lenses 13 in the horizontal direction is Lw, and the number (vertical number) of the display element lenses 13 in the vertical direction is Lh.
Further, the pixel number (horizontal pixel number) of the element image 81 in the horizontal direction is EIx, and the pixel number (vertical pixel number) of the element image 81 in the vertical direction is EIy.
Further, the number of pixels of the image pickup device 25 in the horizontal direction (horizontal pixel number) is EIsx, and the number of pixels of the image pickup device 25 in the vertical direction (vertical pixel number) is EIsx.

Hereinafter, in order to simplify the explanation, as shown in FIG. 20B, the horizontal pixel number EIx of the element image 81 is 2 pixels, and the vertical pixel number EIy of the element image 81 is 2 pixels. Therefore, as shown in FIG. 20A, the number of the imaging cameras 21 is two, which is the same as the number of horizontal pixels EIx of the element image 81 in the horizontal direction, and two that is the same as the number of vertical pixels EIx of the element image 81 in the vertical direction. , Total 4 units. In FIG. 20 (a), the illustrated imaging element 25 ₍₂₅ 1 to 25 ₄₎ for each imaging camera 21 is provided.

Further, it is assumed that the horizontal number Lw of the display element lenses 13 is four and the vertical number Lh of the display element lenses 13 is three. Therefore, as shown in FIG. 20B, four element images 81 are arranged in the horizontal direction, the same number as the horizontal number Lw of the display element lenses 13 and the vertical number Lh of the display element lenses 13 in the vertical direction. Three of the same number are arranged.

The number of horizontal pixels EIsx of the image sensor 25 is a value (8 pixels) obtained by multiplying the number of horizontal pixels Lw of the display element lens 13 by the number of horizontal pixels EIx of the element image 81, as shown in the following formula (7). The vertical pixel number EIsy of the image sensor 25 is a value (6 pixels) obtained by multiplying the vertical number Lh of the display element lens 13 by the vertical pixel number EIx of the element image 81.

In this case, in the horizontal direction, the pixel position conversion table is set so that the pixels of the image sensor 25 are acquired at an interval (4 pixel intervals) obtained by doubling the horizontal pixel number EIx of the element image 81. Further, in the vertical direction, the pixel position conversion table is set so that the pixels of the image sensor 25 are acquired at intervals (between two pixels) equal to the vertical pixel number EIy of the element image 81.

Incidentally, as shown in FIG. 20 (a), the position of the pixel to be obtained from all of the imaging device ₂₅ 1 to 25 ₄ are the same. Moreover, it subjected to the pixel of the imaging device 25, for explaining the correspondence relationship between each pixel of the element image 81, the characters of A~D to each pixel of the image pickup device ₂₅ 1 to 25 ₄ and the element image 81.

Further, the pixel position conversion table, in accordance with the positional relationship between the imaging camera 21, associating pixels A~D the imaging device ₂₅ 1 to 25 ₄ and the element image 81. Specifically, in the pixel position conversion table associates the pixel A in the upper left of the imaging device 25 ₁ and a pixel A in the upper left of the element image 81. Further, the pixel position conversion table associates the pixel B in the upper right of the image pickup element 25 _2, and the upper right of the pixel of the element image 81 B. Further, the pixel position conversion table associates the pixel C at the lower left of the image pickup device 25 _3, a pixel C at the lower left of the element image 81. Further, the pixel position conversion table associates the pixel D of the imaging device 25 ₄ in the lower right, and a pixel D at the lower right of the element image 81.

Here, since the display lens array 12 is in the delta arrangement, as shown in FIG. 20B, the element image 81 _EV in the even-numbered column is more vertical than the element image 81 _{OD in the} odd-numbered column by the vertical pixel number EIy/2 (for example, It is shifted by 1 pixel) in the vertical direction. Therefore, for the even-numbered element images 81 _EV , the pixels 26 of the image sensor 25 located at the double EIx pixel interval+1 pixel in the horizontal direction and the EIy pixel interval+EIy/2 pixel in the vertical direction are associated with each other. .. In other words, in the image sensor 25, the pixel positions corresponding to the element images 81 _EV in the even columns and the element images 81 _{OD in the} odd columns are shifted by EIx pixels/2 in the horizontal direction and EIy pixels/2 in the vertical direction. For example, as shown in FIG. 20A, the upper left pixel 26 in the image sensor 25 corresponds to the element image 81 _OD in the odd-numbered column. In this case, the element image 81 _EV in the even-numbered column corresponds to the pixel 26 that is moved by 2 pixels in the horizontal direction and 1 pixel in the vertical direction from the pixel 26 at the upper left.

Since the pixel positions of the element image 81 are different between the odd-numbered column and the even-numbered column, it is necessary to change the setting of the pixel position conversion table depending on whether the vertical pixel number EIy/2 of the element image 81 is an integer or a decimal number. ..
First, when the vertical pixel number EIy/2 of the element image 81 is an integer, the pixel position conversion table may be set in the above procedure.

Next, when the vertical pixel number EIy/2 of the element image 81 becomes a decimal number, there is no pixel corresponding to the image pickup devices 25 _{1 to} 25 ₄ in a one-to-one manner, and thus the average pixel value of the upper and lower pixels is calculated as the element image 81. Is obtained as the pixel value of the relevant pixel. Further, the average pixel value of pixels according to the vertical and horizontal distances may be used as the pixel value of the pixel of the element image 81.

Further, if the vertical pixel number EIy/2 of the element image 81 does not become a decimal number, that problem does not occur. That is, the condition that the vertical pixel number EIy/2 of the element image 81 is an integer is when the vertical pixel number EIy is an even number. Therefore, when the vertical pixel number EIy is an odd number, the vertical pixel number EIsy of the image sensor 25 may be defined by using the following equation (8). Note that the horizontal pixel number EIx of the element image 81 is also doubled in order to make the aspect ratio of the image sensor 25 equal.

As described above, the pixel position conversion table may be set, and the set pixel position conversion table may be stored in the pixel position conversion table storage means 32 in advance. Even when the display element lens 13 is a circular lens, the setting of the pixel position conversion table is the same.

<Setting of pixel position conversion table: rectangular lens>
Hereinafter, the setting of the pixel position conversion table will be specifically described with reference to FIGS. 19 and 21 when the display element lens 13 is a rectangular lens.

Here, it is assumed that the display element lens 13 is a rectangular lens (for example, a rectangular convex lens) as shown in FIG. The display element lens 13 may be a honeycomb lens (for example, a regular hexagonal convex lens) as long as the height and width are different. The regular hexagonal display element lens 13 has a width-height ratio of 2:√3.

To simplify the explanation, as shown in FIG. 21B, it is assumed that the horizontal pixel number EIx of the element image 81 is 3 pixels and the vertical pixel number EIy of the element image 81 is 2 pixels. Therefore, as shown in FIG. 21A, the number of the imaging cameras 21 is three, which is the same as the number of horizontal pixels EIx of the element image 81 in the horizontal direction, and two that is the same as the number of vertical pixels EIx of the element image 81 in the vertical direction. , Total 6 units. In FIG. 21 (a), the illustrated six of the imaging device 25 to the imaging camera 21 is provided ₍₂₅ 1 to 25 _6).

Further, it is assumed that the horizontal number Lw of the display element lenses 13 is four and the vertical number Lh of the display element lenses 13 is three. Therefore, as shown in FIG. 21B, four element images 81 are arranged in the horizontal direction, the same number as the horizontal number Lw of the display element lenses 13, and the vertical number Lh of the display element lenses 13 in the vertical direction. Three of the same number are arranged.

The number of horizontal pixels EIsx of the image pickup device 25 is a value (12 pixels) obtained by multiplying the number of horizontal pixels Lw of the display element lens 13 by the number of horizontal pixels EIx of the element image 81, as shown in the equation (7). The vertical pixel number EIsy of the image sensor 25 is a value (6 pixels) obtained by multiplying the vertical number Lh of the display element lens 13 by the vertical pixel number EIx of the element image 81.

In this case, in the horizontal direction, the pixel position conversion table is set so that the pixels of the image sensor 25 are acquired at an interval (6 pixel intervals) obtained by doubling the horizontal pixel number EIx of the element image 81. Further, in the vertical direction, the pixel position conversion table is set so that the pixels of the image sensor 25 are acquired at intervals (two pixel intervals) equal to the vertical pixel number EIy of the element image 81.

Incidentally, as shown in FIG. 21 (a), the position of the pixel to be obtained from all of the imaging device ₂₅ 1 to 25 ₆ are identical. Further, in FIG. 21, an imaging device 25, for explaining the correspondence between the pixels of the element image 81, denoted by A~F characters to each pixel of the image pickup device ₂₅ 1 to 25 ₆ and the element image 81.

Further, the pixel position conversion table, in accordance with the positional relationship between the imaging camera 21, associating pixels A~F the imaging device ₂₅ 1 to 25 ₆ and the element image 81. Specifically, in the pixel position conversion table associates the pixel A in the upper left of the imaging device 25 ₁ and a pixel A in the upper left of the element image 81. Further, the pixel position conversion table associates the pixel B of the imaging device 25 ₂ on the central and the pixel B on the center of the element image 81. Further, the pixel position conversion table, and associates the pixel C at the top right of the image pickup element 25 _3, and a pixel C at the top right of the element image 81. Further, the pixel position conversion table associates the pixel D at the lower left of the image pickup device 25 _4, and a pixel D at the lower left of the element image 81. Further, the pixel position conversion table, and a pixel E of the imaging device 25 ₅ lower center, and a pixel E of the bottom center of the element images 81 and associates. Further, the pixel position conversion table associates the pixel F of the imaging device 25 ₆ the lower right, and the pixel F in the lower right of the element image 81.

Here, since the display lens array 12 is in the delta arrangement, as shown in FIG. 21B, the element image 81 _EV in the even-numbered column is more vertical than the element image 81 _{OD in the} odd-numbered column by the vertical pixel number EIy/2 (for example, It is shifted by 1 pixel) in the vertical direction. Therefore, in the image sensor 25, the pixel positions corresponding to the element images 81 _EV in the even columns and the element images 81 _{OD in the} odd columns are EIx pixels/2 in the horizontal direction and EIy pixels/2 in the vertical direction in the vertical direction. It shifts. For example, as shown in FIG. 21A, the pixel 26 at the upper left of the image sensor 25 corresponds to the element images 81 _OD in the odd columns. In this case, the even-numbered element image 81 _EV corresponds to the pixel 26 that is moved from the upper left pixel 26 by 3 pixels in the horizontal direction and 1 pixel in the vertical direction.

In addition, since the rectangular display element lens 13 has a different width-to-height ratio, if the length and width of the element image group 80 are reduced and expanded at the same ratio, the element image group 80 is distorted. Therefore, the width-height ratio of the element image group 80 is changed in advance in consideration of reducing or enlarging the length and width of the element image group 80 at the same ratio. Specifically, in the display element lens 13, the pixel position conversion table may be set so that the element image group 80 is reduced or enlarged according to the width-height ratio. For example, when the horizontal pixel size of the image sensor 25 is used as a reference, the vertical pixel size of the image sensor 25 may be set to 2/3. When the vertical pixel size of the image sensor 25 is used as a reference, the horizontal pixel size of the image sensor 25 may be set to 3/2. As a result, the pixel position of the image sensor 25 that contributes to the generation of the elemental image group 80 and the lens position of the display lens array 12 correspond to each other in the horizontal direction and the vertical direction.
Note that it is preferable to reduce the size rather than the enlargement because the three-dimensional image quality is improved.

As described above, the pixel position conversion table may be set, and the set pixel position conversion table may be stored in the pixel position conversion table storage means 32 in advance. Even when the display element lens 13 is a honeycomb lens, the setting of the pixel position conversion table is the same.

[Operation of stereoscopic image generation device]
The operation of the stereoscopic image generation device 30 will be described with reference to FIG. 22 (see FIG. 15 as needed).
Here, it is assumed that the pixel position conversion table storage means 32 stores the pixel position conversion table in advance.

As shown in FIG. 22, the parameter input unit 31 receives input of various parameters necessary for generating the element image group 80 (step S1).
The position calculating unit 331 calculates the position of each imaging camera 21 based on the maximum depth display distance, the focal length of the display element lens 13, and the inter-pixel distance by referring to the parameters input in step S1. Step S2).

The axis shift amount calculation means 333 refers to the parameters input in step S1, and refers to the inter-pixel distance, the difference distance, the focal length of the display element lens 13, the size of the image sensor 25, and the size of the display element 11. The amount of axial deviation between the image pickup lens 27 and the image pickup device 25 is calculated based on the above (step S3).

The magnification calculation means 335 refers to the parameter input in step S1 and determines the magnification that is the ratio of the distance from the depth control lens 94 to the imaging lens array 92 and the distance from the depth control lens 94 to the virtual lens array 95. Calculate (step S4).

The angle-of-view calculation unit 337 refers to the input parameters input in step S1 and determines the angle of view of each imaging camera 21 based on the maximum depth display distance, the size of the image sensor 25, and the size of the display device 11. Calculate (step S5).

The imaging camera control means 34 controls each imaging camera 21 based on the position calculated in step S2, the amount of axial deviation calculated in step S3, the magnification calculated in step S4, and the angle of view calculated in step S5. The imaging camera 21 is instructed to capture a subject image (step S6).
The subject image input means 35 accepts the input of the subject image captured by each of the image capturing cameras 21 in response to the instruction in step S6 (step S7).

The element image group generation means 36 generates an element image group 80 from the subject image input in step S7. Here, the element image group generation unit 36 refers to the pixel position conversion table and converts the pixel position of the subject image to generate the element image group 80 (step S8).

[Action/effect]
As described above, the stereoscopic image display system 1 can compress the depth of the stereoscopic image by shifting the axes of the imaging lens 27 and the imaging element 25. Accordingly, in the stereoscopic image display system 1, even when the display lens array 12 is in the delta arrangement, the stereoscopic image display device 10 can display a stereoscopic image with high resolution, and the image quality of the stereoscopic image can be improved.

Further, the stereoscopic image display system 1 does not require a high resolution camera or a high density lens array, and can be realized with a simple configuration. Further, the stereoscopic image display system 1 can provide a higher quality stereoscopic image without causing lens distortion due to the depth control lens 94.

(Second embodiment)
[Stereoscopic image display system]
Returning to FIG. 15, the differences between the stereoscopic image display system 1A according to the second embodiment and the first embodiment will be described.
In the first embodiment, the number of pixels of the image sensor 25 is a value obtained by multiplying the number of display element lenses 13 by the number of pixels of the element image 81.
On the other hand, the second embodiment is different from the first embodiment in that the number of pixels of the image sensor 25A is twice the number of the display element lenses 13.

The stereoscopic image display system 1A displays a stereoscopic image of the subject α, and includes a stereoscopic image display device 10, an imaging camera array 20A, and a stereoscopic image generation device 30A, as shown in FIG.
The imaging camera array 20A is an array of a plurality of imaging cameras 21A. The image pickup camera 21A is the same as that of the first embodiment except for the number of pixels of the image pickup element 25A, and therefore further description will be omitted.

[Configuration of stereoscopic image generation device]
The stereoscopic image generation device 30A generates an element image group 80 displayed by the stereoscopic image display device 10 from the subject image captured by the imaging camera array 20A. As shown in FIG. 15, the stereoscopic image generation device 30A includes a parameter input unit 31, a pixel position conversion table storage unit 32A, a calculation unit 33, an imaging camera control unit 34, a subject image input unit 35, and an element image. And a group generation means 36.
Note that each unit other than the pixel position conversion table storage unit 32A is the same as that in the first embodiment, and a description thereof will be omitted.

<Setting of pixel position conversion table: Square lens>
The setting of the pixel position conversion table when the display element lens 13 is a square lens will be specifically described with reference to FIG.
Here, it is assumed that the number of pixels of the element image 81 and the number of display element lenses 13 are the same as those in FIG. That is, the number of imaging cameras 21A and the number of element images 81 are the same as those in FIG.

The horizontal pixel number EIsx of the image pickup device 25A is a value (8 pixels) obtained by doubling the horizontal number Lw of the display element lens 13 as shown in the following expression (9). Further, the vertical pixel number EIsy of the image sensor 25A is a value (6 pixels) obtained by doubling the vertical number Lh of the display element lens 13.

In this case, the pixel position conversion table is set so that the pixels of the image sensor 25A are acquired at intervals of 4 pixels in the horizontal direction and at intervals of 2 pixels in the vertical direction.

Note that the setting of the pixel position conversion table itself is the same as that of FIG. 20 described in the first embodiment, so further description will be omitted.
Further, for convenience of explanation, the number of pixels of the image pickup device 25 of FIG. 20 and the number of pixels of the image pickup device 25A of FIG. 22 are the same. In reality, it is extremely rare that the element image 81 has two pixels, and the image sensor 25A in FIG. 22 has a smaller number of pixels than the image sensor 25 in FIG.

<Setting of pixel position conversion table: rectangular lens>
Hereinafter, the setting of the pixel position conversion table when the display element lens 13 is a rectangular lens will be specifically described with reference to FIG.
Here, it is assumed that the number of pixels of the element image 81 and the number of display element lenses 13 are the same as those in FIG. That is, the number of imaging cameras 21A and the number of element images 81 are the same as those in FIG.

The horizontal pixel number EIsx of the image pickup device 25A is a value (8 pixels) obtained by doubling the horizontal number Lw of the display element lens 13 as shown in the equation (9). Further, the vertical pixel number EIsy of the image sensor 25A is a value (6 pixels) obtained by doubling the vertical number Lh of the display element lens 13.

In this case, the pixel position conversion table is set so that the pixels of the image sensor 25A are acquired at intervals of 4 pixels in the horizontal direction and at intervals of 2 pixels in the vertical direction.

Note that the setting itself of the pixel position conversion table is the same as that of FIG. 21 described in the first embodiment, and therefore further description is omitted.
Further, the operation of the stereoscopic image generation device 30A is similar to that of the first embodiment, and thus the description thereof will be omitted.

[Action/effect]
As described above, since the stereoscopic image display system 1A can reduce the number of pixels of the imaging camera 21A in addition to the same effect as the first embodiment, the configuration can be simplified.

(Third Embodiment)
[Stereoscopic image display system]
Returning to FIG. 15, the differences between the stereoscopic image display system 1B according to the third embodiment and the second embodiment will be described.
In the second embodiment, the number of pixels of the image sensor 25A is twice the number of the display element lenses 13 in both the horizontal and vertical directions.
On the other hand, the third embodiment is different from the second embodiment in that the number of pixels of the image sensor 25B is twice the number of the display element lenses 13 only in the vertical direction.

The stereoscopic image display system 1B displays a stereoscopic image of the subject α, and includes a stereoscopic image display device 10, an imaging camera array 20B, and a stereoscopic image generation device 30B, as shown in FIG.
The imaging camera array 20B is an array of a plurality of imaging cameras 21B. The image pickup camera 21B is the same as that of the second embodiment except for the number of pixels of the image pickup element 25B, and therefore further description will be omitted.

[Configuration of stereoscopic image generation device]
The stereoscopic image generation device 30B generates the element image group 80 displayed by the stereoscopic image display device 10 from the subject image captured by the imaging camera array 20B. As shown in FIG. 15, the stereoscopic image generation device 30B includes a parameter input unit 31, a pixel position conversion table storage unit 32B, a calculation unit 33, an imaging camera control unit 34, a subject image input unit 35, and an element image. And a group generation means 36.
Note that each unit other than the pixel position conversion table storage unit 32B is the same as that in the second embodiment, and therefore its explanation is omitted.

<Setting of pixel position conversion table: Square lens>
The setting of the pixel position conversion table when the display element lens 13 is a square lens will be specifically described with reference to FIG.
Here, it is assumed that the number of pixels of the element image 81 and the number of display element lenses 13 are the same as those in FIG. That is, the number of imaging cameras 21B and the number of element images 81 are the same as those in FIG.

The horizontal pixel number EIsx of the image sensor 25B is the same as the horizontal number Lw (4 pixels) of the display element lens 13 as shown in the following expression (10). Further, the vertical pixel number EIsy of the image sensor 25B is a value (6 pixels) obtained by doubling the vertical number Lh of the display element lens 13.

In this case, the pixel position conversion table is set so that the pixels of the image sensor 25B are acquired at intervals of two pixels in the horizontal and vertical directions. That is, the pixel position conversion table is set so that the pixels of the image sensor 25B are acquired in a "fifth die roll" shape.
Note that the setting of the pixel position conversion table itself is the same as in the second embodiment, so further description is omitted.

<Setting of pixel position conversion table: rectangular lens>
The setting of the pixel position conversion table when the display element lens 13 is a rectangular lens will be specifically described below with reference to FIG.

As shown in FIG. 19B, the display element lenses 13 having different aspect ratios have the same aspect ratio of each pixel of the image sensor 25B as shown in FIG. 26A. Therefore, in the horizontal direction, the pixel spacing of the image sensor 25B does not have the same ratio as the lens spacing of the display element lens 13. Therefore, it is necessary to change the pixel interval of the image sensor 25B in the horizontal direction.

Specifically, when the vertical pixel size Sh of the image sensor 25B in FIG. 26A is used as a reference, the horizontal pixel size Sw of the image sensor 25B may be doubled as shown in FIG. 26B. .. Further, when the horizontal pixel size Sw of the image sensor 25B in FIG. 26A is used as a reference, the vertical pixel size Sh of the image sensor 25B may be halved as shown in FIG. 26C.
Note that the operation of the stereoscopic image generation device 30B is the same as that of the second embodiment, so description thereof will be omitted.

[Action/effect]
As described above, in the stereoscopic image display system 1B, in addition to the effects similar to the second embodiment, the number of pixels of the imaging camera 21A can be further reduced, so that the configuration can be further simplified.

(Fourth Embodiment)
[Stereoscopic image display system]
Returning to FIG. 15, the differences between the stereoscopic image display system 1C according to the fourth embodiment and the first embodiment will be described.
The fourth embodiment is different from the first embodiment in that the image pickup camera 21C includes two image pickup elements 25C arranged so as to be shifted by half a pixel in the diagonal direction.

The stereoscopic image display system 1C displays a stereoscopic image of the subject α, and includes a stereoscopic image display device 10, an imaging camera array 20C, and a stereoscopic image generation device 30C, as shown in FIG.
The imaging camera array 20C is an array of a plurality of imaging cameras 21C. The image pickup camera 21C is the same as that of the first embodiment except that the image pickup camera 21C is provided with two image pickup elements 25B, and therefore further description will be omitted. Note that by using a prism or a half mirror, the two image pickup devices 25B can be arranged so as to be shifted by half a pixel.

[Configuration of stereoscopic image generation device]
The stereoscopic image generation device 30C generates an element image group 80 displayed by the stereoscopic image display device 10 from a subject image captured by the imaging camera array 20C. As shown in FIG. 15, the stereoscopic image generation device 30C includes a parameter input unit 31, a pixel position conversion table storage unit 32C, a calculation unit 33, an imaging camera control unit 34, a subject image input unit 35, and an element image. And a group generation means 36.
Note that each unit other than the pixel position conversion table storage unit 32C is the same as that in the first embodiment, and a description thereof will be omitted.
Further, the operation of the stereoscopic image generation device 30C is similar to that of the first embodiment, and thus the description thereof is omitted.

<Setting of pixel position conversion table: Square lens>
The setting of the pixel position conversion table in the case where the display element lens 13 is a square lens will be specifically described with reference to FIG.
Here, it is assumed that the number of pixels of the element image 81 and the number of display element lenses 13 are the same as those in FIG. That is, the number of imaging cameras 21A and the number of element images 81 are the same as those in FIG.

Each imaging camera 21C is provided with two imaging devices _{25C (25C} 11 _{~25C 42).} That is, the first image pickup camera 21C includes the image pickup devices 25C ₁₁ and 25C ₁₂ , and the second image pickup camera 21C includes the image pickup devices 25C ₂₁ and 25C ₂₂ (also the third and fourth image pickup cameras 21C are included). As well).
In each of the image pickup cameras 21C, the two image pickup elements 25C are arranged such that the pixel positions thereof are diagonally displaced from each other by half a pixel. That is, the two image pickup devices 25C forming a pair are displaced by half a pixel in the horizontal and vertical directions.

The horizontal pixel number EIsx of the image pickup device 25C is 1/2 (2 pixels) of the horizontal number Lw of the display element lenses 13 per one sheet. That is, when the horizontal pixel number EIsx of the two image pickup devices 25C is summed up, it becomes equal to the horizontal number Lw of the display element lenses 13. Further, the vertical pixel number EIsy of the image pickup device 25B is the same number (3 pixels) as the vertical number Lh of the display element lens 13 per one sheet.

Here, as shown in FIG. 27A, when the vertical pixel size Sh of the image sensor 25C is used as a reference, the horizontal pixel size Sw of the image sensor 25C may be doubled. Further, as shown in FIG. 27B, when the horizontal pixel size Sw of the image sensor 25C is used as a reference, the vertical pixel size Sh of the image sensor 25C may be halved.

In this case, in the pixel position conversion table, the pixels of the two image pickup cameras 21C are arranged to alternately correspond to each other according to the positional relation of the respective image pickup cameras 21C and the positional relation of the two image pickup elements 25C forming a pair. , Pixel position conversion table is set.

27(c) and 27(d), in order to describe the correspondence between the pixels of the image sensor 25C and the element image 81, the characters A1 to D2 are added to the pixels of the image sensor 25C and the element image 81. Attached. In the first image pickup camera 21C, the characters A1 and A2 are attached to the pixels of the two image pickup elements 25C ₁₁ and 25C ₁₂ that form a pair, respectively. Further, in the second unit of the imaging camera 21C, the two pixels of the image sensor _25C 21, _{25C 22} in the pair, denoted by B1, B2 letter respectively (third car and four second imaging camera 21C The same).

Specifically, the pixels A1, B1, C1, D1 of the first image pickup device 25C ₁₁ , 25C ₂₁ , 25C ₃₁ , 25C ₄₁ and the element image 81 _OD of the odd number column correspond to each other. Further, a second sheet of the imaging device _{25C 12, 25C 22, 25C 32} , 25C 42, the pixel A2 between the even column of the element images _{81 EV,} B2, C2, D2 correspond.

First, the element images 81 _OD in odd columns will be described. In the pixel position conversion table, the pixel A1 of the first image pickup device 25C _{11 in} the upper left image pickup camera 21C is associated with the upper left pixel A1 of the element image 81 _OD . Further, in the pixel position conversion table, the pixel B1 of the first image pickup device 25C ₂₁ of the upper right image pickup camera 21C and the upper right pixel B1 of the element image 81 _OD are associated with each other. Further, the pixel position conversion table associates the pixel C1 of the first sheet of the imaging device _{25C 31} at the lower left of the image pickup camera 21C, and a lower left pixel C1 element image _{81 OD.} Further, in the pixel position conversion table, the pixel D1 of the first image sensor 25C _{41 in} the lower right imaging camera 21C and the lower right pixel D1 _{OD of} the element image 81 are associated with each other.

Next, the even-numbered element images 81 _EV will be described. The pixel position conversion table associates the pixel A2 of the imaging device _{25C 12} of the second sheet in the upper left of the imaging camera 21C, the pixel A2 in the upper left of the element image _{81 EV.} Further, the pixel position conversion table associates the pixel B2 at the top right of the image pickup camera 21C with the second sheet of the imaging device _{25C 22,} the pixel B2 at the top right of the element image _{81 EV.} Further, in the pixel position conversion table, the pixel C2 of the second image sensor 25C _{32 in} the lower left imaging camera 21C is associated with the lower left pixel C2 of the element image 81 _EV . Further, in the pixel position conversion table, the pixel D2 of the second image sensor 25C _{42 in} the lower right imaging camera 21C is associated with the lower right pixel D2 of the element image 81 _EV .

[Action/effect]
As described above, the stereoscopic image display system 1C can improve the image quality of the stereoscopic image even when the display lens array 12 is the delta array, as in the first embodiment.

(Fifth Embodiment)
With reference to FIG. 28, the difference between the stereoscopic image display system 1D according to the fifth embodiment of the present invention and the first embodiment will be described.
In the first embodiment, a real image of the imaging camera array 20 is processed, whereas in the fifth embodiment, a CG image of a virtual imaging camera array (virtual camera array) is processed. different.

The virtual camera array is a two-dimensional array of virtual imaging cameras (virtual cameras).
The virtual camera is a reference for setting the viewpoint position, the viewpoint direction, the angle of view, etc. in the three-dimensional space in which the three-dimensional model of the subject α is arranged. That is, the CG generates the element image group 80 that looks like the subject α is captured by the virtual camera. Since each virtual camera is the imaging camera 21 (FIG. 21) described in the first embodiment arranged in a three-dimensional virtual space, further description is omitted.

[Configuration of stereoscopic image generation device]
The stereoscopic image display system 1D displays a stereoscopic image of the subject α, and includes a stereoscopic image display device 10 and a stereoscopic image generation device 30D as shown in FIG. 28.
The stereoscopic image generation device 30D generates the element image group 80 displayed by the stereoscopic image display device 10 from the subject image captured by the virtual camera array.

As shown in FIG. 28, the stereoscopic image generation device 30D includes a parameter input unit 31, a pixel position conversion table storage unit 32, a calculation unit 33, a virtual camera setting unit (imaging camera control unit) 34D, and an element image group. It includes a generation unit (stereoscopic image generation unit) 36D and a three-dimensional model input unit 37.
Note that, except for the virtual camera setting unit 34D, the element image group generation unit 36D, and the three-dimensional model input unit 37, the description is omitted because it is the same as the first embodiment.

The virtual camera setting means 34D sets a virtual imaging camera (virtual camera) based on the control signal input from the calculating means 33.
Specifically, the virtual camera setting means 34D arranges each virtual camera at the position of the control signal in the virtual three-dimensional space. Further, the virtual camera setting unit 34D sets the image pickup element and the image pickup lens of each virtual camera by shifting by the amount of the axis deviation of the control signal. Further, the virtual camera setting means 34D sets the view angle of each virtual camera so as to match the view angle of the control signal.

The element image group generation means 36D generates a subject image via the virtual camera array set by the virtual camera setting means 34D, and generates an element image group 80 from the subject image.
Specifically, the element image group generation unit 36D arranges the three-dimensional model input from the three-dimensional model input unit 37 described below in the virtual three-dimensional space in which the virtual cameras are arranged. Here, the element image group generation unit 36D enlarges or reduces the size of the three-dimensional model by the magnification m so that the size of the subject does not change before and after the depth compression on the lens surface of the virtual lens array. Then, the element image group generation unit 36D generates a subject image by capturing the three-dimensional model with each virtual camera. Further, the element image group generation unit 36D generates the element image group 80 by referring to the pixel position conversion table similar to that of the first embodiment and converting the pixel position of the subject image.

The three-dimensional model input means 37 inputs a three-dimensional model representing the three-dimensional shape and color information of the subject, and outputs the input three-dimensional model to the element image group generation means 37D.
The method for generating the three-dimensional model is not particularly limited, but can be generated by performing texture mapping on polygons or the like.

[Operation of stereoscopic image generation device]
The operation of the stereoscopic image generation device 30D will be described with reference to FIG.
Here, it is assumed that the pixel position conversion table storage unit 32D stores the pixel position conversion table in advance.
Note that the processing of steps S1 to S5 is the same as that of the first embodiment, and thus the description thereof is omitted.

The virtual camera setting means 34D sets a virtual imaging camera (virtual camera) based on the control signal (step S10).
The three-dimensional model input means 37 receives input of a three-dimensional model representing the three-dimensional shape and color information of the subject (step S11).
The element image group generation unit 36D generates the subject image of the three-dimensional model input in step S11 via the virtual camera set in step S10, and generates the element image group 80 from the subject image (step S12).

[Action/effect]
As described above, the stereoscopic image generation device 30D has the same effect as that of the first embodiment even when the subject image is generated by CG. As a result, in the stereoscopic image generation device 30D, even when the display lens array 12 is in the delta arrangement, the stereoscopic image display device 10 can display a stereoscopic image with high resolution, and the image quality of the stereoscopic image can be improved.

In the present embodiment, the stereoscopic image generation device 30D has been described as being applied to the first embodiment, but the stereoscopic image generation device 30D may be applied to any of the second to fourth embodiments. it can.

(Sixth Embodiment)
[Stereoscopic image display system]
With reference to FIG. 15, a stereoscopic image display system 1E according to a sixth embodiment of the present invention will be described regarding differences from the first embodiment.
In the first embodiment, the axes of the image pickup lens 27 and the image pickup device 25 are displaced from each other (FIG. 17), whereas in the sixth embodiment, the pixel at the pixel position corresponding to the axis deviation amount dy is acquired from the subject image. Is different.

The stereoscopic image display system 1E displays a stereoscopic image of the subject α, and includes a stereoscopic image display device 10, an imaging camera array 20E, and a stereoscopic image generation device 30E, as shown in FIG.
The imaging camera array 20E is an array of a plurality of imaging cameras 21E (FIG. 30). The imaging camera 21E is the same as that of the first embodiment except that it does not have a lens shift function, and therefore further description is omitted.

[Configuration of stereoscopic image generation device]
The stereoscopic image generation device 30E is for generating an element image group 80 displayed by the stereoscopic image display device 10 from a subject image captured by the imaging camera array 20E. As shown in FIG. 15, the stereoscopic image generation device 30E includes a parameter input unit 31, a pixel position conversion table storage unit 32C, a calculation unit 33, an imaging camera control unit 34, a subject image input unit 35, and an element image. And a group generation means 36E.
Note that each unit other than the element image group generation unit 36E is the same as that in the first embodiment, and a description thereof will be omitted.

The element image group generation means 36E is for generating an element image group from the subject image input from the subject image input means 35. At this time, the element image group generation unit 36E acquires the pixel value of the pixel position corresponding to the axial deviation amount dy calculated by the axial deviation amount calculation unit 333 from each subject image.

<Pixel value acquisition method at pixel position according to the amount of misalignment>
With reference to FIG. 30, a method of acquiring the pixel value of the pixel position according to the axis deviation amount dy will be described.
First, as shown in FIG. 30A, consider a case where there is no axis deviation (axis deviation amount dy=0). In this case, the element image group generation means 36E cuts out the image area 29 ₁ of a predetermined size from the subject image 29 with the center position of the image sensor 25 as a reference. That is, the element image group generation unit 36E, the center position of the image pickup element 25 because it corresponds to the center position of the object image 29, cuts out the image area 29 ₁ relative to the center position of the subject image 29.

Note that the image area 29 _1, taking into account the amount of axial deviation dy, preset size to fit in the subject image 29. Therefore, the size (the number of pixels) of the image pickup device 25 is set larger than that of the image region 29 _{1 in} consideration of the axis deviation amount dy.

Next, as shown in FIG. 30(b), consider the case where there is an axis deviation. In this case, the element image group generation means 36E cuts out the image area 29 ₁ from the subject image 29 with reference to the position moved from the center of the image sensor 25 by the amount of axial deviation dy. That is, the element image group generation unit 36E cuts out the image region 29 ₁ with the position offset from the center of the subject image 29 as the axial shift amount dy.

In this way, the element image group generation unit 36E obtains the pixel value of the pixel position deviated from the center of the image pickup device 25 by the amount of axial deviation dy, so that the optical axis of the image pickup lens 27 and the central axis of the image pickup device 25 are The same process as shifting is performed.

After that, the element image group generation means 36E uses the cut-out image area 29 ₁ as a subject image to generate an element image group. The method itself for generating this element image group is the same as that in the first embodiment, and therefore its explanation is omitted.

[Action/effect]
As described above, the stereoscopic image generation device 30E achieves the same effect as that of the first embodiment by shifting the pixel position acquired from the subject image according to the amount of axial deviation dy. As a result, in the stereoscopic image generation device 30E, the stereoscopic image display device 10 can display a stereoscopic image with high resolution, and the image quality of the stereoscopic image can be improved.

Note that, in the present embodiment, the stereoscopic image generation device 30E has been described as being applied to the first embodiment, but the stereoscopic image generation device 30E can be applied to any of the second to fifth embodiments. it can.

(Modification)
Although the respective embodiments of the present invention have been described above in detail, the present invention is not limited to the above-described embodiments, and includes design changes and the like within a range not departing from the gist of the present invention.

In each of the above-described embodiments, one axial direction is described as a horizontal direction and the other axial direction is described as a vertical direction, but they may be reversed. That is, the positions of the adjacent display element lenses may be shifted in the vertical direction and aligned in the horizontal direction. In this case, the arrangement of each element image in the element image group is also adjusted to the arrangement of the display element lenses.

Consider a case where each pixel of the display element of the stereoscopic image display device is arranged in stripes in the horizontal direction for each color. In this case, in order to suppress color misregistration, the horizontal interval of the image pickup camera array is set to 1/3, the horizontal axis shift amount is set to 1/3, and the number of image pickup cameras in the horizontal direction is tripled. Then, an imaging camera may be assigned for each color. Then, the color shift can be suppressed by interpolating for each color from the pixel information from the adjacent cameras. For example, in each pixel of a display element of a stereoscopic image display device, an RGB stripe structure in which red, green, and blue are arranged in a stripe pattern in order, and when the value of the green pixel is used as it is, the red and blue pixels are used. You need to calculate the value. The red pixel uses the interpolated value recalculated from the first pixel value of the red pixel to be calculated as the red pixel value of the adjacent pixel on the left, and the blue pixel is calculated as the blue pixel value of the adjacent pixel on the right. The interpolated value recalculated from the first pixel value of the desired blue pixel may be used. Specifically, when the red pixel value is calculated, if the initial value of the red pixel value to be obtained is v1 and the left adjacent red pixel value is v0, the new red pixel value v1′ is , Is calculated by the following formula.
v1′=(v1×2/3)+(v0×1/3)

In some cases, the above-mentioned interpolation method may not provide a sufficient effect. Therefore, a pixel position conversion table when the number of cameras in the horizontal direction is tripled will be described. Since the pixel position conversion table described above does not consider sampling for each color, it is a conversion table corresponding to each pixel of imaging/display. For example, three pixel position conversion tables corresponding to RGB colors are required in order to match the display elements with the RGB stripe structure. Therefore, when the horizontal interval of the image pickup camera array is set to 1/3, the horizontal axis shift amount is set to 1/3, and the number of units in the horizontal direction is tripled, conversion is performed by one pixel position conversion table. The method will be described. Specifically, three times as many camera arrays in the horizontal direction are sequentially assigned as R, G, and B cameras. Here, it is assumed that the image pickup camera array assigned to the G camera has the same arrangement as the above-mentioned image pickup camera array. The data of the R and B cameras on both sides of the G camera are replaced with the R and B data of the G camera. As a result, since the image pickup camera array assigned to the G camera holds data that matches the color position of the display element having the RGB stripe structure, the pixel position conversion table is applied to each of RGB. As a result, pixel position conversion without color misregistration can be performed. According to this method, it is not necessary to prepare a pixel position conversion table corresponding to each of RGB, and the addresses of the R data area and the B data area in the data area for interpolating the image acquired by the G camera are respectively set. The data areas of the R camera and the B camera are simply referred to and replaced, and there is an advantage that high-speed processing becomes possible.

In each of the above-described embodiments, the optical element array is described as a display lens array in which optical elements such as display element lenses are two-dimensionally arranged, but the present invention is not limited to this. For example, the optical element array may be a lens array in which element lenses are one-dimensionally arranged. Further, the optical element array may be a lenticular lens array in which kamaboko-shaped element lenses (lenticular lenses) are one-dimensionally arranged. Further, the optical element array may be a pinhole array in which pinholes are arranged one-dimensionally or two-dimensionally. Furthermore, the present invention can be applied not only to the IP method but also to the parallax barrier method.

When the subject image captured by each imaging camera and each pixel of the element images forming the element image group have a one-to-one correspondence, the number of pixels of the subject image is equal to the number of element images. On the other hand, when the number of pixels of the subject image exceeds the number of element images, oversampling is performed. In this case, the ray width of the ray to be acquired is narrowed and it is possible to sample up to a high frequency. Further, since the effective diameter of the pixel becomes small, the sensitivity is lowered in the case of actual shooting, but it is not necessary to consider the sensitivity in CG. That is, by obtaining the pixel closest to the desired light ray from the oversampled subject image and converting the pixel position, it is possible to suppress a decrease in accuracy due to the pixel size.

In each of the above-described embodiments, the stereoscopic image generation device has been described as independent hardware, but the present invention is not limited to this. For example, the present invention can also be realized by a program that causes hardware resources such as a CPU, a memory, and a hard disk included in a computer to cooperate with each other as the stereoscopic image generation device described above. These programs may be distributed via a communication line, or may be distributed by being written in a recording medium such as a CD-ROM or a flash memory.

1, 1A to 1E Stereoscopic image display system 10 Stereoscopic image display device 11 Display element 12 Display lens array (optical element array)
13 Display element lens (optical element)
20, 20A to 20C, 20E Imaging camera array 21, 21A to 21E Imaging camera 22 Elevating member 23 Supports 25, 25A to 25C Imaging device 26 Pixel 27 Imaging lens 30, 30A to 30E Stereoscopic image generation device 31 Parameter input means 32, 32A 32D Pixel position conversion table storage means 33 Calculation means 331 Position calculation means 333 Axis deviation amount calculation means 335 Magnification calculation means 337 Field angle calculation means 34 Imaging camera control means 34D Virtual camera setting means (imaging camera control means)
35 subject image input means 36, 36D, 36E element image group generation means (stereoscopic image generation means)
37 3D model input means 80 Element image group 81 Element image 90 Stereoscopic image pickup device 91 Image pickup element 92 Image pickup lens array 93 Image pickup element lens 94 Depth control lens 95 Virtual lens array

Claims (11)

An image pickup camera array including an image pickup camera in which each pixel of the image pickup element is squarely arranged,
A stereoscopic image generation device that generates an element image group to be displayed by the stereoscopic image display device according to the depth of the stereoscopic image display device that includes a display element and an optical element array in which a plurality of optical elements are arranged in a delta shape, and a stereoscopic image including the stereoscopic image generation device. An image generation system,
In the image pickup camera, the number of pixels of the image pickup element is equal to or more than the number of the optical elements,
The stereoscopic image generation device,
The maximum depth display distance of the stereoscopic image display device, the focal length of the optical element, the inter-pixel distance from the central pixel of the elemental images forming the elemental image group to the calculation target pixel of each of the imaging cameras, and the optical Parameter input means for inputting a differential distance from the element array to the maximum rear depth display position, the size of the image pickup device included in the image pickup camera, and the size of the display device,
Position calculating means for calculating the position of the imaging camera based on the maximum depth display distance, the focal length of the optical element, and the inter-pixel distance;
Based on the inter-pixel distance, the difference distance, the focal length of the optical element, the size of the image pickup element, and the size of the display element, the axes of the image pickup lens and the image pickup element included in the image pickup camera. Axis deviation amount calculating means for calculating the deviation amount,
An imaging camera control unit that moves the imaging camera to the position calculated by the position calculation unit and shifts the imaging lens and the image sensor by the axial displacement amount calculated by the axial displacement amount calculation unit,
A stereoscopic image generation unit that generates an element image group in which the element images are arranged in a delta shape from a subject image captured by the imaging camera array;
A stereoscopic image generation system comprising: An image pickup camera array including an image pickup camera in which each pixel of the image pickup element is squarely arranged,
A stereoscopic image generation device that generates an element image group to be displayed by the stereoscopic image display device according to the depth of the stereoscopic image display device that includes a display element and an optical element array in which a plurality of optical elements are arranged in a delta shape, and a stereoscopic image including the stereoscopic image generation device. An image generation system,
In the image pickup camera, the number of pixels of the image pickup element is equal to or more than the number of the optical elements,
The stereoscopic image generation device,
The maximum depth display distance of the stereoscopic image display device, the focal length of the optical element, the inter-pixel distance from the central pixel of the elemental images forming the elemental image group to the calculation target pixel of each of the imaging cameras, and the optical Parameter input means for inputting a differential distance from the element array to the maximum rear depth display position, the size of the image pickup device included in the image pickup camera, and the size of the display device,
Position calculating means for calculating the position of the imaging camera based on the maximum depth display distance, the focal length of the optical element, and the inter-pixel distance;
Based on the inter-pixel distance, the difference distance, the focal length of the optical element, the size of the image pickup element, and the size of the display element, the axes of the image pickup lens and the image pickup element included in the image pickup camera. Axis deviation amount calculating means for calculating the deviation amount,
Imaging camera control means for moving the imaging camera to the position calculated by the position calculation means,
A stereoscopic image generating unit that acquires a pixel value of a pixel position corresponding to the amount of axis deviation from a subject image captured by the imaging camera array, and generates an element image group in which the element images are arranged in a delta shape;
A stereoscopic image generation system comprising: The imaging camera array includes the same number of the imaging cameras as the number of pixels of the element image,
Two image pickup cameras are arranged such that the image pickup elements having the number of pixels equal to half the number of the optical elements are shifted by half a pixel,
The stereoscopic image generation system according to claim 1 or 2, wherein the stereoscopic image generation unit alternately acquires each pixel of the two image pickup devices. The imaging camera array includes the same number of the imaging cameras as the number of pixels of the element image,
In the imaging camera, the number of pixels of the imaging element is equal to the number of the optical elements in one axial direction in which the adjacent optical elements are aligned, and the number of pixels of the imaging element in the other axial direction in which the adjacent optical elements are aligned. The number of pixels is equal to twice the number of the optical elements,
3. The stereoscopic image generating means acquires each pixel of an image captured by the imaging camera at intervals of two pixels in the one axial direction and the other axial direction. The stereoscopic image generation system described. The optical element array is a honeycomb structure in which the regular hexagonal optical elements are arranged in a delta shape,
The three-dimensional image generation means reduces or enlarges the elemental image group in accordance with the ratio of the width and the height of the optical element, according to any one of claims 1 to 4. 3D image generation system. Generated by an imaging camera array consisting of a virtual imaging camera in which each pixel of the imaging element is squarely arranged according to the depth of a stereoscopic image display device including a display element and an optical element array in which a plurality of optical elements are arranged in a delta shape. A stereoscopic image generation device for generating an element image group displayed by the stereoscopic image display device from the subject image,
The maximum depth display distance of the stereoscopic image display device, the focal length of the optical element, the inter-pixel distance from the central pixel of the elemental images forming the elemental image group to the calculation target pixel of each of the imaging cameras, and the optical Parameter input means for inputting a differential distance from the element array to the maximum rear depth display position, the size of the image pickup device included in the image pickup camera, and the size of the display device,
Position calculating means for calculating the position of the imaging camera based on the maximum depth display distance, the focal length of the optical element, and the inter-pixel distance;
Based on the inter-pixel distance, the difference distance, the focal length of the optical element, the size of the image pickup element, and the size of the display element, the axes of the image pickup lens and the image pickup element included in the image pickup camera. Axis deviation amount calculating means for calculating the deviation amount,
An image pickup camera control means for setting the image pickup camera in which the number of pixels of the image pickup element is equal to or more than the number of the optical elements, with the position calculated by the position calculation means and the axial deviation amount calculated by the axial deviation amount calculation means;
A stereoscopic image generation unit that generates the subject image via the imaging camera array, and generates, from the generated subject image, a group of element images in which the element images are arranged in a delta shape;
A stereoscopic image generation apparatus comprising: A virtual image pickup camera including a virtual image pickup camera in which each pixel of the image pickup element is squarely arranged according to the depth of a stereoscopic image display device including a display element and an optical element array in which a plurality of optical elements are arranged in a delta shape. A stereoscopic image generation device for generating an element image group displayed by the stereoscopic image display device from a subject image generated by an array,
The maximum depth display distance of the stereoscopic image display device, the focal length of the optical element, the inter-pixel distance from the central pixel of the elemental images forming the elemental image group to the calculation target pixel of each of the imaging cameras, and the optical Parameter input means for inputting a differential distance from the element array to the maximum rear depth display position, the size of the image pickup device included in the image pickup camera, and the size of the display device,
Position calculating means for calculating the position of the imaging camera based on the maximum depth display distance, the focal length of the optical element, and the inter-pixel distance;
Based on the inter-pixel distance, the difference distance, the focal length of the optical element, the size of the image pickup element, and the size of the display element, the axes of the image pickup lens and the image pickup element included in the image pickup camera. Axis deviation amount calculating means for calculating the deviation amount,
Image pickup camera control means for setting the image pickup camera in which the number of pixels of the image pickup element is equal to or more than the number of optical elements, to the position calculated by the position calculation means,
The subject image is generated via the imaging camera array, the pixel value of the pixel position corresponding to the axial shift amount is obtained from the subject image, and the element image group in which the element images are arranged in a delta shape is generated. Stereoscopic image generation means,
A stereoscopic image generation apparatus comprising: The imaging camera array includes the same number of the imaging cameras as the number of pixels of the element image,
Two image pickup cameras are arranged such that the image pickup elements having the number of pixels equal to half the number of the optical elements are shifted by half a pixel,
The three-dimensional image generation device according to claim 6 or 7, wherein the three-dimensional image generation means alternately acquires each pixel of the two image pickup devices. The imaging camera array includes the same number of the imaging cameras as the number of pixels of the element image,
In the imaging camera, the number of pixels of the imaging element is equal to the number of the optical elements in one axial direction in which the adjacent optical elements are aligned, and the number of pixels in the imaging element in the other axial direction in which the adjacent optical elements are aligned. The number of pixels is equal to twice the number of the optical elements,
8. The stereoscopic image generation means acquires each pixel of an image captured by the imaging camera at intervals of two pixels in the one axial direction and the other axial direction. The stereoscopic image generation device described. The optical element array is a honeycomb structure in which the regular hexagonal optical elements are arranged in a delta shape,
10. The stereoscopic image generating means reduces or enlarges the elemental image group according to the ratio of the width and the height of the optical element, according to any one of claims 6 to 9. 3D image generation device. A program for causing a computer to function as the stereoscopic image generation device according to any one of claims 1 to 10.

Images