JP6388435B2 - Image generating apparatus, image generating method, and program - Google Patents

Description

  The present invention relates to an image generation device, an image generation method, and a program.

As one of the methods for displaying a stereoscopic image, there is an IP (Integral Photography) system. The IP method is a method for reproducing a three-dimensional image by using a lens array in which a plurality of minute lenses (hereinafter referred to as “element lenses”) are arranged for both imaging and display.
In the generation of a display image in the IP system, a subject is imaged through a lens array configured by arranging element lenses. At this time, the camera images the focal plane of the lens array. Each element lens constituting the lens array functions in the same manner as a minute camera. As a result, an image obtained by imaging the subject through the lens array is an image in which minute images (hereinafter referred to as “element images”) corresponding to the positions of the element lenses constituting the lens array are arranged. Hereinafter, the image acquired in this way is referred to as an IP image (integral photography image). A stereoscopic image is reproduced by displaying this IP image through a lens array similar to that at the time of imaging.
Further, it is known that in the IP system, it is possible to realize an image that appears to protrude from the display surface of the display device by using a depth control lens that controls light rays from a subject.

Spyros S.M. Athineos et al., “Physical modeling of a microlens array setup for use in computer generated IP”, Proc. of SPIE-IS & T Electronic Imaging, Vol. 5664 pp. 472-479, 2005

  In recent years, a technique for generating an IP image by CG (Computer Graphics) has been studied (for example, Non-Patent Document 1). In this technique, a virtual three-dimensional space (hereinafter referred to as “virtual space”) is generated in a computer, and a subject and a set are reproduced in the virtual space so as to operate in the same manner as in the real world. Then, the computer generates an IP image by tracking light rays from the subject in the virtual space.

  In the conventional technology for generating an IP image by CG, a virtual lens array and a virtual depth control lens are arranged as objects different from the virtual camera and the subject between the virtual camera and the subject in the virtual space. . Then, the image data generation device generates the image data of the element image group by calculating the light rays incident on the virtual camera from the subject through these virtual lenses. However, in this case, the positional relationship between the virtual camera, the virtual lens array, and the virtual depth control lens may change for each frame of the video. Therefore, in the conventional technique, there are cases where the light rays passing through the virtual lens array and the virtual depth control lens have to be calculated for each frame. As described above, in the conventional technique, the amount of calculation may increase in the generation of the image data of the IP image by CG.

  The present invention has been made in view of the above points, and provides an image generation apparatus, an image generation method, and a program that can generate image data of an IP image with a small amount of calculation.

  (1) The present invention has been made to solve the above problems, and one aspect of the present invention is an image generation apparatus that generates an integral photography image of an object in a virtual space, the virtual An object attribute acquisition unit that acquires object attribute information including the position of the object in space; virtual lens array attribute information that includes at least the position and refractive characteristics of a virtual lens array that exists in the virtual space; and A virtual camera attribute storage unit that stores image sensor attribute information including at least the position of the virtual image sensor present in the optical image sensor, and a refraction method when a light beam incident on the virtual image sensor passes through the virtual lens array. A lens processing unit that calculates based on the virtual lens array attribute information read from the virtual camera attribute storage unit; Based on the object attribute information acquired by the object attribute acquisition unit, and based on the calculation result by the lens processing unit, the object at the original position of the light ray incident on the virtual imaging device is rendered and the integral And a rendering processing unit that generates image data of a photographic image.

  (2) Further, according to one aspect of the present invention, the image generation device according to (1) stores a light beam direction information indicating a light beam direction in the virtual space corresponding to a pixel in the virtual imaging device. A direction storage unit, and the lens processing unit calculates a direction of a light beam corresponding to each pixel of the virtual imaging device based on the calculated refraction method, and the direction of the pixel and the light beam Is stored in the ray direction storage unit, and the rendering processing unit stores the ray direction when the correspondence relationship between the pixel and the ray direction is already stored in the ray direction storage unit. The image data is generated based on the light direction information read from the storage unit.

  (3) Further, according to one embodiment of the present invention, the image generation device according to any one of (1) and (2) described above is opposite to the virtual imaging element with the virtual lens array interposed in the virtual space. A depth control lens attribute storage unit that stores virtual depth control lens attribute information indicating an attribute of the virtual depth control lens existing on the side of the lens, and the lens processing unit reads the depth control lens attribute storage unit from the depth control lens attribute storage unit Based on the virtual depth control lens attribute information, a refraction method is calculated when a light beam that passes through the virtual lens array and enters the virtual imaging device passes through the virtual depth control lens.

  (4) Further, according to one aspect of the present invention, in the image generation device according to (3), the virtual depth control lens attribute storage unit includes attributes of the two virtual depth control lenses constituting the 4f optical system. It memorizes.

  (5) According to one aspect of the present invention, at least the virtual lens array attribute information including at least the position and refractive characteristics of the virtual lens array existing in the virtual space, and the position of the virtual imaging element existing in the virtual space are included. An image generation method in an image generation apparatus, comprising: a virtual camera attribute storage unit that stores image sensor attribute information including an image, and generates an integral photography image of the object in the virtual space, wherein the object in the virtual space The virtual lens for reading out the object attribute information including the position of the object, and reading from the virtual camera attribute storage unit the refraction method when the light ray incident on the virtual image sensor passes through the virtual lens array Lens processing process for calculating based on array attribute information, and the object attribute Based on the object attribute information acquired in the acquisition process, and on the basis of the calculation result in the lens processing process, the object at the original position of the light ray incident on the virtual imaging device is rendered, and the integral photography And a rendering process for generating image data of the image.

  (6) According to one aspect of the present invention, at least the virtual lens array attribute information including at least the position and refractive characteristics of the virtual lens array existing in the virtual space, and the position of the virtual imaging element existing in the virtual space are included. A virtual camera attribute storage unit that stores imaging element attribute information including the image sensor attribute information, and a computer of an image generation apparatus that generates an integral photography image of the object in the virtual space includes the position of the object in the virtual space Object attribute acquisition procedure for acquiring object attribute information, based on the virtual lens array attribute information that reads from the virtual camera attribute storage unit how the light rays incident on the virtual image sensor pass through the virtual lens array. In the lens processing procedure to calculate and the object attribute acquisition procedure Based on the obtained object attribute information and on the basis of the calculation result in the lens processing procedure, the object at the original position of the light ray incident on the virtual imaging device is rendered, and the image of the integral photography image This is a program for executing a rendering processing procedure for generating data.

  According to the present invention, image data of an IP image can be generated with a small amount of calculation.

It is a conceptual diagram of the process by the image generation apparatus which concerns on the 1st Embodiment of this invention. It is a schematic diagram which shows the virtual lens array and IP image which concern on the 1st Embodiment of this invention. 1 is a block diagram illustrating a schematic functional configuration of an image generation apparatus according to a first embodiment of the present invention. It is a schematic diagram which shows the input screen of the parameter for virtual lens arrays in the image generation apparatus which concerns on the 1st Embodiment of this invention. It is a flowchart which shows the flow of the process by the image generation apparatus which concerns on the 1st Embodiment of this invention. It is a conceptual diagram of the process by the image generation apparatus which concerns on the 2nd Embodiment of this invention. It is a block diagram which shows schematic structure of the image generation apparatus which concerns on the 2nd Embodiment of this invention. It is a schematic diagram which shows the input screen of the parameter for virtual lens arrays in the image generation apparatus which concerns on the 2nd Embodiment of this invention. It is a flowchart which shows the flow of the process by the image generation apparatus which concerns on the 2nd Embodiment of this invention. It is a conceptual diagram which shows arrangement | positioning of the lens etc. in the virtual camera in the image generation apparatus which concerns on the 3rd Embodiment of this invention. It is FIG. 1 for demonstrating the calculation method of the direction of the light beam corresponding to a pixel by the image generation apparatus which concerns on each embodiment of this invention. It is FIG. 2 for demonstrating the calculation method of the direction of the light beam corresponding to a pixel by the image generation apparatus which concerns on each embodiment of this invention. It is FIG. 3 for demonstrating the calculation method of the direction of the light beam corresponding to a pixel by the image generation apparatus which concerns on each embodiment of this invention. It is a figure which shows the example of the light direction information of this invention.

Hereinafter, embodiments of the present invention will be described with reference to the drawings.
(First embodiment)
A first embodiment of the present invention will be described.
First, an outline of the present embodiment will be described with reference to FIG.
FIG. 1 is a conceptual diagram of processing by the image generation apparatus 10 (FIG. 3) according to the present embodiment.
The image generation device 10 is a device that generates an image (video) data of an IP image by rendering an object in a virtual space, and is an electronic device such as a personal computer or a server device, for example.

  The image generation device 10 includes at least the position and refractive characteristics of the object attribute acquisition unit 155 (FIG. 3) that acquires object attribute information including the position of the object 40 in the virtual space, and the virtual lens array 220 that exists in the virtual space. And a virtual camera attribute storage unit 141 (FIG. 3) that stores virtual lens array attribute information including virtual image sensor attribute information including at least the position of the virtual image sensor 210 existing in the virtual space. . Thereby, the image generation device 10 arranges the object 40 in the virtual space based on the object attribute information acquired by the object attribute acquisition unit 155. In addition, the image generation apparatus 10 includes the virtual camera 20 including the virtual imaging element 210 and the virtual lens array 220 based on the virtual lens array attribute information and the virtual imaging element attribute information stored in the virtual camera attribute storage unit 141. Place in the virtual space.

  In addition, the image generation device 10 calculates a refraction method when a light beam incident on the virtual imaging element 210 passes through the virtual lens array based on virtual lens array attribute information that is read from the virtual camera attribute storage unit 141. Based on the object attribute information acquired by the unit 152 and the object attribute acquisition unit 155, and based on the calculation result by the lens processing unit 152, the object 40 at the original position of the light ray incident on the virtual imaging device is rendered. And a rendering processing unit 154 that generates image (video) data. Thereby, the image generation device 10 calculates how the light rays are refracted in the virtual lens array 220 included in the virtual camera 20, renders the object 40, and generates image (video) data of the IP image.

  The video data is data in which frames and image data are associated with each other. In other words, the video data is time-series data of image data. The image generation device 10 may generate either image data or video data of an IP image. In the following, an aspect in which the image generation apparatus 10 generates video data of an IP image will be described. However, in order to simplify the description, generation of image data of each frame constituting video data will be described, and time series of image data will be described. A description of the conversion will be omitted.

The video signal of the IP image generated by the image generation device 10 is output to the IP stereoscopic video display device 30 which is an actual device. The IP stereoscopic video display device 30 includes a display element 310 and a lens array 320. The display element 310 of the IP stereoscopic video display device 30 displays the IP image indicated by the video signal of the IP image acquired from the image generation device 10. Light rays from the IP image displayed on the display element 310 form a three-dimensional image 50 corresponding to the object 40 through the lens array 320.
Note that the IP stereoscopic video display device 30 may be a virtual device.

Next, the virtual imaging device 210 and the virtual lens array 220 will be described with reference to FIG. 2. FIG. 2 is a schematic diagram showing the virtual lens array 220 and the IP image.
The attributes of the virtual image sensor 210 and the virtual lens array 220 can be arbitrarily set. The attributes of the virtual imaging device 210 and the virtual lens array 220 are set in accordance with, for example, the attributes of the display device 310 and the lens array 320 when the IP stereoscopic video display device 30 (FIG. 1) is arranged in the virtual space.
The attributes of the virtual image sensor 210 include the position, size, number of pixels, and the like, which are set to the same values as the position, size, number of pixels, etc. of the display element 310, respectively. In other words, the virtual imaging surface of the virtual imaging device 210 corresponds to the position of the display surface on which the IP image is displayed by the display unit 11 when the IP stereoscopic video display device 30 is arranged in the virtual space.

  As described above, the attribute of the virtual lens array 220 is set in accordance with, for example, the attribute of the lens array 320 of the IP stereoscopic video display device 30. The lens array 320 is configured, for example, by arranging N virtual element lenses having the same attribute so that their main surfaces are located on the same plane. Therefore, the virtual lens array 220 is configured by arranging N virtual element lenses 22-1 to 22-N having the same attribute so that their main surfaces are located on the same plane.

  The attributes of the virtual lens array 220 include the position of the virtual lens array 220, the size of the virtual lens array 220, the number of virtual element lenses 22-1 to 22-N, the arrangement of the virtual element lenses 22-1 to 22-N, and There are shapes of the virtual element lenses 22-1 to 22-N and the like. The position of the virtual lens array 220, the size of the virtual lens array 220, the number of virtual element lenses 22-1 to 22-N, the arrangement of the virtual element lenses 22-1 to 22-N, and the virtual element lenses 22-1 to 22- The shape of N is the position of the lens array 320, the size of the lens array 320, the number of element lenses constituting the lens array 320, the arrangement of the element lenses constituting the lens array 320, and the elements constituting the lens array 320, respectively. It is set to the same value as the shape of the lens. The positions of the principal points of the virtual element lenses 22-1 to 22-N of the virtual lens array 220 are set so that the lens array 320 of the IP stereoscopic video display device 30 is located when the IP stereoscopic video display device 30 is arranged in the virtual space. This corresponds to the position of the principal point of each component lens.

  In addition, the IP stereoscopic video display device 30 includes, for example, a display element 310 and a lens array 320 that have the same display surface or main surface size. Further, the IP stereoscopic image display device 30 displays, for example, at the distances that the normals passing through the centers of the respective display surfaces or main surfaces coincide with each other and that are separated by the focal lengths of the element lenses constituting the lens array 320. The element 310 and the lens array 320 are arranged. In this case, the image generation apparatus 10 sets the size of the imaging surface of the virtual imaging device 210 and the size of the main surface of the virtual lens array 220 in the virtual camera 20 so as to be aligned. Further, in the image generation device 10, the virtual imaging element 210 and the virtual lens array 220 are configured so that the normals passing through the centers of the respective surfaces coincide with each other, and the virtual element lens 22-1 constituting the virtual lens array 220 is used. It is set to be arranged at a distance of ˜22-N by the focal length.

Thereby, the light rays that pass through each of the virtual element lenses 22-1 to 22-N form element images 21-1 to 21-N on the imaging surface of the virtual imaging element 210, respectively. That is, these element images 21-1 to 21 -N are images in which light rays from the subject are recorded, and the number of light rays that can be recorded depends on the resolution of the virtual image sensor 210. The image generation device 10 (FIG. 3) generates IP image data indicating the images of the element images 21-1 to 22-N.
Note that the positions of the principal points of the virtual element lenses 22-1 to 22-N constituting the virtual lens array 220 and the position of the virtual imaging surface of the virtual imaging element 210 may change with the passage of time in the virtual space.

Next, the configuration of the image generation apparatus 10 will be described with reference to FIG.
FIG. 3 is a block diagram illustrating a schematic functional configuration of the image generation apparatus 10.
The image generation apparatus 10 includes a display unit 11, an input unit 12, a communication unit 13, a storage unit 14, and a control unit 15.
The display unit 11 displays a user interface for accepting input of attributes of the virtual camera 20. The display unit 11 includes a display device such as a liquid crystal display or an organic EL (Electro-Luminescence) display.
The input unit 12 receives input of attributes of the virtual camera 20 from the user. The input unit 12 includes an input device such as a mouse, a keyboard, or a touch panel, for example. The contents of operations that can be accepted by the input unit 12 are displayed by, for example, a display device included in the display unit 11.
The communication unit 13 includes a communication interface, and transmits the video signal output from the control unit 15 to the IP stereoscopic video display device 30.

  The storage unit 14 includes, for example, a ROM (Read Only Memory), a RAM (Random Access Memory), and the like. In addition, the storage unit 14 may include an HDD (Hard Disc Drive), an EEPROM (Electrically Erasable Programmable Read Only Memory), a flash memory, and the like. The storage unit 14 stores various programs to be executed by a CPU (Central Processing Unit (not shown)) included in the image generation apparatus 10, results of processing executed by the CPU, and the like.

The storage unit 14 functions as a virtual camera attribute storage unit 141, an object attribute storage unit 145, and a video signal storage unit 146.
The virtual camera attribute storage unit 141 stores virtual camera attribute information indicating attributes of the virtual camera 20 (FIG. 1).
The attribute information of the virtual camera 20 is information including virtual image sensor attribute information indicating the attributes of the virtual image sensor 210 included in the virtual camera 20 and virtual lens array attribute information indicating the attributes of the virtual lens array 220.

The virtual camera attribute storage unit 141 functions as a virtual image sensor attribute storage unit 142, a virtual lens array attribute storage unit 143, and a light direction storage unit 144.
The virtual image sensor attribute storage unit 142 stores virtual image sensor attribute information among information included in the virtual camera attribute information. The attributes of the virtual image sensor 210 represented by the virtual image sensor attribute information are, for example, the number of pixels in the horizontal and vertical directions of the virtual image sensor 210, the aspect ratio of the virtual image sensor 210, and the aspect ratio of each pixel of the virtual image sensor 210. The horizontal and vertical offsets of the virtual image sensor 210, the size of the virtual image sensor 210, and the like.

  The virtual lens array attribute storage unit 143 stores virtual lens array attribute information among the virtual camera attribute information. The attributes of the virtual lens array 220 represented by the virtual lens array attribute information are, for example, the focal lengths of the element lenses 22-1 to 22-N, the numbers of the element lenses 22-1 to 22-N in the horizontal direction and the vertical direction, and adjacent. The distance (pitch) of the element lenses 22-1 to 22-N, the arrangement of the element lenses 22-1 to 22-N, the shape of the element lenses 22-1 to 22-N, and the virtual lens array 220 and the virtual imaging device 210 Distance.

The light beam direction storage unit 144 stores light beam direction information representing the light beam direction in the virtual space corresponding to the pixels in the virtual image sensor 210.
The light direction information is light that is incident on each pixel of the virtual image sensor 210 and is information indicating the direction of the light beam in the virtual lens farthest from the virtual image sensor 210 among the virtual lenses included in the virtual camera 20. is there. In other words, the light direction information is information indicating a direction when the light viewed from each pixel of the virtual image sensor 210 is refracted through each lens included in the virtual camera 20 and exits from the virtual camera 20. . In the present embodiment, among the virtual lenses included in the virtual camera 20, the virtual lens located farthest from the virtual image sensor 210 is the virtual lens array 220. Therefore, in the present embodiment, the light direction storage unit 144 determines the direction in which the light corresponding to each pixel of the virtual image sensor 210 is emitted from the element lenses 22-1 to 22-N constituting the virtual lens array 220 for each pixel. To remember.

The object attribute storage unit 145 stores object attribute information indicating the attributes of the object 40 (FIG. 1).
The object attribute information is information that represents the attribute of the object 40 in the virtual space, and is information that represents, for example, a position, a shape, an optical characteristic, and a color. The position and shape of the object 40 represented by the object attribute information in the virtual space are represented by, for example, a polygon (polygon) or a free-form surface. Moreover, the data format may be arbitrary, for example, it is expressed as CAD (Computer Assisted Drafting) data.

Note that attributes such as the position, shape, optical characteristics, and color of the object 40 in the virtual space may change over time in the virtual space. That is, the object attribute information may be time-series information that represents the attribute of the object 40.
The video signal storage unit 146 stores IP image data in each frame constituting the video.

Some or all of the functions of the control unit 15 are realized by, for example, a CPU (not shown) included in the image generation apparatus 10 executing a program stored in the storage unit 14.
The control unit 15 functions as a display processing unit 151, a lens processing unit 152, a rendering processing unit 154, and a video signal output unit 156.
The display processing unit 151 causes the display unit 11 to display a user interface for receiving input of attributes of the virtual camera 20.

  The lens processing unit 152 causes the virtual lens array attribute storage unit 143 to store information about the attributes of the virtual camera 20 received from the user via the input unit 12. Specifically, the lens processing unit 152 sets the virtual image sensor attribute information indicating the attribute of the virtual image sensor 210 based on the input about the attribute of the virtual image sensor 210 received from the user by the input unit 12. The data is stored in the storage unit 142. Further, the lens processing unit 152 receives virtual lens array attribute information indicating the attribute of the virtual lens array 220 based on the input about the attribute of the virtual lens array 220 received from the user by the input unit 12, and the virtual lens array attribute storage unit 143. Remember me.

In addition, the lens processing unit 152 calculates how the light rays incident on the virtual image sensor 210 travel in the virtual space. At this time, the lens processing unit 152 functions as the virtual lens array processing unit 153, for example.
The virtual lens array processing unit 153 reads from the virtual lens array attribute storage unit 143 of the virtual camera attribute storage unit 141 the refraction method when the light rays incident on the virtual image sensor 210 pass through the virtual lens array 220. Calculate based on attribute information. The virtual lens array processing unit 153 calculates the direction of light rays corresponding to each pixel of the virtual image sensor 210 based on the calculated refraction in the virtual lens array 220, and The correspondence relationship with the direction of the light beam is stored in the light beam direction storage unit 144.

The rendering processing unit 154 functions as an object attribute acquisition unit 155 that acquires object attribute information. In the present embodiment, the object attribute acquisition unit 155 acquires the object attribute information by reading it from the object attribute storage unit 145.
The rendering processing unit 154 reads the light direction information from the light direction storage unit 144 and, based on the read light direction information and the object attribute information, the object 40 at the original position of the light incident on the virtual imaging device 210. Render to generate IP image data. In the present embodiment, the rendering processing unit 154 renders the object 40 based on a ray tracing (ray tracing) method.
In addition, about a ray tracing method, since a well-known technique is applicable, the detail is abbreviate | omitted. In the present embodiment, for the sake of brevity, it is assumed that the light beam is a principal light beam, but the image generation apparatus 10 may also calculate a sub-light beam.

  The video signal output unit 156 sequentially reads the IP image data in each frame constituting the video from the video signal storage unit 146, and sequentially outputs the video signal indicating the read IP image data via the communication unit 13 to the IP stereoscopic video. It transmits to the display device 30.

Next, a user interface for inputting attributes of the virtual camera 20 displayed on the display unit 11 will be described with reference to FIG.
FIG. 4 is a schematic diagram illustrating an input screen for parameters for the virtual lens array 220 in the image generation apparatus 10.
In the example shown in FIG. 4, information of six items “1” to “6” can be set in the display area 111 of the display unit 11 as attributes of the virtual lens array 220 (FIG. 1).

  The item “1” is an item for receiving and setting the input of the focal lengths of the element lenses 22-1 to 22-N constituting the virtual lens array 220. The focal lengths of the element lenses 22-1 to 22-N are attributes that affect the emission angle and the incident angle of light rays, and affect the viewing area and resolution of the stereoscopic image 50 displayed by the IP stereoscopic video display device 30. The item “2” is an item for accepting and setting the number of element lenses 22-1 to 22-N constituting the virtual lens array 220. In the example shown in FIG. 4, the number of lenses in the horizontal direction and the number of lenses in the vertical direction can be specified for the element lenses 22-1 to 22-N.

  The item “3” is an item for receiving and setting the input of the intervals of the element lenses 22-1 to 22-N constituting the virtual lens array 220. The item “4” is an item for accepting and setting the designation of the arrangement of the element lenses 22-1 to 22-N constituting the virtual lens array 220. In the example shown in FIG. 4, in the item “4”, “delta arrangement” and “square arrangement” can be specified as the arrangement of the virtual lens array 220.

  Item “5” is an item for adjusting the positional relationship between the virtual lens array 220 and the virtual imaging device 210. Specifically, the item “5” is an item for enabling the distance between the virtual lens array 220 and the virtual imaging element 210 to be set to an arbitrary distance, for example. When the distance between the virtual lens array 220 and the virtual imaging device 210 is set shorter than the focal length of the element lenses 22-1 to 22-N, the stereoscopic image 50 displayed by the IP stereoscopic video display device 30 is a virtual lens. The image on the display element 310 side becomes clearer with respect to the array 220. When the distance between the virtual lens array 220 and the virtual image sensor 210 is set longer than the focal length of the element lenses 22-1 to 22-N, the stereoscopic image 50 displayed by the IP stereoscopic video display device 30 is: The image on the side opposite to the display element 310 with respect to the virtual lens array 220 becomes clearer. Further, the depth position at which the image becomes sharpest varies depending on the distance between the virtual lens array 200 and the image sensor 210.

The item “6” is an item for receiving and setting an input regarding the shapes of the element images 21-1 to 21 -N. Since the shape of the element images 21-1 to 21-N largely depends on the shape of the element lenses 22-1 to 22-N, the item “6” is an input regarding the shape of the element lenses 22-1 to 22-N. It is also an item for accepting and setting. In the example shown in FIG. 4, the item “6” designates any one of “circle”, “hexagon”, “square”, “rectangle”, etc. as the shape of the element images 21-1 to 21 -N. It shows possible.
The items “1” to “6” described above are basically set in association with the attributes of the IP stereoscopic video display device 30, but set to values different from the attributes of the IP stereoscopic video display device 30. May be.

Next, the operation of the image generation apparatus 10 will be described with reference to FIG.
FIG. 5 is a flowchart showing the flow of processing by the control unit 15 (FIG. 3) of the image generation apparatus 10.
(Step S101) First, the rendering processing unit 154 (FIG. 3) of the control unit 15 determines whether or not the light direction information is stored in the light direction storage unit 144. When the light direction information is stored in the light direction storage unit 144 (step S101; YES), the control unit 15 advances the process to step S107. When the light direction information is not stored in the light direction storage unit 144 (step S101; NO), the control unit 15 advances the processing to step S102.
(Step S <b> 102) Next, the lens processing unit 152 (FIG. 3) of the control unit 15 executes the processes of steps S <b> 103 to S <b> 106 for each pixel of the virtual image sensor 210. The lens processing unit 152 repeats the processes in steps S103 to S106 until there are no unprocessed pixels. Thereafter, the lens processing unit 152 ends the process shown in FIG.

(Step S <b> 103) Next, the lens processing unit 152 specifies an element lens corresponding to the pixel to be calculated among the element lenses 22-1 to 22-N constituting the virtual lens array 220. The element lens corresponding to the pixel to be calculated is an element lens that transmits a ray traced from the pixel to be calculated (hereinafter referred to as “pixel-corresponding ray”). For example, among the element lenses 22-1 to 22-N, the lens processing unit 152 identifies the element lens having the closest distance between the principal point and the calculation target pixel as the element lens corresponding to the calculation target pixel. Thereafter, the lens processing unit 152 proceeds to step S104.
(Step S <b> 104) Next, the lens processing unit 152 refers to the virtual lens array attribute information stored in the virtual lens array attribute storage unit 143 (FIG. 3), and starts from the center coordinates of the main surface of the virtual lens array 220. The position of the element lens corresponding to the pixel to be calculated is calculated. Thereafter, the lens processing unit 152 proceeds to step S105.

(Step S105) Next, the lens processing unit 152 calculates the direction and transmission position of the pixel-corresponding light beam when transmitting the element lens from the positional relationship between the center of the element lens and the pixel to be calculated. Thereafter, the lens processing unit 152 proceeds to step S106.
(Step S106) Next, the lens processing unit 152 associates the identification information of the pixel to be calculated with the direction and transmission position of the pixel-corresponding light ray calculated in Step S105, and uses the light ray direction information indicating this correspondence as the light ray. It memorize | stores in the direction memory | storage part 144 (FIG. 3).
(Step S107) Next, the rendering processing unit 154 reads the ray direction information read from the ray direction storage unit 144 and the object attribute information acquired from the object attribute storage unit 145, and renders the object 40 by the ray tracing method. Image data of an IP image is generated. And the control part 15 complete | finishes the process shown by FIG.

Note that the direction of the pixel-corresponding light beam is represented by two angles, for example. These two angles are, for example, orthogonal triaxial coordinates formed by two axes X and Y parallel to the virtual imaging surface of the virtual imaging device 210 and orthogonal to each other, and an axis Z perpendicular to the virtual imaging surface of the virtual imaging device 210. When the system is used as a reference, it is represented by an angle formed by the pixel-corresponding light beam in the ZX direction and an angle formed in the ZY direction.
Note that the transmission position of the pixel-corresponding light beam is represented by coordinates in the above-described orthogonal triaxial coordinate system, for example.

As described above, the image generation apparatus 10 according to the present embodiment is disposed in the three-dimensional virtual space, and is virtually imaged by the virtual camera 20 including the virtual imaging element 210 and the virtual lens array 220. The image data of the IP image is generated.
Thereby, even if the positional relationship between the virtual camera 20 and the object 40 changes, the image generation device 10 does not change the positional relationship between the virtual imaging element 210 and the virtual lens array 220 in the virtual camera 20. It is not necessary to newly calculate how the light rays are refracted inside the virtual camera 20. For example, even when rendering a plurality of frames, the image generation apparatus 10 only needs to perform the calculation once. Therefore, the image generation apparatus 10 can generate image data of an IP image with a small calculation amount.

(Second Embodiment)
A second embodiment of the present invention will be described.
About the same structure as embodiment mentioned above, the same code | symbol is attached | subjected and description is used.
First, an outline of the present embodiment will be described with reference to FIG.
FIG. 6 is a conceptual diagram of processing by the image generation apparatus 10a (FIG. 7) according to the present embodiment.
The image generation apparatus 10a is an apparatus that renders an object in the virtual space and generates image data of an IP image in the same manner as the image generation apparatus 10 (FIG. 3) according to the first embodiment. However, the image generation apparatus 10 generates image data of an IP image virtually captured by the virtual camera 20 including the virtual imaging element 210 and the virtual lens array 220, whereas the image generation apparatus 10a includes a virtual depth control lens. 230, the virtual lens array 220, and the virtual camera 20a including the virtual image sensor 210 generate image data of an IP image virtually captured.

  Note that the depth control lens is a lens arranged on the opposite side of the image sensor with the lens array interposed in capturing an IP image. Normally, it is not possible to take an image by placing a subject on the imaging surface side of the lens array, and therefore it is not possible to acquire information on a stereoscopic image protruding from the display surface of the IP stereoscopic video display device. However, since the position of the optical image of the subject can be shifted to the vicinity of the imaging surface by using the depth control lens, the same light beam as that assumed when the subject is arranged and imaged on the imaging surface side of the lens array. Space can be obtained equivalently. Thus, the depth control lens can adjust the depth position of the reproduced image. The above-described functions are the same in the virtual space, and the image generation apparatus 10a generates IP image data of an IP image that is virtually captured by the virtual camera 20a including the virtual depth control lens 230. The depth position of the stereoscopic image 50 displayed by the display device 30 can be adjusted.

Next, the configuration of the image generation device 10a will be described with reference to FIG.
FIG. 7 is a block diagram illustrating a schematic functional configuration of the image generation apparatus 10a.
The image generation device 10a includes a storage unit 14a instead of the storage unit 14 (FIG. 3) in the image generation device 10 (FIG. 3). The image generation device 10a includes a control unit 15a instead of the control unit 15 (FIG. 3) in the image generation device 10.

The storage unit 14a functions as a virtual camera attribute storage unit 141a instead of the virtual camera attribute storage unit 141 (FIG. 3) in the storage unit 14. Similar to the virtual camera attribute storage unit 141, the virtual camera attribute storage unit 141 a functions as a virtual imaging element attribute storage unit 142, a virtual lens array attribute storage unit 143, and a light ray direction storage unit 144, and a virtual depth control It functions as the lens attribute storage unit 147.
The virtual depth control lens attribute storage unit 147 stores virtual depth control lens attribute information representing the attributes of the virtual depth control lens 230 (FIG. 6). The attributes of the virtual depth control lens 230 represented by the virtual depth control lens attribute information are, for example, the diameter of the virtual depth control lens 230, the focal lengths of the virtual depth control lens 230 on the image sensor side and the subject side, and the virtual depth control lens 230. The position of the principal point, the number of virtual depth control lenses 230, and the like.

The control unit 15a functions as a lens processing unit 152a instead of the lens processing unit 152 (FIG. 3) in the control unit 15.
In addition to functioning as a virtual lens array processing unit 153a instead of the virtual lens array processing unit 153 (FIG. 3), the lens processing unit 152a functions as a virtual depth control lens processing unit 157.

  Similar to the virtual lens array processing unit 153 (FIG. 3), the virtual lens array processing unit 153a indicates how the light incident on the virtual imaging device 210 is refracted when passing through the virtual lens array 220, as a virtual camera attribute storage unit. Calculation is performed based on virtual lens array attribute information read from the virtual lens array attribute storage unit 143 of 141a. The virtual lens array processing unit 153 a outputs information indicating how the light rays are refracted in the calculated virtual lens array 220 to the virtual depth control lens processing unit 157.

  The virtual depth control lens processing unit 157 reads, from the virtual depth control lens attribute storage unit 147 of the virtual camera attribute storage unit 141a, how the light incident on the virtual image sensor 210 is refracted when passing through the virtual depth control lens 230. Calculation is performed based on the virtual depth control lens attribute information. The virtual depth control lens processing unit 157 performs virtual processing based on the calculated method of refraction of light rays in the virtual depth control lens 230 and the method of refraction of light rays in the virtual lens array 220 acquired from the virtual lens array processing unit 153a. The direction of the light beam corresponding to each pixel of the image sensor 210 is calculated, and the correspondence relationship between each pixel of the virtual image sensor 210 and the direction of the light beam is stored in the light beam direction storage unit 144.

Next, a user interface for inputting attributes of the virtual camera 20a (FIG. 6) displayed by the display unit 11 will be described with reference to FIG.
FIG. 8 is a schematic diagram illustrating an input screen for parameters for the virtual depth control lens 230 in the image generation apparatus 10a.
In the example shown in FIG. 8, in the display area 112 of the display unit 11, information of three items “1” to “3” can be set as attributes of the virtual depth control lens 230 (FIG. 6). .
The item “1” is an item for receiving and setting the input of the focal length of the virtual depth control lens 230. The item “2” is an item for receiving and setting an input of the size of the virtual depth control lens 230. The item “3” is an item for receiving and setting the input of the distance between the virtual depth control lens 230 and the virtual lens array 220.
When the virtual camera 20a includes a plurality of virtual depth control lenses 230, the image generation device 10a can set the above-described items “1” to “3” for each of the plurality of virtual depth control lenses 230. May be shown.

Next, the operation of the image generation device 10a (FIG. 7) will be described with reference to FIG.
FIG. 9 is a flowchart showing the flow of processing by the control unit 15 (FIG. 7) of the image generation apparatus 10a.
Of the processes in steps S201 to S209 shown in FIG. 9, the processes in steps S201, S203 to S205, S208, and S209 are the same as steps S101, S103 to S105, S106, and S107 shown in FIG. , Use the explanation.
(Step S202) The lens processing unit 152a (FIG. 7) of the control unit 15a executes the processing of Steps S203 to S208 for all the pixels of the virtual imaging device. The lens processing unit 152a repeats the processes in steps S203 to S208 until there are no unprocessed pixels. Thereafter, the control unit 15a advances the process to step S209.

(Step S206) The lens processing unit 152a executes the process of step S207 for each virtual depth control lens included in the virtual camera 20a. The lens processing unit 152a repeats the process of step S207 until there is no unprocessed virtual depth control lens. Thereafter, the control unit 15a advances the process to step S208.
(Step S207) Next, the lens processing unit 152a determines the positional relationship between the center position of the virtual depth control lens and the position of the pixel-corresponding light beam when the previous virtual lens is transmitted, and when the lens light is transmitted before the pixel-corresponding light beam. From the direction of, the direction and transmission position of the light beam corresponding to the pixel at the time of transmission through the virtual depth control lens are calculated.
(Step S208) Next, the lens processing unit 152a, like the lens processing unit 152 in step S106 (FIG. 5), identifies the pixel to be calculated, and the direction and transmission position of the pixel-corresponding ray calculated in step S206. And the light direction information indicating the correspondence is stored in the light direction storage unit 144 (FIG. 7).

As described above, the image generation apparatus 10a according to the present embodiment is virtually arranged by the virtual camera 20 that is arranged in the three-dimensional virtual space and includes the virtual imaging device 210, the virtual lens array 220, and the virtual depth control lens 230. The image data of the IP image of the object 40 to be imaged is generated.
Thereby, similarly to the image generation device 10, even when the positional relationship between the virtual camera 20 a and the object 40 changes, the image generation device 10 a and the virtual imaging element 210 and the virtual lens array 220 in the virtual camera 20 a. Therefore, it is not necessary to recalculate how the light rays are refracted inside the virtual camera 20. Therefore, the image generation apparatus 10a can generate image data of an IP image with a small calculation amount. In addition, the image generation device 10a calculates a refraction method in the virtual depth control lens 230, and thus reproduces the stereoscopic image 50 that protrudes in front of the display screen when displayed on the IP stereoscopic video display device 30. The image data of the IP image can be generated.

(Third embodiment)
A third embodiment of the present invention will be described.
In the present embodiment, a mode in which specific settings are used in the image generation apparatus 10a according to the second embodiment will be described. Therefore, in the following, the same reference numerals as those of the second embodiment are given to the components of the present embodiment, and the description is incorporated.

  In the IP method, in order to prevent inversion of depth information during imaging and display, each element lens constituting the lens array at the time of imaging is made a concave lens, and the function of the GRIN lens having a wavelength of 3/4 period Or a reversal process is performed on an element image acquired by each element lens of the lens array. The lens processing unit 152a (FIG. 7) of the image generation apparatus 10a according to the present embodiment includes the principal points of the element lenses 22-1 to 22-N of the virtual lens array 220 included in the virtual camera 20a and each of the virtual imaging elements 210. With respect to the light rays connecting the pixels, the respective light rays are principal points of the element lenses 22-1 to 22-N in a direction rotated 180 degrees with respect to the optical axis passing through the principal points of the element lenses 22-1 to 22-N. The direction after passing through is calculated. By this calculation, the image generation device 10a can obtain a calculation result that is optically equivalent to the concave lens and the GRIN lens having a 3/4 period in the direction of the light beam. In some cases, the depth of the stereoscopic image 50 can be prevented from being reversed.

In addition, the virtual depth control lens attribute storage unit 147 of the image generation device 10a according to the present embodiment is a virtual depth control lens that represents the attributes of the two virtual depth control lenses 230-1 and 230-2 constituting the 4f optical system. Store attribute information.
The 4f optical system will be described with reference to FIG.
In the example shown in FIG. 10, virtual imaging of the virtual imaging device 210 is performed by matching the optical axes of the two virtual depth control lenses 230-1 and 230-2 having the same attribute and the virtual lens array 220. It is arranged parallel to the surface. The distance between the virtual image sensor 210 and the virtual lens array 220 is the focal length f of the element lens of the virtual lens array 220. The distance between the two virtual depth control lenses 230-1 and 230-2 is twice the focal length f1 (2 × f1). In other words, the two virtual depth control lenses 230-1 and 230-2 are arranged with their focal planes coincided with each other in the plane A1. Further, the distance between the virtual depth control lens 230-1 close to the virtual image sensor 210 and the virtual lens array 220 is the focal length f1. Since the depth position is not compressed in the 4f optical system, the virtual depth control lens 230 is located between the focal plane A2 on the right side of the virtual depth control lens 230-1 and the focal plane A3 on the left side of the virtual depth control lens 230-2. -1, 230-2, an optically equivalent light beam can be acquired. In the example shown in FIG. 10, since the virtual lens array 220 is disposed on the focal plane A2 on the right side of the virtual depth control lens 230-1, the virtual lens array 220 is disposed on the focal plane A3. Is optically equivalent.

  Accordingly, in the virtual camera 20a having the 4f optical system shown in FIG. 10, when the object 40 (FIG. 6) is arranged outside the virtual camera 20a (FIG. 6), the virtual camera is based on the focal plane A3 (FIG. 10). The part of the object 40 close to 20a protrudes from the display surface of the IP stereoscopic video display device 30 when displayed on the IP stereoscopic video display device 30 (FIG. 6). In addition, the portion of the object 40 far from the virtual camera 20a with respect to the focal plane A3 is deeper than the display surface of the IP stereoscopic video display device 30 when displayed on the IP stereoscopic video display device 30.

  The display processing unit 151 of the image generation device 10a according to the present embodiment displays an auxiliary line or surface representing the focal plane A2 in the virtual space when receiving the input of the position of the object 40 in the virtual space from the user. It has the function to do. Thereby, the user can easily grasp the depth position of the object 40 with respect to the virtual lens array 220.

Next, a specific example of a method of calculating the direction of the pixel-corresponding light beam will be described with reference to FIGS.
FIGS. 11 to 13 are diagrams for explaining a method of calculating the direction of the light beam corresponding to the pixels by the image generation apparatuses 10 and 10a, respectively. 11 to 13 show an example of pixel-corresponding light rays in the virtual 4f optical system shown in FIG.

  Hereinafter, as an example, calculation in the XZ direction of the pixel-corresponding light beam will be described, but the calculation can be similarly performed in the YZ direction. The XZ direction is represented by an angle formed by the pixel-corresponding light beam and the Z axis in the XZ plane. 11 to 13, the coordinate axes in the virtual space are the same as the coordinate axes shown in FIG. 10. In the example shown in FIGS. 10 to 13, the element lenses of the virtual lens array 220 are lenses having a radius Lx1 and a focal length f, and are arranged at a pitch interval D. In the example shown in FIGS. 10 to 13, the virtual depth control lenses 230-1 and 230-2 are lenses having a radius DLx and a focal length f1.

First, with reference to FIG. 11, the calculation of the direction of the pixel-corresponding light beam that passes through the virtual lens array 220 (FIG. 10) will be described.
FIG. 11 shows an example of a pixel-corresponding light beam that passes through the virtual lens array 220 shown in FIG.
In the example shown in FIG. 11, the element lens 22-1 of the virtual lens array 220 corresponds to the pixel 211-1 of the virtual imaging device 210. Accordingly, the pixel-corresponding light beam 61 of the pixel 211-1 travels from the pixel 211-1 to the principal point 221-1 of the element lens 22-1. The pixel-corresponding light beam 61 of the pixel 211-1 travels straight without being refracted by the element lens 22-1. Accordingly, in the virtual lens array 220, the coordinates (tx2, tz2) and the direction (tdx2) of the transmission position of the pixel-corresponding ray of the pixel 211-1 are the position (tx1, tz1) of the pixel 211-1, and the element lens 22-1. Based on the position (Lx1, f) of the principal point 221-1, the following expressions (1) to (3) are used.

The lens processing unit 152a calculates the transmission position and direction of the pixel-corresponding light beam on the main surface of the virtual lens array 220 based on the above formulas (1) to (3). The lens processing unit 152a stores the light direction information indicating the calculation result in the light direction storage unit 144.
The lens processing unit 152a acquires the position (tx1, tz1) of the pixel 211-1 based on the virtual imaging element attribute information stored in the virtual imaging element attribute storage unit 142. In addition, the lens processing unit 152 a acquires the position (Lx1, f) of the principal point 221-1 of the element lens 22-1 based on the virtual lens array attribute information stored in the virtual lens array attribute storage unit 143.

  10 to 13, the pitch of the virtual lens array 220 is D. Accordingly, the lens processing unit 152a determines the pixel-corresponding ray that passes through the element lens 22-2 adjacent to the element lens 22-1 based on the position (tx1 + D, tz1) of the principal point of the element lens 22-2. The transmission position and direction are calculated. As described above, the lens processing unit 152a calculates the transmission position and direction of the pixel-corresponding light beam on the main surface of the virtual lens array 220 for the pixel-corresponding light beam of each pixel of the virtual imaging device 210, and the light beam direction indicating the calculation result Information is stored in the beam direction storage unit 144.

Next, with reference to FIG. 12, the calculation of the direction of the pixel-corresponding light beam that passes through the virtual depth control lens 230-1 (FIG. 10) will be described.
FIG. 12 shows an example of a pixel-corresponding light beam that passes through the virtual depth control lens 230-1 shown in FIG.
The pixel-corresponding light beam 61 of the pixel 211-1 (FIG. 11) emitted from the element lens 22-1 of the virtual lens array 220 enters the virtual depth control lens 230-1. At this time, the pixel-corresponding light beam 61 of the pixel 211-1 intersects the main surface of the virtual depth control lens 230-1 at a point 232-1. The pixel-corresponding light beam 61 of the pixel 211-1 is refracted by the virtual depth control lens 230-1 and proceeds in the direction of the point 71. A point 71 is an intersection of the focal plane A1 on the left side of the virtual depth control lens 230-1 and the straight line 62. The straight line 62 is a straight line that is parallel to the pixel-corresponding light beam 61 incident on the virtual depth control lens 230-1 and that passes through the principal point 231-1 of the virtual depth control lens 230-1. Accordingly, the coordinates (tx3, tz3) and the direction (tdx3) of the transmission position of the pixel-corresponding light beam 61 of the pixel 211-1 on the main surface of the virtual depth control lens 230-1 are the mains of the virtual lens array 220 of the pixel-corresponding light beam 61. Based on the transmission position (tx2, tz2) and direction (tdx2) on the surface and the position (DLx, tz2 + f1) of the principal point of the virtual depth control lens 230, the following expressions (4) to (6) are used. .

  The lens processing unit 152a calculates the transmission position and direction of the pixel-corresponding light beam on the main surface of the virtual depth control lens 230-1 based on the above equations (4) to (6). The lens processing unit 152a stores the light direction information indicating the calculation result in the light direction storage unit 144. At this time, the lens processing unit 152a uses the light direction information about the transmission position and direction of the pixel-corresponding light beam on the main surface of the virtual lens array 220 as the transmission position of the pixel-corresponding light beam on the main surface of the virtual depth control lens 230-1. You may overwrite with the light direction information about the direction.

Next, with reference to FIG. 13, the calculation of the direction of the pixel-corresponding light beam that passes through the virtual depth control lens 230-2 (FIG. 10) will be described.
FIG. 13 shows an example of a pixel-corresponding light beam that passes through the virtual depth control lens 230-2 shown in FIG.
The pixel-corresponding light beam 61 of the pixel 211-1 (FIG. 11) emitted from the virtual depth control lens 230-1 is incident on the virtual depth control lens 230-2. At this time, the pixel-corresponding light beam 61 of the pixel 211-1 intersects with the main surface of the virtual depth control lens 230-2 at a point 232-2. Then, the pixel corresponding light beam 61 of the pixel 211-1 is refracted by the virtual depth control lens 230-2 and proceeds in the direction of the point 72. A point 72 is an intersection of the focal plane A3 on the left side of the virtual depth control lens 230-2 and the straight line 63. The straight line 63 is a straight line that is parallel to the pixel-corresponding light beam 61 incident on the virtual depth control lens 230-2 and that passes through the principal point 231-2 of the virtual depth control lens 230-2. Accordingly, the coordinates (tx4, tz4) and the direction (tdx4) of the transmission position of the pixel-corresponding light beam 61 of the pixel 211-1 in the virtual depth control lens 230-2 are the main of the virtual depth control lens 230-1 of the pixel-corresponding light beam 61. Based on the transmission position (tx3, tz3) and direction (tdx3) on the surface and the position (DLx, tz3 + 2f1) of the principal point of the virtual depth control lens 230-2, the following expressions (7) to (9) are used. Is done.

The lens processing unit 152a calculates the transmission position and direction of the pixel-corresponding light beam on the main surface of the virtual depth control lens 230-2 based on the above equations (7) to (9). The lens processing unit 152a stores the light direction information indicating the calculation result in the light direction storage unit 144. At this time, the lens processing unit 152a obtains the light direction information about the transmission position and direction of the pixel-corresponding light beam on the main surface of the virtual depth control lens 230-1 and the pixel-corresponding light beam on the main surface of the virtual depth control lens 230-2. You may overwrite with the light direction information about the transmission position and direction.
Note that the calculation result of the transmission position of the pixel-corresponding light beam in each lens (virtual lens array 220, virtual depth control lens 230-1, virtual depth control lens 230-2) is outside each lens and does not pass through the lens. In this case, the rendering processing unit 154a processes the pixel corresponding to the pixel-corresponding ray as having no output.

FIG. 14 shows an example of ray direction information corresponding to each lens (virtual lens array 220, virtual depth control lens 230-1, virtual depth control lens 230-2) in the virtual 4f optical system shown in FIG. FIG.
In the example shown in FIG. 14, the light direction information includes pixel identification information (pixel ID), input X coordinate information (input X coordinate), input Y coordinate information (input Y coordinate), and input Z coordinate information (input Z coordinate). , Input ZX angle information (input ZX angle), input ZY angle information (input ZY angle), output X coordinate information (output X coordinate), output Y coordinate information (output Y coordinate), output Z coordinate information (output Z coordinate) , Output ZX angle information (output ZX angle), and output ZY angle information (output ZY angle). Among these, the input X coordinate information, the input Y coordinate information, the input Z coordinate information, the input ZX angle information, and the input ZY angle information are input information used for calculating the refraction method in each lens, and the output X coordinate. The information, the output Y coordinate information, the output Z coordinate information, the output ZX angle information, and the output ZY angle information are output information indicating calculation results of how to refraction in each lens based on the input information. Note that the input information may not be stored in the light direction storage unit 144.

The pixel identification information is identification information for identifying each pixel included in the virtual image sensor 210 (FIG. 10).
The input X coordinate information is information representing the X coordinate of the position of the emission source of the pixel corresponding light beam corresponding to the pixel indicated by the pixel identification information.
The input Y coordinate information is information representing the Y coordinate of the position of the emission source of the pixel corresponding light beam corresponding to the pixel indicated by the pixel identification information.
The input Z coordinate information is information indicating the Z coordinate of the position of the emission source of the pixel corresponding light beam corresponding to the pixel indicated by the pixel identification information.
The input ZX angle information is information representing an angle formed by the pixel-corresponding light beam with respect to the Z axis in the ZX plane at the position where the pixel-corresponding light beam corresponding to the pixel indicated by the pixel identification information is emitted.
The input ZY angle information is information indicating an angle formed by the pixel-corresponding light beam with respect to the Z axis in the ZY plane at the position where the pixel-corresponding light beam corresponding to the pixel indicated by the pixel identification information is emitted. In the present embodiment, when the position of the emission source is the virtual image sensor 210, the input ZX angle information and the input ZY angle information are NULL (−).

The output X coordinate information is information representing the X coordinate of the transmission position in the lens for the pixel corresponding light beam corresponding to the pixel indicated by the pixel identification information.
The output Y coordinate information is information representing the Y coordinate of the transmission position in the lens for the pixel corresponding light beam corresponding to the pixel indicated by the pixel identification information.
The output Z coordinate information is information representing the Z coordinate of the transmission position in the lens for the pixel corresponding light beam corresponding to the pixel indicated by the pixel identification information.
The output ZX angle information is information representing an angle formed by the pixel-corresponding light beam corresponding to the pixel indicated by the pixel identification information with respect to the Z axis in the ZX plane when emitted from the lens.
The output ZY angle information is information representing an angle formed by the pixel-corresponding light beam corresponding to the pixel indicated by the pixel identification information with respect to the Z axis in the ZY plane when emitted from the lens.

14A, 14B, and 14C show the virtual lens array 220 (FIG. 10), the virtual depth control lens 230-1 (FIG. 10), and the virtual depth control lens 230-2 (FIG. 10), respectively. The light direction information which shows the permeation | transmission position and direction of the light beam corresponding to a pixel in a main surface is shown.
The output information of the light beam direction information shown in FIG. 14A indicates the calculation result for the light beam corresponding to the pixel on the main surface of the virtual lens array 220 described with reference to FIG. Since the pixel-corresponding light ray that enters the virtual lens array 220 is emitted from the virtual imaging device 210, the values of the input X coordinate information, the input Y coordinate information, and the input Z coordinate information are the values of the coordinates of each pixel. . Further, the values of the input ZX angle information and the input ZY angle information are NULL (−) for any pixel.

The output information of the light direction information shown in FIG. 14B indicates the calculation result for the pixel-corresponding light beam on the main surface of the virtual depth control lens 230-1 described with reference to FIG. Since the pixel-corresponding light beam incident on the virtual depth control lens 230-1 exits from the virtual lens array 220, the input information corresponds to the output information shown in FIG.
The output information of the light direction information shown in FIG. 14C indicates the calculation result for the pixel-corresponding light beam on the main surface of the virtual depth control lens 230-2 described with reference to FIG. Since the pixel-corresponding light ray that enters the virtual depth control lens 230-2 exits from the virtual lens array 220, the input information corresponds to the output information shown in FIG.

  In the case of a moving image, the camera moves or changes the focal length so that the subject moves or deforms. In the case of a camera capable of taking an IP image, the depth position of the reproduced image may be changed. When this processing is performed by the image generation device 10a, the virtual camera 20a changes the position of the virtual depth control lens 230, but stores the calculation results for the pixel-corresponding light rays from the virtual imaging element 210 to the virtual lens array 220. As a result, the calculation of the refraction of the light beam corresponding to the pixel in the virtual lens array 220 can be omitted, and the change of the depth position of the reproduced image can be dealt with only the calculation for the position change of the virtual depth control lens 230. In other words, the image generation apparatus 10a stores light ray direction information for each lens of the virtual camera 20a in the light ray direction storage unit 144, so that even when the position of the lens in the virtual camera 20a is changed, the calculation is small. The image data of the IP image can be generated by the amount.

  Further, for example, when the focal length can be changed in conjunction with the lens array of both imaging and display, as can be realized by a liquid crystal lens array using a liquid crystal lens for both imaging and display, the virtual imaging device If the correspondence between each pixel 210 and the element lenses 22-1 to 22-N of the virtual lens array 220 is recorded, it is only necessary to calculate the difference in the light beam direction due to the difference in focal length. In other words, the image generation devices 10 and 10a perform less calculation by storing the identification information of each pixel of the virtual imaging element 210 and the identification information of the element lens in the virtual lens array attribute storage unit 143, for example. The image data of the IP image can be generated by the amount.

  In each of the above-described embodiments, the arrangement of the element lenses 22-1 to 22-N of the virtual lens array 220 stored in the virtual lens array attribute storage unit 143 may be a delta arrangement or a square arrangement. The type of the virtual lens array 220 stored in the virtual lens array attribute storage unit 143 is not limited to a lens array in which element lenses are arrayed two-dimensionally, but may be a lens array in which vertically long lenticular lenses are arrayed in the horizontal direction. Good.

  In each of the above-described embodiments, the number of virtual lenses that the image generation apparatuses 10 and 10a include in the virtual cameras 20 and 20a may be arbitrary. For example, the image generation apparatus 10a may include a plurality of virtual depth control lenses 230 in the virtual camera 20a.

In each embodiment described above, the number of objects 40 arranged in the virtual space may be plural. The object 40 may be, for example, a virtual lens.
In the above-described embodiments, the various attributes of the illustrated virtual cameras 20 and 20a are merely examples, and may include other attributes.
In each embodiment described above, the object attribute acquisition unit 155 acquires the object attribute information by reading the object attribute information from the object attribute storage unit 145. However, the object attribute information by the object attribute acquisition unit 155 is different from the object attribute information. The route may be acquired. For example, the object attribute acquisition unit 155 may acquire object attribute information from an external device that can communicate with the image generation apparatuses 10 and 10a via communication.

  In each of the above-described embodiments, the light direction information stored in the light direction storage unit 144 is the direction of the pixel-corresponding light beam in the virtual lens farthest from the virtual image sensor 210 among the virtual lenses included in the virtual cameras 20 and 20a. However, the ray direction information may represent the direction of the pixel-corresponding ray in another virtual lens included in the virtual cameras 20 and 20a. For example, in the image generation device 10a, the light direction information may represent the direction of the pixel corresponding light beam in the virtual lens array 220. However, the amount of calculation of how light is refracted in the virtual cameras 20 and 20a when the light direction information indicating the direction of transmission through the virtual lens far from the virtual image sensor 210 is stored in the light direction storage unit 144. Is desirable because it reduces

  In addition, you may make it implement | achieve a part of image generation apparatuses 10 and 10a in embodiment mentioned above, for example, the control parts 15 and 15a, etc. with a computer. In that case, a program for realizing this function may be recorded on a computer-readable recording medium, and the program recorded on this recording medium may be read into a computer system and executed. Here, the “computer system” is a computer system built in the image generation apparatuses 10 and 10a and includes hardware such as an OS (Operating System) and peripheral devices.

  The “computer-readable recording medium” refers to a storage device such as a flexible medium, a magneto-optical disk, a portable medium such as a ROM and a CD-ROM, and a hard disk incorporated in a computer system. Furthermore, the “computer-readable recording medium” is a medium that dynamically holds a program for a short time, such as a communication line when transmitting a program via a network such as the Internet or a communication line such as a telephone line, In this case, a volatile memory inside a computer system that serves as a server or a client may be included that holds a program for a certain period of time. The program may be a program for realizing a part of the functions described above, or may be a program capable of realizing the functions described above in combination with a program already recorded in a computer system.

  Moreover, you may implement | achieve part or all of the image generation apparatuses 10 and 10a in embodiment mentioned above as integrated circuits, such as LSI (Large Scale Integration). Each functional unit of the image generation apparatuses 10 and 10a may be individually made into a processor, or a part or all of them may be integrated into a processor. Further, the method of circuit integration is not limited to LSI, and may be realized by a dedicated circuit or a general-purpose processor. In addition, when an integrated circuit technology that replaces LSI appears due to the advancement of semiconductor technology, an integrated circuit based on the technology may be used.

  As described above, the embodiment of the present invention has been described in detail with reference to the drawings. However, the specific configuration is not limited to the above, and various design changes and the like can be made without departing from the scope of the present invention. It is possible to

  DESCRIPTION OF SYMBOLS 10, 10a ... Image generation apparatus, 11 ... Display part, 12 ... Input part, 13 ... Communication part, 14, 14a ... Memory | storage part, 141 ... Virtual camera attribute memory | storage part, 142 ... Virtual imaging element attribute memory | storage part, 143 ... Virtual Lens array attribute storage unit, 144 ... ray direction storage unit, 145 ... object attribute storage unit, 146 ... video signal storage unit, 147 ... virtual depth control lens attribute storage unit, 15, 15a ... control unit, 151 ... display processing unit, 152, 152a ... Lens processing unit, 153 ... Virtual lens array processing unit, 154 ... Rendering processing unit, 155 ... Object attribute acquisition unit, 156 ... Video signal output unit, 157 ... Virtual depth control lens processing unit, 20, 20a ... Virtual Camera 210 ... Virtual imaging device, 211-1 to 21-N Element image, 220 Virtual lens array, 22-1 to 22-N Element element 'S, 30 ... IP stereoscopic image display apparatus, 310 ... display device, 320 ... lens array, 40 ... object, 50 ... three-dimensional image

Claims (6)

An image generation device that generates an integral photography image of an object in a virtual space,
An object attribute acquisition unit that acquires object attribute information including the position of the object in the virtual space;
A virtual for storing virtual lens array attribute information including at least a position and refractive characteristics of a virtual lens array existing in the virtual space, and virtual image sensor attribute information including at least a position of a virtual image sensor existing in the virtual space. A camera attribute storage unit;
A lens processing unit that calculates, based on the virtual lens array attribute information, reading from the virtual camera attribute storage unit, how to refract the light incident on the virtual imaging element when passing through the virtual lens array;
Based on the object attribute information acquired by the object attribute acquisition unit, and based on a calculation result by the lens processing unit, the object at the original position of the light ray incident on the virtual imaging device is rendered and the integral A rendering processing unit for generating image data of a photography image;
An image generation apparatus comprising: A ray direction storage unit that stores ray direction information representing the direction of rays in the virtual space corresponding to the pixels in the virtual imaging device;
Further comprising
The lens processing unit calculates a direction of a light beam corresponding to each pixel of the virtual imaging device based on the calculated refraction method, and determines a correspondence relationship between the pixel and the direction of the light beam as the light beam direction. Memorize it in the memory,
When the correspondence between the pixel and the direction of the light beam is already stored in the light beam direction storage unit, the rendering processing unit is configured to store the image based on the light beam direction information read from the light beam direction storage unit. Generate data,
The image generating apparatus according to claim 1. A virtual depth control lens attribute storage unit that stores virtual depth control lens attribute information indicating an attribute of a virtual depth control lens existing on the opposite side of the virtual imaging element across the virtual lens array in the virtual space;
Further comprising
Based on the virtual depth control lens attribute information read from the virtual depth control lens attribute storage unit, the lens processing unit transmits a light beam that passes through the virtual lens array and enters the virtual imaging device to the virtual depth control lens. Calculate how to refract when transmitting,
The image generation apparatus according to claim 1, wherein the image generation apparatus is an image generation apparatus. The virtual depth control lens attribute storage unit stores attributes of two virtual depth control lenses constituting the 4f optical system.
The image generating apparatus according to claim 3. Virtual camera attribute for storing virtual lens array attribute information including at least the position and refractive characteristics of the virtual lens array existing in the virtual space, and image sensor attribute information including at least the position of the virtual image sensor existing in the virtual space An image generation method in an image generation apparatus that includes a storage unit and generates an integral photography image of an object in the virtual space,
An object attribute acquisition process for acquiring object attribute information including the position of the object in the virtual space;
A lens processing step of calculating, based on the virtual lens array attribute information, reading from the virtual camera attribute storage unit, how to refract light when the light beam incident on the virtual image sensor passes through the virtual lens array;
Based on the object attribute information acquired in the object attribute acquisition process and based on the calculation result in the lens processing process, the object at the original position of the light ray incident on the virtual image sensor is rendered and the integrator is rendered. A rendering process for generating image data of a photo photographic image;
An image generation method comprising: Virtual camera attribute for storing virtual lens array attribute information including at least the position and refractive characteristics of the virtual lens array existing in the virtual space, and image sensor attribute information including at least the position of the virtual image sensor existing in the virtual space A computer of an image generation apparatus that includes a storage unit and generates an integral photography image of an object in the virtual space.
An object attribute acquisition procedure for acquiring object attribute information including the position of the object in the virtual space;
A lens processing procedure for calculating, based on the virtual lens array attribute information, reading from the virtual camera attribute storage unit, how to refract light rays incident on the virtual image sensor through the virtual lens array;
Based on the object attribute information acquired in the object attribute acquisition procedure, and on the basis of the calculation result in the lens processing procedure, the object at the original position of the light ray incident on the virtual imaging device is rendered and the integrator is rendered. Rendering procedure to generate image data
A program for running

Images