JP2018074463A - Image generation device, image display system and program - Google Patents

Abstract

PROBLEM TO BE SOLVED: To efficiently enlarge a vision in which a stereoscopic image is visible.SOLUTION: Virtual camera image acquisition means acquires a virtual camera image at each of multiple view points. Observation position acquisition means acquires an observation position. Pixel value determination means calculates a displacement amount from a center point of a display part to an intersection of a straight line that passes the observation position and a center point of a lens plate of an image generation device, with the display part of the image generation device, identifies an element lens corresponding to a display pixel that is a pixel of a display image, combines a value that is obtained by multiplying a pixel value of a pixel of the virtual camera image at the same position as the display element by a weight coefficient based on a position of the display pixel and a position of the virtual camera image between the virtual camera images and calculates a pixel value of a display pixel at a position where a displacement amount from the display pixel becomes the calculated displacement amount.SELECTED DRAWING: Figure 2

Description

  The present invention relates to an image generation device, an image display system, and a program.

  The integral type stereoscopic display device is a kind of stereoscopic display device that allows an observer to visually recognize a natural stereoscopic image without using dedicated glasses. The stereoscopic display device has a mechanism for selectively presenting light rays of an image according to the positions of the left and right eyes of the observer within a predetermined viewing zone in front of the display surface. This mechanism reproduces the difference in images (binocular parallax) presented to the viewer's left and right eyes and changes in appearance according to the viewing position (motion parallax). Therefore, the viewer can perceive a stereoscopic image of the displayed subject.

  The integral type stereoscopic display device is a stereoscopic display device having an optical mechanism including a lens. This mechanism realizes selective presentation of light at the viewing position. More specifically, the integral type stereoscopic display device includes a lens plate and a display panel. The lens plate is formed by arranging a number of fine convex lenses in a two-dimensional array. The main surface of the lens plate is arranged in parallel with the display surface of the display panel at a position that is separated by a distance corresponding to the focal length of the convex lens constituting the lens plate. The display panel is formed by arranging finer pixels in a two-dimensional plane than the convex lens constituting the lens plate. The display panel is a liquid crystal display, for example. Accordingly, a plurality of pixels are assigned to each one convex lens. The display panel is arranged behind the viewer rather than the lens plate. A viewer located in front of the lens plate selectively observes light rays emitted from one pixel from a group of pixels assigned to each one convex lens constituting the lens plate.

  An image displayed on a display panel disposed behind the lens plate relative to the viewer is referred to as a display image. In addition, each convex lens constituting the lens plate is referred to as an element lens, and an image presented for each one element lens is referred to as an element image. One element image consists of a plurality of pixels. That is, the display image is an element image group including a plurality of element images. The display image is configured to include information on images observed at a plurality of viewpoints within a predetermined viewing area on the front surface of the stereoscopic display device. Therefore, the viewer can recognize the stereoscopic image by observing the display image displayed on the display panel through the lens plate.

However, when the viewer's position is out of the predetermined viewing area, the stereoscopic image may be disturbed. One reason is that at that position, a pixel observed through an element lens is not a pixel in the element image facing the element lens but a pixel in an element image facing an element lens different from the element lens. Is considered. In the example shown in FIG. 4, the pixel on the position _Pn observed through the element lens El at the viewing position Va3 faces the element lens adjacent above the element lens El. This is because the pixels facing the different element lenses are pixels that should be observed from different viewing positions. For example, if the position of the viewer is to the left of the left end of a predetermined viewing area, the pixels that form part of the image observed at that position are originally observed near the right end of the viewing area It may be a pixel constituting an image. A part of this image is visually recognized as image disturbance.

  In order to enlarge the viewing zone, the position of the viewer is measured, and the positional relationship between the lens plate and the display is mechanically moved by the servo mechanism according to the measured position, and the refractive index distribution of the element lens is changed. Have been proposed (see Patent Documents 1-3). Any of these methods is merely a method for reducing the disturbance of the observed image, and the disturbance of the image observed outside the predetermined viewing zone cannot be basically eliminated. For example, an image close to a stereoscopic image viewed by front view may be visually recognized from a viewing position obliquely with respect to the main surface of the stereoscopic display device that is outside the viewing area. Therefore, in order to truly enlarge the viewing zone, it is desirable not only to eliminate image disturbance but also to visually recognize a stereoscopic image that should be viewed at the viewpoint in the enlarged viewing zone.

  In addition to the integral method, the three-dimensional display method includes methods such as a lenticular method and a parallax barrier method. Both the lenticular method and the parallax barrier method are three-dimensional display methods that do not require special glasses. Each of them represents a common subject image, and a display image is displayed on the display panel, in which two images having parallax between each other are alternately arranged in the horizontal direction into element images obtained by cutting the strips in the vertical direction. It is a method to make it. However, the lenticular three-dimensional display device has a lens plate in which the longitudinal direction of a plurality of long and semi-cylindrical convex lenses is oriented in the vertical direction and the convex lenses are arranged in the horizontal direction at the same cycle as the element image. Do it. The parallax barrier type stereoscopic display device is formed by arranging slits in which elongated openings are arranged in the vertical direction in the horizontal direction at the same cycle as the element image. Therefore, in the lenticular type and parallax barrier type stereoscopic display device, the viewing zone can be enlarged by controlling the relative positional relationship between the lens plate or slit and the display. Even with these methods, the image viewed from the viewpoint outside the predetermined viewing area is not disturbed, but is different from the image that should be visually recognized.

Japanese Patent No. 7-38926 JP 2005-175993 A JP 2014-112757 A Japanese Patent No. 5522794

Athineos, Spyros S., et al. “Physical modeling of a microlens array setup for use in computer generated IP”, Proceedings of the SPIE, Vol. 5664, pp. 472-479, 2005 Huy Hoang Tran, et al., “Interactive 3D Navigation System for Image-guided Surgery”, International Journal of Virtual Reality, 8 (1), pp. 9-16, 2009 Nakajima, et al., “High-speed image creation method for 3D display using the principle of Integral Photography”, Journal of the Institute of Image Media Sciences, Vol. 54, No.3, pp. 420-425, 2000 Takafumi Koike, “Rendering Integral Photography Images Using Programmable Graphics Hardware”, Information Processing Society of Japan Research Report, 2003-CG-113, pp. 70-74, 2003

  In order to view the original image from a viewpoint outside the predetermined viewing area, it is conceivable to display a display image corresponding to the viewing position on the flat display. A display image to be displayed on the flat display can be generated by synthesizing images of subjects acquired at a plurality of viewpoints. However, according to the conventional display image generation method, for example, the methods described in Non-Patent Documents 1 to 4, a large calculation cost is required. This makes it difficult to follow the real time to the viewing position that can change from time to time.

  The present invention has been made in view of the above points, and an object thereof is to provide an image generation device, an image display system, and a program capable of efficiently expanding a viewing area where a stereoscopic image is visually recognized.

The present invention has been made to solve the above-described problems. [1] In one aspect of the present invention, the distance from a lens plate in which a display unit for displaying an image is formed by arranging a plurality of element lenses is as described above. An image generation device that generates a display image to be displayed on an image display device that is disposed facing a position that is a focal length of an element lens, and a virtual camera image acquisition unit that acquires a virtual camera image of each of a plurality of viewpoints Viewing position acquisition means for acquiring a viewing position; and a displacement amount from the center point of the display unit to the intersection of the viewing position and a straight line passing through the center point of the lens plate and the display unit is calculated. The element lens corresponding to the display pixel that is the pixel of the display image is specified, and the pixel value of the pixel of the virtual camera image at the same position as the display pixel is set to the position of the display pixel and the position of the virtual camera image. Weighter based And a pixel value determining unit that calculates a pixel value of a display pixel at a position where the displacement amount is calculated by combining a value obtained by multiplying the virtual camera images between the virtual camera images. It is a generation device.
According to the configuration of [1], the pixel value of the pixel of the virtual camera image at the same position as the display pixel constituting the display image, the position of the display image based on the position of the element lens corresponding to the display pixel, and the virtual The pixel value of the display pixel at the position where the displacement amount from the original display pixel is the calculated displacement amount is calculated using the weighting coefficient determined by the viewpoint of the camera image. Therefore, as an image in the region corresponding to each element lens in the virtual camera image, an element image is generated by synthesizing the image of the region displaced from the viewing position toward the center point of the lens plate between the virtual cameras. . Even if the movement amount of the viewing position becomes larger than the diameter of each element lens, each element image is presented through the corresponding element lens at the viewing position. Therefore, the viewing area where the intended stereoscopic image is viewed is enlarged. Further, since it is possible to directly generate a display image without generating an intermediate image based on a virtual camera image, it is possible to improve memory utilization efficiency. Since the pixel value of the display pixel is calculated independently between the display pixels, the generation of the display image is speeded up by processing these operations in parallel.

[2] One aspect of the present invention is the above-described image generation device, wherein the camera parameter recording unit sets the positions of the plurality of viewpoints so that the positions of the center points of the plurality of viewpoints are the viewing positions. The image generating apparatus according to [1] including:
According to the configuration of [2], a plurality of virtual camera images observed from viewpoints distributed around the viewing position are used for generating a display image. Therefore, the entire viewpoint for acquiring the virtual camera image follows the viewing position. Therefore, a plurality of virtual camera images acquired for each viewpoint are effectively used for generating a display image, and the quality of a stereoscopic image visually recognized for the displayed display image is improved.

[3] One aspect of the present invention is the above-described image generation apparatus, which includes the viewing position measurement unit, the image display apparatus, and the image generation apparatus according to [1] or [2]. In the image display system, the viewing position acquisition unit acquires the viewing position measured by the viewing position measurement unit.
According to the configuration of [3], the display image is generated using the virtual camera image displaced by the displacement amount calculated based on the viewing position measured by the viewing position measurement unit. The image display device displays the generated display image. Therefore, the viewer can visually recognize a stereoscopic image corresponding to the viewing position for the display image displayed on the image display device.

[4] According to one aspect of the present invention, an image display in which a display unit that displays an image is arranged to face a position where a distance from a lens plate in which a plurality of element lenses are arranged is a focal length of the element lens An image generation apparatus that generates a display image to be displayed on the apparatus, the virtual camera image acquisition means for acquiring virtual camera images for each of a plurality of viewpoints, the viewing position acquisition means for acquiring a viewing position, and the viewing Calculate the amount of displacement from the center point of the display unit up to the intersection of the position and the straight line passing through the center point of the lens plate and the display unit, and identify the element lens corresponding to the display pixel that is the pixel of the display image Then, a value obtained by multiplying the pixel value of the pixel of the virtual camera image at the same position as the display pixel by a weighting factor based on the position of the display pixel and the position of the virtual camera image is synthesized between the virtual camera images. Before As an image generating apparatus and a pixel value determining means for calculating a pixel value of the display pixel position serving as the displacement amount of the displacement amount of the display pixel is calculated, a program for causing the computer.
According to the configuration of [4], the pixel value of the pixel of the virtual camera image at the same position as the display pixel constituting the display image, the position of the display image based on the position of the element lens corresponding to the display pixel, and the virtual The pixel value of the display pixel at the position where the displacement amount from the original display pixel is the calculated displacement amount is calculated using the weighting coefficient determined by the viewpoint of the camera image. Therefore, as an image in the region corresponding to each element lens in the virtual camera image, an element image is generated by synthesizing the image of the region displaced from the viewing position toward the center point of the lens plate between the virtual cameras. . Even if the movement amount of the viewing position becomes larger than the diameter of each element lens, each element image is presented through the corresponding element lens at the viewing position. Therefore, the viewing area where the intended stereoscopic image is viewed is enlarged. Further, since it is possible to directly generate a display image without generating an intermediate image based on a virtual camera image, it is possible to improve memory utilization efficiency. Since the pixel value of the display pixel is calculated independently between the display pixels, the generation of the display image is speeded up by processing these operations in parallel.

  According to the present invention, it is possible to efficiently expand the viewing zone in which a stereoscopic image is observed.

It is a block diagram which shows the structural example of the image display system which concerns on embodiment of this invention. It is a block diagram which shows the structural example of the image generation apparatus which concerns on embodiment of this invention. It is a figure which shows the example of arrangement | positioning of the virtual camera which concerns on embodiment of this invention. It is a figure which shows an example of the positional relationship of a display image and a viewpoint. It is a figure which shows the other example of the positional relationship of a display image and a viewpoint. It is a figure which shows an example of a display image. It is an enlarged view which shows a part of display image. It is a figure which shows the example of arrangement | positioning of an element lens. It is explanatory drawing which shows an example of the identification method of the element lens nearest to a display pixel. It is a figure which shows the example of a virtual camera image. It is a flowchart which shows an example of the image generation process which concerns on embodiment of this invention. It is a flowchart which shows an example of the pixel value determination process which concerns on embodiment of this invention.

Hereinafter, embodiments of the present invention will be described with reference to the drawings.
FIG. 1 is a diagram illustrating a configuration example of an image display system 1 according to the present embodiment.
The image display system 1 includes an image generation device 10, a viewing position measurement unit 20, and an image display device 30.

  The image generation apparatus 10 acquires a viewing position signal indicating the viewing position Vp and virtual camera image data indicating a virtual camera image observed at each of a plurality of viewpoints from the viewing position measurement unit 20. The viewing position Vp is a position where one viewer views an image displayed on the image display device 30. The image generation device 10 displaces the virtual camera image at each viewpoint based on the viewing position Vp, and generates a display image by combining the displaced virtual camera images between the virtual cameras. The display image indicates an image displayed on the image display device 30. The display image is also called an integral image. The image generation device 10 outputs display image data indicating the generated display image to the image display device 30 and causes the image display device 30 to display the display image.

The viewing position measurement unit 20 measures the viewing position Vp in the real space. When the image display device 30 is movable (for example, a tablet-type image display device), the position of the center point O _d of the image display device 30 is also measured. This is because the positional relationship between the viewing position measurement unit 20 and the image display device 30 is generally not constant. When the viewing position measurement unit 20 is configured integrally with the image display device 30, the position of the center point _Od of the image display device 30 may not be measured. The viewing position measuring unit 20 may be a stationary type or a part thereof may be a portable type. Any method may be used for measuring the viewing position V _p and the position of the center point O _d of the image display device 30. Examples of the measuring method include an optical method and a magnetic method. When the optical measurement method is used, the viewing position measurement unit 20 includes, for example, an image sensor (image sensor). The imaging element captures an image represented by visible light or infrared light from the subject. Viewing position measurement unit 20 measures the midpoint of the center point or the left and right eyes of the viewer's head from the captured image as a viewing position V _p. The viewing position measurement unit 20 refers to image data indicating a depth map obtained by photographing, and extracts the contour of the image based on a difference in depth value between coordinates close to each other. The viewing position measuring unit 20 collates the extracted contour with a predetermined human body model and performs coordinate conversion on the two-dimensional coordinates and the depth value on the image of the center point of the viewer's head, and is thus captured. The position represented by the coordinate value is specified as the viewing position Vp. The human body model is data indicating the position of the center point of each part constituting the human body. The human body region includes the upper arm, chest, lower limbs, etc. in addition to the head. Even if the viewing position measurement unit 20 performs image recognition processing on the image data obtained by shooting, the two-dimensional coordinate value and the depth value on the image of the left eye and right eye of the viewer and the respective center points are specified. Good. Then, the viewing position measurement unit 20 calculates the average value of the two-dimensional coordinate value and the depth value of each of the identified left eye and right eye, and calculates the calculated two-dimensional coordinate value and the average value of the depth value in the imaging space. identifying the position represented by the coordinate values as a viewing position V _p.

When the magnetic measurement method is used, the viewing position measurement unit 20 includes, for example, a magnetic field generator and a magnetic sensor whose positions are fixed. The magnetic sensor is fixed to a fixture that can be worn on the viewer's head, and detects the magnitude and direction of the magnetic field propagated from the magnetic field generator. Viewing position measuring unit 20 identifies the position of the head of the viewer corresponding to the magnitude and direction of the magnetic field by the magnetic sensor detects a viewing position V _p.
Viewing position measurement unit 20 outputs the viewing position signal indicating the specified viewing position V _p to the image generating apparatus 10.
Above, has been mainly described forceps position V _p, the position of the center point O _d of the image display device 30 can also be measured by viewing the position V _p and similar techniques. When measuring the position of the center point O _d of the image display device 30, the viewing position measurement unit 20 sends a display position signal indicating the measured position of the center point O _d of the image display device 30 to the image generation device 10. Output.

  The image display device 30 is an integral type stereoscopic display device. The image display device 30 includes a display panel 31 (FIG. 5) and a lens plate 32 (FIG. 5). The display panel 31 and the lens plate 32 are arranged such that their main surfaces face each other in parallel. The main surfaces of the display panel 31 and the lens plate 32 are two-dimensional planes and have a rectangular shape. On the main surface of the display panel 31, a plurality of pixels are arranged at equal intervals in the horizontal direction and the vertical direction. The numbers of pixels in the horizontal direction and the vertical direction of the display panel 31 are 7680 and 4320, respectively. The area where the pixels are arranged corresponds to the display area of the display image. Therefore, the display panel 31 functions as a display unit that displays a display image. On the main surface of the lens plate 32, a plurality of element lenses are arranged in a predetermined format. The arrangement format of the element lenses is, for example, a delta arrangement. The plurality of element lenses are convex lenses having the same shape and size. The focal length f of the element lens corresponds to the distance between the display panel 31 and the lens plate 32. The viewer can visually recognize the stereoscopic image of the subject by observing the display image displayed on the display panel 31 through the lens plate 32. The image display device 30 may be realized as, for example, a table top type or tablet type integral type stereoscopic display device. The size of the table top type or tablet type integral type stereoscopic display device is suitable when the number of viewers viewing a display image at one time is one. The size of a typical table top type or tablet type integral type stereoscopic display device is, for example, in a range from 100 mm long × 150 mm wide × 20 mm deep to 400 mm × 560 mm wide × 100 mm deep.

(Configuration example of image generation apparatus)
Next, a configuration example of the image generation apparatus 10 according to the present embodiment will be described.
FIG. 2 is a block diagram illustrating a configuration example of the image generation apparatus 10 according to the present embodiment.
The image generation device 10 generates a display image to be displayed on the image display device 30. The image generation apparatus 10 includes a camera parameter recording unit 110, an input unit 120, a camera parameter / virtual camera image association unit 130, a pixel value determination unit 140, and an output unit 150.

  The camera parameter recording unit 110 acquires camera parameters related to each of the plurality of virtual cameras from the outside of the image generation apparatus 10 and records the acquired camera parameters. The virtual camera means an image acquisition unit that is virtually installed in a three-dimensional space. The camera parameter is a parameter related to image acquisition by the virtual camera. The camera parameter includes the position of the optical center of the virtual camera. This position corresponds to the position of the viewpoint, and may be expressed as a relative position with a predetermined position as a reference. The predetermined position as a reference is, for example, the center of gravity of the distribution area of the plurality of virtual cameras.

  Note that the camera parameter recording unit 110 may calculate the coordinates of the center point of each viewpoint of the plurality of virtual cameras, and determine each viewpoint so that the calculated center point coordinates are equal to the coordinates of the viewing position. Good. The viewing position is notified to the camera parameter recording unit 110 by a viewing position signal input from a viewing position acquisition unit 121 (described later) of the input unit 120. In setting the viewpoint, the camera parameter recording unit 110 may keep the relative positional relationship between the viewpoints unchanged. The set viewpoints are arranged in a two-dimensional plane perpendicular to the viewing position and the straight line passing through the center point of the lens plate 32 of the image display device 30, and the relative viewpoints of the viewpoints with respect to the center point of the viewpoint. The coordinates may be kept constant without changing before and after the movement.

  Various data are input to the input unit 120 from the outside of the image generation apparatus 10. The input unit 120 includes a viewing position acquisition unit 121, a virtual camera image acquisition unit 122, a stereoscopic display parameter input unit 123, and a display image adjustment parameter input unit 124.

A viewing position signal is input from the viewing position measurement unit 20 to the viewing position acquisition unit 121. The viewing position acquisition unit 121 outputs the input viewing position signal to the camera parameter recording unit 110 and the pixel value determination unit 140.
The viewing position acquisition unit 121 may receive a display position signal from the viewing position measurement unit 20. In that case, the viewing position acquisition unit 121 outputs the input display position signal to the pixel value determination unit 140.

  The virtual camera image acquisition unit 122 acquires image data indicating an image of a common subject observed from a viewpoint where each virtual camera is installed. The acquired image data may be image data indicating an image of a subject actually taken at that viewpoint, or may be image data indicating an image of a subject that is synthesized using a known CG technique and can be observed at that viewpoint. Good. Further, when the camera parameter recording unit 110 moves each viewpoint, the virtual camera image acquisition unit 122 may acquire image data indicating an image of a subject observed at each set viewpoint.

  In the following description, an image acquired for each virtual camera is referred to as a virtual camera image, and data indicating the virtual camera image is referred to as virtual camera image data. Virtual camera image data is data indicating a pixel value for each pixel. The pixel value may be either a luminance value indicating brightness for each pixel or a color space value indicating color. As the color space value, for example, an RGB value representing a color in the RGB color system can be used. The virtual camera image acquisition unit outputs the virtual camera image data acquired for each virtual camera to the camera parameter / virtual camera image association unit 130.

The stereoscopic display parameter input unit 123 acquires a stereoscopic display parameter. The stereoscopic display parameter is a parameter related to display of a display image by the image display device 30 which is a stereoscopic display device. The stereoscopic display parameters include, for example, parameters of the lens plate 32 that constitutes the image display device 30 and parameters of the display panel 31. The parameters of the lens plate 32 include the diameter, focal length, arrangement interval (pitch) of the individual element lenses constituting the lens plate 32, the number of element lenses in the horizontal and vertical directions, and the like. The parameters of the display panel 31 include the coordinates of the center point _Od of the lens plate 32 as the center point of the image display device 30 described above, the coordinates of the center point of the display area of the display panel 31, the pixel pitch of the pixels displaying the image, This includes the number of pixels in the horizontal and vertical directions. The stereoscopic display parameter input unit 123 outputs the acquired stereoscopic display parameter to the pixel value determination unit 140. When the display position signal indicating the coordinates of the center point _Od of the lens plate 32 is input from the viewing position measurement unit 20 to the viewing position acquisition unit 121, the center point O of the lens plate 32 is included in the stereoscopic display parameter. The coordinates of _d may not be included.

  The display image adjustment parameter input unit 124 acquires display image adjustment parameters. The display image adjustment parameter is a parameter for adjusting the display position and orientation of the display image displayed on the stereoscopic display device. The display image adjustment parameter includes an attachment error adjustment parameter for adjusting an attachment error of the lens plate to the display device, for example, a vertical displacement from a predetermined reference position, a horizontal displacement, and a predetermined reference direction. Includes rotation angle. The display image adjustment parameter input unit 124 outputs the acquired display image adjustment parameter to the pixel value determination unit 140.

  Virtual camera image data is input from the virtual camera image acquisition unit 122 to the camera parameter / virtual camera image association unit 130, and camera parameters related to the virtual camera corresponding to the input virtual camera image data are input from the camera parameter recording unit 110. read. The camera parameter / virtual camera image associating unit 130 associates the read camera parameters with the input virtual camera image data for each virtual camera, that is, for each viewpoint, and forms a set of camera parameters and virtual camera image data. The camera parameter / virtual camera image association unit 130 outputs a set of camera parameters and virtual camera image data formed for each virtual camera to the pixel value determination unit 140.

A set of camera parameters and virtual camera image data is input to the pixel value determination unit 140 from the camera parameter / virtual camera image association unit 130 for each virtual camera. Further, the pixel value determination unit 140 receives a viewing position signal from the viewing position acquisition unit 121 and a stereoscopic display parameter from the stereoscopic display parameter input unit 123.
The pixel value determination unit 140 calculates the coordinates of the intersection point between the normal line of the display area passing through the center point of the display area of the display panel 31 and the main surface of the lens plate 32 as the coordinates of the center point of the lens plate 32. The pixel value determination unit 140 calculates the coordinates of the intersection of the viewing position indicated by the viewing position signal and the straight line passing through the center point of the lens plate 32 included in the stereoscopic display parameter and the display area of the display panel 31. The pixel value determining unit 140 calculates a displacement amount (displacement amount) from the coordinates of the center point of the display area of the display panel 31 to the calculated coordinates. The amount of displacement will be described later.

  The pixel value determining means 140 is based on the three-dimensional display parameter and the camera parameter, and the position of the element lens constituting the display panel 31 that displays the display image and the lens plate 32 that is arranged facing the display panel 31 in parallel. Determine the relationship. Then, the pixel value determination unit 140 specifies an element lens corresponding to the display pixel for each display pixel. The element lens corresponding to the display image is an element lens whose center point is closest to the display pixel. In the stereoscopic display device, when there is an element lens facing the display pixel, the facing element lens is a corresponding element lens.

The pixel value determination unit 140 extracts pixel values of pixels having the same coordinate value indicating the position in the image for each display pixel from the input virtual camera image data.
The pixel value determining unit 140 determines a weighting coefficient for the extracted pixel value based on the position of the display pixel with respect to the position of the element lens corresponding to each display pixel and the position of the virtual camera. The position of the virtual camera corresponds to the position of the corresponding viewpoint. The pixel value determination unit 140 specifies a display pixel at a position where the displacement amount from each display pixel is equal to the calculated displacement amount. The specified display pixel corresponds to a display pixel arranged at a position displaced by a displacement amount calculated from the original display pixel. Then, the pixel value determining means 140 determines the pixel value of the specified display pixel, that is, the value obtained by taking the sum between the virtual camera images for the multiplication value obtained by multiplying the extracted pixel value by the determined weight coefficient, that is, The pixel value of the display pixel at the position displaced by the calculated displacement amount is determined.

The image represented by the determined pixel value is generated based on the image displaced by the displacement amount calculated from the original virtual camera image. Therefore, the displaced position may be outside the display area of the image display device 30. Pixel value determining means 140 rejects the pixel value of the display pixel at a position outside the display area. Further, even if the displaced position is within the display area, if the position before the displacement is outside the display area, the original display image does not exist. Therefore, the pixel value determination unit 140 may set the pixel value of the display pixel whose position before the displacement is outside the display area to a predetermined pixel value. The predetermined pixel value is, for example, a pixel value corresponding to the lowest luminance (reference black luminance) that the display panel 31 can display.
The pixel value determination unit 140 generates display image data indicating the pixel value for each determined display pixel, and outputs the generated display image data to the output unit 150.

  The pixel value determining unit 140 may further receive a display image adjustment parameter from the display image adjustment parameter input unit 124. In that case, the pixel value determination unit 140 corrects the position and orientation of the lens plate using the display image adjustment parameter to which the position and orientation of the predetermined lens plate are input. This correction corrects the position and orientation of each element lens constituting the lens plate. The pixel value determining unit 140 specifies an element lens corresponding to the display pixel from the plurality of element lenses whose positions and orientations are corrected. Therefore, a display image indicated by the pixel value corresponding to the attachment error adjustment parameter that is changed along with the adjustment of the attachment error of the lens plate is obtained. Since the attachment error adjustment parameter can be adjusted based on the display image, the work efficiency related to the calibration of the attachment error adjustment parameter is improved. A more specific method for determining the pixel value in the pixel value determining means 140 will be described later.

  The output unit 150 temporarily stores the display image data input from the pixel value determination unit 140 in an output buffer included in the output unit 150, and outputs the display image data to the display panel 31 of the image display device 30. The display panel displays a display image indicated by display image data input from the output unit 150. The display image data may be output to a device other than the image display device 30, for example, an image database.

(Virtual camera placement example)
Next, an arrangement example of the virtual camera according to the present embodiment will be described. FIG. 3 is a diagram illustrating an arrangement example of the virtual cameras according to the present embodiment. Planar image representing a subject O _b observed from each of the plurality of viewpoints in the generation of the display image is used. In the example shown in FIG. 3, the position of the optical center of the virtual camera corresponding to each viewpoint is arranged in an array in a two-dimensional plane facing the subject. In order to ensure the image quality of the stereoscopic image, it is desirable that the number of virtual cameras arranged in each direction is equal to or greater than the number of pixels that can be accommodated in the region of one element lens constituting the lens plate. For example, when the diameter of the element lens and the pixel pitch are 1 mm and 0.05 mm, respectively, it is desirable that the number of virtual cameras in the horizontal direction and the vertical direction is 20 or more, respectively. Further, the distribution of the virtual cameras may be an array corresponding to the element lenses, or may not be associated with the element lenses. In the example shown in FIG. 3, a plurality of virtual cameras are arranged at equal intervals in the horizontal direction and the vertical direction in a substantially circular area.

Coordinates of the virtual camera V _c to be used as camera parameters (cx, cy) are the coordinates of the virtual camera arranged in their center may be represented by the relative position and the origin O (0,0). With respect to FIG. 3, the coordinates of the virtual camera adjacent to the left side from the origin O are (−p, 0). p indicates the pitch (interval) of the virtual camera. The horizontal coordinate value cx and the vertical coordinate value cy may be values normalized by setting the horizontal and vertical distribution widths of a plurality of virtual cameras to 2C. In that case, the maximum and minimum values of the coordinate values cx and cy are C and -C, respectively. Therefore, 2 × C / p, which is the number of virtual cameras in the horizontal and vertical directions, only needs to be larger than the ratio of the diameter of the element lens to the pixel pitch. When the radius of the element lens is 1, C may be a value larger than 1 depending on an interpolation method used in pixel value calculation described later. For example, C = 1 + 2p may be set. Therefore, the pixel values of the pixels related to the virtual camera whose coordinate values cx and cy are both −1 or more and 1 or less are reliably calculated. Note that the virtual camera image acquisition unit 122 does not necessarily need to acquire virtual camera image data for all of the plurality of virtual cameras all at once. The virtual camera image acquisition unit 122 may acquire virtual camera image data related to one moving virtual camera. Therefore, virtual camera image data indicating a plurality of images observed from different viewpoints at different times is acquired.

As described above, the camera parameter recording unit 110 maintains the positional relationship between the plurality of virtual cameras based on the viewing position indicated by the viewing position signal input from the viewing position acquisition unit 121. The coordinates of the viewpoint may be determined. In the example illustrated in FIG. 3, the camera parameter recording unit 110 determines the center point O of the viewpoint for each of the plurality of virtual cameras as the viewing position, and sets the center point O and the center point O _d of the lens plate 32 of the image display device 30. the straight line passing through (hereinafter, referred to as O-O _d line) in a in a two-dimensional plane to normal, determining the position of the viewpoint of each virtual camera. The camera parameter recording unit 110 sets the original coordinate value (cx, cy) as the coordinate value in the two-dimensional plane of each determined viewpoint. Thereby, the relative positional relationship between viewpoints is kept constant. Note that the camera parameter recording unit 110 may determine the position of each viewpoint so that the angle formed between each viewpoint and the straight line passing through the center point _Od and the O- _Od line is kept constant.
Then, the virtual camera image obtaining means 122 obtains the virtual camera image data representing an image of an object O _b observed in each viewpoint defined. The virtual camera image acquisition unit 122 synthesizes the virtual camera image data using, for example, CG technology.

(Displacement)
Next, the displacement amount will be described. 4 and 5 are diagrams showing examples of the positional relationship between the display image displayed on the image display device 30 and the viewpoint. In the example shown in FIG. 4, the vertical direction (x direction) and the depth direction (z direction) of the display area of the display panel 31 are shown on the lower side and the left side, respectively. The horizontal direction (y direction) of the display area of the display panel 31 corresponds to the depth direction with respect to the drawing. The main surface of the display panel 31 is disposed at a position separated from the main surface of the lens plate 32 by a focal length f in the z direction. The lens plate 32 is composed of element lenses whose diameters are d and whose main surface has a circular shape and are arranged at intervals d in the vertical direction in the xy plane (see FIGS. 8 and 9). When the displacement amount is 0, the element image is displayed in the area facing the element lens in the display area of the display panel 31. Va1, Va2, and Va3 indicate examples of viewing positions, respectively. Viewing position Va1 is on the straight line _{P d} passing through the center point _{O d} of the center point _{O e} and the lens plate 32 in the display area of the display panel 31. The center point of the element lens El is disposed at the same position as the center point O _d of the lens plate 32. Therefore, the viewing position Va1, light from the center point O _e of the element images to be displayed in a region facing the element lenses El arrives via the element lens El. The viewing position Va2 is a position shifted in the vertical direction from the viewing position Va1. The viewing position Va2 is on a straight line passing through the upper end of the element image corresponding to the element lens El and the center point of the element lens El. Therefore, the light beam from the upper end of the element image displayed in the area facing the element lens El arrives at the viewing position Va2 via the element lens El. At the viewing positions Va1 and Va2, since the light rays from the pixels in the element image facing the element lens El arrive, the observed stereoscopic image is not disturbed. On the other hand, the viewing position Va3 is further shifted in the vertical direction than the viewing position Va2. The viewing position Va3 is on a straight line passing through one point _Pn in the element image displayed in the area facing the element lens Em adjacent to the element lens El and the center point of the element lens El. Therefore, the light beam of the pixel arranged at the point _Pn in the element image arrives at the viewing position Va3 via the element lens El. In the example shown in FIG. 4, the point P _n is above the upper end of the element image corresponding to the element lens El, but is located below the center point of the element image corresponding to the element lens Em. Therefore, at the viewing position Va3, unlike the viewing positions Va1 and Va2, a stereoscopic image that should originally be observed is not observed, and a disorder of the stereoscopic image is observed.

In contrast, in the present embodiment, the pixel value determining unit 140, and a straight line P _d passing through the center point O _d of viewing positions Va and the lens plate 32, the coordinates of the intersection point O _f of the display area of the display panel 31 calculate. Pixel value determining means 140 calculates the displacement amount of up coordinates of the intersection point O _f calculated from the center point O _e of coordinates of the display area of the display panel 31 (shift amount). The displacement amount is a two-dimensional vector amount including the displacement amount dx in the x direction and the displacement amount dy in the y direction as elements. The displacement amounts in the x and y directions are calculated using the relationship shown in equation (1). However, in FIG. 5, illustration in the y direction is omitted.

In Expression (1), Dx and Dy indicate the x and y coordinates of the viewing position Va with the center point _Od as the origin. f indicates the focal length of each element lens. z represents the coordinate value (depth value) in the depth direction of the viewing position Va with the center point _Od as the origin. That is, the displacement amounts Dx and Dy in the x and y directions are calculated by multiplying the x coordinate Dx and y coordinate Dy of the viewing position Va by the ratio of the focal length f. By specifying the pixel at the position displaced by the calculated displacement amount, the element lens corresponding to the pixel before displacement is associated with the specified pixel. In the example shown in FIG. 5, the displacement amount in the x direction from the pixel to be in the center point O _e is identified pixel position O _f is dx, and the identified pixels and element lens El is associated. Light from the pixel position O _f in the associated image element in viewing position Va arrives via the element lens El. Therefore, a desired stereoscopic image is observed at the viewing position Va.

(Display image)
Next, an example of a display image will be described. FIG. 6 is a diagram illustrating an example of a display image. Display image shown in FIG. 6, the display image observed at the positions of a plurality of virtual cameras subject O _b shown in FIG. 3, obtained by integration and synthesized in the pixel value determining unit 140. This display image is a two-dimensional image displayed on the display panel 31 of the image display device based on the display image data finally output from the output unit 150. A viewer can visually recognize the stereoscopic image by observing the display image that has passed through the lens plate 32.
In FIG. 6, (u, v) indicates the coordinate value of the pixel in the display image. The origin is the lower left pixel of the display image. u and v are values indicating coordinate values in the horizontal direction and the vertical direction, respectively, and are values normalized to 0 or more and 1 or less. For example, the coordinates of the pixels at the lower left and upper right corners of the display image are (0, 0) and (1, 1), respectively.

FIG. 7 is an enlarged view showing the periphery of the coordinate values (u, v) shown in FIG. The element image constituting the display image is associated with one element lens of the lens plate facing the display pixel. A circle centered on the + mark indicates a region of the element image corresponding to one element lens. In order to determine the pixel value of the display pixel, the pixel value determining unit 140 specifies the center point on the display panel of the element lens closest to the coordinate value (u, v) of the display pixel. The pixel value determining unit 140 converts the coordinate value (u, v) into a coordinate value (rx, ry) of the lens center coordinate system having the center point _Or as the origin (0, 0).

  More specifically, the pixel value determining unit 140 converts the coordinate value (u, v) into a coordinate value (Px, Py) with the display pixel as a unit using the relationship shown in Expression (2).

In Expression (2), width and height indicate the number of pixels in the horizontal direction and the vertical direction of the element image, respectively. The number of pixels in the horizontal and vertical directions of the element image usually corresponds to the resolution of the display panel. For example, when the display panel of the stereoscopic display device is an 8K display having a resolution of 8K, the number of pixels in the horizontal direction and the vertical direction of the display panel is 7680 and 4320, respectively.
In addition, when the coordinate value (Px, Py) is acquired instead of the coordinate value (u, v), the pixel value determination unit 140 determines the coordinate value (Px, Py) from the coordinate value (u, v). The conversion to is omitted.

  When the display image adjustment parameter is input, the pixel value determination unit 140 corrects the coordinate value (Px, Py) using the input display image adjustment parameter. The corrected coordinate values (Px ′, Py ′) are calculated using the relationship shown in Expression (3).

  In Expression (3), h, v, and θ represent a horizontal displacement (displacement) from a predetermined reference point, a vertical displacement from the reference point, and a rotation angle from the predetermined reference direction, respectively.

  The pixel value determining means 140, for the position represented by the coordinate values (Px ′, Py ′) as shown in the equation (4), the coordinate values (Px ″, P) of the position displaced by the displacement amount (dx, dy). Py ″) is calculated. By this calculation, the position displaced by the displacement amount (dx, dy) from the position of the original display pixel is specified.

  If no display image adjustment parameter is input, the coordinate values (Px ′, Py ′) cannot be obtained. Therefore, the pixel value determination unit 140 calculates the coordinate values (Px ″, Py ″) using the coordinate values (Px, Py) instead of the coordinate values (Px ′, Py ′) in the equation (4). To do.

  Next, the pixel value determination unit 140 specifies the center point of the element lens that is closest to the coordinate value (u, v) of the display pixel. A procedure for specifying the center point of the closest element lens will be described by taking as an example a case where the arrangement of the element lenses is a delta arrangement. The delta arrangement is an array of objects arranged at equal intervals in the column direction for each row, the position of the array object arranged in a row in the column direction, and the arrangement object arranged in a row adjacent to the row. In this arrangement, the deviation from the position in the column direction is half the arrangement interval in the column direction. In the example shown in FIG. 8, the array target is an element lens, and the center point of each element lens is arrayed at each vertex of the triangular lattice. Accordingly, with respect to the position of the element lens in the column direction, the deviation between adjacent rows is d / 2, and the distance between the element lens columns is d√3 / 2. Here, d indicates the diameter of each element lens, and it is assumed that the diameter of the element lens is equal to the arrangement interval in the column direction. Note that d is a value (unit: pix) normalized by the pixel pitch pp (unit: mm / pix) of the display panel with respect to the diameter dl (unit: mm) of the actual element lens.

  Then, the pixel value determination unit 140 identifies an area including the position of the display pixel among the square areas inscribed with the element lenses. A square indicated by a broken line in FIG. 8 indicates one of the regions. Assuming that the specified region is the Nth row from the bottom row of the lens plate, the pixel value determining means calculates N using the relationship shown in Expression (5).

In the formula (5), (int)... Indicates an integer value obtained by rounding down values after the decimal point of the real numbers. The pixel value determining means 140 can determine the column number of the specified region by applying the same method as the calculation of the row number N of the region to the element lens column.
The pixel value determining unit 140 converts the coordinate value (Px ″, Py ″) into a coordinate value (Qx, Qy) having the origin at the lower left corner of the specified area. The pixel value determining unit 140 calculates the coordinate value (Qx, Qy) using the relationship shown in Expression (6).

  In Equation (6), e is a variable indicating whether the row number N is an even row or an odd row. Specifically, the variable e is calculated as a remainder obtained by dividing the line number N by 2 as shown in Expression (7). Accordingly, the variable e is 0 when the line number N indicates an even line, and the variable e is 1 when the line number N indicates an odd line.

  The pixel value determining unit 140 converts the coordinate value (Qx, Qy) into a coordinate value (Qx ′, Qy ′) having the center point of the specified square area as the origin. The coordinate values (Qx ′, Qy ′) are calculated using the relationship shown in Expression (8). In the following description, the coordinate value (Qx ′, Qy ′) or the coordinate point at the position represented by the coordinate value may be expressed as Q ′.

  Next, the pixel value determining unit 140 determines the center point of the element lens closest to the coordinate point Q ′ whose position is represented by the converted coordinate values (Qx ′, Qy ′). In the example shown in FIG. 9, the center point of the candidate element lens is the coordinate point pl0 of the center point of the specified region and the center points of six adjacent element lenses whose distance from the center point is d. is there. The coordinate values pl0 to pl6 of these seven center points are expressed by Expression (9). In this example, the coordinate values (Qx ', Qy') and pl0 to pl6 of the coordinate point Q 'are represented by a coordinate system having the center point pl0 as the origin.

  The coordinate value plc of the center point of the element lens closest to the coordinate value Q ′ is expressed using the relationship shown in Expression (10).

In Expression (10), i is an index of the center point taking any value from 0 to 6. argmin _pri ... indicates a coordinate value pri that minimizes.
Then, the pixel value determining means 140 converts the coordinate value (Qx ′, Qy ′) into a coordinate value Q ″ having the origin at the center point plc of the element lens closest to the coordinate point Q ′. The coordinate value Q ″ is expressed using the relationship shown in Expression (11).

  The pixel value determination means 140 calculates the coordinate value (rx, ry) by normalizing the coordinate value Q ″ converted as shown in Expression (12) by the radius d / 2 of the element lens.

  Through the above processing, the coordinate value (u, v) of the display pixel is converted into the coordinate value (rx, ry) of the lens center coordinate system, and the element lens closest to the original coordinate value (u, v) The converted coordinate values (rx, ry) are associated with each other. This conversion includes correction based on the above-described viewing position and display image correction parameters.

Next, the pixel value determining unit 140 sets each pixel value of the pixel arranged at the position represented by the coordinate value (u, v) of the original virtual camera image as the pixel arranged at the corrected position. Extract about the camera. The pixel value determining means 140 determines a value obtained by synthesizing between the virtual cameras a multiplication value obtained by multiplying the extracted pixel value by a weighting factor corresponding to the pixel value as a pixel value of the display pixel. Specifically, the pixel value determination unit 140 calculates the pixel value col ^{(u, v)} of the display pixel arranged at the coordinate value (u, v) using the relationship shown in Expression (13).

In Expression (13), w (cx, cy, rx, ry, p) represents a weighting coefficient. The weight coefficient w (cx, cy, rx, ry, p) depends on the coordinate value (cx, cy) indicating the position of the virtual camera, the coordinate value (rx, ry) of the lens center coordinate system, and the pitch p of the virtual camera. To do. In Equation (16), Σ is a symbol indicating the sum. The coordinate values (cx, cy) attached to this symbol indicate the relative position of the viewpoint that originally has the center point of the coordinate value of the virtual camera as the origin. In Expression (14), the pixel value col ^{(u, v)} _{(cx, cy)} is the coordinate value of the display image among the pixels constituting the virtual camera image of the virtual camera arranged at the coordinate value (u, v). The pixel value of the pixel arrange | positioned in the position (rx, ry) corrected from the position represented by (u, v) is shown. The sum is calculated independently for each display pixel between coordinate values (cx, cy) indicating the virtual camera.

  The weighting factors w (cx, cy, rx, ry, p) are the factors w ′ (cx, rx, p), w ′ (cy, ry, p) in the horizontal direction and the vertical direction as shown in the equation (15). ).

  As a factor w ′ (s, t, p) for each direction, an interpolation coefficient used in a known image interpolation method is applicable. As an example of the interpolation method, a bicubic method can be used. The bicubic method is an interpolation method in which a factor w ′ (s, t, p) calculated using the relationship shown in Expression (15) is used as a weighting factor.

  X shown in Expression (15) is obtained by normalizing the coordinate value based on the center point of the element lens of the pixel constituting the image from the virtual camera with the interval p of the element lens. a represents an adjustment coefficient for adjusting the factor w ′ (s, t, p). The adjustment coefficient a is a real number within a range of −0.5 to −1.0, for example. Accordingly, the factor w ′ (s, t, p) has a value within a predetermined range (in this case, −4/27 to 1). The factor w ′ (s, t, p) takes a positive or negative value except when x is 0 to 2 and x = 1, and becomes 0 when x is 2 or more. As a result, pixels outside the region that is at least twice the element lens interval p from the center point of the element lens closest to the coordinate value (u, v) are ignored in the calculation of the pixel value of the display pixel. Pixels in the region are targets for calculation of pixel values of display pixels. Further, the factor w ′ (s, t, p) has a smaller absolute value of x, that is, a pixel closer to the center point of the element lens closest to the coordinate value (u, v), and is larger, and the pixel passes from the center point. It gets smaller. Thereby, the component of the virtual camera image with respect to the pixel value calculated between display pixels is smoothed. Therefore, image quality degradation due to spatial aliasing is alleviated. Specifically, artifacts such as unnatural lines and double images are reduced or eliminated.

  As another example of the interpolation method, a nearest neighbor interpolation method is also applicable. The nearest neighbor interpolation method is an interpolation method in which a factor w ′ (s, t, p) calculated using the relationship shown in Expression (16) is used as a weighting factor.

  In Expression (16), x is obtained by normalizing the coordinate value based on the center point of the element lens of the pixel with the interval p of the element lens as shown in the lower part of Expression (15). Therefore, the factor w ′ (s, t, p) is 1 for pixels in which the absolute value of x is 0.5 or less, that is, within a range of a distance p / 2 that is half the distance from the center point of the element lens. A pixel outside the range is 0. As a result, pixels outside the area that is less than half of the distance p between the element lenses from the center point of the element lens that is closest to the coordinate value (u, v) are ignored in the calculation of the pixel value of the display pixel. This pixel is a calculation target of the pixel value of the display pixel.

FIG. 10 shows an example of a virtual camera image related to generation of a display image. FIGS. 10A, 10B, and 10C show images acquired from virtual cameras installed at different positions. In the examples shown in FIGS. 10A, 10B, and 10C, the coordinate values (cx, cy) of the virtual camera are (1.1, −0.1), (0.0, 0.0), respectively. ), (−1.1, 0.1). That is, the image shown in FIG. 10A is obtained by observing from the lower left viewpoint than the image shown in FIG. The image shown in FIG. 10C is obtained by observing from the upper right viewpoint than the image shown in FIG. The pixel value col ^{(u, v)} _{(cx, cy)} corresponding to the display pixel at the coordinate value (u, v) and the pixel of the image shown in FIGS. 10 (a), (b), and (c _{) Are} col ^{(u, v)} _{(1.1, -0.1)} , col ^{(u, v)} ₍₀ , ₀₎ , col ^{(u, v)} _{(-1.1, 0.1)} , respectively. Is done. In addition, the pixel value corresponding to the display pixel at the coordinate value (u, v) and the pixel value col ^{(u, v)} _{(cx, The} weight coefficients w (cx, cy, rx, ry, p ₎ for _cy) are w (1.1, -0.1, rx, ry, p) and w (0, 0, rx, ry, p), respectively. , W (−1.1, 0.1, rx, ry, p). As described above, the pixel value col ^{(u, v)} of the display pixel arranged at the coordinate value (u, ^v) is the pixel value col ^{(u, v)} _{(1.1, −0.1)} and the weighting factor. The product of w (1.1, −0.1, rx, ry, p), the pixel value col ^{(u, v)} _{(0, 0)} and the weighting factor w (0, 0, rx, ry, p) And the product of the pixel value col ^{(u, v)} _{(−1.1, 0.1)} and the weight coefficient w (−1.1, 0.1, rx, ry, p) is added between the virtual cameras. It is calculated by doing.

The calculation of the pixel value col ^{(u, v)} of each display pixel does not store data in the middle of calculation in an intermediate image buffer or the like, and the required pixel among the pixel values of the pixels constituting each virtual camera image The value is extracted and executed. Since this calculation is performed independently for each display pixel, the entire processing relating to the generation of the display image is accelerated by performing them in parallel.

(Image generation processing)
Next, image generation processing according to the present embodiment will be described. FIG. 11 is a flowchart illustrating an example of image generation processing according to the present embodiment.
(Step S <b> 101) The camera parameter recording unit 110 acquires information on the position of each of a plurality of virtual cameras in a three-dimensional virtual space, and sets the acquired information. Thereafter, the process proceeds to step S102.
(Step S102) The viewing position measurement unit 20 measures the viewing position in real space. The viewing position acquisition unit 121 acquires the viewing position measured by the viewing position measurement unit 20. Thereafter, the process proceeds to step S103.

(Step S103) The camera parameter recording unit 110 sets the viewpoint position of each of the plurality of virtual cameras based on the measured viewing position. Here, the camera parameter recording unit 110 sets the position of each viewpoint such that the center point of the viewpoints of the plurality of virtual cameras in the virtual space is the viewing position. Thereafter, the process proceeds to step S104.
(Step S104) The camera parameter recording unit 110 calculates a relative position with the center point of the viewpoint of each of the plurality of virtual cameras as the origin, and records relative position information indicating the calculated relative position as part of the camera parameter. Thereafter, the process proceeds to step S105.

(Step S105) The three-dimensional display parameter input means 123 acquires a three-dimensional display parameter. The display image adjustment parameter input unit 124 acquires display image adjustment parameters. The display image adjustment parameter includes a lens plate attachment error adjustment parameter. Thereafter, the process proceeds to step S106.
(Step S106) The pixel value determining means 140 calculates the coordinates of the intersection of the viewing position and the straight line passing through the center point of the lens plate 32 and the display area of the display panel 31, and from the coordinates of the center point of the display area The amount of displacement up to the calculated coordinates is calculated. Thereafter, the process proceeds to step S107.

(Step S107) The pixel value determination unit 140 performs display processing of pixel value determination processing using the calculated displacement amount, the relative position information of the virtual camera, and the virtual camera image data for each virtual camera to display the display image data. Generate. The pixel value determination process will be described later. Thereafter, the process proceeds to step S108.
(Step S <b> 108) The output unit 150 outputs the display image data generated by the pixel value determination unit to the display panel 31 of the image display device 30. A display image based on the display image data supplied from the output unit 150 is displayed on the display panel 31. Then, the process shown in FIG. 11 is complete | finished.

Next, pixel value determination processing according to the present embodiment will be described.
FIG. 12 is a flowchart illustrating an example of a pixel value determination process according to the present embodiment.
The process shown in FIG. 12 is executed in step S107 in FIG.
(Step S111) The pixel value determining means 140 converts the coordinate value (u, v) for each display pixel constituting the display image into a coordinate value (Px, Py) in units of pixels. Thereafter, the process proceeds to step S112.
(Step S112) The pixel value determining means 140 corrects the coordinate values (Px, Py) of the converted display pixels to the coordinate values (Px ′, Py ′) using the display image adjustment parameters. Thereafter, the process proceeds to step S113.

(Step S113) The pixel value determining means 140 adds the displacement (dx, dy) calculated based on the viewing position to the corrected coordinate value (Px ′, Py ′) of the display pixel, thereby obtaining the coordinate value (Px '', Py ''). Thereafter, the process proceeds to step S114.
(Step S114) The pixel value determining unit 140 specifies the element lens including the position represented by the corrected coordinate value (Px ″, Py ″) of the display pixel as the corresponding element lens. The pixel value determining unit 140 converts the coordinate value (Px ″, Py ″) into the coordinate value (rx, ry) of the lens center coordinate system with the specified center point of the element lens as a reference. Thereafter, the process proceeds to step S115.

(Step S115) The pixel value determination means 140 acquires the pixel value of the pixel arranged at the position of the coordinate value (u, v) of the display pixel in the virtual camera image from each virtual camera. Thereafter, the process proceeds to step S116.
(Step S116) The pixel value determining means 140, based on the relative position (cx, cy) of each virtual camera and the coordinate value (rx, ry) of the display pixel, the weight coefficient w (cx) for the corresponding pixel of the virtual camera. , Cy, rx, ry, p). Thereafter, the process proceeds to step S117.

(Step S117) The pixel value determining means 140 is the pixel value col ^{(u, v} ) of the pixel arranged at the position represented by the coordinate value (u, v) of the display image among the images acquired by each virtual camera. ⁾ Calculate the product of _{(cx, cy)} and the weight value w (cx, cy, rx, ry, p) for this pixel, add the calculated products between the virtual cameras, and obtain the pixel value of the display pixel col ^{(u, v)} is defined. The pixel value determining unit 140 generates display image data representing a pixel value determined for each display pixel. Thereafter, the process proceeds to step S108 (FIG. 11).

  As described above, the image generation device 10 according to the present embodiment generates a display image to be displayed on the image display device 30. The image display device 30 is disposed so as to face a position where a distance from a lens plate 32 in which a plurality of element lenses are arranged on a display panel 31 that is a display unit that displays an image is a focal length of the element lens. Become. The image generation apparatus 10 includes a virtual camera image acquisition unit 122 that acquires virtual camera images of a plurality of viewpoints, and a viewing position acquisition unit 121 that acquires a viewing position. Further, the image generation device 10 calculates the amount of displacement from the center point of the display panel 31 up to the intersection of the viewing position and the straight line passing through the center point of the lens plate 32 and the display panel 31, and is a pixel of the display image. The element lens corresponding to the display pixel is identified, and the value obtained by multiplying the pixel value of the pixel of the virtual camera image at the same position as the display pixel by the weighting coefficient based on the position of the display pixel and the position of the virtual camera image is virtually A pixel value determining unit 140 is provided that calculates a pixel value of a display pixel at a position where the displacement amount from the display pixel is calculated by combining the camera images.

  With this configuration, the pixel value of the pixel of the virtual camera image at the same position as the display pixel constituting the display image, the position of the display image based on the position of the element lens corresponding to the display pixel, and the viewpoint of the virtual camera image The pixel value of the display pixel at the position where the displacement amount from the original display pixel becomes the calculated displacement amount is calculated using the determined weight coefficient. Therefore, as an image in the region corresponding to each element lens in the virtual camera image, an element image is generated by synthesizing the image of the region displaced from the viewing position toward the center point of the lens plate 32 between the virtual cameras. The Therefore, even if the displacement amount is larger than the diameter of the element lens, each element image is presented through the corresponding element lens at the viewing position, so that the viewing area where the stereoscopic image is visually recognized is enlarged. In addition, since it is possible to directly generate a display image without generating an intermediate image based on a virtual camera image, it is possible to increase the generation speed of the display image and improve the memory utilization efficiency. Furthermore, since the pixel value of the display pixel is calculated independently between the display pixels, the generation of the display image can be further speeded up by processing these operations in parallel. Since the increase in the number of virtual cameras is allowed with an increase in speed, the image quality of an image to be viewed is improved by setting the viewpoint for acquiring the virtual camera images more densely. Therefore, the application of a stereoscopic image by an image display device to an interactive medium such as a computer game and video communication is promoted as well as a unidirectional medium such as a normal television program.

The image generation apparatus 10 includes camera parameter recording means 110 that sets the positions of a plurality of viewpoints so that the positions of the center points of the plurality of viewpoints become viewing positions.
With this configuration, a plurality of virtual camera images observed from viewpoints distributed around the viewing position are used to generate a display image. Therefore, the entire viewpoint for acquiring the virtual camera image follows the viewing position. Therefore, a plurality of virtual camera images acquired for each viewpoint are effectively used for generating a display image. As a result, the image quality of the stereoscopic image visually recognized for the display image is improved.

The image display system 1 includes a viewing position measurement unit 20, an image display device 30, and an image generation device 10. The viewing position acquisition unit 121 of the image generation apparatus 10 acquires the viewing position measured by the viewing position measurement unit 20.
With this configuration, the image generation apparatus 10 generates a display image using the virtual camera image displaced by the displacement amount calculated based on the viewing position measured by the viewing position measurement unit 20. The image display device 30 displays the generated display image. Therefore, the viewer can visually recognize a stereoscopic image corresponding to the viewing position with respect to the display image displayed on the image display device 30.

  As described above, the embodiment of the present invention has been described in detail with reference to the drawings. However, the specific configuration is not limited to the above, and various design changes and the like can be made without departing from the scope of the present invention. It is possible to

For example, the image generation device 10 may be configured as a single device integrated with one or both of the viewing position measurement unit 20 and the image display device 30.
The above-described lens plate 32 is an example in which the center point of each element lens is arranged on each lattice point of a triangular lattice whose unit graphic is a regular triangle. However, the present invention is not limited to this. . The center point of the element lens may be arranged at each lattice point of a lattice having parallel movement symmetry that is spatially repeated with a predetermined period. Examples of such a lattice include a square lattice, a rectangular lattice, and an isosceles triangular lattice. The shape of the main surface of each element lens is not limited to a circle, and may be, for example, a rectangle or a triangle.

Further, part or all of the image generation apparatus 10 described above may be realized by a computer including an integrated circuit capable of executing data processing such as GPU (Graphics Processing Unit), particularly parallel processing. In that case, the program for realizing the control function may be recorded on a computer-readable recording medium, and the program recorded on the recording medium may be read by the computer system and executed. Here, the “computer system” is a computer system built in the image generation apparatus 10 and includes an OS and hardware such as peripheral devices. The “computer-readable recording medium” refers to a storage device such as a flexible medium, a magneto-optical disk, a portable medium such as a ROM and a CD-ROM, and a hard disk incorporated in a computer system. Furthermore, the “computer-readable recording medium” is a medium that dynamically holds a program for a short time, such as a communication line when transmitting a program via a network such as the Internet or a communication line such as a telephone line, In this case, a volatile memory inside a computer system that serves as a server or a client may be included that holds a program for a certain period of time. The program may be a program for realizing a part of the functions described above, and may be a program capable of realizing the functions described above in combination with a program already recorded in a computer system.
Further, a part or all of the image generation apparatus 10 in the above-described embodiment may be realized as an integrated circuit such as an LSI (Large Scale Integration). Each functional block of the image generation apparatus 10 may be individually made into a processor, or a part or all of them may be integrated into a processor. Further, the method of circuit integration is not limited to LSI, and may be realized by a dedicated circuit or a general-purpose processor. In addition, when an integrated circuit technology that replaces LSI appears due to the advancement of semiconductor technology, an integrated circuit based on the technology may be used.

DESCRIPTION OF SYMBOLS 1 ... Image display system, 10 ... Image generation apparatus, 20 ... Viewing position measurement part, 30 ... Image display apparatus, 110 ... Camera parameter recording means, 120 ... Input part, 121 ... Viewing position acquisition means, 122 ... Virtual camera Image acquisition means, 123 ... stereoscopic display parameter input means, 124 ... display image adjustment parameter input means, 130 ... camera parameter / virtual camera image association means, 140 ... pixel value determination means, 150 ... output unit

Claims (4)

An image for generating a display image to be displayed on an image display device arranged so as to face a position where a distance from a lens plate in which a plurality of element lenses are arranged is a focal length of the element lens. A generating device,
Virtual camera image acquisition means for acquiring virtual camera images of a plurality of viewpoints;
Viewing position acquisition means for acquiring the viewing position;
Calculating the amount of displacement from the central point of the display unit up to the intersection of the viewing position and the straight line passing through the central point of the lens plate and the display unit;
Identifying an element lens corresponding to a display pixel that is a pixel of the display image;
A value obtained by multiplying the pixel value of the pixel of the virtual camera image at the same position as the display pixel by a weighting factor based on the position of the display pixel and the position of the virtual camera image is synthesized between the virtual camera images. Pixel value determining means for calculating a pixel value of a display pixel at a position where the displacement amount calculated from the display pixel is the displacement amount;
An image generation apparatus comprising: The image generation device according to claim 1, further comprising camera parameter recording means configured to set positions of the plurality of viewpoints such that positions of center points of the plurality of viewpoints become the viewing positions. Viewing position measurement unit, the image display device, the image generation device according to claim 1 or 2,
An image display system comprising:
The viewing position acquisition means includes
An image display system, wherein the viewing position measured by the viewing position measurement unit is acquired. An image for generating a display image to be displayed on an image display device arranged so as to face a position where a distance from a lens plate in which a plurality of element lenses are arranged is a focal length of the element lens. A generating device,
Virtual camera image acquisition means for acquiring virtual camera images of a plurality of viewpoints;
Viewing position acquisition means for acquiring the viewing position;
Calculating the amount of displacement from the central point of the display unit up to the intersection of the viewing position and the straight line passing through the central point of the lens plate and the display unit;
Identifying an element lens corresponding to a display pixel that is a pixel of the display image;
A value obtained by multiplying the pixel value of the pixel of the virtual camera image at the same position as the display pixel by a weighting factor based on the position of the display pixel and the position of the virtual camera image is synthesized between the virtual camera images. Pixel value determining means for calculating a pixel value of a display pixel at a position where the displacement amount calculated from the display pixel is the displacement amount;
A program for causing a computer to function as an image generation apparatus.