JP2015222892A - Display image generation device and program therefor - Google Patents

Abstract

PROBLEM TO BE SOLVED: To provide a display image generation device capable of adding a variety of visual effects.SOLUTION: A display image generation device 1 includes: modeling parameter input means 11; normalization image set means 12; a visual effect addition means 13 for setting a function which associates a view point position, a display range of a three-dimensional shape model, a display time of the three-dimensional shape model and an additional information pixel region for displaying additional information, and obtaining a view point position on the basis of the function, and imaging the three-dimensional shape model with a virtual camera of the obtained view point position, to generate a visual effect image; normalized image generation means 14 for allocating the pixel value of the pixel of the visual effect image to the pixel of the normalized image, to generate a normalized image; and display image generation means 15 for generating a display image from the normalized image.

Description

  The present invention relates to a display image generation apparatus that generates a display image of a stereoscopic display apparatus using an integral photography (IP) system and a program therefor.

  In recent years, research and development of stereoscopic display devices have progressed, and naked-eye type stereoscopic display devices that allow an observer to visually recognize a stereoscopic image without using special glasses have been developed. For example, an IP stereoscopic display device includes a high-definition image display panel and a two-dimensional lens array disposed on the front surface thereof, and a stereoscopic image is reproduced when an element image is displayed on the high-definition image display panel.

  In the conventional IP method, a method specialized in displaying a stereoscopic image has been proposed (for example, Patent Document 1). The invention described in Patent Document 1 is suitable for each observer by judging an appropriate depth value (amount of popping out) by referring to personal information such as age when a plurality of persons observe an integral stereoscopic image at the same time. A stereoscopic image is displayed. For example, in the invention described in Patent Document 1, a stereoscopic image with an appropriate depth can be displayed according to the age of an observer such as a child or an adult.

JP 2006-262191 A

  However, the conventional IP method merely reproduces a virtual three-dimensional space and faithfully displays a stereoscopic image. Therefore, in the conventional IP system, there is a demand for adding various visual effects such as displaying the entire circumference of the subject on a stereoscopic display device whose viewing angle is limited to about 20 degrees or displaying the temporal change of the subject. There is.

  It is an object of the present invention to provide a display image generation apparatus and a program for adding various visual effects.

  In view of the above-described problems, the display image generation device according to the present invention is a display with a visual effect added to display on a stereoscopic display device including a lens array in which element lenses are two-dimensionally arranged and image display means. A display image generation device for generating an image, comprising a normalized image setting unit, a normalized image pixel position calculation unit, a function setting unit, a visual effect image generation unit, a normalized image generation unit, and a display image generation And means.

  According to this configuration, the display image generation device sets a normalized image having a pixel size that is a natural number with respect to the distance between the element lenses by the normalized image setting unit. In the display image generation device, the normalized image pixel position calculation unit calculates the pixel position of the normalized image based on the pixel interval of the image display unit and the pixel size of the normalized image.

In addition, the display image generation device is related to the display range of the three-dimensional shape model based on the visual effect parameter representing at least one of the display range or the display time of the three-dimensional shape model representing the subject by the function setting means. A function of the viewpoint position of the virtual camera and the subject state representing the display time of the three-dimensional shape model related to the viewpoint position is set.
As described above, the display image generation apparatus interposes a function between the viewpoint position and the state and time change of the three-dimensional shape model without directly matching the three-dimensional shape model to be displayed with the viewpoint position.

  The display image generation device obtains the viewpoint position of the virtual camera based on the function set by the function setting means by the visual effect image generation means, and shoots the three-dimensional shape model with the virtual camera at the obtained viewpoint position. Thus, a visual effect image to which a visual effect is added is generated.

  In addition, the display image generation device generates a normalized image by assigning the pixel value of the pixel of the visual effect image to the pixel of the normalized image using a predetermined coordinate conversion formula by the normalized image generation unit. Then, the display image generating device generates a display image from the normalized image generated by the normalized image generating unit by the display image generating unit.

  The present invention does not directly correspond to the viewpoint position of the 3D shape model to be displayed, but intervenes a function between the viewpoint position and the state and time change of the 3D shape model, thereby providing various visual effects. Can be added. In the present invention, by moving the viewpoint, it is possible to display a three-dimensional shape model or display an animation of the three-dimensional shape model beyond the limit of the viewing angle of the stereoscopic display device.

(A) is a schematic diagram of a conventional stereoscopic imaging device, and (b) is a schematic diagram of a conventional stereoscopic display device. It is a block diagram which shows the structure of the stereo image production | generation apparatus which concerns on embodiment of this invention. It is a conceptual diagram which shows modeling of the three-dimensional display apparatus in embodiment of this invention, and is explanatory drawing which shows the positional relationship of an element image, an image display panel, and a lens array. In embodiment of this invention, it is explanatory drawing which shows the positional relationship of a normalized image and a lens array. It is a block diagram which shows the structure of the visual effect addition means of FIG. In embodiment of this invention, it is explanatory drawing which shows the positional relationship of a normalized image, a lens array, and a projection surface. In embodiment of this invention, it is explanatory drawing which shows the positional relationship of a three-dimensional shape model and a viewpoint position. (A) And (b) is explanatory drawing which shows the relationship between a viewpoint position and the display range of a three-dimensional shape model in embodiment of this invention. In embodiment of this invention, it is explanatory drawing which shows the relationship between a viewpoint position and the display time of a three-dimensional shape model. In embodiment of this invention, it is explanatory drawing explaining imaging | photography with a virtual camera, and is a figure which shows the positional relationship of a lens array, a normalized image, and a three-dimensional shape model. In embodiment of this invention, it is explanatory drawing explaining imaging | photography with a virtual camera, and is a figure which shows the positional relationship of the sample point on a lens array, and a normalized image. In embodiment of this invention, it is explanatory drawing explaining imaging | photography with a virtual camera, and is a figure which shows the positional relationship of a normalized image, a lens array, and a virtual camera. In embodiment of this invention, it is explanatory drawing explaining the production | generation of a display image. In the embodiment of the present invention, (a) shows an example of a three-dimensional shape model, (b) shows an example of additional information, and (c) is an explanatory diagram showing an example of an additional information pixel region. (A)-(c) is explanatory drawing explaining insertion of additional information in embodiment of this invention. It is a flowchart which shows operation | movement of the display image generation apparatus of FIG. It is a flowchart which shows the visual effect image generation process of FIG.

[Outline of IP method]
Hereinafter, embodiments of the present invention will be described in detail with reference to the drawings as appropriate.
First, an outline of a general IP system will be described as a premise of the display image generation apparatus 1 (see FIG. 2) according to the embodiment of the present invention.

As shown in FIG. 1 (a), the stereoscopic imaging device 2 is for imaging an IP scheme subject U, a lens array L _A placing the element lenses L _P two-dimensionally, and a high-definition camera 21 Prepare. Stereoscopic imaging apparatus 2 via the lens array L _A, records the light from the subject U high definition camera 21. Thus, the stereoscopic imaging device 2, according to the specifications of the lens array L _A, generates a display image I lined image formed by the element lenses L _P.

As shown in FIG. 1B, the stereoscopic display device 3 displays a display image I (an image in which a plurality of element images G are arranged vertically and horizontally) generated by the stereoscopic imaging device 2 in an IP system. It includes an image display panel (image display means) 31 such as a display, and a lens array L _a placing the element lenses L _P two-dimensionally.

The stereoscopic display device 3 displays the display image I to be displayed generated by the stereoscopic imaging device 2 on the display surface of the image display panel 31. In this case, the element lenses L _P corresponding to the element image G is uniquely determined. Then, the stereoscopic display device 3 forms a three-dimensional image Z and projected by the element lenses L _P elements image G in the air. At this time, the viewer A, through element lenses L _P of the lens array L _A, by viewing each element image G, it is possible to visually recognize a stereoscopic image Z. As described above, in the IP method, since the light rays in the subject space are acquired and reproduced, the stereoscopic image Z corresponding to the viewpoint position can be displayed even if the viewpoint moves. In other words, the IP method reproduces a stereoscopic image Z as if the subject U actually exists.

  Here, if the display image generating apparatus 1 (FIG. 2) is used, various visual effects can be added. For example, the display image generation device 1 displays the entire circumference of the subject U on the stereoscopic display device 3 by moving the viewpoint (first example of visual effect), or displays the subject U as an animation (visual effect). Second example), additional information different from the subject U can be displayed (third example of visual effects).

[Configuration of display image generation apparatus]
With reference to FIG. 2, the structure of the display image generation apparatus 1 which concerns on embodiment of this invention is demonstrated.
The display image generating apparatus 1 generates a display image to which a visual effect is added, and includes a modeling parameter input unit 11, a normalized image setting unit 12, a visual effect adding unit 13, and a normalized image generating unit. 14, display image generation means 15, and additional information insertion means 16.

The modeling parameter input means 11 receives, for example, modeling parameters (F, R _H , R _V , P) for modeling the stereoscopic display device 3 from the user of the display image generation device 1, This modeling parameter is output to the normalized image setting means 12. This modeling parameter is defined as shown in FIG.

F: focal length of the element lenses _{L P} (distance between the image display panel 31 and the lens array _{L A)}
R _H: horizontal spacing of the lens array _{L A} element lenses in _{L P} (horizontal spacing)
R _V: vertical spacing of the lens array _{L A} element lenses in _{L P} (vertical spacing)
P: Pixel interval of the image display panel 31

In this embodiment, the element lens L _P are offset by the horizontal distance R _{H /} 2 per line, the vertical distance R _V is described as a √3 / 2 times the horizontal spacing R _H.
Further, the description will be made assuming that the horizontal interval _RH and the vertical interval _RV are not natural numbers times the pixel interval of the element image G.

Here, since the horizontal interval _RH and the vertical interval _RV are not a natural number times the pixel interval of the element image G (this natural number does not include 0), even if shooting is performed using a virtual camera (orthographic camera), Light rays cannot be directly acquired at the pixel position of the element image G. Therefore, as shown in FIG. 4, in the process of taking a picture with a virtual camera (calculation process), the pixel interval of the element image G is defined, and the horizontal interval _RH and the vertical interval _RV are natural numbers of the pixel interval of the element image G. Prepare an intermediate image that doubles. Hereinafter, this intermediate image is referred to as a normalized image I ′.

It also defines world coordinates that indicate the physical location. The world coordinates as the origin the center of the lens element lens L _P located in the center of the lens array L _A, X axis horizontal direction of the lens array L _A, the vertical direction in the vertical direction of the Y-axis and the lens array surface Z Axis.
Further, (i, j) is defined as the image coordinates of the entire element image G. It is assumed that the origin of the image coordinates is the pixel closest to the origin of the world coordinates.

The normalized image setting unit 12 sets the normalized image I ′ based on the modeling parameter input from the modeling parameter input unit 11. Specifically, the normalized image setting unit 12, the normalized image I', as represented by the following formula (1), pixels between horizontal spacing R _H and the vertical distance R _V elements lens L _P By dividing by the number M, the pixel size P _{H in} the horizontal direction of the normalized image I ′ and the pixel size P _V in the vertical direction of the normalized image I ′ are calculated. Hereinafter sometimes abbreviated as horizontal pixel size P _H and a vertical pixel size P _V.

In this case, the normalized image setting unit 12, as represented by the formula (2) below, based on the value obtained by dividing the horizontal distance R _H at the pixel spacing P, and the number of pixels M between elements lens L _P calculate. Then, the normalized image setting unit 12 outputs these modeling parameters (F, R _H , R _V , P, P _H , P _V ) to the visual effect adding unit 13.
In formula (2), “<a>” means rounding off a to a natural number. _{That, "<R H / P>} " is a number of pixels between the element lenses _{L P} in FIG.

As shown in FIG. 5, the visual effect adding unit 13 adds a visual effect, and includes a normalized image pixel position calculating unit 131, a lens center position calculating unit 132, a function setting unit 133, and a visual effect image. Generating means 134.
Further, the visual effect adding means 13 receives modeled parameters (F, R _H , R _V , P, P _H , P _V ) from the normalized image setting means 12. Further, the visual effect adding means 13 receives, for example, a visual effect parameter, a three-dimensional shape model obj (FIG. 8), and a constant N, which will be described later, from the user of the display image generating device 1.

Normalized image pixel position calculating means 131, the pixel pitch P of the image display panel 31, based on the horizontal pixel size P _H and a vertical pixel size P _V, and calculates the pixel positions of the normalized image I'.

Here, using equation (1), the (world coordinates) of the normalized image I ′ can be expressed by the following equation (3). Accordingly, the normalized image pixel position calculation unit 131 divides the pixel interval P of the image display panel 31 by the horizontal pixel size P _H and the vertical pixel size P _V as represented by the following expression (4). Thus, the pixel coordinates of the normalized image I ′ are calculated. That is, the pixel coordinates of the display image I can be converted into the pixel positions of the normalized image I ′ using the equation (4).
In equation (4), (i ′, j ′) represents the pixel coordinates of the normalized image I ′.

Then, the normalized image pixel position calculation unit 131 calculates the lens center position using the modeling parameters (F, R _H , R _V , P, P _H , P _V ) and the calculated pixel coordinates of the normalized image I ′. Output to the means 132.

The lens center position calculating unit 132 calculates the modeling parameters (F, R _H , R _V , P, P _H , P _V ) from the normalized image pixel position calculating unit 131 and the calculated pixel coordinates of the normalized image I ′. Is entered. Then, the lens center position calculating means 132 using Equation (5) below, the center position H (n, m) of the m-th element lens L _P at the n-th in the horizontal direction in the vertical direction is calculated.

Thereafter, the lens center position calculation unit 132 uses the pixel coordinates of the normalized image I ′, the center position H, and the modeling parameters (F, R _H , R _V , P, P _H , P _V ) as the visual effect image. The data is output to the generation unit 134.

  The function setting unit 133 sets a function F (V, T, S) indicating the contents of the visual effect. As shown in FIG. 5, the viewpoint setting unit 133A, the time setting unit 133B, and the region setting unit 133C.

The function F (V, T, S) is a function of the viewpoint position and the subject state in the virtual three-dimensional space, and a display related to the viewpoint position V as a variable representing the viewpoint position V and a variable representing the subject state. Time T and scene identifier S are included.
The subject state is a state where a three-dimensional shape model is displayed. For example, the subject state affects the state in which the 3D shape model is displayed, such as the time or scene at which the 3D shape model obj is displayed.

The viewpoint position V represents a position where the observer A observes the three-dimensional shape model obj, and is the same position as the virtual camera Vc (FIG. 10) arranged in the virtual three-dimensional space.
The display time T represents the display time of the three-dimensional shape model obj that changes with time (animation).
The scene identifier S is information for identifying a scene displaying the three-dimensional shape model obj and a scene displaying additional information described later.

  The three-dimensional shape model obj is a three-dimensional model representing the shape of the subject U, and can be generated by, for example, computer graphics (CG) or a range sensor. The three-dimensional shape model obj is arranged in the virtual space.

The visual effect parameter is a parameter necessary for setting the function F (V, T, S), and includes, for example, the display range and display time of the three-dimensional shape model obj, and the additional information pixel region.
The display range of the 3D shape model obj is information representing the range in which the 3D shape model obj is displayed on the stereoscopic display device 3. For example, the display range of the three-dimensional shape model obj represents a rotation angle when the three-dimensional shape model obj wraps around from the front side to the back side (maximum 360 ° = 2π).
The display time of the three-dimensional shape model obj is information representing from the start to the end of the display time T.
The additional information is additional information other than the three-dimensional shape model obj.
The additional information pixel area is information representing an area for displaying additional information on the display image I.

<First example of visual effects: all-round display of 3D shape model>
With reference to FIGS. 6 to 8, as a first example of the visual effect, a description will be given of the all-around display of the three-dimensional shape model obj (see FIG. 5 as appropriate).

In Figure 6, the integer x _u, y _v is in the local coordinate system of each element image G whose origin is the center position H of the element lenses L _P, represents a pixel position in the horizontal and vertical directions.
Further, the distance between the lens array L _A and the projection plane and L. The projection plane is a plane on which shooting planes (that is, viewpoint positions) of the virtual camera are gathered.

In FIG. 6, the triangle R1 has a vertex at the center position H, the pixel position (x _u , y _v ), and the position corresponding to the center position H in the normalized image I ′. Furthermore, the triangle R2 has a vertex at the center position H, the viewpoint position V (x ′, y ′, z ′), and a position corresponding to the center position H on the projection plane.
In FIG. 6, the triangles R1 and R2 are hatched.

As shown in FIG. 6, since L: F = x ′: x _u is established due to the similarity of the triangles R1 and R2, from the pixel position (x _u , y _v ) using the following equation (6). The viewpoint position V (x ′, y ′, z ′) can be obtained.

Consider a case where the viewpoint position V (x ′, y ′, z ′) is arranged in a three-dimensional space with the origin of the three-dimensional shape model obj (“●” in FIG. 7) as shown in FIG. In this case, in the xz plane, the angle θ1 between the viewpoint position V (x ′, y ′, z ′) and the z axis is ωx _u / M. Further, the angle θ2 between the viewpoint position V (x ′, y ′, z ′) and the xz plane is ωy _v / M. Here, ω represents the display range of the three-dimensional shape model obj. For example, when the display range is ω = 2π, the entire circumference of the three-dimensional shape model obj is displayed.

For a triangle on the xz plane, the side length L1 is L′ cos (ωy _v / M), and the side length L2 is L′ cos (ωx _u / M) cos (ωy _v / M). . Note that the distance between the three-dimensional shape model obj and the viewpoint position V (x ′, y ′, z ′) is L ′.

  As described above, the formula (6) can be expressed by the following formula (7). That is, the viewpoint setting unit 133A uses the expression (7) in the function F (V, T, S) to display the viewpoint position V (x ′, y ′, z ′) and the display range ω of the three-dimensional shape model obj. Can be related. Further, the viewpoint setting unit 133A may convert the viewpoint position V (x ′, y ′, z ′) to the viewpoint position V (−x ′, −y ′, z ′) in order to reverse the line-of-sight direction. .

  Here, consider an example of a triangular prism-shaped three-dimensional shape model obj. As shown in FIG. 8A, in the conventional IP system, since the three-dimensional shape model obj is faithfully displayed within the range of the viewing angle θ of about 20 degrees, the back side of the three-dimensional shape model obj cannot be displayed. On the other hand, as shown in FIG. 8B, the display image generating apparatus 1 displays the back side of the three-dimensional shape model obj in order to relate the entire circumference of the three-dimensional shape model obj within the range of the viewing angle θ. Can do.

<Second example of visual effects: animation display>
With reference to FIG. 9, an animation display of the three-dimensional shape model obj will be described as a second example of the visual effect (see FIG. 5 as appropriate).

  In the function F (V, T, S), the time setting unit 133B is configured to move the three-dimensional shape model obj at the display time T away from the center of the image display panel 31 as the display time T moves away from the predetermined reference time. It is related to V (x, y, z). That is, the time setting unit 133B represents a temporal change of the three-dimensional shape model obj by a spatial change of the viewpoint position V (x, y, z).

  As shown in FIG. 9, consider a case where a rising balloon is used as a three-dimensional shape model obj and an animation is displayed with nine frame images. The reference time is the display time T of the frame image located in the middle among all the frame images. That is, the reference time is the display time T of the fifth frame image.

  In this case, the time setting unit 133 </ b> B relates the frame image at the display time T- 4 at the head to the left viewpoint position V (x, y, z) farthest from the center of the image display panel 31. The time setting unit 133B relates the viewpoint position V (x, y, z) from the left to the center in the order of the frame images at the display times T-3, T-2, and T-1. The time setting unit 133B associates the frame image at the display time T with the central viewpoint position V (x, y, z) of the image display panel 31. Further, the time setting means 133B relates the viewpoint position V (x, y, z) that is located to the right from the center in the order of the frame images at the display times T + 1, T + 2, and T + 3. Thereafter, the time setting unit 133B relates the frame image at the display time T + 4 at the end to the right viewpoint position V (x, y, z) farthest from the center of the image display panel 31.

Also, consider a case where 20 frame images are allocated within a range of a viewing angle θ of about 20 degrees. In this case, the time setting means 133B can be related using the following formula (8). This T ′ represents the local time (local time T _local ) of the three-dimensional shape model obj to be displayed at each viewpoint position V (x, y, z).

In a general stereoscopic video (still image), T ′ = T, and a three-dimensional shape model obj at the same time is displayed at any viewpoint position V (x, y, z). Further, in the case of a general stereoscopic video (moving image), 3 at the global time T _global (time determined by the moving image display speed, frame number or time code) regardless of the viewpoint position V (x, y, z). A dimensional shape model obj is displayed.

Here, when the time change designed by the viewpoint position V (x, y, z) is _Xu , the equation (8) can be expressed by the following equation (9).
In the example of FIG. 9, the time change X _u is in the range of −4 ≦ X _u ≦ 4.

<Third example of visual effects: additional information>
Hereinafter, display of additional information will be described as a third example of visual effects (see FIG. 5 as appropriate).
The area setting unit 133C relates the additional information pixel area and the viewpoint position V (x, y, z) in the function F (V, T, S).

  Specifically, the region setting means 133C displays the scene sa of the three-dimensional shape model obj when the viewpoint position V (x, y, z) is included in the region Sa as in the following formula (10). As described above, the region Sa and the viewpoint position V (x, y, z) are related to each other. In addition, when the viewpoint position V (x, y, z) is included in the area Sb as in Expression (10), the area setting unit 133C displays the area Sb and the viewpoint position so as to display the additional information scene sb. Associate V (x, y, z). That is, Expression (10) defines a three-dimensional shape model pixel area Sa that displays a three-dimensional shape model obj and an additional information pixel area Sb that displays additional information.

Thereafter, the function setting unit 133 outputs the set function F (V, T, S) to the visual effect image generation unit 134 and the additional information insertion unit 16 (FIG. 2).
When the function F (V, T, S) = 1, it indicates that the three-dimensional shape model obj is faithfully displayed as in the conventional IP method.

  The visual effect image generation unit 134 obtains the viewpoint position based on the function F (V, T, S) input from the function setting unit 133, and photographs the three-dimensional shape model obj with the virtual camera at the obtained viewpoint position. Thus, a visual effect image is generated.

Here, the visual effect image generation means 134 receives the three-dimensional shape model obj and the constant N.
Constant N is a value indicating a sampling number of element lenses L _P constant times to have been set in advance.

<Shooting with a virtual camera>
Hereinafter, photographing with the virtual camera Vc will be described in detail with reference to FIGS. 10 to 12 (see FIG. 5 as appropriate).
In FIG. 10, only a partial surface of the three-dimensional shape model (display target) obj is shown for the sake of simplicity.
In FIG. 11, “●” and “△” indicate sample points, “●” is a sample point at the center position H, and “Δ” is an increased sample point other than “●”. is there. In FIG. 11, “■” is a pixel to be calculated that has the same relative position with respect to the center position H in each element image G. That is, the visual effect image generation unit 134 can represent the pixel position of each element image G by local coordinates (u, v) with the center position H as the origin.
Further, in FIG. 11 and FIG. 12, the number of pixels the virtual camera, the lens array L _A "●" are and "△" the sample points and the same number of combined. Further, in FIG. 12, each of the light rays is indicated by an arrow.

  First, the visual effect image generation unit 134 arranges the virtual camera Vc at the viewpoint position obtained by the function F (V, T, S). For example, in the case of the first example, the visual effect image generation unit 134 arranges the virtual camera Vc at a viewpoint position where the entire circumference of the three-dimensional shape model obj can be photographed. Further, for example, in the case of the second example, the visual effect image generation unit 134 arranges virtual cameras at different viewpoint positions according to time times T-4 to T + 4 of the three-dimensional shape model obj.

Also, the visual effect image generating means 134, as shown in FIG. 10, around the center position H present in the reverse direction r _z unit vector, with angle of view including the central position of the entire lens array L _A, virtual The arranged stereoscopic display device 3 is virtually photographed. In other words, the visual effect image generating means 134, as the center of the reverse r _z a visual effect image the position of the center position H in the presence of a unit vector, with each pixel position of the element image G and the optical principal point of the virtual camera Vc A visual effect image is generated by projecting (orthographic projection) the color of the surface of the three-dimensional shape model obj present on the straight line connecting to the virtual plane (the focal position of the virtual camera Vc).

At this time, as shown in FIGS. 11 and 12, the visual effect image generation unit 134 collectively samples rays that pass through the center position H in photographing by the virtual camera Vc. Here, the visual effect image generation unit 134 samples the number of light rays obtained by multiplying the number of the center positions H by N in order to apply the first filter described later (for example, N = 4). Then, the visual effect image generation unit 134 acquires a light beam corresponding to the pixel located at the local coordinates (u, v) of each element image G. Thus, in the visual effect image, the spacing of the element lenses L _P is to meet the natural number times the pixel size of the elemental image G, the direction of the light beam emitted from each element lens L _P is a finite It is possible to collect rays in the same direction. That is, the visual effect image generation unit 134 can represent the emitted light group by a finite number of visual effect images.

That is, in the element lens L _P with (n, m) = (0, 0), if the world coordinates of a pixel (u, v) in the element image G are (X _P , Y _P , −F), the virtual The matrix R indicating the attitude of the camera Vc can be expressed by the following equation (11). Therefore, the center position H can be projected (orthographically projected) on the image plane of the visual effect image using the following formula (11).

In this case, assuming that the three-dimensional position of the center position H is (X _R , Y _R , 0), the image coordinates (k, l) of the point projected on the visual effect image are expressed by the following equation (12) and It is calculated | required using Formula (13).

  Here, (W ′, H ′) is the vertical and horizontal lengths of the visual effect image, and (K, L) is the number of pixels of the visual effect image. In this case, when the sample points “●” and “Δ” in FIG. 11 are projected onto the visual effect image, the projected points are as shown in FIG.

As described above, the visual effect image generating unit 134, using Equation (12) and (13), projecting the arbitrary point on the lens array L _A plane virtual camera Vc, visual effect is added A visual effect image can be generated. Thereafter, the visual effect image generation unit 134 outputs the generated visual effect image to the normalized image generation unit 14 (FIG. 2).

Returning to FIG. 2, the description of the configuration of the display image generation device 1 will be continued.
The normalized image generating means 14 receives the visual effect image from the visual effect adding means 13 (visual effect image generating means 134), and converts the pixel values of the pixels of the visual effect image into the normalized image I by a predetermined coordinate conversion formula. The normalized image I ′ is generated by allocating to the pixel “′”.

Specifically, the normalized image generating means 14 assigns the sample point (“●” in FIG. 12) of the visual effect image to the calculation target pixel (“■” in FIG. 11) of the normalized image I ′. At this time, the normalized image generating means 14 needs to obtain pixel coordinates on the visual effect image at the center position H whose position is represented by (n, m). Therefore, the normalized image generation unit 14, the physical location H (n, m) on the lens array L _A plane by the equation (5) is obtained using equation (12) and (13) Obtain the image coordinates of the visual effect image. Further, the pixel coordinates of the normalized image I ′ corresponding to these coordinates are expressed by the following formula (14). In this case, the first term on the right side of Equation (14) indicates the position of the center position H.

Thereafter, in the display image generation device 1, each pixel of the element image G is designated as local coordinates (u, v), and the imaging processing with the virtual camera by the visual effect addition unit 13 and the normalized image generation unit 14 are specified. The pixel value assignment process is performed on all the pixels of the element image G. Therefore, in the display image generating apparatus 1, it is only necessary to repeat these processes as many times as the number of pixels of the element image G, and it is not necessary to repeat the processes for all the pixels existing in the image as in the related art.
In this way, the normalized image generation unit 14 can obtain all pixel values of the normalized image I ′. Thereafter, the normalized image generation unit 14 outputs the generated normalized image I ′ to the display image generation unit 15.

<Generation of display image>
Hereinafter, generation of the display image I will be described with reference to FIG. 13 (see FIG. 2 as appropriate).
The display image generating means 15 receives the normalized image I ′ from the normalized image generating means 14 and generates the display image I from the normalized image I ′.

  The display image generation means 15 performs low-pass filter processing on the normalized image I ′ because the normalized image I ′ has M times the vertical and horizontal pixels of the display image I displayed on the image display panel 31. . Specifically, as shown in FIG. 13, the display image generation unit 15 includes the pixel value of one pixel of the display image I in a region corresponding to one pixel of the display image I as low-pass filter processing. The average value of all the pixels of the normalized image I ′ is processed. Further, the display image generation unit 15 may obtain a weighted addition value or a median value instead of obtaining an average value as low-pass filter processing.

  That is, since the display image generation unit 15 can perform coordinate conversion by the above-described equation (4), the display image I is generated by performing the low-pass filter processing represented by the following equation (15). Then, the display image generation unit 15 outputs the generated display image I to the additional information insertion unit 16.

In Expression (15), I ′ (i ′ + s, j ′ + t) represents the pixel value of the pixel (i ′ + s, j ′ + t) of the normalized image I ′, and I (i, j) Indicates the pixel value of the pixel (i, j) of the display image I. D is the number of I (i ′ + s, j ′ + t) that satisfies the condition of Σ, and is the number of pixels that contributed to the sum. Moreover, s and t is an integer from 0 to half the number of pixels between the element lenses L _P.

The additional information inserting unit 16 inserts additional information into the additional information pixel region of the display image I input from the display image generating unit 15 based on the function F (V, T, S).
Here, the additional information inserting unit 16 receives the function F (V, T, S) from the visual effect adding unit 13 (function setting unit 133) and the display image I from the display image generating unit 15. The additional information insertion unit 16 receives additional information from, for example, a user of the display image generation device 1.

For example, as shown in FIG. 14A, the three-dimensional shape model obj is a 3D character. For example, as illustrated in FIG. 14B, the additional information is a two-dimensional image of characters “Hello!”. This additional information is composed of element lenses L _P the same number of pixels (e.g., horizontal 73 × vertical 48 pixels).

For example, as shown in FIG. 14 (c), the function F (V, T, S) , among areas included in the element lenses L _P each element image G included in the display image I, 1 pixel region Is defined as the additional information pixel region Sb. Further, for example, the function F (V, T, S) in the display image among the regions included in the element lenses L _P each element image G that constitutes the I, additional information pixel region other than the pixel area Sb is 3-dimensional shape It is defined as a model pixel area Sa.

As shown in FIGS. 15A and 15B, attention is paid to the upper left portion of 'H' of the additional information. In this case, additional information insertion means 16, the number of pixels additional information for as many as the element lenses L _P, as shown in FIG. 15 (c), inserting the additional information one for each element image G. For example, the additional information insertion unit 16 inserts the pixel K1 of the second additional information from the upper left column into the region Sb1 of the element image G at the position corresponding to the pixel K1 (second from the upper left column). Similarly, the additional information inserting unit 16 inserts the pixels K2 to K5 of the additional information into the regions Sb2 to Sb5 of the element image G at positions corresponding to the pixels K2 to K5.
Thereafter, the additional information insertion unit 16 outputs the display image I in which the additional information is inserted to the stereoscopic display device 3 (FIG. 1).

  Since the display image I includes the additional information and the three-dimensional shape model obj, different images may be incident on the eyes of the observer A. In this case, it is preferable that the additional information and the three-dimensional shape model obj do not accompany a somewhat similar color or a large depth change. Thereby, it is possible to suppress a shock that the additional information and the three-dimensional shape model obj are switched.

[Operation of Display Image Generation Device]
The operation of the display image generation device 1 in FIG. 2 will be described with reference to FIG.
The display image generating apparatus 1 sets a normalized image based on the modeling parameter by the normalized image setting unit 12 (step S1).

The display image generation apparatus 1 performs visual effect image generation processing by the visual effect adding means 13 (step S2). Details of this visual effect image generation processing will be described later.
The display image generation device 1 generates a normalized image by assigning the pixel value of the pixel of the visual effect image generated in step S2 to the pixel of the normalized image by the coordinate conversion formula by the normalized image generation unit 14. (Step S3).

The display image generation device 1 generates a display image from the normalized image generated in step S3 by the display image generation means 15 (step S4).
The display image generating apparatus 1 inserts additional information into the additional information pixel area of the display image generated in step S4 based on the function by the additional information inserting unit 16 (step S5).

[Visual effect image generation processing]
The details of the visual effect image generation process (step S2) of FIG. 16 will be described with reference to FIG. 17 (see FIG. 5 as appropriate).

The visual effect adding unit 13 calculates the pixel position of the normalized image based on the pixel interval, the horizontal pixel size, and the vertical pixel size of the image display panel 31 by the normalized image pixel position calculating unit 131 (step S21).
Visual effect adding means 13, the lens center position calculating means 132, by using the equation (5), calculates the center position of the element lenses _{L P} (step S22).

  The visual effect adding unit 13 sets a function by the function setting unit 133. Specifically, the viewpoint setting unit 133A relates the viewpoint position and the display range of the three-dimensional shape model using the above-described equation (6). Further, the time setting unit 133 </ b> B associates the three-dimensional shape model at the display time with the viewpoint position far from the center of the image display panel 31 as the display time is further away from the reference time. Further, the region setting unit 133C associates the additional information pixel region with the viewpoint position using the above-described equation (10) (step S23).

  The visual effect adding means 13 obtains the viewpoint position by the visual effect image generating means 134 based on the function set in step S23, and images the three-dimensional shape model with the virtual camera at the obtained viewpoint position. In this way, the visual effect image generation unit 134 generates a visual effect image (step S24).

  As described above, the display image generation device 1 according to the embodiment of the present invention does not directly correspond the displayed three-dimensional shape model to the viewpoint position, and does not change between the viewpoint position and the state and time change of the three-dimensional shape model. A variety of visual effects can be added by interposing a function in. Thereby, the display image generating device 1 displays the 3D shape model beyond the limitation of the viewing angle of the stereoscopic display device 3 by moving the viewpoint, displays the 3D shape model in an animation, Additional information different from the shape model can be displayed.

(Modification)
The display image generating apparatus 1 is not limited to the above-described embodiments, and can be modified without departing from the spirit thereof.

  In the above-described embodiment, the display image generation device 1 adds three types of visual effects such as displaying a three-dimensional shape model exceeding the viewing angle limit, displaying an animation of the three-dimensional shape model, and displaying additional information. However, it is not limited to this. Here, the display image generation device 1 may add only one or more of these three types of visual effects.

  As a first example of the visual effect, the entire circumference of the three-dimensional shape model is displayed. However, the present invention is not limited to this. For example, the display image generation apparatus 1 may set the display range ω = π in Expression (6) so as to display a half circumference of the three-dimensional shape model.

Although the additional information image area Sb has been described as an area for one pixel, the present invention is not limited to this. Additional information image area Sb, among areas included in the element lenses L _P each element image G included in the display image I, may be at less than half.

  In the above-described embodiment, the display image generation device 1 has been described as independent hardware, but the present invention is not limited to this. For example, the display image generation apparatus 1 can also be realized by a display image generation program that causes hardware resources such as a CPU, a memory, and a hard disk included in a computer to operate cooperatively as the above-described units. This program may be distributed through a communication line, or may be distributed by writing in a recording medium such as a CD-ROM or a flash memory.

DESCRIPTION OF SYMBOLS 1 Display image generation apparatus 2 Stereoscopic imaging apparatus 3 Stereoscopic display apparatus 11 Modeling parameter input means 12 Normalized image setting means 13 Visual effect addition means 14 Normalized image generation means 15 Display image generation means 16 Additional information insertion means 21 High-definition camera 31 Image display panel (image display means)
131 normalized image pixel position calculating means 132 lens center position calculating means 133 function setting unit 133A viewpoint setting unit 133B time setting means 133C area setting unit 134 visual effect image generating unit _{L A} lens array _{L P} element lenses

Claims (5)

A display image generation device that generates a display image with a visual effect added thereto for display on a stereoscopic display device including a lens array in which element lenses are two-dimensionally arranged and an image display means,
A normalized image setting means for setting a normalized image having a pixel size that is a natural number with respect to the interval between the element lenses;
Normalized image pixel position calculating means for calculating a pixel position of the normalized image based on a pixel interval of the image display means and a pixel size of the normalized image;
Based on a visual effect parameter representing at least one of a display range or a display time of a three-dimensional shape model representing a subject, the viewpoint position of the virtual camera related to the display range of the three-dimensional shape model, and the viewpoint position Function setting means for setting a function with the subject state representing the display time of the related three-dimensional shape model;
A visual effect image to which the visual effect is added by obtaining the viewpoint position of the virtual camera based on the function set by the function setting means and photographing the three-dimensional shape model with the virtual camera at the obtained viewpoint position Visual effect image generating means for generating
A normalized image generating means for generating the normalized image by assigning a pixel value of the pixel of the visual effect image to the pixel of the normalized image by a predetermined coordinate transformation formula;
Display image generating means for generating the display image from the normalized image generated by the normalized image generating means;
A display image generating apparatus comprising: In the function, the function setting unit includes a distance L ′ from the three-dimensional shape model to the viewpoint position of the virtual camera, the number M of pixels between the element lenses, and local coordinates with the center of the element lens as an origin. Display of the viewpoint position (x ′, y ′, z ′) of the virtual camera and the three-dimensional shape model using the following formula (7) including integers x _u , y _v representing pixel positions in the system A viewpoint setting means for relating the range ω,
The visual effect image generation unit generates the visual effect image by allocating different display ranges of the three-dimensional shape model for each viewpoint position of the virtual camera based on the function related to the viewpoint setting unit. The display image generation apparatus according to claim 1, wherein the display image generation apparatus is a display image generation apparatus.

The function setting means relates to the viewpoint position of the virtual camera farther from the center of the image display means, as the display time moves away from a preset reference time in the function. Time setting means for attaching,
The visual effect image generating unit generates the visual effect image by assigning the three-dimensional shape model having a different display time for each viewpoint position of the virtual camera based on a function related to the time setting unit. The display image generation apparatus according to claim 1 or 2, wherein the display image generation apparatus is a display image generation apparatus. The function setting means includes area setting means for associating an additional information pixel area for displaying additional information and a viewpoint position of the virtual camera in the function,
The apparatus further comprises additional information insertion means for inserting the input additional information into the additional information pixel area of the display image based on a function to which the additional information is input and the area setting means is related. The display image generation device according to any one of claims 1 to 3.   The visual effect addition program for functioning a computer as a display image production | generation apparatus as described in any one of Claims 1-4.

Images